The Computer Journal presents: Hedging Predictions in Machine Learning by Alexander Gammerman and Vladimir Vovk.

The lecture took place in the BCS London office, Southampton Street, on the 12th June 2006. It was followed by a debate and Q&A from the floor.

Discussants included Alexey Chervonenkis (Moscow), Vladimir Vapnik (Columbia), Xiaohui Liu (Brunel) and Sally Mcclean (Ulster).

The Talk

A new machine learning technique, based on transductive (as opposed to inductive) inference, makes it possible to complement many prediction algorithms with quantitative measures of their accuracy and reliability.

The technique, called conformal prediction, has already been applied in a number of areas including clinical diagnosis, bioinformatics, financial modelling, oil exploration and ecology. The lecture described this approach to machine learning.

Abstract

A new machine learning technique, called conformal prediction, makes it possible to complement many modern prediction algorithms with quantitative measures of their accuracy and reliability.

The fundamental idea used is that of transductive (as opposed to inductive) inference, where, instead of estimating functions over the whole object space, predictions for new objects are made by analysing how well possible future observations conform to known data.

The main feature of conformal predictors is that they allow us to control (up to statistical fluctuations) the number of errors made in classification and regression problems. 

Their advantages include the following:

• as they can be used on top of many state-of-the-art learning algorithms, they can deal with learning problems involving huge numbers of attributes;
• unlike many conventional statistical techniques they do not make any additional assumptions about the data beyond the standard assumption of randomness (the examples are generated independently from the same probability distribution);
• their measures of accuracy and reliability can be tailored to the individual objects to be predicted;
• they can be used in both "on-line" and "batch" modes, as well as in several intermediate modes.

This new approach in machine learning has already been applied in a number of areas, including clinical diagnosis, bioinformatics, financial modelling, energy supply, oil exploration and ecology, and no doubt will find new areas of application.

The Speakers

Alexander Gammerman is Professor of Computer Science at Royal Holloway, University of London. His current research interest lies in field of machine learning, algorithmic randomness and intelligent data analysis. Applications of these techniques include medical diagnosis, forensic science, proteomics, environment and finance.

Vladimir Vovk is Professor of Computer Science at Royal Holloway, University of London. His research interests include predictive and Kolmogorov complexity, algorithmic randomness, machine learning, and the foundations of probability.

The Discussion

The vote of thanks and first contribution was by Glenn Shafer (Rutgers University, New Jersey, USA). He traced historical antecedents in this work, in particular in statistics. 

In discussing how machine learning and statisics share a common perspective, he  illustrated how "we [in joint work with Gammerman and Vovk] have stayed true to Bernoulli in [...] what statistics is about". 

He then commented on statistical education, encompassing machine learning and/or statistics. 

In a written contribution, Philip Long from Google Inc. commented on the importance of conformal prediction.

Xiaohui Liu, Brunel University, discussed applications to  high-dimensional DNA microarray data analysis. 

Harris Papadoupoulos, of the Frederick Institute of Technology, Nicosia, Cyprus, spoke of the powerfulness of applying inductive conformal prediction. 

In a written contribution, Sally McClean, University of Ulster, asked for elaboration on the measure of data "strangeness". 

Drago Indjic of the London Business School asked if confidence and credibility regression estimates has been applied in statistical experiment design.  

Glenn Hawe (Vector Fields Ltd., Oxford and School of Electronic and Computer Science, Southampton) linked machine learning to optimization. He related inductive and transductive inference to one-stage and two-stage optimization algorithms. 

Alexey Chervonenkis (Russian Academy of Sciences, Moscow) spoke of the general context. He flagged the benefits of this approach to assessing confidence in data analysis and raised the question as to  how optimal the approach was.  

Zhiyuan Luo and Tony Bellotti (Royal Holloway, University of London) also listed areas where these new results are of major importance.  

In a written contribution David Bell (Queen's University Belfast) sketched out how transductive and inductive learning could be combined to handle inference and prediction in robotics and video mining.

In a written contribution, David Dowe, Monash University, Australia, linked the work presented with MML and MDL (minimum message length and minimum description length). 

Also in a written contribution, Vladimir Vapnik linked the work presented with VC (Vapnik-Chervonenkis) theory for constructing predictive models, and how this in turn was based on work by Kolmogorov. He raised the question as to how objective verification is related to transductive inference.

Finally, Alan Hutchinson, King's College London, raised a number of issues related to prediction and the reliability of prediction. 

The paper and the discussion from this lecture was published in the Computer Journal.

Computer Journal Lectures