bayesian methods for machine learning

Bayesian Methods for Machine LearningZoubin Ghahramani * Tutorial Notes Now Available Here * Topic Many topics in Machine Learning e.g.. kernel methods, clustering, semi-supervised learn

Bayesian Methods for Machine Learning

Zoubin Ghahramani

* Tutorial Notes Now Available Here *

Topic

Many topics in Machine Learning (e.g kernel methods, clustering, semi-supervised learning, feature selection, active learning, reinforcement learning) can be addressed within the framework of Bayesian statistics While the proportion of work in machine learning based on statistical modelling has grown over the past few years, there remains a good degree of skepticism with respect to taking a fully Bayesian approach This tutorial aims at introducing fundamantal topics in Bayesian statistics as they apply to machine learning problems, and addressing some misconceptions about Bayesian approaches The tutorial will also attempt to present a balanced view of the limitations of Bayesian approaches Finally the tutorial will delve into some of the practical issues in current Bayesian machine learning including the role of approximation algorithms, sampling methods, and nonparameterics

Intended audience

The tutorial is intended for the broad ICML community Little prior knowledge

is assumed (other than basic probability theory) The participants will hopefully get both a big picture of Bayesian approaches to machine learning and some insight into specific state-of-the-art methods

Detailed Outline

[total 180 minutes, outline subject to revision]

1. Three canonical problems [10 minutes]

o Linear Classification

o Coin Toss

o Clustering with Gaussian Mixtures

2. Foundations [30 minutes]

o Representing beliefs and the Cox Axioms

o Dutch Book Theorem

o Asymptotic Convergence and Consensus

o Occam's Razor

o Priors: Objective, Subjective, Hierarchical and Empirical Bayes

o Exponential Family and Conjugate Priors

o How to choose priors?

3. Intractability [10 minutes]

o Bayesian inference in Gaussian mixtures and linear classifiers

o Hidden variables, parameters and partition functions

4. Approximation Tools [40 minutes]

o BIC

o Laplace Approximation

o Variational Approximations

o MCMC

o Exact Sampling

break

5. Feature Selection, Model Selection and Bayesian Methods [20 minutes]

o Do we need to select features?

o Automatic Relevance Determination

o Model selection criteria and model averaging

6. Bayesian Discriminative Modelling [20 minutes]

o Myth: Bayesian methods = Generative models

o Bayes Point Machines vs Support Vector Machines

o Bayesian Neural Networks

7. From Parametric to Nonparametric Bayes [20 minutes]

o Gaussian Processes

o Dirichlet Processes and Infinite Mixtures

o Other non-parametric Bayesian models

8. Further Topics [15 minutes]

o Bayesian Active Learning and Bayesian Decision Theory

o Bayesian Semi-supervised Learning

o Reconciling Bayesian and Frequentist Views

9. Open Discussion of Limitations and Criticisms [10 minutes]

o Philosophical

o Practical

o Computational

10. Other Questions from the Audience

Format

The format will be data-projected slides; I will also occasionally use the whiteboard and have some simple Matlab demos to illustrate some ideas However, the focus won't be on algorithms but rather on concepts

Zoubin Ghahramani is a Reader in Machine Learning at the Gatsby Unit in London, and an Associate Research Professor at CALD at CMU He has given tutorials at NIPS, ICANN, the Machine Learning Summer School in Canberra, and various other summer schools He is interested in Bayesian machine learning, computational approaches to sensorimotor control, and applications of machine learning to bioinformatics

Zoubin Ghahramani

Gatsby Computational Neuroscience Unit

University College London

17 Queen Square, Room 403

London WC1N 3AR

United Kingdom

Tel +44 (0)20 7679 1199

Fax +44 (0)20 7679 1173

Email zoubin "AT" gatsby.ucl.ac.uk

http://www.gatsby.ucl.ac.uk/~zoubin