boosting and tree - structured classifier

Object Tracking b Fast Re Detection From Time t to ™ Online discriminative feature selection [Collins et al 03], Ensemble tracking [Avidan 07]... Structure of this talk™ Second half Š

Boosting and Tree-structured g Classifier

Presenter: Nguyen Dang Binh

The classification speed is not just for time-efficiency but good accuracy

Object Detection

b a Cascade of Classifiers

/76 3

Pictures from Romdhani et al. ICCV01 

Object Tracking

b Fast (Re ) Detection

From  Time  t to 

™ Online discriminative feature selection [Collins et al 03], Ensemble 

tracking [Avidan 07]

Semantic Segmentation

™ Requiring pixel‐wise classification

/76 5

Structure of this talk

™ Second half

Š Unified Boosting framework

Things not covered

Introduction to Boosting Classifiers

A brief history (continued)

Bagging (Bootstrap AGGregatING)

™ Bootstrap

Š For each set randomly draw examples from the uniform

Š For each set, randomly draw examples from the uniform dist. allowing duplication and missing

In classification:   \([) = the majority class in {         ,…,         } \([) \7([)

Randomized Decision Forest [Breiman 01, Geurts et al 06]]

Randomized Tree Learning

left splitright split

™ Features IY chosen from a random feature pool I  )

h h ld h i

™ Thresholds Wchosen in range

™ Choose I and Wto maximize gain in information

/76 16

Random Forest – Summary

Š G li ti th h b i ( d l ) &

Š Generalization through  bagging (random samples) &  randomised tree learning (random features)

Š Very f ast classification

Š Very  f ast classification

™ Iteratively reweighting training samples. 

™ Higher weights to previously misclassified samples.

/76 18

1 round

2 rounds

50 rounds

In classification:   = the majority class in {         ,…,         } 

according to the weights 

AdaBoost [Freund and Schapire 04]

Existence of weak learners

™ XOR problems (Matlab demo)

/76

™ XOR problems (Matlab demo) 

21

Does AdaBoost generalize?

Multiple classifier system

Mixture of Experts [Jordan, Jacobs 94]

Gating Network

Ensemble learning: Boosting and

Robust real-time object detector

Boosting Simple Features

[Viola and Jones CVPR 01]

™ Adaboost classification

Weak  classifier

Strong  classifier

™ W k l ifi H b i lik f ti ( )

™ Weak classifiers: Haar‐basis like functions (45,396 in total)

/76 27

Boosting Simple Features

[Viola and Jones CVPR 01]

Boosting as a Tree-structured Classifier

Boosting (very shallow network)

g

B i Bagging

Tree hierarchy

Inspired by Yin, Crinimisi CVPR07

BREAK !!

Unified Boosting Framework

AnyBoost : an unified framework

[Mason et al 00]

™ Most boosting algorithms have in common that they iteratively update sample weights and select the next hypothesis based on p p g ypthe weighted samples

Tree-structured Classifiers

Multi-view and multi-category object detection

™ Images exhibit multi modality

Multiclass object detection [Torralba et

Multiclass object detection [Torralba et

al PAMI 07]

/76 41

Multiclass object detection [Torralba et

al PAMI 07]]

/76 42

ClusterBoost [Wu et al ICCV07]

ClusterBoost result

/76 44

AdaTree (Grossmann CVPRW04) Result

1% Noise in data 10% Noise in data

0.5 0.4

0.4 0.3 0.2

0.2 0.1

/76 45

Mean computation cost

0

0        90

Boosting for XOR

- MCBoost [T-K Kim, R Cipolla NIPS 08]

Face cluster 1

/76 47

K‐means  clustering

Face cluster 2 MCBoost

ICCV09 Tutorial Tae-Kyun Kim University of Cambridge

MCBoost: Multiple Strong Classifier

Toy XOR classification problem

™ Discriminative clustering positive samples

™ Discriminative clustering positive samples

™ It doesn’t partition an input space

/76 49

Toy XOR classification problem

classifier1 classifier2 classifier3

1.2 1.1

1.0 0.9

0.5 0.4

0 3

Boosting round

10       20       30

0.3 0.2

/76

™ Matlab demo

50

Pedestrian detection by MCBoost

[Wojek, Walk, Schiele et al CVPR09]

Pedestrian detection by MCBoost [Wojek, Walk, Schiele et al CVPR09]

/76 54

Speeding up

A sequential structure of varying

Classify x as ‐ and exit

Making a shallow network deep

- Super tree [Kim, Budvytis, Cipolla, 09]

Y

Y

Converting a boosting classifier to a decision tree

™ Many short paths for speeding up

™ Preserving (smooth) decision 

regions for good generalisation

/76 58

Converting a boosting classifier to a decision tree

™ Many short paths for speeding up

8 2

7

13 2

4

5 times  speed up

/76 59

Boosting

Super tree

/76 60

Boolean optimisation formulation

0 1

0 1

W3

R8 1 1 1 x

Boolean expression minimization

0 1

1 1

R4

R3

/76 62

W1W2W3 v W1W2W3 v W1W2W3 vW1W2W3 W1 v W1W2W3

Boolean optimisation formulation

Synthetic data exp1

Examples  generated 

from GMMs

/76 64

Face detection experiment

False negativer ate

Average path  length

False positive rate

False negativer ate

Average path  length

False positiv

e rate

False negative  rate

Average path  length

/76 65

ST vs Random forest

/76

™ ST is about 2 to 3 times faster than RF at similar accuracy

66

Experiments with tracking and

segmentation by ST g y

/76 67

More segmentation experiments by Boosting and RF

MCBoost building class segmentation

Building class: average class accuracy = 77% 

/76

Ground Truth Segmentation Ground Truth Segmentation

Building Non‐building Error Road Non‐road Error

global :  74.50% 

average: 

79.89%

global:  88.33%,  average:  87.26%

80.45% 

85.55 %,  average:  85.24 % 

/76 70

Boosting for Segmentation g g

Conclusions

B i Bagging

Tree hierarchy

Inspired by Yin, Crinimisi CVPR07

• Slower than RF

Stumps • Fast classification

• Theoretical backgrounds

• Slow training

/76 75

Th k

/76

Thanks

77

Boosting as a Tree- structured Classifier< /h3>

Boosting. .. class="page_container" data-page="25">

Ensemble learning: Boosting and