FISITA nof2008 08 118 v2

Pedestrian crossingbridgePlaying on road Duringcrossinganotherplaces 2000 Car drivers and passengers Motorcycle riders Small-scale motorcycle riders Bicyclists Pedestrians OncomingOperat

Pedestrian Recognition by Single Camera for Driver Assistance

Hirotomo Muroi, Ikuko Shimizu Pongsathorn Raksincharoensak* and Masao Nagai

Faculty of Engineering Tokyo University of Agriculture and Technology

FISITA World Automotive Congress 2008

14th -18th September 2008, ICM Munich, Germany

16th Sep.2008, S-7 Advanced Safety Systems II

▶ Motivation and Objectives of the Research

▶ Recognition Algorithm of Pedestrian near Crosswalk

▶ Crosswalk Detection Algorithm

▶ Pedestrian Classification Algorithm

▶ Experimental Results

▶ Summary and Conclusion

Motivation ( Pedestrian accidents )

▶ International comparison of distribution of traffic accident

death classified by condition (2000)

Pedestrian crossingbridge

Playing

on road

Duringcrossinganotherplaces

2000

Car drivers and passengers

Motorcycle riders

Small-scale motorcycle riders Bicyclists

Pedestrians

OncomingOperating

 Pedestrian recognition near crosswalk is effective for pedestrian protection.

Statistical data of traffic accidents in Japan (2007)

State of the Art

▶ Characteristics of sensors for pedestrian recognition

▶ Current driver assistance systems for pedestrian protection

(Toyota)

Sensor Environment Information

amount Cost performance Algorithmic simplicity

Millimeter Wave

Stereo cameraMillimeter Wave radar

Headlight with Infrared beams

Toyota LEXUS LS460 Pre-Crash Safety System

Using the multipurpose single camera is still in development process.

Multiple functions of single camera

Pedestrian detection Traffic sign recognition for speed pilot

Pedestrian crosswalk Vehicle

Objectives

Development of a new recognition algorithm for detection

of pedestrians near crosswalk by on-vehicle single

camera

Strategy of Pedestrian Recognition

▶The computational results of two algorithms are integrated to detect

pedestrians :

1 Crosswalk detection

2 Pedestrian classification

Crosswalk detection

Pedestrian classification

Crosswalk Detection Using Cross Ratio

▶ Perspective projection: the relation

between the real world and its image

▶ Assumption: The long sides of the crosswalk

are almost parallel to the vehicle moving direction

Crosswalk in the camera image (2D)

Crosswalk in the real world (3D)

AB ,CD (Thickness) [m] AC (Width) [m] Crosswalk

(Japan) 0.45 ~ 0.50 0.90 ~ 1.00

The cross ratio of parallel lines are

preserved under the perspective projection.

AD

BD BC

AC ABCD ] = ⋅

[

] [

]

The key idea of crosswalk detection

31 0

~ 20 0

Range of cross ratio in Japan:

Definition of cross ratio

Flow Diagram for Crosswalk Detection

Image sequence

Binarization Edge extraction

Two kinds of vertical

edge lines extraction

Flow Diagram of Crosswalk Detection

Image sequence

Binarization Edge extraction

Two kinds of vertical

edge lines extraction

Framework of Pedestrian Recognition

near the Crosswalk

▶Results of two algorithms are integrated:

▶ 1 Crosswalk detection

▶ 2 Pedestrian classification

Crosswalk detection

Pedestrian classification

Integration

Pedestrian near

a crosswalk

Crosswalk

Pedestrian Classification Algorithm

Pedestrian features

Features extraction

Training Sets

Pattern Learning

Test sets

Features

Pedestrian 　 Classification

pedestrian

Non-Images

Pedestrian features

Pattern Learning

Features extraction Classifier

Pedestrian Non- pedestrian

Features extraction

Literature Survey

Reference: S.Munder and D.M Gavrila,”An experimental Study on Pedestrian Classification”

IEEE Transactions on Pattern Analysis and Machine Intelligence,

vol.28, pp.1863-1868, 2006

Haar-like features with cascaded AdaBoost classifier

seems to be a suitable method for real-time processing.

Features extraction and learning

algorithm

Recognition accuracy Computational cost

PCA (Principal Component Analysis) + Feed-forward

Haar wavelets + SVM (Support Vector Machine) ○ △

LRFs (Local Receptive Fields) + SVM ◎ ×

LRF + K-nearest neighbor classifier △ △

Haar-like feature + A cascade of AdaBoost classifier ○ ○

Pedestrian Classification Technique

10 5

10 5 5

5

10 5

10 5 5

5

10 5

10 5 5

5

Haar-like Features

Pedestrian features

Pattern Learning

Features extraction Classifier

Pedestrian Non- pedestrian

Features extraction

▶Characteristic of Haar-like features

▶ Local feature extraction of pedestrian

(e.g hands ， legs and etc.)

▶Extraction of edge and line feature on the image

▶ Feature value is calculated by

brightness difference between “black”

and “white” rectangular regions

▶ AdaBoost (Adaptive Boost)

▶ Training by selecting small sets of features

▶ Classification by a weighted majority vote whether the interested object is

pedestrian

▶ Cascaded AdaBoost classifier

▶ Elimination of a large amount of non-pedestrian images at early stages

2 … n

Pedestrian

pedestrian

Pattern Learning

Features extraction Classifier

Pedestrian Non- pedestrian

Features extraction

This technique can reduce the processing time of pedestrian classification.

Pedestrian Recognition Algorithm

▶ To detect a pedestrian near crosswalk, the information

obtained from both algorithms are required

1 Crosswalk detection

2 Pedestrian classification

Crosswalk detection

Pedestrian classification

Integration

Pedestrian near

a crosswalk

Crosswalk

Each candidate for pedestrian region by the algorithm 2 is verified

by checking whether it is located near the crosswalk region

determined by the algorithm 1

Validation of Crosswalk Detection Algorithm

▶Offline validation of the crosswalk detection in real-world traffic

▶ There are 55 crosswalks on the specified driving route.

Experimental Results of Crosswalk Detection

Experimental validation of the pedestrian recognition algorithm

▶Pedestrian is walking on an artificial

crosswalk perpendicular to the vehicle direction

Vehicle

Open Source Computer Vision Library* was used.

*Reference : http://www.intel.com/technology/computing/opencv/index.htm

Experimental Results

▶ The pedestrian detectable range is about 25 m.

▶ Large number of false recognized pedestrian

contains many frontal images of pedestrians.

Computational time: 150ms

Examples of the recognized pedestrian near the crosswalk

Concluding Remarks

This paper describes a pedestrian recognition method

based on single visible camera image information.

▶ The existence of crosswalk is detected based on the

cross ratio of the real world crosswalk.

▶ Pedestrian is recognized by using Haar-like features

and the cascade of AdaBoost classifier.

▶ The preliminary experiments show the effectiveness of

the proposed algorithm.

Future plans

Crosswalk detection algorithm :

▶ Tuning parameters to enhance the robustness under various

conditions

▶ Using information from other sensors (speed, steering angle, etc.)

Pedestrian classification algorithm :

▶ Learning the database including the side images of pedestrians for more robust classification.

System improvement with other modules :

▶ Extraction of ROI (Region of Interests) to reduce the false

detections

▶ Optical flow (moving object detection by camera image)

▶ Sensor fusion with millimeter wave radar (high accuracy of moving object detection)

An Example of Sensor Fusion

proposed method

Thank you

for your kind

attention

Appendix

▶Statistics data about traffic accidents in Japan

Car drivers and passengers Motorcycle riders

Small-scale motorcycle riders Bicyclists

Pedestrians

Traffic accident fatalities 5,732

Conventional method for crosswalk detection

▶Points at issue

Undetection and false

detection

▶ Shadow of the building

under the sunny weather

▶ Corruption of white lines

forming a crosswalk

▶ Extraction of low intensity edge

▶ Binarization

▶ The periodicity of

a crosswalk

Experimental Results of Crosswalk Detection

Different between Single Camera and Sensor Fusion

▶Single camera

▶ Camera can verify what the interested objects are

▶ Camera is cheaper than the other sensors

▶ The distance to the interested objects can’t be measured with accuracy

▶ Computational cost is high for an amount of information obtained single camera

▶Sensor fusion (Image sensor and millimeter wave radar)

▶ Radar can directly measure the distance to the interested objects with high precision

▶ Radar information can be used for determining ROI in image

processing

Overview of Pedestrian Recognition

Pedestrian features

Features extraction

Training Sets

Pattern Learning

Test sets

Features extraction Classifier

Pedestrian

　 Classification

pedestrian

Non-Images

ROI

Method for Determining ROI

(Region Of Interest)

▶Recognition accuracy and computational cost

change by extraction accuracy of ROI

▶Single camera

– High computational cost

▶Sensor fusion (Camera + Millimeter wave radar)

Continental corporation

– Low computational cost

Pedestrian features

Features extraction Learning Pattern

Features extraction Classifier Pedestrian Non-

pedestrian ROI