A robust algorithm for detection and identification of traffic signs in video data

In their system, color segmentation extracted traffic sign candidates, and the candidates were recognized by two phases: shape classification using linear SVMs and sign classification b

DUBLIN CITY UNIVERSITY SCHOOL OF ELECTRONIC ENGINEERING

A Robust Algorithm for Detection

and Identification of Traffic Signs in

Video Data Final Report

Student: Thanh Bui Minh

ID: 10212575

August 2011

MASTER OF ENGINEERING

IN ELECTRONIC SYSTEMS

Supervised by Dr Ovidiu Ghita

First, I would like to thanks my supervisor Dr Ovidiu Ghita The decision he made to be my supervisor, gave me the opportunity to do the project which I am interested in I am grateful

to his support, guidance and advices

I would also like to thanks Dr Martin Collier and Dr David Molloy for helping me to change my old project and give me a chance to work with Dr Ovidiu Ghita in this project Finally, my personal thanks are extended to my family, friends who give me encouragement, inspiration to finish this project

I hereby declare that, except where otherwise indicated, this document is entirely my own

work and has not been submitted in whole or in part to any other university

Signed: Date:

Page

Abstract—A traffic sign recognition system is presented in

this paper The system is able to recognize circular, octagonal

and triangular signs which nearly cover all important Irish and

UK traffic signs There are three main stages in this system:

color segmentation, sign detection and sign classification HSV

and HSI color spaces are used for color segmentation to extract

the candidate of traffic sign And RGB color space is used in

case of achromatic (black and white) color segmentation In the

second stage, shape analysis and the area of the inner of the

candidates are employed to identify the traffic sign Finally, the

identified sign is pre-processed and converted into attributes in

order to input to SVMs for sign classification The approach is

tested in about 650 images and the number of videos under

many different environment conditions and it shows high

robustness

Index Terms—Color Segmentation, Classification, Support

Vector Machines (SVMs), Tracking

I I NTRODUCTION

he traffic signs have primarily the role to regulate the

traffic and to provide information for drivers about

road quality, traffic restrictions, warnings, possible

directions, etc The vast majority of the road (traffic) signs

are standardized and they have distinct shapes and color

patterns to facilitate their easy identification in all traffic

conditions With the increased traffic congestion that was

witnessed over the past decade, the correct recognition of the

road signs plays an important role in preventing accidents In

particular the signs that regulate the traffic at intersections

are of utmost importance since their avoidance can result in

collisions with extreme consequences Moreover, the human

visual perception abilities depend on the individual’s

physical and mental conditions These abilities can be

affected by many factors such as tiredness and driving

tension The availability of an automatic vision-based system

that is able to provide the drivers with the approaching

(incoming) traffic signs will be a useful tool that will help the

drivers to prevent accidents, save lives and increase the

driving performance especially in situations when they are

placed in unexpected locations

The detection and recognition of traffic signs may face

some potential challenges due to the complex environment of

roads and the scenes around them The color of the sign fades

with time due to the long exposure to sunlight and the

reaction of the paint with the air [1] Visibility can be

affected by the local light variations such as shadows, clouds

and the sun Weather conditions such as fog, rain, clouds and

snow is also an obstacle to the detection of the traffic signs

[1] The presence of scene objects with similar colors as the

traffic signs such as vehicles or building can generate

additional computations to the sign detection process Signs

may be found damaged, occluded or attached together If the

image is captured from a moving car, then it usually suffers

from motion blur and car vibration

Various methods for traffic signs recognition have been proposed, some of dominant methods are covered in references [2]-[5]-[8] Escalera et al [8] detected signs by using color thresholding to segment and analyze the image The traffic images were presented as input patterns to multilayer perceptron neural networks for the classification Traffic sign detection and recognition based on Support Vector Machines (SVMs) was proposed by Maldonado- Bascó et al [2] In their system, color segmentation extracted traffic sign candidates, and the candidates were recognized

by two phases: shape classification using linear SVMs and sign classification based on Gaussian-kernel SVMs using gray sign image as input attributes Fleyeh et al [5] introduced a sign detection method based on color and shape

of traffic sign Binary sign image and moments such as Zernike or Legendre were inputted to the SVMs for classification Moreover, Fleyeh [3] presented color constancy method for color segmentation in case of poor light conditions

The paper is organized as follows The system overview is presented in Section 2 The system consists of three main stages, namely color segmentation, sign detection and sign classification which are described in detail in Section 3, Section 4 and Section 5, respectively Finally, the performance evaluation of the system is presented in Section

6 and section 7 provides concluding remarks

II S YSTEM OVERVIEW The overview of our recognition system is shown in Fig 1 The system consists of three main blocks Input image is provided to the color segmentation block The segmentation

is an important step that is applied to eliminate all background objects and irrelevant information in the image

It generates a binary image containing the road signs and any other objects which are similar to the color of the road signs Subsequently, the binary image is analyzed in the sign detection block Noise and small objects are discarded to find the possible traffic sign blobs by applying connected component labeling algorithm and size filtering Then, shape analysis and the consideration of inner sign are applied for these blobs to identify the traffic sign After the traffic sign is determined, it is scaled to the same dimension and the attribute of each traffic sign is extracted to serve as the input patterns for SVMs in classification stage The sign classification block is responsible for classifying the candidate traffic sign using the attributes of the candidate and the training database based on SVMs The training database

is obtained in the offline mode Finally, the classified result is then outputted via a graphic interface

In this work, we chose Irish and UK traffic signs as our case study The research focused on four groups of traffic sign namely prohibitory, mandatory, warning and stop & yield as shown in Fig 2 These signs can be classified according to color and shape as shown in Table 1 Some

A Robust Algorithm for Detection and

Identification of Traffic Signs in Video Data

Student: Thanh Bui-Minh; Suppervisor: Dr Ovidiu Ghita

T

other categories such as information signs are ignored,

because comparing to those categories, the four mentioned

groups of traffic signs are more important and more

challenging to be classified by computers

TABLE I

M EANING OF THE SIGNS WITH RESPECT TO COLOR AND SHAPE

Red Triangle downward Yield

Red Triangle upward Warning

Red Circle Prohibitory

III C OLOR SEGMENTATION

The main role of this computational stage is to generate the

binary image containing the road signs and objects similar to

the color of road signs as well as to eliminate background

and irrelevant objects

Hue played the central role in the color detection because

it is invariant to the variations in light conditions as its scale

invariant, shift invariant and invariant under saturation

changes [3] So, the hue is the component that is invariant to

shadow and highlights [3] Two color segmentation methods

have been developed our work which are referred to Shadow

and Highlight Color Segmentation Algorithm [3] and

modification of Escalera method [8]

In the first method, RGB image is converted to HSV

image to takes advantage of the characteristic of hue

component The thresholds of H, S, V components are used

to determine the image areas that are predominantly red and

blue Histogram analysis and an exhausted testing are carried

out to find the thresholds as presented in Table 2

TABLE II

C OLOR THRESHOLD FOR HSV METHOD

Color Hue [0:179] Saturation[0:255] Value[0:255] Red [0-12] & [145-179] [35-255] [30-250] Blue [96-128] [75-255] [65-254] The second method is a modification of Escalera’s method This approach uses the information of Hue and Saturation of HSI color space So the RGB image is converted to HSI image and similarly the thresholds are used to extract the color of interest Table 3 shows the thresholds to obtain the red and blue color using this method

TABLE III

C OLOR THRESHOLD FOR HSI METHOD

Color Hue [0:179] Saturation[0:255]

Red [0-12] & [145-179] [35-255]

Blue [96-129] [75-255]

The inner of sign has achromatic (black and white) colors Furthermore, the binary images generated by achromatic and white color segmentation are used in sign detection and sign classification stages, respectively Unfortunately, hue and saturation is not contained enough information to segment white color In this case, the decomposition of image’s achromatic helps to detect achromatic color as shown in equation 1 [2]

An example of red color segmentation is shown in Fig 3

IV S IGN DETECTION This stage plays an important role in the whole recognition system It implements connected component labeling to form candidate blobs, size filtering to discard noisy and small objects, shape and sign’s inner analysis to identify traffic signs

A Connected Component Labeling Algorithm

The objects are identified by using component-labeling algorithm that employs contour tracing technique [9] This method scans a binary image only once and traces each contour pixel no more than a constant number of times The method is proved to outperform other methods in term of computational speed, so it is effective in the development of real time applications

Fig 1 The overview of our traffic sign recognition system

Fig 3 Red color segmentation of traffic sign image Fig 2 Traffic signs of interest

B Size filtering

Size filtering is the process of selecting of labeled objects

based on the size of the object The size of the object is

calculated by the number of white pixels in the binary image

The noise objects, very small or very big traffic signs

corresponding to very close or very far signs and the

background are eliminated by using the MIN and MAX

thresholds The size filtering process improves a lot of time

in real time processing as un-important objects are not

processed in the image for traffic sign recognition

C Shape analysis and sign’s inner consideration

There are three types of traffic sign shape: octagon,

triangle and circle Three types of shape measures are used to

decide the shape of the sign They are ellipticity, triangularity

and rectangularity In our experiment, we discovered that the

octagonal shape is easily confused with the circular one at

medium and high distances, so octagonal signs are included

in circular signs class and then using the sign’s inner to

identify it

Ellipticity can be obtained by applying an affine transform

to a circle The simplest way is using the Affine Moment

moments, and µ00is the zero order central moment The

ellipticity (E) and triangularity (T) are measured using

equation (3) and (4), respectively [6]:

2

2

1 1

2 1

otherwise I

otherwise I

Perfect elliptic has ellipticity E of 1 Similarly, perfect

triangle has triangularity T of 1

The rectangularity R is measured by the calculating the

area of the region under considering to the area of its

minimum bounding rectangle (MBR)

The three shape measures assume that the object is

homogeneous (does not contain any holes) Since traffic

signs have two different colors, one for the rim and the other

for the inner, the rim color is used for the segmentation and

then the holes are filled with the same grey level In case the

object under analysis is occluded by another object, the

convex hull of the object is calculated to generate a

homogenous object Table 4 shows the value of E, T, R to

classify the shapes

In order to increase the accuracy of traffic sign detection,

the interior of sign (sign’s inner) is evaluated by calculating

the ratio of white area of the sign’s inner and the whole area

of the sign For instance, if the object has red color, circle

shape and the ratio is greater than 0.35, then the object

belongs to prohibitory sign group The thresholds to

recognize the STOP and NO ENTRY signs are smaller than

that of the prohibitory signs The same method is used to

identify mandatory and warning sign group

Since the shape measures are computed using the Affine Moment Invariant method which is invariant to general affine transformation, the approach is invariant to rotation, scaling and translation

TABLE IV SHA PE MEASURE VALUE FOR CIRCLE AND TRIANGLE SHAPE

Shape Ellipticity (E) Triangularity (T) Rectangularity (T) Circle 0.99 < E < 1.03 T > 1.43 R > 0.69 Triangle E< 0.8 0.99< T< 1.17 0.49< R< 0.7

A problem occurs in case of clustered signs, that is the groups of signs share the same pole and they are often attached or partly occluded to one another When the image

is color segmented, the signs become attached to each other and hence connected component labeling creates a single object Obviously, this new object does not belong to any of the expected sign shapes which the algorithm deals with The Hough circle transform is primarily used to solve this problem, however it has two disadvantages: only detect the circle signs and it fails to recognize the circle signs when the sign is rotated and/or damaged A more robust method is developed in this thesis The method uses the inner of the sign to identify the shape of the sign The reason is that when the traffic sign has a triangular or circle shape, the inner of the sign also has similar shape with some exceptions The advantage of this approach is that the inner of clustered signs are always detached and it is illustrated in Figure 4

V S IGN CLASSIFICATION The traffic signs are classified according to their attributes using SVMs

A Support Vector Machines (SVMs)

SVMs is a pattern classification and regression techniques based on mathematical foundations of statistical learning theory, which was first proposed by Vapnik in 1992 [5] The basic training principle of SVMs is to find an optimal hyper- plane to linearly separate two classes The optimal hyper- plane is formed in such a way to minimize the expected classification error for unseen test samples In binary classification, the training data are labeled {x i , y i }, where i = [1 n], y i Є {-1, 1}, x ∈ ℜ i d, d is the dimension of the

vector, and n is the number of training vector The

classification of a new pattern x can be obtained by solving

the decision function f(x) as shown in equation (5), where α i

are the Lagrange multipliers and b is the bias offset

Fig 4 The result of the detection of attached sign (from left to right and top

to bottom: attached sign image, red color segmentation image, sign’s inner and sign identification result)

linear function A solution is to map the input data into a

higher-dimension feature space φ( )x where the

classification can be performed by linear SVMs So the

decision function is now expressed as:

1

n

i i i i

f x α y K x x b

=

=∑ + (6)

where x is the input vector to be classified, K() is kernel

function A kernel constructs an implicit mapping from the

input space in to a feature space There are four types of

kernels: Linear, Polynomial, Radial Basis Function (RBF)

and Signmoid

For classification problems, the optimal hyper-plane could

not be able to separate the input vectors completely, so

different classification types have been proposed for SVMs

The most common types are C-support vector classification

(C-SVC) and ν-support vector classification (ν-SVC)

Appendix 3 describes more detail about Kernel and

C-SVC and ν-SVC

B Classification

When the candidate blob of sign is identified, the gray sign

image is normalized to 31x31 In order to reduce the number

of attributes and discard the effect of noise (outside the sign

but belong to the bounding box), only those pixels that must

be part of the sign (pixel of interest, PoI) are used For

example, in case of circular sign, only pixels that are inside

the inscribed circle, which belong to the normalized

bounding box, are computed to get the attributes to input to

SVMs for classification

Both training and testing are done according to the color of

each candidate region So every candidate blob is only

compared to those signs that have the same color as the blob

to increase the accuracy and reduce the complexity of the

problem

Every image in the training database is 31x31 pixels and is

invariant the in-plane transformations, that include scale,

translation and rotation In this work, we use 12 classes for

red color signs and 5 classes for blue color signs

One-versus-all SVMs strategy are used So the number of

classifiers M needed is equal to the number of classes that

belong to the case considered that is 12 and 5 classifiers for

red and blue color signs, respectively The amount of training

samples per class is 30 To search for the decision region, all

attributes of a specific class are grouped together against all

attributes corresponding to the rest of classes

The traffic signs are located in outdoor, so they are often

affected by the variation of illumination conditions

Consequently, the gray image of original sign does not

provide a robust attributes to SVMs In our work, three kinds

of the attributes of the sign are used for classification:

• Gray level attributes

• The original gray image is pre-processed by applying

histogram equalization method then the gray level

attributes are extracted

• Binary attributes: The attributes of the sign are extracted

from the binary image which is obtained by applying

white color segmentation to the original image

Figure 5 presents three different types of the attributes of

30km/h speed limit sign and some classification results are

shown in Figure 6 Appendix C-4 presents more detail about the classes for classification

VI R ESULTS More than 650 images which were captured from many different environmental conditions are used to evaluate the performance of the system The experiments are carried out for each stage of the system, color segmentation, sign detection and sign classification

A Color Segmentation

As described in Section 3, two methods for color segmentation have been developed in this thesis The first method uses three components H, S, V of HSV color space and the second method uses only H, S components of HSI color space Table 5 presents the success rate of color segmentation of signs in images taken under different light conditions and different effects of two methods The successful color segmentation means that the method generates the complete binary object of the traffic signs The overall success rate of the first method and the second methods are 91.5% and 91.8%, respectively As can be seen from the table, the second method which uses H, S components gives much better segmentation result than the first method in case of Bad lighting, High lighting and Night lighting conditions The reason is that H and S are components that are invariant to shadow and high light conditions [3] In occlusion condition when the signs are occluded by other objects, the result is not good because it highly depends on the state of occlusion For instance, the color segmentation fails when the sign is almost covered more than 20% by other objects

B Sign Detection

The success of the sign detection stage significantly depends on color segmentation, so the result of sign detection Fig 5 Three different types of the attributes of speed limit sign

Fig 6 Some results of sign classification

is evaluated based on two color segmentation methods The

accuracy of sign detection in images under different

conditions is illustrated in Table 6 The overall success rates

of two analyzed methods are 86.5% and 87.6%, respectively

In this work, a robust method has been introduced to solve

the problem of clustered signs by using the inner of the signs,

the accuracy rate of this method is 90% compared to 10%

when this method is not used Appendix C-2 shows more

results of this method

TABLE V

S UCCESS RATE OF COLOR SEGMENTATION UNDER DIFFERENT CONDITIONS

Conditions No of Signs First method

(HSV)

Second method (HSI) Clustered sign 90 95.6% 95.6%

S UCCESS RATE OF S IGN DETECTION UNDER DIFFERENT CONDITIONS

Conditions No of signs First method

(HSV)

Second method (HSI) Clustered sign 90 90% 90%

After referring from [5], [10] and carrying many

experiments, we recognize that C-SVM with Linear kernel

gives the best performance for sign classification The 31x31

pixels block of sign is used for both training and testing, so

the number of attributes per sign is 961 As described in

section 5, three types of attributes of sign are used for SVMs

classification to find the highest accuracy Eight classes of

red color signs are used to evaluate the performance of

classification stage Table 7 presents the result of

classification corresponding to three types of input attributes

TABLE VII

S UCCESS RATE OF S IGN CLASSIFICATION UNDER DIFFERENT INPUT

ATTRIBUTES USING C - SVM WITH LINEAR KERNEL

Input Attributes Training Testing Overall

Origin gray signs 100% 95.9%

Sign after applying Histogram Eq 100% 98.67%

Binary signs 100% 99.5%

As can be seen from the table, binary attribute gives the

highest accuracy rate (99.5%), followed by the attributes

after applying histogram equalization (98.67%) However, in

order to get binary attributes, the white color segmentation

method is applied to original image as shown in detection

stage, and the white color segmentation is often sensitive to

the illumination of light because it used RGB color space,

this is a disadvantage of binary attributes The attributes

extracted from sign after applying histogram equalization to

original image gives higher success rate than original gray

sign The reason is that histogram equalization increases the

local contrast of the image, especially when the data of the image is represented by close contrast values More result images are presented in Appendix D

TABLE VIII

A COMPARISON OF OUR WORK AND FLEYEH ’ S WORK [5]

Color segmentation

Sign Detection

Sign Classification Fleyeh’s work 88.15% 82.3% 100% Our work 91.8% 87.6% 99.5% Table 8 shows a comparison of our work and Fleyeh’s work [5] We get higher success rates in case of color segmentation and sign detection stages Meanwhile in sign classification stage, Fleyeh’s work used binary attributes which obtained from yellow color segmentation using HSV color, so the result is quite perfect (100%)

For demonstration, 12 reds signs and 5 blue signs are trained and classified correctly in this thesis A number of videos with traffic signs were captured to test the recognition system and a final video result has been created The whole system has been implemented in C++ language using OPENCV library version 2.2 The mean processing time of 0.27s per frame using a 2.2GHz Intel Core 2 Duo CPU T6600 with 448x336 pixels of frame

VII C ONCLUSION

A complete method for traffic sign recognition which takes into consideration almost difficulties regarding to object recognition in outdoor environment has been presented in this paper The system can recognize all traffic signs of interest with high accuracy A robust method has been developed to overcome the problem of clustered signs using sign’s inner

In order to improve the performance of the system, tracking method using Kalman filter has been investigated The tracking helps to estimate the traffic sign position in successive frames by using the information of inter-frame However, due to shortage of time, the tracking method is listed in the future work

ACKNOWLEDGMENT

I would like to express my sincere thanks to Dr Ovidiu Ghita for his precious advices and helps for this thesis

REFERENCES [1] H Fleyeh, “Traffic Sign Recognition by Fuzzy Sets”, 2008 IEEE Intelligent Vehicles Symposium Eindhoven University of Technology Eindhoven, The Netherlands, June 4-6, 2008

[2] Saturnino Maldonado-Bascó, Sergio Lafuente-Arroyo, Pedro Jiménez, Hilario Gómez-Moreno and Francisco López-Ferreras,

Gil-“Road-Sign Detection and Recognition Based on Support Vector Machines”, IEEE Transaction on Intelligent Transportation Systems,

Vol 8, No 2, June 2007 [3] Fleyeh, H., "Shadow and Highlight Invariant Colour Segmentation Algorithm for Traffic Signs" second IEEE Conference on Cybernetics and Intelligent Systems, Bangkok, Thailand, June, 2006

[4] S Maldonado-Bascón, J Acevedo-Rodríguez, A Caballero, F López-Ferreras, “An optimization on pictogram identification for the road-sign recognition task using SVMs”,

Fernández-Computer Vision and Image Understanding, 2009

[5] H Fleyeh, “Traffic and road sign recognition”, Dalarna University, Sweden, 2008

[6] P Rosin, "Measuring shape: ellipticity, rectangularity, and triangularity", Machine Vision and Applications, vol 14, pp 172-184,

A Robust Algorithm for Detection

and Identification of Traffic Signs in

Supervised by Dr Ovidiu Ghita

Acknowledgements

First, I would like to thanks my supervisor Dr Ovidiu Ghita The decision he made to be my supervisor, gave me the opportunity to do the project which I am interested in I am grateful

to his support, guidance and advices

I would also like to thanks Dr Martin Collier and Dr David Molloy for helping me to change my old project and give me a chance to work with Dr Ovidiu Ghita in this project Finally I would like to express my appreciation to Dr Martin Collier again for preparing the original version of this documentation, from which this document is adapted

iii

Declaration

I hereby declare that, except where otherwise indicated, this document is entirely my own

work and has not been submitted in whole or in part to any other university

Signed: Date:

Abstract

The traffic signs have primarily the role to regulate the traffic and to provide information for drivers about road quality, traffic restrictions, warnings, possible directions, etc The vast majority of the road (traffic) signs are standardized and they have distinct shapes and colour patterns to facilitate their easy identification in all traffic conditions With the increased traffic congestion that was witnessed over the past decade, the correct identification of the road signs plays an important role in preventing accidents In particular the signs that regulate the traffic at intersections are of utmost importance since their avoidance can result

in collisions with extreme consequences The availability of an automatic vision-based system that is able to warn the drivers about the approaching (incoming) traffic signs will be

a useful tool that will help the drivers in the process of identifying the road signs especially

in situations when they are placed in unexpected locations

The aim of this project is the development of a robust computer-vision algorithm (system) that is able to detect and identify the traffic signs in low to medium resolution video data The system consists of two stages: the traffic sign detection, the traffic sign identification

(recognition) Shadow and Invariant Algorithm is intended to use in detection stage, while Fuzzy Shape Recognizer and Linear Support Vector Machines are considered to

use in the second stage

Table of Contents

ACKNOWLEDGEMENTS II DECLARATION III ABSTRACT IV TABLE OF CONTENTS V TABLE OF FIGURES VI GLOSSARY VII

CHAPTER 1 - INTRODUCTION 1

1.1WHAT IS TRAFFIC SIGN RECOGNITION? 1

1.2TRAFFIC SIGN RECOGNITION APPLICATIONS 2

1.3POTENTIAL CHALLENGES 3

1.5AIMS AND OBJECTIVES OF THE RESEARCH PROJECT 6

CHAPTER 2 - TECHNICAL BACKGROUND 7

2.1COLOUR-BASED DETECTION OF TRAFFIC SIGNS 7

2.2SHAPE-BASED DETECTION OF TRAFFIC SIGNS 9

2.3COLOUR-SHAPE-BASED DETECTION OF TRAFFIC SIGNS 10

2.4RECOGNITION AND CLASSIFICATION 10

CHAPTER 3 – ANALYSIS TECHNIQUES 13

3.1SYSTEM OVERVIEW 13

3.2COLOUR SEGMENTATION 14

3.3IMAGE RECOGNITION AND SIGNS EXTRACTION 16

CHAPTER 4 - CONCLUSIONS AND PROJECT SCHEDULE 19

REFERENCES 20

Table of Figures

FIGURE 1.1.BMW’S TRAFFIC SIGN RECOGNITION SYSTEM 1

FIGURE 1.2.FADED SIGNS 3

FIGURE 1.3.BAD WEATHER CONDITIONS (RAIN AND SNOW) 4

FIGURE 1.4.BAD LIGHTING GEOMETRY 4

FIGURE 1.5.THE PRESENCE OF OBSTACLES IN THE SCENE 5

FIGURE 1.6.DAMAGED SIGNS 5

FIGURE 1.7.MOTION BLUR PROBLEM 5

FIGURE 1.8.COLOURS IN DIFFERENT COUNTRIES: LEFT,NETHERLANDS; RIGHT,SWEDEN.6 FIGURE 3.1.A BLOCK DIAGRAM OF TRAFFIC SIGN RECOGNITION 13

FIGURE 3.2.RESULTS FROM SHADOW AND HIGHLIGHT INVARIANT ALGORITHM [3] 15

FIGURE 3.3.THE RESULT OF USING COLOUR CONSTANCY ALGORITHM [8] 16

FIGURE 3.4.SOME RESULTS OF THIS ALGORITHM [3] 17

FIGURE 4.1.PROJECT SCHEDULE 19

Glossary

OpenCV: Open Computer Vision Library

SVMs: Support Vector Machines

TSR: Traffic Sign Recognition

Chapter 1 - Introduction

1.1 What is Traffic Sign Recognition?

Road Sign Recognition (TSR) is a field which is concerned with the detection and recognition of road and traffic signs in traffic scenes acquired by a camera It is a technique which uses computer vision and artificial intelligence to extract the road signs from outdoor images taken in uncontrolled lighting conditions where these signs may be occluded by other objects, and may suffer from different problems such as colour fading, disorientation, and variations in shape and size [5]

The first paper on the subject was published in Japan in 1984 [5] The aim was to try various computer vision methods for the detection of road signs in outdoor scenes Since that time many research groups and companies have shown interest, conducted research in the field, and generating an enormous amount of work Different techniques have been used to cover different application areas, and vast improvements have been achieved during the last decade The first TSR systems which recognize speed limits were developed in cooperation

by Mobileye and Continental AG They first appeared in late-2008 on the redesigned BMW 7-Series, and the following year on the Mercedes-Benz S-Class Currently these systems only detect speed limits [wiki]

Figure 1.1 BMW’s traffic sign recognition system

In the recognition stage, each of the candidates is tested against a certain set of features (a pattern) to decide whether it is in the group of road signs or not, and then according to these features they are classified into different groups These features are chosen so as to emphasize the differences among the classes The shape of the sign plays a central role in this stage and the signs are classified into different classes such as triangles, circles, octagons Pictogram analysis allows a further stage of classification By analysing pictogram shapes together with the text available in the interior of the sign, it is easy to decide the individual class of the sign under consideration The system can be implemented

by either colour information, shape information, or both Combining colour and shape may give better results if the two features are available, but many studies have shown that detection and recognition can be achieved even if one component, colour or shape, is missing [5]

1.2 Traffic Sign Recognition Applications

Techniques for traffic sign detection and recognition have been developed in a range of application areas which include [5]:

Driver Support System (DSS) can detect and recognise road signs in real time This

helps to improve traffic flow and safety, and avoid hazardous driving conditions, such

as collisions Traffic sign detection and classification is one of the subjects which are not studied deeply Future Intelligent Vehicles would take some decisions about their speed, trajectory, etc depending on the signs detected Although, in the future, it can

be part of a fully automated vehicle, now it can be a support to automatically limit the speed of the vehicle, send a warning signal indicating excess speed, warn or limit illegal manoeuvres or indicate earlier the presence of a sign to the driver The general idea is to support the driver in some tasks, allowing him or her to concentrate on driving

Highway maintenance: This is used to check the presence and condition of the signs

Instead of an operator watching a video tape, which is a tedious work because the signs appear from time to time and the operator should pay a great attention to find the damaged ones, the road-sign detection and recognition system can do this job automatically for the signs with good conditions and alerts the operator when the sign

is located but not classified

Sign inventory: The many millions of roadway signs necessary to keep roadways safe

and traffic flowing present a particular logistical challenge for those responsible for the installation and maintenance of those signs Road signs must be properly installed

in the necessary locations and an inventory of those signs must be maintained for future reference

Mobile Robots: Landmarks similar to road and traffic signs can be used to

automatically mobilise robots depending on the detection and recognition of these landmarks by the robot

1.3 Potential Challenges

Smart vehicles will operate in real traffic conditions The algorithm should be, therefore, robust enough to get good results under adverse illumination and weather conditions which are a great challenge for the developers Identification of traffic signs at correct time and place is very important Due to the change of weather conditions or viewing angles, traffic signs are difficult to be identified Consequently, the detection and recognition of these signs may face one or more of the following difficulties:

1) The colour of the sign fades with time as a result of long exposure to sun light, and the reaction of the paint with the air, figure 1.2

Figure 1.2 Faded signs [5]

4) Colour information is very sensitive to the variations of the light conditions such as shadows, clouds, and the sun It can be affected by illuminant colour (daylight), illumination geometry, and viewing geometry, as shown in figure 1.4

5) The presence of obstacles in the scene, such trees, buildings, vehicles and pedestrians or even signs which occlude other signs, as shown in figure 1.5

Figure 1.3 Bad weather conditions (Rain and Snow) [5]

Figure 1.4 Bad lighting geometry [5]

6) Signs may be damaged, disoriented or occulted, figure 1.6

7) The size of the sign depends on the distance between the camera and the sign itself 8) The acquired image often suffers from motion blur and car vibration, figure 1.7

9) Sign boards often reflect the light from the sky or from an approaching car during weak daylight hours or generate highlight

10) Different countries use different colours; and different pictograms Figure 1.8 shows two images for the YIELD sign

Figure 1.5 The presence of obstacles in the scene [5]

Figure 1.6 Damaged signs

Figure 1.7 Motion blur problem

1.5 Aims and Objectives of the Research Project

The aim of the project is to develop a robust algorithm that can detect and recognize traffic signs in the video So the utmost aspects considered is reliability that means the accuracy in traffic sign recognition

The objectives are thus:

• To understand the properties of road and traffic signs and their implications for image processing for the recognition task

• To understand colour, colour spaces and colour space conversion

• To develop robust colour segmentation algorithms that can be used in a wide range

Chapter 2 - Technical Background

Road sign recognition has become one of the important research fields since the appearance

of the first paper in Japan in 1984 From that time many research groups have been active in the field and have tried to solve this problem using different approaches Although initially the main steps towards a solution seem very well defined and straightforward, the details of the approaches used show that there are several alternatives and many ideas as to how better solutions, better robustness, or a better classification rate can be achieved So far, no one solution method has dominated, and it will clearly take some time before systems are seen

in the market [5]

The identification of road signs can be carried out by two main stages: detection and recognition In detection, research groups are categorised into three groups [5] The first group of researchers believes that traffic sign colours are important information by which traffic signs can be detected and classified The second group believes that detection of traffic signs can be achieved by traffic sign shape only, and the third believes that colour together with shape make the backbone for any road sign detection Thus, there are three major approaches to detecting traffic signs: detection using colour information, detection using shape information, and detection using both colour and shape information Instead of presenting each reviewed paper according to author, I will analyze their achievements and conclude our approach

2.1 Colour-Based Detection of Traffic Signs

Colour is an important source of information in traffic signs recognition The first part of colour detection is colour space conversion in which colour gathered by a camera in RGB form can be converted into another colour space so that the colour information can be separated from the intensity information Some researchers prefer to use RGB colour space

or a modified version of this colour space while others prefer to undertake colour space conversion to get better results The major colour-based techniques are summarised below [5]:

a) Colour Threshold Segmentation: This is one of the earliest techniques used for segmentation of colour images The method uses a threshold value to classify image pixels to traffic sign pixels or background A reference colour is used to judge whether a pixel is considered as a traffic sign pixel or not

b) Dynamic Pixel Aggregation: Segmentation in this method is performed by introducing a dynamic threshold in the pixel aggregation process on HSV colour space The main advantage of dynamic threshold is to reduce hue instability in real scenes depending on external brightness variation

c) HSI/HSV Transformation: These two colour spaces separate colour information (Hue and Saturation) out of the overall intensity value which makes them more immune to light changes The transformation from RGB colour space to HSI colour space makes the separation between chromatic information and intensity information useful for colour segmentation as the HSI colour space is very similar to human perception of colours

d) Region Growing: This approach uses a seed in a region as a starting point and expands as groups of pixels with a certain colour similarity The approach can be implemented in the HSI colour space As it requires a seed to start and ends when certain criteria are met, it may run into a problem when ending conditions are not satisfied

e) Colour Indexing: Colour histograms are used to compare colours in two images The method is fast and straightforward The colour histogram is used to index the images stored in a database Computations will increase greatly in complex traffic scenes The accuracy achieved by invoking these methods varies between 37% up to 100% [3] Many of these algorithms are developed for real time applications One or two systems have been tested for rain conditions

Four major issues are missing in these algorithms and they are necessary to be investigated

in future work

• The lacking of algorithms dealing with poor light conditions As is the case for Sweden and other Scandinavian countries where winter is long and daylight hours are few

• The absence of algorithms dealing with severe rain showers and snow fall

• There are no algorithms that handle road signs located under trees in which different parts of the sign are exposed to different levels of illumination,

• There are no colour segmentation algorithms which are immune to ‘highlights’ In

‘highlights’ signs function as a mirror to reflect some of the source light to the camera

2.2 Shape-Based Detection of Traffic Signs

In shape-based sign detection grey scale images are used to avoid different problems faced when dealing with colours The outer edges of the signs are used in many studies in this field Among the techniques used to extract road signs are the following [5]:

a) Hierarchal Spatial Feature Matching: The search for geometrical shapes is carried out based on spatial features of signs within the traffic scene Once these shapes are found a list is created and passed to a classification module

b) Hough Transform: The classical Hough transform has been used to detect regular features such as lines and circles It is used because of its ability to isolate features of

a particular shape within an image The method is computationally complex and memory hungry which does not make it a good choice for real-time applications However, these constraints are not crucial issues for road sign inventory

c) Similarity Detection: This approach is performed by finding a similarity factor between a segmented region and a set of binary images which represent each road sign shape The method assumes that both sampled and segmented image have the same dimensions

d) Distance Transform Matching: In this approach a template hierarchy is used to capture the variety of object shapes Efficient hierarchy can be generated offline for given shape distributions using stochastic optimisation techniques In the online mode, a simultaneous coarse-to-fine approach is involved over the shape hierarchy and over the transformation The approach is capable of checking objects of arbitrary shapes which is an advantage over other techniques when dealing with non-rigid objects

It has been proved that it is enough to use road sign shapes to detect them One of the points

to support this theory is the lack of a standard colour system among the different countries even within the European Union Systems relying on colour need to be tuned when moving from one country to another The other point in this argument is the fact that colours vary as daylight and reflectance properties vary In situations in which it is difficult to extract colour information such as twilight time and night time shape detection will be a good alternative However, using shapes to detect road and traffic signs may suffer from some difficulties Among these difficulties are the following:

• Objects similar to traffic signs, such as windows, mail boxes and cars, may exist in the scene

2.3 Colour-Shape-Based Detection of Traffic Signs

It is clear that combining colour information with that of shapes gives a good source of information for traffic sign detection Shape information is valuable as much as that of colour This kind of combination reduces the number of false alarms since every object, not just road and traffic signs, can posses these specifications Alternatively, adaptive traffic sign detectors can be built by using this colour-shape combination When colours are available they can be used to detect the signs Otherwise a shape-based algorithm can be invoked This kind of traffic sign detector may need some kind of rule to control which method is used depending on the availability of colour information or shape information In addition to that, combining colour and shape in one algorithm can also reduce false alarms

by avoiding some of the problems which can arise due to the nature of either of these approaches

2.4 Recognition and Classification

In general, word recognition is used to point out that a sign is identified while word classification is invoked to indicate that the sign is assigned in a certain category based on

certain features Sometimes recognition implies classification, however, a complete separation between recognition and classification is made in this project

From the review a number of parameters should be taken into consideration when a classifier is designed:

a) The recogniser should present a good discriminative power and low computational cost

b) It should be robust to the geometrical status of sign, such as the vertical or horizontal orientation, the size, and the position of the sign in the image

c) It should be robust to noise

Neural Networks are a suitable alternative for the recognition and classification of road

signs There are two distinct advantages of using neural networks First, the input image does not have to be transformed into another representation space Second, the result depends only on the correlation between the network weights and the network However, neural networks have their own problems The training overhead still exists, and the multi-layer neural networks cannot be adapted for on-line application due to their architecture Since this architecture is fixed, there is no provision for an increase in the number of classes without a severe redesign penalty, and they cannot recognise the new patterns without retraining with the entire network In this respect, they do not offer significant advantages over template matching Other types of neural networks, such as reconfigurable neural networks, ART1, ART2, Hopfield, Cellular neural networks, try to offer more flexibility and adaptation of neural networks Kohonen maps were trained for signs partially occluded

by other objects and signs which were rotated by small angles in the outdoor images They can have capabilities to adapt to new signs without need of new training

Template matching (Pattern Matching) is a second alternative in the recognition stage It

is used to classify the inner regions of traffic signs, and in some cases, combined with wavelets to extract the local features of the sign Complex-log transform and 2D-FFT are also combined with template matching to achieve better classification results

Genetic Algorithm can be used to search for traffic sign in a scene image The image is

matched by giving the gene information The gene of individuals can be represented by expression using a set of equation to determine its characteristic

Nearest Neighbour Classification is a straightforward and classic type of classification An

image in the test set is recognised by assigning to it the label of most of the closest points in the learning set All images are then normalised to certain value The image in the learning set that best correlates with the test image is then the result

Support Vector Machines (SVMs) classifier is a potential classifier which shows good

abilities to classify patterns for different applications This classifier which has only been introduced in recent years to the field of traffic sign recognition is chosen as a classifier in this thesis Other classifiers such as classical classifier, weighted distance classifier, Angular

A-12histographic, matching pursuit classifier, Laplace kernel classifier, and Euclidian distance have also been used for road sign classification