Face recognition using local patterns and relation learning

129 7.15 Experimental results of face recognition of three facial representations on ORL.. 133 7.16 Experimental results of face recognition using LBP-based facial fea-tures across the w

Doctor of Philosophy at The University of Canberra

FACE RECOGNITION USING LOCAL PATTERNS AND RELATION LEARNING

Len Bui January, 2013

The study area of this thesis is face recognition, one of the important fields in puter vision Although face recognition has recently achieved many advances, theprocess is still not able to meet the accuracy requirements of many applicationsthat are affected by variations in pose and illumination The aim of this thesis is todevelop a more advanced approach that can handle the challenges in pose and illu-mination in face recognition The thesis proposes Robust Multi-Scale Block LocalBinary Pattern as a new facial representation that is sufficiently robust to acceptvariations in pose and illumination and yet contains rich discriminative informa-tion The thesis also investigates the metrics or scores in general used to measuresimilarity/dissimilarity in face recognition and contributes two novel classificationmethods, namely Extended Bayesian Learning and Relation Learning, to overcomedifficulties such as the Small-Sample-Size problem and gain good performance forface recognition systems.

com-Except where clearly acknowledged in footnotes, quotations and the bibliography, Icertify that I am the sole author of the thesis submitted today entitled -

Face Recognition Using Local Patterns and Relation Learning

I further certify that to the best of my knowledge the thesis contains no materialpreviously published or written by another person except where due reference ismade in the text of the thesis

The material in the thesis has not been the basis of an award of any other degree

or diploma except where due reference is made in the text of the thesis

The thesis complies with University requirements for a thesis as set out in Higher

Degree by Research Examination Policy, Schedule Two (S2).

First and foremost, I would like to thank my primary supervisor, Associate ProfessorDat Tran, for his enormous support during my study at University of Canberra, in

a variety of ways, both academic and personal

I would also like to thank my co-supervisors, Professor Xu Huang and ciate Professor Girija Chetty, for their consistent encouragement and support on myresearch

Asso-My four-and-half-year research at the Faculty of Information Sciences and neering has been priceless The facilities and environment for studying are excellent

Engi-A special thanks to the Faculty and my friends for their support and discussions

I would like to thank Vietnam Ministry of Education and Training for granting

me the scholarship to study at University of Canberra

My study could have not taken without the endless love, support of my family,father, sisters, brothers and nephews This thesis is dedicated to my beloved mother

1.1 Face Recognition 2

1.2 Challenges to Face Recognition 4

1.3 Current Approaches to Face Recognition 6

1.4 Motivation 7

1.5 Problem Statement 7

1.6 Thesis Contribution 9

1.7 Thesis Organization 12

2 Literature Review 15 2.1 Literatures on Facial Feature Representation 17

2.1.1 Data Independent Feature Extraction 17

2.1.2 Data Dependent Feature Extraction 20

2.1.3 Mixed Feature Extraction 23

2.2 Literatures on Face Detection 24

2.3 Literatures on Face Recognition 27

2.3.1 General Face Recognition System 27

2.3.2 Unsupervised Distance Learning-Based Approaches 27

2.3.3 Supervised Distance Learning-Based Approaches 29

2.3.4 Support Vector Machines-Based Approaches 30

2.3.5 Approaches for Face Recognition across Pose and Illumination 30 2.4 Summary 33

3 Facial Representation 35 3.1 Introduction 36

3.2 Pixel-based Facial Features 36

3.3 Fourier-Based Facial Features 37

3.4 Cosine-Based Facial Features 38

3.5 Gabor-Based Facial Features 40

3.6 Haar-like Features 43

3.7 Local Pattern-Based Facial Features 46

3.7.1 Local Binary Patterns 46

3.7.2 Generic Local Binary Patterns 49

3.7.3 Local Ternary Patterns 50

3.7.4 Invariants of Local Binary Patterns 51

3.7.5 Multi-scale Local Binary Patterns 52

3.7.6 Multi-Scale Block Local Binary Patterns 52

3.8 Complex Facial Representations 53

3.9 My Proposed Robust Multi-Scale Block Local Binary Patterns 54

3.10 My Proposed Compact Histogram Representation 55

3.11 Summary 56

4 Face Detection 57 4.1 Description of Face Detection System 58

4.2 Neural Networks 59

4.2.1 Overview 59

4.2.2 Neural Network-Based Classification 60

4.2.3 Neural Network-Based Face Detection 61

4.3 AdaBoost Algorithm 64

4.3.1 AdaBoost-Based Face Detection 65

4.3.2 Face Detection Using Cascade Structure of Boosted Classifiers 66

4.4 My Proposed Hybrid Approach for Face Detection 67

4.4.1 Structure of Hybrid Face Detector 67

4.4.2 Applying to Face Detector 68

4.5 Summary 68

5 Distance Learning for Face Recognition 69 5.1 Description of Face Recognition System 70

5.2 Face Recognition Using Principal Component Analysis 75

5.3 Face Recognition Using Probabilistic PCA 78

5.4 Face Recognition Using Two-Dimensional PCA 80

5.5 Face Recognition Using LDA 81

5.6 Proposed Face Recognition Using Bayesian Learning 82

5.6.1 Bayesian Decision Theory 82

5.6.2 Bayesian Learning 85

5.6.3 Extended Bayesian Learning 88

5.7 My Proposed Face Recognition Using LBP 91

5.7.1 LBP-Based Face Recognition Framework 91

5.7.2 Distances to Measure Similarities of Histograms 91

5.7.3 Feature Transformation 94

5.7.4 Implementation of Computing Similarity/Dissimilarity Scores 95 5.8 Summary 96

6 Relation Learning for Face Recognition 97 6.1 Introduction 98

6.2 Face Recognition Using Support Vector Machine 99

6.2.1 Support Vector Machine 99

6.2.2 SVM-Based Face Recognition 103

6.3 Proposed Face Recognition Using Relation Learning 104

6.3.1 Theory of Relation Learning 104

6.3.2 Algorithm of Relation Learning 108

6.3.3 Relation Learning-Based Face Recognition 111

6.4 Summary 111

7.1 Introduction 114

7.2 Databases 114

7.3 Performance Measures 120

7.3.1 Introduction to Evaluation Protocol 120

7.3.2 Performance Measures in Face recognition 121

7.3.3 Testing Protocol 122

7.3.4 Performance Measures in Face Detection 123

7.4 Experiments on Face Detection 123

7.4.1 AdaBoost and Neural Network-Based Face Detection System 123 7.4.2 Hybrid Models of AdaBoost and Neural Network 125

7.5 Experiments on Gender Classification 127

7.5.1 Database and Experiment Setup 127

7.5.2 Comparisons of Facial Representation Methods 127

7.5.3 Comparisons of Dimension Reduction Methods 128

7.5.4 Comparisons of Classification Methods 128

7.6 Experiments on Face Recognition 129

7.6.1 Global and Local Facial Representations 129

7.6.2 Gabor-Based Facial Representations 130

7.6.3 Local Pattern-Based Facial Representations 132

7.6.4 Complex Facial Representation 139

7.6.5 Fusion Classifiers 140

7.6.6 Distance Learning 143

7.6.7 Extended Bayesian Learning 146

7.6.8 Relation Learning 151

7.7 Summary 154

8 Conclusions and Suggestions for Future Work 155 8.1 Conclusions 156

8.2 Future Work 157

List of Figures

1.1 Two modes of operation of face recognition 2

1.2 A typical face recognition system 3

3.1 A 2-by-2 image and its pixel-based features, 4-element column vector 36 3.2 A sample image and its DFT 37

3.3 DFT features of 25 image blocks 38

3.4 A sample image and its DCT 39

3.5 DCT coefficients sorted in zigzag path 39

3.6 DCT features of 25 image blocks 40

3.7 Real and imaginary parts of a 2D complex sinusoid with θ = π/2,γ = 64,φ = 0 and image size of 128-by-128 41

3.8 2D Gaussian-shaped function with θ = π/2,σ = 64,γ = 1 and image size of 128-by-128 41

3.9 Real and imaginary parts of a 2D Gabor filter in spatial domain with θ = π/2,γ = 64,φ = 0,σ = 64,γ = 1 and image size of 128-by-128 41

3.10 40 Gabor filters: The first 4-column set are the real parts and the last 4-column set are the imaginary parts 43

3.11 A sample image and its 40 Gabors and corresponding feature vector 44 3.12 14 Haar-like templates for edge, line, center surrounding features 45

3.13 4 Haar-like templates for corner features 45

3.14 An 8-by-8 image in the top left, its integral image in the top right to compute the sum of pixels in the rectangle where columns from 3 to 7 and rows from 3 to 7, using four values of integral at (2,2) (2,7) (7, 2) and (7,7) 46

3.15 A pixel has the intensity 128 and its eight neighborhood pixels have the intensities 128, 126, 123, 121, 127, 129, 134 and 130 respectively Thresholding and multiplying with powers of two to achieve its LBP

label 111000012 47

3.16 A sample image and its LBP 47

3.17 An image is divided into 25 blocks 25 LBP histograms of blocks are concatenated into a feature vectors 49

3.18 Three generic LBP operators with (P, R) being (8,1), (16,2) and (8,2) respectively When a sampling point is not at the center of pixel, its intensity is calculated by a bilinear interpolation 50

3.19 Examples of texture primitives detected by LBP 51

3.20 A multi-scale operator formed by LBP4,1 and LBP8,2 52

3.21 9 blocks of a multi-scale LBP operator of size 9-by-9 53

3.22 Framework of hybrid of Gabor and LBP facial representations (cour-tesy of Zhang et al.(2005b)) 54

3.23 First four regions determined by AdaBoost (courtesy of Zhang et al (2005a)) 54

3.24 A sparse 16-bin histogram 56

4.1 First module uses an extracting window with window size of 24-by-24 and window step of 12, and down sampling with scaling factor of 1.2 58 4.2 Merging multiple detected windows and removing isolated windows 59 4.3 Structure of three-layer feed-forward neural network where input layer has 6 neurons, hidden layer has 4 neurons and output layer has 2 neurons 60

4.4 Preprocessing task (a) original images (b) best fit linear functions (c) lighting correct images (d) histogram equalization for images (cour-tesy of Rowley et al.(1998)) 63

4.5 Hidden layer with three types of neurons: 4 type-1 looking at 4 gions, 16 type-2 looking at 16 regions and 6 type-3 looking at 6 re-gions, output layer has one neuron 63

4.6 A weak classifier and its Haar-like feature 65

4.7 A cascade structure of boosted classifiers 67

4.8 A hybrid model-based face detector 68

5.1 A sample image transformed and masked 71

5.2 A sample image normalized in illumination using histogram equalization 71 5.3 Sample images after illumination normalization 74

5.4 Extracting feature vectors from two images 74

5.5 Three samples and their projects on a subspace with two principal components 77

5.6 Mean face and first three eigenfaces 78

5.7 (a) Shape of metrics L2 and L1 (b) Shape of metrics Mahalanobis 87

5.8 (a) L2 is a “good” metric to measure distances (b) L2 is a “bad” metric to measure distances 87

5.9 xi, i = 1, , N are data points, µ is the mean of data points, the distances of data points to mean and ρ is the residual parameter 89

5.10 Distances from xi to mean µ in subspace ∨ and ¯∨ 90

5.11 Framework of LBP-based face recognition 91

5.12 Two 4-bin histograms and their overlapped 93

5.13 A diagram of score computation without using histograms 96

6.1 Two classes A and B with their elements, a decision hyperplane and its two margins 100

6.2 An element of class A is in incorrect side and its slack variable 101

6.3 Input space and feature space 102

6.4 A binary tree for 8-class face recognition 104

6.5 A wrong classification for unknown x using k-NN 105

6.6 A wrong classification for unknown x using VQ 105

6.7 Three pairs having the same distance but not belonging to the same class 106

6.8 Robustness values of discriminant function and relation function 108

7.1 Sample images from ORL 114

7.2 Sample images from EYFDB 115

7.3 Sample images from CMU-PIE 116

7.4 Sample frontal categories from FERET 117

7.5 Sample pose categories from FERET 117

7.6 Another sample pose categories from FERET 117

7.7 (a) Face images (b) Non-face images from MIT-CBCL 118

7.8 A sample image from the first group of MIT+CMU 118

7.9 A sample image from the second group of MIT+CMU 119

7.10 Sample images from CMU Test Set II 119

7.11 Sample images from CalTech database 120

7.12 The flowchart of FERET testing procedure 122

7.13 Experimental results for ABx and ABNNx on CalTech 126

7.14 (a) A whole facial image (b) Local regions of image (c) Gradient of image 129

7.15 Experimental results of face recognition of three facial representations on ORL 133

7.16 Experimental results of face recognition using LBP-based facial fea-tures across the window size on ORL 135

7.17 Experimental results of face recognition using LBP-based facial fea-tures across the window size on FB 135

7.18 Experimental results of face recognition for facial representations on probe set FB 136

7.19 A classifier for fusing two-class SVM+AdaBoost 140

7.20 A classifier fusing multi-class SVM and AdaBoost/Voting using Ga-bor features 142

7.21 Experimental results of face recognition using three scores across the window size on ORL 144

7.22 Experimental results of face recognition of three scores across the window size on (a) FB (b) FC (c) DUP1 (d) DUP2 145

7.23 Experimental results of face recognition of Soft Chi Square score with respect to k on ORL 146

7.24 Experimental results of face recognition of Soft Chi Square score with respect to k on (a) FB (b) FC (c) DUP1 (d) DUP2 147

7.25 Experimental results of face recognition using feature transformation with respect to k on ORL 148

7.26 Experimental results of face recognition using feature transformation with respect to k on (a) FB (b) FC (c) DUP1 (d) DUP2 149

7.27 Recognition rates over weight w in the range of [0,20] on ORL vali-dation set 150 7.28 Recognition rates over weight parameter w on FERET validation set 151

7.29 Cumulative match score curves of experiments of face recognition on

(a) FB (b) FC (c) DUP I (d) DUP II (using soft score with w = 7.25) 152

List of Tables

5.1 Parameter settings for illumination normalization 74

7.1 Summary of ORL database 114

7.2 Summary of EYFDB database 115

7.3 Summary of CMU-PIE database 115

7.4 Summary of FERET database 117

7.5 Experimental result of AdaBoost-based face detector on CalTech data face 124

7.6 Parameter settings for NNs 124

7.7 Experimental result of NN-based face detector on CMU+MIT data set124 7.8 Experimental result of NN-based face detector on CalTech face data set 124

7.9 Experimental results of ABs on CalTech 126

7.10 Experimental results of ABNNs on CalTech 126

7.11 Experimental results of gender classification of facial representation methods on FERET 127

7.12 Experimental results of gender classification of dimension reduction methods on FERET 128

7.13 Experimental results of gender classification methods on FERET 129

7.14 Experimental results of face recognition of facial representations on ORL 130

7.15 Experimental results of face recognition of Gabor-based, PCA-based and DCT-based features on ORL 131

7.16 Experimental results of face recognition of Gabor-based and PCA-based on FERET 132

7.17 Experimental results of face recognition of three facial representations on FERET 134

7.18 Experimental results of face recognition using facial representations

on ORL database 1367.19 Results of computation time of scores on ORL (in seconds) 1377.20 Results of computation time of scores on FERET (in seconds) 1387.21 Results of computation time of LBP operators on ORL (in seconds) 1387.22 Results of computation time of LBP operators on FERET (in seconds)1387.23 Results of computation time of LBP operators on ORL (in seconds) 1397.24 Results of computation time of LBP operators on FERET (in seconds)1397.25 Results of face recognition for three facial representations on FERET 1407.26 Results of face recognition of fusion classifiers on ORL 1417.27 Results of face recognition of fusion classifiers on FERET 1417.28 Results of face recognition of fusion classifiers on ORL 1427.29 Results of face recognition using fusion classifiers on Extended YaleFace Database 1427.30 Results of face recognition of classification methods on ORL 1487.31 Results of face recognition rate (%) of classification methods on ORL 1537.32 Results of face recognition rate (%) of classfication methods on FERETdatabase 153

List of Abbreviations

DCT Discrete Cosine Transform

DFT Discrete Fourier Transform

HFIF Holistic Fourier Invariant Feature

ICA Independent Component Analysis

k-NN k-Nearest Neighbor

LBP Local Binary Pattern

LDP Local Derivative Pattern

LLE Locally Linear Embedding

MBLBP Multi Block Local Binary Pattern

MLP Multi Layer Perceptron

NLPCA Nonlinear Principal Component Analysis

NN Artificial Neural Network

PCA Principal Component Analysis

PIN Personal Identification Number

RMBLBP Robust Multi Block Local Binary Pattern

SVM Support Vector Machine

List of Publications

Bui, L and Le, T (2008) A hybrid approach of adaboost and artificial neural

network for detecting human faces In Proceedings of the IEEE International

Conference on Research, Innovation and Vision for the Future (RIVF), pages

79–85 IEEE

Bui, L., Tran, D., Huang, X., and Chetty, G (2010) Face gender recognition based

on 2d principal component analysis and support vector machine In Proceedings

of the 4th International Conference on Network and System Security (NSS), pages

579–582 IEEE

Bui, L., Tran, D., Huang, X., and Chetty, G (2011a) Classification of gender and

face based on gradient faces In Proceedings of the 3rd European Workshop on

Visual Information Processing (EUVIP), pages 269–272 IEEE.

Bui, L., Tran, D., Huang, X., and Chetty, G (2011b) Face recognition based on

ga-bor features In Proceedings of the 3rd European Workshop on Visual Information

Processing (EUVIP), pages 264–268 IEEE.

Bui, L., Tran, D., Huang, X., and Chetty, G (2011c) A new approach to bayesian

method for face recognition In practice, 1:3.

Bui, L., Tran, D., Huang, X., and Chetty, G (2011d) Novel metrics for face

recog-nition using local binary patterns Knowledge-Based and Intelligent Information

and Engineering Systems (KES), pages 436–445.

Bui, L., Tran, D., Huang, X., and Chetty, G (2011e) Relation learning-a new

approach to face recognition Proceedings of Advances Concepts for Intelligent

Vision Systems (ACIVS), Belgium, LNCS 6915, pages 566–575.

Bui, L., Tran, D., Huang, X., and Chetty, G (2012) Face gender classification

based on active appearance model and fuzzy k-nearest neighbors In The 2012

International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV’12), USA IEEE.

Chapter 1

Introduction

1.1 Face Recognition

People in modern societies, especially in developed countries, are very familiar withusing smartcards, e-passports and internet banking services in daily life However,these services can cause many severe problems in security and privacy Traditional

identification methods of using passwords or Personal Identification Numbers (PINs)

are usually vulnerable to criminal attack because PIN numbers may be lost, guessed

or even cracked by criminals who have sophisticated hacking tools These tional identification methods do not meet the increasing security requirements inour modern information societies A current authentication system should employnew techniques such as identification methods based on biometric information Facerecognition is one such method of identification, and it needs to be chosen carefully.Face recognition is a visual pattern recognition problem A facial image rep-resenting a three-dimensional human head is affected by illumination, facial pose,facial expression and many other factors, yet it still needs to be identified by search-ing and matching from enrolled faces in databases Most current face recognitionsystems use two-dimensional face images; although, some special applications usethree-dimensional facial images Like other biometric-based recognition systems, aface recognition system has two modes of operation: verification and identification,shown in Figure 1.1

tradi-Figure 1.1: Two modes of operation of face recognition

Face verification is known as a one-to-one matching task; to do the verification

a query face image is compared and matched against a claimed user’s face image

Potential applications for a face verification system are Automated Teller Machine

(ATM) systems or vending machines Such systems would combine the use of words and checks of a client’s face image acquired at transaction with the authenticface image stored in a bankcard The combination of confirming both password andfacial image would increase the ability of the system to protect users from criminalattacks In this research, a typical face verification system is designed in two mod-ules: namely enrolment and verification The enrolment module performs the task

pass-of taking a picture pass-of the client’s face using an imaging device such as a digital era and then storing this information as a representation or template in the relevantbankcard as well as in the bank’s database The verification module performs thetask of matching at, for example, ATMs or vending machines It acquires the clientface image, and using the same representation as the previous phase for the input,then computes the similarity between the input and the face image stored in thebankcard If the resulting score achieves the acceptance threshold, the transaction

cam-is allowed to proceed; otherwcam-ise, the transaction cam-is rejected

Face identification is much more complicated than face verification It is a to-many matching task in which a query face image is compared with an entiredatabase of enrolled face images in order to determine the query face identity Someapplications report only one identity, namely that of the person whose face looksmost like the input face but other applications, such as the video surveillance used ingovernments and law enforcement to maintain social control, recognize and monitorthreats, or prevent/investigate criminal activity can report all identities whose facialsimilarity scores reach a confidence level threshold parameter

one-A typical face recognition system (shown in Figure 1.2) consists of four modules:face localization, face normalization, feature extraction and feature matching

Figure 1.2: A typical face recognition system

A face localization module aims to extract facial regions from the image ground In the case of video clips, the detected faces may need to be tracked acrossmultiple frames using a face-tracking component The task provides a rough esti-mate of the location and scale of a face It also includes face land marking to localizefacial landmarks e.g eyes, nose, mouth and facial outline

back-A face normalization module will normalize the resulting face geometrically andphotometrically Such is definitely necessary because most the current face recogni-tion systems, in reality, are expected to deal with face images in varying conditions

of pose and illumination The geometrical normalization process transforms theface into a standard frame using the transformations of scaling and cropping Othertransformations of warping or morphing may be used for more sophisticated geo-metric normalization The photometric normalization task normalizes image pixelsusing several image-processing techniques such as histogram equalization

A feature extraction module is performed on the normalized face to extractsalient information that is useful for distinguishing faces of different people, androbust with respect to geometric and photometric variations The extracted facialfeatures are used for face matching

In a face-matching module, the extracted features from the input face arematched against one or many of the enrolled faces in the relevant database, de-pending on its operation mode The module reports ’yes’ or ’no’ for the verificationmode and for the identification mode It will report the identity of the input face ifthe top match is found with sufficient confidence or unknown if the confidence value

is below a predetermined threshold

1.2 Challenges to Face Recognition

An ordinary person does not need to put much effort into identifying a familiarhuman face even under severely degraded conditions of viewpoint, illumination andexpression However, current face recognition systems cannot achieve a visual iden-tification ability comparable to human beings The difficulties are caused by manyfactors and are still challenging the development of any face recognition algorithm.These difficulties can be classified into three issues: a large variability in facial ap-pearance of the same person; the high dimensionality of data and small sample size;and the presence of highly complex and nonlinear manifolds (Arca et al., 2006; Liand Jain,2011; Sun et al., 2006)

Large variability in facial appearance of the same person Although face

recog-nition algorithms have recently gained achievements in handling the problem, thechallenge of large variability remains, and still causing many problems in face recog-

nition A face, which is part of a non-rigid and dynamic human head, comes with alarge diversity in shape, color and texture, and is affected by multiple factors such

as head pose, lighting conditions (contrast, shadows), facial expressions, occlusions(glasses) and other facial features (make-up, beard) There are also various imag-ing device parameters such as aperture, exposure time, lens aberrations and sensorspectral response All these factors are mixed together causing many problems forextraction of the intrinsic information of the face object from the image Eventhough well-known, the statement below about the challenge still holds true:

“The variations between the images of the same face due to illumination andviewing direction are almost always larger than the image variation due to change

in facial identity” (Moses et al., 1994)

The implication here is that the intra-personal variations are usually larger thanthe image variation due to change in the facial identity named inter-personal vari-ation This variability makes it difficult to build a sophisticated model capable ofdescribing an individual using a small number of sample images

High dimensionality and small sample size A typical image, for example one

in the FERET face database has the size of 150-by-130 and is directly represented

by a 19500-dimensional vector in an image space However, the number of imagesfor each person (often less than ten) is much less than the dimensionality of theimage space The limitation is due to the high cost of building a face database.Therefore, it is hard for the system to construct reliable models for each personbecause the training data does not contain all the intra-personal variants This iscalled the generalization problem In addition, processing a high-dimensional data

is computationally expensive while the enrolment and identification tasks in real-lifeapplications need to be fast

High complex and nonlinear manifolds The face manifolds of any individual

and also the entire face manifold that is the combination of all these manifoldsare very complex and highly non-convex Linear subspace-based methods, such as

Eigenface, Fisherface or Independent Component Analysis, attempt to project data

in a high-dimensional image space into a low-dimensional subspace with respect

to some optimal constraints, such as preserving the variances of data as much aspossible, and maximizing discriminative information However, these methods do

not account for the non-convex variations of face manifolds to differentiate among

different individuals Common distance-based similarity measures, such as Euclidean

or City-block, which are widely used in face recognition algorithms do not perform

well at discriminating between manifolds of a face and non-face, or among manifolds

of different individuals Therefore, they limit the ability of the linear methods toachieve highly accurate face detection and recognition in many practical scenarios

1.3 Current Approaches to Face Recognition

Over the last thirty years, face recognition has continued to be one of the hot topics

in the study of computer vision due to the high benefits from its practical cation Challenging problems in face recognition have attracted researchers frommany different disciplines such as pattern recognition, computer vision and com-puter graphics A single system often involves techniques developed from differentprinciples and the use of a combination of techniques This makes it difficult tocategorise these systems based on the techniques that they use for representation orclassification However, there are three basic types of method in use:

appli-Holistic or appearance-based methods Using a whole face image as the input to a

face recognition system Typical examples are Eigenface (Turk and Pentland,1991b)

using Principal Component Analysis, Probabilistic Eigenface (Moghaddam et al.,

2000) using Bayesian learning, Fisherface (Belhumeur et al., 1997) using Fisher

Linear Discriminant Analysis and Bartlett (2001) using Independent Component

Analysis Although their systems are not as complicated and often have a high

performance time, in practice these methods are rarely implemented independentlyand uniquely in a current face recognition system because they cannot satisfy currentrecognition rate requirements

Feature-based methods Facial local features such as eyes, nose and mouth; data

about these features are localized and extracted, and the information relating to such

as locations or geometric properties is then sent to a classification module Typical

examples of feature-based methods are Geometry methods As appearance-based

methods, they often combine with other approaches to improve their performance

Hybrid methods Most current face recognition systems use both local features and

the whole face region for classification It is argued that these methods potentially

offer the best model Typical examples are Modular Eigenface (Pentland et al.,

1994) using Eigenface and Eigenfeature such as Eigeneye, Eigennose, Eigenmouth,

and Component-based method using face region and components Recently, hybrid

methods using Local Patterns such as Local Binary Pattern (Ahonen et al., 2004,

2006; Ahonen and Pietikäinen, 2009) have been widely used in face recognitionand detection due to their good performance and relatively simple and efficientcomputation tasks

1.4 Motivation

Face recognition is the major biometric technique in use It is more natural andnon-intrusive than the other biometric techniques such as fingerprinting, and iris andspeech recognition Compared to fingerprinting, for example, it is more practicalbecause it does not require special equipment or any expert to verify the results.Any part of a face can be used, and many images of a person’s face, includingdifferent expressions and poses, can be merged to make a composite image that

is more meaningful to the machine In addition, the most important advantage

of the face is that it can be captured at a distance Among the major biometric

techniques, facial recognition has the highest score compatible to Machine Readable

Travel Documents system based on several criteria for enrolment, renewal, machine

requirements, and public perception Face recognition has recently become veryimportant in modern society owing to recent rapid advances in imaging devices andincreased demands for high levels of security Although face recognition has recentlyadvanced in many respects, it is still not able to meet the accuracy requirements ofmany applications that are affected by variations of illumination and pose

1.5 Problem Statement

The aim of this study is to answer the research question “Can facial representationsbased on discriminative and robust local patterns when combined with discrimina-tive methods handle the challenges of pose and illumination in face recognition?”

My methodology of this study to answer the research question is as follows

1 First I will find proper facial representations available in the literature anduse them to create a set of facial features from raw face images which haveminimum intra-class variations (within-face-differences of the same individ-ual) and maximum extra-class variations (between-face-differences of differentindividuals).

Obviously, even the most sophisticated classifiers cannot accomplish the facerecognition task if inadequate facial representations are in use Therefore, de-ciding what facial representation is, is very important in designing face recog-nition systems Ideally, a facial feature representation should satisfy the twofollowing criteria:

• It must have the ability to discriminate between different individuals wellwhile tolerating within-class variations; and

• The representation must be easily extracted from the raw face images

In fact, it is not easy to find facial features that meet all these criteria because

of the large variability in facial appearance caused by different imaging factorssuch as scale, orientation, pose, facial expression and lighting conditions Inthe literature, numerous methods have been proposed for representing facialimages for recognition purposes The earliest works in the early 70s are based

on representing faces in terms of geometric relationships, such as distances andangles, between facial landmarks such as the eyes, nose and mouth Later,appearance-based techniques were developed that generally considered a face

as a 2D array of pixels and aimed at deriving descriptors for facial appearancewithout the explicit use of face geometry Following along these lines, a num-

ber of different holistic methods such as Principal Component Analysis, Linear

Discriminant Analysis and the more recent 2D Principal Component sis have been widely studied Lately, local descriptors have had increasing

Analy-attention due to their robustness to such challenges as pose and illumination

changes Among these descriptors are Gabor Wavelets and Local Binary

Pat-terns which are showing themselves to be very successful in encoding facial

appearance This research project aimed to inherit and develop

state-of-the-art facial representations based on Gabor Wavelets and Local Patterns.

2 Second, I will find and design optimal classifiers for face recognition

Statistical learning methods play a very important role in building currentfacial recognition systems There have been two main approaches used inpattern recognition in general and face recognition in particular The first,called the generative approach, determines class-conditional density functionsfor each class or person individually and their prior class probabilities, thenuses Bayes theorem to find the posterior class probabilities Decision theory isthen applied to the resulting posterior probabilities to determine class mem-bership for each new input The second, called the discriminative approach,finds a discriminant function which maps each input directly to a class label

or identity In this case, probabilities play no role The thesis aims to applyand combine these two methods in face recognition

1.6 Thesis Contribution

The contribution of this thesis to the methodology of face recognition can be marized as follows

sum-1 Robust Multi Block Local Binary Pattern for facial representation:

Facial representation is one of the major problems in face recognition The

thesis proposes the use of the Robust Multi Block Local Binary Pattern BLBP) for facial representation The original Local Binary Pattern (LBP)

(RM-operator proposed by Ojala et al (1996) labels the pixels of an image bythresholding the 3-by-3 neighborhood of each pixel with itself, and consideredeach result as a binary number or label Then the histogram of the labels is

used as a texture descriptor The Multi-scale Local Binary Pattern operator

proposed byOjala et al.(2002) is an extension of the original LBP, with respect

to the neighborhoods of different sizes A drawback of LBP-based methods,

as well as of local descriptors using vector quantization, is that they are notrobust in the sense that a small change in the input image would always cause

a small change in the output LBP may not work properly for noisy images

or on the flat image areas of a constant gray level The use of the Multi Block

Local Binary Pattern (MBLBP) (Zhang et al., 2007) is proposed to overcomethis problem Details of this contribution are presented in Chapter 3

2 Compact Histogram Representation: One of problems in histogram

rep-resentation is the number of bins of histograms caused by the code size (thenumber of different local patterns that can create) (Hussain and Triggs,2012).This number increases exponentially with the size of the spatial support of thepattern and the number of quantization levels It was observed that a his-togram of local patterns is usually sparse; in other words, most of its bins arezero Therefore, I propose a compact histogram representation that describes

a histogram as an array of local pattern labels and their corresponding number

of instances Details of this contribution are presented in Chapter 3

3 Hybrid Approach for Face Detection: AdaBoost and Neural Network

might not achieve desired results in detection rate An AdaBoost-based facedetector has a rather high false rate because the cascade of boosted classifierssignificantly depends on the ability of weak classifiers To handle the prob-lem, we can increase the number of stage classifiers of the system in order toachieve higher detection rate However, increasing the number of both classi-fiers and features means increasing computation time On the other hand, wecan combine AdaBoost with other classification methods that have the ability

to reject false negative images correctly and efficiently in order to improve theperformance Neural network should be a good candidate to combine withAdaBoost due to its having good detection rate and running time We can use

a hybrid model based on AdaBoost and Neural Network (Bui and Le, 2008).Details of this contribution are presented in Chapter 4

4 Face Recognition Using Bayesian Learning: Classification methods play

an important role in face recognition For a generative approach, the thesisproposes the enhanced Bayesian learning (Bui et al., 2011c) that aims to find

a subspace that can maximize the ratio between two covariance matrices ofbetween-class and within-class, whereas PCA finds a subspace that can cap-ture the most variance of a data set An intra-variation measures the differencebetween images of one individual, while extra-variation measures the differ-ence between images of two different individuals However, often there is notsufficient data to estimate the density distributions for these variations, henceall the intra-variations are assumed to be similar and all the extra-variationsthe same From the density distributions of the two variation subspaces, themaximum likelihood and maximum a posteriori scores to measure the similar-

ity between the face images, are defined This study extends this approach byintroducing a scale factor to measure the contribution of the two subspaces tothe final similarity score Details of this contribution are presented in Chapter5.

5 Face Recognition Using LBP: The comparison operator between single

pixels in LBP is replaced with a comparison between the average gray values

of sub-regions Each sub-region is a square block containing neighboring pixels(or just one pixel particularly) The whole filter is comprised of nine blocks.The size of the filter is considered as a parameter, and denote the scale of theMBLBP operator A 3-by-3 MBLBP is in fact the traditional LBP Because theaverage gray values are still non-robust to noisy images, the thesis proposesusing the median gray values that are more robust to outliers or noise andreplacing them with the average ones The research also investigated metrics

or scores in general to measure the similarity/dissimilarity in face recognition

Two novel scores were examined, Soft Chi, which is a generalization of Chi, and Ratio, which combines similarity and dissimilarity information It is also proposed that Power transformation be used to map a feature space dominated

by a non-Euclidean metric to another feature space dominated by a Euclideanmetric, which is appropriate for metric-based learning methods (Bui et al.,

2011d) Details of this contribution are presented in Chapter 5

6 Face Recognition Using Relational Learning: For the discriminative

ap-proach, relation learning (Bui et al.,2011e) is proposed Unlike the statisticalapproach, there is no need to estimate statistical distributions in the discrimi-native approach A discriminant function can be found that directly maps an

object to a class label There is a variety of discriminative methods such as

k-Nearest Neighbor (k-NN), Artificial Neural Network (NN) and Support Vector Machine (SVM) It is noted that the k-NN, the simplest and oldest method

in pattern recognition, has been widely applied in face recognition systems;however, it is so sensitive to noises and variations But especially, this methoddoes not achieve a good performance if the training data set is limited Inpractical applications such as optical character recognition applications, NN-based approaches are the most powerful method; however, they easily fall intoover-fitting and are sensitive to noises and variations Recently, SVM-based

methods have become popular because they show a high level of robustness.However, the SVM method often requires a large training data set An alter-native approach, one that avoids the insufficient data problem, is the use ofsimilarity learning methods However, these methods have some weak points.

In this study, a new relation learning approach is proposed to deal with theinsufficient data problem It defines a relation between two objects and ap-plies this definition to determine all possible relations in the training data set.The relation between two objects is used to measure the similarity betweenthem Instead of considering objects in their data space as other methods do,

a relation space is used Details of this contribution are presented in Chapter6

7 Fusion Classifier: Face detection is the first important step in a face

recog-nition system It decides the quality of the system Most current favoredmethods used for face detection are based on the face detector built by Violaand Jones (2004) which solve three fundamental problems: firstly, learningeffective features from a very large set of features, secondly, constructing weakclassifiers, finally, boosting the weak classifiers to a strong classifier However,

to achieve high ratios of detecting faces, the system must increase the number

of weak classifiers or Haar-like features, and this can result in a significantincrease in the computation time Thus, to deal with the problem, the study

here uses the combination of AdaBoost and Artificial Neural Networks to still

achieve the same face detecting ratios but also the minimum computation time(Bui and Le,2008) Details of this contribution are presented in Chapter 7

1.7 Thesis Organization

This thesis report the above research It includes eight chapters organized as follows

In Chapter 2, the thesis review the literature on face detection and face nition It focuses on two main problems in face recognition, pose and illumination,and covers the developments from the early works to the state-of-the-art methods

recog-It also proposes a detailed solution to these problems

In Chapter 3, the thesis presents facial representation including the Gabor

wavelet and Local Binary Patterns that are particularly successful in face image

processing It also presents the Robust Multi Block Local Binary Pattern.

In Chapter 4, the thesis presents face detection methods in detail including

the AdaBoost and Artificial Neural Network It also presents a combination of two

methods to achieve the better results

In Chapter 5, Bayesian learning is discussed It covers, in brief, the tive approaches used in pattern classification, and focuses on one of the statisticallearning methods, Bayesian learning It also explains in detail enhanced Bayesianlearning in face classification

genera-In Chapter 6, Relation learning is presented It covers, in brief, the

discrimi-native approaches used in pattern classification, then presents the Support Vector

Machine and my Relation learning in face recognition.

In Chapter 7, the thesis presents the experiments carried out on face detectionand face recognition It covers the investigation and analysis of different facialrepresentations and different classification methods in face recognition

The final chapter presents conclusions and learning resulting from the study andrecommends research works that should be pursued in the future

Chapter 2

Literature Review

This chapter reviews a number of the important works on face recognition ported in the literatures They include studies in three principal areas: facial rep-resentation, face detection and face recognition A face recognition algorithm or aface detection algorithm has two components related to how to represent faces andhow to detect or identify faces The two components are not only dependent eachother but also affect each other.

re-In their major literature review on face recognition, Zhao et al (2003) fied face recognition algorithms in terms of facial representations into three majorapproaches: appearance-based, feature-based and hybrid approaches

classi-Appearance-based approaches These approaches use whole face images as one

facial representation and do not attempt to exploit any information about

struc-tures of faces Eigenpicstruc-tures (Kirby and Sirovich, 1990) and Eigenfaces (Turk andPentland,1991a) are the typical face recognition algorithms of this approach

Feature-based approaches These approaches try to exploit the structure

infor-mation of faces such as eyes, noses and mouths The local feature inforinfor-mation such

as positions or geometric properties is extracted and converted into feature vectorsthen fed to classifiers Kelly (1970) built the first automated face recognition sys-tem He used heuristic, goal-directed methods to measure distances of the bodyand head in normalized images, based on edge information Kanade (1973) thendeveloped the first automated face recognition system to use a top-down controlstrategy directed by a generic model of expected feature characteristics of the face.The system determined a set of facial parameters from a single face image includ-ing normalized distances, areas, and angles between fiducial points The systemused a pattern classification technique to match the face to one of a known set, apurely statistical approach that depended primarily on local histogram analysis andabsolute gray-scale values

Hybrid approaches These approaches use the information of both the whole face

and local features It is argued that these approaches could potentially create the

best face models because of their ability to complete facial representation

Modu-lar Eigenface (Pentland et al., 1994) is a typical example of these approaches; itcombines the global features and local features of faces