approaches to visual feature extraction and fire detection based on digital images

Recently, a new approach to automatic fire detection based on computer vision has lager attractive from researchers; it offers some advantages over the traditional detectors and can be u

MINISTRY OF NATIONAL DEFENCE

MILITARY TECHNICAL ACADEMY

THIS THESIS IS COMPLETED AT MILITARY TECHNICAL ACADEMY - MINISTRY OF NATIONAL DEFENCE

Scientific Supervisor: Assoc Prof Dr Dao Thanh Tinh

Reviewer 1: Assoc Prof Dr Nguyen Duc Nghia

Reviewer 2: Assoc Prof Dr Dang Van Duc

Reviewer 3: Assoc Prof Dr Nguyen Xuan Hoai

The thesis was evaluated by the examination board of the academy by the decision number / ., ./ / of the Rector of Military Technical Academy, meeting at Military Technical Academy on

… /… /………

This thesis can be found at:

- Library of Le Quy Don Technical University

- National Library of Vietnam

ABSTRACT

Automatic fire detection has been interested for a long time because fire causes large scale damage to humans and our properties Until now, some kinds of automatic detection devices, such as smoke detectors, flame or radiation detectors, or gas detectors, were invented Although these traditional fire detection devices have proven its usefulness, they have some limitations; they are generally limited to indoors and require a close proximity to the fire; most of them can not provide additional information about fire circumstance Recently, a new approach to automatic fire detection based on computer vision has lager attractive from researchers; it offers some advantages over the traditional detectors and can be used as complement for existing systems This technique can detect the fire from a distance in large open spaces, and give more useful information about fire circumstance such as size, location, growth rate of fire, and in particularly it is potential to alarm early

This research concentrated on early fire detection based on computer vision Firstly, some techniques that have been used for in the literature of automatic fire detection are reviewed Secondly, some of visual features of fire region for early fire detection are examined in detail, which include a model of fire-color pixel, a model of temporal change detection, a model of textural analysis and

a model of flickering verification; and a novel model of spatial structure of fire region Finally, three models of fire detection based

on computer vision at the early state of fire are presented: a model of early fire detection in general use-case (EVFD), a model of early fire detection in weak-light environment (EVFD_WLE), and a model of early fire detection in general use-case using SVM (EVFD_SVM)

CHAPTER 1 INTRODUCTION 1.1 Automated fire detection problems

Automatic fire detection has been interested for a long time because fire causes large scale damage to humans and our properties Heat or thermal detectors are the oldest type of automatic detection device, originating from the mid-19th century with several types still

in production today Since then, other kinds of automatic detection devices, such as smoke detectors, flame or radiation detectors, or gas detectors, were invented Although these traditional fire detection

devices have proven its usefulness, they have some limitations Despite the advances in traditional fire alarm technology over the last century, losses caused by fire, such as deaths, permanent injuries, properties and environment damages still increase In order to decrease this, timely detection, early fire localization and detection

of fire propagation are essential

The problem of fire detection based on computer vision was initialized in early 1990s by Healey G et al., since then various approach to this issue have been proposed However, vision-based fire detection is not a completely solved problem as in most computer vision problems The visual features of flames and smoke

of an uncontrolled fire depend on distance, illumination and burning materials In addition, cameras are not color and/or spectral measurement devices, they have sensors with different algorithms for color and illumination balancing, and therefore they may produce different images and video for the same scene So that most proposed methods in vision-based fire detection return good results in some conditions of use-case, and may give bad results in other conditions

In particularly, existing vision-based fire detection methods are not adequate attention to alarm early

1.2 Research objective

For all above reasons the author have studied the topic

“Approaches to visual feature extraction and fire detection based on digital images” with the main interest in the problem of vision-based

fire detection at the early state of fire Main question and also be

motivation for this research is can vision-based fire detection give a fire alarm as soon as possible at the early state of fire? This thesis

wants to find out the answers for that question in some different case such as general conditions, weak-light environment, camera is dynamic The objectives of this research include the following issues: 1) Firstly, some techniques that have been used for fire detection based on computer vision are reviewed 2) Secondly, some

use-of visual features use-of fire region such as color, texture, temporal change, flicker and spatial structure are examined in detail so that reducing the computational complexity of algorithm 3) Thirdly, some models of early fire detection based on computer vision are developed The development of each model relies on the analysis of the use-case such as for buildings and office surveillance, for

warehouse with weak light environment, etc It is also applying intelligent classification to make the models more suitable and accurate

1.3 Contributions

This thesis makes the following contributions:

1 Develop and propose some methods of visual features of fire region extraction Develop four new methods of pixel or fire region

segmentation, these include a method of fire-color pixel based on Bayes classification in RGB space, a method of temporal change detection, a method of textural analysis and a method of flickering verification; and propose a novel approach to spatial structure of fire

region by using top and rings features

2 Propose a model of vision-based fire detection for early fire detection in general use-case - EVFD This model is a combination

of temporal change analysis, pixel classification based on fire-color process, and the flickering verification

3 Propose a model of vision-based fire detection for early fire detection in weak-light environment - EVFD_WLE This proposal is

a combination of pixel classification based on fire-color process and analysis of spatial structure of fire region; these processes will be done if the environmental light is weak

4 Propose a model of vision-based fire detection for early fire detection in general use-case using SVM - EVFD_SVM In this

model, the algorithm consists of three main tasks: pixel-based processing using fire-color process for pixel classification, temporal change detection, and recover lack pixel; textural features of potential fire region extraction; and SVM classification for distinguishing a potential fire region as fire or non-fire object

1.4 Thesis outline

This thesis is organized as follows: Chapter 1, Introduction,

presents the need of problem of fire detection based on computer vision, disadvantages of traditional fire detection systems, and advantages of fire detection based on computer vision This chapter also describes problem of research, research question, main

contributions and structure of the thesis Chapter 2, Fire detection techniques based on computer vision: A review, review some

techniques that have been used for fire detection based on computer

vision Chapter 3, Visual feature extraction for fire detection,

presents examining in detail some of visual features of fire region for early fire detection; and then develops four new models of pixel or fire region segmentation and proposes a novel model of spatial

structure of fire region Chapter 4, Early fire detection based on computer vision, presents three models of fire detection based on

computer vision: early fire detection in general use-case, early fire detection in weak-light environment, and early fire detection in

general use-case using SVM Chapter 5, Conclusions and Discussions, states the conclusions, presents the contributions and

summarizes the results obtained throughout the thesis and recommendations future research of problem

CHAPTER 2 FIRE DETECTION BASED ON COMPUTER VISION: A REVIEW 2.1 Introduction

Automatic fire detection has been interested for a long time due

to its large scale damage to humans and our properties Heat or thermal detectors are the oldest type of automatic detection device originating from the mid-19th century Since then, other kind of automatic detection devices; smoke detectors, flame or radiation detectors, or gas detectors for examples have been being developed Although these devices have proven its usefulness in some conditions, they have some limitations They are generally limited to indoors and require a close proximity to the fire Most of them can not provide additional information about fire circumstances and may take a long time to raise alarm

Fire detection based on computer vision can be marked by the research of Healey G et al in the early 1990s Since then, various approaches to this issue were proposed The general scheme of fire detection based on computer vision is a combination of two components: the analysis of visual features and the classification techniques The visual features include color, temporal changes, spatial variance, texture and flickering The classification techniques are used to classify a pixel as fire or as non-fire, or to distinguish a potential fire region as fire or as non-fire object; these techniques include Gaussian Mixture Model (GMM), Bayes classification, Support Vector Machine (SVM), Markov Model and Neural Network, etc

2.2 Visual features analysis

2.2.1 The chromatic color

Color detection is one of the most important and earlier feature used in vision-based fire detection The majority of the color-based approaches in this trend make use of RGB color space, sometimes in combination with HSI/HSV color space Some fire-color models often use in the literature of vision-based fire detection such as statistical generated color models, Gaussian Mixture Models (GMM) Based on the analysis of color of flame in red-yellow rang,

a common type of flame in the real-word, a fire-color model to segment a pixel is proposed as follows:

fire-d(r1, g1, b1, r2, g2, b2) is the measurement distance from (r1, g1, b1) to

(r2, g2, b2) in 3-dimensional RGB space The fire-color model based

in which Ri, Gi, Bi are the mean of red, green blue components of

Gaussian distribution i-th; vi is its standard deviation

2.2.2 The temporal changes

Color model alone is not enough to identify fire pixel or fire region There are many objects that share the same color as fire An important visual feature to distinguish between fire and fire-like objects is the temporal change of fire To analyze temporal changes,

it may cause by flame, almost proposals assume that the camera is

stationary A simple approach to estimate the background is to

average the observed image frames of the video Let I(x,y,n) represent the intensity value of the pixel at location (x,y) in the n-th frame, I, background intensity value, B(x,y,n+1) at the same pixel

position is calculated as follows:

( , , ) (1 ) ( , , ) if ( , ) is stationary ( , , 1)

( , , ) ( , , 1) ( , , )

I x y n I x y n T x y n where I(x,y,n-1) is the intensity value of the pixel at (x,y) in the (n-1)-th frame, T(x,y,n) is a recursively updated threshold at (x,y) of frame n Other method

usually used to analysis temporal changes i s frames difference

2.2.3 The textural and spatial difference

Flames of an uncontrolled fire have varying colors even within a small area since spatial color difference analysis focuses on this characteristic Using range filters, variance/histogram analysis, or spatial wavelet analysis, the spatial color variations in pixel values is analyzed to distinguish between fire and fire-like object Using

wavelet analysis, Toreyin et al compute a value, v, to estimate

spatial variations as follows:

sub-is likely that thsub-is region under investigation sub-is a fire region In other way, Borges et al use a well-known metric, the variance, to indicate the amount of coarseness in the pixel values For a potential fire

region, R, the variance of pixels is computed as

2 ( , )x y R ( , ) ) ( ( , )

c  I x y I p I x y

in which I(x,y) is intensity of pixel at (x,y), p() is the normalized

histogram, and I is the mean intensity in R Therefore, fire is assumed if the region is with a variance c > λσ, where λσ is

determined from a set of experimental analysis

2.3 Classification techniques

Some popular approaches to the classification of the dimensional feature vectors obtained from each candidate flame region are Bayes classification and SVM classification Other classification methods that have been used in the literature of vision-based fire detection include neural networks, Markov models, etc This section introduces two classification methods that used in the research: Bayes and SVM classification

multi-2.4 Conclusion

The development of application based on computer vision for fire detection, which can raise alarm quickly and accurately, is essential However, vision-based fire detection is not a completely solved problem as in most computer vision problems The visual features of flames of an uncontrolled fire depend on the distance, illumination and burning materials In addition, cameras are not color and/or spectral measurement devices, they have sensors with different algorithms for color and illumination balancing, and therefore they may produce different images and video for the same scene For the above reasons, the research of vision-based fire detection is necessary

In general, most proposed methods in vision-based fire detection returns good results in some conditions of use-case, and may give bad results in other conditions In particularly, current vision-based fire detection methods are not adequate attention to alarm early so that research of vision-based fire detection is necessary, and using this technique for early fire detection is an important issue

CHAPTER 3 VISUAL FEATURE EXTRACTION FOR FIRE DETECTION

This chapter presents the examining in detail some of visual features of fire region for early fire detection; and then develop four new models of pixel or fire region segmentation, these include a model of fire-color pixel, a model of temporal change detection, a model of textural analysis and a model of flickering verification; and propose a novel model of spatial structure of fire region

3.1 A new approach to color extraction

3.1.1 Chromatic analysis

The model of fire-color is usually used in the first step of the process and is crucial to the final result The general idea of most proposals in the VFD literature is to determine the fire-color model,

Fire(x, y) for pixel at (x,y), and then using that model to build the potential fire mask, PFM(x,y), as follows:

1 if ( , )( , )

0 Otherwise

Fire x y PFM x y  



After that the mask PFM is used to analyze the other characteristics

of fire such as temporal changes, deformation of the boundary, surface statistical parameters, etc The main drawback of existing model for fire-color detection is fixed; it returns good results in some situations and raise bad results in some others For more flexible, t his study proposes a color model of pixel in fire region using Bayesian classification; rely on the red (R), green (G), and blue (B) components a model of fire-color to classify a pixel into two classes,

fire or non-fire pixel is developed

3.1.2 Classification based on Bayes

For pixel p at (x,y), a vector v = [R, G, B]T is considered in terms

of sample for classification problem; in which R, G, and B are red, green and blue component of p Let g1(v) and g2(v) are two

discriminatory functions based on Bayesian classification for fire and non-fire classes of pixel p; if g1(v)>g2(v) then p belong to fire class, otherwise p belong to non-fire class Denote 1 is set of fire class samples, 2 is set of non-fire class samples, Bayessian discriminatory functions are defined as follows:

where m1 is mean and C1 is covariance matrix of 1 and m2 is mean

and C2 is covariance matrix of 2 Then fire-color model, denote

g1=-.68e-3*R*R+.11e-2*R*G-.44e-3*R*B-.82e-3*G*G

+.98e-3*G*B -.47e-3*B*B+.16*R-.92e-1*G+.33e-1*B-23

g2=-.73e-3*R*R+.26e-2*R*G-.13e-2*R*B-.39e-2*G*G

+.52e-2*G*B-.20e-2*B*B+.30e-1*R-.98e-2*G+.56e-2*B-12 The results of training with prepared samples of group 2:

g1=-.42e-3*R*R+.13e-2*R*G-.56e-3*R*B-.14e-2*G*G+.12e-2*G*B -.41e-3*B*B+.46e-1*R-.44e-1*G+.63e-1*B-17

g2= -.98e-3*R*R+.26e-2*R*G-.10e-2*R*B-.35e-2*G*G

+.46e-2*G*B-.20e-2*B*B+.37e-1*R-.16e-1*G-.37e-2*B-12 The results of training with prepared samples of group 3:

g1=-.19e-2*R*R+.38e-2*R*G-.10e-2*R*B-.40e-2*G*G+.44e-2*G*B -.17e-2*B*B+.29*R-.16*G+.17e-1*B-28

Chen Hor ng Tor eyin Celick Bor ger Color F

Figure 1 The number of misclassified pixels in comparison with ColorF

3.1.3 Experiments

For comparing and testing, the author perform the experiment of color segmentation with the model proposed by T Celik et al., the model proposed by P V K Borges et al., the model of T H Chen et al., the model proposed by W B Horng et al., the model proposed

by B U Toreyin et al., and the model proposed in this study - denote

ColorF The chart in Figure 1 represents the sum of misclassified pixels on tested images of each model The chart shows that the

proposed model - ColorF, and Chen have the lowest of total number

of misclassified pixel

3.2 A new approach to temporal change, texture and flicker extraction

3.2.1 Temporal changes analysis

In order to detect temporal changes, which may be caused from fire, it is necessary to use an effective background-modeling algorithm Temporal change caused by fire is usually slow, so that the existing methods of motion detection, such as background subtraction or frames difference, are often inefficient In this research, temporal change is estimated on regions by regions between two consecutive frames by using correlation-coefficient

The correlation-coefficient between two regions A and B is computed

scheme of temporal change detection between two consecutive

frames, I and J, in this approach is described as follows:

1 Divide I and J into MN regions, denote Ik and Jk for k = 1, , MN (Figure 2);

2 Calculate correlation-coefficient between k-th region Ik and

corresponding region Jk, and then assign CC(Ik,Jk) to CHk

1200 frames are used For comparison, frames difference model, background subtraction model, and proposed model are

implemented The evaluation of time performance is shown in Table

1, and the quality of temporal change detection is shown in Figure 3

Figure 2 The scheme of partition of two frames for temporal analysis

Method Time performance per frame (Milliseconds)

Table 1 The comparison of time performance

Figure 3 An example results of three temporal change detection techniques

Figure 4 The ROC curve of temporal changes detection

Figure 4 shows the ROC (Receiver Operating Characteristic)

curve of temporal changes detection for threshold T Rely on this evaluation, when the threshold T = 0.025 then true positive fraction

equal to 95% and false positive fraction is 6%

d)