Develop a low cost thermal camera using 2d camera and thermal sensor

- Investigate the related works combining a 2D-camera and a thermal sensor to detect human face in an image and estimate body temperature.. Despite all of the advantages this technology

Faculty of Computer Science and Engineering

Graduation Thesis

Develop A Low-cost Thermal Camera Using 2D

Camera and Thermal Sensor

Major: Computer Engineering

Committee: Council 6 Supervisor: Pham Hoang Anh, Ph.D

Reviewer: Le Trong Nhan, Ph.D

Î 1752541

ĐẠI HỌC QUỐC GIA TP.HCM CỘNG HÒA XÃ HỘI CHỦ NGHĨA VIỆT NAM

- Độc lập - Tự do - Hạnh phúc

TRƯỜNG ĐẠI HỌC BÁCH KHOA

KHOA:KH & KT Máy tính _ NHIỆM VỤ LUẬN ÁN TỐT NGHIỆP

BỘ MÔN: _ Chú ý: Sinh viên phải dán tờ này vào trang nhất của bản thuyết trình

HỌ VÀ TÊN: MSSV:

HỌ VÀ TÊN: MSSV:

HỌ VÀ TÊN: MSSV: NGÀNH: _ LỚP:

1 Đầu đề luận án:

2 Nhiệm vụ (yêu cầu về nội dung và số liệu ban đầu):

3 Ngày giao nhiệm vụ luận án:

4 Ngày hoàn thành nhiệm vụ:

5 Họ tên giảng viên hướng dẫn: Phần hướng dẫn:

1) 2) 3) Nội dung và yêu cầu LVTN đã được thông qua Bộ môn

Ngày tháng năm

CHỦ NHIỆM BỘ MÔN GIẢNG VIÊN HƯỚNG DẪN CHÍNH

PHẦN DÀNH CHO KHOA, BỘ MÔN:

Người duyệt (chấm sơ bộ):

1752541 1752295

Computer Engineering

Develop A Low-cost Thermal Camera Using 2D Camera and Thermal Sensor

- Study principal operation of existing thermal cameras.

- Investigate the related works combining a 2D-camera and a thermal sensor to detect human face in an image and estimate body temperature.

- Investigate existing hardware devices to propose a solution and detail design

- Develop a prototype and software applications for evaluation the proposed approach.

TS Phạm Hoˆng Anh PGS.TS Phạm Quốc Cường

-

Ngày 06 tháng 08 năm 2021

PHIẾU CHẤM BẢO VỆ LVTN

(Dành cho người hướng dẫn/phản biện)

1 Họ và tên SV: Trần Đình Tiến, Nguyễn Việt Khoa, Nhan Ngọc Thiện

MSSV: 1752541, 1752295, 1752508 Ngành (chuyên ngành): Kỹ thuật Máy tính

2 Đề tài: Develop A Low-cost Thermal Camera Using 2D Camera and Thermal Sensor

3 Họ tên người hướng dẫn/phản biện: Phạm Hoàng Anh

4 Tổng quát về bản thuyết minh:

- Số bản vẽ vẽ tay Số bản vẽ trên máy tính:

6 Những ưu điểm chính của LVTN:

- The students have demonstrated their excellent capability in self-studying and investigating related works, and then they apply new knowledge and suitable techniques to implement the proposed system

- The students have performed various experiments to evaluate the system, and the experimental results have shown that the proposed solution meets all requirements in this thesis’s scope

- The students have also developed a full IoT-based system that is ready and applicable to deploy the system in practice, including hardware, embedded software, IoT server, and Web-based application

- The proposed solution has been accepted and presented at an international conference where its proceeding will be indexed in Scopus

- The report has been well organized and written

7 Những thiếu sót chính của LVTN:

- The students should improve their presentation

8 Đề nghị: Được bảo vệ o Bổ sung thêm để bảo vệ o Không được bảo vệ o

9 Câu hỏi SV phải trả lời trước Hội đồng:

" VT姶云PI"A萎K"J窺E"DèEJ"MJQC" E浦PI"JñC"ZÊ"J浦K"EJ曳"PIJ C"XK烏V"PCO"

/"Vjg"vjguku"tgrqtv"ku"kortguugf"d{"vjg"gzcokpgt0"Vjku"tgrqtv"ecp"dg"wugf"cu"c"iqqf"tghgtgpeg"hqt"hwvwtg"rtqlgevu."eqpegtpkpi"vjg"hceg"fgvgevkqp"qp"Lgvuqp"Pcpq"dqctf0"

320"A pj"ik "ejwpi"*d茨pi"ej英<"ik臼k."mj "VD+<" Ak吋o"<"32132"

" """"M#"v‒p"*ijk"t "j丑"v‒p+"

"

"

We commit that our topic “Develop A Low-cost Thermal Camera Using 2DCamera and Thermal Sensor” is our thesis We declare that this topic is conductedunder our effort, time, and the recommendation from Dr Pham Hoang Anh asour advisor.

All of the research results are conducted by ourselves and not copied fromany other sources If there is any evidence of plagiarism, we will be responsible forall consequences

Ho Chi Minh City, 2021,Tran Dinh Tien, Nguyen Viet Khoa, Nhan Ngoc Thien

First and foremost, we would like to show our deep and grateful gratitude

to Dr Pham Hoang Anh as an adviser for our thesis and as a beloved teacher

We want to show all of our respect for the precious knowledge, advice, and all thenecessary equipment that he has given to us We sincerely appreciate his kindness,patience, and continuous support throughout this thesis

Next, we would like to show our appreciation to Dr Le Trong Nhan forgiving us advice on which devices and components to choose that are most suitablefor the scope of our thesis We also want to show appreciation to Assoc Prof QuanThanh Tho for all of his advice and comments about our thesis proposal

Moreover, we wish to express our appreciation to all our faculty lecturersfor teaching the authors all the fundamental knowledge and necessary skills andknowledge in each field, without whom we would not have enough knowledge andskills to complete this thesis

Last but not least, we would like to acknowledge the support from ourfamily and classmates in many aspects without hesitation—all of your assistance

in helping us

Despite our commitment, we are aware that the project is still inadequateand contains inevitable errors Therefore, we are looking forward to hearing feedbackfrom lecturers to remove all the mistakes and improve the project furthermore

Engineering - Ho Chi Minh City University of Technology - Viet Nam NationalUniversity Ho Chi Minh City.

Students

The Covid-19 pandemic is spreading worldwide, the demand for controllingand screening fever is increasing very rapidly Some old methods like using ther-mometers or medical checking cannot handle many people in crowded areas such

as airports, train stations, or buildings Some thermographic systems or thermalimaging systems control the people’s ins and outs These kinds of systems arehelping a lot for the government of any nation to decrease and prevent the spread ofviruses by screening human temperature based on their radiation of skin Despiteall of the advantages this technology has had, the exceptionally high cost for thissystem is one of the reasons to prevent some organizations from applying it

Despite all of the advantages this technology has had, this system’s veryhigh cost is one reason to prevent some organizations from applying it despite thehigh potential of image processing, artificial intelligence, and the Internet of Things.This paper will present a system combining a low-cost thermal camera and astandard RGB camera for screening fever combined with attendance checking Theexperimental results show that this proposed system can be applied in many areas,from universities to buildings, with the minimum cost and acceptable performance

2.1 Investigation of Similar Studies 4

2.1.3 Contactless Vital Signs Measurement System Using

RGB-Thermal Image Sensors and Its Clinical Screening Test on

Patients with Seasonal Influenza 6

2.2 Background and Methodologies 7

2.2.1 Thermal Camera Principles 7

2.2.2 Face Detection Methods 8

2.2.3 Facial Landmarks Detection 11

2.2.4 Camera Registration 11

2.2.5 Object Tracking 14

2.2.6 Thermal Camera Calibration 14

2.2.7 Face Recognition Methods 16

3 System Design 21 3.1 Solution Description 21

3.2 Proposed System Architecture 22

3.3 Devices and Components 26

3.3.1 Hardware Components 26

3.3.2 Software Components 32

3.4 Database Design 33

3.4.1 Requirement 33

3.4.2 Entity Relation Diagram 34

3.4.3 Relational Model 35

3.4.4 Database Data Type & Constraint 36

3.5 Software Application Design 38

3.5.1 Device Application 38

3.5.2 Web Application 47

4.1 Thermal Camera System 60

4.1.1 Overall Operation 60

4.1.2 Face Detection 61

4.1.3 Transformation Matrix Determination 62

4.1.4 Thermal Camera Calibration and Measurement 65

4.1.5 User Registration 70

4.1.6 Person Tracking 70

4.1.7 Wi-Fi Connection 71

4.1.8 Offline Mode 71

4.1.9 Device Application UI 71

4.1.10 Calibration screen 76

4.2 Server AI 77

4.2.1 Load Models 77

4.2.2 Face Recognition 78

4.2.3 User Registration 80

4.2.4 Mask Detection 81

4.3 IoT Platform 81

4.3.1 Gateway 81

4.3.2 Store Device’s Recognition and Registration Request 81

4.3.3 Handle Login Into Device 82

4.3.4 Store Records 82

4.3.5 Check Device’s Connectivity 83

4.4.4 Device Management 86

4.4.5 User Management 88

4.4.6 Building Management 90

4.5 Evaluation 92

4.5.1 Evaluation Metrics 92

4.5.2 Experimental Scenarios and Results 93

4.5.3 Comparison With Existing Systems 97

5 Conclusion 99 5.1 Main Contributions 99

5.2 Achievement 100

5.3 Future Plan 101

5.4 Challenges and Risks 101

A Set Up Guide and How To Use 103 A.1 Device Set Up 103

A.1.1 Camera Hardware Set Up 103

A.1.2 Connection 104

A.1.3 Jetson Nano Configuration 105

A.1.4 Jetson Nano Software Set up 107

A.2 Azure Set Up 110

A.2.1 Create Resource Group 110

A.2.2 Create An IoT Hub 114

A.2.3 Register IoT Device 120

A.2.4 Create Service Bus Resource 123

A.2.5 Create Blob Storage 126

A.2.6 Create Function App 129

A.2.7 IoT Hub Message Routing 135

A.2.8 Set Up Database 141

A.3 Project Configuration 146

A.3.1 Get Service Keys 146

A.3.2 Tools Set Up 152

A.3.3 Device Configuration 156

A.3.4 Server AI Configuration 157

A.3.5 Function App Configuration 158

A.3.6 Web Application Configuration and Deployment 165

A.4 User Manual 170

A.5 Device 170

A.5.1 Notification 170

A.5.2 Main Function 171

A.5.3 Activate/Exit Device 171

A.5.4 WiFi Connect 172

A.5.5 Setting Device 173

A.5.6 Calibrate Device 173

A.5.7 Check Connection 174

A.6 Web Application 174

List of Figures

2.1 Image registration with manual mapping 5

2.2 Matching procedure 7

2.3 Spectre wavelength 8

2.4 Three coordinate systems in the computing process 13

2.5 Face recognition pipeline 16

2.6 Google FaceNet pipeline 17

3.1 A proposed system architecture 23

3.2 Jetson nano 27

3.3 AMG8833 27

3.4 Flir Lepton Camera 28

3.5 Camera output vs Camera temperature when radiometry mode enabled 29

3.6 Camera output vs Camera temperature when radiometry mode disabled 29

3.7 Raspberry Pi Camera 30

3.8 Thermal camera system propotype 31

3.9 Database Entity Relation Diagram 34

3.10 Database Relational Model 35

3.11 Use case diagram of thermal camera system 40

3.12 Themal camera system normal flow diagram 44

3.13 Themal camera system user registration flow diagram 45

3.14 Thermal camera system temperature calibration flow diagram 46

3.15 Use case diagram of web application 49

3.16 Login & Register Sequence Diagram 54

3.17 Dashboard Sequence Diagram 55

3.18 Record Sequence Diagram 56

3.19 Device Sequence Diagram 57

3.20 User Sequence Diagram 58

3.21 Building Sequence Diagram 59

4.1 Thermal camera system overall diagram 61

4.2 Transformation matrix determination procedure 64

4.3 Temperature measurement according to different distances 65

4.4 Temperature offset according to different distances 66

4.5 Thermal camera calibration procedure 68

4.6 Wifi device UI 72

4.7 Wifi device on-the-fly UI 72

4.8 Activate device UI 73

4.9 Setting device UI 74

4.10 Main UI 75

4.11 Fullscreen RGB UI 75

4.12 Calibrate thermal UI 76

4.13 Register UI 77

List of Figures

4.18 Main dashboard page 84

4.19 Date picker UI 85

4.20 Live record UI 85

4.21 History record UI 86

4.22 Web displays overview of the devices management 86

4.23 Web displays device’s information and setting options 87

4.24 Web displays the delete device option 87

4.25 Web displays the add new device option 88

4.26 Web displays overview of users management 88

4.27 User management detail 89

4.28 User management add form 89

4.29 Web displays overview of buildings management 90

4.30 Web displays overview of buildings management 91

4.31 Web displays overview of buildings management 91

4.32 The output frame of AMG8833 thermal camera: a) Input RGB frame b) Output thermal frame 94

4.33 Face detection using SSD with ResNet-10 model 94

4.34 Face detection using RFB Net mode 95

4.35 a) Input image, b) PnP transform, c) Affine transform, d) Homog-raphy transform 95

4.36 a) Detected bounding boxes in RGB frame b)Transformed bounding boxes in thermal frame 96

4.37 Experimental results of temperature measurement 97

A.1 Flir breakboard v2 schematic 103

A.2 Flir breakboard pin 104

3.1 Hardware Specification 31

3.2 Use-case tabular of thermal camera system 41

3.3 Use case tabular of web application 50

4.1 Regression statistic 97

4.2 FPS Evaluation (unit: frames/second) 98

4.3 Specification Comparison 98

SVM Support Vector Machine

HOG Histogram of Oriented Gradients CPU Central Processing Unit

GPU Graphics Processing Unit

RANSAC RANdom SAmple Consensus K-NN K-Nearest Neighbors

ROI Region Of Interest

IR Infrared

SSD Single Shot Detection

1.1 Purpose and Motivation

For predicting human body temperature, different face regions such as theforehead, nose, ears, cheeks can be considered Because the face is usually notcovered by clothing items and temperatures from different areas of the face canpredict thermal sensations, recent studies have utilized infrared thermography as

an alternative to monitoring skin temperatures non-intrusively

Since then, thermal imaging techniques have been widely applied for ing temperature in the global outbreak era for supporting initial diagnosis andisolate suspects with typical symptoms of fever, influenza-base, and rise of bodytemperature [1] The advantages of infrared imaging devices, including non-invasiveand time-saving compared to the other traditional thermal measuring methods,lead to high demand for such devices

screen-Most commercial thermal cameras easily measure human temperature andskin temperature However, these cameras are not suitable for small and moderatescale applications because they are operated manually while automatic cameras’

1.1 Purpose and Motivation

The recent Covid-19 pandemic is spreading worldwide, so it is essential

to have a system that helps prevent the pandemic and ensure the safety of theenvironment and the security inside the residences, schools, and industrial areas bymonitoring and detecting abnormal human temperature early Therefore, it inspires

us to develop an IoT-based thermal camera system using low-cost off-the-shelfhardware devices to deploy in small to medium-scale facilities for safety and securitypurposes such as fever screening and surveillance

In a cloud computing system, all data are centralized and processed oncloud servers Initially, devices connect to the IoT platform and send raw telemetrydata for the server post process The cloud is storing all data and information

on cloud assets The cloud system also can handle authentication for users anddevices However, cloud computing depends entirely on the internet connection

to keep the data transmission stable Therefore, the device only works when theinternet connection is maintained continuously

IoT devises data can now be processed at edge devices with the rise of edgecomputing technology before sending to a cloud server This technology can helpincrease response time, reduce latency, and edge devices can backup data whenthere is no internet connection However, there are also some drawbacks to edgecomputing technology It cannot be as powerful as a cloud server and can increasethe cost for each device due to expanding storage and processors

Our system combines cloud computing and edge computing methods toutilize the microcontroller’s resources and make the system flexible For example,suppose there is an internet disconnection In that case, the device will not besuspended entirely It can continue to execute tasks, and the stored data will

be sent back to the server when the internet is available again Models and jobsthat need many computation resources will be handled by the server so that themicro-controller will not be overloaded; therefore, the response time will be severelyaffected

1.2 Scope and Objectives

Although IoT-based thermal camera system has many applications, such asdetecting gas and electricity leakage, detecting critical points, airport surveillance,etc., our thesis scope mainly focuses on healthcare and disease control problems bymonitoring and screening the temperature on human skin

Our thesis objective is to build an end-to-end thermal camera system Thedevice can screen human temperature, recognize the identity, and detect face masks,running in both offline and online mode A backend server is powered by a cloudIoT platform for scalability reaching the actual production Management webapplication is designed and implemented with a friendly experience for all users

1.3 Thesis report structure

The remainder of this report is organized following Chapter 2 brieflypresents our related works, including existing similar studies, background knowledge,and methodologies that have been applied in our proposed solution Then, wepropose an overall system design including hardware, software, and IoT-basedarchitecture in Chapter 3 Our detailed implementation and the evaluation ofthe proposed system will be presented in Chapter 4 Moreover, in Chapter 4, wesummarize some remarkable results that we archive throughout the thesis phaseand compare the results produced from different proposed methods to choose thebest possible methods

2 Related Works

In this chapter, we briefly present some related works which have beeninvestigated Then, we present background knowledge and methodologies havingbeen applied in our study

2.1 Investigation of Similar Studies

2.1.1 Skin temperature extraction using facial landmarks

detection and thermal imaging for comfort ment

assess-The authors in [15] presented a method to calculate the facial temperatureaccurately by extracting the facial landmarks from an RGB camera image andprojecting those coordinates into a high-resolution thermal camera image Amethod is called ROI Temperature Extraction

For the image registration between RGB and thermal images, homographytransformation is applied in the following steps The low-resolution thermal images(80x60) will be interpolated into the image with higher resolution (800x600) Theauthor also made a note for using interpolation images only for the registrationstep, not for observing thermal value Initially, the system is manually set up byselecting the corresponding control points to generate the homography matrix; thesetup steps only need to be performed once [15]

After the image registration, facial landmark detection is applied to identifyfacial landmarks to calculate the desired ROI Then the computed ROI is scaleddown to match the original thermal image The original thermal image of 60 x 80pixels consists of direct temperature measurements Hence the temperatures of thelocated ROIs are extracted from the original thermal image [15].

Figure 2.1: Image registration with manual mapping

Our approach of implementing the camera registration process is based onthe paper’s method, manually choosing points in the RGB frame and correspondingpoints in the thermal frame to compute the transformation matrix The procedure

is performed one time in the setup process of the system However, the temperaturemeasuring method in the paper is different since the paper’s temperature measuringmethod is based on factory calibration, which can cause temperature drift due tospecific environment infrared emissivity

2.1.2 Mobile-platform for Automatic Fever Screening

Sys-tem based on Infrared Forehead Temperature

The authors in [16] proposed a method for image registration between thethermal camera and the RGB camera called image alignment, where the distanceand space between two cameras need to be carefully considered and adjusted to get

2.1 Investigation of Similar Studies

and get the temperature at that position Moreover, the authors experiment toevaluate the accuracy of the temperature measuring method by determining thecorrelation between the temperature measured by the infrared thermometer andthe temperature measured by the system

Our method of measuring the temperature is similar to the proposed proach, which gets the position of the face area with the highest temperature.Moreover, we propose a calibration step to reduce the error when measuring thetemperature at a long distance since distance affects the temperature measuringaccuracy We determine what metrics should be considered when setting up thedevices besides distance

ap-2.1.3 Contactless Vital Signs Measurement System Using

RGB-Thermal Image Sensors and Its Clinical ing Test on Patients with Seasonal Influenza

Screen-In this research [17], Toshiaki Negishi and his colleagues proposed a methodfor screening and detecting influenza based on human temperature and other vitalsigns The system consists of an RGB camera and a high-resolution thermal camera.The image registration was done by collecting corresponding points of two imageswith feature extraction using the BRIEF (ORB) method and matching the featurepoints The homography matrix is calculated using these points for finding relativepositions between two different camera views We are applying facial landmarksdetection for calculating the thermal value at specific points and getting the overalltemperature of humans To conduct the feature extraction such as BRIEF andSIFT, the thermal camera resolution must be substantial to match the RGB imagefeatures, and thereby, the system’s cost is very high

Figure 2.2: Matching procedure

2.2 Background and Methodologies

A thermal imaging camera is a thermal imager that is essentially a heatsensor capable of detecting tiny temperature differences The device collectsinfrared radiation from objects in the scene It creates electronic images based oninformation about the temperature differences, resulting in infrared images thatare capable of showing you what the naked eye or visible light camera cannot [5]

2.2 Background and Methodologies

Figure 2.3: Spectre wavelength

thermal camera systems will follow the principle of Stefan - Boltzmann law:

Where ε is called the emissivity (0 to 1), and human skin will have an emissivity around 0.95 σ is called Stefan - Boltzmann constant, Ais surface area, Tis the

body temperature, and Π is the radiative power

Because the Stefan - Boltzmann law uses temperature There is a conversion

of temperature to voltage and vice versa:

T obj= (V

The V above represents the voltage measured by the raw sensor; the variable

k is an empirical constant that absorbs the A, ε, σ, and electronic noise may exist The T s is the temperature of the sensor itself, and the remaining T obj is thetemperature of the object being measured

Some face detection methods are proposed in the thesis for comparison withsome different metrics, such as accuracy and inference time

2.2.2.1 Haar Cascade

In the paper “Rapid Object Detection using a Boosted Cascade of SimpleFeatures” by Paul Viola and Michael Jones in 2001, Haar Feature-based CascadeClassifiers is an effective object detection method which is a machine learning-basedapproach in which a cascade function is trained from a lot of positive and negativeimages [6] It is then used to detect objects in other images

Initially, Cascade classifiers are trained on a few hundred sample images ofimages that contain the faces to detect (positive images) and other images that

do not contain faces (negative images) Each feature is a single value obtained bysubtracting and the sum of pixels under the white rectangle from the sum of pixelsunder the black rectangle [7] Features with minimum error rates are selected,which means they are the features that best classify the face and non-face images

To increase more time to check for possible face regions, the concept ofCascade of Classifiers is introduced which the features are grouped into differentstages of classifiers If a window fails the first stage, discard it If it passes, applythe second stage of features and continue the process [7] The window which passesall stages is a face region

2.2 Background and Methodologies

classification of images, handwriting recognition Histogram of Oriented Gradients

or HOG is a feature descriptor used for object detection A HOG relies on theproperty of objects within an image to possess the distribution of intensity gradients

or edge directions

The descriptors are calculated over blocks of pixels with 8x8 dimensions.These descriptor values for each pixel over 8 x 8 block are quantized into 9 bins,where each bin represents a directional angle of gradient and value in that bin,which is the summation of the magnitudes of all pixels with the same angle TheSVM model is trained using a number of HOG vectors for multiple faces

• Advantages:

– Fastest method CPU.

– Works very well for frontal and slightly non-frontal faces.

• Disadvantages:

– The major drawback is that it does not detect small faces as it is trained

for a minimum face size of 80x80

– The bounding box often excludes part of the forehead and even part of

the chin sometimes

– Does not work for side face and extreme non-frontal faces, like looking

down or up

Deep Neural Network face detection method is based on the neural networkarchitecture as a backbone model such as Mobile Net, ResNet, etc To extractthe features of faces, they then generated them into pre-trained files for inference.Usually, these methods are very accurate However, the inference time is muchlonger due to complex computation

• Advantages:

– Fast and accurate on GPU.

– Work well on side faces and small faces.

• Disadvantages:

– Very slow on CPU and embedded board.

– Hard to choose best model for the project.

2.2.3 Facial Landmarks Detection

Facial landmarks detection localizes and represents salient regions or facialparts of the person’s face, such as eyes, jaws, nose, and eyebrows There are severalmodels for detecting facial landmarks and regions, from traditional methods likeHaar Cascade to the deep learning method

The application of detecting facial landmarks can be used to estimate headposes, checking face mask detection Our system applies the method to detectwhether the user is wearing a mask or not to send warnings and notify the user toput on a mask Furthermore, facial landmarks can estimate the user’s head posesfor the automatic registration phase

The method of detecting key facial structures in the face region involves using

a training set of labeled facial landmarks on an image, specifying the coordinates ofregions surrounding each facial structure [8] Given this training data, an ensemble

of regression trees are trained to estimate the facial landmark positions directlyfrom the pixel intensities themselves; in other words, no “feature extraction” istaking place [8]

2.2.4 Camera Registration

The thermal camera and RGB camera are placed at different angles; each

2.2 Background and Methodologies

There are some popular transformation methods to compute the mation matrix which are proposed in the thesis

Affine transformation is basically the combination between linear

transfor-mations and translation Let (xÕ, yÕ) be a point’s coordinate in the thermal image,

and (x, y) is the corresponding point’s coordinate in the RGB image Then, the

transformation matrix can be determined from Eq 2.3, in which there are 6 degrees

of freedom Therefore, there must be exactly three pairs of 2D coordinates fromtwo different images to calculate the transformation matrix

correspondence will provide 2 equations; therefore we need a minimum of fourfeature correspondences to estimate the Homography matrix.

Perspective-n-Point Transformation projects 2D coordinates from the RGBcamera to the 3D coordinates and back to 2D coordinates of the thermal camera if

a rotation matrix and a translation vector are known [11]

Figure 2.4: Three coordinate systems in the computing process

The presentation of three coordinate systems is represented in Fig 2.4

Let us assume the location of a 3D point P in World Coordinates is (U, V, W ) If the rotation R (a matrix of 3x3) and translation t (a vector of 3x1) of the world

coordinates are identified We can append the translation vector with the rotation

matrix to form the projection matrix The location (X, Y, Z) of the point P in the

camera coordinate system can be calculated using the following equation

2.2 Background and Methodologies

In expanded form, the Eq 2.6 look like below:







X Y Z

• Single Object Tracking(SOT):

Initially, the bounding box or ROI of the object is given at the first frame.The tracking algorithm will determine the position of the same target in thenext frames

• Multiple Object Tracking(MOT):

Different from single-object tracking, the bounding boxes are provided andthe identity of each object One famous challenge in MOT is the ID swapwhen an object is overlapped on another object

Calibrating a thermal camera is the process of checking what the camera sees(infrared radiation) with known temperatures so that the camera can accuratelymeasure the radiation it detects and convert it to truth temperature

Although all Flir cameras are factory calibrated with high technology ment for ensuring the most accurate measurement, over-usage time and the effectsfrom various environments can cause the calibration shift Therefore, the outputtemperature can have an inaccurate result

equip-2.2.6.1 Flir Calibration Procedure

According to Flir datasheet [12], for the best performance of radiometrywhen integrates Flir camera to the whole system, additional calibration is neededdue to some factors such as scene emissivity and unwanted signal from outsideenvironments

Requirements:

• Two black body with known temperature and emissivity

• Software interface to communicate with Lepton over i2c

• Video to view VoSPI

Procedure: All the required steps are listed in Flir Radiometry Quick

Starts Datasheet The additional notes below are very important, which canstrongly impact the calibration result

1 Before any measurement step, make sure the camera is turned on and set tomanual FFC (Flat Field Calibration) mode

2 Right after the camera is turned on, the FFC step will occur, causing thetemperature to rise Please wait from one to two minutes, then captureabout ten consecutive frames to ensure the temperature is stable before anymeasurement step

The above calibration method can boost up the reliability of the accuracy of

a thermal camera However, the procedure is very complicated with the requirement

2.2 Background and Methodologies

2.2.7 Face Recognition Methods

In order to build a face recognition model, there are included three steps:

- To apply face detection, get the location of the unidentified face in a framewith multiple methods

- Use different neural networks to extract features from the face into 128dimension vectors (called “embeddings”)

- Finally, use some other machine learning algorithms (K-NN, SVM, ) toidentify which embedding belongs to whom

Figure 2.5: Face recognition pipeline

FaceNet is a face recognition system developed by Google researchers in

2015 described by Florian Schroff, et al at Google in their 2015 paper titled

“FaceNet: A Unified Embedding for Face Recognition and Clustering” [13]

Figure 2.6: Google FaceNet pipeline

The model is a deep convolutional neural network trained via a triplet lossfunction that encourages vectors for the same identity to become more similar(smaller distance), whereas vectors for different identities are expected to becomeless similar (larger distance) The focus on training a model to create embeddingsdirectly (rather than extracting them from an intermediate layer of a model) was

an important innovation in this work For the triplet loss process, the input will

2.2 Background and Methodologies

based on Zeiler & Fergus or some smaller architectures for inferencing on mobiledevices

• Advantages:

– High accuracy on common datasets.

– One of state of the art face recognition model.

• Disadvantages:

– Hard to deploy on embedded systems with constraint hardware.

– Slow inference speed compared to other frameworks.

Dlib is a modern C++ toolkit with Python API containing machine learningalgorithms and tools for creating complex software It is used widely in indus-trial applications, including robotics, mobile phones, and large high-performancecomputing environments This is a suitable toolkit for embedded applications

Unlike FaceNet recognition, the model is based on complex neural works Dlib uses its custom neural network for lightweight and robust applications.Moreover, Dlib can be built and run on CUDA (GPU), which is compatible withGPU-powered embedded boards like Nvidia Jetson or Google Coral

net-• Advantages:

– Moderate accuracy on common datasets.

– Fast and robust development and deployment.

• Disadvantages:

– Impossible to modify the architecture.

– Custom build may require knowledge of operating systems and

configu-ration

2.2.7.3 Classifier

After generating embedding vectors for people in the dataset, a classifier iscreated to classify each person’s identity Some traditional algorithms can work wellfor such problems, such as support vector machine (SVM) or K-nearest neighbor(K-NN) We want to investigate more about each algorithm to find out the advantagesand disadvantages then choose the most appropriate one for our system

• Support vector machine:

Support vector machine is considered as a supervised-learning algorithm and

is usually used for classification challenges The basic concept of a supportvector machine is to find out hyperplanes that separate data points intomultiple distinguished classes Therefore, we can choose an infinite number

of hyperplanes The main challenge is to find out which hyperplane hasmaximum margins to each nearest point of each class, and this is also theloss function of the algorithm

– Advantages:

- Works well with clearly seperated dataset

- Effective in high dimension space and number of dimensions is higherthan a number of samples

– Disadvantages:

- The training time is large with large classes

- Sensitive with noise data

- Require a re-train model whenever new data are added

• K-nearest neighbor(K-NN):

K-nearest neighbors are considered the most basic machine learning algorithmthat is not required to train (lazy machine learning algorithm) K-NN isoften used in classification, just like the SVM The concept of K-NN is pretty

2.2 Background and Methodologies

- Work well with multiple classes data

- Small number of hyper parameters for tunning

- Do not need training phase

– Disadvantages:

- Very sensitive to noise

- Hard to decide appropriate K

- Heavy calculation process if the dataset is large

In this chapter, we will propose details of our approach in designing thesystems including camera devices, Cloud IoT platform, and web application.

3.1 Solution Description

This study aims to build and develop a solution to measure non-contactmultiple human temperatures combined with checking identity and face masks.The additional features like face mask checking and face recognition are integratedinto our system to enhance the effectiveness in preventing disease spread andtracing the people with a high-temperature sign Our system will be divided intothree main components: camera device, cloud IoT platform, and management webapplication

Our thermal camera system is designed with compact hardware components

We are combining one RGB camera and one thermal camera for the best opticalprocessing The device will detect people’s entries, measure temperature, recognizethe identity, and then send records to the server for processing and monitoring

3.2 Proposed System Architecture

The web application provides administrators a management dashboardthat can show helpful information about total entries and records each day foreach building Administrators can also add, modify person, building, and deviceinformation

We divide our users into two different roles Take the example scenario inuniversity Students and teachers in each building are registered information tothe system, including some basic contact information and many photos for facerecognition Another role is the administrator The administrators can monitorusers and their records each day in specific locations, buildings The administratorscan even control the state of devices and check if they are working or not

3.2 Proposed System Architecture

There are several ways to implement the proposed approaches of the thermalcamera system mentioned in section 2 The first way is to build a server to handletasks that required high computational resources to improve the total system’sresponse time The other alternative method is to implement the edge-computingsystem The embedding board also plays the micro-server role and performs tasksbefore sending data to the central server There is a trade-off between the twoapproaches; the former requires deploying a server system and entirely depend

on the server’s computational capacity Meanwhile, the latter requires powerfulembedding boards to execute tasks effectively so that the response time can beensured

In Fig 3.1, we propose system architecture comprises three main nents: thermal camera devices, a server, and a web application that is deployed

compo-on an Azure server for users to mcompo-onitor and manage the system The two-waycommunication between the thermal camera devices and the server is constructed

on the Azure IoT platform using its available components because the Azureplatform provides a scalable, flexible and secure solution for our system

- Azure IoT Hub: IoT Hub acts a role as a managing service, cloud-hosted,central message hub for bi-directional communication between your IoTapplication and the devices it manages You can use Azure IoT Hub to build