Luận văn a jetson xavier based smart check in

The application has been written on React Na-tive as mobile application for requesting the room access permission and otherfeatures purpose and Next.js admin website for the management p

FACULTY OF COMPUTER SCIENCE AND ENGINEERING

REPORT GRADUATION THESIS CAPSTONE PROJECT

Assoc Prof Dr Pham Quoc Cuong

MSc Bang Ngoc Bao Tam

REVIEWER:

Assoc Prof Dr Tran Ngoc Thinh

—o0o—

STUDENT 1: Le Tu Ngoc Minh (195284)

STUDENT 2: Vo Pham Hoai Nam (1952863)

HO CHI MINH CITY, 06/2023

ĐẠI HỌC QUỐC GIA TP.HCM CỘNG HÒA XÃ HỘI CHỦ NGHĨA VIỆT NAM

KHOA:KH & KT Máy tính _ NHIỆM VỤ ĐỒ ÁN TỐT NGHIỆP

BỘ MÔN:KHMT _ Chú ý: Sinh viên phải dán tờ này vào trang nhất của bản thuyết trình

HỌ VÀ TÊN:

- Lê Tự Ngọc Minh MSSV: 1952844 Computer Engineering

- Võ Phạm Hoài Nam MSSV: 1910351 Computer Science

NGÀNH: Kỹ thuật Máy tính _ LỚP: MT19CLC _

1 Đầu đề đồ án: A JETSON XAVIER-BASED SMART CHECK IN/CHECK OUT SYSTEM

2 Nhiệm vụ (yêu cầu về nội dung và số liệu ban đầu):

- Conduct a thorough survey on the topic of smart check in/out with edge devices

- Propose a Jetson Xavier-based system architecture

- Implement the proposed architecture on the platform

- Conduct tests and evaluate the system with different dataset

3 Ngày giao nhiệm vụ luận án: 10/01/2023

4 Ngày hoàn thành nhiệm vụ: 01/6/2023

5 Họ tên giảng viên hướng dẫn: Phần hướng dẫn:

1) PGS TS Phạm Quốc Cường 2) ThS Băng Ngọc Bảo Tâm _ 3) Nội dung và yêu cầu LVTN đã được thông qua Bộ môn

Ngày 10 tháng 01 năm 2022

(Ký và ghi rõ họ tên) (Ký và ghi rõ họ tên)

PHẦN DÀNH CHO KHOA, BỘ MÔN:

Người duyệt (chấm sơ bộ):

KHOA KH & KT MÁY TÍNH Độc lập – Tự do – Hạnh phúc

-

Ngày 05 tháng 06 năm 2023

PHIẾU ĐÁNH GIÁ LUẬN VĂN/ ĐỒ ÁN TỐT NGHIỆP

(Dành cho người hướng dẫn)

1 Họ và tên

- Lê Tự Ngọc Minh MSSV: 1952844 Computer Engineering

- Võ Phạm Hoài Nam MSSV: 1910351 Computer Science

2 Đề tài: A JETSON XAVIER-BASED SMART CHECK IN/CHECK OUT SYSTEM

3 Họ tên người hướng dẫn: PGS.TS Phạm Quốc Cường – ThS Băng Ngọc Bảo Tâm

4 Tổng quát về bản thuyết minh:

Số bảng số liệu: 16 Số hình vẽ: 57

Số tài liệu tham khảo: 44 Phần mềm tính toán:

Hiện vật (sản phẩm)

5 Những ưu điểm chính của LV/ ĐATN:

-The thesis is detailed and straightforward Moreover, the authors have a good self-study ability to complete all of the tasks in thesis

- The authors surveyed and illustrated many methods related to the problems before

proposing their method

- The authors have good engineering skills in optimizing and developing AI models to run

on edge devices, especially Jetson Xavier

6 Những thiếu sót chính của LV/ĐATN:

7 Đề nghị: Được bảo vệ  Bổ sung thêm để bảo vệ  Không được bảo vệ 

8 Các câu hỏi SV phải trả lời trước Hội đồng:

9 Đánh giá chung (bằng chữ: Xuất sắc, Giỏi, Khá, TB): Điểm: 9.8/10

Ký tên (ghi rõ họ tên)

Phạm Quốc Cường

TRƯỜNG ĐẠI HỌC BÁCH KHOA CỘNG HÒA XÃ HỘI CHỦ NGHĨA VIỆT NAM

-

Ngày tháng năm

PHIẾU ĐÁNH GIÁ LUẬN VĂN/ ĐỒ ÁN TỐT NGHIỆP

(Dành cho người hướng dẫn/phản biện)

1 Họ và tên SV: Lê Tự Ngọc Minh

MSSV: 1952844 Ngành (chuyên ngành):Kỹ thuật Máy tính

Võ Phạm Hoài Nam

MSSV: 1952863 Ngành (chuyên ngành):Khoa học Máy tính

2 Đề tài: A Jetson Xavier-based smart check in/check out system (Phát triển hệ thống quản lý

ra vào thông minh trên Jetson Xavier)

3 Họ tên người hướng dẫn/phản biện: PGS.TS Trần Ngọc Thịnh

4 Tổng quát về bản thuyết minh:

Số bảng số liệu: 14 Số hình vẽ: 58

Số tài liệu tham khảo: 41 Phần mềm tính toán:

Hiện vật (sản phẩm)

5 Những ưu điểm chính của LV/ ĐATN:

This project proposed a smart check-in/check-out system that includes a comprehensive solution for face authentication and real-time human tracking

For face authentication, three techniques including face detection, face recognition and face spoofing are used

anti-For human tracking and counting, the SORT algorithm with Kalman filter is used

The experiments for face authentication are tested on 3 datasets: WF, VGG Face2, MS-Cleleb-1M for face detection and face recognition, and 2 datasets: CASIA-MFSD and Replay-Attack for face anti-spoofing The results are so good with accuracy up to 95%

The proposed system is successfully implemented on Jetson Xavier with high inference

performance, low energy consumption as compared with Jetson Nano and Rasberry4

The mobile application also works well

6 Những thiếu sót chính của LV/ĐATN:

The system compared with Jetson Nano and Rasberry4 is not fair, the system should be compared with other similar works on Jetson Xavier

The test of the face mask still has false results, but this is just a minor task of this project

7 Đề nghị: Được bảo vệ  Bổ sung thêm để bảo vệ  Không được bảo vệ 

8 Các câu hỏi SV phải trả lời trước Hội đồng:

a Have you tested the system with multiple faces (5-25 faces) per frame?

9 Đánh giá chung (bằng chữ: Xuất sắc, Giỏi, Khá, TB): Good (giỏi) Điểm : 9/10

Ký tên (ghi rõ họ tên)

Trần Ngọc Thịnh

The contents of this capstone project are original, with the exception of cific references to the work of others, and have not been submitted in whole or

spe-in part for any other degree or qualification at any other university This stone project is our own work and contains no results of collaborative work withothers, except as noted in the text and Acknowledgments section

cap-Ho Chi Minh City, May 2023

ii

A CKNOWLEDGMENT

Our supervisor, Dr Pham Quoc Cuong, has been incredibly patient, ful, and supportive throughout our almost 1 year as a student We have greatlyprofited from your extensive expertise and careful editing We are incrediblyappreciative that you accepted me as a student and maintained your confi-dence in me over the years Second, we humbly acknowledge the assistance

help-of Master Bang Ngoc Bao Tam - HCMC University help-of Technology Without theirassistance, we would not be able to complete this important research

Our families are deserving of unending gratitude for their unending love,sacrifice, and support, which keeps us motivated and certain Our achieve-ments and success are a result of their faith in us No words can adequatelyexpress our affection for our families

iii

With the advancement of technology, many plans and projects aiming atbuilding and improving Smart IoT System have been developing at a rapid rate.Along with this and the need to solve the check-in/checkout in the organiza-tion’s room in any part of the world, Smart Accessing Control has been a hotlyresearched and implemented subject As one of the newly emerging and dis-rupting technology, AI (Artificial Intelligence) is being utilized in many aspects

of implementing a Smart Accessing Control system One of the many ways toutilize AI for Smart Accessing Control is to implement 3 models: face detec-tion, face recognition and face anti-spoofing for detecting people by utilizingthe power of Jetson Xavier NX In this capstone project, we will implement afull system consisting of software and hardware implementation of many tech-nology techniques to create a fully functional Smart Accessing Control system

We will develop a mobile application for standard user and a website for nization’s owners and admins The application has been written on React Na-tive as mobile application for requesting the room access permission and otherfeatures purpose and Next.js admin website for the management purpose Weconduct real-time data by keeping the data up-to-date from the server Dataflow using between screen - screen and application - server has been transferedwith the JSON files For the back-end server, we will implement by using NestJs

orga-to handle the HTTP requests and MongoDB will be used as our database mentation The back-end server will first be tested on a local machine and thendeployed on a cloud service provider We leverage the use of edge computingand artificial intelligence (AI) to identify the person accessing the organization’sroom in real-time and updating the event to the back-end server

imple-iv

C ONTENTS

1.1 Problem statement 1

1.2 Scope 1

1.3 Contribution 2

1.3.1 Edge computing development 2

1.3.2 Application development 2

1.3.3 API and database server development 2

1.4 Organization 2

2 Background and Related work 3 2.1 Edge Computing 3

2.1.1 Edge Computing 3

2.1.2 Edge AI 3

2.1.3 GPU technology 4

2.1.4 JetPack 4

2.2 Software Technology 5

2.2.1 NestJS - A progressive Node.js framework 5

2.2.2 Next.js by Vercel - The React Framework 6

2.2.3 React Native - A framework for building native applica-tions using React 6

2.2.4 MongoDB 7

2.2.5 REST API 7

2.2.6 Google Cloud Platform 8

v

2.3 Literature review 10

2.3.1 Access control system: Related work in the world 10

2.3.2 Access control system in Viet Nam 11

2.3.3 Access control system with image processing 12

3 Methodology 13 3.1 Problem statement 13

3.1.1 Hardware - Face authentication 13

3.1.2 Hardware - Real-time tracking 14

3.1.3 Software - Application and Server 14

3.2 Proposed solution 15

3.2.1 System Architecture 15

3.2.2 Edge AI computing 17

3.2.3 System functionalities 18

3.2.4 Use case diagram 21

3.2.5 Use Case Specification 23

3.2.6 Main flows of the system 35

4 Implementation 39 4.1 Hardware 39

4.1.1 Setup 39

4.1.2 Training model 39

4.1.3 Integration to our system 46

4.1.4 Applying CUDA, Deepstrem, TensorRT, CUDNN, GStreamer, onxx to enhance our system 48

4.1.5 Deployement on device 50

4.1.6 Database design 53

5 Result and Evaluation 56 5.1 Hardware 56

5.1.1 Face Authentication 56

5.1.2 Evaluation on Kalman filter with human real-time track-ing 59

5.1.3 System performance evaluation 59

5.2 Mobile Application 63

5.2.1 Authentication feature 63

5.2.2 Weather today and Recent activities features 64

5.2.3 Managing room access permission feature 65

5.2.4 Setting account feature 67

5.2.5 Update profile feature 67

5.2.6 Reset password feature 68

CONTENTS vii

5.2.7 Update face feature 69

5.3 Admin Website Application 70

5.3.1 Login feature 70

5.3.2 Statistical room’s data visualization feature 71

5.3.3 Alert feature 72

5.3.4 User management feature 73

5.3.5 Room management feature 75

5.3.6 Abnormal event management feature 76

5.3.7 Request room access management feature 77

5.3.8 Latency and Throughput Evaluation of Api Server 78

5.3.9 Performance Evaluation of Admin Website 79

5.4 Full System Test and Result 82

5.4.1 Test case 1: Valid Access 82

5.4.2 Test case 2: Stranger Access 82

5.4.3 Test case 3: Fake Access 83

5.4.4 Test case 4: Register using phone 84

5.4.5 Test case 5: Register at the door 85

5.4.6 Test case 6: Tracking and counting 85

6 Conclusion 86 6.1 Summary 86

6.2 Future Work 87

3.1 IoT Based Access Control Architecture 15

3.2 NVIDIA JETPACK SDK for AI at the Edge [17] 17

3.3 Use Case Diagram of Standard user mobile application 21

3.4 Use Case Diagram of Administrator website application 22

3.5 Activity diagram of request room access permission activity 35

3.6 Activity diagram of check-in activity 37

4.1 The architectures of P-Net, R-Net, and O-Net [20] 41

4.2 Error rate comparison 42

4.3 Accuracy rate comparison 42

4.4 Precision comparison 43

4.5 Precision comparison 43

4.6 Overview of our proposed PatchNet framework [4] 44

4.7 Overview of our software in Jetson Xavier NX 46

4.8 The face identification flow 47

4.9 Block diagram for Kalman filter algorithm 48

4.10 OpenCV with CUDA 48

4.11 NCNN with CUDA 49

4.12 Enable cuDNN for optimized GPU computations 49

4.13 Start GSreamer pipeline 49

4.14 Deployment step from NVIDIA documents [36] 50

4.15 TensorRT optimizes trained neural network models to produce a deployment-ready runtime inference engine from NVIDIA docu-ments [36] 51

4.16 TensorRT’s vertical and horizontal layer fusion and layer elimi-nation optimizations simplify the GoogLeNet Inception module graph, reducing the computation and memory overhead from NVIDIA documents [36] 51

4.17 TensorRT optimizes trained neural network models to produce a deployment-ready runtime inference engine from NVIDIA docu-ments [1] 52

4.18 Class diagram of Database design 53

viii

LIST OFFIGURES ix

5.1 Power consumption of Jetson Xavier NX when running through

time 60

5.2 FPS inference performance on the different accelerator-based sys-tem on chip 61

5.3 Memory usage on SoCs 61

5.4 Energy consumption of different accelerator-based SoCs 62

5.5 Authentication screens 63

5.6 Home screen 64

5.7 Make a request room access permission feature 65

5.8 Manage list of room access permissions screen 66

5.9 Account setting screen 67

5.10 Update profile screen 68

5.11 Reset password screen 68

5.12 Update face screen 69

5.13 Login screen 70

5.14 Dashboard page 71

5.15 Alert stranger 72

5.16 Alert overcrowded room 72

5.17 View list of users feature 73

5.18 View detail information and room access history of a specific user feature 73

5.19 Edit information of a specific user feature 74

5.20 Create a user feature 74

5.21 View list of rooms feature 75

5.22 View and Edit information of a specific room feature 75

5.23 Add a new room to the system feature 76

5.24 View list of abnormal events feature 76

5.25 View and Edit list of abnormal events feature 77

5.26 View list of room access requests feature 77

5.27 Check the registered face of user 78

5.28 Result of the evaluation of Api Server 79

5.29 Result of the performance evaluation of Admin Website 80

5.30 All metrics of testing performance Admin Website 80

5.31 Valid access detected by the system 82

5.32 Stranger image captured by the system 83

5.33 Invalid face detected by the system 84

3.1 Mobile application: Login use case specification 23

3.2 Mobile application: Register use case specification 24

3.3 Mobile application: Manage account settings use case specification 25

3.4 Mobile application: View room access permission use case fication 26

speci-3.5 Admin website: Manage rooms use case specification 28

3.6 Admin website: Manage users use case specification 30

3.7 Admin website: Manage abnormal events use case specification 32

3.8 Admin website: Get statistical information use case specification 33

3.9 Admin website: Get alerts use case specification 34

4.1 Comparison between models with Inference(ms) metric 52

4.2 Comparison between models with GPU Inference(ms) and FDDBPrecision (100 errors) 53

5.1 Result on Face detection model on Jetson Xavier through mization inference step 56

opti-5.2 Result on Face anti-spoofing model on Jetson Xavier through timization inference step 57

op-5.3 Result on Face Recognition Model run on Jetson Xavier throughoptimization with dataset from LFW 58

5.4 Result table of evaluation on Kalman Filter in Human detection 59

5.5 Comparison between using a model with and with not using timization 59

op-x

In this capstone project, we focus on real-time AI processing, which meanthat we will apply strong AI embedded computer for specific application toget the best performance The methodology achieved in this work will dealwith several challenges like real-time recognition, portable, and reliable system.NVIDIA Jetson Xavier NX is used in this work It is a tiny, powerful AI computerthat delivers the estimated performance to run advanced AI workloads with asmall size The main goal of this research is to deploy a deep learning model forface authentication and tracking security camera in Jetson Xavier NX, with theintention of real-time performance and high accuracy The results of this studyhave shown great real-time performance in running some trained model whichhave greater FPS

1.1 PROBLEM STATEMENT

Many systems nowadays have negative self-awareness and poor service lack

of self-service check-in/check-out, requiring customers to rely entirely on staffmembers It also has problems with management including tracking room sta-tus, retrieving visitor information, and monitoring access in real-time

In our system, we concentrate on optimising the AI model operating onNVIDIA Jetson Xavier NX Moreover, we focus on capturing information regard-ing check-in procedures Finally, we concentrate on check-in passengers with-out masks

1

1 1.3. CONTRIBUTION

Our aim of this project is to build a real-time access control base on 2 maingoals: authentication and tracking data It will help organization to control thelarge crowd of access instead of small system just for attendance in recently use

in room nowadays Our system will be comprised from 3 main parts

1.3.1 E DGE COMPUTING DEVELOPMENT

We will detect face, check liveliness, face recognition occupancy at the edgedevices and send the results to the server instead of just sending the image offaces to the server and make the server do the detection The parking occu-pancy detection identifies parking spaces whether they are empty or occupied

in real-time

1.3.2 A PPLICATION DEVELOPMENT

In this project, a build of application for users to manage and interact withdata and server become our purpose on this part We will build a mobile appli-cation for interaction between standard users and the system Meanwhile, weaim to build a website for management purposes for the administrators

1.3.3 API AND DATABASE SERVER DEVELOPMENT

We will build a server to host our data as well as manage the connection andinformation from and to edge devices as well as client mobile app This serverwill be deployed on-site for testing purpose then we will move everything to acloud service to aid in security, reliability as well as scalability

1.4 ORGANIZATION

In this capstone project, we divided into 6 main chapters Chapter 1 cludes the introduction of the project Chapter 2 includes the Background knowl-edge about the project’s technologies, frameworks, datasets and the Literaturereviews Chapter 3 includes the Problem and Proposed solutions for each part

in-of the system Chapter 4 shows the implementation for the hardware, end server and applications Chapter 5 consists of the evaluation and analytic

back-of the project Finally, Chapter 6 includes the conclusion and future works back-ofthe project

2.1 EDGECOMPUTING

2.1.1 E DGE C OMPUTING

Edge computing is a distributed computing paradigm that brings tation and data storage closer to the location where it is needed, to improveresponse times and save bandwidth The origins of edge computing lie in con-tent delivery networks that were created in the late 1990s to serve web and videocontent from edge servers that were deployed close to users Other definitiondefined edge computing as all computing outside the cloud happening at theedge of the network, and more specifically in applications where real-time pro-cessing of data is required Edge computing aim to computation away fromdata center toward the edge of the network to perform tasks and provide ser-vices on behalf of the cloud Along with its benefits, we have to deal with somechallenge while applying edge computing such as: privacy and security, scala-bility, reliability, speed, efficiency

compu-2.1.2 E DGE AI

Edge AI is the combination of Edge Computing and AI (Artificial gence) with the aim to accelerate the speed of processing data at the edge nodes

Intelli-3

Instead of sending all the data to the server, local devices only need to send theresult of the computed data

Edge Computing/Edge AI is suitable for systems that do not depend much

on internet connection speed and require intermediate data processing (imageand video analysis) such as surveillance cameras or self-driving vehicles.Support vector machine (SVM), deep learning and neural networks (NNs)are the most deployed algorithms in Egde AI corresponding to different appli-cations Bringing those techniques to devices need to meet the model designand compression which are adapted to the hardware Many hardware plat-forms have been developed to the market by big companies in the industrysuch as Intel, Xilinx, NVIDIA, etc to ensure that edge computing operates AI

in an efficient way and meets system’s requirements There are some populartypes of hardware including central processing unit (CPU), graphics process-ing unit (GPU) and some NVDIA frameworks which provided by jetson xavierwhich have been adapted to perform AI on edge devices

2.1.3 GPU TECHNOLOGY

CUDA (Compute Unified Device Architecture) is a parallel computing form and programming model developed by NVIDIA for use with their graph-ics processing units (GPUs) It enables developers to write programs that canharness the power of the GPU to accelerate computing tasks, especially in data-intensive applications like machine learning, scientific computing, and imageprocessing

plat-With CUDA, developers can write code in a C-like programming languagecalled CUDA C, which is then compiled using the nvcc compiler that is included

in the CUDA toolkit The resulting executable code can run on the GPU, which

is optimized for parallel computing, rather than the CPU

2.1.4 J ET P ACK

The Jetson Xavier is a powerful system-on-a-chip (SoC) designed by NVIDIAfor AI and high-performance computing applications It includes a number ofhardware accelerators, such as a GPU, deep learning accelerators, and a videoencoder/decoder

If you’re asking about a "jetpack" for the Jetson Xavier, then you might bereferring to the NVIDIA JetPack SDK, which is a collection of software compo-nents that enable developers to build and deploy AI and computer vision appli-cations on Jetson platforms

The JetPack SDK includes a number of software components, such as theCUDA toolkit, cuDNN, TensorRT, OpenCV, and others, which are optimized forthe Jetson platform and take advantage of its hardware accelerators

Once the JetPack SDK is installed, you can use it to develop and deploy AIand computer vision applications on the Jetson Xavier, taking advantage of itspowerful hardware accelerators to achieve high performance and efficiency.

frame-A DVANTAGES OF N EST JS

Firstly, Architecture: There are many frameworks for Node.js, but none ofthem effectively solves the main problem of Node.js - the architecture problem.NestJS, on the other hand, was built to offer solutions to code structure andcode organization strategy problems Developers and teams may build highlytested, scalable, loosely linked, and simple-to-maintain apps using Nest’s pre-built application architecture The architecture takes a lot of cues from Angular.Secondly, NestJS has a lot of support modules:

• Contain a lot of built-in modules to work with the database, handle rity, implement streaming, etc

secu-• Allows building Rest API, MVC, Microservices, GraphQL, Web Socket, orCRON job applications

• NestJS’s microservices module supports all kinds of connections: bitMQ, gRPC, Kafka, Redis, etc

Rab-Finally, Community support: NestJS is one of the fastest-growing Node.jsframeworks in recent years NestJS maintains detailed documentation, and itscommunity of developers and contributors is very active and ready to respond

to issues on the fly

2.2.2 N EXT JS BY V ERCEL - T HE R EACT F RAMEWORK

I NTRODUCTION

Next.js is a React front-end framework that enables extra pre-rendering toolsincluding out-of-the-box server-side rendering and static generation to maxi-mize efficiency and user experience Because it necessitates a solid grasp ofboth client-side React and server-side architectures, full-stack developers uti-lize Next.js to create reactive websites

A DVANTAGES OF N EXT JS

Firstly, Community support: One of the widely used React frameworks formany well-known brands is Next.js As it gains popularity, a large community ofpeople naturally contributes to it This implies that there will likely be a solutionfor any issue we encounter

Secondly, Performance: Because Next.js is a React framework, it has all ofReact’s positive attributes One of these is live coding, or hot reload, which pro-vides us with immediate feedback on changes made to our React components.Moreover, Next.js has some improved features, such as:

• Image Optimization (Lazy Loading, Responsive, )

• Server-side Rendering (SSR) and Static Site Generation (SSG) give us theability to serve the entire content on the first load very fast

Thirdly, Built-in CSS support: For quicker rendering, we can import CSSstyles from a JavaScript file into Next.js

Finally, Optimized production deployment on Vercel platform BecauseVercel is a platform created by the creators of Next.js, designed for Next.js ap-plications

2.2.3 R EACT N ATIVE - A FRAMEWORK FOR BUILDING NATIVE APPLICA

-TIONS USING R EACT

Introduction React Native is a JavaScript-based framework for creating tive iOS and Android mobile apps It is built on React, Facebook’s JavaScripttoolkit for designing user interfaces, but targets mobile devices rather than browsers

na-In addition, since the majority of the code you create may be shared across forms, React Native makes it simple to build simultaneously for Android andiOS

2

7

A DVANTAGES OF R EACT N ATIVE

React Native combines the most advantageous aspects of native ming with React, the premier JavaScript toolkit for constructing user interfaces.Instead of directly modifying the browser’s DOM, React produces a virtual DOM

program-in memory, where it does all the required manipulations, before applyprogram-ing thecorresponding modifications to the browser’s DOM

React Native was used to develop cross-platform applications React ponents encapsulate existing native code and communicate with native APIsusing the declarative UI paradigm and JavaScript provided by React This al-lows the creation of native apps by whole new teams of developers and mayallow current native teams to operate far quicker

com-2.2.4 M ONGO DB

I NTRODUCTION

MongoDB is an open-source, cross-platform, document-oriented databaseprogram A NoSQL database program, MongoDB, utilizes documents that re-semble JSON and may or may not contain schemas Some distributions con-sider the Server-Side Public License (SSPL), which is used to distribute Mon-goDB and is produced by MongoDB Inc., to be non-free

A DVANTAGES OF M ONGO DB

Firstly, NoSQL makes it easy to keep all types of data gathering without theneed to determine beforehand the type of data being saved It allows us to storefar more types of data than SQL, including objects and arrays

Secondly, The document database architecture of MongoDB makes it easy

to store both structured and unstructured data It saves documents in a like format This format readily transfers to native objects in the programminglanguage of our choice (Java, Python, etc.), thus we need not bother about nor-malizing data

JSON-Thirdly, MongoDB can scale both horizontally and vertically and managelarge amounts of data

Finally, the MongoDB cloud service makes the deployment simpler thancompeting SQL systems, such as Oracle and PostgreSQL We will save time andmoney by performing this task

2.2.5 REST API

I NTRODUCTION

A REST API (also known as RESTful API) is an application programming terface (API or web API) that complies to the REST architectural style and en-ables interaction with RESTful web services Roy Fielding, a computer scientist,

invented REST, which stands for representational state transfer

REST is neither a protocol or a standard; it is a collection of architectural strictions REST may be implemented in a number of ways by API developers

re-An API is a collection of protocols and standards for developing and integratingapplication software It is sometimes referred to as a contract between an infor-mation supplier and an information user, setting the material needed from theconsumer (the call) and the content required by the producer (the response)

REST APIs provide the development of any kind of application supportingall CRUD (create, retrieve, update, and delete) actions REST principles recom-mend utilizing a given HTTP method for a particular sort of server call (whiletheoretically it is feasible to break this recommendation, it is strongly discour-aged)

There are 5 essential HTTP methods in RESTful API development:

• GET: The GET method is used to request a representation of a resource.GET requests should only fetch data

• POST: The POST method sends an entity to the given resource, often sulting in a server state change or other side effects

re-• PUT: The PUT method overwrites all existing representations of the source with the request payload

re-• DELETE: The DELETE method deletes the resource supplied

• PATCH: The PATCH technique modifies a resource with partial changes

2.2.6 G OOGLE C LOUD P L ATFORM

Google Cloud Platform (GCP) is a suite of cloud computing services offered

by Google It provides a comprehensive set of tools and services that enable ganizations to build, deploy, and scale applications and services in the cloud.GCP encompasses a wide range of products, including computing, storage, net-working, databases, machine learning, and more, all backed by Google’s ro-bust and globally distributed infrastructure In this thesis, we use the Googlecloud Storage to store image and Google Cloud Run to deploy our applications.Therefore, we will dive into the advantages of Google Cloud Storage and GoogleCloud Run

2

9

G OOGLE C LOUD S TORAGE

Google Cloud Storage is an object storage service within the Google CloudPlatform It allows you to store and retrieve unstructured data such as files,images, videos, backups, and logs Here are some key features and advantages

of Google Cloud Storage:

• Scalability: Google Cloud Storage offers virtually unlimited scalability, lowing you to handle massive amounts of data You can start with smallstorage requirements and seamlessly scale up as your needs grow

al-• Durability and Availability: Your data is automatically replicated acrossmultiple devices and locations within a region, ensuring high durabilityand availability This minimizes the risk of data loss and enables reliableaccess to your data

• Security: Google Cloud Storage provides robust security features, ing encryption at rest and in transit You can manage access control us-ing Google Cloud Identity and Access Management (IAM), defining fine-grained permissions and roles to secure your data

includ-• Integration: Google Cloud Storage integrates well with other Google CloudPlatform services, allowing you to easily incorporate it into your applica-tions and workflows For example, you can seamlessly connect it withCompute Engine, BigQuery, or Cloud Functions

• Cost-effectiveness: Google Cloud Storage offers cost-effective pricing tions, with different storage classes to suit various use cases You can opti-mize costs by choosing the appropriate storage class based on the accessfrequency and retrieval requirements of your data

op-G OOGLE C LOUD R UN

Google Cloud Run is a fully managed serverless compute platform on GCP

It enables you to run containerized applications without worrying about frastructure provisioning or management Here are the key features and ad-vantages of Google Cloud Run:

in-• Serverless Architecture: With Google Cloud Run, you can focus solely ondeveloping your application code while the underlying infrastructure ismanaged by Google It automatically scales your application based onincoming requests, ensuring high availability and eliminating the needfor manual scaling

• Container-based: Google Cloud Run allows you to deploy containerizedapplications, making it easy to package and deploy your software in aconsistent and portable manner You can use Docker containers andleverage existing container orchestration tools

• Quick Scalability: Google Cloud Run can rapidly scale your applications

to handle high traffic loads, automatically adjusting the number of stances to match demand This makes it suitable for applications withvarying workloads or unpredictable traffic patterns

in-• Cost Efficiency: You only pay for the actual usage of your applications

on Google Cloud Run It scales down to zero when there is no incomingtraffic, which can result in cost savings compared to maintaining contin-uously running servers

• Integration and Portability: Google Cloud Run integrates with other GCPservices and can be easily connected to tools like Cloud Build, Cloud Log-ging, and Cloud Monitoring It also offers portability, allowing you to de-ploy your containers to Cloud Run on Anthos or other Kubernetes clus-ters

In summary, Google Cloud Platform provides a robust and extensive set ofcloud computing services Google Cloud Storage offers scalable and secure ob-ject storage for your data, while Google Cloud Run enables serverless deploy-ment of containerized applications, allowing you to build and deploy applica-tions efficiently and cost-effectively

2.3 LITERATURE REVIEW

2.3.1 A CCESS CONTROL SYSTEM : R EL ATED WORK IN THE WORLD

Face recognition technology has become increasingly popular in access trol systems due to its high accuracy and convenience

con-HERTA SECURITY

Touchless access control is the perfect solution for controlling user access

to physical locations such as buildings, offices, data centers, and restricted curity areas while maintaining customers and staff safety

se-Face biometrics is used to identify a user from access control lists and toverify their identity Herta’s access control solution enables user managementwhile in use and allows users to be enrolled on-the-fly by video capture

This technology yields excellent performance despite partial occlusions ofthe face, the use of glasses, scarfs or caps, changes of facial expression, and

imper-Some of its main features:

• 90% accuracy of face recognition

• 93% of liveliness detection

• 89% of mask detection

• increased speed over a times

NEC F ACE R ECOGNITION T ECHNOLOGY

NEC’s biometric face recognition technology is used worldwide for fightingcrime, preventing fraud, securing public safety, and improving customer expe-rience across a vast range of locations and industries

Why? Because independent NIST testing has confirmed NEC’s NeoFacerecognition and matching capability as the world’s most accurate several timesacross benchmarks and challenging conditions.*

It stems from over Half-century of biometric technology expertise It ples recognition with real-time identification, verification and situation anal-ysis for quick decision-making, preemptive security, and smoother services.Installed in over 1,000 major systems in more than 70 countries and regionsworldwide, it boasts a stellar track record and wealth of practical experience

cou-And finally, as the long-reputed industry leader, our face recognition tions are also designed to offer solid protection for the public without encroach-ing on their privacy, freedoms or human rights

solu-F ACE F IRST

FaceFirst is a global leader in highly effective face matching systems for tailers, hospitals, casinos, airports, stadiums, and arenas FaceFirst’s softwareleverages artificial intelligence and human oversight to prevent violence, theft,and fraud With FaceFirst, you can provide safer environments for your val-ued customers, patients, guests, employees, and associates We designed ourpatented video analytics platform to be scalable, fast, accurate, and ethical,while maintaining the highest levels of security, privacy, and accountability.FaceFirst is proudly designed, engineered, and supported in the USA

re-2.3.2 A CCESS CONTROL SYSTEM IN V IET N AM

Vingroup’s VinAI: VinAI is a Vietnamese artificial intelligence company thatuses facial recognition technology for a range of applications, including accesscontrol, security, and retail

2.3.3 A CCESS CONTROL SYSTEM WITH IMAGE PROCESSING

Access control systems with image processing use cameras and advancedimage recognition software to authenticate and authorize access to physicalspaces Image processing technology allows access control systems to captureand analyze images, and make decisions about whether or not to grant accessbased on the information in those images Here are some examples of accesscontrol systems with image processing:

• Facial recognition access control systems: These systems use camerasand facial recognition software to identify users and grant access based

on their facial features This technology is becoming increasingly lar for access control in high-security areas such as data centers, airports,and government buildings

popu-• License plate recognition systems: These systems use cameras and age processing software to read the license plates of vehicles and allow ordeny access based on a pre-determined list of authorized vehicles Thistechnology is commonly used in parking garages, gated communities,and toll booths

im-• Video analytics-based access control: These systems use advanced videoanalytics to monitor access to physical spaces, identify and track people,and detect anomalies or suspicious behavior This technology is com-monly used in high-security facilities such as prisons, military bases, andgovernment buildings

• Visitor management systems: These systems use cameras and image nition software to scan identification documents such as driver’s licenses

recog-or passprecog-orts, and verify the identity of visitrecog-ors befrecog-ore allowing them cess to a building or office

ac-Overall, access control systems with image processing technology are coming increasingly popular due to their high level of accuracy and conve-nience However, there are concerns around privacy and data protection, asthese systems collect and store personal information It is important for com-panies and organizations to implement appropriate safeguards and data pro-tection measures to address these concerns

to investigate the effectiveness of using advanced technologies such as facialrecognition, biometric authentication, and machine learning in access controlsystems, and to develop a new access control system that addresses the limita-tions of traditional systems and meets the needs of modern organizations andpublic spaces.

3.1.1 H ARDWARE - F ACE AUTHENTICATION

Face authentication has become an increasingly popular method for fying the identity of individuals for access control, financial transactions, andother applications However, face authentication systems may be vulnerable

veri-to various security threats, such as spoofing attacks, where a malicious acveri-toruses a counterfeit face or facial features to gain unauthorized access Addition-

13

ally, face authentication systems may be susceptible to biases and inaccuracies,particularly for individuals from underrepresented groups As such, there is aneed to develop and improve face authentication systems that are resistant tosecurity threats and provide a high level of accuracy and fairness for all users.This thesis aims to investigate the effectiveness of different face authenticationmethods, such as biometric recognition and machine learning algorithms, and

to develop a new face authentication system that addresses the limitations ofexisting systems and provides a high level of security, accuracy, and fairness

3.1.2 H ARDWARE - R EAL - TIME TRACKING

Real-time people tracking is an important technology for a range of plications, including security, retail, and crowd management However, tradi-tional people tracking systems may be limited by factors such as lighting con-ditions, occlusion, and complex backgrounds, which can lead to inaccuraciesand tracking failures Additionally, traditional people tracking systems may not

ap-be able to handle large crowds or complex environments, such as those found

in airports, stadiums, and public spaces As such, there is a need to develop andimprove real-time people-tracking systems that can overcome these limitationsand provide a high level of accuracy, efficiency, and scalability This thesis aims

to investigate the effectiveness of different people-tracking algorithms, such asobject detection, deep learning, and image processing, and to develop a newreal-time people-tracking system that addresses the limitations of existing sys-tems and meets the needs of modern organizations and public spaces

3.1.3 S OFTWARE - A PPLICATION AND S ERVER

When developing the software component of the BKAccess system, we counter various considerations such as time, performance, user experience,speed, aesthetics, and effectiveness However, attempting to address all possi-ble scenarios and fully satisfy all aspects of the software would be time-consuming.Therefore, our focus is primarily on resolving specific issues outlined below

en-R EAL -T IME D ATA P ROCESSING

Real-time data processing plays a critical role in the software system BKParkoperates by streaming data in real-time to ensure smooth workflow and mini-mize potential errors that may occur during its operation, such as data con-flicts, outdated information, or system congestion Real-time data processingoffers several advantages that optimize our system, including faster responseand action times, reduced risks, and the ability to understand customer behav-ior as it unfolds

3.2.PROPOSED SOLUTION

3

15

W ELL - DESIGNED UX/UI (U SER E XPERIENCE /U SER I NTERFACE )

A well-designed UX/UI design is a specific issue we prioritize because itleads to enhanced user satisfaction, ease of use, accessibility, clear communi-cation, a positive brand image, improved engagement and retention, and errorprevention and recovery

O PTIMIZING R ESPONSE T IME

As our system requires real-time data exchange with multiple clients, ticularly receiving data from various edge devices, it is crucial for the server’sAPIs to respond quickly The performance of the server determines the overallsystem performance, encompassing the mobile app, website, and the speed atwhich data is received from edge devices

par-3.2 PROPOSED SOLUTION

3.2.1 S YSTEM A RCHITECTURE

Figure 3.1: IoT Based Access Control Architecture

After examining the challenges associated with building this system, we fered a solution with the architecture shown in Figure above The architecture

of system is made up of five major parts:

First, Application: Web App for Administrator for management purposes

and Mobile App for Standard User to interact with the system

Second, API Server: Includes 6 core services:

• User management service: Takes responsibility for interactions with the

user data in the system (such as CRUD operations)

• Room management service: Responsible for interactions with the room

data in the system (such as CRUD operations)

• Access requests service: Handle for managing all the requested access

per-mission to the organization’s rooms from the standard users

• Check-in/out events management service: Handles interactions with the

system’s check-in/check-out event data (such as CRUD operations)

• Abnormal events management service: Responsible for tracking all

abnor-mal events occurred in the organization’s rooms during the working time

of the system (such as CRUD operations)

• Notification service: Takes responsibility for managing all system

notifi-cations, such as notifying admin about the abnormal event happened in

a specific room

Provides transaction processing and security when interacting with data

from Application and Hardware.

Third, Database: Stores the data in MongoDB and Image hosting by Google

Cloud Storage

Fourth, Hardware: Includes

• Jetson Xavier for Edge Computing and performing logic, AI operation.

• Camera for capturing faces.

Fifth, use Adafruit IO as MQTT broker to receive messages from Hardware

and send to App via MQTT protocol, vice versa

Sixth, use Google Cloud Platform, which includes:

• Google Cloud Run: to deploy API server and web application

• Google Cloud Storage: to store and host image for public accessing

In this thesis, we proposed a solution using CUDA and TensortRT and VPI

to optimize the model and build an NVinffer engine to speed up the processing

of the engine

Figure 3.2: NVIDIA JETPACK SDK for AI at the Edge [ 17 ]

CUDA is a parallel computing platform and programming model that ables developers to use NVIDIA GPUs for general-purpose computing It pro-vides a set of libraries, tools, and APIs for creating high-performance applica-tions, including those for face recognition With CUDA, developers can lever-age the parallel processing power of GPUs to accelerate computationally inten-sive tasks, such as image and video processing

en-TensorRT is a deep learning inference optimizer and runtime that helps velopers deploy high-performance neural networks on NVIDIA GPUs It opti-mizes and quantizes deep neural networks to improve inference performance,reduce memory usage, and reduce latency TensorRT can be used for a variety

de-of tasks, including face recognition, object detection, and speech recognition

VPI (Vision Programming Interface) is a library that provides a unified gramming interface for accelerating computer vision applications on NVIDIAGPUs It supports a wide range of image and video processing functions, in-cluding face detection, tracking, and recognition VPI provides a simple and ef-ficient API for integrating GPU acceleration into computer vision applications

pro-3.2.3 S YSTEM FUNCTIONALITIES

FACE RECOGNITION

Collect face images: the Labeled Faces in the Wild (LFW) dataset

Train face recognition model: use a deep learning framework like Flow to train your face recognition model on the collected face images, andleverage CUDA to accelerate the training process on NVIDIA GPUs

Tensor-Optimize model with TensorRT: Once you’ve trained your face recognitionmodel, use TensorRT to optimize it for deployment on NVIDIA GPUs This willprovide faster inference performance and reduced memory usage

Process face images with VPI: Before feeding face images into your mized face recognition model, use VPI to perform image processing tasks likeface detection and alignment This will ensure that the images are properlyaligned and cropped before being fed into the model

opti-Deploy your model: deploy your optimized face recognition model on anNVIDIA GPU using TensorRT This will provide fast and accurate face recogni-tion performance on a variety of devices

Overall, using CUDA, TensorRT, and VPI together can significantly improvethe performance of the face recognition system, making it faster and more ac-curate

F ACE ANTI SPOOFING

Face anti-spoofing is a technique to distinguish between a real face and afake one created using a printed photo, a video, or a 3D mask NCNN (NeuralNetwork Computing Network) is a high-performance neural network inferenceframework that can be used for deep learning tasks such as image classification,object detection, and segmentation

Preprocessing: preprocess the input image to make it compatible with thenetwork’s input format We can resize the image and normalize the pixel values.Model loading: load the pre-trained model for face anti-spoofing

Model optimization: use TensorRT to optimize it for deployment on NVIDIAGPUs

Model inference: pass the preprocessed image through the pre-trained model

to get the prediction

R EAL - TIME PEOPLE TRACKING AND COUNTING :

Collect a dataset of video footage of the area you want to track and countpeople in You can use publicly available datasets or collect your own dataset

Train a deep learning model using the collected video footage and an objectdetection framework MobileNetV1-SSD Use CUDA to accelerate the trainingprocess

Once the model is trained, deploy it on a Jetson device for real-time peopletracking and counting

For each input frame, the function first predicts the location of each trackedobject and updates the location based on the new detections from the childthread It then computes the intersection over union (IoU) between each trackedobject and the new detections and associates the detections to the tracked ob-jects based on the IoU value The function updates the position of the trackedobjects based on the new detections and draws bounding boxes around thetracked objects in the input frame

Finally, the function counts the number of people entering and exiting apredefined area (in this case, the line at the center of the frame) and displays thecounts on the input frame It also displays the total number of tracked objects

in the frame

Set up the camera(s) in the area you want to track and count people in

Use the deployed model to track and count people in real time by ing the video footage captured by the camera using the Jetson device

process-U SER M OBILE A PPLICATION AND A DMINISTRATOR W EBSITE A PPLICATION

In our system, applications play a vital role We decided to develop twoseparate applications, one for standard users and one for administrators Themobile application will help users interact with the system such as requestingthe access permission to the organization’s room, while the website will assistadministrators with administrative tasks In this proposed solution chapter,the functional requirements for the user mobile application and administratorwebsite will be outlined

Firstly, The Standard User actor of the Mobile application:

• The standard users can register their information, particularly their facialimage, to be granted access rights to enter the rooms of an organization

• The standard users are able to request the room access’s permission tothe system

• The standard users are able to view their accessing room history

• The standard users are able to view the weather today based on their cation

lo-• The standard users are able to view or update their account information,such as avatar, name,

Secondly, the Administrator actor of the Website application:

• The administrators are able to view the statistical data generated aboutthe abnormal events, check-in/out events and the status of the room

• The administrators are able to manage the users in the system

• The administrators are able to manage the rooms in the system

• The administrators are able to manage the abnormal events and in/out events of the organization’s rooms

check-• The administrators are able to receive notifications including room formation, occurred time, scene pictures, and type of the event when anyabnormal events occur

in-Moreover, there are non-functional requirements of our system:

• Reliability: The system should have about 1% probability of system

fail-ure when a service request is made

• Security: The system should have at least one for the authentication

tech-nique (such as with usernames and passwords) and one for the rization technique when the user accesses the system

autho-• Performance: The request made by the client should take at most 1

sec-onds to send back the response

• Usability: The users can understand how to use and be attracted to the

website and mobile application after 1 minute of access

• Maintenance: The system should be designed with modularity and ease

of maintenance in mind, allowing for updates and improvements withoutsignificant disruption to users

specifi-Figure 3.3: Use Case Diagram of Standard user mobile application

The above use case diagram illustrates the overview of mobile applicationsystem with the standard user actor In the diagram, there are 7 core use cases,including:

• Login and Register

• Manage account settings

• View check-in/out history

• View weather on current time

• Register face

• View room access permission

The detail of those use cases will describe in the next section

Figure 3.4: Use Case Diagram of Administrator website application

The above use case diagram illustrates the overview of website applicationsystem with the administrator actor In the diagram, there are 6 core use cases,including:

• View statistical information

• View real-time information

• Manage rooms

• Manage users

Description User can login to the system.

Pre-condition 1 The User have accessed the system.

2 The User has a valid account

Post-condition User successfully logged into the system

Normal Flow

1 The User chooses the login function

2 The System requires a username and password

3 The User inputs username and password

4 If the username and password are Valid, the System navigatesthe User to the home page

5 End the Use Case

a The account is invalid

b The account does not exist

c Enter the wrong password

d The system is under maintenance

2 End the Use Case

Table 3.1: Mobile application: Login use case specification

Use case name Register

Description User can register to the system

Pre-condition The User have accessed the system

Post-condition User successfully registered

Normal Flow

1 1 The User chooses the register function

2 The System asks the user to enter the necessary information

3 The System checks the valid information and sendsregistration information via email

4 The User activates the account

5 The System navigates the User to the home page

6 End the Use Case

a The information provided is not valid

b The account already exists

c The account has been banned

d The system does not temporarily support the function

2 End the Use Case

Table 3.2: Mobile application: Register use case specification

Description User can view and update their account information.

Pre-condition 1 The User has logged into the system

Post-condition Any changes from the User must be valid to update to the database

Normal Flow

1 The User selects account setting function

2 The System displays some basic user’s information, (such as name,email, avatar) and some interactive options (Update face images,Update information, Log-out)

3 End the Use Case

Alternative Flow

Alternative 1: At step 3

1 The User chooses the Update face images option

2 The System navigates the User to Update face image screen

3 The User put the face in the correct frame that the systemhas already indicated

4 The System validates the user’s face via Face Detectionprovided by Google ML Kit

5 The User presses the button to capture the face image

6 The System updates to the database and raises a updatingsuccessfully message to notify the user

7 End the Use Case

Alternative 2: At step 3

1 The User chooses the Update information option

2 The System navigates the user to Update user information screen

3 The User changes the their information

4 The System validates the changed values

5 The System updates to the databaseand raises updating successfully message to notify the user

6 End the Use Case

Alternative 3: At step 3

1 The User chooses the Logout option

2 The System clears the user state

3 The System navigates the User to the Login screen

4 End the Use Case

Exceptions

Exception 1: At step 4 of Alternative 1.

1 If the user’s face is invalid, the System prevents users

to capture the face image

2 The System shows message to guide the user doing correctly

3 Continue to step 3 of Alternative 1

Exception 2: At step 4 of Alternative 2.

1 If the changed values are invalid, the System prevents users

to update those information

2 The System shows the error message about the invalid cases

3 Continue to step 3 of Alternative 2

Table 3.3: Mobile application: Manage account settings use case specification

1 The User selects managing room access function.

2 The System displays the list of room permissions, which arerequested by the current User

3 The User checks which rooms of the organization they can access

as of the current time

4 End the Use Case

Alternative Flow

Alternative 1: At step 3

1 The User chooses the Create request access in the current screen

2 The System checks if the User has registered their face to the system

3 The System navigates the user to the requested form foraccessing permission

4 The User fills out the form completely

5 The System validates the inputs of the User

6 The User presses the Submit button to make a request

7 The System updates to the database

8 End the Use Case

Exceptions

Exception 1: At step 2 of Alternative 1.

1 If the user’s face is not existed, the System displays a dialog thatasks the User to register their face

2 The user accepts to register face

3 The System navigates the User to Update face image screen

4 The User put the face in the correct framethat the system has already indicated

5 The System validates the user’s face via Face Detectionprovided by Google ML Kit

6 The User presses the button to capture the face image

7 The System updates to the databaseand raises a updating successfully message to notify the user

8 Continue to step 3 of Alternative 1

Exception 2: At step 5 of Alternative 1.

1 If the inputs are invalid, the System prevents users

to create a request accessing permission

2 The System shows the error message about the invalid cases

3 Continue to step 4 of Alternative 1

Table 3.4: Mobile application: View room access permission use case specification

Description Administrator can create, view, edit, terminate the rooms.

Pre-condition Administrator has logged into the system

Post-condition Any changes from the room information are updated to the database

Normal Flow

1 The Administrator clicks the room management in the navigation

2 The System displays all the rooms registered on the system

in tabular form

3 The Administrator views list room’s information

4 End the Use Case

Alternative Flow

Alternative 1: At step 3

1 The administrator clicks the View button on the Action Tab

of the table

2 The system navigates the administrator to the room detail page

3 The system displays the all the information of the room

4 End the Use Case

3 The administrator changes the room information

4 The system validates the changed information

5 The system updates the changes to database andnotifies the administrator of successful updating

6 End the Use Case

Alternative 3: At step 3

1 The administrator clicks the Delete button on the Action Tab

of the table

2 The system raises a dialog to confirm the deletion of the room

3 The administrator approves to delete

4 The system changes the room’s status to "Unavailable" in database

5 End the Use Case