Luận văn a jetson xavier based smart parking

We leverage the use of edge computing and artificial ligence AI to deploy a Deep Neural Network DNN algorithms for ojbect de-tection and Convolutional Neural Network CNN for image classi

FACULTY OF COMPUTER SCIENCE & TECHNOLOGY

CAPSTONE PROJECT

MAJOR: COMPUTER ENGINEERING

Council:

COMPUTER ENGINEERING OISP 02

Superviosrs:

Assoc Prof Pham Quoc Cuong - HCMUT

Assoc Ms Bang Ngoc Bao Tam - HCMUT

Reviewer:

Assoc Prof Tran Ngoc Thinh - HCMUT

Authors:

Nguyen Thanh Loc - 1952330

HO CHI MINH CITY - June-2023

ĐẠI HỌC QUỐC GIA TP.HCM CỘNG HÒA XÃ HỘI CHỦ NGHĨA VIỆT NAM

TRƯỜNG ĐẠI HỌC BÁCH KHOA Độc lập - Tự do - Hạnh phúc

KHOA:KH & KT Máy tính _ NHIỆM VỤ ĐỒ ÁN TỐT NGHIỆP

BỘ MÔN:KHMT _ Chú ý: Sinh viên phải dán tờ này vào trang nhất của bản thuyết trình

HỌ VÀ TÊN:

- Trần Vĩnh Tiến MSSV: 1952493 Computer Engineering

- Nguyễn Thành Lộc MSSV: 1952330 Computer Engineering NGÀNH: Kỹ thuật Máy tính _ LỚP: MT19CLC _

1 Đầu đề đồ án: A JETSON XAVIER-BASED SMART PARKING

2 Nhiệm vụ (yêu cầu về nội dung và số liệu ban đầu):

- Conduct a thorough survey on the topic of smart parking with edge devices

- Propose a Jetson Xavier-based system architecture

- Implement the proposed architecture on the platform

- Conduct tests and evaluate the system with different dataset

3 Ngày giao nhiệm vụ luận án: 10/01/2023

4 Ngày hoàn thành nhiệm vụ: 01/6/2023

5 Họ tên giảng viên hướng dẫn: Phần hướng dẫn:

1) PGS TS Phạm Quốc Cường 2) ThS Băng Ngọc Bảo Tâm _ 3) Nội dung và yêu cầu LVTN đã được thông qua Bộ môn

Ngày 10 tháng 01 năm 2022

(Ký và ghi rõ họ tên) (Ký và ghi rõ họ tên)

Phạm Quốc Cường Phạm Quốc Cường

PHẦN DÀNH CHO KHOA, BỘ MÔN:

Người duyệt (chấm sơ bộ):

Ngày 05 tháng 06 năm 2023

PHIẾU ĐÁNH GIÁ LUẬN VĂN/ ĐỒ ÁN TỐT NGHIỆP

(Dành cho người hướng dẫn)

1 Họ và tên

- Trần Vĩnh Tiến MSSV: 1952493 Computer Engineering

- Nguyễn Thành Lộc MSSV: 1952330 Computer Engineering

2 Đề tài: A JETSON XAVIER-BASED SMART PARKING

3 Họ tên người hướng dẫn: PGS TS Phạm Quốc Cường – ThS Băng Ngọc Bảo Tâm

4 Tổng quát về bản thuyết minh:

Số tài liệu tham khảo: 26 Phần mềm tính toán:

Hiện vật (sản phẩm)

5 Những ưu điểm chính của LV/ ĐATN:

-The thesis is detailed and straightforward Moreover, the authors completely finished all of the tasks in thesis

- The authors have a deep understanding of optimizing AI models on edge devices

- The authors also designed and built an AI system to deploy their solution in reality

6 Những thiếu sót chính của LV/ĐATN:

- There are some minor errors in grammar and format of the thesis

- The thesis also lacks information about how the authors proposed their solution Thus, it

is difficult to understand the intuition behind their methodology In addition, illustrations about the results is not briefly

7 Đề nghị: Được bảo vệ  Bổ sung thêm để bảo vệ  Không được bảo vệ 

8 Các câu hỏi SV phải trả lời trước Hội đồng:

- What is novelty (tính mới) in your research?

9 Đánh giá chung (bằng chữ: Xuất sắc, Giỏi, Khá, TB): Điểm : 9.0/10

Ký tên (ghi rõ họ tên)

Phạm Quốc Cường

TRƯỜNG ĐẠI HỌC BÁCH KHOA CỘNG HÒA XÃ HỘI CHỦ NGHĨA VIỆT NAM KHOA KH & KT MÁY TÍNH Độc lập - Tự do - Hạnh phúc -

Ngày tháng năm

PHIẾU ĐÁNH GIÁ LUẬN VĂN/ ĐỒ ÁN TỐT NGHIỆP

(Dành cho người hướng dẫn/phản biện)

1 Họ và tên SV: Nguyễn Thành Lộc

MSSV: 1952330 Ngành (chuyên ngành):Kỹ thuật Máy tính

Trần Vĩnh Tiến

MSSV: 1952493 Ngành (chuyên ngành):Kỹ thuật Máy tính

2 Đề tài: A Jetson Xavier-based smart parking (Xây dựng hệ thống quản lý giữ xe thông minh

sử dụng Jetson Xavier)

3 Họ tên người hướng dẫn/phản biện: PGS.TS Trần Ngọc Thịnh

4 Tổng quát về bản thuyết minh:

Số bảng số liệu: 0 Số hình vẽ: 48

Số tài liệu tham khảo: 23 Phần mềm tính toán:

Hiện vật (sản phẩm)

5 Những ưu điểm chính của LV/ ĐATN:

 The students proposed solutions for Parking lot detection and License plate detection in smart parking system They also built a customer’s mobile application for booking parking lots

 The Yolov4 AI model is used for both Parking lot detection and License plate detection It is trained and optimized by tensorRT engine of NVIDIA

 The experiments show that the proposed system can run on a Jetson Xavier board with 21-30 frames per second and the mobile application also works well

6 Những thiếu sót chính của LV/ĐATN:

 The dataset for Vietnam License plate detection is small The author should refer the previous thesis “An edge ai hardware acceleration solution for smart parking”

 The system is only applied for car parking lots in HCMUT, not for general parking lots

7 Đề nghị: Được bảo vệ  Bổ sung thêm để bảo vệ  Không được bảo vệ 

8 Các câu hỏi SV phải trả lời trước Hội đồng:

a How can you improve the wrong license plate detection in some cases?

b Have you compared the proposed system with previous works?

9 Đánh giá chung (bằng chữ: Xuất sắc, Giỏi, Khá, TB): Khá (fair good) Điểm : 8/10

Ký tên (ghi rõ họ tên)

Trần Ngọc Thịnh

We hereby declare that except where specific reference is made to the work ofothers, the contents of this dissertation are original and have not been submit-ted in whole or in part for any other degree or qualification in any other uni-versity This dissertation is our own work and contains nothing which is theoutcome of work done in collaboration with others, except as specified in thetext and Acknowledgements.

Tran Vinh Tien

Nguyen Thanh Loc

Ho Chi Minh City, June 2023

iii

of Technology and Faculty of Computer Science and Engineering, whom havetaught us many valuable and significant lessons, knowledge and experiencesduring our time studying in the university.

Last but not least, we thank all of our beloved families and friends for alwayssupporting us

v

Thanks to technological blessings and scientific advancement, many plans anfprojects aiming at building and improving Smart Cities have been developing

at a rapid rate Along with this and the need to accelerate automation, mize manual labor, wasted resource and time, Smart Parking has been a hotlyresearched and implemented subject As one of the newly emerging and dis-rupting technology, IoT (Internet of Things) is being utilized in many aspect ofimplementing a Smart Parking system First of all, AI (Artificial Intelligence)can be applied to implement object detection for detecting vehicles and imageclassification for license plate recognition Another point is, the rapid develop-ment of hardware technology, especially embedded computing hardware such

mini-as NVIDIA Jetson series, can reduce and possibly implement the real-time tem with out performance not only on speed but also on accuracy We will de-velop a mobile application for customer and a website for owners and adminsfor easily interact with out system The mobile application has been written onFlutter for booking and other features purpose and the Django Python frame-work will be used for website and server for management purpose We collectreal time data from hardware and update if any specific event has come and up-date to server The specific event can be the real-time event in parking space or

sys-a trigger signsys-al from higher sys-abstrsys-act levels Dsys-atsys-a flow between screen - screenand application - server has been transferred with JSON file type An API serverwill be used to handle the HTTP request and SQLlite3 will be applied as ourdatabase implementation The server will be completely tested and deployed

on a local machine We leverage the use of edge computing and artificial ligence (AI) to deploy a Deep Neural Network (DNN) algorithms for ojbect de-tection and Convolutional Neural Network (CNN) for image classification intoNVIDIA Jetson Xavier NX Developer Kit for identifying parking occupancy anduser’s license plates in real-time and updating the results to their server Ourwork reach the FPS of 31 in the inference phase and around 17 to 26 FPS forfully handling a single frame

intel-vii

C ONTENTS

1.1 Problem statement 1

1.2 Scope 1

1.3 Goals 2

1.3.1 Edge computing development 2

1.3.2 Application Development 2

1.3.3 API and database server development 3

1.4 Thesis Structure 3

2 Background and Related work 5 2.1 Related Work In Smart Parking System Approaches 5

2.1.1 Parking space management 5

2.1.2 Gate management 6

2.2 Technologies using in Our Smart Parking System 7

2.2.1 iot 7

2.2.2 cv / im 7

2.2.3 ml, dl and nn 7

2.2.4 gps 8

2.2.5 Some solutions for reference 8

2.3 Edge Computing 9

2.3.1 Edge Computing 9

2.3.2 Edge AI 9

2.4 NVIDIA Jetson Xavier NX Developer Kit 9

2.4.1 What and why is NVIDIA Jetson Platform 9

2.4.2 NVIDIA Jetson Xavier NX Developer Kit 14

ix

3 Methodology 17

3.1 Problem Statement Review 17

3.2 General flow of system 17

3.2.1 Customer’s mobile application Workflow 17

3.2.2 Edge device workflow 19

3.3 System Architecture 19

3.4 Hardware 20

3.4.1 License plate recognition 20

3.4.2 Parking lot occupancy detection 21

3.4.3 Cloud-based Server for parking 22

3.4.4 Mobile application 24

4 Implementation 25 4.1 Hardware 25

4.1.1 Set Up 25

4.1.2 Create parking spaces 26

4.2 Deployment 28

4.2.1 License plate recognition 28

4.2.2 Parking lot occupancy detection 30

4.2.3 Check parking lot status 30

4.3 Database and end point server implementation 30

4.3.1 Database design 30

4.3.2 Back-end implementation 31

4.4 Mobile application 32

4.4.1 Introduction 35

4.4.2 Software Development 35

5 Result and Evalation 41 5.1 Hardware 41

5.1.1 Models evaluation 41

5.1.2 FPS evaluation 44

5.1.3 Deployment results 45

5.2 Server implementation result and testing 49

5.2.1 APIs response time testing 49

5.2.2 APIs reliability testing 49

5.3 Mobile application 52

5.3.1 Android application 52

5.3.2 Sign up feature 53

CONTENTS xi

5.4 Full system test and result 57

5.4.1 Testcase 1: Parking Space Detection of Empty Spaces 57

5.4.2 Testcase 2: Parking Space Detection of Occupied Spaces 58

5.4.3 Testcase 3: License Plate Recognization Check in 59

5.4.4 Testcase 4: License Plate Recognization Check out 60

5.4.5 Testcase 5: Creating Booking 60

5.4.6 Testcase 6: Completing Booking 61

5.4.7 Testcase : Edge device stability/connection consistency test 62

6 Conclusion 65 6.1 Summary 65

6.2 Future Work 65

6.2.1 Edge AI detection 65

6.2.2 back-end server 66

6.2.3 mobile application 66

2.1 Edge Computing Infrastructure 10

2.2 CUDA support FPS performance comparison 14

2.3 Optimized models and State of the art models accuracy compari-son 15

2.4 NVIDIA Jetson Xavier NX Developer Kit 15

3.1 Mobile application workflow illustration 18

3.2 Smart Parking System Architecture 19

3.3 License Plate Recognition Module Architecture 21

3.4 Parking Lot Occupancy Detection Module Architecture 22

3.5 Server Role in Smart Parking System Architecture 22

3.6 Android application general work flow 24

4.1 Setup configuration on Edge device 26

4.2 Creating bounding boxes of parking slots 27

4.3 Activity diagram of license plate recognition task 28

4.4 Gray scale image 29

4.5 Processed license plate image 29

4.6 API documentation 31

4.7 API documentation 32

4.8 API documentation schema 33

4.9 API documentation demo fetch parking lot by id 34

4.10 Software system architecture 34

4.11 General use case diagram for Spark smart parking system 35

4.12 General use case diagram for Spark smart parking system 36

4.13 General use case diagram for Spark smart parking system 37

4.14 General usecase diagram for Spark smart parking system 37

4.15 General use case diagram for Spark smart parking system 38

4.16 General use case diagram for Spark smart parking system 38

5.1 Configuration for training 42

5.2 Different architecture comparison result 42

5.3 Configuration for training 43

5.4 Accuracy performance before optimizing 43

xiii

xiv LIST OFFIGURES

5.5 Accuracy performance after optimizing 43

5.6 Speed performance before optimizing 43

5.7 Speed performance after optimizing 44

5.8 Accuracy performance after optimizing 45

5.9 Configuration power mode 45

5.10 Some results for LPR module 46

5.11 Error result in blue color license plate 46

5.13 Test result with slanted angle 47

5.12 Test result with straight angle 48

5.14 Create booking histogram 50

5.15 Update Booking status 51

5.16 Change booking status 51

5.17 Onboard screen 53

5.18 Login screen 53

5.19 Booking steps 54

5.20 Routing screen 55

5.21 Booking history 56

5.22 User setting 57

of saving time and resources, four-wheeled transportation is a more interestingand important aspect in the 21st century.

1.1 PROBLEM STATEMENT

The population explosion is one of the most challenging problems that whelmingly puts significant pressure on many cities around the world One ofits consequences is an increase in the number of vehicles participating in traf-fic, leading to enormous pressure on city parking lots Drivers searching forparking spaces are one of the biggest causes of urban traffic congestion, ac-counting for 30% of all traffic congestion (Intro Smart Parking, 2021) This phe-nomenon not only wastes a large amount of time but also consumes a signifi-cant amount of fuel, contributing to the greenhouse effect Therefore, without

over-a well-plover-anned over-and convenient solution to help mover-anover-age pover-arking spover-aces, theseproblems will continue to worsen

A smart parking system is not only an effective solution for saving time byquickly and easily finding parking spaces but also plays a crucial role in building

a smart city

1.2 SCOPE

There are several constraints that we will handle in this problem:

1

• Application for Parking Space in Ho Chi Minh University of Technology:The smart parking system will be specifically designed and applied forthe parking spaces in Ho Chi Minh University of Technology

• User Account Sign-up: Users will be required to create an account through

a mobile application to access the smart parking system

• Handling Small Cars: The system will be designed to handle parking forsmall cars, specifically 4 to 7-seat cars

• Hardware Performance: The hardware used, specifically the Jetson Xavier

NX Developer Kit, should be able to run stably and process the data at aminimum rate of 17 to 24 frames per second (fps)

1.3 GOALS

This thesis aims to design and implement a smart parking system using Internet

of Things (IoT) technologies The system will optimize the parking process byallowing customers to locate and reserve appropriate parking spots for theirvehicles Additionally, the system will enable parking lot owners to monitorand manage their parking lots more effectively using mobile smart devices Thesystem will consist of three main parts:

1.3.1 EDGE COMPUTING DEVELOPMENT

The system will detect parking occupancy at the edge devices and send the sults to the server This approach eliminates the need to send the entire parkinglot image to the server for detection The module will provide real-time identi-fication of empty and occupied parking spaces It will also include license platerecognition for checking and confirming bookings

re-1.3.2 APPLICATIONDEVELOPMENT

A mobile application will be developed to allow users to interact with the tem Users will be able to create, edit, delete, and modify data through theapplication The application will provide a user-friendly interface for viewingdata and offer efficient advice to customers for selecting the best parking spacesbased on their information It will also provide routing features to guide usersfrom their current location to the parking zone

sys-In terms of management, we want to build a website for managers to view, date, delete and create the object in a database as well as manage easier

1.3.3 APIAND DATABASE SERVER DEVELOPMENT

A server will be built to host the data and manage the connections and

informa-tion between the edge devices and client mobile applicainforma-tion The server will be

thoroughly tested and deployed on local devices, with the ability to be accessed

publicly on the Internet through port forwarding

1.4 THESIS STRUCTURE

There are totally five chapters in this thesis proposal The first chapter will

present the related works about smart parking system and also the background

knowledge The second one introduces obviously about an effective solution

The next one will explain evidently implementation for hardware, server and

the mobile application Our system is evaluated, analyzed and tested in

Sec-tion 5 Finally, the inclusion and discussion in the last secSec-tion

on Edge computing device as well as some Smart Parking solution related toour project.

2.1 RELATEDWORKINSMARTPARKINGSYSTEMAPPROACHES

The development of effective smart parking systems at a reduced cost is nowpossible thanks to the rapid advancement in internet, communication, and in-formation technology Numerous researchers have created alternative way tomanage parking lot based on various methodologies and sensors They can

be divided into 2 main groups: smart car park base on gate management andsmart car park base on lot management Moreover, we also do some commonresearch on technologies that smart parking systems are using, namely: Wire-less Sensors Networks (WSN), Computer Vision (CV)/Image Processing (IM),Internet of Things (IoT), Machine Learning, Deep Learning, Neural Network,Global Positioning System (GPS) In this document, we will analyse some of thesolution in 2 main groups

2.1.1 PARKING SPACE MANAGEMENT

Space parking management can be classified into two categories: multi-parkingmanagement and mono-parking management The multi-parking manage-ment manage many parking lot at the same time to provide user with suitable

5

car parking location This kind of parking management is suitable for car park

in door such as: airport, or outdoor such as: street parks in metropolis Onthe other hand, mono-parking management tend to manage individual park-ing lot and provide guidance for drivers within parking area Overall, many so-lutions belong to mono-parking management, whereas only few solutions areproposed for multi-parking management.[1]

M ONO - PARKING SPACE MANAGEMENT

In this system, wireless sensors are deployed into a car park field, with eachparking lot equipped with one sensor node

Usually, authors use WSN to detect free parking space quickly and efficiently.When a car drive in or out which will trigger a function in sensor node Sensornode will send data to the WSN gateway using Zigbee protocol Then the gate-way which acts as a bridge between sensors and IoT middle-ware, back-endserver or display the info to driver at entrance Although the node rely heav-ily on batteries, the power of the node is kept minimal thank to some methodssuch as: energy efficient techniques[2] [3], TinyOS [4], [5], [6]

Zigbee is chosen due to its efficient energy consumption and flexibility in ogy management so as to increase the life expectancy of devices

topol-In [7], [8] and [9] authors propose a systems using image detection which bine with machine learning and deep learning algorithm

com-M ULTI - PARKING SPACE MANAGEMENT

The author in [10] use VANET to vehicles with real-time parking guidance in bigparking lots Drivers may locate a free parking place fast with the help of real-time parking guidance

The solution in [11] was built around a sensor-based method and designedbased on a multi-layer framework to enable modularity and scalability and toprovide different services to different users of a parking system

These system can extend to larger scale like street parking or car park in urbanarea however it may need collaboration between parking space owners

2.1.2 GATE MANAGEMENT

IParking in [12] is a smart parking framework that monitor the vehicles whenthey drive in and drive out They count the number of lot then display it ontwitter account of iparking When a car driving in the entrance will detect itthen calculating remaining lot in parking space

2.2.TECHNOLOGIES USING INOURSMARTPARKING SYSTEM

2

7

2.2 TECHNOLOGIES USING INOURSMARTPARKINGSYSTEM

2.2.1.IOT

IoT is the current buzzing technology in which all gadgets are networked with

one another over the internet Every gadget connected to the internet has a

unique identification (UID) These devices might be computational,

mechani-cal, or digital in nature They can send data without requiring human-to-human

or human-to-computer contact IoT technology is one of the core essential

technologies used by Smart parking system developers All sensors and

compu-tational equipment in IoT-based Smart parking system are connected through

the internet and may communicate data without the need for human

involve-ment A wired or wireless internet connection can be used to link sensors,

com-puting equipment, and storage units to the internet Almost all recently smart

parking system use this technology

2.2.2.CV /IM

Smart parking system based on CV/IM employs several types of camera

net-works to extract various information such as parking lot occupancy status ,

li-cense plate recognition (LPR), and facial recognition for billing, security

con-cerns, and providing road traffic congestion reports Because these systems

rely on real-time parking lot video data for feature extraction, systems based

on computer vision/IM technologies often have a high data transfer rate from

the camera network to the processing units These kind of Smart parking

sys-tems are typically used in open parking spaces since a single camera can catch

a large portion of the parking lot These technologies, however, are susceptible

to occlusion, shadow effects, distortion, and light shifting [13] [9] [14] [15] use

this technology in the papers to detect, check occupation and security

2.2.3.ML,DL AND NN

Machine learning is a subset of AI that allows a system to learn and improve on

a specific task using dataset or experiences without directly programming the

system A machine learning-based Smart parking system analyzes the

park-ing lot data to determine the parkpark-ing lot state Furthermore, ML and AI-based

Smart parking system can estimate parking lot occupancy status in the

follow-ing days, weeks, or even months and provide a dynamic pricfollow-ing scheme

ML-based systems can monitor traffic congestion on specific roadways and provide

a smart solution to smart parking spots

Deep learning is a subset of ML and an AI function that resembles the human

brain in terms of data processing and feature extraction to make judgments In

an Smart parking system, DL algorithms detect vacantly occupied and special

parking lots instead of ordinary sensors, reducing the number of sensors andcameras required by the system DL is also used to forecast parking lot occu-pancy

Neural network is a set of algorithms that pulls characteristics and

underly-ing relationships from data sets in a manner similar to how the human brainworks In Smart parking system, NN is employed for real-time license plateidentification utilizing video data To identify parking lot occupancy, CNN andmachine vision are used CNNs may also provide road traffic conditions for var-ious routes [16] Deep Neural Network (DNN), also called Deep Nets, is a neu-ral network with a certain level of complexity It can be considered as stackedneural networks, or networks composed of several layers, usually two or more,that include input, output, and at least one hidden layer in between DDNs areoften employed to deal with unlabeled and unstructured data Today, these so-phisticated neural networks have become the standard tool for solving variouscomputer vision tasks

Convolutional Neural Network(CNN) is a type of deep learning algorithm that

is particularly well-suited for image recognition and processing tasks It is made

up of multiple layers, including convolutional layers, pooling layers, and fullyconnected layers

2.2.4.GPS

GPS is a crucial component of several smart parking techniques However, GPSalone is incapable of gathering parking lot occupancy statistics and providingadditional smart parking capabilities It need support from other source likesensors However, GPS can give vehicle guidance for the user to drive towardsvacant parking lots Using CNN or algorithms, several systems can estimateparking lot occupancy and road traffic congestion based on GPS data Thenumber of GPS receivers determines its accuracy GPS accuracy for a single fre-quency receiver is roughly 7.8 m A two-frequency receiver, on the other hand,has an accuracy of roughly 0.715 m Parking spaces that use GPS suitable forthis are open area one Some solution use this technique are [17] [18]

2.2.5 SOME SOLUTIONS FOR REFERENCE

Several solutions such as [19], [1], [4], [8] using wireless sensor with strongstructure of computer network for checking occupancy in parking space Al-though the system witness a good architecture of structure and low power forexecuting, the setup such as routing, setting and maintenance cost are signifi-cantly noticeable On the other hand, if the parking space has the large amount

of parking lots, around 1000 lots, the architecture for routing is very

2

9

cated and consume a lot of sensor In terms of [9] [13], they using an Edge AI

for processing and detecting parking space for extracting some important data

before transferring them to server This solution will save resource for both data

transferring and image processing, make the system real time running

How-ever, they train model by CNPark and PKL datasets which is not very suitable for

parking space in VietNam and these systems do not have new or outstanding

features In industry, they can check license plate by using Edge AI for

pro-cessing an image captured manually by signal from a RFID card This feature

is amazing, however, the system only capture the image when security using

RFID card so that the system missed the automation feature

2.3 EDGE COMPUTING

2.3.1 EDGECOMPUTING

Edge computing [20] is a distributed computing paradigm that allows IoT

hard-ware devices to quickly process and act on its stored data to archive the efficient

performance

Edge computing aim to pre-process, calculations, and post-process real time

data that collected from the environment or data center toward the edge of the

network to perform task and provide services on behalf of the cloud for better

productivity However, along with these advantages, we also deal with several

challenge while applying edge computing such as privacy, security, scalability,

reliability, speed and efficiency

2.3.2 EDGEAI

Edge AI [22] is the combination of Edge Computing and AI (Artificial

Intelli-gence) with the objective to accelerate the speed of processing data at the edge

nodes but perform a suitable performance in accuracy Moreover, the benefit

of this technology is that instead of sending all the data to the server, which

wasting a large amount of resource and time for transferring, local devices only

need to send the results, containing the specific and important data after being

computed Furthermore, these calculations do not depend much on internet

connection speed and require intermediate data processing (image and video

analysis) such as surveillance cameras or self-driving vehicles

2.4 NVIDIA JETSON XAVIER NX DEVELOPER KIT

2.4.1 WHAT AND WHY ISNVIDIA JETSON PLATFORM

NVIDIA® Jetson™ developer kits [23] are used commonly around the World for

developing and testing software for products based on Jetson modules, and for

Figure 2.1: Edge Computing Infrastructure.

[21]

2.4.NVIDIA JETSONXAVIERNX DEVELOPERKIT

2

11

projects and learning by professionals and students, in turn Moreover, these

devices get a lot of support and newly technology from NVIDIA for deploy the

most efficient Edge Computing System with combination with Edge AI These

tools and technologies can be listed as:

Opencv with CUDA and cuDNN support: A collection of classes and functions

are included in this module to make use of CUDA’s computing power It only

supports NVIDIA GPUs and is implemented using the CUDA Runtime API from

NVIDIA The OpenCV CUDA module consists of high-level algorithms,

low-level vision primitives, and utility functions While the high-low-level functionality

includes some cutting-edge algorithms (such as stereo correspondence, face

and people detectors, and others) that are ready to be used by the application

developers, the utility functions and low-level primitives provide a powerful

in-frastructure for developing fast vision algorithms using CUDA

TensorRT engine: An SDK that enables high-performance machine learning

in-ference is called NVIDIA® TensorRTTM TensorFlow, PyTorch, and MXNet are

just a few examples of training frameworks that it is intended to function in a

complimentary way with It focuses primarily on rapidly and effectively

oper-ating a trained network on NVIDIA hardware

A feature of NVIDIA GPUs with NVIDIA Ampere Architecture or subsequent

ar-chitectures called Multi-Instance GPU, or MIG, enables user-directed division

of a single GPU into numerous smaller GPUs Physical partitions offer

dedi-cated computation and memory slices with QoS and autonomous parallel job

execution on portions of the GPU MIG may increase performance for TensorRT

applications with low GPU usage with little to no latency effect The ideal

par-titioning strategy depends on the application

Both C++ and Python provide language bindings for TensorRT’s API, and both

have roughly identical features The NumPy and SciPy Python data processing

toolkits and libraries may be used together more easily thanks to the Python

API The C++ API may be more effective and may better satisfy various

compli-ance standards, such as in applications for the automobile industry

In this project, we will use Yolov4 and Yolov4 tiny model which are pretrained

on darknet platform and convert them into trt file model for target GPUs and

CPUs Therefore, there are an extra is that convert these weight file into ONNX

file before completely converting them into trt files

ONNX: The ONNX transfer standard is the main way TensorRT imports a trained

model from a framework An ONNX parser library is included with TensorRT to

make importing models easier When feasible, the parser is backward

compat-ible with opset 7; inconsistencies can be fixed with the use of the ONNX Model

Opset Version Converter TensorRT operates in two phases In the first phase,

usually performed offline, you provide TensorRT with a model definition, and

TensorRT optimizes it for a target GPU In the second phase, you use the mized model to run inference

opti-Building phase:The Builder (C++, Python) is the highest-level interface for

Ten-sorRT’s construction process The builder is in charge of crafting an Engine andoptimizing a model

To construct an engine, you must:

• Create a network definition

• Specify a configuration for the builder

• Call the builder to create the engine

The model is defined using the NetworkDefinition interface (C++, Python) Themost popular method for transferring a model to TensorRT is to export it inONNX format from a framework and then populate the network specificationusing TensorRT’s ONNX parser The Layer (C++, Python) and Tensor (C++, Python)interfaces of TensorRT may also be used to build the specification step-by-step

Whichever method you use, you must additionally specify which tensorsserve as the network’s inputs and outputs Tensors that are not designated asoutputs are thought of as transitory values that the builder can remove throughoptimization In order for TensorRT to know how to connect the input and out-put buffers to the model at run time, input and output tensors must be givennames

The method TensorRT should use to optimize the model is specified via theBuilderConfig interface (C++, Python) You may limit the selection of CUDA®kernels TensorRT can use, regulate the ability of TensorRT to minimize compu-tation precision, and adjust the trade off between memory and run time exe-cution speed You may choose how long the constructor runs for as it can takeseveral minutes

You may call the builder to create the engine once you have defined the networkand configured the builder To make operations perform more quickly on theGPU, the builder gets rid of unnecessary calculations, folds constants, and re-orders and combines processes By executing them in 16-bit floating point or byquantizing floating point data so that calculations may be done using 8-bit in-tegers, it is optional to lower the precision of floating-point computations Themodel is then executed according to an ideal schedule that minimizes the cost

of the combined kernel executions and format transformations It also times

2.4.NVIDIA JETSONXAVIERNX DEVELOPERKIT

2

13

various implementations of each layer with different data formats

The engine is built by the builder in a serialized form known as a plan, which

may either be instantly deserialized or stored to disk for later

Running phase: The highest-level interface for the execution phase of

Ten-sorRT is the run time (C++, Python)

When using the run time, you will typically carry out the following steps:

• Deserialize a plan to create an engine

• Create an execution context from the engine

Then, repeatedly:

• Populate input buffers for inference

• Call enqueueV3() on the execution context to run inference

The Engine interface (C++, Python) is a model that has been improved You can

ask an engine about the network’s input and output tensors, including the

an-ticipated dimensions, data type, data format, and other details

The primary interface for calling inference is the ExecutionContext

inter-face (C++, Python), produced by the engine You may have numerous contexts

connected to a single engine and use them simultaneously since the execution

context holds all of the state connected to a certain invocation

The input and output buffers must be configured correctly before using

in-ference This might reside in the CPU or GPU memory, depending on the type

of data If it isn’t clear from your model where to place the buffer, you can ask

the engine for guidance

Inference can be enqueued (enqueueV3) once the buffers have been

config-ured Control is quickly restored to the application after the necessary kernels

are enqueued on a CUDA stream Control might not come back right away in

some networks because numerous transfers of control between CPU and GPU

are necessary Utilize cudaStreamSynchronize to synchronize on the stream

and wait for asynchronous execution to finish

Plugins: Applications can offer implementations of operations that TensorRT

does not natively support using the Plugin interface provided by TensorRT The

ONNX parser can locate plugins developed and registered with TensorRT’s

Plug-inRegistry while interpreting the network

You can also write your own plugin library and serialize it with the engine.Also, TensorRT supports FP32, FP16, INT8, INT32, UINT8, and BOOL data types.Due to lack of time (training AI model consuming a lot of time), we just usingFP16 in this thesis and using it for testing on other devices

Based on NVIDIA official website, applied these tools and technologies willutilize our AI models which are deployed on the Jetson Platform and get the outperformance in terms of accuracy and speed

In details, it can be seen in the picture 2.2 and 2.1 that the speed (presented

by FPS) on NVIDIA Jetson Platform outweight other platforms However, withTensorRT engine, the accuracy of AI models after being optimized

Figure 2.2: CUDA support FPS performance comparison.

[24]

2.4.2 NVIDIA JETSON XAVIERNX DEVELOPERKIT

The NVIDIA® Jetson Xavier NX™ Developer Kit enables development of featured, multi-modal AI applications for products based on the Jetson Xavier

full-NX module

According to NVIDIA Forums and Experts, the optimized models with full nologies support on Jetson Xavier NX Developer Kit and Jetson Nano DeveloperKit are around 20 to 30 FPS and 14 to 20 FPS, respectively On this thesis, we aim

tech-2.4.NVIDIA JETSONXAVIERNX DEVELOPERKIT

to archive the best performance we can do based on this information

3.1 PROBLEM STATEMENT REVIEW

The Smart Parking System is aimed to reduce the human resource needed to pervise parking lots and provide drivers a convenience way for finding places topark their car To meet the requirement, some basics problem that we need todeal with are: Detect parking’s lot status, suggesting and navigating to the park-ing place and booking the parking space and securely check in and check outvehicles Moreover, real-time data transfer is critical problem not only in smartparking system but also in many internet of things system If there is a signifi-cant delay in transferring information about the actual status of a parking lot tothe customer, the customer may receive inaccurate information This can re-sult in the customer taking actions that conflict with the actual data flow in thesystem, potentially causing errors or bugs Therefore, it is crucial to minimizethe delay in transferring information to ensure that customers receive accurateand timely updates about the parking lot’s status

su-3.2 GENERAL FLOW OF SYSTEM

Our workflow of this systems can be separated into 2 main parts:

3.2.1 CUSTOMER’S MOBILE APPLICATIONWORKFLOW

In can be seen from the Illustration Figure below that the whole process rience 6 steps:

expe-• Step 1: Customer will book the ticket for using the parking service, in thisstep, the mobile apps will analyst data from customers and hardware de-vices for suggestion the most suitable parking zone

17

• Step 2: The application will guide the user to the parking lot through GPStechnology

• Step 3: User will be checked in by Edge devices on the parking zone

• Step 4: This is the hardware part which Edge devices collect data andcomputed into results such as parking time

• Step 5: User will pay bill based on their parking time

• Step 6: Customer’s vehicle is checked out again by Edge devices on ing zone

park-Figure 3.1: Mobile application workflow illustration.

3

19

3.2.2 EDGE DEVICE WORKFLOW

The workflow of Edge device simply collect data from environment and center

data forward by looping and compute to specific data before updating these

data on server

3.3 SYSTEM ARCHITECTURE

After analyzing the problems for building this system, we proposed a solution

with the architecture as in Figure 3.1

• Hardware: Hardware includes a camera for capturing video purpose and

a main board for image processing

• Server: Server receives data from hardware part then gathering

informa-tion and synchronizing data to applicainforma-tion part

• Application: A mobile application for interacting with data from server

We plan to build a website for management in the future

Figure 3.2: Smart Parking System Architecture.

3.4 HARDWARE

The parking area will be photographed in real-time by a video camera that isequipped with an edge device, and the edge device is in charge of collectingvital parking lot data from the frame that was captured and transferring it to theserver The server controls all of the system’s data and synchronizes it with userinteraction and edge device data to prevent conflicts The server also serves

as a resource for customers to get helpful information and aids the owner inmanaging their parking spaces via the application Users now have access to

a mobile application that allows them to communicate with both the serverand the parking space By utilizing this program, users may determine wherethey are right now and identify the best parking spots The mobile applicationalso allows users to view and update several information related to the parkingplaces such as location, price and status (free and booked lots)

3.4.1 LICENSE PLATE RECOGNITION

License plate detection is a component of a smart parking system that allows forautomated entry and exit of vehicles The license plate detection system mustoperate in real-time to provide up-to-date information about parking spaceavailability.The license plate detection system must be able to operate in dif-ferent lighting conditions and handle various types of license plates, includingthose that may be obscured or partially covered The license plate detectionsystem should be integrated with other components of the smart parking sys-tem, such as the server and user interface, to ensure smooth communicationand seamless functionality

On this project, we will use Yolov4 tiny model which is trained and optimized bytensorRT engine with FP16 of NVIDIA For license plate recognition, we use thepretrain simple CNN model All of data for license plate detection, recognitionand CNN model are based on this report[26]

In this thesis, we proposed a solution using Jetson Xavier NX developer Kit withCUDA, tensorRT support to accelerate for archiving the good performance notonly on speed but also on accuracy Our system will get picture from usb cam-era for capture user vehicle before preprocessing this picture Next, the pro-cessed picture will be detected and classified by Yolov4 tiny custom and basicpretrain CNN model to get the license plate number of vehicle Finally, this re-sults will be update to server for further steps

3

21

Figure 3.3: License Plate Recognition Module Architecture.

3.4.2 PARKING LOT OCCUPANCY DETECTION

The primary and critical function in a smart parking system is the accurate

de-tection of parking space availability This information is essential for drivers to

choose the most suitable and available parking space It is necessary that the

detection task operates in real-time and updates the server where all the

sys-tem’s data, including parking places, spaces, prices, and user information, is

collected The detection must provide highly precise and accurate information

to avoid conflicts during the booking process for drivers

To begin, our system will get the picture of whole view parking zone that device

handle and manually create the parking lots in rectangle shape These parking

lots will be collected their 4 coordinates Then, the picture from usb camera

will capture real-time event in the parking zone, used for detecting vehicles in

that picture by Yolov4 model and get their coordinates of bounding boxes for

comparing and computing the free parking slots in the zone Eventually, these

output again update to the server

Once again, we still optimize Yolov4 model by tensorRT engine to get the best

speed and accuracy performance

Figure 3.4: Parking Lot Occupancy Detection Module Architecture.

3.4.3 CLOUD-BASEDSERVER FOR PARKING

Figure 3.5: Server Role in Smart Parking System Architecture.

The Figure 3.5 illustrates the basic architecture of the server The serverserves as the main hub which receiving data from the edge devices and themodify the databases using Django framework accordingly to data received In

3

23

addition, the server will transfer the data from databases to parking app for

customers as well as admin The communication between the devices and

ap-plications will be through a set of API we implement using REST API python for

Edge devices and HTTP API for applications

Our server serves as the central communication hub for the system It receives,

process and send information between edge devices and client mobile apps

The server also serves as the admin control center for system The core

func-tionalities of the server is at follow:

• Process data from edge devices

- Receive real-time parking data in parking lots

- Modify parking database based on received data

• Process data from client mobile application

- Receive booking information from user by mobile app

• Allow admin to modify parking lot information

- Parking lot name - Location - Fee

• Send parking lot data to mobile app

- Parking lot basic information

- Parking lot availability

• Send parking lot data to edge devices

- Booking list’s information to process check in and check out

• Secure user interactions (Login/Sign Up, )

• Manage user information

- History of Booking

- Penalty

- Miscellaneous information

3.4.4 MOBILE APPLICATION

In term of user application, we choose android as base platform for ing and it has been written in Flutter Our Proposal application will follow thegeneral structure as figure below:

develop-Figure 3.6: Android application general work flow

There are three main procedure that occur in the system:

• User - Application: User request through user interface then mobile plication will request to 3rd party api service or back-end server

ap-• Application - Back-end server/Database: Back-end server receive user quest and response to mobile application accordingly Then applicationhandle the data and display to user

re-• Application - 3rd Party API: Applicatin use map, routing and searchingfrom 3rd party API All response will send to application to process

4.1 HARDWARE

4.1.1 SET UP

C ONFIG THE HARDWARE DEVICES

As mention above, we will use Opencv with CUDA and cuDNN, tensorRT ports for our project Moreover, tensorflow is required for our pretrain CNNmodel, however, with the latest Jetpack of Jetson Xavier NX Developer Kit, thetensorflow-gpu is not supported, causing a minor issue, we will explain it clearlylater

sup-The detail of configure on Jetson Xavier NX Developer Kit is illustrated on theFigure below It can be seen that tensorRT, CUDA and cuDNN are already in-stalled with NVIDIA Jetpack Moreover, we install Opencv 4.6.0 with CUDA sup-port is ON mode, Figure 4.1

T RAINING MODEL

On the previous chapter, the parking space occupancy detection model andlicense plate recognition model are pretrained The parking lot occupancy de-tection use the basic model trained in COCO data with 80 classes for any Yolomodel However, we do not have any standard dataset for license plate in Viet-Nam, so that the dataset in this project will be combination of dataset in thisreport [26] and several extra image of ours Furthermore, the training set datadoes not fully cover three common type of license plate in VietNam (blue, whiteand yellow) leading to being pre-processed these data by gray-scaling is verysignificant for eliminating the aspect of color but keep the size and shape as-pect After that, we train the model based on darknet guide before optimizingboth models by tensorRT engine

25

For the basic CNN model, the data include 31 letters in license plates of Nam and one class named Background for trash data from segmentation phase

Viet-Figure 4.1: Setup configuration on Edge device

4.1.2 CREATE PARKING SPACES

For parking lot occupancy detection module, first of all, we will get the picture

of the parking zone that the device handling Then, we draw several boundingboxes in rectangle shape for each parking slot and get 4 coordinates of theseboxes These data with be written into yml file for using later, Figure 4.2

4.2 DEPLOYMENT

4.2.1 LICENSE PLATE RECOGNITION

The activity of this module is described as follow

Figure 4.3: Activity diagram of license plate recognition task.

P RE - PROCESSING IMAGES FOR LICENSE PLATE DETECTION

Firstly, we will capture the image of user’s vehicle and then convert it into scale color for input of our Yolov4 tiny custom, Figure 4.4

gray-L ICENSE PLATE DETECTION AND PRE - PROCESSING FOR LICENSE PL ATE RECOG

-NITION

After that, the image will be detected by our Yolov4 tiny custom model for tecting license plate Next, the license plate will be cropped from the originalimage by its bounding box’s coordinates before convert to HSV and converteach bit of number again to white color for segmentation numbers in the plate.Then, these number will be inputs of CNN model for classification, Figure 4.5