Khóa luận tốt nghiệp Hệ thống thông tin: A mobile-based application for smart tiny parking lot

21 Figure 3-5 SSD MobileNet v2 Archiite€fIIFr€...--e-sceeeceeeeeeeeerrrrrrreeeeeeeeeee 22 Figure 3-6 Single shot detector Network Process ws 2 Figure 3-7 License plate detected after eva

VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY

UNIVERSITY OF INFORMATION TECHNOLOGY

ADVANCED PROGRAM IN INFORMATION SYSTEMS

TRAN DUC THINH BUI TAN LOC

THESIS GRADUATION

A MOBILE-BASED APPLICATION FOR SMART TINY

PARKING LOT

BACHELOR OF ENGINEERING IN INFORMATION SYSTEMS

HO CHI MINH CITY, 2023

VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY

UNIVERSITY OF INFORMATION TECHNOLOGY

ADVANCED PROGRAM IN INFORMATION SYSTEMS

TRAN DUC THINH - 18521450

BUI TAN LOC - 18521002

HO CHI MINH, 2023

ASSESSMENT COMMITTEE

The assessment committee is established under the Decision

date oo by Rector of the University of Information Technology.

For this graduate thesis, we would like to express our sincere gratitude to

my advisor, Ph.D Nguyen Thanh Binh, who wholeheartedly taught and

supported us during our university studies and completion of this graduate thesis In addition to teaching, suggesting knowledge for the thesis, giving

presentations, and commenting on everything, he also cares about students'

psychology, and listens, inspires, and motivates us Without her support and

advice, we probably wouldn't have done my best job.

At the same time, we also want to send to the teachers and teaching

assistants at the University of Information Technology, especially the teachers

in the Faculty of Information Systems, who have imparted valuable knowledge

to me during these 4 years here Therefore, we have improved ourselves and are ready for the future career path.

We consider this an important milestone in my education and development.

We will try to use the skills and knowledge gained in the best possible way and we will continue to improve more in the future Once again, we would like

to thank everyone for giving me the motivation and creating the most

convenient conditions for us to complete the graduation thesis.

Sincerely,

Tran Duc Thinh

Bui Tan Loc

TABLE OF CONTENTS

Chapter 1 PROBLEM STATEMENT 0 cccccecescesessesesesesteseseseeseseesesesneenenes 3

lì No ha

1.2 Aims and ObJ€CfIV€S: ¿+ 1t 1121211 T HH HH HH HH 4

1.3 Scope Of StUdy cành HH HH HH HH TH Hư 4

1.4 Structure of th€SiS - 552222 32222 2 11112121 11211101 re 5

Chapter 2 THEORETICAL BACKGROUND AND LITERATURE REVIEW 6

2.1 Solutions available on the market -+-5+5s+5+cc+sxeeererxerrrrrrrrrer 6 2.2 Framework and technỌOgIGS - - ¿+52 2 2S S*+E£keEeEvErkrkekererkrkee 8 2.3 Near field communication (NFC) ccccccccscseeseseeseseeseseeseeseseeeesseseeeeseeassee 0

2.4 Automatic license plate đeteCtiOn ¿ ¿+55 5c+2++x+xexexerexersrrree 5

2.5 Automatic license plate reCOgnitiOn 5:52 5+5+2c+c+c+scesxzxsrerre 6

Chapter 3 METHOD AND EXPERIMENT RESULTS OF LICENSE PLATE

RECOGNITION 55221 12t 1 HT” H001 TH HH gi 8

3.1 Automatic license plate detection model ‹s <+s<5s+s+x+s+s++ 8

3.1.1 Preparing experiment đa(aSe( Ăn 8 3.1.2 _ Image annotat(iOn - «thư 9 3.1.3 TensorFlow Record .S: cty 21 3.1.4 Single shot detector MobileNet V2 (SSD MobileNet V2) 21 3.1.5 Automatic license plate detection model - - «+ +++ 24 3.1.6 Evaluate detection model oo ¿+ + + + +k+k+xexekrrkrkekereree 24 3.2 Auto License Plate Recognition method -. ¿s5555++++s+s+++ 27

3.2.1 Convert TensorFlow model to TensorFlow LIte - 27

3.2.2 CameraX and image analySis - «¿65 +ccsrzteksrersre 29 3.2.3 Object cropping and grayscale

3.2.4 Text recognition — Machine learning KÍt - -555+<<+ 31

Chapter 4 DESIGN AND DEVELOPMENT MOBILE APPLICATION

LIST OF FIGURES

Figure 2-1 Smart parking system Simulation css |

Figure 2-2 Compare NFC and RFID frequency LL Figure 2-3 Principle NFC architecture

Figure 2-4 Working principle of the automatic road toll collection system 13

Figure 2-5 Types of NFC tag on Idrkef ccă «cu LA Figure 3-1 Full process of License Plate Detection and Recognifion 18 Figure 3-2 Stage 1: License Plate Detection pr0C€§S e LB Figure 3-3 Dataset sdIHpÏe sieu LO

Figure 3-4 Labellmg annotation tool to label map image 21

Figure 3-5 SSD MobileNet v2 Archiite€fIIFr€ e-sceeeceeeeeeeeerrrrrrreeeeeeeeeee 22 Figure 3-6 Single shot detector Network Process ws 2)

Figure 3-7 License plate detected after evaluate Iéodel -.-. -«e«««««< «2È

Figure 3-8 Evaluate result dete€fion -.‹.c«eesc-secerkeeieieerrirriiiiirrrrro 2D) Figure 3-9 Stag 2: License Plate Recognition Proce sss 27 Figure 3-10 TensorFlow to TensorFlow Lite conversion Architecture 28 Figure 3-11 Scan activity with CaImer@X -ccccccecerseeeeriieerseerri.uue 2D Figure 3-12 Cropping image process.

Figure 3-13 Gray-scale image process

Figure 3-14 Result of license plate recognition cece 3/2 Figure 3-15 Best case with correct recognition reSuÏf 3⁄4 Figure 3-16 Worst case incorrect recognition F€SHÏI - OD Figure 4-1 System Architecture wccessscssssssssssessesssesssessssnsessesessesssnissssessssnsneesessnnns OO Figure 4-2 Parking-in session workflow diagram vse 37 Figure 4-3 Parking-out session workflow diagraim 3Ö Figure 4-4 Report lost card workflow diagram uses OD

Figure 4-5 Main use case diagram of Smart Tiny Parking Lot 40

Figure 4-6 Manage Card Use case DqBTđI - «+ AO

Figure 4-7 Manage Membership Use case Figure 4-8 Manage Profile Use case DÌ4BTAIH ccccccseereriiiiririeerier

Diagraim ò -Figure 4-9 Manage Staff Use case Did gram crests

Figure 4-10 Manage Parking History Use case Did gram Figure 4-11 Entity Relationship Diagram vues Figure 4-12 Scan license plate User Interface -eccceeeeceecrrrr

64

.65

Figure 4-13 Detailed checkout information User Interface

Figure 4-14 Login User Interface

Figure 4-15 Register staff account User ÏnfeFƒA4C€ -eeeeeeeeeeeerrr Figure 4-16 Manage card User Interface wiccccscsssssssssescssssssessssssssnsnsiesssessssnniesess

Figure 4-17 Manage parking history User Ïnf€ï/ƒC€ ecccceceeecveeeeeeeerer

Figure 4-18 Manage staff User Tnter face ceecssscssccscsssessssssesssssssssessessnsnsnessessessnnns Figure 4-19 Manage profile User Interface cccscssssssssssesorsssssesessssnsniisseessessssnnnsesieenss Figure 4-20 Detailed information of each parking history User Interƒace Figure 4-21 Detailed parking history of report lost card User Interface

Figure 4-22 Recovery lost card User Interface sesesssssssssesssesssssssssesssssesssessssesssseesssessets

Figure 4-23 Filter parking history User Interface -ee-eeecce> Figure 4-24 Edit your profile User Interface occurrences

Figure 4-25 Change your password User Interface sess

Figure 4-26 Change password successfully notification User Interface

.78

.79

Figure 4-27 Update profile successfully notification User Interface

Figure 4-28 Add new NFC card User Interface

41

41

42

42 60

63

66

67 68 69 70

77

LIST OF TABLES

Table 3-1 Perforamnce evaluation of license plate recognifioni 33

Table 4-1 Login Use case SpecifTcdfi0n -ceeeseeeeerseaeo FA

44

46

46

Table 4-2 Logout Use case Specification

Table 4-3 Insert new card Use case Specification

Table 4-4 Recovery lost card Use case Specification

Table 4-5 Vehicle parking-in session Use case Specification ics +9 Table 4-6 Vehicle parking-out session Use case Specificafion 0 Table 4-7 Filter parking history Use case Specification «0 D1

Table 4-8 Report lost card Use case Specification e«eeceecereeceeeee DL

Table 4-9 Change password staff Use case Specificafion - DD Table 4-10 Register new staff account Use case Specifïcafi0H OA

Table 4-11 Search staff Use case Specification rcs Ð

Table 4-12 Change password Use case Specification wees 90 Table 4-13 Edit profile Use case Specification ou 7 Table 4-14 Get membership information Use case Specification 5Ö Table 4-15 Reset point Use case Specification ssscsssicccccsessneesennneesennnn 29

Table 4-16 Database description for users table .61

-61

Table 4-17 Database description for card table

Table 4-18 Database description for customer table wien OZ

Table 4-19 Database description for parking history table O3

LIST OF ACRONYMS AND ABBREVIATIONS

No Acronyms Meaning

1 LPR License Plate Recognition

2 VLP Vietnamese License Plate

3 LPD License Plate Detection

4 ALPD Automatic License Plate Detection

5 ALPR Automatic License Plate Recognition

6 RFID Radio Frequency Identification

7 NFC Near Field Communication

8 VLP Vietnam License Plate

9 ID Identification

10 JVM Java Virtual Machine

11 MVC Model-View-Controller

12 P2P Pear to Pear

13 ECDSA Elliptic Curve Digital Signature

14 OCR Optical Character Recognition

15 ML Kit Machine Learning Kit

17 SSD MobileNet V2_ | Single Shot Detector Mobile Network version 2

18 FPS Frame per second

Derived from economic and social development, the number of vehicles increases leads to an increase in people's participation in traffic Smart traffic management systems are gradually appearing and developing, to reduce work for people Many technology solutions in the field of transportation for vehicle management and monitoring, one of them is automatic license plate recognition technology This technology helps to recognize license plates faster and more accurately by processing

a photo of the vehicle's license plate The tollbooth uses this system to recognize

license plates to collect tolls for vehicles, CCTV system on highways to recognize speed and license plate information of moving vehicles to monitor traffic.

In recent years, the traffic of vehicles concentrated in public places is very large Management of vehicles in public places and parking lots is a problem that needs to

be solved This system requires high-cost specialized equipment such as cameras, PC,

barriers, and RFID/NFC card readers Therefore, it is not feasible for cafes,

restaurants, eateries, and convenience stores to use this smart parking system because

of the installation area and operating costs Smartphones combine all components of the parking system, so the solution to this problem is to build a mobile application that integrates the license plate recognition algorithm.

In this graduate thesis, I focused on detecting and recognizing Vietnamese license plates from images and videos in real-time based on a mobile application and NFC technology to store vehicle management history at parking lots This application

helps to solve the problem of mobility and flexibility for parking staff, and area and

saves installation and operating costs, with high accuracy Along with that, I use

customer information to visit the store to support the ERP, CRM system for managing the membership reward of that store.

Chapter 1 PROBLEM STATEMENT

1.1 Rationale

The license plate is an identifier for each vehicle Current popular solutions for

managing vehicles, parking lot guard will write the license plate on the parking pass, then they will give you the parking pass The disadvantage of paper parking card is that it is easy to lose, get wet, or tear, and it is not difficult for bad guys to fake it.

License plate recognition is reading and analyzing, recognizing the content of the

license plate to manage vehicle information And the fact that in terms of recognizing

and checking license plates humans can make mistakes sometimes and can't do as

well as machines, at least humans can only identify well in the number of vehicles.

In the current digital age, it is important for a store to collect customer information

to serve many purposes such as marketing, understanding satisfaction, and

membership management Stores often use the method of accumulating points with

a member loyalty card containing personal information or by requesting a customer's

phone number to accumulate points However, personal information is easily

collected and leaked by the development of technology So, customers are very concerned about the security of personal information and feel uncomfortable about

sharing too much information for many stores Our application uses the customer's

parking information, each license plate, and a number of personal identifiers of each

customer, combined with NFC cards such as membership cards to manage membership, make customers feel comfortable and surprised when they don't have to

share information and still receive membership point.

Therefore, the study of this problem is necessary In my major is Information

Systems, this problem has certain relevance in the system definition, the mobile

application is a vehicle management system, and we can use the data source of the application to be integrated ERP/CRM system for managing the membership reward

of that store.

1.2 Aims and Objectives:

With this graduation thesis, the objectives that I aim for when doing this is as

follows:

e Applying NFC technology to create communication between an NFC tag and

android-powered device to store and query data stored in NFC tag.

e Learning about license plate recognition, how to apply TensorFlow Lite and

OCR to solve that problem.

e Understanding and applying the model to solve the problem of license plate

recognition, thereby building an android application using this model and NFC technology to smart vehicle parking management application.

1.3 Scope of study

This course focuses on the study of the Automatic License Plate Recognition

method The automatic license plate recognition system is a system capable of analyzing and processing images license plate on the images taken directly from the

camera The input to the system is a photo of a vehicle with a license plate taken

directly from the smartphone's camera The output is the character string on the number license plate This Automatic License Plate Recognition method applies the

knowledge of Machine Learning and TensorFlow Lite tool to create license plate

detection machine learning model We also use Labellmg tool graphical image

annotation to draw visual boxes around license plate in each image.

With the application of this Automatic License Plate Recognition method to

practical applications for tiny parking lot management This system was developed

to accurately recognize Vietnamese license plates If used to recognize foreign license

plates, the accuracy will be reduced or not recognized because we use the Vietnamese

license plate dataset to train and draw the detection frame according to the size of the Vietnamese license plate Besides that, the system only processes license plate images with white background and black text The system uses NFC tags to store and retrieve

the vehicle's parking access history instead of using a paper parking pass.

1.4 Structure of thesis

The thesis report is organized as follow:

e Chapter 2 shown the theoretical background and literature review.

se Chapter 3 is method and experiment result of license plate recognition

processing.

e Chapter 4 is design and implementation mobile application.

¢ Chapter 5 conclusion after study about license plate recognition with

machine learning.

Chapter 2 THEORETICAL BACKGROUND AND LITERATURE

REVIEW

2.1 Solutions available on the market

With the continuous economic and social development along with the rapidlyincreasing number of cars and motorbikes in our country, a shortage of qualityparking lots has arisen Therefore, a smart parking system was born and graduallyreplaced traditional parking lots to improve operational efficiency and better control

access.

Smart parking system includes components such as: the guard cabin contains 1computer, RFID cards, 2 RFID card readers, 4 cameras and 2 barriers This systemrequires a large area to install and must install electrical infrastructure The functions

of the above components:

e Cabin with computer: This is an important part of the system located at the

input and output ports The main function is to house the computer and parking

management software and is attached with 2 RFID card readers

e Camera system: Includes license plate recognition camera and panoramic

camera The license plate recognition camera will specialize in reading and

recording license plates, so it must be connected to the smart parking

management system Able to clearly see license plates in different lightingconditions The panoramic camera supports driver control and the overallparking gate area Therefore, it is necessary to choose cameras with facialrecognition, and action analysis,

e RFID card and card reader device: The main function will replace paper

parking pass Each RFID card will have a unique ID to make security more

secure.

e Barriers: Acts as a barrier gate to help reduce the speed of vehicles in and out

and better control security

The working principle of smart parking:

e When entering the parking lot, you will be issued with an RFID card registered

in the system When swiping the Barrier card, it will open, and the camera willcapture the license plate, analyze it into a number and identify the driver Atthis time, the license plate and card number are stored with the image of thedriver Captured by 2nd Camera to execute query when needed

e When taking out the car, the driver will give the RFID card to the staff At this

time, the software will automatically calculate the parking time, the camera atthe exit gate will capture the license plate to compare with the license platewhen the car enters If there is a difference in license plates when entering andexiting, the system will notify you by sound or voice for security to know andcheck again The picture of the driver at the time of entry is also shown to theconfirmation staff To support the most convenient and accurate control

Computer RFID CardReader RFIDCard Switch Network ups Camera IP

Figure 2-1 Smart parking system simulation

Scan License Plate Camera

This parking system is often installed in apartments, hotels, hospitals, and malls.This system requires a large installation area, and the cost is quite expensive.Smartphones combine all components of the parking system, so the solution to this

problem is to build a mobile application that integrates the license plate recognitionalgorithm.

2.2 Framework and technologies

Android is currently the most popular mobile operating system today with anaverage market share of more than 50% worldwide It is an open-source Linux-basedoperating system designed for touch screen mobile devices such as smartphones andtablets It is the open-source code coupled with the unfettered license that allowsdevice developers, mobile networks, and programmers to adapt and distributeAndroid freely These factors have helped make Android the most popularsmartphone platform in the world Android accounted for 75% of the worldwide

smartphone market share as of Quarter 3 2012, with a total of 500 million devices

activated and 1.3 million activations per day

Developing an Android app that runs well on a particular Android phone or tablet

is simple However, developing an app that runs well on all Android devices can beextremely challenging because of the huge differences in operating system versions,

screen sizes, and device capabilities

Advantages of android:

e As an open-source operating system, it is highly customizable, can be freely

edited without interference or prohibition from Google

e Diverse products, many phone companies, technology devices have favored

Android for their devices, the price is reasonable from popular to high-end

e The massive Google Play Store app store

e Friendly and easy to use

e Ability to multitask, run multiple applications at the same time

Disadvantages of android:

e Vulnerable to malware and viruses Due to its open-source nature, many

unregulated software of poor quality or security flaws are still used

e Too many app stores lead to difficulty in quality control, lack of good apps

e Update not automatically for all devices When a new version of the operating

system is released, not all products are updated, even if you want to experience

it you often have to buy a new device

Android devices are increasingly upgraded and support more hardware sensors.The advancement of technology and user experience is driving the growth of themobile app design market Each new update of new operating systems will create

new improvements in mobile app design and features to better meet user needs Here

are some trends in mobile application development:

e 5G network

e Artificial Intelligence (AI)

e Machine learning, deep learning

e Computer vision

e Cloud computing

e Blockchain mobile app

Through mobile application development trends, in this thesis graduation, we

focused on machine learning, computer vision, near field communication (NFC),cloud computing This is the hot trend technology

Java is a very famous and widely used object-oriented programming language.Based on the advantages of Java, it has gained popularity and many configurationshave been built to suit various types of platforms including Java SE for Macintosh,Windows and UNIX, Java ME for Mobile Applications and Java EE for Enterprise

Applications

Java provides cross-functionality and greater portability because programs written

on one platform can run on desktop computers, mobile phones, embedded systems

It is free, simple, object-oriented, distributed, supports multithreading, and providesmultimedia and network support The Java class library enables cross-platformdevelopment Unlike C and C++, Java programs are compiled platform-independently using a bytecode language, allowing the same program to run on anymachine with a JVM installed.

Laravel is a free and open-source PHP framework, built to support thedevelopment of software and applications, following the MVC architecture.Currently, Laravel is the most popular and best PHP framework

The PHP framework is a library that makes the development of web applicationswritten in the PHP language more fluid By providing a basic structure to build thoseapplications In other words, the PHP framework helps you speed up the applicationdevelopment process Help you save time, increase the stability of the application.Reduce the number of times you have to rewrite code for programmers

MySQL is an Oracle-backed open-source relational database management

system (RDBMS) MySQL is used as a database at the webserver and PHP is used

to fetch data from the database on your android application It was designed to be

compatible with other systems, it supports deployment in virtualized environments

such as cloud, [1]

2.3 Near field communication (NFC)

Near field communication — NFC is a set of short-range wireless technologies,

typically requiring 4cm or less to initiate a connection It enables two devices to

communicate with one another when they are in close together NFC tags are passivedevices that can be read by a smartphone or other NFC-enabled device It is a branch

of HF RFID (High Frequency RFID) technology and operates under a frequency of

13.56 MHz in the globally available unlicensed radio frequency ISM band using theISO/IEC 18000-3 air interface standard at data rates ranging from 106 to 424 kbit/s

10

Figure 2-2 Compare NFC and RFID frequency

The principal operation of NFC is similar to RFID It uses magnetic induction(based on the NFC chip available in the device) to make a connection between devices(smartphone, tablet, speaker, headset ) when there is direct contact (touch, heremeans 2 devices are very close to each other, touching each other)

Figure 2-3 Principle NFC architecture

Android-powered devices with NFC simultaneously support three main modes ofoperation: [2]

11

e Reader/writer mode, allowing the NFC device to read and/or write passive NFC

tags and stickers

e P2P mode, allowing the NFC device to exchange data with other NFC peers; this

operation mode is used by Android Beam

e Card emulation mode, allowing the NFC device itself to act as an NFC card

The emulated NFC card can then be accessed by an external NFC reader, such as

an NEC point-of-sale terminal

The automatic road toll collection system is applied RFID (Radio FrequencyIdentification) technology using radio waves to automatically identify motorvehicles When the vehicle enters the recognition area of the RFID reader system, theRFID tag will communicate with the reader system to retrieve the tag's information.Then it will transmit the data to the server to check the card status and vehicleinformation (priority card, lock, ban, account balance, license plate number) Thenwill transmit the signal to open/close the barrier according to the status of the card.Thanks to the above method, toll collection at the stations is no longer difficult: there

is no longer waiting and congestion like the tradition; saving time for vehicle driversand station attendants; saving money in hiring personnel, station maintenance costs,other arising; create a change in awareness and civilized and modern behavior forpeople

12

Reader System

Figure 2-4 Working principle of the automatic road toll collection system

NFC devices have the advantage of limiting radio frequencies in proximity.Because NFC devices must be in close contact with each other, usually no more than

a few centimeters, it has become a popular choice for secure communicationsbetween devices, such as smartphones With NFC, only one tag can be scanned andreceived at a time While RFID can identify multiple tags at once within a certain

range.

Each NFC parking card has a unique ID and paper parking pass also has a uniqueidentity number and a logo that identifies the shop However, the disadvantage of apaper parking card is that it is easy to lose, get wet, or tear, and it is not difficult forbad guys to fake it So, NFC tag combined with an android device's NFC tagread/write technology is the solution

The type of NFC tag used in this project is NTAG215, it is a medium-size memorycapacity; the available memory is 504 bytes NFC Tags are rewritable by default.Potentially, the NFC Tag can be rewritten endlessly They are guaranteed to berewritten up to 100,000 times (depending on the IC) This function allows you to lock

13

the encoding of an NFC Tag with a password This means that those who'll find apassword-protected NFC Tag will be able to read it, but not to edit or delete the storeddata.

=

NFC Ntag215 ——————

NFC Ntag215 circle NEC Keyfob

Figure 2-5 Types of NFC tag on market

NTAG2Ix features a cryptographically supported originality check This checkcan be performed on personalized tags as well NTAG21x digital signature is based

on standard Elliptic Curve Cryptography (curve name secp128r1), according to the

ECDSA algorithm Each NTAG21x UID is signed with an NXP private key, and the

resulting 32-byte signature is stored in a hidden part of the NTAG21x memory during

IC production [3]

The working process of NFC tag with Android device is NFC tag and android

device communicate with each other, android device will get the ID of the NFC tagand turn on the camera to ready detect and recognize the license plate number After

14

license plate detection and recognition, the system will be mapping the ID of the NFCcard and license plate number and store it into the database When customers want toget out of the parking lot, staff will touch the NFC card to the device, the system willget the ID card and query with the condition ID card on the database to get parkinghistory information of this vehicle with ID NFC card, license plate number, time get

in the parking lot

2.4 Automatic license plate detection

Nowadays, object detection is a computer vision technology related to imageprocessing that deals with detecting instances of semantic objects of a certain class indigital images and videos Automatic license plate detection is used not only by thepolice, but also by parking facility managers, security agencies, and other traffic

control institutions ALPD is identifying the part of the car that is predicted to be thenumber plate License plate detection and recognition is the technology that uses

computer vision and machine learning to detect and recognize a license plate from aninput image of a motorbike License plate detection is a necessary procedure beforelicense plate recognition [4]

There are some challenges that affect accuracy detection First challenges are theimage quality taking by camera in real time may be affect by severe weatherconditions, poor lighting conditions, and low camera resolutions The aperture time

of the camera will cause the blurring effect on the license plate The second challenge

is the complex background containing many objects on the captured image [5]

License plate detection dataset is a data set used to train the model to detect licenseplates in the image The dataset sample is a set of images of vehicles parked in theshopping mall parking lot The images captured by the parking system camera will

be used to draw the bounding box annotation, which is the process of labeling objects

in an image with rectangular boxes that can quickly identify the license plate

15

In this thesis, I focused on Vietnam motorbike license plate with black characterand white background Our target is to create a detection with 80% or more accuracy

and a high recognition rate per frame

2.5 Automatic license plate recognition

Automatic license/number plate recognition is a specific application of optical textrecognition ALPR can be used to cite people who ignore traffic signals or drive toofast, as a method of electronic toll collection, or to locate a suspect at a crime scene

ALPR is the next procedure after ALPD processing successfully

To achieve the highest accuracy for license plate number recognition, we not onlyapply existing solutions such as optical character recognition, text recognitionMachine Learning kit API, and character segment but also have pre-cognition stepslike cropping license plate number region, converting RGB image to gray-scaleimage Cropping image help to focus on region want to recognize, avoid recognizingunnecessary objects of complex background image Next step of pre-cognition isconverted RGB to Grayscale in image This step help to reduce color complexity,noise reduction, and high speed and accuracy to recognize license plate [5]

The important step in ALPR is text recognition Currently, the application of

machine learning to large systems that support many management and recognitiontasks is increasing Applying machine learning to solve everyday problems is alsointeresting We expect mobile apps to be smart - adapting to user activity or delighting

us with amazing features That's why applying machine learning has become such animportant part of mobile app development Bringing machine learning to mobileapplications is now supported by many libraries such as Tesseract, or Android'sMobile Vision However, the disadvantage of the above methods is that the accuracy

is not high, if you want to improve it, it takes time to create sample data as Machine

Learning has evolved Machine learning can be used to increase OCR accuracy We

used MLKit of Google in this project, it provides Google’s machine learningtechnologies many years of research to mobile developers, allowing them to integrate

16

some machine learning models into their apps more easily Currently ML Kit isproviding 5 base APIs and custom model support such as Text recognition, Imagelabeling, Barcode scanning, Face detection, Landmark recognition Therefore, we arefocus on text recognition API to extract characters in the license plate region.

TensorFlow Lite is a lighter version of TensorFlow also developed by Google

TensorFlow Lite is designed to run models on mobile and embedded devices UsingTensorFlow Lite models helps you integrate with ML Kit for easy customization inyour mobile application

Automatic license plate recognition is expected to achieve with the following goals:

e Fast recognition speed

e Ability to recognize plate numbers from noisy, low-quality images

e Ability to work for tilted license plates

17

Chapter3 METHOD AND EXPERIMENT RESULTS OF LICENSE

PLATE RECOGNITION

We start using the above technologies to create models and methods to get the

results of license plate recognition In order to get the result, this chapter consists of

2 stages: License Plate Detection and License Plate Recognition The steps are

shown in the figure below

Figure 3-1 Full process of License Plate Detection and Recognition

3.1 Automatic license plate detection model

Automatic license plate detection method has two main steps First, label map image

or frame from video and create TensorFlow record from it Second, build a training

model to detect license plate

SSD MobileNet v2

TensorFlow Record

3.1.1 Preparing experiment dataset

The license plate detection dataset used to train the model detection; thedetected license plate image used to extract text in the plate region in next

18

procedure ALPR In object recognition research to accumulate large volumes

of image features to improve performance I have a dataset around 400 images

of vehicles parked in the shopping mall parking lot in Vietnam were taken to

create the experiment dataset [4] We need the dataset to train the machine torecognize object ‘license’ — license plate, we will be able to teach the machinehow the motorbike license plate looks like After training model using set ofimages, we will split them into two categories: train and test in an 80:20 ratio.Next, we will start training with 80% of pictures and 20% of pictures will beused for testing the dataset

3.1.2 Image annotation

There are many reasons why your model is underperforming But there aresome causes related to image annotation such as poor-quality training data,lack of accuracy and consistency, or too many duplicate points Therefore, weshould use methods to annotate with bounding boxes to increase accuracydetection

19

Object detection consists of identifying a target object, determining itslocation, and annotating an image of that specific target The technique usesfor object detection is bounding box annotation Annotation tag objects forimage means that the process of adding label or tags to image, it draws the boxaround the objects and makes them more informative, so that machine learningalgorithms can interpret data and help to train a license plate model Thebounding boxes help localize the license plates in the image while license platedetection process.

Here are some things to do when working with annotation: [7]

e The edges of bounding boxes should touch the outermost pixels of the

object that is being labeled

e Box size variations in your training data should be consistent

e Avoid overlap due to the nature of your images, consider labeling the

object with polygons and using an instance segmentation model

Now, we are using Labellmg — flexible open-source data graphical imageannotation tool to start annotating image dataset [5] It is written in Python anduses Qt for its graphical interfaces Annotations are saved as XML file for each

image in PASCAL VOC format

20

Box Labels

File tiết

Figure 3-4 LabelImg annotation tool to label map image

After annotation image, the data set will be divided to train and test for modelbuilding

3.1.3 TensorFlow Record

After using Labellmg tool exported annotations in the VOC format Theannotations were saved as XML files, and then converted it into TF Recordsformat The TF Record annotations will be generated for storing a sequence ofbinary records to use a MobileNet SSD v2 model

TensorFlow Hub is a repository of trained machine learning models that

are ready to be refined and deployed anywhere Reuse trained models like

BERT and Faster R-CNN with just a few lines of code It provides pre-trained,

mobile optimized models that can detect common objects [9]

3.1.4 Single shot detector MobileNet V2 (SSD MobileNet V2)

SSD MobileNet V2 is a one-stage object detection model which has gainedpopularity for its lean network and novel depth wise separable convolutions

It is a model commonly deployed on low compute devices such as mobile(hence the name MobileNet) with high accuracy performance The network

21

predicts the bounding box (region) and the class simultaneously in a singleshot.

The SSD method consists of two components: a base network model (BaseNetwork) and auxiliary neuron layers (Auxiliary Layers) In particular, thebasic network is responsible for extracting image features, then these featuresare used by auxiliary layers of neurons to predict the position of objects in theimage

For the base network, a truncated version of VGG 16 was chosen, which isused for image classification purposes An input image passes through all theconvolution layers to generate multiple convolutional feature maps of differentsizes Non-maximum suppression and Hard Negative Mining (HNM) are used

in this network SSD predicts the bounding boxes and class probabilities in

one shot, which makes it faster than the region-based algorithms [6]

22

The input of the SSD is the bounding box coordinates of the object (alsoknown as the offsets of the bounding box) and the label of the object contained

in the bounding box The special feature that makes the speed of the SSDmodel is that the model uses a single neural network Its approach is based onobject recognition in feature maps have different resolutions The modelcreates a grid of squares called grid cells on the feature map, each cell is called

a cell and from the center of each cell defines a set of default boxes to predictthe frame capable of enclosing objects At the time of prediction, the neuralnetwork will return two values: the probability distribution of the label of theobject contained in the bounding box and a coordinate called offsets of thebounding box The training process is also the process of refining the label andbounding box probabilities to match the model's ground truth input values(including labels and bounding box offsets) [7]

While not the most accurate model, SSD-MobileNet is well suited forapplications that need high speed The more complex the networks, the betterthe feature extraction, but the processing speed is slow The faster the networkcomputes, the lower the accuracy Real-time recognition problems have fastrecognition speed but reduced accuracy So we decided to use ssd networkbecause it is one of the algorithms with high accuracy and relatively fastprocessing speed

23

3.1.5 Automatic license plate detection model

After using TensorFlow and SSD MobileNet v2 to build ALPD model, we

evaluate model to get the detection result We are setting up the frame rate of

camera is 60FPS In 1 second, camera captured 60 frames and detect

localization license plate object The bounding box will appear real-time onthe preview camera The percentage of object predictions set in the model is80% or more If it's from this level or higher, the bounding box will appear onthe preview camera

100

200

300

400 +

Figure 3-7 License plate detected after evaluate model

3.1.6 Evaluate detection model

Precision is a metric that measures the accuracy of a classifier in terms ofpositive predictions In object detection, precision measures the fraction ofdetected objects that are actually correct It is calculated as the ratio of truepositive detections to the sum of true positive and false positive detections

24

A true positive is a detection that correctly identifies an object in the image.

A false positive is a detection that identifies an object in the image, but theobject is not actually present

Precision = (True Positive)/(True Positive + False Positive)

For example, suppose you have a set of 100 images and you are trying todetect apples in these images Your object detection model detects 8 apples in

an image, but only 5 of them are correct The precision of the model on thatimage is 5/8 = 0.625, which means that out of the 8 detections, 5 of them aretrue positives and the remaining 3 are false positives

Higher precision indicates that the model is better at detecting true objects

and avoiding false detections Precision should be balanced with recall, whichmeasures the ability of a model to detect all instances of the objects of interest

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.888

Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=199 ] = 1.000

Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 1.000

Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=@.50:0.95 | area=medium | maxDets=100 ] = @.888

Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000

Average Recall (AR) @[ IoU=@.50:0.95 | area= all | maxDets= 1 ] = @.909

Average Recall (AR) @[ ToU=Ø.50:0.95 | area= all | maxDets= 10 ] = 0.909 Average Recall (AR) @[ ToU=@.50:0.95 | area= all | maxDets=100 ] = @.909

Average Recall (AR) @[ ToU=0.50:0.95 | area= small | maxDets=100 ] = -1.000

Average Recall (AR) @[ IoU=Ø.50:0.95 | area=medium | maxDets=100 ] = 9.909

Average Recall (AR) @[ ToU=@.50:0.95 | area= large | maxDets=100 ] = -1.000

Figure 3-8 Evaluate result detection

The above metrics are evaluation scores for an object detection model

e AP: The Average Precision (AP) is a measure of the accuracy of the

model's bounding box predictions It ranges from 0 to 1, with 1indicating perfect performance

e IoU: The Intersection over Union (IoU) threshold is used to determine

if a predicted bounding box is a true positive or a false positive The

25

IoU is calculated as the ratio of the area of intersection between thepredicted and ground truth bounding boxes and the area of their union.area: This indicates the size of the object instances in the dataset that

is considered when calculating the AP "all" means that the AP iscalculated over all object instances in the dataset, regardless of their

size "small", "medium", and "large" mean that the AP is calculated

only for instances of the corresponding size

maxDets: This is the maximum number of detections per image thatare considered when calculating the AP For example, "maxDets=100"means that the AP is calculated based on the top 100 predictedbounding boxes per image, even if the model generates more than 100predictions

AR: The Average Recall (AR) is a measure of the recall of the model'sbounding box predictions, which is defined as the number of truepositive detections divided by the number of ground truth instances

The AR ranges from 0 to 1, with 1 indicating perfect recall

Based on the metrics, the model has a good average precision of 0.888 (AP

@ [IoU=0.50:0.95 | area=all | maxDets=100]), and an average recall of 0.909

(AR @ [IoU=0.50:0.95 | area=all | maxDets=100]) However, the AP for small

and large instances is not available as it is equal to -1.000 In this case, we

collect a dataset from the parking lot of a supermarket with a large numberplate compared to the photo Therefore, in the data set, small objects will notappear, leading to the average precision and average recall of the model in the

"small" region of -1 And in reality, the guard will point the phone camera atthe license plate area, so the size of the object in the picture will be large In

this context, the object detection model will work well

In summary, the metric AP @ [IoU=0.50:0.95 | area=all | maxDets=100] =

0.888 indicates that the object detection model has an average precision of

26

0.888, which is a good performance for an object detection model A value of0.888 means that the model can accurately detect objects in 88.8% of the cases,

on average, when considering multiple recall values and IoU thresholdsranging from 0.50 to 0.95

3.2 Auto License Plate Recognition method

ALPR is the next procedure after ALPD processing successfully The image afterdetection with bounding box around the object-by-object training detection model of

ALPD After the implementation detection model, the location from the regional

proposal was treated as the bounding box, ALPR will crop image by the boundingbox to localization the license plate After that, converting the image from RGB color

to grayscale during image processing help speed up recognition Next step, from theimage that has been processed through the previous steps, we are using Machine

learning kit text recognition API of Google to extract text from the license plate

region and store text result into database Here are the steps of stage 2:

3.2.1 Convert TensorFlow model to TensorFlow Lite

TensorFlow Lite is a compact version of TensorFlow for mobile It is across-platform machine learning technology library that has been optimized

27

for running machine learning models on edge devices such as Android and

iOS mobile devices

Some advantages of TensorFlow Lite with android application:

e Faster, because TensorFlow Lite enables machine learning to be

performed on-device with low latency

e TensorFlow Lite consumes less storage space

e Optimize kernels for mobile apps for low latency

e Pre-fused activations

We will use Tensorflow for model training, on powerful hardware devices

It is also possible to do mobile and embedded training, but it will take a lot of

time So I will use Tensorflow for model training and Tensorflow Lite forInference After we have the model, we will convert it to Tensorflow Lite, putthat model on the mobile application for inference

To be able to run the model with TensorFlow Lite you must convert themodel into a model (.tflite) which is the format accepted by TensorFlow Lite.You now have your model (.tflite) and label file You can start using the modeland label files in your Android application to load models and predict outputsusing the TensorFlow Lite library

3.2.2 CameraX and image analysis

CameraX in Java is a Jetpack library, designed to make camera appdevelopment easier For new apps, you should start with CameraX Thisapplication provides a unified API, easy to use, works on the majority ofAndroid devices and is backward compatible with Android 5.0 (API level 21).CameraX support some use case such as: preview, image analysis, videocapture, image capture In this project I used video capture and image analysis

use case [13]

The camera management in the application is not easy, for example: aspectratio, orientation, rotation, preview size, high-resolution image size But withCameraX it will be much simpler We are setting up for camera with1920x1080 resolution, 60FPS to be able to perform the detection andrecognition process In a second, camera will capture 60 frames and added it

to model Therefore, the result will appear within | second

Figure 3-11 Scan activity with CameraX

Image analysis provides CPU cache to perform image analysis jobs using

image processing, computer vision, machine learning.

29

3.2.3 Object cropping and grayscale

The act of selecting and extracting the region of interest is known ascropping Cropping object from image help to focus on region want torecognize, avoid recognizing unnecessary objects of complex backgroundimage We crop the license plate are from the video frames We are using thecamera with frame rate of 60fps Therefore, in a second we were taken 60frames and cropped all it based on the bounding box around this

1000 2000

License plate detected Cropped image

Figure 3-12 Cropping image process

A Grayscale Image is an image in which the value of each pixel is a uniquesample representing an amount of light, which means it contains only intensityinformation Grayscale image, a monochrome black-and-white or grayscale,consisting only of shades of gray Contrast ranges from black at its weakest towhite at its strongest [8]

Grayscale images contain only intensity information, which can beunderstood as the brightness level of a particular pixel The higher the value,

the greater the intensity, consisting of 256 different gray levels, each only

slightly lighter than the previous one To store a pixel in a grayscale image, it

takes 1 byte (8 bits) to cover all 256 possible gray levels, from 0 to 255

Some advantages when convert RGB to Grayscale in image after cropped:

30

e Color complexity: When it comes to image processing, you can either

go the traditional route and control the camera's color calibration,brightness, contrast, lighting, and other factors, or you can simplyconvert it to grayscale

e Noise reduction: Color information does not assist the algorithm in

identifying all the image's important edges (changes in pixel value) orother features

e High performance: Grayscale helps to speed up the processing of

RGB images, highlighting the characters that need to be recognized in

the license plate region

Cropped image Gray-scale image

Figure 3-13 Gray-scale image process

Gray-scale reduces color bits, increases contrast for characters, and improves

recognition speed

3.2.4 Text recognition — Machine learning kit

We used Machine Learning Kit of Google in this project, it provides Google’s machine learning technologies many years of research to mobile

developers, allowing them to integrate some machine learning models into

their apps more easily Currently ML Kit is providing 5 base APIs and custom

model support such as Text recognition, Image labeling, Barcode scanning,

31