Anti theft system design using opencv on raspberry pi 4

MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITYUNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT ELECTRONIC AND COMMUNICATION ENGINEERING AN

MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY

UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR

HIGH QUALITY TRAINING

GRADUATION PROJECT ELECTRONIC AND COMMUNICATION ENGINEERING

ANTI-THEFT SYSTEM DESIGN USING OPENCV ON RASPBERRY PI 4

LECTURER: Ph.D DO DUY TAN SONSTUDENT: NGUYEN XUAN THANG

LUU DINH HUONG NGUYEN

SKL009360

Ho Chi Minh City, July 2022

HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION

FACULTY FOR HIGH QUALITY TRAINING

Major: ELECTRONIC AND COMMUNICATION ENGINEERING

Advisor: Ph.D DO DUY TAN

Ho Chi Minh City, July 2022

HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION

FACULTY FOR HIGH QUALITY TRAINING

Major: ELECTRONIC AND COMMUNICATION ENGINEERING

Advisor: Ph.D DO DUY TAN

Ho Chi Minh City, July 2022

THE SOCIALIST REPUBLIC OF VIETNAM

Independence – Freedom– Happiness

*** Ho Chi Minh City, July 29 th , 2021

ADVISOR’S EVALUATION SHEET

Major: Electronics and Communication Engineering

Project title: Anti-theft system design using OpenCV on Raspberry Pi 4

Advisor: Ph.D Do Duy Tan

EVALUATION

1 Content of the project:

The content and workload of the project meets the initial requirements of the project

2 Strengths:

The topic of the project is practical and applicable Extensive experimental results are

conducted to evaluate the performance of the proposed design

− There are many grammatical mistakes and typos in the report

− The performance of the proposed design might be unstable in environments that

light intensity is low

4 Approval for oral defense? (Approved or denied)

Approved

5 Overall evaluation: (Excellent, Good, Fair, Poor) Good.

6 Mark: 8.0 (in words: Eight )

Ho Chi Minh City, July 29 th , 2022

ADVISOR

(Sign with full name)

THE SOCIALIST REPUBLIC OF

VIETNAM

Independence – Freedom– Happiness

-Ho Chi Minh City, July 29 th , 2022

PRE-DEFENSE EVALUATION SHEET

Major: Electronics and Communication Engineering

Project title: Anti-theft system design using OpenCV on Raspberry Pi 4

THE SOCIALIST REPUBLIC OF VIETNAM

Independence – Freedom– Happiness

-Ho Chi Minh City, July 29 th , 2022

EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER

Major: Electronics and Communication Engineering

Project title: Anti-theft system design using OpenCV on Raspberry Pi 4

Advisor: Ph.D Do Duy Tan

Name of Defense Committee Member:

EVALUATION

1 Content and workload of the project:

4 Overall evaluation: (Excellent, Good, Fair, Poor):

Ho Chi Minh City, day… month… year

COMMITTEE MEMBER

(Sign with full name)

HCMC University of Technology and Socialist Republic of Vietnam

EducationFaculty for High Quality training Independence - Liberty – Happiness

Ho Chi Minh City, July 29 th , 2021

PROJECT ASSIGNMENT

Name of Members: Nguyen Xuan Thang

Luu Dinh Phuong Nguyen

Student ID: 18161038Student ID: 18161027Training system: Regular university

Major: Electronics and communication engineering

Class: 18161CLA

ON RASPBERRY PI 4

− Recognize the urgency and importance of the project

− Conducting research, studying materials, current models related to the topic forcomparison and improvement

2 Content of implementation

Task 1: Brainstorm and research related topics

Task 2: Research and select components

Task 3: Draw algorithm flowchart

Task 3: Design and build a complete system model Run tests and calibrate the system.Task 4: Write a report

CHAIR OF THE PROGRAM ADVISOR

(Sign with full name) (Sign with full name)

i

HCMC University of Technology and Socialist Republic of Vietnam

EducationFaculty for High Quality training Independence - Liberty – Happiness

Ho Chi Minh City, July 29 th , 2022

PROJECT IMPLEMENT SCHEDULE

Luu Dinh Phuong Nguyen Student ID: 18161027Class: 18161CLA

Name of project: Anti-theft system design using OpenCV on Raspberry Pi 4

1st - Meet the Instructor to disseminate regulations:

(14/3-20/3) choose the topic and working time

- Instructor conducts a review of the topic

2st - Write the plan and detailed outlines for the

ii

10th - Write a program to alert users via Gmail

(17/5-23/5) account and web server

11th

- Model construction and design(24/5-30/5)

12th

(31/5-6/6) - Implement, test and fix the project

13th - Write the report

(7/6-13/6)

14th

- Complete the report and send it to Instructor toreview and comment for the last time before(14/6-20/6)

printing the report

iii

Student 1: NGUYEN XUAN THANG

Student 1’s ID: 18161038 Class: 18161CLA

Email: 18161038@student.hcmute.edu.vn

Student 2: LUU DINH PHUONG NGUYEN

Student 2’s ID: 18161027 Class: 18161CLA

Email: 18161027@student.hcmute.edu.vn

STUDENTS

Luu Dinh Phuong Nguyen Nguyen Xuan Thang

After the process of learning, researching, implementing capstone project andcompleting on schedule, we sincerely thank our teachers, family and my friends whowholeheartedly help, give advice and encouragement in difficult times

We would like to thank Mr Do Duy Tan, a lecturer at HCMC University ofTechnology and Education, for standing side by side as well as creating all conditions forour group to be able to perform well on their topic

And we would like to extend our gratitude to the teachers in the Electrical andElectronic Faculty of the University of Technology and Education, Ho Chi Minh Citywho are dedicated to teach us the basic knowledge and give us a solid knowledge base tocomplete this topic

Family is always a source of motivation and inspiration for us to continually striveand complete the topic, thank you for the birth that always encouraged us in the mostfrustrating times

And finally, we would like to thank the brothers, sisters and friends in the class fortheir sincere advice and suggestions so that our topic can be more complete

Once again, we would like to thank everyone for always being by our side andhelping us, creating the motivation for us to perfect this topic

Sincerely!

STUDENTS

Luu Dinh Phuong Nguyen Nguyen Xuan Thang

TABLE OF CONTENT

PROJECT ASSIGNMENT i

PROJECT IMPLEMENT SCHEDULE ii

DISCLAIMER iv

ACKNOWLEDGMENTS v

LIST OF FIGURES ix

LIST OF TABLES x

ABSTRACT xi

CHAPTER 1: OVERVIEW 1

1.1 PROBLEM STATEMENT 1

1.2 PROJECT OBJECTIVES 2

1.3 RESEARCH CONTENT 2

1.4 LIMIT OF THE TOPIC 2

1.5 THESIS OUTLINE 2

CHAPTER 2: BACKGROUND 4

2.1 RASPBERRY PI 4 COMPUTER 4

2.1.1.Introduction 4

2.1.2.Applications for Your Raspberry Pi 4

2.1.3.Exploring the Raspberry Pi 4 Model B Board 5

2.1.4.GPIOs Pinout 6

2.1.5.Raspberry Pi OS 7

2.2 RASPBERRY PI CAMERA MODULE – 5MP 8

2.2.1.General 8

2.2.2.Characteristics 8

2.2.3.Pinout Description 8

2.3 PIR MOTION SENSOR MODULE 9

2.3.1.Overview 9

2.3.2.Characteristics 10

2.3.3.Working Principle 10

2.4 INTRODUCTION TO DEEP LEARNING 11

2.4.1.Deep learning 11

2.4.2.Artificial Neural Network 12

2.5 INTRODUCTION TO IMAGE PROCESSING

2.6 HAAR CASCADE CLASSIFIER 15

2.6.1.Introduction to Haar Cascade 15

2.6.2.The algorithm Haar Cascade Classifier 15

2.7 PYTHON PROGRAMMING LANGUAGE 17

2.7.1.Introduction to the Python programming language 17

2.7.2.Introduction to OpenCV 18

2.7.3.Dlib library 18

2.7.4.Flask Framework 19

CHAPTER 3: BLOCK DIAGRAM AND ALGORITHMS 21

3.1 HARDWARE DESIGN 21

3.1.1.General diagram of the system 21

3.1.2.Block diagram and function of blocks in anti-theft system 21

3.2 DESIGN DETAILS EVERY BLOCK OF THE MONITORING SYSTEM 22

3.2.1.Power supply 22

3.2.2.Central control block 23

3.2.3.Motion detection block 24

3.2.6 Communication block 25

3.3 INSTALL OPERATING SYSTEM FOR RASPBERRY PI 4 26

3.3.1.Equipment to be installed 26

3.3.2.Install the Raspbian OS 26

3.4 INSTALL LIBRARY FOR RASPBERRY PI 4 27

3.4.1.Install the OpenCV library 27

3.4.2.Enabling Camera Function on Raspberry Pi 29

3.5 STEPS TO USE THE SOFTWARE 30

CHAPTER 4: EXPERIMENTAL RESULTS 34

4.1 THE SYSTEM MODELING 34

4.1.1 Hardware results 34

4.1.2 Software results 34

4.1.3 Result of system test execution 37

4.2 REVIEW AND EVALUATION 46

4.2.1 Review 46

4.2.2 Evaluation 46

CHAPTER 5: CONCLUSION 47

5.1 CONCLUSION 47

5.2 DISCUSSION 47

APPENDIX 48

REFFERENCES 52

LIST OF FIGURES

Figure 2.1 Raspberry Pi 4 Model B 4

Figure 2.2 Raspberry Pi 4 Model B hardware design 5

Figure 2.3 Raspberry Pi 4 pinout 6

Figure 2.4 Debian 10 installation at the end 7

Figure 2.5 Connection of Camera Module on Raspberry Pi board 8

Figure 2.6 Camera module pinout 9

Figure 2.7 A typical PIR sensor 10

Figure 2.8 Working principle of PIR 11

Figure 2.9 Performance comparison between deep learning and old algorithms 12

Figure 2.10 Artificial Neural Network 12

Figure 2.11 Basic steps in image processing 14

Figure 2.12 Sample of Haar features 15

Figure 2.13 Haar cascade features 16

Figure 2.14 Illustration for how an integral image works 16

Figure 2.15 Basic Structure of OpenCV 18

Figure 2.16 The 68 landmarks detected by Dlib library 19

Figure 2.17 Overview of the Flask architecture 20

Figure 3.1 General model of anti-theft system 21

Figure 3.2 Block diagram of the anti-theft system 22

Figure 3.3 System power supply block diagram 23

Figure 3.4 Flowchart of the system block 24

Figure 3.5 Flowchart of the sensor block 25

Figure 3.6 Flowchart of the communication block 26

Figure 3.7 Latest version of Raspbian Buster 27

Figure 3.8 Enable the camera module 30

Figure 3.9 Run the web server program file 31

Figure 3.10 Run the Gmail program file 31

Figure 3.11 Web server interface 32

Figure 3.12 Gmail notification received from the Raspberry Pi 32

Figure 3.13 Web server notification received from the Raspberry Pi 33

Figure 4.1 Overview of the system 34

Figure 4.2 VNC Viewer overview 35

Figure 4.3 Motion detection and email notifications 35

Figure 4.4 Program to create web server using Flask 36

Figure 4.5 Email notification received 36

Figure 4.6 Web server home screen 37

Figure 4.7 Web server notification to user 37

Figure 4.8 Camera captures when motion is detected 38

Figure 4.9 Camera captures when motion is detected 38

Figure 4.10 Motion detection system 39

Figure 4.11 The system only detects faces in the case of looking straight ahead 39

Figure 4.12 Object detection system looking to the left 40

Figure 4.13 The system captures the face of the subject looking to the left 40

Figure 4.14 The system captures the subject's face looking to the right 41

Figure 4.15 The face is tilted so much that the system does not recognize the object 41

Figure 4.16 The object is not recognized by the system 42

Figure 4.17 Unrecognizable case due to sitting position 42

Figure 4.18 Two object recognition system 43

Figure 4.19 The system recognizes only one object 43

Figure 4.20 Multiple object recognition system 44

Figure 4.21 One person with mask 44

LIST OF TABLES Table 3.1 List of consumed current and rated voltage of components 22

Table 4.1 Testing conducted with 2 students 46

Currently, home security is always a top priority and helps a lot in implementinghome surveillance This is because the increase in home theft has helped homeowners toimprove awareness and vigilance Anti-theft security systems have been on the marketfor many years but most are just Internet protocol (IP), door sensor alarm systems, orclosed-circuit television (CCTV) systems that are not real-time because the system alertshomeowners after the incident unless they were at home at the time of the incident Theabove methods are much better when applied facial recognition of people who comeclose to the camera within the allowed distance Our team decided to implement the topic

" Anti-theft system design using OpenCV on Raspberry Pi 4"

In this project, our team will build and design a program on Raspberry Pi 4 using a5MP camera for the observation and PIR sensor for motion detection The module usesOpenCV algorithm and Dlib library to solve facial recognition problems which isoperated on the Internet of Things (IoT) system When the infrared sensor module detects

a person moving in front of the system, the camera module will capture person's face andsend the obtained data to the system The person's face images are sent to your Gmailaccount and the information is continuously updated on the web server in real-time Theplace where the incident happened is also warned of illegal entry by a buzzer to helpprevent and warn people This system depends on the PIR sensor to trigger when itcaptures The introduced system above is provided a low-cost solution to monitor andcontrol small spaces such as houses, offices, factories, etc

CHAPTER 1: OVERVIEW

Currently, the situation of property theft is very sophisticated and complicated takingplace in provinces and cities across the country, especially Ho Chi Minh City The thefts notonly cause great damage to the property of families and businesses but also create anxietyand insecurity for many people and affect social order and safety After such incidents, manyfamilies have applied some techniques to increase the installation of burglar alarm systemsfor households Some surveillance systems currently are on the market such as CCTVcameras and digital video recorders (DVRs) which can record burglars' break-in activities,but they cannot be able to identify faces of humans [1] In the past year 2021, the rate ofproperty theft increased gradually and accounted for a high rate with 37.6% in the crimechart [2] At night, detecting and identifying thieves in the dark will be a big obstacle forvideo camera systems that do not have night vision capabilities

Accordingly, smart home technology is receiving much interest to improve andenhance our life with full convenience, comfort, and security In particular, security is alwaysthe top concern and must ensure the safety of family members Currently, there are alsomany research papers by students who also work on smart homes such as author TangNguyen Cong Thien who has studied the topic "Application of image processing, design andstructure of smart anti-theft circuits" using modules Arduino Uno R3, 800A sim module andLogitech HD Webcam C270 Webcam are used to send alarms and SMS alert at the sametime when opening and closing the door and was written by a MATLAB program [3].Author Duong Chi Hung has studied the topic "Designing an anti-theft circuit" using theArduino NANO Board as the central microcontroller, using the Logitech HD Webcam C270camera to recognize faces If the recognition is correct, it will send a unlock signal andtrigger the PWM pulse for the 9G RC Servo motor to open and close the door and use thematrix keypad to enter the password in case the circuit camera has a problem [4] AuthorsTran Quoc Huy and Tran Ngoc Vinh research on the topic "Intelligent fire alarm system"using Arduino Mega 2560 control board as the central control block, ESP8266 Node MCU,MQ-2 gas sensor, sensor PIR HC - SR501 motion, fire detection sensor to monitor homeburglar alarm system anywhere via a web server and phone, controlled by Android App Thesystem can control the door which opens and closes via Android App and SMS notificationvia phone sim when there is a problem [5]

Through the above references, our team has chosen the topic " Anti-theft systemdesign using OpenCV on Raspberry Pi 4" The microcontroller and sensor that we use inthis project is a Raspberry Pi 4 Model B module, a Raspberry 5MP camera module, a PIR

HC - SR501 motion detection sensor The system has a home monitoring function, alertsusers when the sensor detects motion and uses a 5MP camera to collect images of thievesthrough the OpenCV algorithm and Dlib library The system will warn with a buzzer,send facial recognition images via Gmail account, and update image and current time tothe web server in real-time To create a better anti-theft system, our team's topic are usingRaspberry Pi as a fully functional computer that will have a faster processing speed thanthe Arduino microcontroller While Arduino only supports C and C++ languages,Raspberry is very flexible and supports most programming languages The Raspberry Pi

4 is also a lot faster than the Arduino (1.5 GHz compared to 16 MHz), making it faster toprocess images and alert users

1.2 PROJECT OBJECTIVES

In the overview of the topic, our team will design a system with high applicability inthe security system that warning of thefts Module camera can identify the thief's face afterthat the data images will be sent to the system and alert directly to the user It includes:

− Use a PIR motion sensor to detect the thief

− Use a 5MP Raspberry Pi camera to capture images of thief

− Apply the OpenCV algorithm to detect the thief ’s face

− Use the system of led lights to warn

− Send images of thieves to the web server for real-time updates, including the current dates and times

− Send an alert to the user's Gmail

1.3 RESEARCH CONTENT

During the implementation of the project " Anti-theft system design using OpenCV

on Raspberry Pi 4", our team focused on solving and completing the following contents:

− Content 1: Investigate about the theories, methods and algorithms used for object recognition problems Study about image processing, machine learning

− Content 2: Research and process images from cameras

− Content 3: Identity moving objects in the image

− Content 4: Research algorithms to process and send data to web server and Gmail

− Content 5: Complete the system and conduct test runs

− Content 6: Evaluation of performance results

1.4 LIMIT OF THE TOPIC

Due to limited time conditions as well as subjective factors (limited professionalknowledge, financial conditions), the topic content is only implemented in the followingscope:

− Depend on the accuracy of the PIR sensor

− The 5MP Pi camera has low resolution and poor image quality in conditions of strong or too weak light environment

1.5 THESIS OUTLINE

The project report consists of 5 chapters:

− Chapter 1: Overview: our team learn about the reasons for choosing the topic,the objective of the topic, the research content, the limitation and the layout of the topic

− Chapter 2: Background: this chapter presents knowledge of hardware such asRaspberry Pi 4 Model B, 5MP Pi camera module and PIR sensor Theories on AI, Deeplearning, image processing, face recognition methods, Python programming language andthe introduction of the libraries used in the topic

− Chapter 3: System design: this chapter presents and analyzes the general blockdiagram of the system, shows the function of each block in the system and details of theflowchart of each block

2

− Chapter 4: Experimental results: this chapter presents the hardware designprocess for parts of the system Presentation about the results of building the system model,comment and evaluation of the system in all aspects.

− Chapter 5: Conclusions and recommendations: this chapter presents the resultsachieved after completing the thesis, thereby giving directions for future development

CHAPTER 2: BACKGROUND 2.1 RASPBERRY PI 4 COMPUTER

2.1.1 Introduction

The Raspberry Pi is one of the most versatile devices that has emerged in recentyears, with the ability to help customers explore and learn to program in multipleprogramming dialects Raspberry Pi is still considered an overall PC that can provideusers with all-important elements by connect peripherals such as USB ports to plug in akeyboard and mouse, use HDMI ports to plug into your TV or display, we can evenconnect it to wireless network and Bluetooth onboard although the speed is a bit slowerthan today's laptops and computers Raspberry Pi can run many operating systemsincluding Raspbian Linux, Ubuntu Mate, and Windows 10 IoT Core

Figure 2.1 Raspberry Pi 4 Model B [6]

Figure 2.1 shows the Raspberry Pi 4 Model B, the newest member of theRaspberry Pi family of computers Regular computers don't have General Purpose InputPins (GPIOs), which is unique to the Raspberry Pi's board These GPIOs allow us toresearch and develop IoTs projects in our lives When compared to previous models, Pi4improves on the following features: processing speed, multimedia performance, bettermemory and functional connectivity while consuming low power

2.1.2 Applications for Your Raspberry Pi

Raspberry Pi is a credit-card-sized computer but can serve many different uses andtarget diverse users such as research students, professionals or even hackers The mostpopular applications that can be used for general programming purposes, especially thoserelated to IoT or AI Here is one of the simplest applications with great meaning:

− Building a home automation system: The system using a Raspberry Pi module toeasily host several home automation applications It can be used to combine sensors, LEDs,relays and cameras with phones or computers We can easily control our home remotely

− Building a retro gaming console: If we want to play games, we will turn Raspberry Piinto a classic game console This is a program that runs on a custom SD card calledRetroPie which allow us to use the controller to select emulators and games without

touching the keyboard or mouse

4

− Using your Pi as a desktop: Running with Raspberry Pi Desktop is a great way when

we use an old computer that is not capable of running the latest version of Windows ormacOS With 8GB model option the Raspberry Pi 4 will be great for basic everyday taskslike using a web browser, watching video streams, and editing documents, as well as setting

up a personal computer quickly

− Making your own CCTV system: The surveillance system built includes: Raspberry

Pi module acting as a HUB with 1TB SATA and a camera that can be monitored via theHUB All devices are connected to the local network and we can access all cameras usingyour computer's browser or another device

2.1.3 Exploring the Raspberry Pi 4 Model B Board

The user can see the peripheral communication ports arranged on the Raspberry Pi

4 Model B shown in Figure 2.2

Figure 2.2 Raspberry Pi 4 Model B hardware design [7]

Figure 2.2 shows the ports arranged on the Raspberry Pi 4 Model B and theirfunctions are announced by the Raspberry Pi Foundation in June 2019 with significantupgrades The new Raspberry Pi 4 features a high-speed of 1.5GHz Broadcom both CPU andGPU which can be configured with 1GB, 2GB, and 4GB memory configurations compared

to the 1GB memory of the previous Raspberry Pi 3 [8] Raspberry Pi 4 uses Broadcom'sBCM2711 processor by a completely new 28nm process with more powerful processorcores, which can perform tasks smoothly The pipeline depth of the Cortex A72microarchitecture is 15 instructions, 7 instructions more than previous models As a result,the Cortex-A72 processor will be much faster and consume more power than the Pi 3 Withthese configurations, the Pi 4 can diversify usage needs from simple math problems to AIand other computer applications In addition, Raspberry Pi 4 is upgraded to Bluetooth 5.0compared to 4.2 in previous Pi 3 versions, Micro-USB Type-B is changed to Type-C toallow compatibility with the connection environment of current Macbook laptops andsupports 802.11ac Wi-Fi An Ethernet port is fitted on the top right side of the board for fastnetwork speed without bottleneck Two USB 3.0 ports are fitted in the center of the deviceproviding a high-speed connection to external devices This new USB port provides amassive bandwidth boost and has a big impact on the performance of external storagedevices with SSDs connected via USB adapters with the average read and write throughput

measured in MB/s The Pi 4's HDMI connection is changed to a micro port with the ability toconnect two monitors Furthermore, there are two micro-HDMI ports that enable supportdual monitor up to 4K resolutions, driving two 4K displays at resolution up to 4Kp30 or asingle display at 4Kp60 SD memory card is used to storage information, all information will

be put into this memory card, audio jack for audio hardware communication

2.1.4 GPIOs Pinout

The GPIO pinout of Raspberry Pi 4 is depicted as shown in Figure 2.3

Figure 2.3 Raspberry Pi 4 pinout [9]

Figure 2.3 shows the pinout and key functions available on the Raspberry Pi 4 Model

B The 40-pin GPIO header is included with the Raspberry Pi 4, with 26 GPIO (GeneralPurpose Input/Output) pins Because GPIO pins can only have two states, off or on, and have

no specific function, the software will be used to alternate them The Raspberry Pi 4 deviceuses the 5V voltage level to output the 5V power supply via the Type-C interface, whichincludes pins 2 and 4 External components, such as pins 1 and 17, receive 3.3V power fromthe 3.3V voltage level Ground pins are used to connect electrical circuits together Theground pins (0V) safeguard the Pi 4 module from overheating and are crucial to the circuit(ground pins are numbered 6, 9, 14, 20, 25, 30, 34 and 39) I2C and EEPROM protocols, aswell as digital pins 27 and 28, are communication via dedicated pins PWM is a spectrumtechnique used to convert digital signals to analog signals Only hardware PWM is supported

by a few pins, including GPIO pins 12, 13, 18, and 19 All pins are capable of runningsoftware PWM The Serial Peripheral Interface (SPI) is a set of pins that allow devices likesensors, LEDs, and actuators to communicate with one another The Raspberry Pi uses theMaster Out Slave Pin (MOSI) to deliver data to a device, while the device uses the Master InSlave Out (MISO) pin to connect with the Raspberry Pi GND, SCLK (GPIO 11), MOSI(GPIO 10), MISO (GPIO 9) and CE (GPIO 7) are the five GPIO pins required for SPcommunication The choices of MISO (GPIO 19), MOSI (GPIO 20), SCLK (GPIO 21), CE0(GPIO 18), CE1 (GPIO 17), and GPIO 16 as CE2 will be modified in the case of the SPI1pin

2.1.5 Raspberry Pi OS

Raspberry Pi OS, also known as Raspbian, is one of the Officially DistributedOperating Systems (ODOS) that can be downloaded and used for free Raspbian is well-suited to low-performance ARM CPUs of the Raspberry Pi version This system is based

on Debian Linux and is designed to interact with Raspberry Pi machines efficiently.Debian is a lightweight operating system that is suitable for the Raspberry Pi Raspbiancomes with browser, python programming, and graphical user interface (GUI) desktop.Raspberry Pi Operating System has recently published two Debian 10 and Debian 11versions, are codenamed "Buster" and "Bullseye," respectively In there:

− A released Raspberry Pi OS “Buster” is said to inherit the previous version Raspbian

“Stretch” for compatibility with Raspberry Pi models Buster is based on Debian Linux

10 Like previous Raspberry OS operating systems, Buster also provides users with the PIXEL desktop environment (including LXDE)

Figure 2.4 Debian 10 installation at the end [10]

Based on Figure 2.4 above, the Debian 10 desktop virtual machine creation processhas been successfully installed to the virtual hard disk on proxmox While there is not

a big difference between Debian “Stretch” and Debian “Buster”, the big difference isthe security changes that makes Buster becomes harder to hack The “Eject” icon isused to remove a USB device appears only when a USB device needs to be ejected.Other difference is mostly minor changes that most users may not notice

− Raspberry Pi OS 'Bullseye' is the latest Raspberry Pi operating system which based onDebian Linux predecessor “Bullseye” Debian 10 used Linux Kernel 4.19 whileDebian 11 is upgraded to Linux Kernel 5.10 LTS to improve new bug fixes, supportsnew hardware and improves performance KMS video driver (kernel mode setting),formerly an experimental option, is now a standard video driver with this release Thischange makes it possible for applications that use the standard Linux Display API torun on the Raspberry Pi without modification

2.2 RASPBERRY PI CAMERA MODULE – 5MP

2.2.1 General

The Raspberry 5MP camera module is a low-cost, lightweight camera module isdesigned to work with the Raspberry Pi in range of computers The module is attached tothe Raspberry Pi's CSI connection via a ribbon wire and communicates with it via theMIPI camera serial interface protocol CSI bus capable of extremely high data rates and itexclusively carries pixel data to the BCM2835 processor It can provide clear 5MPimages, capture still images at 2592 x 1944 pixels and support video recording in HD1080p at 30fps, 720p at 60fps With the latest version of Raspbian, the Raspberry Pi'sDebian-based computer operating system, the camera module is always supported

Figure 2.5 Connection of Camera Module on Raspberry Pi board

[11] 2.2.2 Characteristics

The Raspberry Pi Camera 5MP module is compatible with the following Raspberry Picamera models: Pi 4, Pi 3 B+, Pi 3, Pi 2, Pi B+, Pi A, and Pi Zero/Zero W The cameramodule is 25mm x 24mm x 9mm in size and weighs around 3g A 5MP OmniVision 5647sensor is used in the module [11] When shooting still photographs, the resolution is 2592 x

1944 pixels, and video can be recorded in three different modes: 1080p @ 30fps, 720p

@ 60fps, and 640x480p 60/90 The MIPI camera serial interface has 15 pins and is connected to the Raspberry Pi directly

2.2.3 Pinout Description

User can see the pinout of Pi camera module shown in Figure 2.6

Figure 2.6 Camera module pinout [12]

Figure 2.6 shows the pinout and function of each pin on the camera module Theground pins are used to close the electrical circuits, protect the Pi camera module fromburning and play an important role in the electrical circuit (the ground pins are numbered

1, 4, 7, 10) A voltage level of 3.3V is used to provide power for the device to operate,including digital pin 15 Digital pin 2 (CAM1_DN0) and pin 3 (CAM1_DP0) are MIPIPositive and Negative MIPI data, respectively counts for data 0, pin 5 (CAM1_DN1) and

6 (CAM1_DP1) represent level 1 Pins 8 (CAM1_CN) and 9 (CAM1_CP) provide clockpulses for the MIPI data lanes Pin 11 (CAM_GPIO) is an optionally used pin and pin 12(CAM_CLK) is an optional clock pin Pins 13 (SCL0) and pin 14 (SDA0) are used forI2C communication

2.3.1 Overview

Passive InfraRed sensor (PIR sensor) is a passive sensor that allow you to sensemotion, which almost always used to detect the movement of a person entering or leavingthe sensor's range PIR sensors are always a good choice in projects when they have a lot

of advantages such as small, inexpensive, quite sturdy, wide lens range, easy to use andlow power consumption [13] However, the PIR sensor will not tell us how many peopleare around the area, the lens used here is usually fixed at a certain scanning angle anddistance using the infrared excitation source Infrared (IR) rays are heat rays that emitted

by hot objects In our body, the body temperature is usually 37 ° C and always emits heatrays The manufacturer uses a galvanic cell to convert heat rays into electrical signals anddetect hot moving objects The PIR sensor consists of 3 pins as shown in the figure 2.7

Figure 2.7 A typical PIR sensor [13]

Figure 2.7 introduces the type of PIR sensor HC-SR501 contains 3 pins, which areVCC, Input pin and ground Pin 1 of the sensor is connected to the ground Pin 3 is the inputvoltage of +5V for typical applications and ranges from 4.5V-12V Pin2 is provided 3.5Vlogic high output if a moving object is detected and low when no motion is detected

2.3.2 Characteristics

The PIR sensor module can be powered from voltage 4.5V to 20V to createflexibility for choosing the right operating source When the PIR sensor detects infraredmotion, the output will be HIGH-level 3.3V, if the module does not detect motion, it will

be LOW 0 V after fixed period The delay time is adjustable from 0.3 seconds to 200seconds and can be customized up to 10 minutes The operating temperature of the PIRsensor is impressive from -15 to +70 degrees Celsius Sensitivity range: The angle of thedetection area is about 120 degrees cone and the range varies up to 7 meters The sensormodule has compactly designed with dimensions of 32 mm x 24 mm, the diameter size ofthe Fresnel lens is 23 mm, which increases the detection range and it looks opaque

2.3.3 Working Principle

Passive infrared sensors use a pair of pyroelectric sensors to detect heat energy inthe surrounding environment The sensor does not use spontaneous heating but only usesinfrared radiation from human and animal bodies to alarm Any individual with a stabletemperature will be radiated infrared to the outside world (average human bodytemperature is always maintained at 36°C - 37°C) We can understand the operatingprinciple of the PIR sensor as shown in Figure 2.8

Figure 2.8 Working principle of PIR

Figure 2.8 shows that the PIR sensor is made up of two slots, each slot is made of

a special material that is sensitive to IR and linked in such a way that they cancel eachother out When the sensor does not detect motion, both slots will detect an equal amount

of IR, the amount of ambient emanating from walls or the outdoors To increase thesensitivity of the slot, we use a Fresnel lens with a large detection angle, which filters outinfrared rays and prevents ultraviolet rays When the object moves, the subject's bodytemperature will always emit heat rays, strongly condense with the Fresnel lens and thenfocus on the infrared sensor target, these 2 slots will give 2 signals and these signals will

be amplified for a high amplitude and put into a voltage comparator circuit to affect acontrol or alarm device

2.4 INTRODUCTION TO DEEP LEARNING

2.4.1 Deep learning

Deep learning is a sub-branch of Machine Learning, used with the idea that based

on the human neural network (Neural Network) Deep Learning has helped computersperform tasks that were thought impossible many years ago, such as classifying differentobjects in an image, creating captions for images, recognizing human voices andhandwriting, etc even in making movies and music [14] Deep learning is also animportant application of the technology behind self-driving cars, allowing them torecognize traffic signs or distinguish pedestrians from traffic lights Deep learning isattracting a lot of attention recently because of its positive results that were impossiblewith other methods before The performance of deep learning compared to previous oldlearning methods is shown in Figure 2.9

Figure 2.9 Performance comparison between deep learning and old algorithms [15]

Figure 2.9 shows us that performance is the big difference between the twoalgorithms When the data is small, deep learning does not work well Therefore, whenthere is a large amount of data, deep learning performs perfectly

2.4.2 Artificial Neural Network

One of those Deep Learning methods is Artificial Neural Network - the layer fully-connected neural layers Artificial Neural Network usually refers to thenumber of hidden layers in a neural network Traditional neural networks contain only 2-

multi-3 hidden layers, while deep networks can have up to hundreds of layers Artificial NeuralNetworks can be viewed as cascade models of cell types inspired by biologicalobservations They are a combination of layers of perceptrons or multilayer perceptrons

as shown in Figure 2.10

Figure 2.10 Artificial Neural Network [16]

Based on Figure 2.10, the general architecture of an Artificial Neural Network includes 3 components: Input layer, Hidden layer and Output layer.

− Input layer: this layer is located on the leftmost side of the network to represent input data Each neuron corresponds to an attribute or feature of the input data

− Hidden layer: this layer is located in the center between the input and output layers,where the data is processed before outputting It represents the logical reason processing of thenetwork This layer can have one or more Hidden Layers that are intermediate layers betweenthe input and output layer and process the data by applying complex non-linear functions tothem These layers are the key component that enables a neural network to learn complex tasksand achieve excellent performance

− Output layer: this layer is on the right and it represents the output of the neuralnetwork After passing through the last hidden layer, the data will be transformed by afunction called the activation function and given the final result

2.4.2.1 Application

Applications of deep learning are everywhere such as virtual assistants that help tounderstand the user's speech and the language people use when they interact with themachine, translation can help tourists, business people and people working in government.For the transportation sector, drones and self-driving cars can move, stop, avoid obstacles onthe road or other vehicles with the ability to handle and act like humans during the process.receive information Many companies can take good care of customers with optimalresponses to customer questions thank to Chatbots deep learning applications

2.5 INTRODUCTION TO IMAGE PROCESSING

Image processing is one of the most important parts of the field of computervision, the premise for many studies in this field is rapidly developing Image processing

is the process of changing from an original image to a new image with the desiredcharacteristics and requirements of the user Image processing includes the processes ofanalysis, classification of objects, enhancement, segmentation and separation, labeling ofregions or the compilation of image information

Digital image processing includes transformation techniques and methods,intending to transmit or encode natural images The purposes of image processinginclude: improving and enhancing image quality, automatically recognizing images andevaluating image content Image processing is widely applied to practical problems,bringing high efficiency in many areas such as:

− Military: target detection and tracking, vehicle navigation, wide-area surveillance and automatic target identification

− Human-machine interface: image recognition, sound processing, graphics

− Safety and security: human face recognition, fingerprint recognition, eye pattern recognition

− Entertainment: video games

− Medical: Computed Tomography (CT) or Magnetic Resonance Imaging (MRI) fordiagnosis of medical conditions or interventions such as surgical planning or researchpurposes

− Industrial production: identification and classification of products and fruits

Previously, the images which obtained from the camera were analog Currently, along with the continuous development of technology, color images or black and white

images taken from the camera will be directly converted to digital images to facilitate thenext processing steps Fundamental steps in image processing are shown in Figure 2.11.

Figure 2.11 Basic steps in image processing

Figure 2.11 introduces the map in image processing including 5 steps [17]:

− Image Acquisition is an important first stage in the image processing process Inputimages can be acquired through devices: cameras and scanners, which can be color or blackand white images Then, these signals will be digitized with important parameters such asimage resolution, color gamut quality, and image acquisition speed of the devices Thequality of a captured image will be depended on the receiving device and the environment(light, weather, landscape)

− Image Processing: at this stage, the image may have low contrast noise, so it needs to

be put into a preprocessor to improve quality, improve contrast, and remove noise for thepurpose of making the image more beautiful, sharp and quality This pre-processing stage isperformed by filters with different noise filtering functions but will have the same purpose

of improving contrast, clarity and matching user requirements

− Segmentation is the process of separating the input image into component regions torepresent analysis and image recognition This is the most complicated and difficult step inimage processing and can make errors and loss of image accuracy The result of imagerecognition depends a lot on this step

− Image representation: after the segmentation process containing pixels of the imagearea will come to represent the image The use of converting these data into an appropriateform is necessary for further processing by the computer The selection of features torepresent the image is called Feature Extraction which associated with the extraction ofimage features in the form of quantitative information or like a to distinguish this featureclass from other objects in the received image

− Image Recognition and Interpretation: Image recognition is defined as the process ofidentifying an image to obtain an image by comparing it with a previously learned (orsaved) standard sample Interpolation is defined as making sense of judgment based onidentification There are different ways to classify images The theory of recognition willinclude mathematical models of images and can be classified into two basic imagerecognitions: parameter recognition and structure recognition Identification objects arecommonly applied in science and technology today: character recognition (print,handwriting, electronic signature), text recognition (Text), fingerprint recognition,identification barcode and facial recognition

2.6 HAAR CASCADE CLASSIFIER

2.6.1 Introduction to Haar Cascade

Haar Cascade, also known as the Viola-Jones method, is a machine learning objectdetection algorithm created with features for object detection and recognition proposed

by Paul Viola and Michael Jones in 2001 It includes four essential features to identify anobject in an image With Haar cascading, objects can be found in an image regardless ofthe object's position or size of the image Initially, the application was used to recognizefaces and human features: eyes, nose and mouth However, many Haar features aredeveloped to apply full body recognition, upper body, lower body, smile, and manyobjects including vehicles, and industrial products Cascade window extractions are used

by Haar features to compute the features in each window and determine if it can be anobject using rectangular features as shown in Figure 2.12

Figure 2.12 Sample of Haar features [18]

Figure 2.12 shows the types of rectangles which applied for Haar Featureextraction Firstly, the feature using two rectangles is the difference in the total number ofpixels in two rectangular regions, which is used mainly for edge detection (a, b) Thethree-rectangle feature performs the sum of the two outermost rectangles minus the sum

of one middle rectangle to detect lines (c, d) The feature uses four calculated rectangles

to represent the difference between diagonal pairs of rectangles (e)

2.6.2 The algorithm Haar Cascade Classifier

Using the Haar classifier, the faces of objects are detected from real-time images,recordings, or videos There are four stages of the Haar Cascade classifier presented:

− Haar features selection: the first stage is collection of Haar features The commonfeatures that appear on a human face: the bridge of the nose is lighter than the eyes,the eye area is darker than the upper cheek, and the mouth is known as Haar Features.Each Haar feature is a combination of two black and white rectangles which dividedinto 3 features shown in Figure 2.13

Figure 2.13 Haar cascade features

Figure 2.13 shows the features of the rectangle that are sorted in order on the image.The total number of pixels in the white part is subtracted from the total number ofpixels in the black part Where edge features are filters used to detect edges in animage Line features will capture three straight lines in the image and Four-rectanglefeatures have four squares to detect inclined lines in the image

− Creating an integral image: Calculating the features of a rectangle by using the

built-in multipliers can be very time-consumbuilt-ing Therefore, the author has proposed anintermediate representation: the integral image which will allow to sum of any rectanglesimply using four values which gives a significant speedup using the integral image throughfigure 2.14

Figure 2.14 Illustration for how an integral image works [19]

Figure 2.14 shows the total number of pixels contained in the rectangle calculatedaccording to the four array references A, B, C and D At position 1, the value of theintegral will be the sum of the pixels contained in rectangle A Next, the integral value

at position 2 is the sum of the two values A + B Position 3 will be the sum of thepixels contained in the two rectangles A + C, and the integral value at position 4 is thesum of 4 values A + B + C + D The sum of pixel values present in area D can besimply calculated as 4 + 1 - (2 + 3)

− AdaBoost training (Adaptive Boost): AdaBoost is used to select a small set of featuresand train the classifier The algorithm helps to deploy and build a strong classifier bycombining the good features of many weak classifiers The final classifier is a weightedcombination of several weak classifiers This matches a weak sequence of learners ondifferent weighted training data and increases in accuracy.

− Cascading classifiers: Although the above procedures are quite efficient, Viola andJones improve with a reduction in computation time and an increase in detection rate usingCascading Classifiers The main purpose is correctly to identify the face and minimize thefalse ratio by rejecting the undefined and non-faced sub-windows while defining theregions The result will be a highly accurate classifier from the average prediction of allweak learners

2.7.1 Introduction to the Python programming language

Python is an easy-to-learn, high-level object-oriented programming language thathas a simple effective approach to object-oriented programming and was created byGuido van Rossum Python's syntax and statements are huge strengths: its clearpresentation, ease understanding, and flexible typing make it quickly an ideal languagefor scripting and application development used in many fields and appeared on almostevery platform Features of Python include:

− Simple Programming Language: Python is a high-level programming language with asimple, clear syntax Python programs are easier to read and write than other programminglanguages: C++, Java, C# Python made programming fun, making programmers morefocused on solutions, not syntax

− Free and open-source: Python is an open source and there is no restriction on its use.Users can not only use but also change the source code of Python as a community of users

is constantly improving them

− Portability: Python programs can be moved from one operating system to another andexecuted without any changes It runs seamlessly on almost all platforms like Windows,Mac OS, Linux and UNIX

− Extensible and embeddable: when applications need high performance, users cancombine parts of the code C, C++, and other programming languages into Python code tomake the program more efficiency at the level that other programming languages cannotprovide

− Interpreted, high-level language: unlike C/C++ languages, Python programmers donot need to worry about difficult tasks like memory management, cleaning up meaninglessdata, etc When running the Python code, it will automatically convert the code into alanguage that the computer can understand and save the user from worrying about any low-level operations

− Large standard library: Python has a large number of standard libraries that make theuser's programming job a lot easier, simply because you do not have to write all the codeyourself

− Object-Oriented: Everything in Python is object-oriented Object-OrientedProgramming (OOP) helps to solve complex problems visually With OOP, users can breakdown complex problems into smaller groups by creating objects

2.7.2 Introduction to OpenCV

Open Source Computer Vision Library (also known as OpenCV) is an source library for the research or development of computer vision and machine learning.OpenCV has been existed for 1 period and was taken over by the nonprofit OpenCVFoundation in 2012 OpenCV is written natively from C and C++ programming, it can beused in other languages: Java, Python, Ruby and Matlab They are easily compatible withWindows, Linux and Mac OS operating systems OpenCV automatically uses theappropriate IPP library at runtime if the library is installed One of the goals of OpenCV

open-is became the basopen-is for computer vopen-ision to help users build fairly complex applicationsquickly It is mainly used for Computer Vision Applications and implemented additionalfeatures with GPU acceleration for real-time operations, this includes algorithms used formotion tracking, facial recognition, imaging, robotics, autonomous vehicles, structures3D self-motion, object detection and more OpenCV library has outstanding featuressuch as 2D and 3D support toolset, face recognition, gesture, object and behaviorrecognition, human-computer interaction and robot control

The basic structure of OpenCV is shown in Figure 2.15:

Figure 2.15 Basic Structure of OpenCV [20].

Figure 2.15 shows the basic structure diagram of OpenCV including 5 parts, which

4 parts are presented First, CV (Computer vision) is the component that includes imageprocessing, image structure analysis, high-level computer vision algorithms, etc MLL(machine learning library) is a machine learning library, which includes many functionsfor clustering, classification, and data analysis HighGui provides I/O commands and I/Ohas functions of accessing and storing image and video files CXCore contains datastructures, matrix algebra, basic data content, error handling, basic drawing and math, etc.The figure above will not include CvAux, which contains experimental algorithms ordiscontinued functions