Anti theft system design using opencv on raspberry pi 4

MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING Ho Chi Minh City, 2022 CAPSTONE PROJECT ELECTRONIC AND COMM

MINISTRY OF EDUCATION AND TRAINING

HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION

FACULTY FOR HIGH QUALITY TRAINING

Ho Chi Minh City, 2022

CAPSTONE PROJECT ELECTRONIC AND COMMUNICATION

ANTI-THEFT SYSTEM DESIGN USING OPENCV ON

RASPBERRY PI 4

LECTURER: Ph.D DO DUY TAN STUDENT: NGUYEN XUAN THANG LUU DINH PHUONG NGUYEN

SKL 0 0 9 3 6 0

HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION

FACULTY FOR HIGH QUALITY TRAINING

- -

GRADUATION PROJECT

Ho Chi Minh City, July 2022

ANTI-THEFT SYSTEM DESIGN USING

OPENCV ON RASPBERRY PI 4

Student ID: 18161027 Major: ELECTRONIC AND COMMUNICATION ENGINEERING

Advisor: Ph.D DO DUY TAN

NGUYEN XUAN THANG Student ID: 18161038 LUU DINH PHUONG NGUYEN

THE SOCIALIST REPUBLIC OF VIETNAM

Independence – Freedom– Happiness

***

Ho Chi Minh City, July 29 th , 2021

ADVISOR’S EVALUATION SHEET

Major: Electronics and Communication Engineering

Project title: Anti-theft system design using OpenCV on Raspberry Pi 4

Advisor: Ph.D Do Duy Tan

EVALUATION

1 Content of the project:

The content and workload of the project meets the initial requirements of the project

2 Strengths:

The topic of the project is practical and applicable Extensive experimental results are conducted to evaluate the performance of the proposed design

3 Weaknesses:

− There are many grammatical mistakes and typos in the report

− The performance of the proposed design might be unstable in environments that light intensity is low

4 Approval for oral defense? (Approved or denied)

Approved

5 Overall evaluation: (Excellent, Good, Fair, Poor)

Good

6 Mark: 8.0 (in words: Eight )

Ho Chi Minh City, July 29 th , 2022

ADVISOR

(Sign with full name)

THE SOCIALIST REPUBLIC OF VIETNAM

Independence – Freedom– Happiness

-

Ho Chi Minh City, July 29 th , 2022

PRE-DEFENSE EVALUATION SHEET

Major: Electronics and Communication Engineering

Project title: Anti-theft system design using OpenCV on Raspberry Pi 4

THE SOCIALIST REPUBLIC OF VIETNAM

Independence – Freedom– Happiness

-

Ho Chi Minh City, July 29 th , 2022

EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER

Major: Electronics and Communication Engineering

Project title: Anti-theft system design using OpenCV on Raspberry Pi 4

Advisor: Ph.D Do Duy Tan

Name of Defense Committee Member:

5 Mark: ………… (in works:…….………….….………….………)

Ho Chi Minh City, day… month… year

COMMITTEE MEMBER

(Sign with full name)

HCMC University of Technology and

Education Faculty for High Quality training

Socialist Republic of Vietnam Independence - Liberty – Happiness

Ho Chi Minh City, July 29 th , 2021

PROJECT ASSIGNMENT

Name of Members: Nguyen Xuan Thang

Luu Dinh Phuong Nguyen

Student ID: 18161038 Student ID: 18161027 Training system: Regular university

Major: Electronics and communication engineering

Class: 18161CLA

ON RASPBERRY PI 4

1 The original figures

− Recognize the urgency and importance of the project

− Conducting research, studying materials, current models related to the topic for comparison and improvement

2 Content of implementation

Task 1: Brainstorm and research related topics

Task 2: Research and select components

Task 3: Draw algorithm flowchart

Task 3: Design and build a complete system model Run tests and calibrate the system Task 4: Write a report

CHAIR OF THE PROGRAM

(Sign with full name)

ADVISOR

(Sign with full name)

HCMC University of Technology and

Education Faculty for High Quality training

Socialist Republic of Vietnam Independence - Liberty – Happiness

Ho Chi Minh City, July 29 th , 2022

PROJECT IMPLEMENT SCHEDULE

Name of Members: Nguyen Xuan Thang

Luu Dinh Phuong Nguyen

Class: 18161CLA

Student ID: 18161038 Student ID: 18161027

Name of project: Anti-theft system design using OpenCV on Raspberry Pi 4

1st

(14/3-20/3)

- Meet the Instructor to disseminate regulations:

choose the topic and working time

2st

(21/3-27/3)

- Instructor conducts a review of the topic

- Write the plan and detailed outlines for the topic

- Write a summary of the requirements of the selected topic

3nd

(28/3-4/4)

4rd

(5/4-11/4)

- Learn some topics similar to the selected topic

- Learn the components used in the selected topic

5th

(12/4-18/4)

6th

(19/4-25/4)

- Design block diagram

- Design the principal diagram and explain the function of each block

- Implement, test and fix the project

- Write the report

14th

(14/6-20/6)

- Complete the report and send it to Instructor to review and comment for the last time before printing the report

Student 1: NGUYEN XUAN THANG

Student 1’s ID: 18161038 Class: 18161CLA

Email: 18161038@student.hcmute.edu.vn

Student 2: LUU DINH PHUONG NGUYEN

Student 2’s ID: 18161027 Class: 18161CLA

Email: 18161027@student.hcmute.edu.vn

STUDENTS

Luu Dinh Phuong Nguyen Nguyen Xuan Thang

ACKNOWLEDGMENTS

After the process of learning, researching, implementing capstone project and completing on schedule, we sincerely thank our teachers, family and my friends who wholeheartedly help, give advice and encouragement in difficult times

We would like to thank Mr Do Duy Tan, a lecturer at HCMC University of Technology and Education, for standing side by side as well as creating all conditions for our group to be able to perform well on their topic

And we would like to extend our gratitude to the teachers in the Electrical and Electronic Faculty of the University of Technology and Education, Ho Chi Minh City who are dedicated to teach us the basic knowledge and give us a solid knowledge base to complete this topic

Family is always a source of motivation and inspiration for us to continually strive and complete the topic, thank you for the birth that always encouraged us in the most frustrating times

And finally, we would like to thank the brothers, sisters and friends in the class for their sincere advice and suggestions so that our topic can be more complete

Once again, we would like to thank everyone for always being by our side and helping

us, creating the motivation for us to perfect this topic

Sincerely!

STUDENTS

Luu Dinh Phuong Nguyen Nguyen Xuan Thang

TABLE OF CONTENT

PROJECT ASSIGNMENT i

PROJECT IMPLEMENT SCHEDULE ii

DISCLAIMER iv

ACKNOWLEDGMENTS v

LIST OF FIGURES ix

LIST OF TABLES x

ABSTRACT xi

CHAPTER 1: OVERVIEW 1

1.1 PROBLEM STATEMENT 1

1.2 PROJECT OBJECTIVES 2

1.3 RESEARCH CONTENT 2

1.4 LIMIT OF THE TOPIC 2

1.5 THESIS OUTLINE 2

CHAPTER 2: BACKGROUND 4

2.1 RASPBERRY PI 4 COMPUTER 4

2.1.1.Introduction 4

2.1.2.Applications for Your Raspberry Pi 4

2.1.3.Exploring the Raspberry Pi 4 Model B Board 5

2.1.4.GPIOs Pinout 6

2.1.5.Raspberry Pi OS 7

2.2 RASPBERRY PI CAMERA MODULE – 5MP 8

2.2.1.General 8

2.2.2.Characteristics 8

2.2.3.Pinout Description 8

2.3 PIR MOTION SENSOR MODULE 9

2.3.1.Overview 9

2.3.2.Characteristics 10

2.3.3.Working Principle 10

2.4 INTRODUCTION TO DEEP LEARNING 11

2.4.1.Deep learning 11

2.4.2.Artificial Neural Network 12

2.5 INTRODUCTION TO IMAGE PROCESSING 13

2.6 HAAR CASCADE CLASSIFIER 15

2.6.1.Introduction to Haar Cascade 15

2.6.2.The algorithm Haar Cascade Classifier 15

2.7 PYTHON PROGRAMMING LANGUAGE 17

2.7.1.Introduction to the Python programming language 17

2.7.2.Introduction to OpenCV 18

2.7.3.Dlib library 18

2.7.4.Flask Framework 19

CHAPTER 3: BLOCK DIAGRAM AND ALGORITHMS 21

3.1 HARDWARE DESIGN 21

3.1.1.General diagram of the system 21

3.1.2.Block diagram and function of blocks in anti-theft system 21

3.2 DESIGN DETAILS EVERY BLOCK OF THE MONITORING SYSTEM 22

3.2.1.Power supply 22

3.2.2.Central control block 23

3.2.3.Motion detection block 24

3.2.6 Communication block 25

3.3 INSTALL OPERATING SYSTEM FOR RASPBERRY PI 4 26

3.3.1.Equipment to be installed 26

3.3.2.Install the Raspbian OS 26

3.4 INSTALL LIBRARY FOR RASPBERRY PI 4 27

3.4.1.Install the OpenCV library 27

3.4.2.Enabling Camera Function on Raspberry Pi 29

3.5 STEPS TO USE THE SOFTWARE 30

CHAPTER 4: EXPERIMENTAL RESULTS 34

4.1 THE SYSTEM MODELING 34

4.1.1 Hardware results 34

4.1.2 Software results 34

4.1.3 Result of system test execution 37

4.2 REVIEW AND EVALUATION 46

4.2.1 Review 46

4.2.2 Evaluation 46

CHAPTER 5: CONCLUSION 47

5.1 CONCLUSION 47

5.2 DISCUSSION 47 APPENDIX 48 REFFERENCES 52

LIST OF FIGURES

Figure 2.1 Raspberry Pi 4 Model B 4

Figure 2.2 Raspberry Pi 4 Model B hardware design 5

Figure 2.3 Raspberry Pi 4 pinout 6

Figure 2.4 Debian 10 installation at the end 7

Figure 2.5 Connection of Camera Module on Raspberry Pi board 8

Figure 2.6 Camera module pinout 9

Figure 2.7 A typical PIR sensor 10

Figure 2.8 Working principle of PIR 11

Figure 2.9 Performance comparison between deep learning and old algorithms 12

Figure 2.10 Artificial Neural Network 12

Figure 2.11 Basic steps in image processing 14

Figure 2.12 Sample of Haar features 15

Figure 2.13 Haar cascade features 16

Figure 2.14 Illustration for how an integral image works 16

Figure 2.15 Basic Structure of OpenCV 18

Figure 2.16 The 68 landmarks detected by Dlib library 19

Figure 2.17 Overview of the Flask architecture 20

Figure 3.1 General model of anti-theft system 21

Figure 3.2 Block diagram of the anti-theft system 22

Figure 3.3 System power supply block diagram 23

Figure 3.4 Flowchart of the system block 24

Figure 3.5 Flowchart of the sensor block 25

Figure 3.6 Flowchart of the communication block 26

Figure 3.7 Latest version of Raspbian Buster 27

Figure 3.8 Enable the camera module 30

Figure 3.9 Run the web server program file 31

Figure 3.10 Run the Gmail program file 31

Figure 3.11 Web server interface 32

Figure 3.12 Gmail notification received from the Raspberry Pi 32

Figure 3.13 Web server notification received from the Raspberry Pi 33

Figure 4.1 Overview of the system 34

Figure 4.2 VNC Viewer overview 35

Figure 4.3 Motion detection and email notifications 35

Figure 4.4 Program to create web server using Flask 36

Figure 4.5 Email notification received 36

Figure 4.6 Web server home screen 37

Figure 4.7 Web server notification to user 37

Figure 4.8 Camera captures when motion is detected 38

Figure 4.9 Camera captures when motion is detected 38

Figure 4.10 Motion detection system 39

Figure 4.11 The system only detects faces in the case of looking straight ahead 39

Figure 4.12 Object detection system looking to the left 40

Figure 4.13 The system captures the face of the subject looking to the left 40

Figure 4.14 The system captures the subject's face looking to the right 41

Figure 4.15 The face is tilted so much that the system does not recognize the object 41

Figure 4.16 The object is not recognized by the system 42

Figure 4.17 Unrecognizable case due to sitting position 42

Figure 4.18 Two object recognition system 43

Figure 4.19 The system recognizes only one object 43

Figure 4.20 Multiple object recognition system 44

Figure 4.21 One person with mask 44

LIST OF TABLES Table 3.1 List of consumed current and rated voltage of components 22

Table 4.1 Testing conducted with 2 students 46

ABSTRACT

Currently, home security is always a top priority and helps a lot in implementing home surveillance This is because the increase in home theft has helped homeowners to improve awareness and vigilance Anti-theft security systems have been on the market for many years but most are just Internet protocol (IP), door sensor alarm systems, or closed-circuit television (CCTV) systems that are not real-time because the system alerts homeowners after the incident unless they were at home at the time of the incident The above methods are much better when applied facial recognition of people who come close

to the camera within the allowed distance Our team decided to implement the topic "theft system design using OpenCV on Raspberry Pi 4"

Anti-In this project, our team will build and design a program on Raspberry Pi 4 using a 5MP camera for the observation and PIR sensor for motion detection The module uses OpenCV algorithm and Dlib library to solve facial recognition problems which is operated

on the Internet of Things (IoT) system When the infrared sensor module detects a person moving in front of the system, the camera module will capture person's face and send the obtained data to the system.The person's face images are sent to your Gmail account and the information is continuously updated on the web server in real-time The place where the incident happened is also warned of illegal entry by a buzzer to help prevent and warn people.This system depends on the PIR sensor to trigger when it captures The introduced system above is provided a low-cost solution to monitor and control small spaces such as houses, offices, factories, etc

CHAPTER 1: OVERVIEW

1.1 PROBLEM STATEMENT

Currently, the situation of property theft is very sophisticated and complicated taking place in provinces and cities across the country, especially Ho Chi Minh City The thefts not only cause great damage to the property of families and businesses but also create anxiety and insecurity for many people and affect social order and safety After such incidents, many families have applied some techniques to increase the installation of burglar alarm systems for households Some surveillance systems currently are on the market such as CCTV cameras and digital video recorders (DVRs) which can record burglars' break-in activities, but they cannot be able to identify faces of humans [1] In the past year 2021, the rate of property theft increased gradually and accounted for a high rate with 37.6% in the crime chart [2] At night, detecting and identifying thieves in the dark will be a big obstacle for video camera systems that do not have night vision capabilities

Accordingly, smart home technology is receiving much interest to improve and enhance our life with full convenience, comfort, and security In particular, security is always the top concern and must ensure the safety of family members Currently, there are also many research papers by students who also work on smart homes such as author Tang Nguyen Cong Thien who has studied the topic "Application of image processing, design and structure of smart anti-theft circuits" using modules Arduino Uno R3, 800A sim module and Logitech HD Webcam C270 Webcam are used to send alarms and SMS alert

at the same time when opening and closing the door and was written by a MATLAB program [3] Author Duong Chi Hung has studied the topic "Designing an anti-theft circuit" using the Arduino NANO Board as the central microcontroller, using the Logitech

HD Webcam C270 camera to recognize faces If the recognition is correct, it will send a unlock signal and trigger the PWM pulse for the 9G RC Servo motor to open and close the door and use the matrix keypad to enter the password in case the circuit camera has a problem [4] Authors Tran Quoc Huy and Tran Ngoc Vinh research on the topic "Intelligent fire alarm system" using Arduino Mega 2560 control board as the central control block, ESP8266 Node MCU, MQ-2 gas sensor, sensor PIR HC - SR501 motion, fire detection sensor to monitor home burglar alarm system anywhere via a web server and phone, controlled by Android App The system can control the door which opens and closes via Android App and SMS notification via phone sim when there is a problem [5]

Through the above references, our team has chosen the topic " Anti-theft system design using OpenCV on Raspberry Pi 4" The microcontroller and sensor that we use in this project is a Raspberry Pi 4 Model B module, a Raspberry 5MP camera module, a PIR

HC - SR501 motion detection sensor The system has a home monitoring function, alerts users when the sensor detects motion and uses a 5MP camera to collect images of thieves through the OpenCV algorithm and Dlib library The system will warn with a buzzer, send facial recognition images via Gmail account, and update image and current time to the web server in real-time To create a better anti-theft system, our team's topic are using Raspberry

Pi as a fully functional computer that will have a faster processing speed than the Arduino microcontroller While Arduino only supports C and C++ languages, Raspberry is very flexible and supports most programming languages The Raspberry Pi 4 is also a lot faster than the Arduino (1.5 GHz compared to 16 MHz), making it faster to process images and alert users

1.2 PROJECT OBJECTIVES

In the overview of the topic, our team will design a system with high applicability

in the security system that warning of thefts Module camera can identify the thief's face after that the data images will be sent to the system and alert directly to the user It includes:

− Use a PIR motion sensor to detect the thief

− Use a 5MP Raspberry Pi camera to capture images of thief

− Apply the OpenCV algorithm to detect the thief ’s face

− Use the system of led lights to warn

− Send images of thieves to the web server for real-time updates, including the current dates and times

− Send an alert to the user's Gmail

1.3 RESEARCH CONTENT

During the implementation of the project "Anti-theft system design using OpenCV

on Raspberry Pi 4", our team focused on solving and completing the following contents:

− Content 1: Investigate about the theories, methods and algorithms used for object recognition problems Study about image processing, machine learning

− Content 2: Research and process images from cameras

− Content 3: Identity moving objects in the image

− Content 4: Research algorithms to process and send data to web server and Gmail

− Content 5: Complete the system and conduct test runs

− Content 6: Evaluation of performance results

1.4 LIMIT OF THE TOPIC

Due to limited time conditions as well as subjective factors (limited professional knowledge, financial conditions), the topic content is only implemented in the following scope:

− Depend on the accuracy of the PIR sensor

− The 5MP Pi camera has low resolution and poor image quality in conditions of strong or too weak light environment

1.5 THESIS OUTLINE

The project report consists of 5 chapters:

− Chapter 1: Overview: our team learn about the reasons for choosing the topic, the objective of the topic, the research content, the limitation and the layout of the topic

− Chapter 2: Background: this chapter presents knowledge of hardware such as Raspberry Pi 4 Model B, 5MP Pi camera module and PIR sensor Theories on

AI, Deep learning, image processing, face recognition methods, Python programming language and the introduction of the libraries used in the topic

− Chapter 3: System design: this chapter presents and analyzes the general block diagram of the system, shows the function of each block in the system and details

of the flowchart of each block

− Chapter 4: Experimental results: this chapter presents the hardware design process for parts of the system Presentation about the results of building the system model, comment and evaluation of the system in all aspects

− Chapter 5: Conclusions and recommendations: this chapter presents the results achieved after completing the thesis, thereby giving directions for future development

Figure 2.1 Raspberry Pi 4 Model B [6]

Figure 2.1 shows the Raspberry Pi 4 Model B, the newest member of the Raspberry

Pi family of computers Regular computers don't have General Purpose Input Pins (GPIOs), which is unique to the Raspberry Pi's board These GPIOs allow us to research and develop IoTs projects in our lives When compared to previous models, Pi4 improves on the following features: processing speed, multimedia performance, better memory and functional connectivity while consuming low power

2.1.2 Applications for Your Raspberry Pi

Raspberry Pi is a credit-card-sized computer but can serve many different uses and target diverse users such as research students, professionals or even hackers The most popular applications that can be used for general programming purposes, especially those related to IoT or AI Here is one of the simplest applications with great meaning:

− Building a home automation system: The system using a Raspberry Pi module to easily host several home automation applications It can be used to combine sensors, LEDs, relays and cameras with phones or computers We can easily control our home remotely

− Building a retro gaming console: If we want to play games, we will turn Raspberry Pi into a classic game console This is a program that runs on a custom SD card called RetroPie which allow us to use the controller to select emulators and games without touching the keyboard or mouse

− Using your Pi as a desktop: Running with Raspberry Pi Desktop is a great way when

we use an old computer that is not capable of running the latest version of Windows or macOS With 8GB model option the Raspberry Pi 4 will be great for basic everyday tasks like using a web browser, watching video streams, and editing documents, as well

as setting up a personal computer quickly

− Making your own CCTV system: The surveillance system built includes: Raspberry Pi module acting as a HUB with 1TB SATA and a camera that can be monitored via the HUB All devices are connected to the local network and we can access all cameras using your computer's browser or another device

2.1.3 Exploring the Raspberry Pi 4 Model B Board

The user can see the peripheral communication ports arranged on the Raspberry Pi

4 Model B shown in Figure 2.2

Figure 2.2 Raspberry Pi 4 Model B hardware design [7]

Figure 2.2 shows the ports arranged on the Raspberry Pi 4 Model B and their functions are announced by the Raspberry Pi Foundation in June 2019 with significant upgrades The new Raspberry Pi 4 features a high-speed of 1.5GHz Broadcom both CPU and GPU which can be configured with 1GB, 2GB, and 4GB memory configurations compared to the 1GB memory of the previous Raspberry Pi 3 [8] Raspberry Pi 4 uses Broadcom's BCM2711 processor by a completely new 28nm process with more powerful processor cores, which can perform tasks smoothly The pipeline depth of the Cortex A72 microarchitecture is 15 instructions, 7 instructions more than previous models As a result, the Cortex-A72 processor will be much faster and consume more power than the Pi 3 With these configurations, the Pi 4 can diversify usage needs from simple math problems to AI and other computer applications In addition, Raspberry Pi 4 is upgraded to Bluetooth 5.0 compared to 4.2 in previous Pi 3 versions, Micro-USB Type-B is changed to Type-C to allow compatibility with the connection environment of current Macbook laptops and supports 802.11ac Wi-Fi An Ethernet port is fitted on the top right side of the board for fast network speed without bottleneck Two USB 3.0 ports are fitted in the center of the device providing a high-speed connection to external devices This new USB port provides

a massive bandwidth boost and has a big impact on the performance of external storage devices with SSDs connected via USB adapters with the average read and write throughput

measured in MB/s The Pi 4's HDMI connection is changed to a micro port with the ability

to connect two monitors Furthermore, there are two micro-HDMI ports that enable support dual monitor up to 4K resolutions, driving two 4K displays at resolution up to 4Kp30 or a single display at 4Kp60 SD memory card is used to storage information, all information will be put into this memory card, audio jack for audio hardware communication

2.1.4 GPIOs Pinout

The GPIO pinout of Raspberry Pi 4 is depicted as shown in Figure 2.3

Figure 2.3 Raspberry Pi 4 pinout [9]

Figure 2.3 shows the pinout and key functions available on the Raspberry Pi 4 Model B The 40-pin GPIO header is included with the Raspberry Pi 4, with 26 GPIO (General Purpose Input/Output) pins Because GPIO pins can only have two states, off or

on, and have no specific function, the software will be used to alternate them The Raspberry Pi 4 device uses the 5V voltage level to output the 5V power supply via the Type-C interface, which includes pins 2 and 4 External components, such as pins 1 and

17, receive 3.3V power from the 3.3V voltage level Ground pins are used to connect electrical circuits together The ground pins (0V) safeguard the Pi 4 module from overheating and are crucial to the circuit (ground pins are numbered 6, 9, 14, 20, 25, 30,

34 and 39) I2C and EEPROM protocols, as well as digital pins 27 and 28, are communication via dedicated pins PWM is a spectrum technique used to convert digital signals to analog signals Only hardware PWM is supported by a few pins, including GPIO pins 12, 13, 18, and 19 All pins are capable of running software PWM The Serial Peripheral Interface (SPI) is a set of pins that allow devices like sensors, LEDs, and actuators to communicate with one another The Raspberry Pi uses the Master Out Slave Pin (MOSI) to deliver data to a device, while the device uses the Master In Slave Out (MISO) pin to connect with the Raspberry Pi GND, SCLK (GPIO 11), MOSI (GPIO 10), MISO (GPIO 9) and CE (GPIO 7) are the five GPIO pins required for SP communication The choices of MISO (GPIO 19), MOSI (GPIO 20), SCLK (GPIO 21), CE0 (GPIO 18), CE1 (GPIO 17), and GPIO 16 as CE2 will be modified in the case of the SPI1 pin

2.1.5 Raspberry Pi OS

Raspberry Pi OS, also known as Raspbian, is one of the Officially Distributed Operating Systems (ODOS) that can be downloaded and used for free Raspbian is well-suited to low-performance ARM CPUs of the Raspberry Pi version This system is based

on Debian Linux and is designed to interact with Raspberry Pi machines efficiently Debian

is a lightweight operating system that is suitable for the Raspberry Pi Raspbian comes with browser, python programming, and graphical user interface (GUI) desktop Raspberry Pi Operating System has recently published two Debian 10 and Debian 11 versions, are codenamed "Buster" and "Bullseye," respectively In there:

− A released Raspberry Pi OS “Buster” is said to inherit the previous version Raspbian

“Stretch” for compatibility with Raspberry Pi models Buster is based on Debian Linux

10 Like previous Raspberry OS operating systems, Buster also provides users with the PIXEL desktop environment (including LXDE)

Figure 2.4 Debian 10 installation at the end [10]

Based on Figure 2.4 above, the Debian 10 desktop virtual machine creation process has been successfully installed to the virtual hard disk on proxmox While there is not a big difference between Debian “Stretch” and Debian “Buster”, the big difference is the security changes that makes Buster becomes harder to hack The “Eject” icon is used

to remove a USB device appears only when a USB device needs to be ejected Other difference is mostly minor changes that most users may not notice

− Raspberry Pi OS 'Bullseye' is the latest Raspberry Pi operating system which based on Debian Linux predecessor “Bullseye” Debian 10 used Linux Kernel 4.19 while Debian

11 is upgraded to Linux Kernel 5.10 LTS to improve new bug fixes, supports new hardware and improves performance KMS video driver (kernel mode setting), formerly an experimental option, is now a standard video driver with this release This change makes it possible for applications that use the standard Linux Display API to run on the Raspberry Pi without modification

2.2 RASPBERRY PI CAMERA MODULE – 5MP

2.2.1 General

The Raspberry 5MP camera module is a low-cost, lightweight camera module is designed to work with the Raspberry Pi in range of computers The module is attached to the Raspberry Pi's CSI connection via a ribbon wire and communicates with it via the MIPI camera serial interface protocol CSI bus capable of extremely high data rates and it exclusively carries pixel data to the BCM2835 processor It can provide clear 5MP images, capture still images at 2592 x 1944 pixels and support video recording in HD 1080p at 30fps, 720p at 60fps With the latest version of Raspbian, the Raspberry Pi's Debian-based computer operating system, the camera module is always supported

Figure 2.5 Connection of Camera Module on Raspberry Pi board [11] 2.2.2 Characteristics

The Raspberry Pi Camera 5MP module is compatible with the following Raspberry

Pi camera models: Pi 4, Pi 3 B+, Pi 3, Pi 2, Pi B+, Pi A, and Pi Zero/Zero W The camera module is 25mm x 24mm x 9mm in size and weighs around 3g A 5MP OmniVision 5647 sensor is used in the module [11] When shooting still photographs, the resolution is 2592

x 1944 pixels, and video can be recorded in three different modes: 1080p @ 30fps, 720p

@ 60fps, and 640x480p 60/90 The MIPI camera serial interface has 15 pins and is connected to the Raspberry Pi directly

2.2.3 Pinout Description

User can see the pinout of Pi camera module shown in Figure 2.6

Figure 2.6 Camera module pinout [12]

Figure 2.6 shows the pinout and function of each pin on the camera module The ground pins are used to close the electrical circuits, protect the Pi camera module from burning and play an important role in the electrical circuit (the ground pins are numbered

1, 4, 7, 10) A voltage level of 3.3V is used to provide power for the device to operate, including digital pin 15 Digital pin 2 (CAM1_DN0) and pin 3 (CAM1_DP0) are MIPI Positive and Negative MIPI data, respectively counts for data 0, pin 5 (CAM1_DN1) and

6 (CAM1_DP1) represent level 1 Pins 8 (CAM1_CN) and 9 (CAM1_CP) provide clock pulses for the MIPI data lanes Pin 11 (CAM_GPIO) is an optionally used pin and pin 12 (CAM_CLK) is an optional clock pin Pins 13 (SCL0) and pin 14 (SDA0) are used for I2C communication

2.3 PIR MOTION SENSOR MODULE

2.3.1 Overview

Passive InfraRed sensor (PIR sensor) is a passive sensor that allow you to sense motion, which almost always used to detect the movement of a person entering or leaving the sensor's range PIR sensors are always a good choice in projects when they have a lot

of advantages such as small, inexpensive, quite sturdy, wide lens range, easy to use and low power consumption [13] However, the PIR sensor will not tell us how many people are around the area, the lens used here is usually fixed at a certain scanning angle and distance using the infrared excitation source Infrared (IR) rays are heat rays that emitted

by hot objects In our body, the body temperature is usually 37 ° C and always emits heat rays The manufacturer uses a galvanic cell to convert heat rays into electrical signals and detect hot moving objects The PIR sensor consists of 3 pins as shown in the figure 2.7

Figure 2.7 A typical PIR sensor [13]

Figure 2.7 introduces the type of PIR sensor HC-SR501 contains 3 pins, which are VCC, Input pin and ground Pin 1 of the sensor is connected to the ground Pin 3 is the input voltage of +5V for typical applications and ranges from 4.5V-12V Pin2 is provided 3.5V logic high output if a moving object is detected and low when no motion is detected

2.3.2 Characteristics

The PIR sensor module can be powered from voltage 4.5V to 20V to create flexibility for choosing the right operating source When the PIR sensor detects infrared motion, the output will be HIGH-level 3.3V, if the module does not detect motion, it will

be LOW 0 V after fixed period The delay time is adjustable from 0.3 seconds to 200 seconds and can be customized up to 10 minutes The operating temperature of the PIR sensor is impressive from -15 to +70 degrees Celsius Sensitivity range: The angle of the detection area is about 120 degrees cone and the range varies up to 7 meters The sensor module has compactly designed with dimensions of 32 mm x 24 mm, the diameter size of the Fresnel lens is 23 mm, which increases the detection range and it looks opaque

2.3.3 Working Principle

Passive infrared sensors use a pair of pyroelectric sensors to detect heat energy in the surrounding environment The sensor does not use spontaneous heating but only uses infrared radiation from human and animal bodies to alarm Any individual with a stable temperature will be radiated infrared to the outside world (average human body temperature is always maintained at 36°C - 37°C) We can understand the operating principle of the PIR sensor as shown in Figure 2.8

Figure 2.8 Working principle of PIR

Figure 2.8 shows that the PIR sensor is made up of two slots, each slot is made of a special material that is sensitive to IR and linked in such a way that they cancel each other out When the sensor does not detect motion, both slots will detect an equal amount of IR, the amount of ambient emanating from walls or the outdoors To increase the sensitivity of the slot, we use a Fresnel lens with a large detection angle, which filters out infrared rays and prevents ultraviolet rays When the object moves, the subject's body temperature will always emit heat rays, strongly condense with the Fresnel lens and then focus on the infrared sensor target, these 2 slots will give 2 signals and these signals will be amplified for a high amplitude and put into a voltage comparator circuit to affect a control or alarm device

2.4 INTRODUCTION TO DEEP LEARNING

2.4.1 Deep learning

Deep learning is a sub-branch of Machine Learning, used with the idea that based

on the human neural network (Neural Network) Deep Learning has helped computers perform tasks that were thought impossible many years ago, such as classifying different objects in an image, creating captions for images, recognizing human voices and handwriting, etc even in making movies and music [14] Deep learning is also an important application of the technology behind self-driving cars, allowing them to recognize traffic signs or distinguish pedestrians from traffic lights Deep learning is attracting a lot of attention recently because of its positive results that were impossible with other methods before The performance of deep learning compared to previous old learning methods is shown in Figure 2.9

Figure 2.9 Performance comparison between deep learning and old algorithms [15]

Figure 2.9 shows us that performance is the big difference between the two algorithms When the data is small, deep learning does not work well Therefore, when there is a large amount of data, deep learning performs perfectly

2.4.2 Artificial Neural Network

One of those Deep Learning methods is Artificial Neural Network - the multi-layer fully-connected neural layers Artificial Neural Network usually refers to the number of hidden layers in a neural network Traditional neural networks contain only 2-3 hidden layers, while deep networks can have up to hundreds of layers Artificial Neural Networks can be viewed as cascade models of cell types inspired by biological observations They are a combination of layers of perceptrons or multilayer perceptrons as shown in Figure 2.10

Figure 2.10 Artificial Neural Network [16]

Based on Figure 2.10, the general architecture of an Artificial Neural Network includes 3 components: Input layer, Hidden layer and Output layer

− Input layer: this layer is located on the leftmost side of the network to represent input data Each neuron corresponds to an attribute or feature of the input data

− Hidden layer: this layer is located in the center between the input and output layers, where the data is processed before outputting It represents the logical reason processing of the network This layer can have one or more Hidden Layers that are intermediate layers between the input and output layer and process the data by applying complex non-linear functions to them These layers are the key component that enables

a neural network to learn complex tasks and achieve excellent performance

− Output layer: this layer is on the right and it represents the output of the neural network After passing through the last hidden layer, the data will be transformed by a function called the activation function and given the final result

2.4.2.1 Application

Applications of deep learning are everywhere such as virtual assistants that help to understand the user's speech and the language people use when they interact with the machine, translation can help tourists, business people and people working in government For the transportation sector, drones and self-driving cars can move, stop, avoid obstacles

on the road or other vehicles with the ability to handle and act like humans during the process receive information Many companies can take good care of customers with optimal responses to customer questions thank to Chatbots deep learning applications

2.5 INTRODUCTION TO IMAGE PROCESSING

Image processing is one of the most important parts of the field of computer vision, the premise for many studies in this field is rapidly developing Image processing is the process of changing from an original image to a new image with the desired characteristics and requirements of the user Image processing includes the processes of analysis, classification of objects, enhancement, segmentation and separation, labeling of regions or the compilation of image information

Digital image processing includes transformation techniques and methods, intending to transmit or encode natural images The purposes of image processing include: improving and enhancing image quality, automatically recognizing images and evaluating image content Image processing is widely applied to practical problems, bringing high efficiency in many areas such as:

− Military: target detection and tracking, vehicle navigation, wide-area surveillance and automatic target identification

− Human-machine interface: image recognition, sound processing, graphics

− Safety and security: human face recognition, fingerprint recognition, eye pattern recognition

− Entertainment: video games

− Medical: Computed Tomography (CT) or Magnetic Resonance Imaging (MRI) for diagnosis of medical conditions or interventions such as surgical planning or research purposes

− Industrial production: identification and classification of products and fruits

Previously, the images which obtained from the camera were analog Currently, along with the continuous development of technology, color images or black and white

images taken from the camera will be directly converted to digital images to facilitate the next processing steps Fundamental steps in image processing are shown in Figure 2.11

Figure 2.11 Basic steps in image processing

Figure 2.11 introduces the map in image processing including 5 steps [17]:

− Image Acquisition is an important first stage in the image processing process Input images can be acquired through devices: cameras and scanners, which can be color or black and white images Then, these signals will be digitized with important parameters such as image resolution, color gamut quality, and image acquisition speed of the devices The quality of a captured image will be depended on the receiving device and the environment (light, weather, landscape)

− Image Processing: at this stage, the image may have low contrast noise, so it needs to

be put into a preprocessor to improve quality, improve contrast, and remove noise for the purpose of making the image more beautiful, sharp and quality This pre-processing stage is performed by filters with different noise filtering functions but will have the same purpose of improving contrast, clarity and matching user requirements

− Segmentation is the process of separating the input image into component regions to represent analysis and image recognition This is the most complicated and difficult step in image processing and can make errors and loss of image accuracy The result of image recognition depends a lot on this step

− Image representation: after the segmentation process containing pixels of the image area will come to represent the image The use of converting these data into an appropriate form is necessary for further processing by the computer The selection of features to represent the image is called Feature Extraction which associated with the extraction of image features in the form of quantitative information or like a to distinguish this feature class from other objects in the received image

− Image Recognition and Interpretation: Image recognition is defined as the process of identifying an image to obtain an image by comparing it with a previously learned (or saved) standard sample Interpolation is defined as making sense of judgment based on identification There are different ways to classify images The theory of recognition will include mathematical models of images and can be classified into two basic image recognitions: parameter recognition and structure recognition Identification objects are commonly applied in science and technology today: character recognition (print, handwriting, electronic signature), text recognition (Text), fingerprint recognition, identification barcode and facial recognition

2.6 HAAR CASCADE CLASSIFIER

2.6.1 Introduction to Haar Cascade

Haar Cascade, also known as the Viola-Jones method, is a machine learning object detection algorithm created with features for object detection and recognition proposed by Paul Viola and Michael Jones in 2001 It includes four essential features to identify an object in an image With Haar cascading, objects can be found in an image regardless of the object's position or size of the image Initially, the application was used to recognize faces and human features: eyes, nose and mouth However, many Haar features are developed to apply full body recognition, upper body, lower body, smile, and many objects including vehicles, and industrial products Cascade window extractions are used by Haar features to compute the features in each window and determine if it can be an object using rectangular features as shown in Figure 2.12

Figure 2.12 Sample of Haar features [18]

Figure 2.12 shows the types of rectangles which applied for Haar Feature extraction Firstly, the feature using two rectangles is the difference in the total number of pixels in two rectangular regions, which is used mainly for edge detection (a, b) The three-rectangle feature performs the sum of the two outermost rectangles minus the sum of one middle rectangle to detect lines (c, d) The feature uses four calculated rectangles to represent the difference between diagonal pairs of rectangles (e)

2.6.2 The algorithm Haar Cascade Classifier

Using the Haar classifier, the faces of objects are detected from real-time images, recordings, or videos There are four stages of the Haar Cascade classifier presented:

− Haar features selection: the first stage is collection of Haar features The common features that appear on a human face: the bridge of the nose is lighter than the eyes, the eye area is darker than the upper cheek, and the mouth is known as Haar Features Each Haar feature is a combination of two black and white rectangles which divided into 3 features shown in Figure 2.13

Figure 2.13 Haar cascade features

Figure 2.13 shows the features of the rectangle that are sorted in order on the image The total number of pixels in the white part is subtracted from the total number of pixels

in the black part Where edge features are filters used to detect edges in an image Line features will capture three straight lines in the image and Four-rectangle features have four squares to detect inclined lines in the image

− Creating an integral image: Calculating the features of a rectangle by using the built-in multipliers can be very time-consuming Therefore, the author has proposed an intermediate representation: the integral image which will allow to sum of any rectangle simply using four values which gives a significant speedup using the integral image through figure 2.14

Figure 2.14 Illustration for how an integral image works [19]

Figure 2.14 shows the total number of pixels contained in the rectangle calculated according to the four array references A, B, C and D At position 1, the value of the integral will be the sum of the pixels contained in rectangle A Next, the integral value

at position 2 is the sum of the two values A + B Position 3 will be the sum of the pixels contained in the two rectangles A + C, and the integral value at position 4 is the sum of

4 values A + B + C + D The sum of pixel values present in area D can be simply calculated as 4 + 1 - (2 + 3)

− AdaBoost training (Adaptive Boost): AdaBoost is used to select a small set of features and train the classifier The algorithm helps to deploy and build a strong classifier by combining the good features of many weak classifiers The final classifier is a weighted combination of several weak classifiers This matches a weak sequence of learners on different weighted training data and increases in accuracy

− Cascading classifiers: Although the above procedures are quite efficient, Viola and Jones improve with a reduction in computation time and an increase in detection rate using Cascading Classifiers The main purpose is correctly to identify the face and minimize the false ratio by rejecting the undefined and non-faced sub-windows while defining the regions The result will be a highly accurate classifier from the average prediction of all weak learners

2.7 PYTHON PROGRAMMING LANGUAGE

2.7.1 Introduction to the Python programming language

Python is an easy-to-learn, high-level object-oriented programming language that has a simple effective approach to object-oriented programming and was created by Guido van Rossum Python's syntax and statements are huge strengths: its clear presentation, ease understanding, and flexible typing make it quickly an ideal language for scripting and application development used in many fields and appeared on almost every platform Features of Python include:

− Simple Programming Language: Python is a high-level programming language with a simple, clear syntax Python programs are easier to read and write than other programming languages: C++, Java, C# Python made programming fun, making programmers more focused on solutions, not syntax

− Free and open-source: Python is an open source and there is no restriction on its use Users can not only use but also change the source code of Python as a community of users is constantly improving them

− Portability: Python programs can be moved from one operating system to another and executed without any changes It runs seamlessly on almost all platforms like Windows, Mac OS, Linux and UNIX

− Extensible and embeddable: when applications need high performance, users can combine parts of the code C, C++, and other programming languages into Python code

to make the program more efficiency at the level that other programming languages cannot provide

− Interpreted, high-level language: unlike C/C++ languages, Python programmers do not need to worry about difficult tasks like memory management, cleaning up meaningless data, etc When running the Python code, it will automatically convert the code into a language that the computer can understand and save the user from worrying about any low-level operations

− Large standard library: Python has a large number of standard libraries that make the user's programming job a lot easier, simply because you do not have to write all the code yourself

− Object-Oriented: Everything in Python is object-oriented Object-Oriented Programming (OOP) helps to solve complex problems visually With OOP, users can break down complex problems into smaller groups by creating objects

2.7.2 Introduction to OpenCV

Open Source Computer Vision Library (also known as OpenCV) is an open-source library for the research or development of computer vision and machine learning OpenCV has been existed for 1 period and was taken over by the nonprofit OpenCV Foundation in

2012 OpenCV is written natively from C and C++ programming, it can be used in other languages: Java, Python, Ruby and Matlab They are easily compatible with Windows, Linux and Mac OS operating systems OpenCV automatically uses the appropriate IPP library at runtime if the library is installed One of the goals of OpenCV is became the basis for computer vision to help users build fairly complex applications quickly It is mainly used for Computer Vision Applications and implemented additional features with GPU acceleration for real-time operations, this includes algorithms used for motion tracking, facial recognition, imaging, robotics, autonomous vehicles, structures 3D self-motion, object detection and more OpenCV library has outstanding features such as 2D and 3D support toolset, face recognition, gesture, object and behavior recognition, human-computer interaction and robot control

The basic structure of OpenCV is shown in Figure 2.15:

Figure 2.15 Basic Structure of OpenCV [20]

Figure 2.15 shows the basic structure diagram of OpenCV including 5 parts, which

4 parts are presented First, CV (Computer vision) is the component that includes image processing, image structure analysis, high-level computer vision algorithms, etc MLL (machine learning library) is a machine learning library, which includes many functions for clustering, classification, and data analysis HighGui provides I/O commands and I/O has functions of accessing and storing image and video files CXCore contains data structures, matrix algebra, basic data content, error handling, basic drawing and math, etc The figure above will not include CvAux, which contains experimental algorithms or discontinued functions

and academia: face recognition Dlib's deep matrix learning engine is intended to be used for face recognition with applications such as face detection, face alignment, and face recognition The linear algebra tools in Dlib are all handled by dense matrices Two main types of containers in Dlib are used for working with linear algebra: matrices and vector layers The matrix operations included in Dlib are implemented by the expression pattern technique, which allows them to remove temporary matrix objects and returned from expressions A pre-trained facial landmark detector included in the Dlib library is used to estimate the position of the 68(x,y) coordination that map the facial structures on the face shown in figure 2.16

Figure 2.16 The 68 landmarks detected by Dlib library

Figure 2.16 presents the indices of 68 datums (numbered 0 to 67) displayed on 68 landmark locations for each face image stitched together into a one-dimensional feature vector This model was trained with the iBUG 300-W dataset From this picture, we can see the exact positions of the eyes, lips, and nose and imagine the shape of the person's face

2.7.4 Flask Framework

Flask is a simple web micro-framework that requires no tools or libraries, making it easy for developers to access and build websites quickly Extension components are always supported by Flask for applications, including database integration, form validation, upload handling, authentication technology, templates, email, and RESTful Web Server Gateway Interface (WSGI) is used as a standard for Python web application development including

a test client to simulate HTTP requests Werkzeug and Jinja are the two essential core libraries that Flask built into a micro framework with an extremely simple, accessible, easy-to-learn style of writing In it, Werkzeug is a WSGI toolkit to execute requests, object responses, and utility functions which help to build a web framework on it Jinja2 is a full-fledged template engine with features: proper execution, powerful XSS prevention, template inheritance and easy debugging

Flask features include the development of servers and development of debuggers It depends on two libraries including the Jinja2 template engine and the Werkzeug WSGI toolkit It provides secure supporting for cookies, unit testing, and HTTP request handling Flask also provides secure request submission, high flexibility, ease of execution, a unique