INTRODUCTION
OVERVIEW
The global e-commerce sector has experienced remarkable growth in recent years, with industry giants like Amazon and Alibaba leading the charge As of April 2021, Jeff Bezos, the founder of Amazon, was the world's wealthiest individual, highlighting the immense financial success of online retail Companies such as eBay and Walmart also generate billions annually from e-commerce To maintain their competitive edge, these companies leverage advanced technology and a deep understanding of market trends and consumer preferences The significance of effective supply chain management has been underscored by the Covid-19 pandemic, which accelerated online shopping as people stayed home Amazon reported shipping a record 1 billion items during the holiday season, while Alibaba can handle up to 60 million orders daily, showcasing their operational prowess in the evolving digital marketplace.
Vietnam is rapidly emerging as one of Southeast Asia's fastest-growing e-commerce markets, with 53% of the population expected to shop online by 2020, according to the Ministry of Industry and Trade Despite the challenges posed by the COVID-19 pandemic, Vietnam's e-commerce revenue rose by 18% to $11.8 billion, accounting for 5.5% of total retail sales However, technology remains a significant barrier to unlocking the market's full potential As consumer demand for online shopping increases, key factors such as accurate product classification and prompt delivery will determine customer retention To compete effectively with global brands, Vietnamese e-commerce companies must leverage advanced technologies.
Modern equipment ensures accurate classification and safe, swift delivery of goods to clients Unlike many units that rely on manual classification by workers, leading to potential delivery and safety issues, major companies like Shopee Vietnam, Lazada Vietnam, and Tiki have adopted more advanced methods To address these challenges, I have developed an autonomous goods classification system utilizing QR codes, specifically designed for logistics and e-commerce storage needs.
OBJECTIVES
A disdistinctive QR code on packaged goods includes the order number and the delivery zip code Three goals are outlined within the context of this thesis, including:
Developing a product classification system by delivery region using a robotic arm and machine vision Labor costs will be cut with the use of robots in production
Create a database management system to supervise commodities in a warehouse before export
Create a user interface to track and track the system's
SCOPES
The mechanical design of the robot is inadequate, leading to challenges in orbital planning for the group robots, which results in the robots failing to achieve the desired trajectory.
Open-loop driven stepper motors lead to the accumulation of control errors, with occasional slippage when the motor is hot
The measurement of the robot's parameters is done with a straightedge, so the accuracy is not high
The image processing part obtained from the camera is sensitive to sudden brightness changes, the tracking algorithm is not continuous due to noise
The user interface software sometimes crashes while in operation.
RESEARCH CONTENT
Content 1: Learn QR code and set parameters
Content 2: Write a program to recognize image processing using OpenCv Python
Content 3: Build a database system to store data upon identification
Content 4: Design GUI user interface
Content 5: Hardware design of conveyor belt and robot arm
Content 6: Write an Arduno program to control an automatic robot
Content 7: Connect hardware and software, run model execution, evaluate results
Content 8: Write an implementation report
Content 9: Defend the graduation thesis.
METHODOLOGY
These are the approaches used to accomplish these objectives:
- Understand the format and decoding of QR codes
- Recognize image processing techniques for input
- Robot arm control algorithm; robot arm model controller design:
- Acquire knowledge about Microsoft SQL platform data management.
BACKGROUND INFORMATION
Overview OF QR CODE
A QR code, or quick response code, is a two-dimensional matrix barcode developed by Denso Wave in Japan in 1994 These codes are designed to be scanned quickly and efficiently, allowing for easy access to information.
"Quick Response" (QR) codes, designed for rapid decoding, are predominantly utilized in Japan and China, where they have become the most widely used two-dimensional codes.
Originally solely employed to track automobile components during manufacture,
QR codes have become increasingly popular across various sectors, including brand tracking and convenience applications They not only display text but also enable users to save vCards, open URLs, send emails or messages, and facilitate electronic payments In China, QR code payments are widely adopted, making it a common payment method Users can create and print their own QR codes, allowing others to scan them to access both free and paid websites This versatility has made QR codes one of the most favored formats within the 2D barcode family.
2.1.2 The development history of QR codes
In 1970, IBM developed the 13-digit UPC symbols to allow automatic input into the computer These UPC symbols are still widely used for point-of-sale (POS) systems
In 1974, Code 39 was introduced, capable of encoding around 30 alphanumeric characters By the early 1980s, advancements led to the creation of multistaged symbol codes, such as Code 16K and Code 49, which could store approximately 100 characters As the information technology industry evolved, the demand for symbols that could hold more data and represent multiple languages, beyond just English, became increasingly important.
To accommodate the need for a symbol code type that can hold more characters than traditional multistaged symbol codes, the QR Code was developed This innovative code can store up to 7,000 alphanumeric characters, including Kanji characters used in Japan.
1994 Fig.2.2 describes the development process of these types of code
Figure 2 2 The history of the development of symbol codes [5]
Figure 2 3 The evolution of QR codes[6]
In Figure 2.3, QR codes offer the perfect combination of three main characteristics: large data capacity, high reading speed, and minimal printing sp
Structure and specifications of the QR code
The Finder pattern enables accurate detection of a QR code's location by positioning it in the three corners of the icon, allowing for the identification of its position, size, and angle This design ensures that the QR code can be scanned from any direction, providing 360-degree readability.
The timing pattern in a QR Code defines the central coordinates of each cell using alternating black and white patterns This feature is crucial for correcting the coordinates of data cells in cases of distortion or cell pitch errors The timing pattern is designed to function effectively in both portrait and landscape orientations.
The alignment pattern effectively corrects nonlinear distortions in QR Codes by determining the center coordinates of the symbol To facilitate this detection, a black isolating cell is incorporated into the alignment pattern, enhancing the accuracy of distortion correction.
Quiet zone: margin space required to read QR Code This static area makes it easy to detect symbols among the images read by the sensor/camera
The data area of a QR Code is where information is encoded, represented by the gray section in Figure 2.4 This data is transformed into binary digits, '0' and '1', following specific encoding rules These binary numbers are then converted into black and white cells, which are organized accordingly Additionally, the data area incorporates Reed-Solomon codes to enhance data storage and provide error correction capabilities.
2.2.2 The specifications of the QR code
Table 1 Specifications of QR code
Symbol size Min is 21x21 squares and Max is 177x177 squares
Information type and information storage volume
Simple number Maximum 7,089 characters Numbers and letters Maximum 4,296 characters Binary numbers (8 bits) Maximum 2,953 bytes Kanji/Kana Maximum 1,817 characters
Level M 15% of the codewords can be recovered
Level Q 25% of the codewords can be recovered
Level H 30% of the codewords can be recovered
Link function A QR code can be divided into up to 16 small symbols, capable of linking information together
Practical applications of QR codes
- Inventory of goods, product information, personal information on visiting cards
To access the store URL, simply scan the QR Code with your phone, which will automatically open the website in your browser This convenient method is increasingly common in public spaces, particularly for medical declarations.
- Application at bus, train, and subway stations: when users scan the QR Code of the bus station, they will know information about the bus trips
- Use at museums: users only need to scan the QR Code
Placed next to an exhibit is to get detailed and up-to-date information about it
- Use at supermarkets: for information, cooking instructions as well as nutritional content of the food to be purchased, origin, expiry date,
- Use at seminars, presentations, events, music festivals, live shows, bars, clubs: participants can use QR Code instead of their Business Card
- Use at restaurants, hotels, coffee shops: to know the recipe and how to prepare food, drinks, hotel information
- Use in marketing advertising communication: gradually replace advertising in the form of printing and distributing leaflets QR Code will attach brand information on publications, store signs, billboards, billboards
- Use QR codes to differentiate common email signatures; on gifts, to create special and surprise for the recipient, and make the recipient curious about the content of the attached
Figure 2 5 QR code image placed in public places to declare health for people [8]
Background information of image processing
The classification of goods in carrier warehouses relies heavily on the effective processing of images, as each image contains vital information To extract this information, an image processing technique is necessary, involving various operations on the input image to achieve the desired outcome The results of image processing can range from enhanced image quality to specific conclusions about the items being classified.
In recent years, advancements in computer hardware and technology have led to a growing integration of image processing techniques into everyday life, driven by the need to modernize and replace human labor with machines This evolution not only enhances quality processing and image recovery but also merges with artificial intelligence, resulting in significant applications such as object recognition, autonomous vehicle technology, disease identification and diagnosis, and robotics.
2.4.2 Open Source Computer Vision (OpenCV - Open Source Computer Vision)
OpenCV is a premier open-source library for computer vision and image processing, offering over 500 functions and 2500 optimized algorithms for efficient image handling Built with optimized C/C++, it utilizes multi-core processing to deliver rapid real-time performance across various platforms, including Windows.
Linux, Mac, OS… and are free to use for both commercial and non-commercial purposes
The OpenCV project began in 1999 and was introduced at an IEEE conference in 2000, focusing on challenges in computer vision and identification It wasn't until 2006 that OpenCV 1.0 was officially released Initially supported by Intel, the library later received backing from Willow Garage, a robotics technology lab Today, OpenCV remains an open-source library, developed by a non-profit foundation.
Before extracting the necessary content, it is necessary to take steps to improve the input image quality to optimize, improve the accuracy and output quality of the processing
Image acquisition is the essential first step in processing and system operation, primarily achieved through devices like cameras and smartphones The effectiveness of the system heavily relies on the quality of the acquired images, which can be influenced by external factors such as lighting and weather conditions Therefore, selecting the right image acquisition equipment is critical, taking into account the system's requirements, including object characteristics, operating environment, and processing speed Additionally, budget considerations play a vital role in choosing equipment that meets quality standards Leading manufacturers like Keyence, Omron, and Cognex offer specialized cameras designed to fulfill practical needs in various applications.
The image obtained from the camera will be a digital image and transmitted to the central processor to perform the next steps of the preprocessing
Digital images captured by a camera typically utilize the RGB color system To facilitate easier image processing, it is essential to convert these RGB color images into grayscale images.
The RGB color system is a matrix of pixels where each pixel represents a specific color point, composed of three primary color layers: red, green, and blue In a 24-bit encoded image, each pixel is represented by 1 byte (8 bits), allowing for values ranging from 0 to 255 This results in a total of 16.7 million possible colors, as each pixel can have combinations of values across the three color channels For instance, a pixel can be represented as (0…255, 0, 0) for red, (0, 0…255, 0) for green, and (0, 0, 0…255) for blue.
A gray image consists of a matrix of pixels, where each pixel is assigned a value ranging from 0 to 255 In this scale, a pixel with a value of 0 represents the darkest color, while a pixel with a value of 255 signifies the lightest color.
Converting RGB color image to Gray image will be done according to the formula:
• 𝐼(𝑥, 𝑦): Is the luminous intensity at the point with coordinates (𝑥, 𝑦) of the gray image
• 𝑅𝑒𝑑(𝑥, 𝑦): Is the value at the pixel with coordinates (𝑥, 𝑦) of the Red color channel in the RGB color image
• 𝐺𝑟𝑒𝑒𝑛(𝑥, 𝑦): Is the value at the pixel with coordinates (𝑥, 𝑦) of the Green color channel in the RGB color image
• (𝑥, ): Is the value at the pixel with coordinates (𝑥, ) of the channel that Blue in the RGB color image
• The coefficients 0.3086, 0.6094, 0.0820 are proportional to each color channel The value of the 3 coefficients can be different depending on the programmer's wishes and their sum is always equal to 1
Figure 2 6 Color image is converted to grayscale image
- Filter noise and enhance image quality:
Camera images often fall short of expected quality due to various types of noise, including salt-and-pepper noise and Gaussian noise, which can lead to blurriness and loss of detail To improve the quality of input images, specific enhancement methods can be applied.
The median filter operates by scanning each pixel of the input image using a filter window, typically sized 3x3 or 5x5 The pixel at the center of the filter window is designated as the filtered pixel, which is determined by calculating the median of the values within the filter window area of the original image For a series of numbers containing (2n+1) values, the median is identified as the middle number when the sequence is arranged in either ascending or descending order.
Threshold refers to a value that divides two regions, specifically below and above the threshold This concept is applicable not only in image processing but also in various functions, where different expressions are used depending on whether the value is below or above the threshold.
Image thresholding is an effective image segmentation technique that transforms grayscale or color images into binary images This method is primarily used to distinguish between foreground (object) pixels and background pixels, facilitating enhanced image processing.
The input image is a grayscale image and the output image is a binary image for the purpose of separating the foreground and background To perform thresholding use the following formula:
𝐼(𝑥, 𝑦) is the pixel value at (𝑥, ) of the output image
𝑓(𝑥, 𝑦) is the pixel value at (𝑥, ) of the input image
Note: If you choose inappropriate threshold, the result will be wrong and cannot distinguish background and foreground
Step 1: Choose a threshold 𝑇 (𝑇 can be any value, for example choosing the average gray level of pixels is one option)
Step 2: We divide pixels into 2 groups as follows:
• G1: is the set of pixels with gray level >T
• G2: is the set of pixels with gray level ≤ 𝑇
Step 3: Calculate the average gray level of the pixels of 𝐺1 and 𝐺2:
Step 4: Calculate the new threshold:
Step 5: Repeat steps 2 to 4 so that the value between 2 consecutive time is less than a given value
Figure 2 8 Adaptive Global Threshold Image
Detect objects and scan QR codes with the camera
In Python, a valuable tool for object detection is contour, which is defined as a continuous curve made up of points that share similar color values Contours are characterized by their seamless, gapless nature, making them essential for identifying shapes and objects within images.
To accurately identify the contour of an image, begin by converting the color image to a grayscale image, followed by transforming the grayscale image into a binary image Utilize the appropriate syntax for the contour detection method to achieve the desired results.
• contours: List of contours in the binary image Every one contour is stored as a vector of points
• hierarchy: A list of vectors, containing relationships between contours
• _ : Image after using contour, usually we don't use this argument
The original binary image, referred to as binaryImage, is crucial for image processing tasks It is important to note that utilizing the findContours function alters the binaryImage's value To preserve the original data, it is advisable to use binaryImage.copy() before applying any modifications.
• typeofContour: has the following forms: RETR_EXTERNAL, RETR_LIST, RETR_CCOMP, RETR_TREE, RETR_FLOODFILL
• methodofContour: The following methods are available: CHAIN_APPROX_NONE, CHAIN_APPROX_SIMPLE,CHAIN_APPROX_TC89_L1,
Python supports QR code reading library pyzbar The image obtained from the camera containing the QR code will be decoded through the decode command
The content of the scanned QR code will be xxx.SGQy.z
In there: xxyyyyyy is the distributor code x is the district code of Ho Chi Minh City (Go Vap District y = 1, Thu Duc District y = 2, District 10 y = 3) y is the order ID (000001, 000002, )
Through reading the QR code, we will classify products by area delivery to facilitate distribution to the buyer.
Robot control programming
1 The base coordinate system: is located at the origin of the robot The basic coordinate system has Zero at the origin of the robot, making movements predictable for stationary robots It is therefore very useful for moving from one location to another
The right-hand rule is a method for determining the orientation of the x, y, and z axes in three-dimensional space To apply this rule, hold your right hand with your pinky and ring fingers down while extending your thumb, index finger, and middle finger so that they are all perpendicular to each other In this configuration, your thumb points in the direction of the z-axis, your index finger indicates the x-axis, and your middle finger shows the direction of the y-axis.
Figure 2 9 The right hand determines the direction of the axes
2 The word object coordinate system: with targets and paths when programming robots brings a lot of advantages
Repositioning a robot's work on the conveyor is simplified by adjusting the work object's coordinate system, which automatically updates all associated paths simultaneously.
Allows working on work pieces that are moved by external axes or conveyors, since the entire work object with its paths can be moved
3 The tool coordinate system: Determines the position of the tool that the robot uses when performing the programmed targets The tool coordinate system has the origin O at the midpoint of the tool It determines the position and orientation of the instrument
4 The world coordinate system: Used to define the robot cell, all other coordinate systems are related to the World coordinate system in a direct or indirect way This is a very useful coordinate system for handling some robots or when the robot moves by following external axes
5 The user coordinate system: The user coordinate system can be used to represent fixed devices such as work platforms This is useful for handling robots to move goods
Components Used To Design Hardware
Figure 2 11 Robot Arm Table 2 Basic parameters of the robot arm
Link1 (cm) Link2 (cm) Link3 (cm) Link4 (cm) Link5
The 17HS4401S stepper motor is designed specifically for 3D printers, laser engraving machines, and other motion control applications Its use in production enhances machine performance by ensuring smooth operation and minimizing noise during use.
Figure 2 12 17HS4401S Stepper Motor Table 3 Parameters of stepper motor 17HS4401S [Datasheet]
The Arduino UNO utilizes three 8-bit AVR microcontrollers: ATmega8, ATmega168, and ATmega328, with the UNO R3 specifically featuring the ATmega328 This versatile board is capable of executing simple tasks, including controlling LED flashing, processing signals for remote control cars, creating temperature and humidity measurement stations with LCD displays, and managing engine control functions.
Figure 2 13 Board Arduino UNO R3 Table 4 Basic parameters of Arduino UNO R3 [datasheet]
Operating voltage 5VDC (powered via USB only)
Number of Digital I/O pins 14 (6 pins PWM)
Flash memory 32KB (ATmega328) với 0.5KB used by bootloader
The CNC Shield V3 is an expansion board designed for the Arduino UNO R3, enabling the control of mini CNC machines It features four slots for A4988 stepper motor control modules, allowing for precise management of the X, Y, and Z axes, along with the option to control a fourth axis on these machines.
Figure 2 14 Board CNC Shield V3 Salient features:
- GRBL compatible (open source running on Arduino UNO R3 to control mini CNC)
- Supports up to 4 axes (optional X, Y, Z and fourth axis)
- Supports up to 2 Endstops (terminal sensors) for each axis
- Spindle control feature, control the cooling solution when the machine works
- Using stepper motor control module saves replacement and upgrade costs
- Set the stepper motor resolution by simple jump
- Power supply voltage is diverse from 12 - 36V
Driver is an extremely compact stepper motor driver circuit, supporting many working modes, adjusting the output current for the motor, automatically disconnecting the power when it is overheated
Figure 2 15A4988 Driver Table 5 Basic parameters of A4988 Driver [datasheet]
Continuous feed current for each phase
1A (no need for heatsink, cooling) 2A (need heat sink, cooling)
Selecting to resolution will depend on 3 pins MS1, MS2, MS3 If these 3 pins are floating, it means that the driver is operating in full step mode
Table 6 Microstep configuration for A4988 driver [datasheet]
MS1 MS2 MS3 Microstep Resolution
Low Low Low Full step
High Low Low Half step
Low High Low Quarter step
High High Low Eighth step
High High High Sixteenth step
The 12V - 5A honeycomb source is designed to convert voltage from 180V - 240V
AC source into 12VDC DC source to supply operating devices In industry, honeycomb power is often used to power some equipment of electrical cabinets
Figure 2 16 12V 5A Power voltage Table 7 Basic parameters of 12V 5A power voltage [datasheet]
The LS220 geared motor features all-metal gears in its gearbox, ensuring durability and reliability It is designed with robust brushes that enhance its longevity, providing high torque while operating with minimal noise This motor is specifically utilized for efficiently pulling conveyor belts.
Figure 2 17 12V geared motor LS220 [datasheet]
Table 8 LS220 Basic parameters of 12V LS220 geared motor
No-load speed at 12VDC 30rpm
A limit switch is an electromechanical device functioning as a contact sensor, commonly utilized across various industries to detect the position or presence of objects.
Figure 2 18 Limit switch KW11-3Z [datasheet]
Table 9 Basic parameters of the KW11-3Z limit switch
Module 1 Relay 5VDC with compact size, low cost is used to switch AC or DC equipment, the module uses 5VDC voltage with only 3 pins for easy connection
Figure 2 19 High-powered 5V relay module [datasheet]
Table 10 Basic parameters of high-powered 5V relay module
Snail hole size line 3.1mm
The Pyp -370 vacuum pump features a compact design and operates with low noise, making it ideal for various applications, including baby products, medical care, and household appliances Its durability and efficiency make it suitable for devices such as massage machines, oxygen generators, breast pumps, and vacuum packaging machines.
Figure 2 20 Vacuum Pump 12VDC Pyp – 370 Table 11 Basic parameters of Pyp - 370 vacuum pump
Flow Higher than 2LPM/min
Vacuum pressure Lower than -58 Kpa
Noise level Lower than 50dB
The mini conveyor model uses a 12VDC reducer motor, measuring 8cm x 50cm, using a 2020 aluminum frame
The Logitech C310 webcam offers a maximum resolution of 720p and a frame rate of 30 fps, making it an affordable and well-configured option for basic computer vision applications.
Figure 2 22 Webcam Logitech c310 Table 12 Basic parameters of Logitech webcam
High: 71.15 mm Width: 31.1 mm Thickness: 25.95 mm
Operating system support Window 8 or later, MAC - OS,
Program used for software
PyCharm, created by JetBrains, is a powerful Integrated Development Environment (IDE) designed specifically for Python application development Major companies such as Twitter, Facebook, Amazon, and Pinterest utilize PyCharm for their Python programming needs.
PyCharm is a versatile integrated development environment (IDE) that operates on Windows, Linux, and Mac OS, making it accessible for a wide range of users It includes essential modules and packages that streamline Python software development, allowing programmers to create applications more efficiently Additionally, PyCharm offers customization options to meet the specific needs of developers, enhancing their coding experience.
The outstanding features of Pycharm can be mentioned as: smart program editor, color scheme for keywords, classes and functions, autocomplete and easy error correction,
Qt Designer is a tool for quickly building graphical user interfaces with widgets from the
Qt GUI framework It provides users with a simple drag and drop interface to layout elements like buttons, text fields, combo boxes and more
Qt Designer creates ui files, which are XML-based formats that organize widgets in a tree structure These files can be loaded during runtime or converted into programming languages such as C++ or Python.
Matlab, developed by MathWorks, is a powerful numerical computing and programming environment that excels in matrix arithmetic, graphing functions, and information visualization It enables users to implement algorithms, create user interfaces, and connect with various programming languages, making it a versatile tool for engineers and scientists.
The Arduino IDE utilizes the widely-used C/C++ programming language, allowing programmers to easily recognize and implement any C/C++ code This compatibility makes it convenient for developers to create programs for Arduino boards.
The Arduino IDE offers access to over 700 libraries created by Arduino Software and its community members, allowing users to utilize these resources for their projects at no cost.
System analysis and design
System requirement
The computer processes the webcam image using algorithms to detect and classify objects based on QR codes, determining the coordinates of moving objects Once an object is within a predefined distance, the robot performs inverse kinematic calculations and transmits the corresponding variable values to the Arduino The Arduino then controls the stepper motor attached to the robot's joints, positioning it accurately through the A4988 driver and CNC Shield V3.
Block Diagam
Figure 3 1 Block diagram of system
Power Block: This block is responsible for providing power to the controller block Controller Block: Includes Arduino Uno R3 and CNC Shield V3, driver A4988
Arudino after receiving data from the user interface on the computer, will pulse the cnc
27 shield to make driver A4988 work When the cnc shield is pulsed, it will make the robot's motors rotate in the selected position
User Interface Block: This block is used to send data to the controller block There are 2 modes: automatic mode and manual mode
The data block holds the QR code's ID, which, upon scanning by the webcam, allows the user interface to extract information from the data block This information is then utilized for object recognition, with the coordinates of the identified object being sent to the controller block.
Scan QR Code Block : This block is a webcam used to scan the qr code on each object
Pick and Place Block : This is the last block of the system The robot arm is responsible for picking and placing the object after the object has been classified
Flowchart
Figure 3 2 Flowchart of the system
The robot will be activated and returned to its starting position to prepare for object retrieval Once orders with QR codes are placed on the conveyor belt and recognized by the camera, the robot arm will sort them to their designated locations.
If the order belongs to type 1, it will be delivered to the position of type 1 similar to types
The robot is programmed to only pick up objects that fall into specific categories; if an order does not match these categories, the robot will not retrieve the item Once it successfully picks up an object, the robot will return to its original position to await the next command.
Hardware Design
3.4.1.1 Calculate the angel of the robot
Figure 3.3 Flowchart of the robot arm
To be able to calculate the robot's angles, I followed 2 rules of robot kinematics:
Forward kinematics used to calculate the coordinates of the end effector
To develop kinematic equations that characterize the motion of a robotic manipulator, each link is treated as a rigid body that connects the axes of two adjacent joints These joint axes are represented by straight lines in three-dimensional space, and a coordinate frame is assigned to each link of the robot.
Construct a general transformation between adjacent coordinate frames
− describe the relationship between the adjacent links Find the relationship describing the end-effector relative to the coordinate system attached to the origin {0}
Two parameters of the bar (link parameters): a i , i Perform the transformation along the axis X i
1The length of the i bar (kinematic length of link): a i is the distance from the axis Z i − 1 to
2 Twist angle (link twist) i is the rotation angle of the axis Z i − 1 around the axis X i to parallel (or coincide) with the axis Z i In other words i is the angle between the axis
The transformation along the axis Z i − 1 is defined by the joint parameters Z i and Z i, which are oriented around the axis X i The positive direction of this transformation is established by the right-hand rule, indicating the relationship between two adjacent bars through the parameters i and i d θ.
3 Distance between two joints d i is the distance between the axis X i − 1 and axis X i measure along the axis Z i − 1 This distance is also called the offset of the bar
4 Joint rotation angle i is the axis rotation angle X i − 1 around the axis Z i − 1 come parallel to the axis X i In other words: i is the angle between the axis X i − 1 and axis X i rotate around the axis Z i − 1 The positive direction is determined by the right-hand rule
Figure 3 3 Denavit-Hertenberg parameter sets
3.4.1.3 Solve the forward kinematics problem
Forward kinematics is a method used to calculate the position and orientation of a robotic tool based on known joint angles By utilizing experimental measurements, we can effectively determine the parameters of the robot.
Table 13 Basic parameters of the robot arm d 1(cm) a 2 (cm) a 3 (cm) a 4 (cm) a 5 (cm)
Attach coordinate systems to each joint of the Robot
Figure 3 4 Mount coordinate systems on robots Building a Denavit - Hartenberg
Table 14 Parameter table Denavit - Hartenberg
Form the Denavit-Harenberg matrix formular
Infer the transformation matrix of End-effector vs Base.
3.4.1.4 Solving the inverse kinematics problem
Inverse kinematics is a method used to calculate the fitting variable values when the position and orientation of a device's end-effector are known, contrasting with the forward problem approach.
From the forward kinematics calculation, we have the coordinates of the end - effector:
Control method: Perform pulse feed to drive the stepper motors to the desired position Stepper motor 1 will drive joint 1 of the robot to rotate with the gear ratio:
Stepper motor 2 will drive joint 2 of the robot to rotate with gear ratio:
Stepper motor 3 will drive joint 3 of robot to rotate with gear ratio:
Gear ratio = 20The A4988 driver is configured with microstepping in 1/16 mode, so the number of pulses to drive the stepper motor is:
Pulse = Gear ratio = Gear ratio
Suppose initially the values of the robot's matching variables are 1 , 2 , 3 We need to perform control of the robot to the position
Then the number of pulses needed to drive the stepper motors is:
Pulse = Gear ratio − = Gear ratio
Pulse = Gear ratio − = Gear ratio
Pulse = Gear ratio − = Gear ratio
The mechanical design of the robot requires that when the second joint rotates, the third joint's variable remains constant, necessitating that both the second and third motors rotate at the same angle.
Thus, the number of pulses needed to control the stepper motor 1, 2, and 3 are respectively:
The L298N driver operates on a 12V supply from the honeycomb source, with the IN1 and IN2 pins connected to digital pins 7 and 8 of the Arduino, enabling control over the motor's direction.
35 the motor's rotation The ENA pin is connected to the Arduino PWM pin 5 to control the output voltage value OUT1 OUT2
Figure 3 5 Hardware connection diagram for position control 3 Stepper Motor of Robot
The CNC Shield V3 operates on a 12V, 5A power supply, controlled by a 3-pin switch for easy power management Stepper motors for joints 1, 2, and 3 of the robot connect to the Z, X, and Y axes on the CNC Shield V3 Additionally, the travel switches for these joints are linked to the corresponding Z, X, and Y limit switch pins on the CNC Shield V3, enabling accurate determination of the robot's initial joint variables.
The hardware connection diagram for the On/Off control of a robot effector illustrates that the relay operates on a 5V power supply from the CNC Shield V3 The SpnDir pin on the CNC Shield V3 is linked to the signal pin on the relay, enabling the control of the vacuum pump's On/Off function through relay switching.
Figure 3 7 Connect the webcam to the conveyor
The webcam, mounted on a bar connected to the conveyor, detects incoming orders with QR codes in its identification zone Upon recognition, it sends a signal to the user interface, enabling the classification of the order.
Connection Diagram Of The System
Figure 3 8 System Hardware Connection Diagram
The computer processes the webcam image using algorithms to detect and classify objects based on color, shape, and QR codes, while also determining the coordinates of moving objects Once an object is within a designated distance, the robot conducts inverse kinematic calculations and transmits the corresponding variable values to the Arduino The Arduino then controls the stepper motor, attached to the robot's joints, to achieve the desired position using the A4988 driver and CNC Shield V3.
Software Design
Figure 3 9 Block diagram of image processing algorithm
The frame-by-frame processing algorithm utilizes background subtraction to differentiate between the foreground image captured continuously by the camera and the background image taken in the absence of objects By focusing on the area of interest, specifically the conveyor area, the algorithm narrows the detection region and processes any noise present in the frames Once an object is identified, the algorithm determines its center and contour, classifying it based on features such as shape, color, or QR code Finally, the object's coordinates are converted into the robot's coordinate system and transmitted to the Arduino for object retrieval.
* Determine the center of the object and the contour of the object
After performing noise filtering and background subtraction, we get an image containing only the object to be tracked in the area of interest Use OpenCV's findContours algorithm
To achieve accurate results with the findContours algorithm, it is essential to limit the object's size to minimize noise from surrounding areas Additionally, determining the center of the object is crucial for effective contour detection.
The center of a shape is the average of all the points in that shape For example, a figure consists of n distinct points x_1,x_2,…,x_n then the center is determined by the formula:
In image processing and computer vision, images consist of pixels, and the center of a shape within an image is determined by the weighted average of these pixels Image moments serve as the weighted average of pixel intensity, enabling the extraction of key features such as the center, area, and radius of the image The formula for calculating the moment of an image area plays a crucial role in analyzing these characteristics.
The OpenCV library provides the cv2.moments function, which is essential for locating the center of a shape identified by the findContours algorithm This function returns a dictionary containing the coordinates of the object's center on the image plane, calculated using a specific formula.
Using the moments function, we can determine the object's center coordinates and convert them to the robot's coordinate system These coordinates are then transmitted to the microcontroller, directing the robot to pick up the specified object.
Figure 3 10 Example of defining the contour and center of an object
3.6.2 Create customer database using mysql workbench
The database we use here is MySQL Workbenh We will create 2 tables named info_customer and robots
The info_customer table contains nine columns: ID, Nguoi_Gui, Nguoi_Nhan, Noi_Gui, Noi_Nhan, Ngai_Gui, Ngan_Nhan, San_Pham, and Phi_Code Its primary function is to generate unique QR code IDs that encapsulate all relevant order information.
The Robot table features seven columns: ID, Ngai_Xuat, Nguoi_Gui, Noi_Gui, Ngai_Gui, Ten_SP, and So_Tien It is designed to present information and timestamps after the system has processed and classified the data.
Figure 3 13 Manual mode control interface Connection : make connection with arduno
Configuation : to configure the speed and acceleration for the stepper motor
Basic control : Calib the robot angle to the original position, on off griper
Joint mode : solve the forward kinematics problem
World mode : solve inverse kinematics problem
Current position : current position and orientation of the switch head
Set position : control the robot to the desired position
Figure 3 14 Auto mode console Star cam : turn on the camera
Stop cam : turn off the cam
RESULTS AND ASSESSMENT
Result
4.1.1 Programming the control of the robot arm
Understanding the characteristic parameters of the robot arm model Mk2, solving the problem of inverse kinematics to control the robot to the desired position
Understand how to control a stepper motor, from which the joint can be controlled simultaneously
I have measured 3 times with a straightedge and collected the above data table, because there is no specialized measuring device, the measurement results will not be absolutely accurate
Figure 4.1 complete model of the system
I use 3 baskets respectively District 1, District 2, District 3
1.Identify products going to District 1
Figure 4 2 Product go to district 1
2 Identify products going to District 2
Figure 4.3 Product go to district 2
3.Identify products going to District 3
Figure 4 3 Product go to district 3
The information of the order is displayed on the user's automatic mode after being recognized and classified
Table 15 Position of end-effector desired to control and position of end-effector measured from reality
STT Position of the desired end-effector
The position of the end- effector measured
4.1.2 Classification based on QR code
Table 16 Classification results based on QR code using reverse kinematics method
STT Time Number of things put on the conveyor
Number of correctly classified objects
1 Morning 30 29 96.67% Wrong 1 time due to not recognizing
2 Morning 30 29 96.67% Wrong 1 time due to not recognizing
3 Morning 30 28 93.33% Wrong 2 time due to not recognizing
Figure 4.5 Order information is displayed on the database
After being classified, the order information will be displayed on the user interface in the data base section We can filter orders based on ID or product name
Comments and ratings
4.2.1 Algorithm to control the robot to pick up and classify objects
4.2.1.1 Robot control by traditional inverse kinematics method
The errors observed in the robot's performance stem from inadequate mechanical design and subpar 3D printing quality, compounded by transmission inaccuracies between the gear on the stepper motor shaft and the joints Acceptable control errors for the x, y, and z axes are approximately 0.3 cm, 0.4 cm, and 0.3 cm, respectively.
The robot's hit rate is quite high, depending on the coordinates of the object detected by the camera and the stability of the conveyor speed
Realizing the control of 3 joints of the robot simultaneously helps to increase product sorting time rather than controlling each joint
4.2.2 Object classification algorithm using computer vision
Tracking objects, finding coordinates in pixels and converting them to the coordinates of the robot, calculates the rotation angle of the joints and sends it to the Arduino
Rounding errors and calculation inaccuracies can accumulate, resulting in significant discrepancies when transmitting the object's center coordinates to the robot.
Poor camera quality can hinder accurate object detection, especially in environments with significant brightness changes, causing robots to misidentify the position of noisy objects.
The frame rate is still slow, so the object tracking is not stable
CONCLUSION
Results achieved
- Build and understand the robotic system model and the equipment components required for a robotic arm model
- Programming the robot arm control by controlling stepper motors with Arduino UNO R3, using driver circuits
- The rate of picking up objects is almost 100% accurate, the recognition ability of the computer vision system is quite high, for QR code recognition is 98.05%
- Calculate coordinates from the coordinate system on the picture frame to the coordinate system of the robot to perform accurate picking
- Programming interface to connect conveyor system and camera with robot system, making operation and observation more intuitive.
Futher work
Learn about the G-code programming language to solve the problem of orbital planning for robots
Change the current motor of the conveyor with a motor with greater torque and speed to optimize the robot's product sorting time
Replace the travel switches with optical sensors to calibrate the robot's joints more accurately
[1]L Crane, "Battle of the billionaires," ed: Elsevier, 2021
[2]T J J S J Soon, "QR code," vol 2008, pp 59-78, 2008
[3]G Bradski and A J D D s j o s t Kaehler, "OpenCV," vol 3, 2000
[4]https://www.shutterstock.com/vi/image-vector/qr-code-vector-attention-realistic- detail-89748973
[5]https://www.researchgate.net/figure/The-history-of-symbols-used-in-barcode-image- recognition_fig15_281190312
[6] https://www.researchgate.net/figure/The-Development-of-QR-Code_fig3_267828104 based Artificial Neural Network (ANN) Method”
[7] http://qrcode.meetheed.com/question7.php
[8] https://kinhtedothi.vn/ha-noi-day-manh-trien-khai-viec-quet-ma-qr-de-quan-ly-thong- tin-nguoi-ra-vao-co-quan-dia-diem-phuc-vu-phong-chong-dich.html
[9] Adrian-Vasile Duka (Dec 2014) “Neural network based inverse kinematics solution for trajectory tracking of a robotic arm” Presented in The 7th International Conference Interdisciplinarity in Engineering Available: www.sciencedirect.com
[10] W S Mada Sanjaya, Dyah Anggraeni, Madinatul Munawwaroh, M Yusuf S
Nurasyidiek, Darmawan Setia Rahayu, Ahmad Samsudin, Ikhsan Purnama Santika, and Endah Kinarya Palupi (2018) “Colored Object Sorting using 5 DoF Robot Arm based Artificial Neural Network (ANN) Method”
[11] Bruno Siciliano, Lorenzo Sciavicco, Luigi Villani, Giuseppe Oriolo (Aug 2008)
“Robotics: Modelling, Planning and Control”
[12] Documents and Tutorials of OpenCV Available: https://docs.opencv.org/4.x/d4/db1/tutorial_documentation.html
[13]TS Nguyễn Đức Thành, Kỹ thuật Robot
[14]PGS TS Nguyễn Trường Thịnh (2016), Kỹ thuật robot, NXB Đại Học Quốc Gia TP HCM
APPENDIX https://drive.google.com/drive/folders/1yTSqBGcILKwvYEkrFL-ER_TWuY5yr4fi?usp=share_link