Development of machine vision systems on object classification and measurement for robot manipulation

國國國國國國國國國國國國國國國國國國國國國國國 Development of machine vision systems on object classification and measurement for robot manipulation 國國國國國國國 Graduate student: Ngo Ngoc Vu 國國國國國國國國 國國 Advisor: P

國國國國國國國國國國國國國國國國國國國國國國國

Development of machine vision systems on object classification and

measurement for robot manipulation 國國國國國國國 Graduate student: Ngo Ngoc Vu 國國國國國國國國 國國 Advisor: Prof Quang-Cherng Hsu

國國國國國國國國 國國國國

國 國國國國

A Dissertation Submitted to Department of Mechanical Engineering National Kaohsiung University of Science and Technology

in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy

in Mechanical Engineering

January, 2019 Kaohsiung, Taiwan, Republic of China

國國國國 108 國 1 國

國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國 國國國國國國國國 國國

國國國國國國國國 國國國國國 國國國

國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國 國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國 國國國國國國國國國 6 國國國國國DOF國國國國國國國國國國國國國國國國國國國 國國國國國國國國國國國國國 CMOS 國國國國國國國國國國國國國國國國國國國國 國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國 國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國 國國國國國國國國國國國國國國國國國國國國國國國國國國國國

國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國 國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國 國國國國國國國 X 國 Y 國國國國國國國國國國 0.48 mm 國 0.38 mm 國國國國國

國國 國國-0.34 mm 國-0.43 mm國

國國國國國國國國國國國國國國國國國國國國(DOF)國國國國國國國國國國國 國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國 國國國國國國國國國國(2-D)國國國(3-D)國國國國國國國國國國國國國國國國國國國 國國國國國國國國國國國國國國國國國國國國國國國國國國國國 X 國 Y 國國國國國 國國國國國國 1.29 mm 國 1.12 mm國國國國國國國國國國-1.48 mm 國-0.97 mm國國 國國國國國國國國X國Y 國 Z 國國國國國國國國 0.07國-0.418國-0.063 mm國國國國 國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國

國國國國國國國國國國國國國國國國國國國國國國國國國國國國國國

i

Development of machine vision systems on object classification and

measurement for robot manipulation

Graduate student: Ngo Ngoc Vu Advisor: Prof Quang-Cherng Hsu

Department of Mechanical EngineeringNational Kaohsiung University of Science and Technology

ABSTRACT

This research presents development of machine vision systems on objectclassification and measurement for robot manipulation Firstly, a machine visionsystem for the automatic metal part classification and measurement process isdeveloped under different lighting conditions, and has been applied to the operation

of a robot arm with 6 degrees of freedom (DOF) In order to obtain accuratepositioning information, the overall image is captured by a CMOS camera which ismounted above the working platform The effects of back-lighting and front-lighting conditions to the proposed system were investigated With the front-lighting condition, four different conditions were performed For each condition,global and local threshold operations were used to obtain good image quality Therelationship between the image coordinates and the world coordinates wasdetermined through Zhang’s method, the linear transformation and the quadratictransformation during the calibration process Experimental results show that in aback-lighting environment, the image quality is improved, such that the positions ofthe centers of objects are more accurate than in a front-lighting environment.According to the calibration results, the quadratic transformation is more accurate

than other methods By calculating the calibration deviation using the quadratictransformation, the maximum positive deviation is 0.48 mm and 0.38 mm in the Xand Y directions, respectively The maximum negative deviation is -0.34 mm and-0.43 mm in X and Y directions, respectively The proposed system is effective,robust, and can be valuable to industry

The second, a machine vision system for color object classification andmeasurement process for robot arm with six degree of freedom (DOF) is developed

In order to obtain accurate positioning information, the overall image is captured by

a double camera C615 and a camera C525 which are mounted above the workingplatform The relationship between the image coordinate and the world coordinate

is performed through calibration procedure The quadratic transformation andgeneralized perspective transformation algorithms were used to transformcoordinates in 2-D and 3-D calibration process, respectively According tocalibration results, with 2-D calibration, the positive maximum deviation is 1.29

mm and 1.12 mm in X and Y directions, respectively The negative maximumdeviation is -1.48 mm and -0.97 mm in X and Y directions, respectively With 3-Dcalibration, the deviation is 0.07 mm, -0.418 mm, and -0.063 mm in X, Y and Zdirections, respectively The proposed system can catch the three dimensionalcoordinates of the object and perform classification and assembly automaticoperations by the data from visual recognition system

Keywords: machine vision, robot arm, camera calibration, image analysis, object

recognition, lighting source

The fulfillment of over three years of study at National Kaohsiung University ofScience and Technology (NKUST) has brought me into closer relations with manyenthusiastic people who wholeheartedly devoted their time, energy, and support tohelp me during my studies Therefore, this is my opportunity to acknowledge mygreat debt of thanks to them

I wish to express my thanks and gratitude to my academic supervisor,Prof Dr Quang-Cherng Hsu, for his continuous guidance, valuable advice, andhelpful supports during my studies He has always been supportive of my researchwork and gave me the freedom to fully explore the different research areas relatedwith my study

I wish to acknowledge my deepest thanks to Vietnam Ministry of Education andTaiwan Ministry of Education for giving me a great opportunity, necessaryscholarships to study at NKUST via VEST500 scholarship which is corporationbetween Vietnam Government and Taiwan Government, and many enthusiastichelps during my time in NKUST I am also particularly grateful to Thai NguyenUniversity of Technology (TNUT) provided me unflagging encouragement,continuous helps and support to complete this course

My gratitude also goes to all of the teachers, Dean and staffs of Department ofMechanical Engineering at NKUST for their devoted teaching, great helping andthoughtful serving during my study

I would also like to express my sincere gratitude to all of my colleagues at thePrecision and Nano Engineering Laboratory (PANEL), Department of MechanicalEngineering, NKUST

I want to express my sincere thanks to all my Vietnamese friends in NKUST fortheir helpful sharing and precious helping me over the past time

I also wish to express my gratitude to all those who directly or indirectlyhelped me during my study in NKUST

Finally, my special thanks to my dad Ngo The Long and my mom Vu Thi Hai,

to my older sister Ngo Thi Phuong, to my adorable wife Duong Thi Huong Lien, totwo lovely little daughters Ngo Duong Anh Thu and Ngo Phuong Linh, who are themost motivation for me over years in Taiwan!

國國國國……….i

ABSTRACT……… ii

ACKNOWLEDGMENTS……… iv

CONTENTS……… vi

LIST OF FIGURES……… xii

LIST OF TABLES……… xvi

NOMENCLATURE……… xvii

Chapter 1 Introduction……….1

1.1 Motivation of the research………1

1.2 Scopes of the research……… 8

1.3 Contributions………9

1.4 Organization of the dissertation……… 9

Chapter 2 Theory of image processing and machine vision………12

2.1 Image processing system………12

2.1.1 Basics in image processing……… 13

2.1.1.1 Pixels……… 13

2.1.1.2 Resolution of image……… 13

2.1.1.3 Gray level……… 14

2.1.1.4 Histogram……… 15

2.1.1.5 Image presentation……….16

2.1.1.6 Color models……… 17

2.1.1.7 Neighbors of pixels………18

2.1.2 Morphological image processing ……… 19

2.1.2.1 Erosion operation……… 19

2.1.2.2 Dilation operation……… 19

2.1.2.3 Opening and closing operations ………20

2.1.3 Blob analysis……… 21

2.1.3.1 Goal of blob analysis ………21

2.1.3.2 Feature extraction ………21

2.1.3.3 Steps to perform blob analysis ………22

2.2 Machine vision system ……….22

2.2.1 Lighting design ……… 23

2.2.2 Lens ……… … 25

2.2.3 Image sensors ……… … 27

Chapter 3 Coordinate calibration methods and camera calibration………….30

3.1 Two-Dimensional coordinate calibration……… 30

3.1.1 Linear transformation ……… 30

3.1.2 Quadratic transformation……….32

3.2 Three-Dimensional coordinate calibration ………36

3.2.1 Stereo imaging ……… 36

3.2.2 The generalized perspective transformation ……… 37

3.2.3 The perspective transformation with camera model ……… 39

3.3 Camera calibration……… 44

3.3.1 Intrinsic parameter ………44

3.3.2 Extrinsic parameter ……… 45

3.4 Lens Distortions ……… 46

3.4.1 Radial distortion ……….46

3.4.2 Tangential distortion ……… 47

3.5 Camera calibration using Matlab tool……….48

Chapter 4 Development of a metal part classification and measurement system under different lighting conditions………51

4.1 Materials and Experimental Setup……… 51

4.1.1 Platform Description ……… 51

4.1.2 Robot arm ……… 53

4.2 Research methodology………54

4.2.1 Image processing……….54

4.2.2 Camera calibration……… 55

4.2.3 Coordinate calibration method………57

4.2.4 Resolution of the Measurement System…….……….58

4.2.5 Calculating the Coordinate Deviation ……… 58

4.3 Experimental Procedures………59

4.3.1 Illumination conditions ……… 59

4.3.2 Calibration work ……….59

4.4 Image Segmentation……… 61

4.4.1 Using backlighting ……… 61

4.4.2 Using front-lighting ……… 63

4.5 Algorithm for object classification……… 65

4.6 Algorithm for the bolt head determination ……… 66

4.7 Results and Discussion ……….68

4.7.1 Calibration results…… ……… 68

4.7.2 Coordinate deviation among different lighting conditions ……….70

4.8 Implementation ……… ………73

Chapter 5 Development of a color object recognition and measurement system… 77

5.1 Platform Description……… 77

5.1.1 Experimental setup … ……… 77

5.1.2 The implementation process of the proposed system……… 79

5.2 Calibration work……….80

5.2.1 2-D coordinate calibration using the quadratic transformation 80

5.2.2 3-D calibration coordinate using the perspective transformation 81

5.3 Image analysis and object segmentation……….83

5.4 Algorithm for color object classification………86

5.4.1 Algorithm for solid classification………86

5.4.2 Algorithm for hole classification……….87

5.5 Determination of orientation for triangular holes……… 90

5.6 Results and discussion………93

5.6.1 2-D calibration and spatial position measurement results……… 93

5.6.2 3-D calibration and spatial position measurement results……… 95

5.6.3 Determination process for the first points of triangular holes………….97

5.7 Implementation……… 98

Chapter 6 Conclusions and future works……… 100

6.1 Conclusions……… 100

6.1.1 Conclusion for the classification and measurement system for metal parts 100

6.1.2 Conclusion for the classification and measurement system for color

objects101

6.2 Future works……….102

List of publications………103

References……… 105

Appendices……….113

LIST OF FIGURES

Figure 1.1 Flowchart of dissertation 10

Figure 2.1 Flowchart of image processing system 12

Figure 2.2 Positions and gray-scale values within an image 15

Figure 2.3 Histogram between intensity and frequency 15

Figure 2.4 The conversion from gray-scale image into binary image 16

Figure 2.5 Representation of image 17

Figure 2.6 Relationship between pixels 18

Figure 2.7 Erosion operation 19

Figure 2.8 Dilation operation 20

Figure 2.9 Closing operation 20

Figure 2.10 Result of blob analysis 21

Figure 2.11 Architecture of machine vision system 23

Figure 2.12 Front lighting source 24

Figure 2.13 Back lighting source 24

Figure 2.14 Side lighting source 25

Figure 2.15 Lens model 25

Figure 2.16 Depth of field 26

Figure 2.17 Field of view 27

Figure 2.18 Chip size 29

Figure 3.1 The schematic diagram of a stereo imaging system 36

Figure 3.2 The specific perspective projection model 37

Figure 3.3 Pinhole model 38

Figure 3.4 The generalized perspective projection model 39

Figure 3.5 Converting from object to camera coordinate system 45

Figure 3.6 Radial distortion 46

Figure 3.7 Tangential distortion 47

Figure 3.8 Chessboard pattern 49

Figure 4.1 Structure of experimental system 52

Figure 4.2 Articulated and Cartesian coordinate system 53

Figure 4.3 The Camera Calibration using Matlab 60

Figure 4.4 Calibration board 61

Figure 4.5 Binary thresholding operation 62

Figure 4.6 Classification and determining coordinates of centers of objects 62

Figure 4.7 Setting the global threshold and binary threshold result 64

Figure 4.8 Setting the local threshold 65

Figure 4.9 Result of local threshold process 65

Figure 4.10 Using the closed function with 3 times 65

Figure 4.11 Flowchart of algorithm for object classification process……… 66

Figure 4.12(a) Determining the head of bolts left side 67

Figure 4.12(b) Determining the head of bolts right side 67

Figure 4.13 Result of determining the head of bolts 67

Figure 4.14 The calibration error, using Matlab toolbox 69

Figure 4.15 Diagram of calibration error of linear transformation method ………69

Figure 4.16 Diagram of calibration error of the quadratic transformation ……….70

Figure 4.17 Diagram of deviation percentage of bolts 71

Figure 4.18 Diagram of deviation percentage of washers 72

Figure 4.19 Diagram of deviation percentage of nuts 72

Figure 4.20 Flowchart of the implementation process 74

Figure 4.21 Model of robot arm 74

Figure 4.22 End effector of robot arm 75

Figure 4.23 Experiment result for top view 75

Figure 4.24 Experiment result for front view 76

Figure 5.1 Experimental system 78

Figure 5.2 Flowchart of the implementation process of the proposed system 80

Figure 5.3 2-D calibration board 81

Figure 5.4 3-D calibration pattern 82

Figure 5.5 Determining coordinates of 3-D calibration part on CMM 82

Figure 5.6 Flowchart of algorithm for shape classification process……….87

Figure 5.7 Flowchart of algorithm for hole classification process………88

Figure 5.8 Assembly parts 88

Figure 5.9 Recognition results of assembly parts 89

Figure 5.10 Assembly models 89

Figure 5.11 Recognition results of assembly models 89

Figure 5.12 Scan line to find the first point of triangular holes 91

Figure 5.13 All orientations of triangular holes 91

Figure 5.14 Determination of first point for triangular holes of the first world orientation 91

Figure 5.15 Determination of first point for triangular holes of the second world orientation 92

Figure 5.16 Determination of first point for triangular holes (Left image plane) 92

Figure 5.17 Diagram of calibration error 94

Figure 5.18 The calibration system verification screen 95

Figure 5.19 Calibration parameters and calibration accurate checking 96

Figure 5.20 Determining the orientations of triangular holes 97

Figure 5.21 The robot arm implemented in this work 98

LIST OF TABLES

Table 2 1 The type of the image forming device 28

Table 2 2 Advantages and shortcomings of scan modes 28

Table 4 1 Specifications of the CMOS camera (C910) 52

Table 4 2 Specifications of the Robot arm 54

Table 4 3 Determination of bolt orientation 68

Table 5 1 Specifications of the CMOS camera (C525) 79

Table 5 2 Specifications of the CMOS camera (C615) 79

Table 5 3 Specifications of CMM 83

Table 5 4 The world coordinates of calibration points measured by CMM 83

Table 5 5 Color recognition algorithms 85

Table 5 6 Threshold values using in this study 86

Table 5 7 The world coordinates of shapes 93

Table 5 8 The world coordinates of holes 96

Table 5 9 The world coordinates of first points of triangular 97

NOMENCLATURE

WCS The World Coordinate System

CCS Camera Coordinate System

ICS Image Coordinate System

CCDs Charge Coupled Devices

CMOS Complementary Metal Oxide Semiconductor2-D Two Dimension space

3-D Three Dimension space

CMM Coordinate Measuring Machine

ATOS Advanced Topo-metric Optical Sensor SCARA Selective Compliance Assembly Robot ArmDOF Degrees of Freedom

RGB Red Green Blue

CMYK Cyan, Magenta, Yellow, and Black

HSB Hue, Saturation, and Brightness

PC Personal Computer

DFOV Diagonal Field Of View

J Joint

LSM Least Square Method

DLT Direct Linear Transformation

VB 6.0 Visual Basic 6.0

MIL Matrox Image Library

Re Resolution

BMP Bitmap

ppi Pixel per inch

dpi Dot per inch

bit Binary digit

MP Mega Pixel

Sx Summation of the error square in X direction

Sy Summation of the error square in Y direction

w h The world coordinate system

c h The camera coordinate system

λ The focal length

θ The angle between the x-axis of the camera's image plane and the

X-axis of the world coordinate system

α The angle between the z-axis of the camera's image plane and the Z

axis of the world coordinate

R The rotation matrix

C, G The translation matrix

P The perspective specific transformation matrix

T The extrinsic matrix

M The intrinsic matrix

tij The parameters of extrinsic matrix

 The angle between z and Z

s The scale factor

k1 The first order distortion coefficient

k3 The third order distortion coefficient

LED Light Emitting Diode

 The percentage of deviation

(%)

l j The real size of the objects

x n The coordinates of the center points of objects in x-direction

y n The coordinates of the center points of objects in y-direction

x i The coordinates of the targeted centers of the objects in x-direction

y i The coordinates of the targeted centers of the objects in y-direction

Re(X) The resolutions of the measurement system in the x-direction

Re(Y) The resolutions of the measurement system in the y-direction

Pixel(X) The pixel values between the first point and the last point of the

calibration image in the x-direction

Pixel(Y) The pixel values between the first point and the last point of the

calibration image in the y-direction

X The world coordinates between the first point and the last point in

the x-direction

Y The world coordinates between the first point and the last point in

y-direction

Chapter 1 Introduction

1.1 Motivation of the research

Nowadays, flexible automatic assembly systems are one of the most usefultools for automated manufacturing processes These systems are expected tointegrate machine vision systems for various applications within automationsystems and production lines Therefore, machine vision is considered as a valuablesystem for automation process in industry

During the currently several years, the development of machine vision has beenhelped improving significantly both quality and productivity in manufacturing.Machine vision uses illumination, image processing and blobs analysis to obtainfeatures and positions of objects The machine vision technology is based on thehuman vision to detect objects in the physical world The information is processed

by a personal computer installed image processing software, and then the worldcoordinates of objects can be determined

Machine vision method is often applied in a lot of fields, such as: qualityinspection [1, 2, 3], optical measurement systems [4, 5], industrial automation [6],electronic semiconductors [7, 8], medical [9, 10], defect inspection [11, 12, 13, 14],etc Among applications of machine vision in manufacturing and industry, Dworkin,

S B and Nye, T J [15] investigated an algorithm of machine vision to measureparts in hot forming process In this study, to acquire images with visible light,charge coupled device (CCD) cameras was used After getting the images,thresholding operations were performed Finally, the images were analyzed andprocessed Derganc, J et al [16] presented a machine vision system for determining

automatic measurement eccentricity of bearings The proposed system canobtain 100%

inspection of bearings and it is a valuable system for testing quality of thehigh-end-product Shuxia, G et al [17] presented a machine vision system for acutting device’s diameter and the maximum rotating diameter of a mini-millingmachine was measured A CCD camera which is Sony XC-55 was used forcapturing images In this study, to identify the edges of mini milling cutter, asub-pixel threshold segmentation algorithm was applied using gray level histogram.These features are robustly and accurately determined by the Hough transform andlinear regression Their system was fast and accurate Hsu, Q C et al [18]presented an automatic optical inspection system (AOI) for defect detection ofdental floss picks In this study, five webcams and seven sets of white LED modules

as lighting to support the vision environment were used to perform imageacquisition With this system, the dental floss picks were measured and thenclassified

Furthermore, applications of machine vision have also used for non-contactprecision measurement Precision measurement is very important in manufacturingtechnology using machine vision Traditional measurement methods include contactand non-contact measurement With contact measurement method, CMM(Coordinate Measuring Machine) is used normally This equipment has very highprecision and to be widely used in research institutes, factories and universities.With non-contact measurement method such as White Light Interference method,likes ATOS (Advanced Topo-metric Optical Sensor) is a selection with highprecision and to be used for parts which has complex geometries However, bothCMM and ATOS are very expensive and take a long time to measure Today, withdevelopment of sciences and technology, automated optical measurement method

using machine vision technology is a suitable selection for precision measurementwith low cost and high performance In a study of Ngo, N V et al [19], a machine

vision system was developed for non-contact measurement in three dimensional(3-D) sizes This study used a 6-point calibration algorithm to transform the imagecoordinate into the world coordinate The image data from a double CMOS wassent to a computer to measure the sizes With this system, sizes of part are fast andaccurately determined Specially, this system is not complex and it is easy foroperation The study of Ali M.H et al [20] is very useful for improving precisionmeasurement The proposed system was to measure gear profile to replacetraditional methods which may face danger in measurement process Besides, theexisting methods are either time consuming or expensive Experimental results ofthe proposed system were compared with the existing systems These resultsshowed that their method has great advantages over existing methods in practicalapplication In a study of Rejc J et al [21], the authors measured dimension of aprotector using automated visual inspection system They used a linear and apolynomial approximation for defining edges of selected structures of the protector.Pixel to metric unit transformation was performed by using a higher orderpolynomial approximation The measurement accuracy of the proposed system is inthe range of ±0.02 mm The measurement time is less than a second However, itcannot replace the current measurement system It can be only use in the companytesting laboratory In a study of Martínez, S et al [22], they presented a qualityinspection system for machined metal parts using an image fusion technique Themachine vision system can perform the detection of flaws on textured surfaces Thissystem works effectively with a low value of false rejections In a study of Abhinesh

B et al [23], an automatic inspection algorithm of lead-frames to identify stampingdefects was presented Blobs analysis using morphological closing and imagesubtraction for detecting defects of the stamping was performed The vision-based

system was integrated with a conveyor The experimental results showed that thesystem can detect defects of the stamping on the lead-frames success detecting rate

of this system is close to 98.7% In a research of Quinn, M.K et al [24], anautomated inspection system for pressure-sensitive paint (PSP) was developedusing machine vision camera systems This research presented relevant imagingcharacteristics and the applicability of such imaging technology for PSP Theexperimental results show that this machine vision system has advanced to apply forquantitative measurements of a PSP system In a study of Zhang, X et al [25] ahigh precision quality inspection system for steel bars using machine vision waspresented In this study, the sub-pixel boundary location method (SPBLM) and faststitch method (FSM) were proposed Steel bar diameter, spacing, and quantity weredetected The results show that this system has a high accuracy for measuringdiameter and spacing

Among applications of machine vision in agriculture field, a research of Cubero,

S et al [26] has been developed a machine vision system to classify the objects thatreach the line into four categories, finding broken fruit attending, basically, to theshape of the fruit The image acquisition system used two cameras and a computerand back-lighting source By extracting morphological features, pieces of skin andother raw material were automatically identified The classified results of thesystem based on machine vision obtained 93.2% of sound segments In study ofJarimopas, B and Jaisin, N [27], they developed a machine vision experimentalsorting system for sweet tamarind pods based on image processing techniques Thesorting system used a CCD camera, a TV card, microcontrollers, sensors, and amicrocomputer The proposed vision sorting system could separate Sitong tamarindpods at an average sorting efficiency of 89.8%

In recent years, machine vision is also applied for robots Various types ofimage processing hardware have been studied to increase the performance of visionsystem In particular, a machine vision feedback procedure can be used to guide anobject held in a robot manipulator Movement instructions of a robot is programmedwith a general set In a research of Blasco, J et al [28], they installed two visionsystems on a machine One of them was to identify weeds and send theircoordinates to control of the robot arm, and the other for calibrating inertialperturbations induced in the position of the gripper These systems weredemonstrated to be capable of properly locating 84% of weeds and 99% of lettuces

Di Fulvio, G et al [29] proposed a stereoscopic vision system for conductingdimensional measurements in an industrial robotics application In their research, acamera mounted on a six-axis robotic manipulator was able to calculate the worldcoordinates of the target objects Experimental results have shown that the accuracy

of the measurements were strongly influenced by the accuracy of the calibration.Phansak, N et al [30] developed a flexible automatic assembly system for robotarm (SCARA - Selective Compliance Assembly Robot Arm) using machine vision

A prototype of the flexible automatic pick and place assembly system weredesigned With this vision system, the robot could also pick each correct assemblypart and place it into the assembly location perfectly In a research of Tsarouchi, P

et al [31], to detect of randomly placed objects for robotic handling, the authorsproposed a 2-D machine vision system including a high resolution camera Thiscamera was mounted on a fixed position from the conveyor In their study, Matlabsoftware was used for image processing, determination of objects’ poses,calculation and transformation of the 2-D coordinates Among those applications ofmachine vision for robot arm, Iscimen, B et al [32] developed a smart robot arm

motion using machine vision This system could detect objects from imagesautomatically and implement given tasks Specially, artificial neural network wasused recognize objects and 98.30 % overall accuracy of recognition was achieved.

In addition, Rai, N et al [33] used a computer vision system for detection andautonomous object manipulation placed randomly on a target surface and controls

an educational robotic arm with 6 degree of freedom (DOF) to pick it up and place

it to target location The system applied Centre-of-Mass based computation,filtering and color segmentation algorithm to determine the target and the positionfor movement of the robotic arm In the other research, Tsarouchi, P et al [34] usedalso a vision system to recognize objects whose location are random in workspacefor a robot This research proposed a method to calculate the coordinates of object

in the world system from images captured by camera It was performed using a tool

of Matlab software and shaver handles’ feeding was used as a case study Juang, J

et al [35] presented an application of optical word recognition and fuzzy control to

a smartphone automatic test system The proposed system consists of a robot armand two cameras Two cameras is to capture and recognize the commands and thewords on the monitor of control panel and the tested smartphone Objectcoordinates provided by the camera which is used to recognize the testedsmartphone will be sent to robot arm, and then robot arm moves to the targetpositions and presses the desired buttons

Within a machine vision system, calibration work is very important Itsperformance depends on accuracy of calibration process Shin, K Y et al [36]proposed a new calibration method for a multi-camera setup using a wand danceprocedure With this method, the calibration for the 3-D frame parameters were firstestimated using the Direct Linear Transformation (DLT) method Then, the

parameters estimated in the first step were improved iteratively through a nonlinearoptimization using the wand dance procedure Experimental results showed that theproposed calibration method holds great promise to increase the overall accuracy ofthe DLT algorithm and provide better user convenience Y Ji et al [37] proposed anautomatic calibration method for camera sensor network using a 3-D texture mapinformation of environment given in this research A new image descriptor waspresented It based on quantized line parameters in the Hough space (QLH) toperform a particle filter-based matching process between line features and the 3-Dtexture map information Deng, Li, et al [38] investigated a relationship model forcamera calibration In this study, the geometric parameters and the lens distortioneffect of the camera were taken The study results show that the proposed algorithmhad the ability to avoid local optimums, and could complete the visual identificationtasks accurately You, X et al [39] presented a two vanishing point calibrationmethod for a roadside camera This method is accurate and practical To improvethe accuracy of the calibration results, they also employed the multiple observations

of the vanishing points and presented the other calibration method which is dynamic

to calibrate the camera’s parameters This method is suitable for outdoors Ivo S et

al [40] presented a novel structured light pattern for 3-D structured light scanner toobtain accurate anthropometric body segment parameters in a fast and reliablemanner Volumetric parameters of both an artificial object and a human bodysegment were obtained through 3-D scanning

Beyond the applications mentioned, machine vision provides innovativesolutions in the direction of industrial automation [41] and benefits in industrialapplications These applications include manufacturing the electronic components[42], productions for textile [43], metal products [44], glass manufacturing [45],

machined products [46], products of printing [47] and automatic optical inspectionfor quality of granite [48], products of integrated circuits (IC) advancedmanufacturing technology [49] and many others Machine vision technologyimproves both productivity and quality management and provides many advantages

to industries

In summary of the literature survey, we can see that machine vision has applied

in many fields, especially in the area of industrial automation With machine visiontechnology, we can obtain an effective solution towards improving both theflexibility of performance and the accuracy of measurement, when applying amachine vision for various applications Therefore, in this study, an attempt hasbeen produced for development of object classification and measurement systemsusing machine vision This proposed systems use CMOS cameras with highresolution, personal computers with Visual Basic 6.0 and MIL - Matrox ImagingLibrary Coordinate calibration algorithms were investigated and compared withother traditional algorithms

1.2 Scopes of the research

Scopes of this research include two mainly objects The first object studies onapplication of machine vision for metal part classification and measurement system

in 2-D spatial space under different lighting conditions for robot manipulation usingquadratic transformation algorithm in calibration process The second object studies

on application of machine vision for the color object classification and measurementsystem in space using different calibration algorithms including the quadratictransformation for 2-D and the perspective transformation for 3-D

These vision systems use CMOS cameras with high resolution, personalcomputer with Visual Basic 6.0 and MIL for image processing, extracting an

In this dissertation, the research procedure is organized as shown in Figure

1.1

9

object’s features within an image

1.3 Contributions

Contributions of this study can described as follows:

The classification and measurement system for metal parts including bolts, nutsand washers in 2-D coordinate system was developed successfully under differentlighting conditions This system can work in different lighting conditions which isthe same with industrial environments and can affect the recognition andmeasurement of target objects For this system, the quadratic transformation which

is plane-to-plane transformation was applied for calibration process This proposedcalibration method is accurate and simple to use in industrial environment Toassess accuracy of the proposed system, camera calibration using Camera Calibratortool of Matlab and traditional calibration methods of Tsai and Zhang are alsoinvestigated and compared

Along with the above vision system, a classification and measurement systemfor color objects in 3-D coordinate system was also presented This system canrecognize and classify exactly different color objects such as sphere, square cubic,hexagonal cubic and triangle cubic which are in red, green, blue and yellow color in2-D physical space Besides, circle, triangle and square holes in 3-D physical spacewere also recognized and classified by this system Both the perspectivetransformation and plane-to-plane transformation algorithms were used forcalibration processes

These systems were used for robot manipulation in determination of positionand shape of 2-D and 3-D objects in the physical world

1.4 Organization of dissertation

Chapter 1 presents the brief content and background of the research The

motivation, scope of research work, purposes and contributions of the project arealso discussed

Chapter 2 describes a theory of image processing and machine vision such as main

components of machine vision, and the basic concepts

Chapter 3 presents coordinate calibration methods which is important in machine

vision The generalized perspective transformation and plane-to-planetransformation algorithms which are to determine the relationship between imagecoordinate system and the world coordinate system were shown Besides, cameracalibration work is also presented

Figure 1.1 Flowchart of dissertation

Chapter 4 gives development of a metal part classification and measurement

system under different lighting conditions, and has been applied to robotmanipulation

Chapter 5 presents development for an automatic classification and measurement

process for color objects during working processes of robot arm with six degree offreedom (DOF)

Chapter 6 shows conclusions of this research and suggestions for future works.

Chapter 2 Theory of image processing and machine vision

The purpose of this chapter is to introduce several concepts related to digital imageand machine vision This chapter is divided into two main sections Section 2.1briefly summarizes the basics in digital image and image processing Section 2.2discusses components of a machine vision system, lights, lens and image sensors

2.1 Image processing system

Image processing techniques have been developed tremendously during thepast six decades It has become increasingly importance because digital devices,relating automated vision detection and inspection and object recognition, arewidely used today [50] Typical image processing systems usually include thefollowing components:

Figure 2.1 Components of an image processing system

a Image acquisition: Image acquisition is the first stage of any image processingsystem Images can be received through color camera or black and white camera.Normally, these images are digital images

b Image preprocessing: After image acquisition stage, images are noise and theircontrast is low Therefore, images need to be processed to enhance quality usingimage preprocessing

c Image segmentation: Image segmentation operation subdivides an image intoconstituent regions or objects to analysis Segmentation is one of the most difficulttasks in image processing technique

d Image representation: After an image has been segmented into regions, the resultsaggregated of segmented pixels are usually represented and described in a suitableform for further computer processing

e Image recognition and interpretation: Image recognition and interpretation are theimage determination processes based on preparation with stored images before

2.1.1 Basics of image processing

2.1.1.1 Pixels

In digital imaging, a pixel (picture element) is a physical point in a raster image

It is the smallest controllable element of a picture represented on the screen Eachpixel is a sample of an original image The intensity of each pixel is variable Incolor imaging systems, a color is typically represented by three or four componentintensities such as red, green, and blue, or cyan, magenta, yellow, and black

2.1.1.2 Resolution of image

Resolution refers to the number of pixels in a unit length within an image,which is usually calculated with several pixels per inch Resolution can be dividedinto the following three categories:

a Image resolution [53] is a number of pixels contained in a unit of length within apicture or an image

For example: 300 ppi means 300 pixels per inch.

b Display resolution: It indicates the range which can display

For example, if the size of the Windows Desktop is set from (640× 480) to(800× 600), it can accommodate more things, things displayed will become smaller

c Printer resolution: It indicates the print quality of a printer

For example: 720 dpi means that 720 dots can be ejected in one inch

Image resolution affects the print quality and it does not affect the display on the screen

The printer resolution does not affect the print size The print size is determined

by the resolution of the image, as shown in Equation 2.1 Increasing printerresolution improves the print quality

is white color

Figure 2.2 Positions and gray-scale values within an image

2.1.1.4 Histogram

Figure 2.3 Histogram between intensity and frequency

The characteristic distribution of an image is the distribution of frequencywhich is appearance of a digital set group, and the gray distribution is thedistribution of the number of pixels appearing at various gray levels The horizontalaxis of the grayscale distribution is the grayscale value (0255), and the vertical