Nghiên cứu điều khiển rô bốt tay máy di động bám mục tiêu trên cơ sở sử dụng thông tin hình ảnh tt tiếng anh

With the above reasons, the author has chosen the topic: “A research into the control of mobile robot manipulator to track target based on visual information to develop some control al

MINISTRY OF EDUCATION VIETNAM ACADEMY OF

CODE: 9.52.02.16

SUMMARY OF DOCTORAL THESIS IN TECHNICAL

Ha Noi - 2019

The thesis was completed in: Graduate University of Science and Technology (GUST) - Vietnam Academy of Science and Technology

Supervisors: 1 Assoc Prof Pham Thuong Cat

2 Dr Pham Minh Tuan

Referee 1:

Referee 2:

Referee 3:

The thesis will be examined by Examination Board of Vietnam Academy of Science and Technology, at:

The thesis can be found at:

LIST OF WORKS RELATED TO THE THESIS HAS BEEN

PUBLISHED

[1] Le Van Chung (2018), “Robust Visual Tracking Control of Pantilt

-Stereo Camera System”, International Journal of Imaging and

Robotics, Vol 18 (1), pp 45 – 61

[2] Le Van Chung, Pham Thuong Cat (2015), “A new control method for

stereo visual servoing system with pan tilt platform”, Journal of

Computer Science and Cybernetics,Vol 31 (2), pp 107 – 122

[3] Le Van Chung, Pham Thuong Cat (2016), “Optimal tracking a

moving target for integrated mobile robot – pan tilt – stereo camera”,

Advanced Intelligent Mechatronics AIM IEEE Conference, Banff, Canada July 12-15, pp 530 – 535

[4] Le Van Chung, Duong Chinh Cuong (2016), “Design Adaptive-CTC

Controller for Tracking Target used Mobile Robot-pan tilt-stereo camera system”, International Conference on Advances in

Information and Communication Technology, ICTA, Thai Nguyen, Dec 12-14, pp 217 – 227

[5] Le Van Chung, Pham Thuong Cat (2014), “Robust visual tracking control of pan tilt – stereo camera system g i

pp.167-173

[6] hu g h h g t i u h 013), “Phương

pháp điều khiển hệ servo thị giác stereo sử dụng bệ Pan-Tilt -

- g - 382

[7] hu g “Phát triển hệ pan/tilt – nhiều camera bám mục tiêu

di động”, Thai Nguyen University Journal of Science and

Technology, 116(02), tr 41-46

In recent years, there has been a great deal of research on robotics control using visual information But the achieved results still reveal some limitations For example, using a camera on a mobile robot only allows full tracking of the target's information when knowing the target's moving plane As with the use of two cameras on the pan tilt system, but not considering the deterioration of the Jacobian matrix affecting the grip capacity of the system Besides, the mathematical model of a robot is often difficult to achieve the required accuracy because there are many unspecified parameters in the system such as measurement of parameters or coefficient of friction, inertia mo e t etc…, usually changes during operation In addition, it is difficult to optimize the parameters in the robot controllers to achieve the desired accuracy

With the above reasons, the author has chosen the topic:

“A research into the control of mobile robot manipulator to track

target based on visual information to develop some control

algorithms that use image information with many uncertain parameters

The objective of the thesis

The main object of the research is to focus on pan-tilt robots and mobile robots with wheels

Scope of research

Researching methods of controlling pan-tilt tracking moving target using image information from 2 cameras with many uncertain parameters

Developing algorithms for controlling integrated system include mobile robot, pan-tilt and two cameras tracking moving targets

New findings of the doctoral dissertation

1- The application of artificial neural networks in compensate for uncertainties in a pan-tilt system model with two cameras Based on that, the kinematic and dynamic controllers are constructed for the pan-tilt two cameras system to track moving targets with uncertain parameters

2- The thesis has developed a dynamic model for the integrated system that including mobile robot, pan-tilt and stereo cameras Also,

the thesis has built two control methods, sliding mode controller and quadratic perfofmance optimal controller for the above integrated system

The layout of the thesis

Chapter 1: Overview

Chapter 2: Developing controller for pan-tilt stereo camera system

to track the moving target

Chapter 3: Some improvements in controlling servo system to

track the target

Chapter 4: Developing control method for mobile robot

CHAPTER 1 OVERVIEW 1.1 Problem?

In order to control a robot system using two cameras to work better, the problem is:

The first: Developing methods to control pan-tilt systems using image information to track moving targets when there are uncertain parameters

The second: Build a Jacobian matrix image is a square matrix for the system to track the moving target for the better system

The third: Developing some methods to control the integrated system combining a mobile robot with a pan-tilt robot that carries two cameras to track the moving target and be able to move closer to the target in the space

1.2 Overview of controlling robots using visual information

When using a pan-tilt robot with two cameras, another problem poses the degradation of the Jacobi matrix when taking inverse pseudocode When using a camera, the Jacobian matrix of the pan-tilt system - a camera is square and invertible But when using 2 cameras, the Jacobian matrix of the system will be (3x6), the Jacobian matrix of the system will be (3x2) So in transformations we have to take the inverse pseudo cause that is the cause of singularities

Hình 1.2 Some methods of controlling robots

1.3 The research issues of the thesis

- Building kinematic/dynamic model of pan-tilt robot and developing classical control method combining with Neural Network

to get better results including:

- Research and improve to build a square image Jacobian matrix

- Developing advanced control methods for robots when having uncertain parameters

- Building dynamic models for the integrated system that including mobile robots, pan-tilt with 2 cameras and controllers for the above system

- Using Lyapunov stability method and Barbalat's lemma proves stability and Matlab to verify results

Chapter 2 DEVELOPING CONTROLLER FOR PAN TILT STEREO CAMERA TO TRACK MOVING TARGET

In this chapter, the kinematic control algorithm combined with the neural network is built to control the rotation angle of the pan - tilt robot so that the target image is always maintained at the desired

Optimal paramet ers

Optimal output

Control robots using visual information

Combined with neural network

Optimal control, adaptable

Modern methods with neuron network

Classical methods with neuron network

position on the model of the image frames The content of the chapter consists of 4 main parts: building a kinematic model of the pan-tilt stereo camera system, designing control algorithm, verifying and comparing with the controller not associated with Neural network and conclusions

2.1 Kinematic model of stereo visual servoing system with uncertain parameters

2.1.1 Determination of Image Jacobian matrix:

Fig 2 1 Pan-tilt PTU-D48E-Series & camera coordinates

The velocity of t rget’s i ge o c er s:

( )

imag

- v is the velocity vector of the camera system

2.1.2 Kinematic equations of Pan-Tilt platform:

Denote Jrobot the Jacobian of the Pan-Tilt platform, we have:

Fig 2 2 Camera system model

2.1.3 Formulation of stereo visual servoing problem with uncertain parameters:

We calculate the image feature error: ε m md

The kinematic control problem of stereo visual servoing is to find control law q = K ε( ) to control the system track the moving target so the tracking error ε converges to zero

imag

m = J (m)u;v = x = Jrobot(q)q; q = K(ε) 2.2 Control law design

In this chapter, a neural control method is proposed for Pan - Tilt - stereo camera system to track a moving object when there are many uncertainties in the parameters of both camera and Pan-Tilt platform

ε = m - m = Jq + f - m -Kε + f + u (2.21) The structure of the chosen artificial neural network is of RBF type as shown in Fig.6 It has three layers: the input layer includes the three components of error ε, the output layer includes 3 linear neurons and hidden layer contains neurons with Gaussian output function:

2

2exp j j

j

j

c

Fig 2 6 Structure of proposed visual tracking system with many

uncertain parameters

Control Method 1:

The stereo camera system described by the model in Esq (2.9), (2.11), with uncertain parameters controlled by the neural network defined by Eqs (2.22), (2.23) will track moving targets with the error ,ε ε0 if the speed of the Pan-Tilt joints is determined by the Eqs (2.25), (2.26), (2.27) and learning rules (2.28):

Simulation 1: Fixed target

Camera center at the initial time: m(0) = [-40, 30, 0] (pixel);

Image coordinates of the target stand still at m t= [-20, 0, 20] (pixel);

Fig 2 7 Image feature error when

using neural control u , 1 0

Fig 2 8 Image feature error when

no neural control is used u = 0 1 ,

Simulation 2: Moving target in a straight line

Moving target from point A(0m,1.8m,0m) to B(0.3m, 1.8m, 0.5m) on the plane ZCOCXC

-Fig 2 9 Tracking error coordinates when the target moves along a straight

line

along a straight line

Fig 2 11 Tracking error

coordinates when the target

moves along a straight line and

no neural control is used

coordinates when the target moves along a straight line and

no neural control is used

Simulation 3: Moving target in an arc

The target follows the circular arc with center coordinates at the origin O(0, 0), radius r = 1 on the plane ZCOCXC

simulation 3 However, in the first 1/6 arc, the velocity increases steadily with acceleration 1cm/s2, after 3 seconds it will move with the constant speed 3cm/s In 1/6 end of the arc, its moving speed reduces with the deceleration -1cm/s2

arc with changing

along an arc with changing

2.4 Conclusion of chapter 2

This chapter proposes a new control method for the stereo visual servoing system using neural networks with on-line learning rules to compensate for the impacts of uncertain parameters such as inertial torque, Jacobian matrix, friction in the joints, noise effects, etc … he ro osed ethod gu r tees the st bility of the over ll system and eliminates tracking error Control algorithm is highly adaptable and is able to resist the noise impact on the system The global asymptotic stability of the whole system is proved by the

Lyapunov stability theory The simulation results with the uncertainties up to 20% in the case of fixed target, moving target in a straight line or circular arc, show that the tracking error converges to zero These results are consistent with the theoretical

CHAPTER 3 SOME IMPROVEMENTS IN CONTROLLING

SERVO SYSTEM TO TRACK TARGET

As in chapter 2, the author has noticed that in order to control the pan- tilt system with two cameras to track target work well, the control problem has some issues to solve as follows:

- Firstly, it is necessary to build a square image Jacobian matrix

so that performing the inverse and avoid singular points leading to losing grip

- The second is building dynamic controllers in combination with neural networks to compensate for the effects of uncertainty parameters inside the model as well as external noise

- The third is to optimize some parameters in the neural network

to get better outputs

In Part 1 of this chapter, the author built a 3D model for two camera system to obtain the full Jacobian matrix In part 2, the author built the dynamic controller using neural network with optimized parameters, the stability of the system is demonstrated by Lyapunov method and Barballat lemma Part 3 is the simulation results Finally, some conclusions

3.1 3D visual model for eye-in-hand stereo camera system

3.1.2 3D virtual stereo camera model systems

A 3D visual space is built according to the following steps:

First step, from geometrical relations between the target and that

feature images we calculated the coordinates of the target point

c x y z

Second step, a reference coordinate frame with the origin located at the same position as is defined In order to

transform OC to OV, the rotation matrix (Fig 3.2) is used The

v v v

v X Y Z O

c c c

c X Y Z O

v

C R

Fig 3.2 3D visual stereo camera model

associated with stereo cameras Their location on Xv and Zv axes are

far away from Ov the distances λ

Last step, the virtual camera model is combined with 3D visual

camera model to construct a 3D virtual Cartesian space having feature

v v v

s  z 1 z 2 x 1

Jacobian matrix: xs f vJvimgxv (3.9)

v

v v

v

v v v

vimg

z z

y

z z

x

x

z x

1 0

) (

0 1

) (

0 ) ( 1

2 2

2

3.1.3 Avoid singularity

c c c

c X Y Z

O O v X v Y v Z v

The singularity of Jvimg that can be avoided by choosing the

r eter λ such s λ > x x v, z v)

3.1.4 Stereo visual servoing problem with uncertain parameters

0 )

ˆ

xs    f v vimg v RC (3.19)

3.2 Dynamics of robot manipulator with uncertainties

The dynamics of a serial n-link rigid robot with friction, and uncertainty can be written as follows:

τ d q g q q q

3.3.1 Construction of robust controller

The control law is chosen as follows:

3.3.2 Layer construction of RBF neural network

The structure of choosing artificial neural network is a Radial Basis Function (RBFNN) network It has 3 layers

Input layer is vecto  T

s s



Hidden layer computation The hidden layer consists of neurons with

output function calculated by Gaussian form

Output layer The output values of the network are approximate

function f 1

Control Method 2: The image error dynamics (3.32b) of the uncertain pan-tilt – Stereo camera tracking system (3.19), (3.20) will be asymptotically stable with the error if the control torque is chosen by following (3.36), (3.37) and online learning rules (3.38):

(3.36)

(3.37)

Fig 3.6 Structure of proposed visual tracking system

Proof: We choose the candidate Lyapunov function as follows:

1 2

1 ) , (

i

i T i T

4 Simulation results

Simulation 1: Moving target in a straight line from point A

(0m,3m,0m) to B (-0.5m, 3m, -0.3m)