Industrial Robotics (Theory, Modelling and Control) - P9 pdf

The SELECT operation thus specifies which robot receives motion instructions for example, DRIVE to move the vision belt in program "drive", or MOVES to move the SCARA in program "track"

3.2 Dynamically altering belt locations for collision-free object picking on-the-fly

The three previously discussed user tasks, when runnable and selected by the system's task scheduler, attach respectively the robots:

• Task 1: robot 1 – a SCARA-type robot Cobra 600TT was considered;

• Task 2 and 3: robot 2 – the "vision conveyor belt" of a flexible feeding

sys-tem

In multiple-robot systems like the one for conveyor tracking, SELECT robot operations select the robot with which the current task must communicate

The SELECT operation thus specifies which robot receives motion instructions

(for example, DRIVE to move the vision belt in program "drive", or MOVES to move the SCARA in program "track") and returns robot-related information (for

example, for the HERE function accessing the current vision belt location in

program "read").

Program "track" executing in task 1 has two distinct timing aspects, which

cor-respond to the partitioning of its related activities in STAGE 1 and STAGE 2

Thus, during STAGE 1, "track" waits first the occurrence of the on-off

transi-tion of the input signal generated by a photocell, indicating that an object passed over the sensor and will enter the field of view of the camera Then, af-ter waiting for a period of time (experimentally set up function of the belt's

speed), "track" commands the vision system to acquire an image, identify an

object of interest and locate it

During STAGE 2, "track" alters continuously, once each major 16 millisecond

system cycle, the target location of the end-effector – part.loc (computed by sion) by composing the following relative transformations:

vi-SET part.loc = to.cam[1]:vis.loc:grip.part

where grip.part is the learned grasping transformation for the class of objects

of interest The updating of the end-effector target location for grasping one moving object uses the command ALTER()Dx,Dy,Dz,Rx,Ry,Rz, which specifies the magnitude of the real-time path modification that is to be applied to the robot path during the next trajectory computation This operation is executed

by "track" in task 1 that is controlling the SCARA robot in alter mode, enabled

by the ALTON command When alter mode is enabled, this instruction should

be executed once during each trajectory cycle If ALTER is executed more ten, only the last set of values defined during each cycle will be used The ar-guments have the meaning:

of-• Dx,Dy,Dz: optional real values, variables or expressions that define the translations respectively along the X,Y,Z axes;

• Rx,Ry,Rz: optional real values, variables or expressions that define the tions respectively about the X,Y,Z axes

rota-The ALTON mode operation enables real-time path-modification mode (alter

mode), and specifies the way in which ALTER coordinate information will be interpreted The value of the argument mode is interpreted as a sequence of two bit flags:

Bit 1 (LSB): If this bit is set, coordinate values specified by subsequent

ALTER instructions are interpreted as incremental and are

accumu-lated If this bit is clear, each set of coordinate values is interpreted as

the total (non cumulative) correction to be applied The program "read"

executing in task 3 provides at each major cycle the updated position information y_off of the robot 2 – the vision belt along its (unique) Y

motion axis, by subtracting from the current contents pos the belt's offset position offset at the time the object was located by vision: y_off

= pos – offset The SCARA's target location will be altered therefore, in

non cumulative mode, with y_off

Bit 2 (MSB): If this bit is set, coordinate values specified by the subsequent

ALTER instructions are interpreted to be in the World coordinate tem, to be preferred for belt tracking problems

sys-It is assumed that the axis of the vision belt is parallel to the Y0 robot axis in its base Also, it is considered that, following the belt calibrating procedure de-

scribed in Section 2.1, the coefficient pulse.to.mm, expressing the ratio between

one belt encoder pulse and one millimetre, is known

The repeated updating of the end-effector location by altering the part.loc

ob-ject-grasping location proceeds in task 1 by "track" execution, until motion

stops at the (dynamically re-) planned grasping location, when the object will

be picked-on-the-fly (Borangiu, 2006) This stopping decision is taken by

"track" by using the STATE (select) function, which returns information about

the state of the robot 1 selected by the task 1 executing the ALTER loop The argument select defines the category of state information returned For the pre-sent tracking software, the data interpreted is "Motion stopped at planned lo-cation", as in the example below:

Example 3:

The next example shows how the STATE function is used to stop the ous updating of the end-effector's target location by altering every major cycle the position along the Y axis The altering loop will be exit when motion stopped

continu-at planned loccontinu-ation, i.e when the robot's gripper is in the desired picking tion relative to the moving part

posi-ALTON () 2 ;Enable altering mode

WHILE STATE(2)<>2 DO ;While the robot is far from the moving

;location

ALTER () ,-pulse.to.mm*y_off ;Continuously alter the

;trajectory generator a chance to execute END

object-placing loc

After alter mode terminates, the robot is left at a final location that reflects both the destination of the last robot motion and the total ALTER correction that was applied

Program "drive" executing in task 2 has a unique timing aspect in both

STAGES 1 and 2: when activated by the main program, it issues continuously motion commands DRIVE joint,change,speed, for the individual joint number

1 of robot 2 – the vision belt (changes in position are 100 mm; several speeds were tried)

Program "read" executing in task 3 evaluates the current motion of robot 2 –

the vision belt along its single axis, in two different timing modes During STAGE 1, upon receiving from task 1 the info that an object was recognised, it computes the belt's offset, reads the current robot 2 location and extracts the

component along the Y axis This invariant offset component, read when the

object was successfully located and the grasping authorized as collision-free, will be further used in STAGE 2 to estimate the non cumulative updates of the

y_off motion, to alter the SCARA's target location along the Y axis.

The cooperation between the tasks on which run the "track", "drive" and

"read" programs is shown in Fig 8

Figure 8 Cooperation between the tasks of the belt tracking control problem

4 Authorizing collision-free grasping by fingerprint models

Random scene foregrounds, as the conveyor belt, may need to be faced in botic tasks Depending on the parts shape and on their dimension along z+,grasping models Gs_m are off line trained for prototypes representing object

ro-classes However, if there is the slightest uncertainty about the risk of collision between the gripper and parts on the belt – touching or close one relative to the

other –, then extended grasping models EG_m ={Gs_m, FGP_m} must be created

by the adding the gripper's fingerprint model FGP_m to effectively authorize part access only after clear grip tests at run time

Definition.

A multiple fingerprint model MFGP_m(G, O) ={FGP_m1(G, O), ,FGP_m k (G, O)} for a

p-fingered gripper G and a class of objects O describes the shape, location and interpretation of k sets of p projections of the gripper's fingerprints onto the

image plane x , vis y vis for the corresponding k grasping styles Gs_m i,i=1, ,k of

O -class instances A FGP_m i (G, O) model has the following parameter structure:

• finger_shape(G)=number,shape i,size i,i=1, , p, expresses the shape of the

gripper in terms of its number p of fingers, the shape and dimensions of each finger Rectangular-shaped fingers are considered; their size is given

"width" and "height"

• fingers_location(G, O)={x ci(O),y ci(O),rz i(O)},i=1, ,p,indicates the relative

loca-tion of each finger with respect to the object's centre of mass and minimum ertia axis (MIA) At training time, this description is created for the object's model, and its updating will be performed at run time by the vision system for any recognized instance of the prototype

in-• fingers_viewing(G, pose_context i,i=1, p) indicating the way how ble" fingers are to be treated; fingers are "invisible" if they are outside the field of view

"invisi-• grip=1, ,k are the k gripper-object Gs_m(G, O) distinct grasping models a priori trained, as possible alternatives to face at run time foreground context situations

A collision-free grasping transformation CF(Gs_m i,O) will be selected at run time

from one of the k grip parameters, after checking that all pixels belonging to

i

FGP_m (the projection of the gripper's fingerprints onto the image

planex , vis y vis, in the O -grasping location) cover only background-coloured

pix-els To provide a secure, collision-free access to objects, the following vision sequence must be executed:

robot-1 Training k sets of parameters of the multiple fingerprints model

O) MFGP_m(G, for G and object class O , relative to the k learned grasping

stylesGs_m i(G, O),i=1, ,k

2 Installing the multiple fingerprint model MFGP_m(G, defining the shape, O)

position and interpretation (viewing) of the robot gripper for clear-grip tests,

by including the model parameters in a data base available at run time This must be done at the start of application programs prior to any image acquisi-tion and object locating

3 Automatically performing the clear-grip test whenever a prototype is recognized

and located at run time, and grips FGP_m i,i=1, k have been a priori fined for it

de-4 On line call of the grasping parameters trained in the Gs_m i(G, O) model, which corresponds to the first grip FGP_m found to be clear i

The first step in this robot-vision sequence prepares off line the data allowing to position at run time two Windows Region of Interest (WROI) around the cur-rent object, invariant to its visually computed location, corresponding to the

two gripper fingerprints This data refers to the size, position and orientation of the gripper's fingerprints, and is based on:

• the number and dimensions of the gripper's fingers: 2-parallel fingered grippers

were considered, each one having a rectangular shape of dimensions

g

g ht

wd , ;

• the grasping location of the fingers relative to the class model of objects of interest

This last information is obtained by learning any grasping transformation for a

class of objects (e.g "LA"), and is described by help of Fig 9 The following frames and relative transformations are considered:

• Frames: (x0,y0): in the robot's base (world); (x vis,y vis): attached to the image plane; (x g,y g): attached to the gripper in its end-point T; (x loc,y loc): default object-attached frame, x loc ≡MIA (the part's minimum inertia axis);

• Relative transformations: to.cam[cam]: describes, for the given camera, the

lo-cation of the vision frame with respect to the robot's base frame; vis.loc:

de-scribes the location of the object-attached frame with respect to the vision frame; vis.obj: describes the location of the object-attached frame with respect

to the vision frame; pt.rob: describes the location of the gripper frame with

re-spect to the robot frame; pt.vis: describes the location of the gripper frame with respect to the vision frame

As a result of this learning stage, which uses vision and the robot's joint

encod-ers as measuring devices, a grasping model

{d.cg alpha z_off rz_off}

) ( G,"LA" = , , ,

centre of mass C and minimum inertia axis MIA (C and MIA are also available

at runtime):

))Gdir(C,,(_

G),dist(T,_

G)),dir(C,MIA,(G),

els, which means that no other object exists close to the area where the gripper's fingers will be positioned by the current robot motion command A negative result of this test will not authorize the grasping of the object

For the test purpose, two WROIs are placed in the image plane, exactly over the areas occupied by the projections of the gripper's fingerprints in the image plane for the desired, object-relative grasping location computed from

) GP_m(G, LA"" ; the position (C) and orientation (MIA) of the recognized object must be available From the invariant, part-related data:

cg d ht wd wd off

rz

alpha, , LA, g, g, , there will be first computed at run time the current coordinates xG, yG of the point G, and the current orientation angle

grasp

angle. of the gripper slide axis relative to the vision frame

Figure 9 Frames and relative transformations used to teach the GP_m(G, LA"" ) rameters

pa-The part's orientation angle.aim=∠(MIA,x vis) returned by vision is added to the learned alpha

alpha angle.aim

x

Once the part located, the coordinates xC, yC of its gravity centre C are

avail-able from vision Using them and beta, the coordinates xG, yG of the G are puted as follows:

com-)sin(

.),

Now, the value ofangle.grasp=∠(x g,x vis), for the object's current orientation and accounting for rz off from the desired, learned grasping model, is obtained from angle.grasp=beta+rz.off

Two image areas, corresponding to the projections of the two fingerprints on the image plane, are next specified using two WROI operations Using the ge-

ometry data from Fig 9, and denoting by dg the offset between the end-tip

point projection G, and the fingerprints centres CWi, ∀i=1,2,

2/2

).cos(

G

cw1 x dg angle grasp

x = − ⋅ ;xcw2 =xG +dg⋅cos(angle.grasp)

(7))

.sin(

G

cw1 y dg angle grasp

y = − ⋅ ; ycw2 = yG +dg⋅sin(angle.grasp)

The type of image statistics is returned as the total number of non-zero ground) pixels found in each one of the two windows, superposed onto the ar-eas covered by the fingerprints projections in the image plane, around the ob-ject The clear grip test checks these values returned by the two WROI-generating operations, corresponding to the number of background pixels not occupied by other objects close to the current one (counted exactly in the grip-per's fingerprint projection areas), against the total number of pixels corre-sponding to the surfaces of the rectangle fingerprints If the difference between

(back-the compared values is less than an imposed error err for both fingerprints –

windows, the grasping is authorized:

If ar1[ ]4 − ar fngprt ≤errANDar2[ ]4 − ar fngprt ≤err,

clear grip of object is authorized; proceed object tracking by ously

continu-altering its target location on the vision belt, until robot motion is pleted

5 Conclusion

The robot motion control algorithms with guidance vision for tracking and grasping objects moving on conveyor belts, modelled with belt variables and 1-d.o.f robotic device, have been tested on a robot-vision system composed from a Cobra 600TT manipulator, a C40 robot controller equipped with EVI vi-sion processor from Adept Technology, a parallel two-fingered RIP6.2 gripper from CCMOP, a "large-format" stationary camera (1024x1024 pixels) down looking at the conveyor belt, and a GEL-209 magnetic encoder with 1024 pulses per revolution from Leonard Bauer The encoder’s output is fed to one

of the EJI cards of the robot controller, the belt conveyor being "seen" as an ternal device

ex-Image acquisition used strobe light in synchronous mode to avoid the

acquisi-tion of blurred images for objects moving on the conveyor belt The strobe light is triggered each time an image acquisition and processing operation is executed at runtime Image acquisitions are synchronised with external events

of the type: "a part has completely entered the belt window"; because these events generate on-off photocell signals, they trigger the fast digital-interrupt line of the

robot controller to which the photocell is physically connected Hence, the VPICTURE operations always wait on interrupt signals, which significantly improve the response time at external events Because a fast line was used, the most unfavourable delay between the triggering of this line and the request for image acquisition is of only 0.2 milliseconds

The effects of this most unfavourable 0.2 milliseconds time delay upon the integrity

of object images have been analysed and tested for two modes of strobe light triggering:

• Asynchronous triggering with respect to the read cycle of the video camera, i.e as soon as an image acquisition request appears For a 51.2 cm width of the image field, and a line resolution of 512 pixels, the pixel width is of 1

mm For a 2.5 m/sec high-speed motion of objects on the conveyor belt the most unfavourable delay of 0.2 milliseconds corresponds to a displacement

of only one pixel (and hence one object-pixel might disappear during the

dist travel above defined), as:

(0.0002 sec) * (2500 mm/sec) / (1 mm/pixel) = 0.5 pixels

• Synchronous triggering with respect to the read cycle of the camera, ing a variable time delay between the image acquisition request and the strobe light triggering The most unfavourable delay was in this case 16.7 milliseconds, which may cause, for the same image field and belt speed a potential disappearance of 41.75 pixels from the camera's field of view

induc-(downstream the dwnstr_lim limit of the belt window)

Consequently, the bigger are the dimensions of the parts travelling on the veyor belt, the higher is the risk of disappearance of pixels situated in down-stream areas Fig 10 shows a statistics about the sum of:

con-• visual locating errors: errors in object locating relative to the image frame

)

,

(x vis y vis ; consequently, the request for motion planning will then not be issued;

• motion planning errors: errors in the robot's destinations evaluated during

motion planning as being downstream downstr_lim, and hence not

author-ised,

function of the object's dimension (length long_max.obj along the minimal inertia axis) and of the belt speed (four high speed values have been considered: 0.5

m/sec, 1 m/sec, 2 m/sec and 3 m/sec)

As can be observed, at the very high motion speed of 3 m/sec, for parts longer than 35 cm there was registered a percentage of more than 16% of unsuccessful object locating and of more than 7% of missed planning of robot destinations (which are outside the CBW) for visually located parts, from a total number of

250 experiments

The clear grip check method presented above was implemented in the V+ gramming environment with AVI vision extension, and tested on the same ro-bot vision platform containing an Adept Cobra 600TT SCARA-type manipula-tor, a 3-belt flexible feeding system Adept FlexFeeder 250 and a stationary, down looking matrix camera Panasonic GP MF650 inspecting the vision belt The vision belt on which parts were travelling and presented to the camera was positioned for a convenient robot access within a window of 460 mm

pro-Experiments for collision-free part access on randomly populated conveyor belt have been carried out at several speed values of the transportation belt, in the range from 5 to 180 mm/sec Table 1 shows the correspondence between the belt speeds and the maximum time intervals from the visual detection of a part and its collision-free grasping upon checking [#] sets of pre taught grasping modelsGs_m i(G, O),i=1, ,#

0 4 8 12 16 20 Object locating errors [%]

long_max.obj [cm]

0.5 m/sec 1m/sec 2m/sec 3m/sec

0 4 8 12 16

20 planning errors [%]

long_max.obj [cm]

0.5 m/sec 1m/sec 2m/sec 3m/sec

Figure 10 Error statistics for visual object locating and robot motion planning

Belt speed [mm/sec] 5 10 30 50 100 180 Grasping time (max) [sec] 1.4 1.6 1.9 2.0 2.3 2.5 Clear grips checked[#] 4 4 4 4 2 1 Table 1 Corespondance between belt speed and collision-free part grasping time

6 References

Adept Technology Inc (2001) AdeptVision User's Guide Version 14.0, Technical

Publi-cations, Part Number 00964-03300, Rev B, San Jose, CA

Borangiu, Th & Dupas, M (2001) Robot – Vision Mise en œuvre en V+, Romanian

Academy Press & AGIR Press, Bucharest

Borangiu, Th (2002) Visual Conveyor Tracking for "Pick-On-The-Fly" Robot Motion

Control, Proc of the IEEE Conf Advanced Motion Control AMC'02, pp 317-322,

Maribor

Borangiu, Th (2004) Intelligent Image Processing in Robotics and Manufacturing,

Roma-nian Academy Press, ISBN 973-27-1103-5, Bucarest

Borangiu, Th & Kopacek, P (2004) Proceedings Volume from the IFAC Workshop

Intelli-gent Assembly and Disassembly - IAD'03 Bucharest, October 9-11, 2003, Elsevier Science, Pergamon Press, Oxford, UK

Borangiu, Th (2005) Guidance Vision for Robots and Part Inspection, Proceedings

vol-ume of the 14th Int Conf Robotics in Alpe-Adria-Danube Region RAAD'05, pp

27-54, ISBN 973-718-241-3, May 2005, Bucharest

Borangiu, Th.; Manu, M.; Anton, F.-D.; Tunaru, S & Dogar, A (2006) High-speed

Ro-bot Motion Control under Visual Guidance, 12th International Power Electronics and Motion Control Conference - EPE-PEMC 2006, August 2006, Portoroz, SLO Espiau, B.; Chaumette, F & Rives, P (1992) A new approach to visual servoing in ro-

botics, IEEE Trans Robot Automat., vol 8, pp 313-326

Lindenbaum, M (1997) An Integrated Model for Evaluating the Amount of Data

Re-quired for Reliable Recognition, IEEE Trans on Pattern Analysis & Machine tell

In-Hutchinson, S A.; Hager, G.D & Corke, P (1996) A Tutorial on Visual Servo Control,

IEEE Trans on Robotics and Automation, vol 12, pp 1245-1266, October 1996

Schilling, R.J (1990) Fundamentals of Robotics Analysis and Control, Prentice-Hall,

Englewood Cliffs, N.J

Zhuang, X.; Wang, T & Zhang, P (1992) A Highly Robust Estimator through

Par-tially Likelihood Function Modelling and Its Application in Computer Vision,

IEEE Trans on Pattern Analysis and Machine Intelligence

West, P (2001) High Speed, Real-Time Machine Vision, CyberOptics – Imagenation, pp

1-38 Portland, Oregon

28

Visual Feedback Control of a Robot in an Unknown Environment

(Learning Control Using Neural Networks)

Xiao Nan-Feng and Saeid Nahavandi

Many methods(1)-(11) have been so far proposed to control a robot with a era to trace an object so as to complete a non-contact operation in an unknown environment e.g., in order to automate a sealing operation by a robot, Hosoda,

cam-K.(1) proposed a method to perform the sealing operation by the robot through off-line teaching beforehand This method used a CCD camera and slit lasers

to detect the sealing line taught beforehand and to correct on line the joint gles of the robot during the sealing operation

an-However, in those methods(1)-(3), only one or two image feature points of the sealing were searched per image processing period and the goal trajectory of the robot was generated using an interpolation Moreover, those methods must perform the tedious CCD camera calibration and the complicated coor-dinate transformations Furthermore, the synchronization problem between the image processing system and the robot control system, and the influences

of the disturbances caused by the joint friction and the gravity of the robot need to be solved

In this chapter, a visual feedback control method is presented for a robot to trace a curved line in an unknown environment Firstly, the necessary condi-tions are derived for one-to-one mapping from the image feature domain of the curved line to the joint angle domain of the robot, and a multilayer neural network which will be abbreviated to NN hereafter is introduced to learn the mapping Secondly, a method is proposed to generate on line the goal trajec-tory through computing the image feature parameters of the curved line Thirdly, a multilayer neural network-based on-line learning algorithm is de-veloped for the present visual feedback control Lastly, the present approach is applied to trace a curved line using a 6 DOF industrial robot with a CCD cam-

era installed in its end-effecter The main advantage of the present approach is that it does not necessitate the tedious CCD camera calibration and the com-plicated coordinate transformations.

Contact object

Robot end-effector

Workspace

camera Rigid tool Tangential direction

Workspace frame

x o

y o z o

O Ȉ

x C

y C C

Ȉ

x B

z B

B

B

Robot base frame

z C

Cameraframe

ȟ

Goal feature

Initial feature

Feature of curved line

Figure 2 Image features and mapping relation

2 A Trace Operation by a Robot

When a robot traces an object in an unknown environment, visual feedback is necessary for controlling the position and orientation of the robot in the tan-gential and normal directions of the operation environment Figure 1 shows a robot with a CCD camera installed in its end-effecter The CCD camera guides the end-effecter to trace a curved line from the initial position I to the target position G Since the CCD camera is being fixed at the end-effecter, the CCD camera and the end-effecter always move together

3 Mapping from Image Feature Domain to Joint Angle Domain

3.1 Visual feedback control

For the visual feedback control shown in Fig 1, the trace error of the robot in the image feature domain needs to be mapped to the joint angle domain of the robot That is, the end-effecter should trace the curved line according to

j

a ,aj+1 in the image domain of the features Aj,Aj+1 shown in Fig 2

Let ȟ,ȟ d∈R6×1be the image feature parameter vectors of aj,aj+1 in the image feature domain shown in Fig 2, respectively The visual feedback control shown in Fig 1 can be expressed as

where || · || is a norm, and e should be made into a minimum

From the projection relation shown in Fig 2, we know

ȟ= ϕ (p tc), (2)

where ϕ∈R6×1 is a nonlinear mapping function which realizes the projection

transformation from the workspace coordinate frame ™ O to the image feature domain shown in Fig 2

It is assumed that ptc∈R6×1is a position/orientation vector from the origin of

the CCD camera coordinate frame ™ C to the gravity center of Aj Linearizing Eq.(2) at a minute domain of p tc yields

ȟ

where Dž ȟ and Dž ptc are minute increments of ȟ and ptc , respectively, and J f=

ptc

Ǘ

∂ ∂ ∈R6×6 is a feature sensitivity matrix

Furthermore, let ș and Dž ș∈R6×1 are a joint angle vector of the robot and its

minute increment in the robot base coordinate frame ™ B If we map Dž ș from

™ B toDž p t c in ™ O using Jacobian matrix of the CCD camera J c∈R6×6, we can get

pt c

Dž =Jc· Dž ș (4) From Eqs.(3) and (4), we have

ș

Dž = (JfJc)-1 · Dž ȟ (5)

Therefore, the necessary condition for realizing the mapping expressed by

Eq.(5) is that (J f J c)-1 must exist Moreover, the number of the independent age feature parameters in the image feature domain (or the element numbers

im-of ȟ) must be equal to the degrees of freedom of the visual feedback control system

3.2 Mapping relations between image features and joint angles

Because J f and J c are respectively linearized in the minute domains of ptc and

ș, the motion of the robot is restricted to a minute joint angle domain, and Eq.(5) is not correct for large Dž ș Simultaneously, the mapping is weak to the

change of (J f J c)-1 In addition, it is very difficult to calculate (J f J c)-1 on line during the trace operation Therefore, NN is introduced to learn such mapping

Firstly, we consider the mapping from ǻptc in ™ O to ǻȟ in the image feature domain ǻȟ and ǻp tcare increments of ȟ and p tc for the end-effecter to move from Aj to Aj+1, respectively As shown in Fig 2, the mapping is depend on p tc,and the mapping can be expressed as

ǻp =ij3(ǻȟ,ȟ), (8)

where ij3( )∈R6×1 is a continuous nonlinear mapping function

Secondly, we consider the mapping from ǻ ș in ™ B to ǻp c in ™ O Let p c∈R6×1be

a position/ orientation vector of the origin of ™ C with respect to the origin of

™ O, ǻ ș and ǻp c be increments of ș and p c for the end-effecter to move from Aj

to Aj+1, respectively ǻp c is dependent on ș as shown in Fig 2, and we obtain from the forward kinematics of the CCD camera

ij ( )∈R6×1 is a continuous nonlinear mapping function

Since the relative position and orientation between the end-effecter and the CCD camera are fixed, the mapping from ǻp c to ǻp tc is also one-to-one We get

Combing Eq.(8) and Eq.(11.b) yields

ș

ǻ =ij6(ǻȟ,ȟ,ș), (12)

where ij6( )∈R6×1 is a continuous nonlinear mapping function In this paper,

NN is used to learn ij6( )

3.2 Computation of image feature parameters

For 6 DOF visual feedback control, 6 independent image feature parameters are chosen to correspond to the 6 joint angles of the robot An image feature parameter vector ( j)

1 [ ξ j , ) 2

j

ξ ,···, j T

] ) 6

ξ is defined at the window j shown in Fig

3 L and W are length and height of the window j, respectively.

᧤

1

᧥ pixel white

᧤

0

the elements of )

ȟ are defined and calculated by the following equations:

(1) The gravity center coordinates

q g

1 1

) (

1 1

) (

Imagefeature domain

u v

j

Window

) 1 (j−

ȟ

W W

j qr

g

r g

1 1

) (

1 1

) (

g

1 1

) (

(3) The lengths of the main and minor axes of the equivalent ellipse

) 02 ) 20 )

( 02 )

20

2

)(4)(

j

j j

j j

j

ξ

λλ

λλ

02 )

20

2

)(4)(

j

j j

j j

j

ξ

λλ

λλ

g

1 1

) ( 1 )

qr r g

) ( 2 )

) ( 11 1

tan

2

1

j j j

λλ

ȟ are image feature parameter vectors in the

win-dow j and j+1 shown in Fig 3 ( 0 )( )

4 Goal Trajectory Generation Using Image Feature Parameters

In this paper, a CCD camera is used to detect the image feature parameters of

the curved line, which are used to generate on line the goal trajectory The

se-quences of trajectory generation are shown in Fig 4 Firstly, the end-effecter is

set to the central point of the window 0 in Fig 4(a) At time t=0, the first image

of the curved line is grasped and processed, and the image feature parameter vectors ( 0 ) ( 0 )

ȟ and ( 2 ) ( 0 )

ȟ in the windows 0,1,2 are computed

respec-tively From time t=mT to t=2mT, the end-effecter is only moved by

ȟ At time t=mT, the second image of the curved line is

grasped and processed, the image feature parameter vector ( 2 ) ( )

− At time t=imT, (i+1)th image is grasped and processed, the

image feature parameter vector ( 2 ) ( )

im

ȟ shown in Fig 4(c) is computed From

)

(im

ǻȟ = ( 1 )[( 1 ) ] ( 0 )[( 1 ) ]

m i m

) 1 (

ȟ

W W

L

)0(

) 0 (

ȟ

)0(

) 2 (

) 2 (

) 1 (

m

ȟ

) (

) 0 (

) (

) 2 (

im

ȟ

W W

L

) (

) 1 ( im

ȟ

) (

) 0 (

im

ȟ

W

0

c) Image feature domain t=imT

Figure 4.Image trajectory generation sequences

5 Neural Network-Based Learning Control System

5.1 Off-line learning algorithm

In this section, a multilayer NN is introduced to learn ij6( ) Figure 5 shows the structure of NN which includes the input level A, the hidden level B, C and the

output level D Let M, P, U, N be the neuron number of the levels A, B, C, D, respectively, g=1,2,ƛƛƛ,M, l=1,2,ƛƛƛ,P, j=1,2,ƛƛƛ,U and i=1,2,ƛƛƛ,N Therefore, NN

.

.

.

.

.

lj

( ),

( ,

, ,

1 1

1

D i D

i C

j C

j B

l B

l A

g

A

g

D i U

j

C j CD ji D

i P

l

C j B l BC lj C

j M

g

B l A g AB

gl

B

l

x f y x

f y x

f y x

y

z y w x

z y w x

z y w

x

, (15)

where x m R, R

m

y and z m R (m=g, l, j, i; R=A, B, C, D) are the input, the output and

the bias of a neuron, respectively w gl AB, w lj BC and CD

( [ )]

( )

( [

2

T n r

f f

CD

ji

w

E w

j D

algo-The present learning control system based on NN is shown Fig 7 algo-The initial

()

1

(

S k

k k

k

n n

n n

n

d

ǻș ș

ș

ǻș ș

ș

, (19)

where d∈R6×1is an output of NN when ș( 0 ), ( 0 ) ( 0 )

ȟ and ǻȟ( 0 ) are used as tial inputs of NN The PID controller G c (z) is used to control the joint an-gles of the robot

ini-5.2 On-line learning algorithm

For the visual feedback control system, a multilayer neural network NNc is troduced to compensate the nonlinear dynamics of the robot The structure and definition of NNc is the same as NN, and its evaluation function is defined as

experiments While NNc is learning, the elements of e(k) will become smaller

and smaller

Figure 7 Block diagram of learning control system for a robot with visual feedback

In Fig 7, I is a 6×6 unity matrix ȁ[ș(k)]∈R6×1and J r[ș(k)]=∂ȁ[ș(k)]/∂ș(k)∈R6×6

are the forward kinemics and Jacobian matrix of the end-effecter (or the rigid tool), respectively Let p t (k)=[p t1, p t2,··· T

de-5.3 Work sequence of the image processing system

The part circled by the dot line shown in Fig 7 is an image processing system The work sequence of the image processing system is shown in Fig 8 At time

t =imT, the CCD camera grasps a 256-grade gray image of the curved line, the

image is binarizated, and the left and right terminals of the curved line are tected Afterward, the image parameters ( 0 ) ( 0 )

ȟ and ǻȟ (k) are solved at time t=kT.

5.4 Synchronization of image processing system and robot control system

Generally, the sampling period of the image processing system is much longer than that of the robot control system Because the sampling period of )

] ) 1 ( [ 2

2

2

m i m

m i

ȟ

), 0 () 1 (

ȟ

) 0 () 2 (

ȟ

Binarization

Goal terminal detection

Start terminal detection

If iฺ 1, compute image feature

parameters in the window 2 ȟ (2)(im) (i=1,2,···)

),(esynchroniz)

im), ( ) at time =(

) 1

ȟ

), (

) 0 (

k

ȟ ȟ(1)( k ),

If i=0, compute image feature

parameters in the windows 0,1,2

Start tracing

Stop tracing

Figure 8 Work sequence of image processing

6 DOF robot

Imageprocessing

Robotcontrol

I/O ports

Switches

6 servo drivers

CCDcameracontroller

CCD camera

Work table

Rigidtool

Operation enverimentHand

Hand controller

Parallelcommunication

Figure 9.Trace operation by a robot

6 Experiments

In order to verify the effectiveness of the present approach, the 6 DOF trial robot is used to trace a curved line In the experiment, the end-effecter (or the rigid tool) does not contact the curved line Figure 9 shows the experimen-tal setup The PC-9821 Xv21 computer realizes the joint control of the robot The Dell XPS R400 computer processes the images from the CCD camera The robot hand grasps the rigid tool, which is regarded as the end-effecter of the robot The diagonal elements of K P,K I and K D are shown in Table 1, the con-trol parameters used in the experiments are listed in Table 2, and the weight-

indus-ing matrix W is set to be an unity matrix I.

Figures 10 and 11 show the position responses (or the learning results of NN and NNc)p t1(k),p t2(k) and p t3(k) in the directions of x, y and z axes of ™ O p t1(k),p t2(k)and p t3k) shown in Fig 10 are teaching data Figure 11 show the trace responses after the learning of NNc Figure 12 shows the learning processes of NN and

NNc as well as the trace errors E f converges on 10-9 rad2, the learning error E*

shown in Fig 12(b) is given by E*= E k N

k c

/ ] ( [N 10

¦−

tri-als (10000 iterations) using NNc, E*converges on 7.6×10-6 rad2, and the

end-effecter can correctly trace the curved line Figure 12(c) shows the trace errors

of the end-effecter in x, y, z axis directions of ™ O , and the maximum error is lower than 2 mm

Table 1 Diagonal ith element of K P,K I andK D

Table 2 Parameters used in the experiment

Sampling period T =5 ms, mT=160 ms

ControlParameters ηf=0.9,ηc=0.9,˟ =1,

K=100, S=732Window size L=256, W=10

0,060,0650,070,0750,08

0 0,000001

Time s

(c)Trace errors in x,y, z directions

Figure 12 Learning processes and trace errors

7 Conclusions

In this chapter, a visual feedback control approach based on neural network is presented for a robot with a camera installed on its end-effecter to trace an ob-ject in an unknown environment Firstly, the necessary conditions for mapping the image features of the object to be traced to the joint angles of the robot are derived Secondly, a method is proposed to generate a goal trajectory of the robot by measuring the image feature parameters of the object to be traced Thirdly, a multilayer neural network is used to learn off-line the mapping in order to produce on line the reference inputs for controlling the robot Fourthly, a multilayer neural network-based learning controller is designed for the compensation the nonlinear robotic dynamics Lastly, the effectiveness of

the present approach is verified by tracing a curved line using a 6 DOF trial robot with a CCD camera installed on its end-effecter

indus-Through the above research, the following conclusions are obtained:

1 If the mapping relations between the image feature domain of the object and the joint angle domain of the robot are satisfied, NN can learn the mapping relations

2 By computing the image feature parameters of the object, the goal tory for the end-effecter to trace the object can be generated

trajec-3 The approach does not necessitate the tedious CCD camera calibration and the complicated coordinate transformation

4 Using the 2nd-order holder and the disturbance observer, the zation problem and the influences of the disturbances can be solved

synchroni-The above research is supported by the Natural Science Foundation of China (No 60375031) and the Natural Science Foundation of Guangdong (No 36552), the authors expresses their heartily thanks to the Foundations

8 References

Hosoda, K & Asada, M., (1996) Adaptive Visual Servoing Controller with

Feed-forward Compensator without Knowledge of True Jacobian, Journal

of Robot Society of Japan, (in Japanese), Vol 14, No 2, pp 313-319

Hosoda, K., Igarashi, K & Asada, M., (1997) Adaptive Hybrid Visual/Force

Servoing Control in Unknown Environment, Journal of Robot Society of pan, (in Japanese), Vol 15, No 4, pp 642-647

Ja-Weiss, L.E & Sanderson, A.C., (1987) Dynamic Sensor-Based Control of

Ro-bots with Visual Feedback, IEEE Journal of Robotics and Automation, Vol.3,

No 5, pp 404-417

Nikolaos, P & Pradeep, K., (1999) Visual Tracking of a Moving Target by a

Camera Mounted on a Robot: A Combination of Control and Vision, IEEE Journal of Robotics and Automation, Vol 9, No 1, pp 14-35

Bernardino, A & Santos-Victor J, (1999) Binocular Tracking: Integrating

Per-ception and Control, IEEE Journal Robotics and Automation, Vol 15, No 6,

pp 1080-1093

Malis, E., Chaumette, F and Boudet, S., (1999) 2-1/2-D Visual Servoing, IEEE Journal of Robotics and Automation, Vol 15, No 2, pp 238-250

Hashimoto, K., Ebine, T & Kimura, H., (1996) Visual Servoing with Hand-Eye

Manipulator-Optimal Control Approach, IEEE Journal of Robotics and Automation, Vol 12, No 5, pp 766-774

Wilson, J.W., Williams, H & Bell, G.S., (1996) Relative End-Effecter Control

Using Cartesian Position Based Visual Servoing, IEEE Trans Robotics and Automation, Vol 12, No 5, pp 684-696

Ishikawa, J., Kosuge, K & Furuta, K., Intelligent Control of Assembling Robot Using Vision Sensor, 13-18 May 1990, Cincinnati, OH, USA, Proceedings

1990 IEEE International Conference on Robotics and Automation (Cat No.90CH2876-1), Vol 3, pp 1904-1909

Yamada, T & Yabuta, T., (1991) Some Remarks on Characteristics of Direct

Neuro-Controller with Regard to Adaptive Control, Trans Soc Inst Contr Eng., (in Japanese), Vol 27, No 7, pp 784-791

Verma, B., (1997) Fast Training of Multilayer Perceptrons, IEEE Trans Neural Networks, Vol 8, No 6, pp 1314-1319

29

Joystick Teaching System for Industrial Robots

Using Fuzzy Compliance Control

Fusaomi Nagata, Keigo Watanabe and Kazuo Kiguchi

Industrial robots have been applied to several tasks, such as handling, bling, painting, deburring and so on (Ferretti et al., 2000), (Her & Kazerooni, 1991), (Liu, 1995), (Takeuchi et al., 1993), so that they have been spread to vari-ous fields of manufacturing industries However, as for the user interface of the robots, conventional teaching systems using a teaching pendant are only provided For example, in the manufacturing industry of wooden furniture, the operator has to manually input a large mount of teaching points in the case where a workpiece with curved surface is sanded by a robot sander This task

assem-is complicated and time-consuming To efficiently obtain a desired trajectory along curved surface, we have already considered a novel teaching method as-sisted by a joystick (Nagata et al., 2000), (Nagata et al., 2001) In teaching mode, the operator can directly control the orientation of the sanding tool attached to the tip of the robot arm by using the joystick In this case, since the contact force and translational trajectory are controlled automatically, the operator has only to instruct the orientation with no anxiety about overload and non-contact state However, it is not practical to acquire sequential teaching points with normal directions, adjusting the tool's orientation only with operator's eyes

When handy air-driven tools are used in robotic sanding, keeping contact with the curved surface of the workpiece along the normal direction is very impor-tant to obtain a good surface quality If the orientation of the sanding tool largely deviates from normal direction, then the kinetic friction force tends to become unstable Consequently, smooth and uniform surface quality can’t be achieved That is the reason why a novel teaching system that assists the op-erator is now being expected in the manufacturing field of furniture

In this paper, an impedance model following force control is first proposed for

an industrial robot with an open architecture servo controller The control law allows the robot to follow a desired contact force through an impedance model

in Cartesian space And, a fuzzy compliance control is also presented for an advanced joystick teaching system, which can provide the friction force acting

between the sanding tool and workpiece to the operator (Nagata et al., 2001) The joystick has a virtual spring-damper system, in which the component of stiffness is suitably varied according to the undesirable friction force, by using

a simple fuzzy reasoning method If an undesirable friction force occurs in teaching process, the joystick is controlled with low compliance Thus, the op-erator can feel the friction force thorough the variation of joystick's compliance and recover the orientation of the sanding tool We apply the joystick teaching using the fuzzy compliance control to a teaching task in which an industrial robot FS-20 with an open architecture servo controller profiles the curved sur-face of a wooden workpiece Teaching experimental results demonstrate the effectiveness and promise of the proposed teaching system

2 Impedance Model Following Force Control

More than two decades ago, two representative force control methods were proposed (Raibert, 1981), (Hogan, 1985) ; controllers using such methods have been advanced and further applied to various types of robots However, in order to realize a satisfactory robotic sanding system based on an industrial robot, deeper considerations and novel designs are needed Regarding the force control, we use the impedance model following force control that can be easily applied to industrial robots with an open architecture servo controller (Nagata et al., 2002) The desired impedance equation for Cartesian-based con-trol of a robot manipulator is designed by

I are the switch matrix diag(S1, S2, S3) and identity matrix It is assumed that

d

M , B d, K d and K f are positive definite diagonal matrices Note that if

S = I , then Eq (1) becomes an impedance control system in all directions; whereas if S is the zero matrix, it becomes a force control system in all direc-

tions If the force control is used in all direction, X =x−xdgives

f d d

M

X =− −1 + −1 − (2)

Figure 1 Block diagram of the impedance model following force control.

In general, Eq (2) is solved as

( d d t) ( )0 texp{ d d(t τ) } d f ( d)dτ

0

1 1

F F K M B

M X

B M

( )k ( M d B d t)x(k ) { M d B d t I}B d K f{F k F d}

x =exp − −1 Δ  −1 − exp(− −1 Δ )− −1 ( )− (5)

where x( )k is composed of position vector [ ( ) ( ) ( ) ]T

k z k y k

variable x( )k is given to the normal direction to a workpiece Figure 1 shows

the block diagram of the impedance model following force control in

s-domain

Profiling control is the basic strategy for sanding or polishing, and it is formed by both force control and position/orientation control However, it is very difficult to realize stable profiling control under such environments that have unknown dynamics or shape Undesirable oscillations and non-contact state tend to occur To reduce such undesirable influences, an integral action is added to Eq (5), which yields

Figure 2 Relation among desired mass M di, damping B di and exp(−M di−1B diΔt)

From Eq (6), the following characteristics are seen Among the impedance rameters, desired damping has much influence on force control response as well as the force feedback gain The larger B d becomes, the smaller the effec-tiveness of force feedback becomes Figure 2 shows the relation among M di,

pa-di

B and diagonal elements of transition matrix exp(−M di−1B diΔt) in the case that

t

Δ is set to 0.01 [s] i denotes the i -th (i =1, 2, 3) diagonal element As can be

seen, for example, if B di is smaller than about 100, then appropriate M di is limited M di over 15 leads exp(−M di−1B diΔt) to almost 1 In selecting the imped-ance parameters, their combinations should be noted

3 Fuzzy Compliance Control of a Joystick Device

3.1 Fuzzy Compliance Control

In our proposed teaching system, the joystick is used to control the orientation

of the sanding tool attached to the top of the robot arm The rotational velocity

of the orientation is generated based on the values of the encoder in x- and

y-rotational directions as shown in Fig 3 Also, the compliance of the joystick is varied according to the kinetic friction force acting between a sanding tool and workpiece As the friction force becomes large, the joint of the joystick is con-trolled more stiffly Therefore, the operator can perform teaching tasks having the change of the friction force with the joystick's compliance

The desired compliance equation for the joint-based control of a joystick is signed by

de-J J J

J

J

~

ș K ș

respec-stiffness matrices of the joystick joints The subscripts x, y denotes x- and

y-directional components in Fig 3, respectively

Figure 3 Coordinate system of a joystick

Further, to adjust the compliance of the joystick according to the friction force,

Jx Jy

Jx

K

K K

00

0

(8)

where K J =diag(K Jx , K Jy) is the base stiffness matrix, ΔK J =diag(ΔK Jx ,ΔK Jy)

is the compensated stiffness matrix whose diagonal elements are suitably given from the following fuzzy reasoning part

3.2 Generation of Compensated Stiffness Using Simple Fuzzy Reasoning

In this section, we discuss how to suitably generate the compensated stiffness according to the undesirable friction force The compensated stiffness is adju-sted by using a simple fuzzy reasoning method, so that the teaching operator can conduct the teaching task delicately feeling the friction force acting betwe-

en the sanding tool and workpiece through the compliance of the joystick In

teaching, x- and y-directional frictions F x and F y in the base coordinate system are used as fuzzy inputs for the fuzzy reasoning, and they are used to

estimate y- and x-rotational compliance of the joystick joints The present fuzzy

rules are described as follows:

Rule 1: If | F x | is A ~ x1 and | F y | is A ~ y1, Then ΔK Jx =B x1 and ΔK Jy =B y1

Rule 2: If | F x | is A ~ x2 and | F y | is A ~ y2, Then ΔK Jx = B x2 and ΔK Jy =B y2

Rule 3: If | F x | is A ~ x3 and | F y | is A ~ y3, Then ΔK Jx =B x3 and ΔK Jy =B y3

#

#

Rule L: If | F x | is A ~ xL and | F y | is A ~ yL, Then ΔK Jx =B xL and ΔK Jy =B yL

Where A ~ xi and A ~ yi are i -th ( i =1,…,L) antecedent fuzzy sets for | F x | and | F y |,

respectively L is the number of the fuzzy rules B xi and B yi are the

conse-quent constant values which represent i -th x- and y-rotational compensated stiffness, respectively In this case, the antecedent confidence calculated by i -

th fuzzy rule is given by

(| F x |) Ayi(| F y |)

Axi

ω = ∧ (9)

where μX(⋅) is the Gaussian type membership function for a fuzzy set sented by

2

2βα

μX x x (10)

where α and β are the center of membership function and reciprocal value of standard deviation, respectively

Figure 4, Antecedent membership function for | F x | and

Table 1 Constant values in the consequent part

In the sequal, the compensated stiffness matrix ΔK Jis obtained from the weighted mean method given by

B

1

1 1

1

diag

ω

ωω

ω

Figure 4 shows the designed antecedent membership functions On the other hand, the designed consequent constants, which represent the compensated values of the stiffness, are tabulated in Table 1 In teaching experiments, the friction force more than 3 kgf is regarded as an overload If such an overload is detected, then the teaching task is automatically stopped and the polishing to-

ol is immediately removed from the workpiece Therefore, the support set of range [0, 3] in Fig 3 is used for the antecedent part

4 Teaching Experiment

4.1 Sanding Robot System

Throughout the remainder of this paper, the effectiveness of the proposed ching method is proved by teaching experiments

tea-Photo 3 Joystick system used in teaching experiments (Impulse Engine2000)

Photo 1 Robotic sanding system Photo 2 Air-driven sanding tool.