Human robot cooperative grasping

In the teleoperation process, motiontracking is carried out by Kinect in real time to detect the positions of human shoul-der, elbow and hand joints such that the robot can imitate human

CHEN NUTAN

NATIONAL UNIVERSITY OF SINGAPORE

2012

HUMAN-ROBOT COOPERATIVE GRASPING

CHEN NUTAN (B.Eng.(Hons.), DUT)

A THESIS SUBMITTED FOR THE DEGREE OF MASTER ENGINEERING DEPARTMENT OF MECHANICAL ENGINEERING NATIONAL UNIVERSITY OF SINGAPORE

2012

DECLARATION

I hereby declare that the thesis is my original work and it has

been written by me in its entirety I have duly acknowledged all the sources of information which have

been used in the thesis

This thesis has also not been submitted for any degree in any

university previously

_

Chen Nutan 6!December!2012!

I would like to express my sincere gratitude to my supervisor, Chew Chee-Meng(Associate Professor, Department of Mechanical Engineering, National University ofSingapore), for his constant support, invaluable suggestions, insightful comments andcontinuous encouragement during this research My interests in the field of roboticsstarted when I joined in Control and Mechatronics Laboratories under Prof Chew.His guidance helped me in all the time of research and writing of this thesis I couldnot have imagined having a better advisor and mentor for my Master study

My sincere thanks also goes to my co-supervisor, Han Boon Siew (Senior ResearchOfficer, Institute for Infocomm Research, A*Star), for providing the opportunity

of developing the world famous robots, Meka and Hubo, to me, and leading meworking on diverse exciting projects His innovative ideas also stimulate me to trynew equipments and methods

Next, I would like to thank Tee Keng Peng (Doctor, Institute for Infocomm Research,A*Star), for his technical guidance I gained the skills of research methodology andalgorithm developments from him His stimulating suggestions, encouragement andguidance helped me during the whole research

Anthony, Chua Yuan Wei, Yan Rui, and Chang Taiwen for the discussions, and forthe days we were diligently working together before deadlines In addition, I thank myseniors in Control and Mechatronics Laboratories of National University of Singapore:Huang Weiwei, Albertus Hendrawan Adiwahono, Shen Bingquan, Li Renjun, and WuNing for enlightening me at the first glance of research With their invaluable advices,

I could finish this project smoother

1.1 Related Research Areas 2

1.1.1 Teleoperation 2

1.1.2 Autonomous Grasping 3

1.1.3 Human-robot Cooperation 5

1.2 Related Work 6

1.3 Motivation and Objective 8

1.4 Dissertation Outline 11

2 Telemanipulation System 12 2.1 Human-teleoperated Pre-grasp Position 13

2.1.1 Human Arm Tracking 13

2.1.2 Human Hand Detection 16

2.1.3 Robot Control 19

2.1.4 Feedback System 25

2.2 Gesture Based Grasp Activation 26

2.2.1 Table And Objects Perception 26

2.2.2 Deciding How to Grasp 30

2.3 IR-based Grasp Assistance 34

2.3.1 Hardware–infrared Sensors with Robot Hand 35

3 Experiment Results and Discussion 44

3.1 Robot Setting 44

3.2 Teleoperation Results 47

3.3 Experimental Results of Gesture-based Grasp Activation 50

3.3.1 Autonomous Grasping Results 50

3.3.2 Combined Method Results 51

3.4 Experimental Results of IR-based Grasp Assistance 56

3.4.1 Comparison of Full Teleoperation with Full Assistance 57

3.4.2 Ratio of Teleoperation to Assistance 58

3.4.3 Graspable Areas 66

3.4.4 Graspable Objects 67

3.4.5 Adjustment of Orientation 69

3.4.6 Tracking Mobile Object 71

4 Discussion 74 5 Conclusion and Future Work 76 Bibliography 78 A Appendix 87 A.1 Appendix 1 87

In the field of robotics, teleoperation and grasping make sense and play a valuablerole, whether in terms of entertainment or practical application It is meaningful forrobots to imitate human and finish some tasks such as grasping With these abilities,robots can really be developed to assist human in our daily life

In order to enable robots to grasp a desired object through teleoperation, this sis describes combined approaches of real time remote teleoperation and autonomousgrasping for human-robot cooperation For providing a user-friendly system with sim-ple operation commands to grasp di↵erent objects successfully, vision-based teleoper-ation and autonomous grasping are combined In the teleoperation process, motiontracking is carried out by Kinect in real time to detect the positions of human shoul-der, elbow and hand joints such that the robot can imitate human Hand gestures aredetected by Kinect or AcceleGlove to control the robot hand gestures Autonomousgrasping is then employed, which is e↵ective and generate more natural grasping ges-tures The autonomous grasping subsystems are robust and can grasp both knownand unknown objects Two approaches of autonomous grasping are employed in thisthesis In the first approach, hand gestures are recognized as the switch of teleopera-tion and autonomy At the end of teleoperation, the person closes his or her hand tosend a signal to the robot, then the system convert to autonomous grasping Afterthat, the second subsystem begins for autonomous grasping for known objects, which

the-requiring premature object contact or regrasping strategies We use three infra-red(IR) sensors that mounted on the robot hand, and design an algorithm that controlsthe robot hand to grasp objects using the information from the sensor readings andthe teleoperator.

To satisfy the requirement, high performance Meka robot is used The robot has 26DOFs in total, 7 DOFs for each arm, 5 DOFs for each hand and 2 DOFs for torso.Experiment results show that it is e↵ective and user-friendly, and it has the capability

to complete the missions of grasping of known and unknown objects For known jects, it can convert from teleoperation to autonomy smoothly with simple commandsfrom hand gestures of the user, and it can find the edge of objects and even trackmobile objects for unknown objects using IR algorithm Both of the two strategiesenhance the success rates compared to pure teleoperation

ob-List of Tables

3.3 Grasping success rates 553.4 Grasping success rates with di↵erent ratio of teleoperation and IR al-gorithm 603.5 Grasping success rates 68

1.1 Overview of the proposed system structure 9

2.1 Overview of the proposed teleoperation structure 13

2.2 Skeleton The right figure shows the view from Kinect, and the left figure the human skeleton extracted from the sensor data 14

2.3 Kinect 14

2.4 Flow chart of detecting joint data 15

2.5 Point cloud of hand 16

2.6 Flow chart of detecting hands 17

2.7 Hand detection The right figure shows the hand detection result from the left figure 17

2.8 acceleglove output signal convention (top view, right hand) [1] 18

2.9 MEKA robot [2] 19

2.10 MEKA robotics arm joints: (a)the posture shown 0 joint angles, arrows show positive rotation and torque directions; (b) Z axis is the axis of joint rotation [2] 19

2.11 Coordinate sets of a person 20

2.12 Shoulder-elbow on {X0, Y0, Z0} frame 21

2.13 Shoulder-elbow on Y Z plane 22

2.14 Shoulder-elbow on Y0 Z0 plane 22

2.15 The whole arm on {X00, Y00, Z00} frame 23

2.16 The whole arm on {X, Y, Z} frame 23

2.17 MEKA robotics hand joint names and directions: the posture shown 0 joint angles, arrows show positive rotation and torque directions [2] 24 2.18 Feedback system 25

2.19 Table and object detection The left image shows a table, a can and MEKA robot from Kinect RGB sensor The right image shows the detection of the table and the can and robot model from RVIZ (A 3d visualization environment for robots using ROS) simulation 30

2.20 Tool frame [2] 31

2.21 Graspit! simulation 32

2.22 Jerk trajectory 34

2.23 Hardware of IR sensors: The red points are the centers of sensor, and two edges of objects are supposed to be on the sensor 1 and sensor 3 centers as the blue rectangular 36

2.24 Conrol block diagram FK and IK denote forward & inverse kinematics maps respectively 41

3.1 Teleoperation with virtual robot 47

3.2 Teleoperation with real robot The first image shows the robot mimics the person to move the arms The second image shows pre-grasp The third image shows the robot mimics the person to grasp an object 48

3.3 Degree of joints during a motion 49

3.4 Simulation on RVIZ It shows the detection of the table, the can, and the robot model The blue cube indicates the selected object 51

the robot lifts the object 52

3.6 Grasping object using the combined method The first image shows the teleoperation The second image shows the switching from the teleop-eration to autonomous grasping The third image shows autonomous grasping 53

3.7 The position of the tool frame during grasping 54

3.8 Grasp postures The left image shows grasping a can The right image shows grasping a tape 55

3.9 Teleoperation with and without the proposed IR Algorithm The blue cylinder represents a soda can, and the points the pre-grasp positions 57 3.10 2D vision feedback 58

3.11 E↵ort in grasp placement for di↵erent ratios of teleoperation to assis-tance Mean values and standard deviations are shown 62

3.12 Error in final position(ef) for di↵erent ratios of teleop to assistance Mean values and standard deviations are shown 63

3.13 The position of IR sensors with respect to robot base frame during the person moves close to an object then moves away 64

3.14 The combined e↵orts of grasp placement and error recovery ability Mean values and standard deviations are shown 65

3.15 Graspable area maps verified experimentally The shaded rectangle and circle are sample objects End ”⇥” represents the position of the center of the surface of the object facing the IR sensors 67

3.16 Failures of grasping 68

3.17 Object orientation test 69

3.18 Orientation result 70

3.19 Object trajectory detection 713.20 Tracking mobile object The black rectangle is an object A is a point

on the corner of the object The blue nearly rectangular line is thetrajectory of the object The green stars are the trajectory of sensor

3 The three red circles represent the three sensors It is with respect

to robot base frame 71

A.1 Grasp experiments: for every object, the top picture shows the initialposition of the sensor detecting an object The middle and the bottompictures show the final grasp from front view and side view respectively 88A.2 Grasp experiments: for every object, the top picture shows the initialposition of the sensor detecting an object The middle and the bottompictures show the final grasp from front view and side view respectively 89A.3 Grasp experiments: for every object, the top picture shows the initialposition of the sensor detecting an object The middle and the bottompictures show the final grasp from front view and side view respectively 90

The use of robots can be traced back to the end of the twentieth Century With manyyears’ trial and error and a large number of researchers’ contribution, many worldfamous robots appear, such as ASIMO [3], PR2 [4] lightweight robot (LWR) [5] andRobonaut [6] Nowadays, robots fulfill some tasks and perform meaningful services,from industrial manufacturing to healthcare, transportation, and exploration of thedeep space and sea It is an increasing research topic due to robots’ useful applicationsand challenging potential Interacting and working with humans, the robots willbecome a part of our lives

1.1 RELATED RESEARCH AREAS

This thesis describes a method in which a robot imitates a human in performing thetask of grasping The three fields related to the work done in this thesis are teleoper-ation, autonomous grasping and human-robot interaction This section describes therelated research areas

Vision-based teleoperation [13, 14] have gained popularity in recent years as they aremore portable and do not require that the user wear any special equipment However,the limitation of vision-based method is that only partial information of human can

be collected due to occlusion.

Although a variety of concepts and methods for grasping have been developed duringthe last decades, grasping an unknown object still remains a challenging problem Forknown object where a full 3-D model can be obtained, various more robust approachescan be used for grasping For example, methods based on friction cones, or form-closure and force-closure [17], or pre-stored primitives [18], etc can be applied

In reality, due to uncertainties of sensor limitations and unpredicted environmentconditions, it is difficult to obtain a full 3-D model of an object through stereo camera

or other sensors such as a laser scanner Therefore, it is necessary to extract grasppoints from partial information of an object

1.1 RELATED RESEARCH AREAS

Some approaches have been explored to grasp unknown objects An approach, scribed in [19], of partial features predicts the grasp position of unknown objectsusing 2D images However, one grasp point is defined per object, which is not generaland may result in an unstable grasp A system for grasping objects using unknowngeometry [20] was developed This system requires a 360 degrees scan of an objectusing a laser scanner on a rotating disc This method is time-consuming and requiresthat the object be placed on the rotary disc

de-A framework of automatic grasping of unknown objects by a laser scanner and asimulation environment is shown in [21] Another method [22] combining onlinesilhouette and structured-light generates a 3D object model with a robust force closuregrasp However, only several simple objects have been tested for both [21] and [22],which cannot demonstrate that they are suitable for complicated and general objects

A vision based approach was presented in [23] Object information was obtained usingmonocular and binocular visual cues and their integration Curvature information [24]was obtained from the silhouette of the object The pose of the robot is then updatedand a suitable grasping configuration is achieved by maximizing the curvature value

A strategy for grasping unknown objects based on co-planarity and color informationwas developed in [25] However, the environments in [25] are simple, which cannot

be applied to the real world

1.1.3 Human-robot Cooperation

Human-robot cooperation is useful for performing special tasks in dangerous, distant

or inaccessible environments in military missions such as clearing nuclear waste [26]and defusing a bomb They are also useful for applications such as serving elderly anddisabled people [27] Such systems take advantage of the ability of both the humanand the robot They reduce human workload, costs, fatigue-driven error and risk [28],and augment human’s abilities Hence, given the present state of robotics, it is one

of the fundamental methods for controlling robots

In the applications stated above, there is synergy between robots and human Theyshare a workspace and goals in terms of achieving the task This close interactionneeds new theoretical models–there is need to improve a robots utility while evaluatingthe risks and benefits of this robot for modern society

There are many investigative studies on robot assistive technology for many tions Specifically, robots are studied as tools to aid in daily tasks, act as guides andbecoming assistants with high communication behavior [29, 30]

applica-The concept of human and robots sharing a common intent without complex nication was mentioned in [31] The system consists of perception, recognition andintention inference The result of the study was positive although toys representedrobots

commu-1.2 RELATED WORK

Teleoperation with haptic feedback was developed to achieve a more natural ande↵ective method for human-robot cooperation This method of interaction allowedfor a more ecological interface [32, 33] Both the human operator and the robot sharecontrol depending on the situation This system is more intuitive for human operatorsand has proven to be more e↵ective

Another method of collaboration is to treat the human as a robot assistant while therobot acts autonomously [34, 35] The robot works autonomously until it encounters

a problem, where the robot will seek assistance from a person Alternatively, therobot performance could be improved through human suggestions

Recently, Robonaut, an assistant humanoid robot designed by NASA [36], was sent

to outer space Robonaut was teleoperated remotely with force feedback integrated

There are few approaches that combine teleoperation with autonomous grasping [37,38] Although there exist some combined approaches for other robot control, such aslocal autonomous formation control [39], and event-based planning [40], they are notfor remote grasping

Middle or long range sensors such as laser scanner [41] and stereo cameras [42] can tect and localize objects fairly accurately, but they are not suitable for teleoperation

de-Firstly, occlusion by the robot arm may occur during manipulation of the object.Secondly, image acquisition and processing are generally not fast enough for onlinereactive response in unstructured environments (e.g when the object is moving).

Tactile sensors are employed as well in the following methods [43, 44], but they arecontact based sensing methods For teleoperation, contacting objects is almost asdifficult as grasping them Besides, it lacks sensitivity and hence not suitable forteleoperation

Short range stereo cameras [45] mounted on the end-e↵ector were developed However,they have a narrow field of view and cannot be positioned at short distances to theobject Otherwise, there may be no good grasp points In addition, the camera isbeing used for grippers, and might fail for humanoid hand due to the larger width,which might cause occlusion of the object

Optical infrared sensors have also been employed for final grasp adjustments though they have less information compared to cameras or lasers, but they are lesssensitive to environmental changes and require less computation The method in [46]detects the orientation of an object surface using the IR sensors that fit inside thefingers Continuous Shared Control [47] combines brain signal and IR sensors to graspobject However, [46] can only adjust the fingers, and [47] can only be used for onedimension of the end-e↵ector [48] equips IR sensors on a gripper to adjust the griperfor a normal force to the object boundary However, the objects are supposed to be

Al-in the gripper before applyAl-ing the method and it uses logic approach which is discrete

1.3 MOTIVATION AND OBJECTIVE

control

Current autonomous robots cannot meet real life expectations because of their limitedabilities for manipulation and interaction with humans These robots could fulfillsome simple tasks, but the process may be time-consuming Moreover, robots cannothandle changes well without user intervention With teleoperation, robots can receivehuman’s commands in real time under human’s assistance to execute tasks However,teleoperation also has limitations For example, simple teleoperation systems maynot be able to collect sufficient information from a person resulting in robots ignoringsome important tasks On the other hand, teleopoeration systems that acquire moredata require intricate instruments or complex operations In addition, even if thesystems can obtain full user information, it is hard to use in the real world without

a trained person to operate the robots

In order to overcome these challenges, we develop two combined approaches, whichprovide user-friendly operation The first approach contains two subsystems for grasp-ing known objects (see Fig 1.1)

The teleoperation subsystem enables the end-e↵ectors to be brought close to thedesired object In this system, the information from the Kinect sensor is continuouslydetected from the human joints and sent to the robot control system As the bridge of

Figure 1.1: Overview of the proposed system structurethe two subsystems, hand gestures play a fundamental role in the switching Whenthe person closes his hand, a signal is sent to the robot to switch to autonomousgrasping The autonomous grasping subsystem contains table and object perceptionand grasp planning.

Another approach is final grasping correction for teleoperation using three IR sors It is e↵ective, light, robust, small size and cheap which are desired qualities forassisting grasping in teleoperation We chose the minimum number of sensors thatyields the most useful information, thus reducing the size of the structure and sim-plifying the algorithm Three sensors are mounted on the robot hand for localizing

sen-a nesen-arby object sen-and providing error signsen-als thsen-at drive the hsen-and in three dimensionsbased on a potential energy algorithm The global minimum potential energy could

be calculated in order to look for a grasp point, as well as teleoperation also a↵ects

1.3 MOTIVATION AND OBJECTIVE

the trajectory of the hand; therefore, the combined result enables the end-e↵ector tofollow teleoperation and track the object at the same time

In this work, Meka robot [2] is chosen, whose manipulators are 7 DOF arms, e↵ectors are 5 DOF hands, and body is a 2 DOF torso

end-Main features of the proposed telemanipulation system:

1, cooperative grasping It combines the advantages of having human initiative, andthe accuracy and robustness of the robotic system

2, need a little object knowledge The proposed IR strategy is to grasp objects withlimited perception data The three IR used in the current approach only gives onedimension data

3, online adjustment It allows online grasp adjustments to estimate a suitable grasppoint of unknown objects without requiring premature object contact or regraspingstrategies

4, robust grasping The system is robust when grasping a wide range of objects andeven tracks mobile objects

5, portable interface component It employs the low cost Microsoft Kinect as aninterface instead of other higher end equipment for human motion capture

1.4 Dissertation Outline

The structure of this thesis is as follows

Firstly, Chapter 2 gives an overview of the teleoperation and autonomous graspingsystems and provides a detailed description of the experimental methods, includingdi↵erent teleoperation techniques

In particular, it consists of three subsections Subsection one describes teleoperation

in this system using Kinect sensor Both the second and third subsections looksinto autonomous grasping, describing di↵erent approaches for grasping known andunknown objects respectively

In Chapter 3, the results of the experiments are discussed A comparison of thetwo di↵erent methods - teleoperation grasping and the combined method grasping, isdone The results demonstrate that the combined method is more e↵ective and has

a higher success rate

Next, Chapter 4 discusses the experimental results and demonstrates the contribution

of this thesis

Finally, Chapter 5 concludes with a section discussing the future work for betterhuman-robot cooperation

Chapter 2

Telemanipulation System

To obtain the benefits of human dexterity and robot accuracy, the telemanipulationsystems consist of human teleoperation and robot autonomous grasping Direct po-sition control is carried out for teleoperation The data detected from human jointpositions are transformed to the shoulder and elbow joint angles after inverse kine-matics For the first apporach, the gestures of hands are also detected continuously.Once one of the hands of the human is closed, the control is switched to autonomousgrasping immediately The robot then detects the nearest object to the tool frame onthe table in front of the robot and generates a suitable grasp for the object Alter-natively, another approach of autonomous grasping that adjusting final grasp usingshort range Infra-Red porximity sensors is also presented In this IR approach, sen-sors can search for the edge of an object Teleoperation and IR signal have combinedcontribution to the robot grasping

2.1 Human-teleoperated Pre-grasp Position

The proposed teleoperation system consists of a person, a MEKA robot and twoKinect sensors The main purpose of this subsystem is to enable the end-e↵ector to

be brought to a good position close to an object for autonomous grasping In thissystem, the person is at a local place while the robot is at a remote place interactingwith the environment The process includes three steps:

(1) human arm tracking;

(2) human hand detection;

(3) robot control

Figure 2.1: Overview of the proposed teleoperation structure

Motion tracking is a process for generating human skeleton information which containsposition data of human joints (see Fig.2.2) It is carried out using Kinect which has acolor image CMOS sensor and depth sensor including an infrared laser projector and

a monochrome CMOS sensor as shown in Fig.2.3

2.1 HUMAN-TELEOPERATED PRE-GRASP POSITION

Figure 2.2: Skeleton The right figure shows the view from Kinect, and the left figurethe human skeleton extracted from the sensor data

Figure 2.3: Kinect

As shown in Fig.2.4, Human joint detection mainly contains three processes, usergeneration, pose detection, and skeleton generation In the process of user generation,human in the view of depth camera are detected using 3D raw data from sensors Inthis process, the main functionalities includes: (1) Detecting the current number ofusers; (2) Calculating the center location of users’ mass; (3) Tracking a new user Posedetection enables the system to check if the user is in the specific position Calibrationwill start after specific positions are detected In this work, we choose T-shape pose

as this specific pose Skeleton generation calibrates the human skeleton, gets the joint

Figure 2.4: Flow chart of detecting joint datadata continuously using tracking algorithm, and enables the application to transfer thedata to other applications [49] In these processes, data of human joints is generatedusing Open Natural Interaction (OpenNI) which communicates vision/audio sensorsand NITE, a motion tracking middleware.

The Kinect sensor could recognize 15 joints using the middleware of NITE, includinghead, neck, torso, left shoulder, left elbow, left hand, right shoulder, right elbow, righthand, left hip, left knee, left foot, right hip, right knee, right foot For every joint,the data contains the position of the human joints, the orientations, and confidences

2.1 HUMAN-TELEOPERATED PRE-GRASP POSITION

In this project, the joints of the arms are employed for robot manipulators

(a) Point cloud of closed hand (b) Point cloud of open hand

Figure 2.5: Point cloud of hand

By hand detection process, hand point clouds and hands’ states can be provided [50]

We assume the palm faces to the camera The first step is to detect the hand’sposition by the skeletal information getting from the above human tracking Thenext step is to estimate the distance of a point of raw data and the hand position

If the distance is less than a threshold, the point will be deemed as one element ofpoint cloud of the hand and split from the whole point cloud of the human Lastly,two states of the gestures (closed and open) are identified The centroid of the hand

the set of hand points in 3 dimensions The eigenvalues of the covariance matrixreflects the distribution of the points in the main directions In other words, theeigenvalues represent the three dimensions of the hand point cloud which containsthe main information of the geometry of the point cloud Therefore the 2 highesteigenvalues eig1(⇣) and eig2(⇣) are used for detecting the hand state

eig2(⇣)/eig1(⇣) = c (2.1)

where, if the ratio c is less than a threshold, the point cloud is considered as a longshaped object, and vice versa Based on the experiment, c is chosen to be 0.4

Left Hand

near the wristposition

segment the hand cluster from the arm

Distinguish whether the hand

is closed or openFigure 2.6: Flow chart of detecting hands

Figure 2.7: Hand detection The right figure shows the hand detection result fromthe left figure

Acceleglove (AnthroTronix, Inc.) is employed as another method of hand detection tocompare with using Kinect Acceleglove could detect the finger and hand information

to be used for some applications such as controlling robots, video games, and lators [51] In this experiment, the Acceleglove enable human and robot to interactwith each others

simu-2.1 HUMAN-TELEOPERATED PRE-GRASP POSITION

First of all, the data of the fingers and palm are detected through 6 accelerometers.Every accelerometer outputs x, y and z signals If the glove is horizontal, z axis

is along gravity vector, as well as x and y axis parallel to a horizontal plane (seeFig 2.8) When x axis is rotated or there is an acceleration in the y direction, thevalue of y axis would be changed Analogously, when y axis is rotated or there is anacceleration in the x direction, the value of x axis would be changed

Figure 2.8: acceleglove output signal convention (top view, right hand) [1]

After detecting the data from sensors, the gestures could be trained recognized usingthe raw signals In the process, four gestures, start, stop, closed and open, are recordedfor each hand in a library Besides, around 20 group data are recorded for everygesture, because more instances trained for one gesture could enhance the recognitionconfidence A probability filter is employed to recognize the gestures; therefore, only

if the probability is more than the threshold can the gesture be recognized

At last, the gesture information are transferred to the robot using TCP/IP, becauseAcceleglove is used under Windows, while Robot uses ROS under Linux in this ex-periment

J4 J5 J6

(a) Joint names and directions

{RT1}

{RT2}

Y Z

Y Z {RT3}

{RT4}

Z Y X {RT5}

X

Y Z {RT6}

X

Y Z {RT7}

X Y Z {T0}

Z Y X

Y

Z X

{RT8}

X

Y Z RIGHT ARM

{LT1} {LT2}

Y Z

X X

Y Z {LT3}

{LT4}

Z Y X {LT5}

X

Y Z {LT6} X

Y {LT7}

Z Y X

Y

Z X

{LT8}

X

Y Z LEFT ARM

(b) Kinematic frames

Figure 2.10: MEKA robotics arm joints: (a)the posture shown 0 joint angles, arrowsshow positive rotation and torque directions; (b) Z axis is the axis of joint rotation [2]Direct position control is used for the arm motion tracking The MEKA robot (seeFig.2.9) is a humanoid robot with 7 DOFs for each arm Its coordinate sets can

2.1 HUMAN-TELEOPERATED PRE-GRASP POSITION

Figure 2.11: Coordinate sets of a personTable 2.1: Modified Denavit-Hartenberg representation of the right arm of MEKA

In 2.12, frame {X0, Y0, Z0} is parallel to the camera frame with (x1, y1, z1) as the

2.1 HUMAN-TELEOPERATED PRE-GRASP POSITION

0

0

 (z1,y1)

= atan2((x2 x1), ((z1 z2) cos(⇡/2 1) + (y1 y2) cos( 0))) (2.5)

From 2.15, frame{X00, Y00, Z00} is parallel to the camera frame with (x2, y2, z2) as theorigin, and BF is projection of BE on X00 Z00 plane 2 can be computed as follow,

We only consider the angles of the joints, so that the frame transformation from joint

0 to joint 1 can be represented by,

0

0BBBB

As shown in From 2.16,

2.1 HUMAN-TELEOPERATED PRE-GRASP POSITION

In order to achieve smooth motion for robot, mean filter is employed to reduce noise

of the robot joint data The positions calculated after the above algorithm at time t1,

t2, t3, , ti, ti+1, ti+2, , ti+k, , are denoted by q(t1), q(t2), q(t3), , q(ti), q(ti+1),q(ti+2), , q(ti+k), , where q(ti) is a vector of the 4 joints’ angles

3 tendon(J2, J3, J4) and a 2-DOF thumb driven by a tendon(J1) and directly driven

by a motor(J0) [2] The encoders measure the tendon position instead of every joint

position Under the circumstances, the true finger pose could neither be measurednor uniquely determined Thus, there are only two gestures of the hands (closed andopen) in this thesis.

2.1.4 Feedback System

(a) cameras mounted on the robot head (b) cameras mounted on the robot hands

(c) vision feedback from the camera of the

2.2 GESTURE BASED GRASP ACTIVATION

and hands, and a monitor at local displays the three views One of the three cameras

is mounted on the head of Meka robot as robot’s eyes, and the rest(Point Grey) areequipped on the end-e↵ectors besides the thumbs The feedback images are shown inFig.2.18 (c) and (d)

The teleoperation behaviors attempt to move the manipulators such that the hand isnear an object and send a signal to start autonomous grasping The grasping behaviorwill search for the nearest object besides the robot hand using another Kinect, andconsequently generate a suitable position and orientation to grasp it

2.2.1 Table And Objects Perception

Before generating grasping strategies, the robot has to get the information of theobjects In this experiment, all graspable objects are assumed on a table Therefore,the table should be detected, then the objects can be separated and be identified

All data are point cloud generated from a Kinect sensor mounted on a table ronment perception contains two main components:

Envi-• table detection: it could segment the table, and extract the clusters above the tableand remove the useless information