Multi-Robot Systems From Swarms to Intelligent Automata - Parker et al (Eds) Part 10 ppt

Restricting attention to managing a team of multiple robots where a single human must be able to analyze video from each robot, we review how neglect time and inter-action time of the in

HUMAN-ROBOT INTERACTION

Michael A Goodrich

Computer Science Department, Brigham Young University, Provo, UT, USA

mike@cs.byu.edu

Morgan Quigley

Computer Science Department, Brigham Young University, Provo, UT, USA.

Keryl Cosenzo

U.S Army Research Laboratory, Aberdeen, MD, USA.

Abstract Determining whether it is possible for a single human to manage a team of

mul-tiple robots is an important question given current trends in robotics Restricting attention to managing a team of multiple robots where a single human must be able to analyze video from each robot, we review how neglect time and inter-action time of the interface-robot system provide a test for feasibility of a team.

We then present a feasibility test that is applicable if the cost of switching at-tention between multiple robots or multiple tasks can become prohibitive We then establish that switch costs can be high, and show that different tasks impose different switch costs.

Keywords: Switch costs, fan-out, human-robot interaction, multiple robot management

Recently, there has been much discussion in the robotics community on creating robot systems that allow a single human to perform multiple tasks, especially managing multiple robots The possibility for such one-to-many human-robot teams is caused by the ever-increasing autonomy of robots As

a robot becomes more autonomous, its human manager has more free time to

do other tasks What better way to use this free time than to have the human manage multiple robots or manage multiple tasks?

The potential impact of this line of reasoning includes some very desirable consequences, but there are some clear upper bounds on the number of robots and the number of tasks that a single human can manage These upper bounds

185

L.E Parker et al (eds.),

Multi-Robot Systems From Swarms to Intelligent Automata Volume III, 185–195.

 c 2005 Springer Printed in the Netherlands.

are created by how long a single robot can be neglected Formally, neglect time is the expected amount of time that a robot can be ignored before its

performance drops below a threshold

During the time that a robot is being neglected, the human manager can con-ceivably be doing any other task However, once the neglect time is exhausted, the human must interact with the robot again The average amount of time required by the human to “retask” the robot once interaction begins is referred

to as the interaction time Formally, interaction time is the expected amount of

time that a human must interact with a robot to bring it to peak performance

In a problem with multiple robots, neglect time and interaction time dictate the maximum number of robots that a single human can manage The upper bound on the number of robots can easily be computed when all robots are homogeneous and independent The idea of determining how many indepen-dent homogenous robots can be managed by a single human is captured by the notion of fan-out (Olsen and Goodrich, 2003) Roughly speaking, fan-out

is one plus the ratio of neglect time to interaction time The ratio represents the number of other robots that the human can manage during the neglect time interval, and the “plus one” represents the original robot Thus,

FanOut = NT

IT + 1

where NT and IT represent neglect time and interaction time, respectively.

This idea can be extended to teams of heterogeneous robots performing in-dependent tasks When a team is made up of heterogeneous robots, then each robot has its autonomy level and interface This, in turn, implies that each

ro-bot has a given neglect time and interaction time Let N N i = (NT T i ,IT T i) denote

the neglect and interaction time of robot i A team of M robots consists of the

setT = {N N i : i = 1 M}.

To determine whether a human can manage a team of robotsT , we can use

the neglect times and interaction times to determine if a team is infeasible

T is

 feasible if∀i NT T i ≥ ∑ j =  i IT T j

The key idea is to find out whether the neglect time for each robot is sufficiently long to allow the human to interact with every other robot in the team If not, then the team is not feasible If so, then there is sufficient time to support the team, though the team performance may be suboptimal, meaning that a different team configuration could produce higher expected performance Fan-out and the feasibility equation are upper bounds on the number of in-dependent robots that can be managed by a single human The purpose of this paper is demonstrate that the amount of time required to switch between robots can be substantial, and can dramatically decrease this upper bound

2 Related Literature

Approaches to measuring switch costs are usually loosely based on fun-damental models of cognitive information processing (Meiran et al., 2002, Leviere and Lee, 2002) These models suggest that procedural memory ele-ments, encoded as modules in long-term memory and sometimes referred to as mental models, dictate how particular stimuli are interpreted and acted upon

by a human When the nature of the task changes, a required switch in mental models is required and this switch comes at a cost even if the stimuli does not change Reasons for this cost include the need to prepare for the new task and inhibit the old task

The experimental methodology typically adopted in the cognitive science literature has been to use the same set of stimuli but switch the task to be done

on the stimuli(Cepeda et al., 2001, Koch, 2003) For example, the digits “1 1 1”, “1”, “3 3 3”, and “3” can be randomly presented to a subject One task requires the subject to name the digit (one, one, three, and three, respectively), and the other task requires the subject to count the number of digits depicted (three, one, three, and one, respectively) Switch cost is given by the extra time required when a trial requires a change from one task to another as compared

to a trial when the task does not change

This approach has limited application to the human-robot interaction do-main for two reasons First, the absolute values of the switch costs are very low; they are on the order of fifty milliseconds Second, human-robot interac-tion domains are not simple stimuli-response tasks, but rather require the use of short term memory and the possible recruitment of multiple mental models to solve a problem As a result, new experimental methodologies must be created

to measure switch costs in human-robot interaction domains

Altmann and Trafton have proposed one technique for measuring switch costs in more complex domains (Altmann and Trafton, 2004) Their approach, which has been applied to problems that impose a heavy burden on working memory and cognitive processing, is to measure the amount of time between when a switch is made to a new task and the first action is taken by the hu-man on this new task In their research, they measure the time between when the working environment causes a switch to a new task and when the human takes their first action on the new task They have found that switch costs in complicated multi-tasking environments can be on the order of two to four sec-onds; this amount can impose serious limitations on the number of robots that

a single human can manage

It is important to note that Altmann and Trafton’s research has included

a study of signaling the human of an impending interrupt They have found that signaling reduces switch costs (on the order of two seconds) because it allows people to prepare to resume the primary task when the interruption is

completed Their experiments suggest that people’s preparation includes both retrospective and prospective memory components

Unfortunately, the experiment approach used by Altmann and Trafton does not go far enough into naturalistic human-robot interaction domains to suit our needs The primary limitation is the primary-secondary task nature of their experiments Multi-robot control will not always have a primary robot with

a set of secondary robots A second limitation is the use of “first action after task resumption” as a metric for switch costs In the multi-robot domain, a person may not take any action when a task is resumed either because the robot

is still performing satisfactorily or because the human does not have enough situation awareness to properly determine that an action is required Despite its limitations, we adopt the primary-secondary task approach to establish that switch costs can be high However, we use a change detection approach for measuring recovery time

The preceding discussion has assumed that interaction time captures all in-teraction costs Unfortunately, there is a cost associated with switching be-tween multiple activities This cost has been studied extensively under the name of “task switching” in the cognitive science literature, but has received considerably less attention in the human-robot interaction literature

The problem with the preceding discussion of fan-out and feasibility is that

it assumes no interference effects between tasks Olsen noted this limitation in the definition of interaction time, and used the more general notion of interac-tion effort to include the actual time spent interacting with the robot as well as the time required to switch between tasks (Olsen, Jr and Wood, 2004, Olsen,

Jr et al., 2004) Unfortunately, this work did not research the costs of task switching and does not, therefore, allow us to make predictions about the fea-sibility of a human-robot system or diagnose the problems with a given system Switch costs are important in domains where a human must manage multi-ple robots because managing multimulti-ple robots entails the need to switch between these robots If the costs due to switching are significant, then the number of robots that can be managed dramatically decreases As autonomy increases and interfaces become better, switch costs may become the bottleneck which limits the number of robots that a single human can manage For example, suppose that a human is managing a set of robots that can be neglected for no more than 20 seconds If the human is asked to manage a team of robots where each robot requires no more than five seconds of interaction time per interac-tion event, then the human can manage no more than five robots If, however, switching between robots comes at a cost of, say, three extra seconds, then rotating between the five robots requires 15 seconds of switch cost which

con-sumes 75% of the total neglect time without actually performing any useful interaction This switch cost makes interaction effort jump from five seconds

to eight seconds, and means that the human can manage at most three robots

Formally, we denote the cost to switch between task i and task j as SC (i, j)

where this cost has units of time; large times mean high costs When a human

begins to neglect task k, the feasibility constraint in Equation (1) demands that all the interaction times of all other tasks j =  k can be accomplished during the neglect time for task k Since the experiment results strongly indicate that the

switch cost can vary substantially depending on the type of secondary task, it

is necessary to address how feasibility is affected by switch costs

To this end, it is necessary to distinguish between what constitutes an in-teraction time and what constitutes a switch cost The precise differentiation

between these terms could be a topic of heated debate, but for our purposes we will use operational definitions of the two terms that are compatible with the

experiment The term switch cost denotes the amount of time between when

one task ends and the operator demonstrates an ability to detect changes in the

environment This relies on the association between the change detection

prob-lem and situation awareness which we will discuss shortly A good description

of change detection can be found in Rensink (Rensink, 2002) “Change de-tection is the apprehension of change in the world around us The ability to detect change is important in much of our everyday lifefor example, noticing

a person entering the room, coping with traffic, or watching a kitten as it runs

under a table.” The term interaction time denotes the amount of time that is

required for the operator to bring the robot to peak performance after a level

of situation awareness is obtained that is high enough to detect changes in the environment

The experiment results presented below indicate the the time to detect changes

is sensitive to the type of tasks involved Therefore, the feasibility equation must be modified to account for the effects of changes The problem with do-ing so is that the total switch costs depends on the order in which the tasks are performed Addressing this issue completely is an area of future work For now, we adopt the most constraining definition and consider worst case switch costs

LetS(i) = {1,2, ,i − 1,i + 1, ,n} denote the set of all tasks different from task i Consider the set of permutations over this set,

P(i) = {permutations over S(i)},

and letπ denote a particular permutation within this set; π(1) denotes the first element in the permutation, and so on Let SC∗

i denote the largest possible

cumulative switch cost for a given set of tasks inS(i), that is

SC∗

i = max

π∈P(i)SC(i,π(1)) + n∑−2

k=1

SC(π(k),π(k + 1)) + SC(π(n − 1),i).

Note that we have included both the cost to switch from primary task i to

the first permutation task in the permutation and the cost to switch from the last task in the permutation to the primary task This is necessary because beginning the set of secondary tasks comes at a cost, and resuming the primary task also comes at a cost

Feasibility of the team is then given by

T is  feasible if∀i NT T i ≥ ∑ j =  i IT T j+ SC∗ i (2)

If, for all tasks, the neglect time exceeds the sum of the interaction times plus the worst case switch costs, then the team is feasible

In the next section, we describe an experiment that demonstrates that switch costs can be high enough (on the order of 5 to 10 seconds) to merit their con-sideration in determining team feasibility We also show that type of switch is also an important consideration because various types of secondary tasks have substantially different switch costs

We adopt the primary task/secondary task formulation in the experiment The primary task was to control a simulated ground robot using a conventional display This display included a video feed from the robot and a plan-view map

of the environment The environment consisted of treeless grass with multiple folds and hills

Throughout the environment, there were ten geometric shapes randomly dis-persed Subjects used a gamepad to teleoperate the robot to within a meter of the geometric shapes They then cycled through a set of geometric shapes (sphere, cube, or tetrahedron) by repeatedly clicking on one of the gamepad’s buttons The selected categorization was shown on the map view by placing a corresponding symbol on the map

We adopted a change detection approach to indirectly measuring situation awareness On approximately 50% of the trials (so that people will not be cued

of a change), one of the geometric shapes changes or disappears from the cam-era view while the subject is performing the secondary task The subject will

be informed that this may occur in some trials, and will be asked to “alert their boss” that something has changed as soon as they detect a change Alerting consisted of clicking one of two buttons to indicate the presence or absence of

a change

Figure 1. The before and after shots from the task switching experiment.

The experiment setup is illustrated in Figure 1 using screen shots from the experiment The top figure shows the camera view (left) and map view (right) along with the geometric shapes (camera view) and their correct categorization (map view) In the top figure, a sphere is prominently displayed in the camera view The corresponding sphere is circled in the map view to highlight its location in the map (The circle is not shown to the subject, but is included in the figure to help highlight an important aspect of the experiment.)

The bottom figure shows the same world after the subject returns from the secondary task Note how the sphere has disappeared from the camera view Note further that the map view retains the information about the sphere Thus, although the camera view has changed during the secondary task, the map view has the same information before and after

The subject’s task is to indicate that something changed while they were performing the secondary task If a person has a good situation awareness after the secondary task, then they should be able to quickly consult the camera view

to detect a change If the situation awareness is poor, then they will need to compare the camera and map views to determine if something has changed This forces the subject to “reconstruct” situation awareness and takes longer to perform Secondary tasks that interfere with situation awareness should require the subject to take a longer time to recover

Measuring the reaction time to detect this change after the task is resumed is

an estimate of situation awareness The time required to detect a change is an estimate of the time to achieve Endsley’s “level 1” situation awareness

(End-sley, 1997) Differences in times caused by various secondary tasks indicate different switch costs

When subjects indicate that a change has occurred, we inform them whether they were correct If they correctly identified a change, we require the subject

to generate a report of what things have changed The time to generate this report and the accuracy of the report will form a second measure of switch costs that we will analyze in future work The nature of the report will be an updated categorization of all geometric shapes in the robot’s camera field of view This report will be made by removing missing shapes and recategorizing shapes that have changed This report will be made by requiring the subject

to (a) click on the shape in the map view that has disappeared if required, and (b) drive to the shapes that have changed or been added and (re)categorize them

We experimented with four different types of secondary tasks

Blank screen: the screen goes blank for a preselected period of time Tone counting: subjects are given a target tone and asked to count the number of times this target tone occurs in a two tone sequence At the end of the sequence, subjects report the number of tones by clicking on the appropriate number in the display

Vehicle counting: subjects are asked to watch a video from a camera mounted on a real or simulated UAV, and to count the number of unique cars observed from the UAV At the end of the sequence, subjects report the number of vehicles by clicking on the appropriate number in the display

Spatial reasoning (tetris): subjects are asked to play a game of tetris for

a preselected period of time

The blank screen serves as the baseline, both tone counting and vehicle count-ing place some burden on attention and workcount-ing memory, and both vehicle counting and spatial reasoning place some burden on visual short term mem-ory Secondary tasks last between 10 seconds and 40 seconds Tasks are presented in a balanced randomized schedule, and changes are generated ran-domly

For this paper, we estimate switch costs by measuring the amount of time between when the secondary task ends and when the subject pushes the but-ton indicating that a change has occurred Results are presented only for those conditions where a change occurred and the subject correctly identified the

change Future work should carefully address error rates as well as the sensi-tivity of these error rates and switch costs to the frequency with which changes occur

Figure 2. Average switch costs as a function of task with 20% confidence intervals.

The results of the experiment are shown in Figure 2 which displays the aver-age switch costs across five subjects and seven one hour experiment sessions Also shown are the 20% confidence intervals

Two important things are worth noting First, note that the average values for the switch costs range from over five seconds to just over twelve seconds This is important because it indicates that switch costs can be very large This indicates that an evaluation of the feasibility of a multi-robot team with inde-pendent robots should include an anaylsis of switch costs

Second, note that the switch costs associated with the UAV are twice as large

as the switch costs associated with the tone counting and blank screen This indicates that there is a potentially large difference in the switch costs between various types of tasks In fact, a two-sided t-test indicates that the different tasks all have statistically significant differences at the 20% level (or below) except for the difference between tone counting and tetris which appears to not be statistically significant This data must be taken with a grain of salt