The international journal of advanced manufacturing technology, tập 59, số 9 12, 2012

ORIGINAL ARTICLETangible user interface of digital products in multi-displays Jae Yeol Lee&Min Seok Kim&Jae Sung Kim& Sang Min Lee Received: 21 February 2011 / Accepted: 7 August 2011 /

ORIGINAL ARTICLE

Tangible user interface of digital products in multi-displays

Jae Yeol Lee&Min Seok Kim&Jae Sung Kim&

Sang Min Lee

Received: 21 February 2011 / Accepted: 7 August 2011 / Published online: 8 September 2011

# Springer-Verlag London Limited 2011

Abstract Early attempts at supporting interaction with

digital products for the design review were based on CAD

and virtual reality (VR) systems However, it is not easy to

build a virtual environment of fine quality and to acquire

tangible and natural interactions with VR-based systems,

which are expensive and inflexible to be adaptable to

typical offices or collaboration rooms We present a new

method for supporting tangible interactions with digital

products in immersive and non-immersive multi-display

environments with inexpensive and convenient optical

tracking The provided environment is more intuitive and

natural to help participants to review digital products

through their functional behavior modeling and evaluation

Although vision-based image processing has been widely

used for interaction tracking, it cannot be effectively used

under a low illumination condition since most of the

collaborations and meetings take place in such conditions

To overcome this problem, the proposed approach utilizes

Wiimote™ for infrared (IR)-based optical tracking and for

capturing user's interactions and intents Thus, users can

easily manipulate and evaluate digital products with

inexpensive tools called IR tangibles in more natural and

user-friendly environments such as large displays,

tab-letops, and situational displays in typical offices and

workspaces Furthermore, the multi-view manager is

suggested to effectively support multi-views of digital products among participants by providing public and private views We will show the effectiveness and useful-ness of the proposed approach by demonstrating several implementation results and by evaluating user study of the proposed approach

Keywords Tangible user interface Human–computer interaction Multi-display Wiimote

IR-based optical tracking

1 Introduction

Design review of digital products is required to test their functionalities and characteristics, which can result in higher stability, better maintainability, and less potential errors of the products before production Shortening of development cycles demands the use of an intelligent interface for testing human–computer interactions of digital products and an efficient method for evaluating their functional behaviors [1] Many attempts at supporting digital product design and evaluation were based on traditional visual environments such as virtual reality (VR) and cave automatic virtual environment (CAVE) However, these environments are very expensive and not flexible Meanwhile, as displays increase in size and resolution while decreasing in price, various types of inexpensive multi-displays will be soon available such as situated displays or tabletops providing high-resolution visual output These large and high-resolution displays will provide the possibility of working up close with detailed information in typical office or workplace environments [2] For example, when people work collaboratively with a

J Y Lee ( *):M S Kim

Chonnam National University,

300 Yongbong-dong, Buk-gu,

Gwangju 500-757, South Korea

e-mail: jaeyeol@chonnam.ac.kr

J S Kim:S M Lee

KISTI,

Daejeon, South Korea

DOI 10.1007/s00170-011-3575-0

digital product, its related information is often placed on a

wall or tabletop, where it is easy to view, annotate, and

organize The information can be rearranged, annotated,

and refined in order to evaluate the functional behavior of

the digital product to solve a problem For this reason,

multi-displays have been considered to be used for

collaboration, robot, engineering, and realistic visualization

and interaction [3–7] Furthermore, the availability of smart

devices and their interactions have increased dramatically

over the last decade, which provides new possibilities of

interacting techniques such as multi-touch and sensor-based

interactions [8,9]

Usually, a single-user design task in a multi-display

environment mainly requires the visualization capabilities

of a large display and demands long hours Similarly, in a

collaborative discussion where users gather around a large

conference room table, various digital contents frequently

need to be displayed on a large screen for others to see

However, it is not sufficient to simply move existing

graphical user interfaces onto multi-displays For example,

large displays afford different types of interactions than

workstations or desktops for several key reasons The large

visual display can be used to work with large quantities of

simultaneously visible material On the other hand,

inter-action is directly on the screen with a pen-like device or by

touch rather than with a keyboard and an indirect pointing

device Also, people often work together at a wall,

interweaving social and computer interactions However,

direct manipulation through pointing and clicking is still

considerably the dominant interaction paradigm in

conven-tional user interfaces [10,11]

Most of the proposals in the previous research works

require expensive display and considerable space [12] In

addition, they cannot be effectively applied to another type

of display to interact with digital products in a typical office

or workspace Furthermore, in order to support

user-oriented and tangible interactions, a vision-based tracking

has been widely used But, the vision tracking cannot be

effectively used under a low illumination condition Note

that most of the collaborations and meetings take place in

such conditions Another related problem with most vision

algorithms is the difficulty experienced when segmenting

objects under varying lighting conditions and shadows and

requires a large amount of processing time [13]

This paper presents a new method for supporting

tangible interactions with digital products in immersive

and non-immersive multi-display environments with

inex-pensive and convenient optical tracking It can be adaptable

to a large set of multi-displays such as a projected display, a

tabletop, and a situated remote display to perform

multi-touch interactions with digital products In addition, the

proposed approach makes participants review and evaluate

the functional behavior of digital products efficiently with

infrared (IR) tangibles To overcome the generic problem of the vision-based image processing approach, the proposed approach utilizes Wiimote [14] for optical tracking and for capturing user's various interactions and intentions effec-tively This approach can support the tangible user interface

of digital products in immersive and non-immersive environments where users can easily manipulate and evaluate digital products, providing more effective and user-friendly circumstances Moreover, the proposed ap-proach can be easily set up to run various environments without any difficulty such as large displays, tabletops, and situational displays in typical offices and workspaces The multi-view manager is suggested to effectively support multi-views of digital products among participants, which provides public and private views of the shared digital product We will show the effectiveness and usefulness of the proposed approach by demonstrating several imple-mentation results and by evaluating user study of the proposed approach Section 2 presents previous work Section 3 explains tangible user interactions in multi-displays with IR tangibles and IR tracking Section 4 proposes how to effectively support interactions with digital products in multi-displays for design review Section 5 presents implementation results Finally, Section 6 con-cludes with some remarks

2 Previous work

Early attempts at supporting interactions with digital products for the design review were based on computer-aided design (CAD) and VR systems Powerful and expensive tools including stereoscopic display systems, head-mounted display, data gloves, and haptic devices have been utilized and combined to construct virtual prototyping systems that provide realistic display of digital products and offer various interaction and evaluation methods[1, 15] Since it is not easy to build a virtual environment of fine quality and to acquire tangible and natural interaction with VR-based systems, many alternative solutions have been proposed

Another type of VR known as augmented reality (AR) is considered to be an excellent user interface Interacting in

AR environments can provide convincing feedback to the user by giving the impression of natural interaction since virtual scenes are superimposed on physical models in a realistic appearance Thus, AR is considered to complement

VR by providing an intuitive interface to a 3D information space embedded within physical reality Lee et al [16] proposed how to provide car maintenance services using

AR in ubiquitous and mobile environments Christian et al [17] suggested virtual and mixed reality interfaces for e-learning which can be applied to aircraft maintenance

Regenbrecht et al [18] proposed a collaborative augmented

reality system that featured face-to-face communication,

collaborative viewing and manipulation of 3D models, and

seamless access to 3D desktop applications within the

shared 3D space However, AR depends on the marker

tracking such that a vision-based image processing is

intensively used This is a severe drawback for supporting

realistic visualization in a large display or tabletop display

since the image processing deteriorates when the resolution

increases In addition, it is very difficult to interact directly

with digital objects in AR environments since it is not

convenient and natural to interact with them through

marker-based paddles which are widely used in AR

applications

Meanwhile, to support effective and natural interactions

with digital objects in various displays, vision-based image

processing techniques have been used [4, 5, 7] The

approach in [7] tracks a laser pointer and uses it as an

input device which facilitates interactions from a distance

While the laser pointer provides a very intuitive way to

randomly access any portion of the wall-sized display, the

natural shaking of the human hand makes it difficult to use

for precise target acquisition tasks, particularly for smaller

targets The VisionWand [19] uses simple computer vision

algorithms to track the colored tips of a simple plastic wand

to interact with large wall displays both close up and from a

distance A variety of postures and gestures are recognized

in order to perform an array of interactions A number of

other systems use vision to track bare, unmarked hands

using one or more cameras, with simple hand gestures for

arms-reach interactions [20, 21] Dynamo was proposed

and implemented for a communal multiuser interactive

surface [21] The surface supported the cooperative sharing

and exchange of a wide range of media that can be brought

to the surface by users outside of their familiar

organiza-tional settings

Recently, mobile devices have been considered as

com-plementing tools to interact with virtual objects in ubiquitous

environments Much work has attempted to bridge the gap

between personal devices and multi-displays Many

researches tried to augment mobile devices of limited

capabilities with enhanced sensing or communication

capa-bilities such as remote controller [8, 9] However, this

approach is hard to effectively provide visual information

to multi-users To overcome this limitation and integrate the

interaction between smartphones and multi-displays, it is

considered to provide visually controlled interaction Few

research work has dealt with how to effectively support

visually controlled views to support individual and

cooper-ative interactions for collaborcooper-ative design review in

multi-display and smartphone environment

Although various ways have been proposed to support

the visualization and evaluation of digital products, more

research is still needed in the following aspects The interaction should be more intuitive and natural to help participants in the digital product design to make a product

of interest more complete and malfunction free before production The environment should be available at low cost without strong restriction of its accessibility and should

be adaptable to various environments and displays Moreover, for effective evaluation of the digital product, we need to define its functional behavior through forms, functions, and interactions In this paper, we address these aspects by proposing a natural interaction approach in multi-displays using convenient tangible interfaces Note that the proposed approach can be easily adaptable to various environments with low cost and much convenience

3 Proposed approach

This section explains how to effectively support tangible interactions with digital products in multi-displays using low-cost IR tracking, which provides much convenience and effectiveness for the design review of digital products

It overviews the proposed system Then, it explains tangible interfaces for directly interacting with digital products

3.1 System overview

The proposed approach consists of four layers: (1) tangible interface layer, (2) resource layer, (3) collaboration and evaluation layer, and (4) visualization layer as shown in Fig 1 The tangible interface layer supports tracking of IR tangibles and interpreting user intent through analyzing IR tangible inputs The result is used for natural and direct interactions with digital products in multi-displays such as large display, tabletop, and remote display For natural user interactions, IR tangibles are devised for the direct multi-touch interfaces To transform the user space to the visualization space, the perspective transform is calculated The collaboration and evaluation layer manages multi-views among participants and generates graphic scenes adaptable to participants' devices and contexts In particu-lar, it provides private and public views of the shared digital product among participants Each view is systematically generated by the multi-view manager which controls all the views of participants and generates adaptive views consid-ering the display context and user's preference To support the design review of digital products, the finite-state machine (FSM)-based functional model is linked to the actions that occur during the interaction with digital products [1, 22] According to the user's actions and functional evaluation, the adaptive rendering of the digital product is executed and the generated scene is sent to the reviewer In addition, according to the actions related to

FSM, the system loads and renders corresponding virtual

objects or guides users to manipulate them on

multi-displays All the necessary digital models and functional models are stored in the resource layer Thus, participants Fig 1 System overview

Fig 2 Overall process for tangible interactions with digital products in multi-displays

can use multi-displays for collaborative and private

inter-actions for the design review and discussion

The overall process involves three stages: (1) digital

product design, (2) tangible interaction in multi-displays,

and (3) collaboration and evaluation In the digital product

design stage, the product designer creates a product model

using a commercial CAD system, and considers design

specification and customers' requirements during the

prod-uct design that correspond to the overall functional

behavior and interface However, this consideration is

limited and subjective, and therefore cannot guarantee the

complete functional evaluation of the digital product

When the product design is completed, it is used to

construct the virtual product model in the interaction stage

The tangible digital model consists of geometry, assembly

relation, and other attributes for visualization and

interac-tion In addition, its functional model and related

multime-dia contents are generated and created Eventually, the

multimedia contents will be overlaid into the virtual model

in multi-display environments The FSM-based functional

model is linked to actions that occurred during the

interaction with the product model Each action can be

linked to a multimedia content visualization and interaction

such as menu manipulation, movie and music playing, and

color change

Then, participant(s) can evaluate and simulate the

functional behavior of the digital product by tangible

interaction in multi-displays at the collaboration and evaluation stage To support a new interface that is able to directly manipulate virtual objects in multi-touch aspect IR tangibles are provided for cost-effective and convenient interactions Two Wiimotes are used to effectively track IR tangibles fast and robustly under low illumination con-ditions Moreover, multi-views are generated since partic-ipants have different views as well as common view of the digital product To evaluate the functional behavior of the product, an FSM is embedded into the tangible interaction and visualization The concrete execution according to each action or activity is conducted during the functional simulation Finally, participants from different areas share their ideas and collaborate to find design problems and revise the overall shape and its functional behavior according the result of the design review

Figure 2 shows a tangible interaction with a digital mobile phone and its functional evaluation in multi-displays based on the above overall process Firstly, the designer designs a digital product of a new mobile phone Then, its functional behavior is modeled with an FSM, and communications between the digital model and the FSM are made by IR tangible-based interaction Each interaction plays a role in linking an action in the FSM and the actual action in the digital model Finally, its virtual model is displayed in multi-displays as a private view or common view, and thus, the user can easily evaluate its functional

Fig 3 Wiimote and infrared

camera: a Wiimote, b IR

camera module in Wiimote

Fig 4 IR tangibles interfaces: a tangible wand, b tangible cube, c tangible ring

behavior as well as the appearance on the prototype During

the evaluation, corresponding virtual objects and

multime-dia contents are overlaid on the digital phone model to help

the evaluation When the collaboration space is

synchro-nized, participants perform collaboration and evaluate the

functional properties using tangible interfaces Normally,

the space shares the visualization model, functional model,

and related multimedia contents Thus, the proposed

tangible interface and visualization can make participants

experiment more touchable and tangible feelings compared

to existing virtual model visualization and its simulation

[15,23,24]

3.2 IR tangibles and multi-displays

Wiimote shown in Fig 3 plays the main role in efficient

and robust tracking of IR tangibles in multi-display

environments It integrates an in-built infrared camera with

on-chip processing and accelerometers, and supports

Blue-tooth communication This characteristic makes it possible

to communicate with external hardware that supports

Bluetooth, while several open-source libraries are available

to capture and process the derived information In

partic-ular, the proposed approach is very flexible since it is easily

adaptable to various displays such as large displays,

tabletops, desktops, and situated displays Moreover, it is

robust in a low illumination condition where most of the

collaborations and discussions occur, since it utilizes

infrared optical tracking (rather than vision-based tracking)

which has the advantage of an enhanced sense of presence and increased interaction among participants Furthermore, the environment setup is quite simple as it is sufficient to mount two Wiimotes on a portable support in front of multi-displays in a typical office or workplace The information derived from the Wiimote camera is used to track an IR tangible and to generate graphics corresponding

to the movements of the user These data are successively shared across multi-display environments

An IR tangible can be easily created from IR light-emitting diodes (LEDs), or alternatively by shining IR light generated via an LED array on a reflective marker attached

to the participant's hand Figure4shows tangible interfaces made by IR LEDs which can be effectively used depending

on the type of display and application The 3D coordinates

of the IR tangible are calculated by a stereo vision technique Using a real-time optical tracking algorithm that simultaneously tracks multiple IR tangibles, we can explore techniques that allow for direct manipulation on multi-displays using multi-touch gestures

By pointing a Wiimote at a projection screen or large display, we can create different types of interactive multi-displays as shown in Fig.5 Since the Wiimote can track up

to four points, up to four pens can be used In particular, using reflective tape and the LED array shown in Fig.6a and

b, we can control and interact with digital products remotely This allows us to interact with various applications simply by waving one's hands in the air, similarly to the interaction as shown in Fig.5c Fig.5demonstrates a variety of interaction Fig 5 Multi-displays and tangible interactions: a large display: direct control, b tabletop, c large display: remote control

Fig 6 Interfaces for remote

control: a reflective tape, b

LED array

techniques that exploit the affordability of the proposed

approach, resulting in effective multi-displays such as large

displays, table tops, and remote displays There are also

circumstances where users cannot easily approach the

display and can interact only from a distance Our work

also investigates potential techniques for pointing and

clicking from a distance using the proposed approach as

shown in Fig.5c This eliminates issues related to acquiring

a physical input device, and transitions very fluidly to up

close touch screen interaction

To support interactions with digital products through IR

tangible interfaces, we need to convert the coordinates of

the IR tangibles to those in the computer that actually

manipulate objects For example, the coordinate (x, y) from

the IR LED should be mapped to the coordinate (X, Y) in

the virtual world of digital products by the perspective

transform as shown in Fig 7 In other words, we need to

find a transform that maps one arbitrary 2D quadrilateral into another [25]

A property of the perspective transform is its ability to map straight lines to straight lines Thus, given the coordinates of the four corners of the first quadrilateral, and the coordinates of the four corners of the second quadrilateral, the task is to compute the perspective transform that maps a new point in the first quadrilateral onto the appropriate position on the second quadrilateral Let us assume that the perspective transform is written as

X = Hx, where x is the vector of multi-display coordinates, and X is the vector of the virtual world of digital products

We can write this form in more detail as:

XW YW W

2 6

3 7

5 ¼

a b c

d e f

g h 1

2 6

3

7 xy 1

2 6

3 7

5 where W ¼ gx þ hy þ 1

We can rewrite the above equations as follows:

X ¼axþ by þ c

gxþ hy þ 1; Y ¼

dxþ ey þ f

gxþ hy þ 1

X ¼ ax þ by þ c  gxX  hXy

Y ¼ dx þ ey þ f  gxY  hyY Since we need to find H which contains eight unknown variables, we need four points that are already known We Fig 7 Perspective transform

Fig 8 Tangible user interfaces

using IR tangibles: a select, b

move, c rotate, d scale

need a calibration step to obtain four points that map (x, y)

into (X, Y) from the user before interactions so that we can

find all the unknown variables in H as follows:

X 1

Y 1

X 2

Y 2

X 3

Y 3

X 4

Y 4

2

6

6

6

6

6

6

6

4

3

7

7

7

7

7

7

7

5

¼

x1 y1 1 0 0 0 x1X 1 X 1y1

0 0 0 x1 y1 1 x1Y1 y1Y1

x2 y2 1 0 0 0 x2X 2 X 2y2

0 0 0 x2 y2 1 x2Y2 y2Y2

x3 y3 1 0 0 0 x3Y3 X 3y3

0 0 0 x3 y3 1 x3Y3 y3Y3

x4 y4 1 0 0 0 x4X 4 X 4y4

0 0 0 x4 y4 1 x4Y4 y4Y4

2

6

6

6

6

6

6

6

4

3 7 7 7 7 7 7 7 5

a b c d e f g h

2 6 6 6 6 6 6 6 4

3 7 7 7 7 7 7 7 5

3.3 User interfaces with IR tangibles

The user manipulates multiple IR tangibles to select, move,

rotate, and scale multimedia and 3D digital products The

system traces the location of IR tangibles while being moved as shown in Fig.8

& Select: When the IR tangible turns on from the off status and the location of the IR tangible is close to the display, it

is considered that the user tries to select an object

& Rotate: When one of the two IR tangibles is rotating around the other, the selected object is rotated

& Scale: When the distance between the two IR tangibles becomes significant, the selected object is zoomed out

On the other hand, when it is closer, it is zoomed in

& Translate: When the IR tangible that selects an object moves, the selected object moves along the IR tangible

To support the design review and functional evaluation

of a digital product, visualization information as well as its functional model and related multimedia contents are generated and visualized The multimedia contents will be overlaid into the digital product on multi-displays To

a

b

c

Fig 9 Tangible user interface of digital products: a rotating, b changing attributes of digital product, c playing multi-media on the digital product

evaluate the functional behavior of the product, an FSM is

embedded into the tangible interaction and visualization

The concrete execution according to each action or activity

is conducted during the functional simulation Finally,

participants from different areas share their ideas and

collaborate to find design problems and revise the overall

shape and its functional behavior according to the result of the

design review As shown in Fig 9, interactions include

playing the multimedia and changing the attribute of a digital

product such as changing color or texture Based on the

above manipulation operators, the user can perform design

review tasks in immersive and non-immersive multi-display

environments effectively

4 Collaborative design review of digital products

in multi-displays

This section explains how the proposed approach can be

utilized for the effective design review of digital products

among participants in multi-displays The user can

manip-ulate multiple IR tangibles in front of multi-displays for

interacting with digital products The interaction is analyzed

and fed into a design view and visualization of digital

products in a large display, tabletop, and remote display

4.1 Functional evaluation

During the design review, modeling and simulation of digital products are essential to test their functionalities and characteristics, which can result in higher stability, better maintainability, and less potential errors of the products before manufacturing [23, 24] To effectively support the design review and collaboration in co-location, VR and AR have been widely used However, there is no cost-effective way to support a tangible user interface of digital products in multi-displays because most of the previous research work requires expensive VR systems and inflexible visualization environments [1] For this reason, we adopt FSM to simulate the functional behavior of a digital product [22]

Every digital product has some part components that are involved in the interaction between the user and the product These include switches, buttons, sliders, indica-tors, displays, timers, and speakers They are called objects making up the basic building blocks of the functional simulation Every object has a pre-defined set of properties and functions that describe everything it can do in a real-time situation The overall behavior of a digital product can

be broken down into separate units of behavior, which are called states The state of the product can be changed to another state as shown in Fig.10a[15] This is called state

Fig 10 State transition chart: a

concept of state transition, b

state transition of a mobile

de-vice

transition Every state transition is triggered by one or more

events associated with it Some tasks called actions can be

performed before transition to a new state In order to

define the actual behavior of the product, all the tasks

performed in each state are specified These tasks are called

activities They only occur when their state becomes active

Actions and activities are constructed using the objects'

properties and functions Each action or activity consists of

a set of statements Each statement can be the assignment of

some value to a variable, the calling of a function of an object,

or a composite statement with a conditional statement The functional behavior model for tangible interactions is used to generate a state transition chart, which represents all the states and the possible state transitions between them Figure10b shows a state-transition chart for a mobile phone [1] When the user creates an input event using IR tangibles

in multi-displays, the proposed approach checks whether or not the event is related to the functional behavior of the

Fig 12 Interaction process for the design review in multi-displays

Fig 11 Multi-view manager