Handbook of Multimedia for Digital Entertainment and Arts- P24 ppt

Facilitator knowledge of the potentials of the digital media is essential touser-experience, successful guidance targeted as via mediating technology, anduser self-determined entertainme

resonance This aesthetic is an inner resonance recognized from achieved experience Having experienced a vocation as a stage performer I relate to emotionsachieved through a satisfying entertainment of an audience Thus, in the therapythe user entertains him or her-self through the mediating interactive digital mediasystem This can be from controlling a game or creating art, i.e music making orpainting via digital tools, and is evident as FUN resulting from the created situ-ations Facilitator knowledge of the potentials of the digital media is essential touser-experience, successful guidance (targeted as via mediating technology), anduser self-determined entertainment.

user-The same system that was created specific for the rehabilitation training wasused by me to perform abstract expressionistic stage art and interactive installations

at various large exhibitions These include ongoing tours in Museums of ModernArt as well as at International-National Cultural events (e g., Danish NeWave, NewYork 1999; Cultural Olympiad 1996, 2000; and European Cultural Capital events

1996, 2000) Underground events were as informative to the concept as the largervenues (see next section) Insight was gained from these situations By combiningself-reflection with collaborative inter-/multi-disciplinary reflective analysis of thehuman performance situations a significant input to designing the system, facilitat-ing sessions, and evaluating outcomes evolved

Underground Non-formal Learning

An example of how learning was apparent from the entertainment and arts is where

I directed, produced and performed in a one-man-show realized as the inauguralAarhus Festival Fringe (‘Festuge’) This 1999 event took place in an emptied storagerooms that were adjacent to my sizable interactive room installation titled Circle ofInteractive Light (COIL), hosted by the Radisson SAS Scandinavia Hotel, Aarhus,Denmark The Festival committee had declined my proposal to bring together Eastand West German artists to feature alongside North and South Korean artists as toopolitically sensitive The festival theme was ‘THE WALL, Ten Years After’ – and

my proposal involved artists of distinction from both sides of the Berlin Wall andthe DMZ to discus and showcase how political issues affect art and artists lives.Understandable that the committee declined!

The eventual “FESTUGE FRINGE” protest performance featured my digital teractive media in the form of the motion sensitive system utilized in various wayswith projections inside and outside the human body The final section of the perfor-mance was where I used fourteen infrared sensors to control one sound and a library

in-of image manipulations The purpose was to explore non-control and subliminalperformance by enabling the feedback to control me and this was done by mappingthe data to maximum parameter control of a specific aspect of the sound envelope

In four of the twelve performances I achieved what I refer to as non-control withsubliminal interaction This was where I, as performer, experienced being as onewith the feedback stimuli If conscious intent or query was involved, the experience

696 A.L Brooksand higher-state of interaction (or inbetweenness – see Kidd) was lost I relate thistargeted non-control and interaction to the therapy work with profoundly disabledwhere often the user’s abilities are of a non-controlled state so my learning experi-ences as performer/user assist by informing my role as facilitator.

Following the performances the wall would open to exhibit the technology used,and then audience walked through the wall into the COIL exhibition space wherethey could experience the virtual interactive spaces and debate over a drink in thecaf´e I had built for the occasion The piece titled “Behind the Wall” targeted beyondBerlin and DMZ issues as it reflected on human barriers across ages and cultures –and especially committees

The next section introduces the ArtAbilitation2workshops from 2007 and 2008.The ArtAbilitation movement has evolved from the SoundScapes work It is wheredigital interactive media is used to create an entertaining user-experience

ArtAbilitation Workshops, Casa da Musica, Porto, Portugal

The “Ao Alcance de Todos - Within Everyone’s Reach” festival was hosted at Casa

da M´usica (Figure 13), Porto, Portugal in April 2008 Eight one hour workshopswere held with 144 disabled children and adults attending with caregivers Theworkshops were created in a room 238 square meters floor area and approximately

20 meters high Additionally, a symposium for professionals was hosted ately following the workshops for 35 professionals (international, national, regional,and local // social workers, psychologists, researchers, teachers, and students : : :)

immedi many of whom had attended the workshops This event was a continuation ofthe previous year’s inaugural event where the theme was ‘Music, Technology andDisability’, and where six workshops were attended by 91 attendees including 61from special care institutes, of which 39 had profound disability 30 student musicteachers also attended A local crew assists in the realization of the workshops (seePetersson and Brooks 2007;Brooks 2008)

This section does not detail the workshops but rather it exemplifies the designissues where digital interactive media, entertainment and the arts are combined toresult in various user-experiences that inform the ongoing research and refinement

of the acknowledged international ArtAbilitation movement (Wiederhold 2007)

I begin by giving an overview of commonalities in designing both workshops.The 2007 workshop is then detailed in more depth followed by the 2008 workshop.Both workshop designs targeted participant learning from interactive experienceswhere active participation formed a context for meaning Offered were opportuni-ties for augmenting learning and awareness Observations were of self-determinedactive participation that established goals that in turn engaged interest, curiosityand play In this way, motivation to achieve goals is considered intrinsic rather thanextrinsic, which can result from an activity, task or goal introduced by someone else,

2 http://www.ArtAbilitation.net

Fig 13 Casa da Musica, Porto, Portugal (photo with permission Casa da Musica)

and or with possibly over zealous guidance/facilitation Contextual design thus came a way to configure learning resources and interaction (Kress & van Leeuwen,2001) Exploration, play and transformation were also targeted in both events suchthat each participant had an ‘action-stillness’ profile that evolved through inter-actions Each cycle of activity was considered a new creation that contributed toform patterning of actions resulting from the activity Each activity was designed

be-to increment challenges of confrontation The resultant ‘play’ and created ity’ scenarios involved manipulation of provided tools The manipulation required

‘creativ-a degree of competence th‘creativ-at w‘creativ-as le‘creativ-arnt through explor‘creativ-ation of the tool’s tr‘creativ-aits(Bruner 1972)

Absence of negative consequences seemingly encouraged participant free pressive explorations, which in turn, can over time result in development of un-employed skill (Beach, 1942) Such development is significant even if it is at amicro scale, i.e micro-development (Yan and Fischer 2002) Responsive environ-ment composition and learning process/outcome evolve as the assessment focusinstrument of observers Realized was new learning spaces and approaches to learn-ing and rehabilitation through emphasizing user’s creation of meaning via seriousplay The approach is that no aspect of the learning process and outcome is taken forgranted Rather it is formulated into play and creative activities that are inherent toe.g., games and art making The activities of play and creativity conceal the embed-ded learning and training involved for the user An emphasis is on a supplementingtool for traditional practices rather than a replacement In this sense, learning is at a

ex-‘subliminal’ level for the user as he or she engages in the responsive environment.Thus, motivation is optimized through action and stillness cycles where the user

698 A.L Brooks

Fig 14 Sala de Ensaio 1, Casa da M´usica, Oporto, Portugal (C D camera) – ( Brooks 2008 )

iteratively explores and transforms the feedback media This process containschoices and decisions that indicate learning, e.g in the form of increased repertoire

of expressions, changes of skills, and new patterns of social interaction (Brooks

2008)

In the 2007 workshop a private (2) and a public (1) area were designed – seeSala de Ensaio diagram (Figure14) This was inspired from the evaluation consul-tations mentioned earlier in this chapter where a caregiver experienced that a privatespace, without any intervention, was desired before a user felt comfortable enough toexpress through the SoundScapes system My previous research (Brooks et al 2002)presented how certain individuals prefer to explore, play and create without any oth-ers being present (Figures 15–16) Exemplifying the SoundScapes open system andconcept a digital video camera was used to create a feedback loop play-space whereparticipant gesture distorted image silhouette and color change to a RGB lightingsystem that reacted to voice input Interactions in the private space were videotaped and analyzed as achieving a positive flow state (Csikszentmih´alyi 1996) Itwas observed in the private space that the chain of exploration-play-creation beganwith a curiosity that evolved out of the isolation and initial stillness that was firstencountered within the created environment Thereby, stillness became part of theaction and vice versa Indications pointed to how interactive play and creativity thatoffered choices between interaction and rest in a silent space enhanced the sense ofcontrol

A created public space (Figures 17–18) questioned participant perception andawareness where peer-support and scaffolding of exploration was evident Over-head infrared camera tracking was mapped to auditory music making (Figure 17)

Figs 15-16 Private space images of total engagement from ArtAbilitation 2007, (photo credit Augusto Brazio, Casa da Musica)

and image effects (Figure 18) Many results occurred from this space One rable instance was when a woman who was profoundly physically disabled insisted

memo-in explormemo-ing the memo-interactive space out of her wheelchair – evident was that crawlmemo-ingaround required immense effort (Figure 18) Her motions were tracked to open

up a digital mask that concealed an image of a famous Portuguese footballer Herdisabled colleagues were supporting by shouting instructions Motivation was stag-gering to achieve the whole erosion of the mask Another instance was where anautistic group was in the workshop and one of them began to move and open up thesame mask The first exposed facial element of the footballer was the chin and im-mediately another autistic male shouted out the name of the recognized footballer

In both instances (and others) the public virtual interactive space empowered diate learning to be exhibited (Petersson and Brooks 2007)

imme-Five physical ‘VIS rooms’ were created in the 2008 workshop Each was signed to offer differing experiences of empowerment through motion creatingimages and auditory feedback Large sub-woofer speakers positioned on the woodenfloor gave tactile stimulation which was especially important for the attending class

de-of young deaf children Findings were that the interactive spaces de-offered a placewhere comprehension of tasks could be shared from the strong to the weak in eachgroup Again wheelchairs were discarded to explore by crawling around the spaces

700 A.L Brooks

Figs 17-18 Public interactive space: two designs utilizing overhead camera tracking

Targeted tangible outcome was that each participant received a painting that theycreated through gesture Reports of how many showed these paintings to family,friends and non-attending staff stated to the powerful effect

Both workshops exhibited action-stillness cycles, which consisted of iterativeloops of exploring and transforming, constitute one part of a theoretical map forthe purposes of analyzing critical incidents in a non-formal learning process Thesecycles are related to the user’s learning experience The other part of the theoreticalmap concerns design issues in the form of use qualities relative to the user’s interac-tive experience; transparency, social-action space, user control/autonomy, pliability,playability and seductivity (L¨owgren and Stolterman 2005) During a whole session

cycle of action and stillness, facilitators have the possibility to reflect upon the dications of learning that occur during the process The user profile influences thefacilitator’s decisions on how to set up the attributes of the responsive environmentsrelative to the desired learning process and the expected outcome of that process.Once these prerequisites are set, the user is expected to ‘experience from the inter-action and learn from the experience’.

in-Hence, the aim of the workshop events was to investigate the user’s performance

in using responsive environments designed to engage participants in learning through action and stillness cycles A sub-question concerns the ability ofperception and the associated learning curve of the attendees with cognitive disabil-ities to be able to easily correlate across dimensions of scale and plane – a matterwhich influences the participant’s interactive as well as learning experience.The final reported event is one that was built upon previous work where thirdparty performance gesture created visuals to complement classical music situations

experience-Visualizing classical music

The previous sections present the motion-sensitive environment and its use as a habilitation training supplement within the community of people with a disability,both profound from birth, and acquired The use of the created interactive system ispresented in intimate institute settings for individuals and a later section shares twoexamples of larger workshop situations where groups attended This section exem-plifies how a similar concept and technique of using digital interactive media wasimplemented to elicit dynamic performance data from a situation that featured the

re-‘Orquestra Nacional do Porto’ - regarded by many as The Portuguese National chestra This was again in Casa da Musica, Porto, Portugal but this time in the main

Or-1200 seat state-of-the-art auditorium See publication Interpretation3(Brooks 2008)

A goal of the exploratory study was to dynamically complement the music byoffering an experience of inter/multi-sensory stimulation for both audience and per-formers This built upon my earlier work in Auckland, New Zealand, and Aarhus,Denmark where the different situations, one with orchestra and one with choir, bothhad a similar stimulation goal An aim of this work is to offer inclusive access sothat even people who are deaf may have an opportunity to appreciate classical mu-sic Important to mention is how in the choir study three sensors mapped to RGBenabled the conductor to paint the choir backdrop through his gestured interactionswith the singers The session was in the Danish Radio TV studio and six ‘takes’were recorded It was evident that his gestures were expanding as he became used

to the interaction and – in his own words - “how it felt that the air around my bodywas activated” This expansion of gesture relates to the work in the disability sectorwhere targeted in subliminal motivated augmented motion

The following Figs.19–26illustrate the result from the visualization experiment.Overhead cameras and stage sensors sourced the performance data and this was

3 http://www.icdvrat.reading.ac.uk/2008/interpretations.htm

702 A.L Brooks

Fig 19 Data from stage performance – conductor gesture, section/musician expressiveness, lights and music - sourced to dynamically affect the synchronized visuals for environment change in real- time (photo credit Jo˜ao Messias, Casa da Musica)

Fig 20 Sourced stage data mixed and matched to digitally mirror performance in abstract – here camera and music stand lights - creating and effecting projected visuals (photo credit Jo˜ao Messias, Casa da Musica)

Fig 21 Projected moving image dynamically matched to digitally mirror performance (photo credit Jo˜ao Messias, Casa da Musica)

mapped by created algorithms to effect the environment variables in real-time tocomplement the music A technical deficit proved the biggest hurdle and resultswere unsatisfactory as image mixing was via switching instead of faders on a mixer.Thus, nuance of performance was unable to be matched as I desired However, as

an experiment to explore the concept it did build my knowledge of the concept Thework is planned to evolve further locally for the next phase of experiments due tothe problems of accessing the specialized equipment required for technical setup.The work has been accepted for the Cultural Olympiad 2012 London

704 A.L Brooks

Fig 22 Projected image (center of screen) shows data capture from camera overhead (photo credit Jo˜ao Messias, Casa da Musica)

Conclusions and Future Directions

This article informs of digital interactive media sculpted into a responsive ture driven environment that is used within rehabilitation training Entertainmentachieved through a user creating and playing acts as a stimulant towards augmentinglife quality and motivating participation in training that otherwise is boring, tediousand mundane – often for both patient and therapist/facilitator Artistic composition

ges-in the form of music-makges-ing and digital pages-intges-ing is empowered through an ges-interfaceand multi-sensory virtual environment content that is adaptive and flexible enough

to address idiosyncratic needs However, whilst I create for another’s creativity, my

Fig 23 Musicians can observe effect of their own performance (photo credit Jo˜ao Messias, Casa

da Musica)

art is not the ‘product’ of the users’ interaction, i.e the music or painting, as manyjudge, but rather it is the aesthetic resonance that is achieved in and of the user.El-lis(1996,2004) describes aesthetic resonance achieved within his ‘Sound Therapy’body of research This was extemporized in a 2002 article (Brooks et al 2002) fol-lowing findings that explored interactive visuals in the form of animations, virtualreality, and robotic lighting devices with colored and patterned gobos This aesthetic

is a representation of a system where affordances were perceived and the targetedaction achieved Resultant sense of self-agency and inner empowerment is realizedcumulating to signify a state of flow that is identified by the facilitator Investiga-tion is of how to optimize user experience toward self-determined development viathe virtual interactive space (VIS) that is created between the user and the system

‘cor-Fig 26 View from behind the control station where author interacts (photo credit Paul Sharkey)

rect way of doing’ instead of ‘just doing’ with a focus on the mediating feedbackcontent rather than on the impaired feed-forward function User achievements arecommonplace

The activities of exploring interactive systems in the field of disability are ingly growing In Denmark commercial suppliers that sell targeted games for thismarket attempt a monopoly with substandard goods and services that tend to dimin-ish opportunities for end users rather than increase After two decades of research

seem-it is clear that open flexible and affordable systems that have selectable libraries ofinput device as well as selectable libraries of content stimuli are optimal Systemsthat can be adapted, and personalized within customized environments tailored to

708 A.L Brooks

an individual profile offer maximum opportunities and benefit That benefit is notrestricted to those with impairment as fun social interactions can motivate in playinggames or making art

Interface technologies have improved and motion detection devices such as theNintendo Wii offer a data device that permits access to data and thus can advance thefield in an affordable flexible package – either as a controller for games or music/artmaking An approach I made to Sony Computer Entertainment Europe (SCEE),London, after the launch of the EyeToy asking such access to interface data wasrejected Their representatives stated discriminatingly that they did not wish to beseen exploiting disabled people With such technology available it is a shame thatsuch a commercial profile prevents opportunity inquiry However, computer visionand graphical card advances makes affordable non-commercial sensor, single ormultiple camera systems, a reality and our students in MEDIALOGY4 in Esbjergare active exploring both user-centered games and art as controlled mediums in theSensoramaLab VR complex

Such diversity in the SoundScapes concept is recognized by Eaglestone &Bamidis(2008) as elements of a hybrid system consisting of networks of complexinter-connected subsystems comprising ‘created technical systems’ and ‘natural hu-man systems’ The MAX software has also undergone huge improvements since Ifirst used it as version 2.5 in the early 1990s Integrated capability to manipulatedigital audio signals in real-time from a computer became a reality via the MSP(Max Signal Processing) add-on, thus there was no longer a requirement for exter-nal audio hardware Subsequently another package developed and released for theMAX software environment in 2003 was Jitter, which enabled control of real-timevideo, 3D and matrix processing Latest MAX activities is a partnering with a loop-based sequencer software named Ableton Live that offers intuitive live performancecapabilities where MIDI manipulations can slot into a playing song such that it re-mains ‘musical’ Whilst developed for DJs and VJs the potentials in my researchfor such a tool is acknowledged and this is a future direction when the packagesare released Other explored software includes Touch Designer by Derivative, andSyntetik’s awesome Studio Artist interfaced with a Wacom Cintiq tablet

In this sense a tool to supplement traditional training becomes available if thecommunity of therapists and medical staff are prepared to accept an alternative ap-proach in the form of a hybrid non-formal learning apparatus and method If they

do and are prepared to collaborate in evolving the concept with other interestedresearchers from the related disciplines then the next-generation of therapists maywell be teamed with digital artists, computer scientists, and fMRI neurologists increating, facilitating, and evaluating the human in new ways, with new methods, andwith new apparatus, where FUN is the targeted user-experience and human benefitthe ultimate goal New virtual interactive spaces for learning are exhibited in thiswork that combines digital interactive media with entertainment and the arts

4 http://www.medialogy.eu

1 K E Misulis “Essentials of Clinical Neurophysiology.” Butterworth-Heinemann, 1997.

2 A Lucier “Statement On:luciermisulis Music for Solo Performer.” In: D Rosenboom (Ed.), Biofeedback and the Arts, Results of Early Experiments Aesthetic Research Center of Canada Publications, 1976.

3 D Warner “Rock ’n roll Science:warner Playing the Body Electric” [online] Available from http://www.virtualgalen.com/virtualhealing/dr-dave.htm.

4 A L Brooks, A Camurri, N Canagarajah and S Hasselblad “Interaction with shapes and sounds as a therapy for special needs and rehabilitation.” In:brooks International Conference

on Disability, Virtual Reality and Associated Technologies (4th) Veszpr´em, Hungary, 18–20 September P Sharkey, C S L´anyi, and P Standen, (Eds.) Reading: University of Reading,

pp 205–212, 2002.

5 A L Brooks Robotic Synchronized to Human Gesture as a Virtual Coach in (Re)habilitation Therapy In: 3rd International Workshop on Virtual Rehabilitation (IWVR2004), VRlab, EPFL, Lausanne, Switzerland, pp 17–16, 2004.

6 E Petersson and A Brooks Non-formal therapy and learning potentials through human ture synchronized to robotic gesture International Journal Universal Access in the Information Society Springer 6(2), pp 167–177, 2007.

ges-7 W M Marston Emotions of Normal People, Routledge, 1928.

8 A L Brooks SoundScapes: A concept of Virtual Interactive Space (V.I.S.) lished] World Summit for Social Development/NGO Forum 6–12 March, Holmen, Copenhagen, 1995.

[unpub-9 A L Brooks Virtual interactive space (V.I.S.) as a movement capture interface tool giving multimedia feedback for treatment and analysis In: “Bridging Cultures” – pro- gram of The 13th International Congress of the World Confederation for Physical Ther- apy, Yokohama Japan, May 23–28, Science Links Japan: http://sciencelinks.jp/jeast/article/ 200110/000020011001A0418015.php, 1999.

10 A L Brooks Virtual interactive space (V.I.S.) In: ‘Pushing the limits: optimising tial through science and technology’, Congress Program and Abstract Book, 5th Scientific Congress, Sydney 2000 Paralympic Games, Oct 11–13, Convention Centre, Darling Harbor, Sydney, Australia, 2000.

poten-11 A L Brooks HUMANICS 1 – a feasibility study to create a home internet based telehealth product to supplement acquired brain injury therapy In: International Conference on Dis- ability, Virtual Reality and Associated Technologies (5th) Oxford University, UK, 20–22 September 2004 Sharkey, P., McCrindle, R and Brown, D (eds.) Reading: University of Reading, pp 43–50, 2004.

12 A L Brooks and E Petersson Recursive reflection and learning in raw data video analysis

of interactive ‘play’ environments for special needs health care In: Proceedings of 7th national Workshop on Enterprise networking and Computing in Healthcare Industry, Korea HEALTHCOM 2005, IEEE Signal Processing Society, USA, pp 83–87, 2005.

Inter-13 A L Brooks and S Hasselblad Creating aesthetically resonant environments for the icapped, elderly and rehabilitation: Sweden Proceedings of 6th International Conference on Disability, Virtual Reality and Associated Technologies (ICDVRAT), Esbjerg, Denmark, 18th– 20th September, pp 191–198, 2004.

hand-14 A L Brooks Enhanced Gesture Capture in Virtual Interactive Space Computers in Art, sign, and Education (CADE), 29 June–01 July, Copenhagen Business School, Denmark and Malm¨o University, Sweden, 2004.

De-15 A L Brooks and E Petersson Stillness design attributes in non-formal rehabilitation: CADE2007 - Computers in Art Design and Education Perth, Curtin University of Technology,

pp 36–44, 2007.