Human behavior understanding 7th international workshop, HBU 2016

3Benjamin Allaert, José Mennesson, and Ioan Marius Bilasco Assessing Affective Dimensions of Play in Psychodynamic Child Psychotherapy via Text Analysis.. Recently, child databases are b

7th International Workshop, HBU 2016

Amsterdam, The Netherlands, October 16, 2016 Proceedings

Human Behavior Understanding

Lecture Notes in Computer Science 9997Commenced Publication in 1973

Founding and Former Series Editors:

Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

More information about this series at http://www.springer.com/series/7412

Mohamed Chetouani • Jeffrey Cohn

Albert Ali Salah (Eds.)

Human Behavior

Understanding

7th International Workshop, HBU 2016

Amsterdam, The Netherlands, October 16, 2016 Proceedings

123

ISSN 0302-9743 ISSN 1611-3349 (electronic)

Lecture Notes in Computer Science

ISBN 978-3-319-46842-6 ISBN 978-3-319-46843-3 (eBook)

DOI 10.1007/978-3-319-46843-3

Library of Congress Control Number: 2016952516

LNCS Sublibrary: SL6 – Image Processing, Computer Vision, Pattern Recognition, and Graphics

© Springer International Publishing AG 2016

This work is subject to copyright All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on micro films or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.

The use of general descriptive names, registered names, trademarks, service marks, etc in this publication does not imply, even in the absence of a speci fic statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made.

Printed on acid-free paper

This Springer imprint is published by Springer Nature

The registered company is Springer International Publishing AG

The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

The HBU workshops gather researchers dealing with the problem of modeling humanbehavior under its multiple facets (expression of emotions, display of complex socialand relational behaviors, performance of individual or joint actions, etc.) This year, theseventh edition of the workshop was organized with challenges of designing solutionswith children in mind, with the cross-pollination of different disciplines, bringingtogether researchers of multimedia, robotics, HCI, artificial intelligence, patternrecognition, interaction design, ambient intelligence, and psychology The diversity ofhuman behavior, the richness of multi-modal data that arises from its analysis, and themultitude of applications that demand rapid progress in this area ensure that the HBUworkshops provide a timely and relevant discussion and dissemination platform.The HBU workshops were previously organized as satellite events to the ICPR(Istanbul, Turkey, 2010), AMI (Amsterdam, The Netherlands, 2011), IROS (Vilamoura,Portugal, 2012), ACM Multimedia (Barcelona, Spain, 2013), ECCV (Zurich,Switzerland, 2014) and UBICOMP (Osaka, Japan, 2015) conferences, with differentfocus themes The focus theme of this year’s HBU workshop was “Behavior Analysisand Multimedia for Children.”

With each passing year, children begin using computers and related devices atyounger and younger ages The initial age of computer usage is steadily getting lower,yet there are many open issues in children’s use of computers and multimedia In order

to tailor multimedia applications to children, we need smarter applications thatunderstand and respond to the users’ behavior, distinguishing children and adults ifnecessary Collecting data from children and working with children in interactiveapplications call for additional skills and interdisciplinary collaborations Subsequently,this year’s workshop promoted research on the automatic analysis of children’sbehavior Specifically, the call for papers solicited contributions on age estimation,detection of abusive and aggressive behaviors, cyberbullying, inappropriate contentdetection, privacy and ethics of multimedia access for children, databases collectedfrom children, monitoring children during social interactions, and investigations intochildren’s interaction with multimedia content

The keynote speakers of the workshop were Dr Paul Vogt (Tilburg University),with a talk entitled“Modelling Child Language Acquisition in Interaction from Cor-pora” and Dr Isabela Granic (Radboud University Nijmegen), with a talk on “BridgingDevelopmental Science and Game Design to Video Games That Build EmotionalResilience.” We thank our keynotes for their contributions

This proceedings volume contains the papers presented at the workshop Wereceived 17 submissions, of which 10 were accepted for oral presentation at theworkshop (the acceptance rate is 58 %) Each paper was reviewed by at least twomembers of the Technical Program Committee The papers submitted by the co-chairswere handled by other chairs both during reviewing and during decisions The Easy-Chair system was used for processing the papers The present volume collects the

accepted papers, revised for the proceedings in accordance with reviewer comments,and presented at the workshop The papers are organized into thematic sections on

“Behavior Analysis During Play,” “Daily Behaviors,” “Vision-Based Applications,”and “Gesture and Movement Analysis.” Together with the invited talks, the focustheme was covered in one paper session as well as in a panel session organized by

Dr Rita Cucchiara (University of Modena and Reggio Emilia)

We would like to take the opportunity to thank our Program Committee membersand reviewers for their rigorous feedback as well as our authors and our invitedspeakers for their contributions

Jeffrey CohnAlbert Ali Salah

Conference Co-chairs

Mohamed Chetouani Université Pierre et Marie Curie, France

Jeffrey Cohn Carnegie Mellon University and University

of Pittsburgh, USAAlbert Ali Salah Boğaziçi University, Turkey

Technical Program Committee

Elisabeth André Universität Augsburg, Germany

Lisa Anthony University of Florida, USA

Oya Aran Idiap Research Institute, Switzerland

Antonio Camurri University of Genoa, Italy

Marco Cristani University of Verona, Italy

Abhinav Dhall University of Canberra, Australia

Hamdi Dibeklioğlu Delft University of Technology, The NetherlandsWeidong Geng Zhejiang University, China

Hatice Gunes University of Cambridge, UK

Sibel Halfon Bilgi University, Turkey

Zakia Hammal Carnegie Mellon University, USA

Dirk Heylen University of Twente, The Netherlands

Andri Ioannou Cyprus University of Technology, Cyprus

Mohan Kankanhalli National University of Singapore, Singapore

Alexey Karpov SPIIRAS, Russia

Heysem Kaya Namık Kemal University, Turkey

Cem Keskin Microsoft Research, USA

Hatice Kose Istanbul Technical University, Turkey

Ben Kröse University of Amsterdam, The Netherlands

Matei Mancas University of Mons, Belgium

Panos Markopoulos Eindhoven University of Technology, The NetherlandsLouis-Philippe Morency Carnegie Mellon University, USA

Florian Mueller RMIT, Australia

Helio Pedrini University of Campinas, Brazil

Francisco Florez Revuelta Kingston University, UK

Stefan Scherer University of Southern California, USA

Ben Schouten Eindhoven University of Technology, The NetherlandsSuleman Shahid University of Tilburg, The Netherlands

Reiner Wichert AHS Assisted Home Solutions, Germany

Bian Yang Norwegian University of Science and Technology,

Norway

Additional ReviewersNecati Cihan CamgözIrtiza Hasan

Giorgio Roffo

Ahmet Alp Kındıroğlu

VIII Organization

Behavior Analysis During Play

EmoGame: Towards a Self-Rewarding Methodology for Capturing

Children Faces in an Engaging Context 3Benjamin Allaert, José Mennesson, and Ioan Marius Bilasco

Assessing Affective Dimensions of Play in Psychodynamic Child

Psychotherapy via Text Analysis 15Sibel Halfon, Eda Aydın Oktay, and Albert Ali Salah

Multimodal Detection of Engagement in Groups of Children

Using Rank Learning 35Jaebok Kim, Khiet P Truong, Vicky Charisi, Cristina Zaga,

Vanessa Evers, and Mohamed Chetouani

Daily Behaviors

Anomaly Detection in Elderly Daily Behavior in Ambient

Sensing Environments 51Oya Aran, Dairazalia Sanchez-Cortes, Minh-Tri Do,

and Daniel Gatica-Perez

Human Behavior Analysis from Smartphone Data Streams 68Laleh Jalali, Hyungik Oh, Ramin Moazeni, and Ramesh Jain

Gesture and Movement Analysis

Sign Language Recognition for Assisting the Deaf in Hospitals 89Necati Cihan Camgöz, Ahmet Alp Kındıroğlu, and Lale Akarun

Using the Audio Respiration Signal for Multimodal Discrimination

of Expressive Movement Qualities 102Vincenzo Lussu, Radoslaw Niewiadomski, Gualtiero Volpe,

and Antonio Camurri

Spatio-Temporal Detection of Fine-Grained Dyadic Human Interactions 116Coert van Gemeren, Ronald Poppe, and Remco C Veltkamp

Vision Based Applications

Convoy Detection in Crowded Surveillance Videos 137Zeyd Boukhers, Yicong Wang, Kimiaki Shirahama, Kuniaki Uehara,

and Marcin Grzegorzek

First Impressions - Predicting User Personality from Twitter Profile Images 148Abhinav Dhall and Jesse Hoey

Author Index 159

Behavior Analysis During Play

EmoGame: Towards a Self-Rewarding

Methodology for Capturing Children Faces

in an Engaging Context

Benjamin Allaert(B), Jos´e Mennesson, and Ioan Marius Bilasco

Univ Lille, CNRS, Central Lille, UMR 9189 - CRIStAL - Centre de Recherche

en Informatique Signal et Automatique de Lille, 59000 Lille, France

benjamin.allaert@ed.univ-lille1.fr,

{jose.mennesson,marius.bilasco}@univ-lille1.fr

Abstract Facial expression datasets are currently limited as most of

them only capture the emotional expressions of adults Researchers havebegun to assert the importance of having child exemplars of the vari-ous emotional expressions in order to study the interpretation of theseexpressions developmentally Capturing children expression is more com-plicated as the protocols used for eliciting and recording expressions foradults are not necessarily adequate for children This paper describesthe creation of a flexible Emotional Game for capturing children faces

in an engaging context The game is inspired by the well-known GuitarHeroTM gameplay, but instead of playing notes, the player should pro-duce series of expressions In the current work, we measure the capacity

of the game to engage the children and we discuss the requirements interms of expression recognition needed to ensure a viable gameplay Thepreliminary experiments conducted with a group of 12 children with agesbetween 7 and 11 in various settings and social contexts show high levels

of engagement and positive feedback

recogni-In most of the proposed recording scenarios the subjects are passive or areacting on demand, but they are not naturally engaged in the interaction The elic-itation of expression is most of the time explicit Some of the recent databases

c

 Springer International Publishing AG 2016

M Chetouani et al (Eds.): HBU 2016, LNCS 9997, pp 3–14, 2016.

4 B Allaert et al.

like SEMAINE [13] or RECOLA [17] are proposing (limited) social interactions:agent to human or human to human Still, the lab context tends to bias expres-sions as the subjects are not in their natural environment In these settings, thevivid/spontaneous expressions are captured in between recording sessions, whenthe subjects are interacting with the lab personnel On the other hand, capturingdatabases in natural environments is challenging as the position of microphonesand cameras is not fully controlled, there is noise like visual backgrounds and it

is difficult to control the emotional content eliciting expressions

Participating to recording session is not perceived as an enjoying task ally people are rewarded explicitly in order to participate to the recordings

Usu-We think, that especially for children, the reward should be implicit as ject should enjoy the session We propose an interactive and non-intrusive toolfor capturing children faces in an engaging context We provide an emo-relatedgame, inspired by the well known Guitar HeroTM game, where children have

sub-to produce expressions in order sub-to score points The application was ported onportable devices (tablet, smartphone) so that it can be easily deployed in-the-wild environments The engaging scenario ensures control over the capturingconditions While playing, the subjects become aware that without visual con-tact, in poor lighting condition, in absence of frontal faces they acquire little or

no points In preliminary experiments, we observed that they strive to conform

to the technology limitation in order to score as much points as possible Viablesessions (frontal poses, good lighting, etc.) can be distinguish from poor ones bythe scores acquired

While the children are using the application, we expect to collect data that

is partially annotated, as subjects in good mental health, will strive to duce the required emotion at expected moments in time Most of the time theexpressions are expected to be acted and exaggerated in order to acquire morepoints But sill, vivid and spontaneous expressions can be elicited when thegameplay is tuned High speed variations in emotions sequence generally pro-duce natural hilarity, especially when the game is played in a social context withfriends Defective expression recognition tends to induce spontaneous negativeexpressions With regard to the current databases, which generally provides aneutral-onset-apex-offset schema, the proposed solution allows to obtain morecomplex patterns, including fading from one emotion to another

pro-The preliminary experiments conducted with a group of 12 children withages between 7 and 11 in various settings and social contexts (home - alone orwith friends or family, school, work) show the rapid adoption of the applicationand the high engagement and enjoyment of the subjects in participating to therecording session Due to privacy and image property, at this stage, no visualdata was collected The recording sessions were conducted in order to measurethe adequacy of the developed tool and protocol for capturing child expression

in various contexts and in an enjoying way

The paper is structured as follows First, we discuss existing gies for capturing expression-related databases Then, we discuss methodologiesand scenarios aiming to increase the attractiveness of recording sessions Anapplication reflecting the proposed methodology is presented, as well as the

methodolo-EmoGame 5

time-efficient expression recognition technologies We report on the preliminaryresults obtained for recording sessions concerning children and young adults.Finally, conclusions and perspectives are discussed

2 Related Works

Motivated by a wide range of applications, researchers in computer vision andpattern recognition have become increasingly interested in developing algorithmsfor automatic expressions recognition in still images and videos Most of theexisting solutions are data-driven and large quantities of data are required totrain classifiers Three main types of interaction scenarios have been used torecord emotionally colored interactions, most of the time for adults:

– Acted behavior presented in CK+ [10] and ADFES [23] database is duced by the subject upon request, e.g., actors Interaction scenarios with astatic pose and acted behavior are the easiest to design and present the advan-tage to have a control on the portrayed emotions However, this approachwas criticized for including (non-realistic) forced traits of emotion, which areclaimed to be much more subtle when the emotion arises in a real-life context[19]

pro-– Induced behavior occurs in a controlled setting designed to elicit an

emo-tional reaction such as when watching movies like in DISFA [2] Active narios based on the induced behavior can influence on subject behavior withindirect control of the behaviors of participants, by imposing a specific con-text of interaction, e.g., four emotionally stereotyped conversational agentswere used in SEMAINE database [13] However, this approach may not pro-vide fully natural behaviors, because the interaction may be restricted to aspecific context, wherein the spontaneous aspect of interaction may be thuslimited or even absent [21]

sce-– Spontaneous behavior appears in social settings such as interactions

between humans as in RECOLA database [17] The spontaneous behaviorscenario guarantees natural emotionally colored interactions, since the set ofverbal and non-verbal cues is both free and unlimited However, this spon-taneous interaction scenario is the hardest to design as it includes severalethical issues, like people discussing about private things, or not knowing theyare recorded

The above scenarios were successfully employed in collecting adult databases[2,10,13,17,23] Child databases are needed to train solution tuned for childexpression analysis Recently, child databases are becoming available [4,7,15].The databases are generally created under the protocols used for capturing adultdatabases and more specifically using the acted and induced behavior protocols.The Dartmouth Database of Children’s Faces [4] is a well-controlled data-base of faces of 40 males and 40 females children between 6 and 16 years-of-age.Eight different facial expressions are imitated by each model During acquisition,children were seated front of a black screen and were dressed in black In order

6 B Allaert et al.

to elicit the desired facial expressions, models were asked to imagine situations(e.g Disgust: “Imagine you are covered with chewing gum”, or, Anger: “Imagineyour brother or sister broke your PlayStation”), and photos were taken whenthe expressions were the best the children could produce

The most extensive children databases is the NIMH Child Emotional FacesPicture Set (NIMH-ChEFS) [7], which includes frontal face images of 60 childrenbetween 10 and 17 years-of-age, posing five facial expressions Each child actorwas instructed to act a specific facial emotion The dataset includes children ofdifferent races, multiple facial expressions and gaze orientations

EmoWinconsin [15] recorded children between 7 and 13 years old while ing a card game with an adult examiner The game is based on a neuropsycho-logical test, modified to encourage dialogue and induce emotions in the playerbecause children are deeply involved in its realization Each sequence is anno-tated with six emotional categories and three continuous emotion primitives

play-by 11 human evaluators The children’s performance and interaction with theexaminer trigger reactions that affect children’s emotional state Influencing thechildren’s mood during the experiment, by imposing for example a specific con-text of interaction (positive and negative sessions), may thus be useful to ensure

a variety of behaviors during the interactions

By analyzing the protocols that have been used to elaborate the childrendatabases, we believe that more natural interactions, with limited technologyconstraints shall be provided in order to increase the children engagement andcollect more vivid expressions Although there are not many databases of childrenwith annotated facial expressions available, games are being used more and more

to eliciting emotions The games provide a useful tool for capturing childrenfaces, while imposing a specific context (social environment, interaction) In[20] Shahid et al explore the effect of physical co-presence on the emotionalexpressions of game playing children They showed that the emotional response

of children varies when they play a game alone or together with their friends.Indeed, children in pairs are more expressive than individuals because they havebeen influenced by their partners

Motivated by the interest of children in games, we propose a methodologyand a tool for capturing interactive and non-intrusive emo-related expressions

It is important to specify that our tool does not retain personal data and, inits current state, it does not record facial images The goal is to evaluate theimpact of a new protocol that encourages children to produce facial expressionsunder close to in-the-wild conditions As illustrated in [15,20], a game seems

an appropriate context because children are strongly involved and they becomemore collaborative Moreover, a mobile support allows the recordings outside

a (living) lab context, hence, children can be recorded in an unbiased fashion

in more natural circumstances (in a spirit of a family home, friendship) Whileconducted in a social context, the pilot is expected to capture vivid expressions

in between or during the game The protocol can be tuned in order to inducenegative expression like frustration in biasing the behavior of the game Moreinsights about the design of the game scenario is provided in the next section

As illustrated in Fig.1, the game interface is composed of seven major ments Each expression is associated with one column and one token with a visualrepresentation of the expression (Fig.1(1–3) represent a positive expression token)

ele-in a particular color When a token is reachele-ing the bottom circle of the board(Fig.1(1–4)), the player face (Fig.1(1–5)) and the expression which should befound are analyzed to assign a grade (Fig.1(1–1) and keep track of the accumu-lated score (Fig.1(1–2)) These individual ratings aggregated provide an overallrating for the game The gauge above the score indicates the overall rating dur-ing the course of the game As visual facial feedback is provided, when expression

Fig 1 EmoGame interface inspired to the Guitar HeroTM gameplay (Color figure

online)

8 B Allaert et al.

recognition fails, the user is implicitly encouraged to control the quality of theimage by correcting the orientation and the position, in order to ensure optimalconditions (no backlight, frontal face, homogeneous illumination, etc.)

In order to enhance user engagement and enjoyment we have added textual(perfect, good, fair, bad) and audio (positive or negative) feedback In order to

be able to collect data for longer times, it is important to provide a positivegame-playing experience The game-playing experience is also influenced by thesequence of tokens presented to the user The sequence of events is fully program-mable and does make a difference in terms of player behavior The speed of tokengoing down the screen, the time distance between consecutive tokens (Fig.1(1–6))and the order of appearance of the token can be customized In preliminary test-ings, we have observed that high speed variations in emotions sequence generallyproduce natural hilarity, especially when the game is played in a social contextwith friends Besides, the deployed technology for expression recognition must berobust enough in order to keep the child committed to the experience However,

an expression palette larger than the one that the application can recognize can

be collected, as long as a minimum set of expressions is recognized and points arecoherently scored for the supported expressions

In the following we focus on the application and on the details of the lying methods for positive, negative and surprise expression recognition

under-4 Expression Recognition

In the context of a video game, the expression analysis must be performed as fast

as possible (in interactive time) in order to keep the attention and involvement

of the player These requirements are even more important when the expressionrecognition is performed in a mobile context where memory, computational capa-bilities and energy are limited We propose a fast analysis process in two stagesillustrated in Fig.2: image pre-processing detailed in Sect.4.1 and expressionrecognition detailed in Sect.4.2

A good technical realization is necessary to keep player attention focusedduring the session and to ensure that the objectives of the scenario are fulfilled.The Fig.3 illustrate the process of capturing the facial expressions Once thetoken scrolls down and reaches the bottom circles, the image provided by thefrontal camera of the hand-held device is extracted and sent through the JNIinterface to the expression from face library The expression analysis is done bythe native library and results are sent back to the application controller whichupdates the score and provides textual and audio feedback to the user

Fig 2 Overview of the facial expression recognition process

EmoGame 9

Fig 3 The EmoGame expression analyzer process Results are calculated from the

metrics estimated on the player face and the required expression

4.1 Image Pre-processing

As soon as the image is received through the JNI interface, image pre-processing

is performed The goal is to detect the face and make it invariant under lation, rotation, scale and illumination

trans-As the application is deployed in a hand-held mobile device, most of thetime the person face is situated in the center of the image A fast face detectorsuch as Boosted Haar classifiers [24] is used to localize the face of the user Thiskind of classifiers presents some drawbacks since they support only a limited set

of near to frontal head-poses Supposing that the user is engaged in the game,each frame of the video contains a single face The absence of a face is a signfor either non supported head-poses (e.g looking somewhere else) or inadequatehand-held device orientation (e.g camera device pointing above the face)

We use a dedicated neural network defined by Rowley [18] (available inSTASM library [14]) in order to detect the eye positions Orientation of theface is estimated using the vertical positions of the two eyes The angle betweenthe two pupil points is used to correct the orientation by setting the face center

as origin point and the whole frame is rotated in opposite direction Finally,the face is cropped and its size is normalized to obtain scale invariance Imageintensity is normalized using histogram equalization which improve its contrast

It aims to eliminate light and illumination related defects from the facial area

10 B Allaert et al.

4.2 Face Expression Analysis

By normalizing the face representation (invariant under translation, rotation andscale), we can compute fast metrics directly on pixels rather than extractingcomplex metrics that can have high computational complexity Positive expres-sion are recognized by considering raw pixel intensities Negative and surpriseexpressions are detected by characterizing changes in small regions of interests

Positive expression detection: In this application, positive expression

detec-tion is performed using method defined in [6] The dataset GENKI-4K [1] is used

as a training set for positive/neutral classification Only the lower part of thenormalized face which maximizes the accuracy for this particular classificationproblem is considered A back propagation neural network having two hiddenlayers (20 and 15 neurons) is used to train pixel intensity values obtained fromthe selected ROI Input layer has 200 neurons and output layer has two neuronsrepresenting the happy and neutral classes Experiments on positive expressiondetection are conducted in [6] and state-of-the art performances are obtained onGENKI-4K [1] (92 %), JAFFE [12] (82 %) and FERET [16] (91 %) databases

Negative expression detection: Negative expressions generally involves the

activation of various degrees of FACS AU4 where eyebrows are lowered anddrawn together [9] We focus on the regions of interests located in the upper part

of the face which include wrinkles between the eyes The wrinkles are extractedusing a Gabor filters bank as in [3] Each pixel of the resulting image corresponds

to the maximum amplitude among the filtered responses Then, the resultingimage is normalized and thresholded to obtain a binary image The featureencoding AU4 activation corresponds to the proportion of white pixels, whichcorresponds to wrinkles A threshold is used to determine if there is a negativeexpression or not To show the role of the threshold, KDEF database [11] isconsidered to perform tests In our experiment, negative expressions cover anger,disgust, afraid and sad expression as in [9] A recall-precision curve obtained byvarying the threshold is shown in Fig.4 It can be easily seen that obtainedresults are good enough to provide a consistent feedback to the user In KDEFexperiments, a proportion of white pixels superior to 0 % gives the best results

in terms of recall-precision as the dataset was captured in control settings andforehead is cleared of artifacts However, while playing an adaptive thresholdhas to be employed in order to better support variations due to shadows andcamera orientation

Surprise expression detection: It is well-known that surprise expression is

closely related to the activation of FACS AU1 and FACS AU2 [22], which respond, respectively, to left and right eyebrows movements In this paper, eye-brows are detected using a Gabor filter applied to a ROI determined experimen-tally considering the eye position and the IPD distance as in [5] The featureencoding AU1 or AU2 activation is the ratio between the distance of the eyecenter and the lower boundary of the eyebrow and the distance between the twoeyes Higher this feature is, more the person raises eyebrows The surprise expres-sion is detected when this feature is higher then threshold This feature has been

cor-EmoGame 11

Fig 4 Recall-Precision curve on KDEF: negative expression detection evaluation (on

the left) and negative expression detection evaluation (on the right)

chosen because it is fast to compute and the obtained results are good enough

in our context To test the stability of the feature against the threshold used,

an experiment has been conducted on KDEF database [11] A recall-precisioncurve obtained by varying the surprise threshold is shown in Fig.4 In KDEFexperiments, a threshold equals to 33 gives the best results in terms of recall-precision As for the negative expression, while playing, an adaptive thresholdhas to be employed in order to take into account camera orientation

5 Preliminary Experiments

In this section, we study the capacity of our application to engage the children inthe scenarized recording sessions Moreover, we want to measure the satisfaction

of subjects and their intention to renew the experience

For the experiments, we used a Samsung Galaxy Tab 2 10.1 digital tablet.The application layout is adapted to the landscape mode as the front camera issituated in the middle of the long side of the tablet

Each game is composed of 15 expressions to mimic (5 Positive, 5 Negative,

5 Surprise) The sequence of expressions is randomized The speed of tokensscrolling down is constant but the gap between them varies randomly

Twelve children and six adults were invited to test the application We havedivided them into three age categories: between 4 and 7 years old, between 8 and

10 years old and adults (> 20 years old) Sessions were recorder either at home

or at school, alone or with friends

Each subject played freely the game once or several times Then, he filled in

a questionnaire measuring the enjoyment, and his intention to play again Wealso asked the player about the ability of the application to detect correctly theexpressions Each response is ranged between 1 for bad and 5 for great Theresults of this experimentation is shown in Fig.5

The perception of the expression recognition performances varies within thechildren groups (see Fig.5A) Older children were challenging more the applica-tion and were able to identify situations were the technology is failing (high pitch,poor lighting conditions, near field of view) However, we have noted that chil-dren were motivated to play again in order to improve their score by correcting

12 B Allaert et al.

Fig 5 Boxplots showing several statistics computed on our experiments

device orientation and trying various ways of producing the required expressions.Finally, we can see that for all testers, positive expression seems to be the bestdetected by our game (see in Fig.5B) Negative and surprise expression detectionresults depends strongly on the facial characteristics of the player and illumina-tion settings In this case, adaptive thresholding could improve results concerningthese features Despite the recognition errors, in Fig.5C, we can clearly see thatall age groups enjoyed the game Children enjoyed better than adults and weremore committed to renew the experience and play again (see Fig.5D) Thesetwo statistics are correlated and show that the children are engaged when theyplay the game

6 Conclusion

In this paper we have proposed a new tool for capturing vivid and spontaneouschildren expressions by means of an engaging expression-related games A mobiledevice is used in order to be able to realize recording session outside a labenvironment We think that capturing data in familiar setting reduces the biasbrought by an unknown context The game play encourages children to implicitlycontrol the device orientation and light exposure in order to obtain high scores.The results of the preliminary study show that the children are enjoying the gameexperience and that they are ready and willing to renew the experience As long

as the facial expression are used as a mean of interaction within a rewardingcontext, engagement from subjects can be expected

Preliminary results encourage us to extend the experiments to larger childrencorpus As large quantity of data can be collected in out-of-lab conditions, it is

EmoGame 13

important to assist the process of selecting viable data Hence, we will focus oncollecting and annotating processing We envision to better quantify the qual-ity of the recorded sessions by means of homogeneous illumination quantifica-tion, head orientation estimation, mobile device stability, etc This metrics willenhance the annotation process by filtering inadequate conditions At longerterm we envision including new expression recognition techniques in order topropose more complex scenarios

References

1 The MPLab GENKI database, GENKI-4K subset (2011)

2 Bartlett, M.S., Littlewort, G.C., Frank, M.G., Lainscsek, C., Fasel, I.R., Movellan,J.R.: Automatic recognition of facial actions in spontaneous expressions J Multi-

media 1(6), 22–35 (2006)

3 Batool, N., Chellappa, R.: Fast detection of facial wrinkles based on gabor features

using image morphology and geometric constraints PR 48(3), 642–658 (2015)

4 Dalrymple, K.A., Gomez, J., Duchaine, B.: The dartmouth database of children’s

faces: acquisition and validation of a new face stimulus set PLoS one 8(11), e79131

tron Sig Process 93(6), 1547–1556 (2013)

7 Egger, H.L., Pine, D.S., Nelson, E., Leibenluft, E., Ernst, M., Towbin, K.E.,Angold, A.: The NIMH child emotional faces picture set (NIMH-CHEFS): a new

set of children’s facial emotion stimuli Int J Methods Psychiatr Res 20(3),

145–156 (2011)

8 Ekman, P., Rosenberg, E.L.: What the Face Reveals: Basic and Applied Studies ofSpontaneous Expression Using the Facial Action Coding System (FACS) OxfordUniversity Press, Oxford (1997)

9 Lablack, A., Danisman, T., Bilasco, I.M., Djeraba, C.: A local approach for negativeemotion detection In: ICPR, pp 417–420 (2014)

10 Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: Theextended cohn-kanade dataset (CK+): a complete dataset for action unit andemotion-specified expression In: CVPR-Workshops, pp 94–101 IEEE (2010)

11 Lundqvist, D., Flykt, A., ¨Ohman, A.: The Karolinska Directed Emotional Faces KDEF, CD ROM from Department of Clinical Neuroscience, Psychology Section.Karolinska Institutet (1998) ISBN 91-630-7164-9

-12 Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions withGabor wavelets In: FG, pp 200–205 IEEE (1998)

13 McKeown, G., Valstar, M., Cowie, R., Pantic, M., Schroder, M.: The semaine base: annotated multimodal records of emotionally colored conversations between

data-a person data-and data-a limited data-agent IEEE Trdata-ans Affect Comput 3(1), 5–17 (2012)

14 Milborrow, S., Nicolls, F.: Locating facial features with an extended active shapemodel In: Forsyth, D., Torr, P., Zisserman, A (eds.) ECCV 2008 LNCS, vol 5305,

pp 504–513 Springer, Heidelberg (2008) doi:10.1007/978-3-540-88693-8 37

14 B Allaert et al.

15 P´erez-Espinosa, H., Reyes-Garc´ıa, C., Villase˜nor-Pineda, L.: EmoWisconsin: anemotional children speech database in Mexican Spanish In: D’Mello, S., Graesser,A., Schuller, B., Martin, J.-C (eds.) ACII, Part II LNCS, vol 6975, pp 62–71.Springer, Heidelberg (2011) doi:10.1007/978-3-642-24571-8 7

16 Phillips, P.J., Wechsler, H., Huang, J., Rauss, P.J.: The feret database and

evalua-tion procedure for face-recognievalua-tion algorithms Image Vis Comput 16(5), 295–306

(1998)

17 Ringeval, F., Sonderegger, A., Sauer, J., Lalanne, D.: Introducing the recola timodal corpus of remote collaborative and affective interactions In: FG, pp 1–8.IEEE (2013)

mul-18 Rowley, H.A., Baluja, S., Kanade, T.: Neural network-based face detection IEEE

Trans Pattern Anal Mach Intell 20(1), 23–38 (1998)

19 Schuller, B., Batliner, A., Steidl, S., Seppi, D.: Recognising realistic emotions andaffect in speech: state of the art and lessons learnt from the first challenge Speech

Commun 53(9), 1062–1087 (2011)

20 Shahid, S., Krahmer, E., Swerts, M.: Alone or together: exploring the effect ofphysical co-presence on the emotional expressions of game playing children acrosscultures In: Markopoulos, P., Ruyter, B., IJsselsteijn, W., Rowland, D (eds.) Funand Games 2008 LNCS, vol 5294, pp 94–105 Springer, Heidelberg (2008) doi:10.1007/978-3-540-88322-7 10

21 Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for

affect recognition and implicit tagging IEEE Trans Affect Comput 3(1), 42–55

Assessing Affective Dimensions of Play

in Psychodynamic Child Psychotherapy

via Text Analysis

Sibel Halfon1, Eda Aydın Oktay2, and Albert Ali Salah2(B)

1 Department of Psychology, Bilgi University, Istanbul, Turkey

sibel.halfon@bilgi.edu.tr

2 Department of Computer Engineering, Bo˘gazi¸ci University, Istanbul, Turkey

{eda.aydin,salah}@boun.edu.tr

Abstract Assessment of emotional expressions of young children

dur-ing clinical work is an important, yet arduous task Especially in naturalplay scenarios, there are not many constraints on the behavior of thechildren, and the expression palette is rich There are many approachesdeveloped for the automatic analysis of affect, particularly from facialexpressions, paralinguistic features of the voice, as well as from the myr-iads of non-verbal signals emitted during interactions In this work,

we describe a tool that analyzes verbal interactions of children ing play therapy Our approach uses natural language processing tech-niques and tailors a generic affect analysis framework to the psychother-apy domain, automatically annotating spoken sentences on valence andarousal dimensions We work with Turkish texts, for which there are farless natural language processing resources than English, and our app-roach illustrates how to rapidly develop such a system for non-Englishlanguages We evaluate our approach with longitudinal psychotherapydata, collected and annotated over a one year period, and show thatour system produces good results in line with professional clinicians’assessments

dur-Keywords: Play therapy·Affect analysis·Psychotheraphy·NaturalLanguage Processing·Turkish language·Valence·Arousal

1 Introduction

Clinical work with young children often relies on emotional expression and gration through symbolic play [58] Play naturally provides a venue in whichchildren can communicate and re-enact real or imagined experiences that areemotionally meaningful to them [23,52] Many child therapists use play therapy

inte-to help children express their feelings, modulate affect, and resolve conflicts [16].Affective analysis of psychodynamic play therapy sessions is a meticulousprocess, which requires many passes over the collected data to annotate differentaspects of play behavior, and the markers of affective displays Both the verbal

c

 Springer International Publishing AG 2016

M Chetouani et al (Eds.): HBU 2016, LNCS 9997, pp 15–34, 2016.

16 S Halfon et al.

and non-verbal content of the interactions contain valuable information, andare analyzed in detail Recent developments in multimedia analysis suggest thatautomatic tools could be used to help the analyst in these tasks The advantagesare many; such tools can support the therapist with immediate and rich feedbackabout the data, highlighting promising patterns for which more effort can bedevoted, and also provide additional quantification of treatment effects Thedisadvantages are that good automatic systems typically require a large amount

of data for training, their generalization abilities may suffer from factors thatmay appear trivial to the experimenter (e.g amount of ambient light, if a camera-based system is employed), and depending on the model used, justification ofthe classifications may be difficult to fathom

In this work, we propose such an automatic, text-based tool for affectivecontent analysis from verbal communications of children during play activity inpsychodynamic treatment Automatic analysis of psychodynamic play therapy

is not a broadly researched subject, and we hope that our contribution willinitiate more research in this domain Another important point is that our tool

is based on the Turkish language, which is spoken by more than 70 millionpeople worldwide, but for which few analysis tools are available1 We make thedeveloped tool available to the research community

1.1 Preliminary Research Questions

It will be useful to put the work presented in this paper into the broader text of our research program Using a naturalistic process-outcome design ofpsychodynamic play therapy with children at an outpatient clinic, our experi-mental study assessed affect expression over the course of treatment using twodifferent kinds of instruments Children’s Play Therapy Instrument is a psycho-dynamically informed measure that aims to assess the structure and narrative

con-of a child’s play activity in psychotherapy [31] The affective dimensions of themeasure allows the rater to code an array of emotions expressed by the childwhile playing The second instrument we use is the automated affective analy-sis model for Turkish language that analyzes affect from text using dimensions

of Valence and Arousal [4] Children’s natural linguistic output over the course

of treatment is assessed with the use of this instrument, and it is this secondinstrument that we describe in detail in this paper

Given the paucity of research with clinical children in treatment, we reporthere a preliminary study which aims to investigate the utility of using an auto-matic text analysis tool to study the relations between affective expression inpsychodynamic play therapy as it relates to different types of psychopathologyand coping and its changes over the course of treatment In terms of the typeand quality of affective expression in play, literature shows that children withbehavioral problems are likely to express more negative affect However, therehave been very few studies that looked at these associations with clinical samples

1 Ethnologue estimates it as 71 millions as of 2006, related Wikipedia content suggests

the numbers to be closer to 80 millions

Affective Dimensions of Play 17

in therapy The first aim of this study was to investigate the relations betweenthe type and quality of affect expressed in play and its relation to type of psy-chopathology Literature shows that different negative emotions relate differently

to Internalizing and Externalizing Problem behaviors In general, irritabilityand anger has been hypothesized to predict Externalizing Problem behaviors,whereas sadness, anxiety, and fear are believed to predict Internalizing Problems(see [20] for a review) Therefore, in our research, we specifically look at Inter-nalizing and Externalizing children’s expression of anger, sadness and anxiety inthe initial stages of treatment, as well as over the course of treatment

Secondly, studies show that the expression of negative affect in play is related

to better coping in the long-run [53] Play provides a context in which a child isable to explore both positive and negative emotional content in a safe, controlledmanner Play ultimately provides the opportunity to increase positive affect andreduce negative affect However, empirical evidence to support this theory withclinical children over the course of treatment is limited The second aim of thisstudy was to assess the type of affect expressed in play over the course of psycho-dynamic play therapy and its relation to different kinds of psychopathologicalfunctioning

Based on literature, several specific hypotheses can be tested for the initialphase of psychotherapy and over the course of treatment The first hypothesis isthat children with Externalizing Problems will show higher levels of anger andlower levels of valence The second hypothesis is that children with InternalizingProblems will show higher levels of sadness, anxiety and lower levels of valence.Finally, we hypothesize that in the initial phase of therapy, both Internalizing andExternalizing children are expected to bring more negative affect (high anger,sadness and low valence) followed by more positive affect (high valence) over thecourse of treatment

The two assessment instruments mentioned earlier, one used by psychologists,the second introduced in this paper, both aim to quantify affect over the course

of the therapy for the investigation of these hypotheses

The paper is structured as follows In Sect.2 we summarize related work

in the area of affective expression in play We broadly describe affect in chotherapy research, specifically discuss the role of text analysis, and then brieflyoverview text analysis for sentiment and affect detection, which is a widelyresearched topic for multimedia and information retrieval Section3 introducesour text analysis system Section4describes the data, and the participants of thestudy Section5 reports our experimental results, including sensitivity analysisfor parameters of the system and ablation study for measuring the contribution

psy-of the different parts psy-of the system Finally, Sect.6concludes the paper

2 Related Work

2.1 Affect in Psychotherapy Research with Children

Affect plays a significant role in psychotherapy, and a model of emotions can

be used to explain different aspects of psychopathology [48] In psychotherapy,

18 S Halfon et al.

the emphasis is on the analysis of affect rather than the elicitation of particularemotions, as the latter is quite difficult Play therapy is one approach to obtainrich behavioral data with affective content

There are numerous studies that link children’s behavior in play to affectivestates Children with disruptive behaviors have been shown to display more neg-ative affect in their play and lower levels of affect regulation [11,17,59] Dunnand Hughes found that children who were hyperactive and displayed conductproblems showed more physical aggression in their pretend games [19] Simi-larly, children with disruptive behavior disorders such as Conduct Disorder andAttention Deficit Hyperactivity Disorder show more hostility and anger in theirplay [14] Von Klitzing et al found that expressing negative and/or aggressiveaffect in disorganized pretend play predicted behavior problems [63]

Russ and Cooperberg found that first and second graders who had morenegative affect in their early play also had more symptoms of depression whenmeasured 10 years later [51] Additionally, in a sample of 322 six year-olds, some

of whom were exposed to cocaine prior to birth, negative affect in play nificantly correlated with both Internalizing and Externalizing behaviors [57].Negative affect in play also correlated significantly with Major Depression Dis-order and Oppositional Defiant Disorder in this study These studies point to theimportance of the relation between negative affect in play and behavioral prob-lems Some studies have also looked at the longitudinal effects of expression ofaffect, especially negative affect in play and behavioral functioning Marcelo andYates evaluated prospective relations among preschoolers’ pretend play, copingflexibility, and behavior problems across varied degrees of child stress expo-sure [35] They found that preschoolers who expressed more negative affect intheir play engaged in more varied coping strategies (i.e., coping flexibility) during

sig-a simultsig-aneous delsig-ay of grsig-atificsig-ation chsig-allenge sig-and fewer Internsig-alizing Problemsone year later These results show that even though expression of negative affectmay initially be related to higher frequency of behavior problems, it may berelated to enhanced coping in the long run [54]

However, there is a gap between the research literature that shows that affect

in play facilitates coping, and the actual process of what happens in play therapywith clinical children in terms of affective changes Bratton, Rhine and Jones,

in a meta-analysis of outcome of play therapy, identified only seven studies thatreported that play overall helped in the reduction of anxiety and fear [7] Thefew empirical studies in the play intervention area that were focused on playwith specific problems found that play reduced fears and anxiety for childrenwith an acute physical illness and separation anxiety [5,41,47] The researchfindings from a variety of studies in the child and adult areas suggest that othertypes of negative affect, like anger should also be helped by play therapies how-ever these studies have not been carried out There is even less research aboutthe kinds of affective transformations that take place over the course of treat-ment Gaensbauer and Siegel found that children who expressed affect in play,especially negative affect, were better able to work through their trauma in play-based therapy [26] According to them, the key element that enables a child to

Affective Dimensions of Play 19

use play adaptively, is the “degree to which the affects can be brought to thesurface so the child can identify them and integrate them in more adaptiveways” (p 297) Singer proposed that children can then increase positive affectand reduce negative affect through play [61] This conceptualization fits withthe idea that play is one way in which children learn to regulate their emotions.However, these ideas need to be empirically investigated

2.2 Assessment Measures of Affect Expressed in Play Therapy

Even though there are many developmental measures to assess children’s tend play skills, there is relatively little evidence-based support for assessmentmeasures that have been developed specifically to assess affective process andchange in child play therapy In particular, self-reported emotions are none tooreliable, as they can be influenced by external factors [56]

pre-Russ and Niec [54], in a review of play therapy assessment measures, talkabout only three measures, which are Play Therapy Observation Instrument(PTOI) [28], the Trauma Play Scale [24] and the Children’s Play Therapy Instru-ment (CPTI) [31], respectively These are specifically designed to study children’sexpression of affect in therapy among other therapeutic indices PTOI includes

an Emotional Discomfort scale to rate child’s comments about worries and blesome events, inappropriate aggression toward the therapist, conflicted play,the quality and intensity of the child’s affect (i.e., mood), and play disruption.The Trauma Play Scale allows for the coding of negative affect or lack of joyduring play CPTI has a more extensive affective component assessing affectregulation strategies as well as the types of affect expressed in play over thecourse of treatment With all these measures, the sessions have to be recorded,transcribed and rated by trained judges on affective components

trou-2.3 Automatic Text Analyses of Affect from Text in Psychotherapy Research

A primary focus of the use of natural language processing (NLP) methods in chotherapy has been to evaluate complex relational/emotional processes usingthe words from treatment sessions Much of this work has involved the use ofcomputerized dictionaries that place specific words in psychologically mean-ingful categories For example, Anderson and colleagues found that when thepatient used more emotion words, therapists obtained better outcomes whenminimizing responses with cognitively geared verbs (e.g., “think,” “believe,”

psy-“know”) [3] Mergenthaler focused on the emotional tone (density of emotionalwords) and level of abstraction (the amount of abstract nouns) within patients’language and found that successful outcome in psychodynamic therapy is asso-ciated with increased use of emotion and abstraction in language, which showsthat the patients have emotional access to conflictual themes and can reflectupon them [38,39]

Bucci’s Referential Process theory is a similar, but more comprehensive chological construct that “concerns the degree to which speakers (or writers)

psy-20 S Halfon et al.

are able to access nonverbal, including emotional experience, in their own mindsand to express this verbally in a form that is likely to evoke a correspondingexperience in the listener” [9]) The affective connection between the languageused and the underlying emotions has been consistently correlated with clinicalratings of psychoanalytic session effectiveness [10] Pennebaker did not specif-ically investigate psychotherapy transcripts; however analyzed the writing fea-tures most strongly associated with enhanced psychological and physiologicalhealth found that people whose stories contained a high rate of what he calledemotional processing words (e.g., “sad,” “hurt,” “guilt,” “joy,” “peace”), insightwords (e.g., “realize,” “understood,” “thought,” “know”) and causal words (e.g.,

“because,” “reason,” “why”) showed the greatest benefit from expressive ing exercises [45] Even though there is substantial research in the application ofNLP methods to specifically assess affective processes in adult treatment, to thebest of our knowledge, no research has been carried out to adapt these measures

writ-to psychodynamic play therapy and there are no such resources in Turkish

2.4 Text Analysis for Sentiment and Affect Detection

In multimedia computing, sentiment analysis and opinion mining refer to thecategorization of a given text into positive, negative, or neutral classes, whichmakes it a relatively restricted and practicable NLP problem On the other hand,detecting affect from text is a more challenging task, as it requires a profoundunderstanding of both semantics and syntax of a language, as well as representingaffect with the appropriate emotion categories or dimensions

There exist several approaches to extract sentiment and opinion from textualmultimedia content such as blogs, tweets, movie reviews and customer reviews.Basic methods include keyword spotting, lexical affinity, statistical NLP, learningbased methods and commonsense-based approaches [13,44] Similarly, methodsfor affective content analysis from text generally blend these approaches withrule-based systems An example is the Affect Analysis Model, which analyzesaffect specifically in informal online communication media [43] This approachhas five main steps; symbolic cue analysis, syntactic structure analysis, word-level, phrase-level and sentence-level analysis, respectively

The majority of research on affect analysis from text relies on lexicon-basedapproaches, in which a set of keywords and associated affect categories are used

to generate features for affect prediction models One of the comprehensive ical resources in this area is the Affective Norms for English Words (ANEW)corpus [6], which includes a set of normative emotional ratings for 1,034 com-monly used English words This tool represents a set of verbal materials thathave been rated in terms of pleasure, arousal, and dominance to support emotionstudies Similarly, WordNet-Affect is a well known linguistic resource for extract-ing emotions from text [62] The starting point of WordNet-Affect is to build

lex-a hierlex-archy of lex-affective domlex-ain llex-abels by llex-abeling synsets (lex-a set of one or moresynonyms) that express affective concepts based on WordNet Domains [34]

A powerful system for text analysis is Linguistic Inquiry and Word Count(LIWC), which has a comprehensive affective dictionary to analyze text based

Affective Dimensions of Play 21

on grammatical, psychological, and content word categorization This dictionaryallows to measure 74 different linguistic dimensions with more than 2,200 wordsand word stems Affect sensing methods that are based on LIWC calculate wordcounts in the input text depending on these linguistic dimensions [27,30,46]

In addition to these lexicon-based approaches, several alternative methodshave been studied in textual affect analysis For example, Liu et al first pro-posed the Commonsense-based approach by using three real-world commonsensedatabases [33] Brooks et al [8] presented an automated affect classification sys-tem in chat logs exploiting NLP and machine learning techniques Their systemsegments the chat data and makes use of an improved bag-of-words model toclassify text into 13 affect categories The basic drawback of machine learningapproaches is that they usually lack linguistic analysis by mainly focusing onstatistical and syntactical features

Recent approaches to text-based sentiment analysis rely on co-occurrencestatistics, and in a multimedia context, typically combine image analysis withtext [65] To derive fixed length descriptors from variable length text fragments,the unsupervised Paragraph Vector approach proposed by Le and Mikolov isfrequently used [32] N-gram based generative approaches have shown somepromise [40] An example work for rule-based systems is Vader, which is tai-lored for social media text [29] A recent review encompassing many applicationdomains is given in [42]

3 The Proposed Text Analysis System

The automated affect analysis tool that is used in this work is designed to analyzeaffect and sentiment in Turkish online communication texts across domains [4].Because of the lack of comprehensive Turkish corpora for affect analysis, we use

an affect lexicon which is adapted from English to Turkish English lemmas weregathered from the study of Warriner et al [64], which evaluated 13,915 Englishlemmas in a nine point scale (1–9) by 1,827 participants through MechanicalTurk For each item, mean and standard deviation values for valence, arousal,and dominance scores are given Our text analysis model linearly transformsthese affect scores to a five point scale [1–5] Mapping to this range makes thescores given by the system directly comparable to the CPTI scores The affec-tive lexicon was expanded with synonym sets (synsets) from a standard Turkishdictionary (by TDK, Turkish Language Organization) As a result, a comprehen-sive affective lexicon for Turkish is developed, which includes valence, arousal,and dominance scores for 15,222 different words and phrases We note here thatthe translation process naturally introduces errors, and ignores cultural aspectsentirely Nonetheless, this approach produces a useful resource with little cost

To deal with written communication, the model uses additional resources,including 120 emoticons, 98 abbreviations, 50 interjections, and 71 modifiers(emotion intensifiers and diminishers) The affect analysis model of the tool isillustrated in Fig.1

In order to calculate sentence-level affect scores, the system first calculatesthe affective values of small units in the sentence, such as words and phrases, by

22 S Halfon et al.

Fig 1 Overview of the affect analysis system.

tokenizing the sentence into trigrams, bigrams and unigrams Next, the systemchecks the modifier list If there is any modifier connected to a verb or a noun asphrasal, the score of the word is updated based on the polarity of the sentence and

on the particular coefficient of the modifier Then, the system handles negationand some morphological alternations and updates the valence and arousal scoresaccordingly

The system exploits some linguistic rules when calculating the overall tence score, based on simplifying assumptions For example, considering thetransitive verbs in Turkish, for NN+VB structures, such as “hayatını kaybetti”(he lost his life), only the affective score of the verb is taken and then the noun

sen-is neutralized Similarly, if there sen-is a NN+ADJ structure such as “kafam karı¸sık”(I’m confused), the noun is neutralized and only the adjective is taken into con-sideration The overall sentence score is computed by summing the scores ofthese units Only words with affective load are considered in the summation

3.1 Adaptations for Psychotherapy

The initial design of this system targeted general online communications [4] As apart of this work, we adapted the system to the psychotherapy domain by updat-ing the affect dictionary During the translation of the dictionary, the primarymeanings were used for each word, but synonyms were also stored as alterna-tives We checked approximately 1,500 words manually and selected the wordwith the most appropriate sense in the psychotherapy domain and discarded theothers Another feature that we added to the system is the detection of frequentstop-words and redundant words in play therapy For example, words such as

“anne” (mother), “baba” (father), “oyun” (play) have high valence scores in ourdictionary, however, these words are mostly present with a neutral tone during

Affective Dimensions of Play 23

the play sessions Therefore, we neutralized the affect scores of these words whencalculating the overall affect score Stop-word and redundant word lists includemore than 500 words that we have treated as neutral words We suggest that toadapt the system for a different domain, expert knowledge should be integrated

at this level The resources and code developed for this work is made available

We next describe the experimental setup We evaluate our approach on datacollected during psychotherapy sessions, and contrast our findings with those ofthe expert psychotherapists

4 Experimental Setup

We describe the experimental setup somewhat extensively here; the reader mayskip to Sect.4.3for the details of data analysis and results

4.1 Data

Patients The source of data used for this study comes from the Istanbul Bilgi

University Psychotherapy Research Laboratory, which provides low-cost tient psychodynamic psychotherapy and professional training at master’s levelfor students in the Clinical Psychology Program Referrals were made by par-ents themselves or by mental health, medical, and child welfare professionals.The parents and the children were interviewed in order to determine whetherthe patients fit the study protocol inclusion criteria: ages between 4–10 yearsold; average intelligence; motivation for treatment; no psychotic symptoms; nosignificant developmental delays; no significant risk of suicide attempts; no drugabuse The patients and their parents were extensively informed before com-mencing therapy and consented to video recordings and data collection at alltimes The parents provided written informed consent and the children providedoral assent concerning use of their data for research purposes

outpa-From September 2014 to September 2015, a group of 26 consecutively ted patients who met inclusion criteria and consented to research were included

admit-in the study 20 patients (76 %) completed the treatment The demographics ofthe children are presented in Table1 Eighty to ninety percent of the childrencome from low to middle socioeconomic status (SES) families and approximately

10 % of the parents are divorced or widowed for both samples Referral lems manifested primarily as anger management issues and behavioral problems,such as disobedience and not taking limits, followed by academic issues such asinattention in class and low grades and finally relational problems such as diffi-culties in family relationships or socialization with friends At intake, 5 patientshad DSM IV Attention Deficit and Hyperactivity Disorder, 3 patients had aMood Disorder, 3 had Separation Anxiety Disorder, 2 patients had Encopresis

Prob-Therapists A total of 12 therapists (all clinical psychology master’s level

grad-uate students) treated the 20 patients, with each therapist generally workingwith one to two patients The therapists were all females with ages ranging

24 S Halfon et al.

Table 1 Subject characteristics at intake

Subject characteristics at intake (N = 20)

Treatment The treatment was psychodynamic play therapy The treatment

was not manualized and the only restrictions placed were regularity and length(once weekly treatment of 50 min for one year) Patients on average received 40sessions Even though there is no unitary model of therapeutic action in psy-chodynamic play therapy [25], the core principles and techniques employed can

be summarized as follows: Central to this approach is the establishment of what

is called a “setting” The psychotherapist sees the child at regular times, in thesame play room with a standard set of play toys This consistency provides asafe context that allows the child to play out difficult and disturbing emotionalexperiences that would be hard to express in the outside world The exploration

of the child’s issues takes place in a largely child-led process way and the pist encourages the child to express and reflect on his perceptions, feelings andthoughts in play This is done by listening actively and inviting the child to con-tinue his communications and asking questions about the play setting, temporalordering, and the details of the characters, their thoughts, feelings and behaviors.The therapist also labels the repetitive themes, conflicts and feelings in play withthe aim of helping the child to synthesize his experience Interpretations aim tohelp the child see links between conflicting needs and emotions about self andothers that find reflection in play behaviors and in the therapeutic relationship

thera-Affective Dimensions of Play 25

with the purpose of bringing to consciousness attitudes, assumptions and beliefs

of which the child is unaware

Session Selection For correlational analyses, the longest play segments of

the first two sessions of psychotherapy were used A total of 40 sessions and 40play segments constituted the data points for the analysis To run Multi-levelModeling and Trend Analyses, a total of six sessions were selected from eachcase To represent different therapy phases, the sessions were divided into early,middle, and late phase by dividing the total number of sessions for each case bythree Two consecutive sessions were selected from early therapy, two from latetherapy, and two from the middle Each session included 1 to 10 play segments(see Sect.4.2), with a mean of 2.3 Up to four play segments were selected from

each session in order to achieve a balance among participants, since the number

of play segments per session varied This sampling resulted in 120 sessions and

289 play segments for 20 children

4.2 Measures

Background Information Demographic information such as socioeconomic

status and marital status were obtained using a standard intake informationform and from information obtained in the initial interview

Outcome Measures The Child Behavior Checklist (CBCL) is a widely used

method of identifying problematic behaviors in children [1] For children ages

4 through 18, a parent or a primary caregiver reports on the child’s academicperformance, social relationships, and indicates how true a series of 112 prob-lem behavior items are for the child on a 3 point scale (0 = not true, 1 =somewhat or sometimes true, and 2 = very true or often true) The followingeight syndromes are scored from the CBCL, Anxious/Depressed, Withdrawn/Depressed, Somatic Complaints, Social Problems, Thought Problems, AttentionProblems, Rule Breaking Behavior, Aggressive Behavior Anxious/Depressed,Withdrawn/Depressed, and Somatic Complaints syndromes comprise an Inter-nalizing group, and the Rule Breaking Behavior and Aggressive Behavior syn-dromes comprise an Externalizing group, and Total Problems is the sum ofscores on all problem items The cut-off points for borderline and clinical desig-nation are based on t–scores formed on a clinical population Back translation,bilingual retest method, and pretest studies were used for the translation of theCBCL [22] The test–retest reliability of the Turkish form was 84 for the TotalProblems, and the internal consistency was adequate (Cronbach’s alpha = 88;[21,22])

Assessment of Affect in Play Activity Children’s Play Therapy Instrument

(CPTI) is a psychodynamically-informed measure of in-session play activity [31].The selected scales of the instrument for the purposes of the study involve Seg-mentation and Affects Expressed in Play (for further definition of play activity

26 S Halfon et al.

categories, see [15]) The CPTI rates children’s behavior in a therapeutic setting

at different levels The first level involves a “Segmentation of the child’s activity”(non-play, pre-play, play and interruption) Going forward, only play segmentsare rated The Affective Component looks at the types emotions brought by thechild to his play Eight types of emotions are rated using a 5-point Likert scale:

5 = Most Characteristic; 4 = Considerable Evidence; 3 = Moderate Evidence;

2 = Minimal Evidence; 1 = No Evidence For the purposes of the study, onlyAnger, Anxiety and Sadness were coded Two masters level clinical psychologystudents, who received 20 hours of training on the CPTI by the first authorand rated 10 training sessions (24 play segments) prior to the study, rated thesessions They were independent assessors who were not associated with thetreating clinicians or the cases, and blind to the purposes of the study In order

to identify the agreement level between judges for subscale ratings, Intra-classCorrelation Coefficients (ICC) were computed Cronbach Alpha was 72 for Seg-mentation, and 81 for Affect Types, suggesting good reliability for all Scales ofCPTI

Valence and Arousal Categorical and dimensional modeling are two main

approaches in representation of affect [12] In dimensional modeling, the tion is that emotions are related to each other and the affective state is investi-gated in a continuous multidimensional space, in generally two or three dimen-sions There is still a lack of consensus on which dimensions are fundamental andwhich dimensions are a mixture of these basic dimensions However, the popularCircumplex model of emotions [49], which defines “valence” and “arousal” as theprincipal axes, is frequently used Valence describes the extent of pleasure (pos-itive) and sadness (negative), and arousal (or activation) describes the extent ofcalmness and excitation [49,55] Valence and arousal are commonly considered

assump-as independent dimensions, however, real-world findings confirm that these twodimensions are correlated most of the time

4.3 Method of Automatic Analysis

As a general rule, linguistic programs need to segment the transcript (typically

in equally sized units) for comparison of the data while analyzing a text Asthe proposed text analysis tool performs sentence level analysis, firstly we had

to segment sessions into smaller units The length of a scoring unit containingthe minimum number of necessary words is determined by statistical proceduresdescribed before [36] In psychotherapy research, an entry with minimum of 150words is required by many linguistic programs such as the therapeutic CycleModel and computer-assisted content analysis [37] Therefore, for the grouping,

we created 150-word chunks of sentences while paying attention to play segmentborders Each 150 word block was processed as a single sentence in our affectanalysis system Then, average scores of these 150 word blocks gave us the overallaffect score of the corresponding therapy session

Affective Dimensions of Play 27

5 Results

5.1 Descriptive Statistics

To examine the association between CBCL problems and affect expressed inplay at the beginning of therapy, play affect scores as measured by CPTI Anger,Anxiety and Sadness scores, VA (Valence and Arousal) scores collected in theinitial two sessions of psychotherapy were calculated Each child’s two longestplay segment affect scores from the initial two sessions were computed, whichgave mean affect scores for the initial phase of psychotherapy The means andstandard deviations for each of the major variables collected at the beginning

of psychotherapy are listed in Table2 The first two rows (Valence and Arousal)are obtained by the proposed automatic analysis approach, and the next threerows are CPTI annotations (Anger, Anxiety and Sadness)

Prior to testing correlations, the possible contribution of background anddemographic variables to the studied variables was examined through prelimi-nary analyses Spearman correlations were conducted to assess the association

of age and gender with the main study variables: CBCL Problems and all CPTIItems and VA No significant differences were found according to these variables

Table 2 Descriptive statistics for affect variables and CBCL problems (N = 20)

Internalizing Problems 59.95 11.14

5.2 Preliminary Results of Affect Analysis at the Beginning

of Treatment

The relationship between the CBCL Problems and play affect scores as measured

by CPTI Anger, Anxiety and Sadness scores and Valence and Arousal scorescollected in the initial two sessions of psychotherapy were examined Due to thelow number of children included in the analysis, Spearman Correlations wereused (see Table2)

Results show that in the first two sessions, CPTI Anger scores were positivelyrelated to Externalizing Problems, Valence scores were negatively related to

28 S Halfon et al.

Table 3 Spearman correlations between the affect scores and CBCL problems.

CPTI anxiety CPTI sadness CPTI anger VA valence VA arousalCBCL

Note:aCorrelation is significant at the 05 level;bCorrelation is significant at the 01 level.

Internalizing and Externalizing Problems, and Arousal Scores were negativelyrelated to Internalizing Problems on the CBCL No significance was observedfor CPTI Sadness and Anxiety scores (see Table3)

While we do not analyze the specific findings of the play therapy sessions indetail here, we note that the high correlations obtained by the proposed auto-matic tool and the manual CPTI coding are very promising The results provideempirical support for two measures of affective assessment that can be usedtowards investigating affective processes in play in psychodynamic play therapy.Both CPTI and Valence-Arousal showed preliminary promise for systematic playobservation

5.3 Preliminary Analyses of Affect Expressed During Treatment

In order to assess affect expressed during the treatment, two sessions from thebeginning, middle and end of therapy were used As such the data consisted of

6 sessions from 20 children resulting in 120 sessions and 289 play segments Weconducted Hierarchical Linear Modelling (HLM) [50] which is used to measuredata that has more than one level Using Hierarchical Linear Growth CurveModeling, affective change over time was modeled This model takes into accountthe hierarchical structure of the data i.e., different measurements in time (level1) are nested within subjects (level 2) Using maximum likelihood, multilevelanalysis allows for missing data [60] Effect-sizes were calculated using R2

To see the variability of mean valence, arousal and anger scores, first null

models were run for each Results showed that Valence (β= 3.39, t(19) = 124.36,

p < 0.001), Arousal (β = 3.54, t(19) = 41.64, p < 0.001) and Anger (β = 2.59, t(19) = 18.09, p < 0.001) significantly varied across participants.

Results also revealed that left over variance was significant for Arousal

(Var(u0) = 0.08, χ2 (19) = 49.49, p < 0.001), Anger (Var(u0) = 0.27, χ2 (19)=

54.36, p < 0.001), but not for Valence (Var(u0)= 0.00, χ2(19)=15.34, p >0.05).

We also calculated how much of the variance is explained by level 2 variables(Externalizing and Internalizing problems) in predicting Valence, Arousal andCPTI Anger To calculate this we used the following formula:

Explained variance = u0(unconditional) − u0(conditional)

Affective Dimensions of Play 29

Because HLM does not give a direct R2 value, the variance explained withthis formula can be used as pseudo R2 [2]

We found that for Arousal 16 % and for Anger 19 % of the variance isexplained by Externalizing and Internalizing Problems For Valence, we couldnot obtain a value because left over variance was not significant at null model

as stated above Together, these results indicated that further analysis usingHierarchical Linear Modeling, was suited

Growth Curve Analyses To investigate the change in Internalizing and

Externalizing children’s Valence, Arousal and Anger scores across sessions, timeand time squared variables were entered into the model at level 1 to see the lin-ear and quadratic growth rates of variables For Internalizing Problems, results

revealed a significant linear increase (β = 0.05, t(161) = 2.98, p < 0.05) in Valence as well as Arousal (β = 0.06, t(161) = 2.17, p < 0.05) scores Effect

sizes for each trend was small (R2 = 0.01) No significance was observed for

linear (β = -0.02, t(161) = 0.18, p >0.05) and quadratic effect for CPTI Anger (β = -0.03, t(161) = 0.58, p >0.05) Growth rates of Valence, Arousal and CPTI

Anger with Externalizing Problems were not significant

5.4 Ablation Study

We assess the impact of different parameters on the accuracy of our affect diction system To achieve that, we setup a sentence-level annotation with 4 dif-ferent play therapy sessions that includes approximately 500 sentences in total.For each sentence, a human annotator assigned a Valence and an Arousal score

pre-by using a 5-point Likert scale After the automated affect analysis, we comparedthe model prediction scores with the ground truth scores that we obtained fromthe annotation Model scores are also scaled continuously between 1 and 5 In

order to calculate the accuracy, we mapped the Valence scores to positive (>3) and negative (< 3) classes to carry out the corresponding classification of the

affect

The first experiment we conducted tested the benefit of using domain tion on the text analysis system As can be seen from Table4, with the updateddictionary and redundant word elimination, we observed a higher correlationand reduced mean square error in both Valence and Arousal dimensions

adapta-Table 4 The effect of adapted dictionary for psychotherapy domain

30 S Halfon et al.

Table 5 The accuracy of the model for binary Valence classification

Accuracy (%)

All features with generic dictionary 74.5

All features without redundant words 81.1

All features without negation 75.1

All features without modifiers 79.8

Contribution of the different parts of the system to the performance is given

in Table5 Our results show that the system gives the best accuracy (83 %)when all features are employed with the adapted dictionary for psychotherapydomain We see that eliminating the domain-specific redundant words improvesthe system performance by 2 %

6 Conclusions

There is relatively little empirical investigation of the measurement of affectexpressed in play and how it relates to psychopathology during the treatmentprocess of children in psychodynamic play therapy We propose in this paper anautomatic rule-based text analysis tool that can quantify Valence and Arousalfor longitudinal transcriptions of therapy sessions We obtain good agreementwith a standard measure used by psychotherapists Result of the study sup-port the relationships between affect expressed in play and behavioral problems

as well as the importance of play in the modulation of negative feelings Thefindings were consistent with our prediction which indicated Internalizing andExternalizing Problems negatively associated with Valence at the beginning oftreatment These findings parallel previous results from the literature that sug-gest a relationship between negative affect in play and maladaptive behavior.Our findings also indicated, in line with previous literature, that children withInternalizing Problems present with a constricted range of negative affect andcan use psychodynamic play therapy towards the modulation of negative affect

in play They are able to express more intense and positive emotions over thecourse of treatment as shown in the increase in Arousal scores These findingsprovide preliminary empirical support for two measures of affective assessmentthat can be used towards investigating affective processes in play in psychody-namic play therapy

One of the main limitations of the study is that none of the existing based sentiment analysis approaches could be directly employed for comparativeassessment, as few approaches are proposed for Turkish (see [4] and referencestherein) It is obvious that improvements in the automatic affect analysis pipelinewill translate to more reliable assessment of the play therapy sessions In par-ticular, a comprehensive affective lexicon prepared for Turkish language would