The Psycholinguistic Dimension in Second Language Writing: Opportunities Jor Research and Pedagogy Using Computer Keystroke Logging

The pattern of cognitive processes cre-ated during writing may differ both between writers and within writers,depending on factors such as writing task Severinson Eklundh, 1994,writing m

The Psycholinguistic Dimension in

Second Language Writing: Opportunities for Research and Pedagogy Using

Computer Keystroke Logging

KRISTYAN SPELMAN MILLER

to observe the process without the obvious intrusion of cameras or server-researchers As a tool for gathering data on aspects of writing,computer recording (or keystroke logging) offers researchers an alter-native to introspective methods such as think-aloud protocol and makes

ob-accessible a mass of detailed information about the processes involved inproducing text (Stevenson, 2005; Strömqvist, Holmqvist, Johansson,Karlsson, & Wengelin, 2006; Sullivan & Lindgren, 2006; van Waes, 1991;Wengelin, 2006).

The aim of the present article is to evaluate the contribution of puter-logging research with reference to second language (L2) writersand to consider possible applications to language teaching We presentdata from a longitudinal study of young L2 writers of English in Sweden

com-in order to illustrate the types of com-insight offered com-into pauscom-ing, fluency,and revision behaviour and to connect these findings to such issues ascognitive capacity, working memory, and automaticity The relevance ofthis tool within language pedagogy and its potential to prompt recall andself-reflection are then considered in relation to the individualisation ofteaching and the promotion of self-assessment, metacognitive awareness,and learner autonomy

PSYCHOLINGUISTICS AND WRITING RESEARCH

Research into writing and particularly into L2 writing has not alwaysdefined itself closely in relation to mainstream psycholinguistic research,with the result that writing is sometimes seen as “a stepchild of psycho-linguistics” (Bonin & Fayol, 1996, p 145) The rise in importance in the1970s and 1980s of the process perspective on writing (e.g., Emig, 1971;Flower & Hayes, 1981; Perl, 1979; Sommers, 1980), however, gave promi-nence to the study of the cognitive processes underlying written textproduction and had significant impact on both research and pedagogy,and in particular L2 writing (Raimes, 1985; White & Arndt, 1991) Thecognitive perspective established writing as involving a complex set ofhierarchically arranged cognitive activities, involving problem-solvingand decision-making within clearly defined goals (Kellogg, 1994, p.13).Flower and Hayes’s (1981) model identified three main composing pro-cesses—planning, formulating, and reviewing—which are “interactive,intermingling, and potentially simultaneous” (Grabe & Kaplan, 1996, p.91), with transition between these processes and their subprocesses oc-curring via a monitor Rather than being a linear sequence of processes,then, composing is recursive, with one process potentially triggering andtriggered by another

More recently, the concern in L2 writing with broader socioculturaland discoursal aspects of composing has distanced some writing re-searchers (Atkinson, 2003; Kent, 1999; Matsuda, 2003; Tobin, 1994;Trimbur, 1994) from the cognitivist stance in favour of a socially situatedperspective on the writing event However, in this so-called postprocess

era, some researchers also acknowledge that a cognitivist stance does notnecessarily imply a reductive, abstract view of writing (Spelman Miller,2005) Rather, researchers recognize that the internal and external con-texts of composing are complementary and mutually reinforcing.The early models of composing (e.g., Bereiter & Scardamalia, 1987;Flower & Hayes, 1981; Hayes, 1996), which have been so influential,identified the interacting processes of planning, translating (formulat-ing), and reading/revising which the writer juggles in order to meet thegoals and intentions of composing Not only do these processes interactmore or less simultaneously with one another during the activity of com-posing, but they also trigger one another (Rijlaarsdam, Couzijn, & Vanden Bergh, 2004) For example, after reading parts of the text composed

so far, inconsistencies can be noticed which lead to revision Reading canalso trigger the discovery of new ideas or relationships in the text andlikewise result in revision (Hayes, 2004) The changes made may in turnresult in a need to plan the following section, so as to fit it in with therevised text, and the planning may in turn result in more revision Cog-nitive processing during writing thus forms a recursive pattern in whichthe different activities—planning, formulation, transcription, reading,and revising—interact These activities can leave traces in the writingprocess in the form of pauses, insertions, deletions, and movements.Taken together, the numerous, detailed, and complex traces form aweb of interacting events, constructed and constrained by one specificwriter under specific conditions The pattern of cognitive processes cre-ated during writing may differ both between writers and within writers,depending on factors such as writing task (Severinson Eklundh, 1994),writing medium (Haas, 1996; Van Waes & Schellens, 2003), writing ex-perience (Bereiter & Scardamalia, 1987), disability (Holmqvist, Johans-son, Stromqvist, & Wengelin, 2002; Strömqvist, Ahlsén, & Wengelin,1999; Wengelin, 2007), and language (first or additional) (New, 1999;Raimes, 1987; Thorson, 2000) In the following section, we briefly con-sider how we might access information about these cognitive processes,focussing on the insights offered by computer recording

COMPUTER-BASED OBSERVATION

A number of publications focussing on writing research methodology(Severinson Eklundh & Kollberg, 1996; Spelman Miller & Sullivan, 2006;Strömqvist & Ahlsén, 1999) have already articulated the main features ofcomputer logging as a research tool In brief, this tool records writingsessions and stores in a log file detailed information about the time andoccurrence of every keystroke, providing indirect evidence of cognitive

activity during online writing As an observational tool, keystroke loggingshares some of the features of face-to-face observation techniques, such

as those used by Matsuhashi (1981, 1987), and more recently by Bosher(1998), Chanquoy, Foulin, and Fayol (1996), Janssen, van Waes, and vanden Bergh (1996), Phinney and Khouri (1993), and Schilperoord(1996), who all use video technology Unlike think-aloud protocols,which are widely used in cognitive research, the data are not generated

by the writers themselves The use of self-generated data, although stillwidely used, is not without the problem of reactivity, that is, interferenceand distortion of the process itself, which can open such data to criti-cisms (see, e.g., Janssen, van Waes, & van den Bergh, 1996, for a discus-sion of this issue) Furthermore, the advent of computer recording asopposed to video or direct observation has provided a more versatile andunobtrusive means of data collection because the resident software re-cords all keystroke presses (characters and actions such as deletions,cursor movements, spacebar presses, and so on) and yet appears as anormal text editor

A variety of logging programmes are used in keystroke studies, ing JEdit, used in the current study (IPLab, 1997; see Severinson Ek-lundh & Kollberg, 1996), Scriptlog (see Strömqvist & Ahlsén, 1999), andInputlog (see Leijten & van Waes, 2006) The data file produced by thelogging programme (see sample in Figure 1 taken from our study1)presents information concerning the location and duration of pauses

includ-1 See Appendix B for the textual output of this logfile.

FIGURE 1 Logfile from the text by Writer 2, Year 2.

(normally identified as those nonwriting occurrences longer than 2 onds in duration, and indicated as a bracketed < > number), characterand punctuation presses, and space bar and other moves including cur-sor movements and deletions Use of the space bar is indicated by theunderscore symbol; the delete key is represented as a crossed blockarrow, with the number of deleted keystrokes presented immediatelypreceding the symbol; the number of spaces moved with the arrow keysand the direction of the movement are also indicated numerically Thelogfiles contain a wealth of information concerning operations on thetext, and this information can be used to replay the writing event revision

sec-by revision to elicit later reflections sec-by the writer on his or her actionsand intentions In this way, composition may be viewed within the broad-est context, that is, in terms of the writer’s response to the demands of

a particular task, including decisions concerning genre, audience, andtopic development in addition to linguistic choices

SECOND LANGUAGE WRITING: APPLICATIONS

AND INSIGHTS

One of the key issues for exploration is the way in which the writerjuggles the complex web of interacting cognitive activities During com-position, writers plan content and form, consider audience and style,reread and revise If writers are inexperienced and/or are writing in aforeign language, the complexity of the task increases As a consequence

of their lack of automaticity, L2 writers can be forced to focus theirattention and cognitive resources on specific aspects of their writing(Graham, Berninger, Abbot, Abbot, & Whitaker, 1997; Graham & Harris,2000; McCutchen, 1996, 2000) Other aspects might then be left unat-tended to, and the quality of the final text could suffer Raised awareness

of which processes interact during writing in the first language (L1) and

in the L2, and how they interact, can assist writers in reducing cognitiveload during writing and promote writing development

One major difficulty writers have to struggle with during writing is that

of distributing their limited cognitive resources between the various straints of writing (McCutchen, 1996, 2000) Language, topic, content,goals, style, and genre are only a few of the items they need to considerwhile writing in order to achieve a good text (Chenoweth & Hayes, 2001;Hayes, 1996) In the case of the L2 writer, attention to lower level lin-guistic demands during writing may take up a large amount of the writ-er’s working memory resources As a result, less cognitive capacity isavailable for higher level processing of, for example, content, audience,and style (e.g., Schoonen et al., 2003; Whalen & Ménard, 1995)

con-Insights into the L2 writer’s cognitive activity have been derived from

a number of studies using keystroke logging These studies have focussed

on a range of variables, including different task types (Severinson lundh, 1994; Spelman Miller, 2000a, 2000b), ages of writer (Pålson,1998), and contexts (quasipedagogic and classroom) (Lindgren, 2004;Sullivan, Kollberg, & Pålson, 1998; Sullivan & Lindgren, 2002; Stevenson,2005; Thorson, 2000) Keystroke logging has also been used in research

Ek-on translatiEk-on (Englund Dimitrova, 2006)

Results from these studies have highlighted differences in pausing andformulating behaviour between first and L2 writers, which are consistentwith Silva’s (1993) conclusion that L2 writers plan less at the global level(Dennett, 1985; Jones & Tetroe, 1987; Moragne e Silva, 1989; Yau, 1989),and display a slower rate of production and lower productivity (Hall,1990; Hildenbrand, 1985) Whalen and Ménard (1995) found that writ-ers in L1 (English) and L2 (French) planned and evaluated in their L2more frequently at a lower linguistic level and less frequently at a higher(textual and pragmatic level) level Spelman Miller’s (2000b) findingsconcerning two groups of L1 and L2 student writers working on aca-demic tasks in a British university context also confirm group-level dif-ferences in fluency, in particular, in the location and duration of micro-and macroplanning pauses and length of text span produced Planningpauses appeared to be more frequent at phrase-level locations, in par-ticular at the point in the clause where the thematic element is intro-duced In L1 writers, by comparison, more significant pausing activityoccurred at the ends of clauses; this behaviour underpins a difference inthe approach to the demands of the task Because L2 writers have tojuggle demands on their attention in relation to the linguistic and dis-course aspects of the text, planning and revision by L2 writers tend tocoincide at sites which are critical in foregrounding or backgroundinginformation and, therefore, in controlling the development of the topic

In this way, the micro analyses reveal sites of conflict and making within the construction of text, so facilitating “new insight intothe interplay of cognition and context” (Flower, 1994, p 55)

decision-Other research into the fluency of L2 writers, both online and offline(Chenoweth & Hayes, 2001; Kellogg, 1994; Ransdell, Levy, & Kellogg,2002; Schoonen et al., 2003; Whalen & Ménard, 1995; Stevenson, 2005),provides further support for these findings Fluency, defined as efficientaccess to linguistic knowledge and retrieval of linguistic form (van Gel-deren & Oostdam, 2002), is a cognitive construct which presupposes theavailability of sufficient linguistic knowledge to formulate ideas into lin-guistic form (Stevenson, 2005, p 136) When the writer has developedhighly proceduralized rules for language production, retrieval can bemore rapid, which frees up cognitive capacity from working memory Byextension, the lack of fluency occurs when increased cognitive effort is

devoted to one component process, such as formulation, with the sequence that a greater burden is placed on working memory for othercomponent processes, such as handling higher level concepts.

con-In a study of L1 (Dutch) and L2 (English) writers, Stevenson (2005)considered the extent to which higher level processing in L2 writers wasinhibited by the need to divert attention elsewhere Levels of fluencywere ascertained using both offline and online measures (respectively,the number of words occurring in the final text divided by the total tasktime and the average number of intact recognisable words producedbetween pauses of 2 seconds or more, regardless of whether the wordsoccurred in the final product) Higher level processing was investigated

in terms of time devoted to processing and the rhetorical structure of thewritten product Stevenson’s findings confirm that writers appearedsomewhat inhibited in their higher level processing in English, with lessattention devoted to conceptualising in English and the English textproducts containing fewer subarguments Extra attention to linguisticprocesses was identified through the greater number of clause-internallanguage revisions carried out at the point of inscription, more localisedreading of the text, and the strategies to solve language problems Lessattention was given to conceptualising in L2 than in L1, and the L2 textswere rhetorically less well developed than the L1 texts However, thestudy was not able to establish that the inhibition of higher level pro-cessing in L2 was related to lack of fluency at a linguistic level This studyappears in general to support capacity accounts of writing (Kellogg,1994; McCutchen, 1996), which assert that inhibition of conceptual pro-cessing occurs in L2 writing However, the specific aspects affected by aswitch of attention to linguistic processes remain complex and call forfurther research

THE CURRENT STUDY

The study which we discuss in this article builds on the research ported in the previous section in considering the cognitive processes ofL2 writers using keystroke logging The data we present are part of alarger study comparing the writing of a cohort of young writers compos-ing in both L1 (Swedish) and L2 (English) over an extended period Thefocus for this particular report is on L2 writing performance over the3-year data collection period It is concerned with pauses, fluency, andrevision, and how they relate to text quality, as measured by a compositescore of content, range, complexity, accuracy, and fluency This longi-tudinal focus adds a new dimension to the studies of L2 process researchand allows us to address the following research questions:

re-1 How does linguistic experience/year of study affect the online ing process in terms of revising, pausing, and fluency?

writ-2 Are there any online writing process variables which can be fied as contributing to text improvement?

identi-Participants

Seventeen high school students took part in the study The pants were all in Year 7 in the Swedish school system (that is, 14 yearsold) at the beginning of the study, and they all attended the same school

partici-in a Swedish town All were monolpartici-ingual native speakers of Swedish who,

at the start of the study, had been learning English for 3.5 years All theparticipants were volunteers and their parents or guardians had givenwritten permission for their participation in the study

The high school students came from two different school classes: the

majority (n = 12) had been on a 50% English immersion program for one term at the start of the data collection; the remaining students (n =

5) studied all their subjects in Swedish The content of the curriculumwas the same for all the students: All 17 received three 40-minute Englishlanguage classes per week The immersion–nonimmersion learning con-text was originally made a variable, although our results indicate that it

is not a significant factor

All the students regularly used computers to assist in their schoolwork,including the preparation of project reports Naturally some of the stu-dents were more interested in computing and had their own personalcomputers at home, and some of the students used English more thanothers outside the classroom, for example, in writing to pen friends or inchatrooms However, as a group, we found the students participating inthis study to represent the spectrum of students typically found in Swed-ish school Year 7 The sample had a small amount of attrition, with 14students completing all three tasks over the 3 years

Materials and Design

For this part of the study, each participant wrote one essay in Englisheach year over a 3-year period Following a briefing about the loggingtool, the essays were composed in JEdit on an Apple Macintosh com-puter with an A4 screen, the international standard size of a sheet ofpaper The participants were taken one by one from their classroom to

a small quiet room to undertake the task Although questions relating toJEdit could be posed at any point during the writing session, questions

relating to English and/or the essay task were not answered The dents were allowed to use pen and paper to jot down ideas.

stu-The instructions given to the students were that they had around 60minutes to write their essays (although in Year 1 students had the chance

to write for longer) and that they were to write approximately one A4page of text They were explicitly told that they were to decide when theirtext was complete and that this could be longer or shorter than one A4page, and could take more or less than 60 minutes (See Appendix A forthe descriptive statistics of how long the writers’ wrote From Appendix

A it is apparent that the change in instruction did not result in writerslimiting their writing to 60 minutes.) The task was then read to thestudents and given to them in writing The tasks over the 3 years were alldescriptive In the first year of the study the writers’ task was to compose

an article about a typical day in a Swedish school In the second year,participants completed a task describing holidays in Sweden, and in thethird year, the students wrote an article describing their town and sur-roundings to a time-traveller Examples of the texts produced by oneparticipant are given in Appendix B

Coding and Analysis

The research questions focus on the potential relationship betweenthe year of writing and a set of variables, which characterize the writingprocess and the individual The data were coded and analyzed as follows.First, information about the writers’ writing processes in terms of onlinetext production, pausing, and revisions was extracted from the keystrokelogfiles Second, the grades for the final texts were used as a text qualitymeasure, and third, fluency measures were calculated from information

in the logfiles

Fluency was calculated on the basis of the definition in Chenoweth

and Hayes’ (2001) study of fluency in L1 and L2 writing They definefluency as the number of words written per minute, and define an ad-

ditional factor, burst, as the number of words written between pauses or revisions In computer-based automatic syntactic analysis, word is a diffi- cult unit to define Therefore, we defined fluency as the number of characters produced per minute and burst as the number of typed char- acters between pauses and/or revisions: Burst = number of typed char-

acters ÷ (number of revisions + number of pauses) We were also

inter-ested in writer fluency during burst This we calculated as Fluency during burst = (total writing time − total pause time) ÷ (number of revision +

number of pauses) The number of characters included spaces as well as

characters that were deleted from the final text version Pauses were

defined as interruptions to keyboard and mouse activity of more than 2

seconds, and revisions were defined as deletions or insertions.

Thus, in order to study the relationship between the writers’ textquality, year of writing, fluency, revision, and pausing, the following

variables were used: text quality, year of study, fluency, burst, fluency during burst, text length, time on task, revisions (total number), pauses (total num- ber), pause time (as a % of total writing time), pause length (average), and gender.

In terms of the offline assessment of the text quality, a grading systemdrawn from Jacobs, Zingraf, Wormuth, Hartfiel, and Hughey (1981) was

used that identified the following weighted criteria: content (coverage of topic, detail, and appropriateness to genre) 20%; range (grammatical structures and vocabulary) 15%; complexity (sentence structure and para- graph organisation) 15%; accuracy (grammar, sentence structure and discourse organisation, spelling, punctuation) 30%; and fluency (feel for

language, appropriateness, use of idioms) 20% For the purposes of thisarticle, we used the composite score of these criteria Two native speakerswith experience of teaching EFL undertook the grading The texts weregraded blind and the order of grading was random The interrater reli-ability was confirmed with Cronbach’s␣ (␣ = 0.95)

Statistical Analysis

Two statistical models were used to explore the data: repeated sures and linear regression Only the data from the 14 students whocompleted all three tasks were included Hence, 42 (14 students × 3tasks) sets of data points were used First, a one-way repeated measures

mea-ANOVA was used to examine the effect of year on the main variables text quality, fluency, burst, fluency during burst, text length, time on task, revisions, pauses, pause time, and pause length Gender was used as between-subjects

variables The results are summarised in Table 1

The results show that, in this study, the year of writing did not affectquality or the length of the written text, or the total number of revisions

made during the writing session Gender had no effect on any of the tested variables However, the results show that year of writing had an effect on several other variables The writers increased their writing flu- ency (F (2, 12) = 8.72; p = 0.006) as well as the length of the writing bursts (F (2, 12) = 18.19; p < 0.001) The writers’ time on task decreased (F (2, 12) = 7.03; p = 0.005) (See Appendix A for time on task descriptive statistics.) The total number of pauses decreased (F (2, 20) = 24.48; p <

0.001) as did the total amount of time the writers’ paused during their

writing session (F (2, 20) = 9.11; p = 0.002).

To further explore which variables affected text quality, two linear

regression models were designed with a reduced number of variables.

Following the results of the ANOVA analyses, gender was excluded

be-cause it did not contribute significantly to the understanding of any of

the tested variables Furthermore, because the variables fluency, burst, and fluency during burst were calculated using time, text length, and

revision measures, collinearity was likely to occur Therefore, two modelswere created One model included only the automatically generated

variables text quality, text length, time on task, revisions, pause time, and pause length The variable pauses was not included in this model because of high

collinearity values (VIF = 9.3) and significant correlation (Pearson) with

several other variables (pause time r = 0.61, pause length r = 0.43, and time

on task r = 0.76) The second model included only the variables burst and fluency during burst, together with year as independent variables The

effects of the variables in the two models are presented in Table 2 Wheninterpreting these tables, the power of the models needs to be consid-

ered The determination coefficient, r2, for Model 1 is 58.60 and forModel 2 is 55.60 We judge the explanatory power of the models to bereasonable, given the data set and number of variables Neither of theregression models distinguish between the different writers; a model thatwould account for individual writers would entail too many variables forthis limited data set

An examination of the results of regression Model 1 shows that (a) the

TABLE 1 Descriptive Statistics and Results of the Repeated Measures ANOVAs

Variable Year 1 Year 2 Year 3 ANOVA