Corpus informed assessment of comprehension of conversational implicatures in l2 english

Edited by ALI SHEHADEH United Arab Emirates University JOHN LEVIS Iowa State University Corpus-Informed Assessment of Comprehension of Conversational Implicatures in L2 English NAOKO TAG

BRIEF REPORTS AND SUMMARIES

TESOL Quarterly invites readers to submit short reports and updates on their work These summaries may address any areas of interest to Quarterly readers.

Edited by ALI SHEHADEH

United Arab Emirates University JOHN LEVIS

Iowa State University

Corpus-Informed Assessment of Comprehension of Conversational Implicatures in L2 English

NAOKO TAGUCHI

Carnegie Mellon University

Pittsburgh, Pennsylvania, United States

& A number of studies have examined second language (L2) learners’ ability to recognize linguistic and contextual cues and use them to make inferences of speakers’ intentions conveyed indirectly (e.g., Bouton, 1992; Ro¨ ver, 2005; Taguchi, 2007) In these studies, learners’ accurate identification of implied intentions has been used to depict pragmatic development and factors affecting the facility and difficulty of comprehension Research has found that implicature type and conventionality affect L2 comprehension and its development Implicatures vary in their degree of strength; some implicatures are strongly conveyed, while others are weakly understood, owing to the number of linguistic and contextual cues to be processed (Sperber & Wilson, 1995) The greater the number of cues to be processed, the more extensive people’s search for meaning becomes, resulting in greater processing load In contrast, when implicatures contain conventional linguistic features or are embedded in predictable discourse patterns, extensive analysis of contextual cues becomes unnecessary, and comprehension becomes immediate and automatic (Sperber & Wilson)

In accordance with these theoretical claims, L2 learners were found to

be sensitive to the differential degrees of implicitness and layers of meaning Empirical evidence exists where learners’ comprehension showed distinct patterns across implicatures that involved different degrees of conventionality (e.g., Bouton, 1992; Ro¨ ver, 2005; Taguchi,

2007) In Bouton’s study, 30 ESL learners took a written multiple-choice test with 33 short written dialogues that had various types of implicatures The learners’ comprehension of relevance-based implica-tures became similar to native speakers’ after 4.5 years However, they still struggled with formulaic implicatures (e.g., sequence implicatures, Pope implicatures such as saying ‘‘Can fish swim?’’ to mean ‘‘yes.’’) Taguchi (2007), on the other hand, examined the comprehension of indirect refusals and opinions among Japanese EFL learners Indirect refusals were considered conventional because they followed a common, predictable discourse pattern (giving a reason for refusal) Indirect opinions were considered less conventional because they did not attach meaning to specific linguistic expressions or predictable patterns (e.g., indicating a negative opinion of a movie by saying ‘‘I was glad when it was over’’) These item types were incorporated into a listening test that contained a series of short dialogues followed by yes–no questions Learners’ comprehension was faster and more accurate for indirect refusals than for indirect opinions Learners made significant gains in both accuracy and comprehension speed (measured by response times) over 7 weeks, with the degree of gain for accuracy outweighing response time These findings suggest that the comprehension of implicatures have developmental patterns When meaning is based on shared conventions,

it is easier to comprehend because it allows the transfer of pragmatic knowledge from the first language (L1) This was the case in Bouton’s studies, where learners’ comprehension of relevance implicatures became native-like over time Similarly, Taguchi’s study found that the comprehension of refusals preceded that of opinions because the means

to encode refusals were conventional and shared between L1 and L2 However, when the convention is culture specific, or not shared between L1 and L2, meaning becomes the most difficult to recognize This phenomenon was shown in Bouton’s studies wherein learners continued

to have difficulty with Pope implicatures and in Taguchi’s study in the comprehension of nonconventional implicatures

These studies used artificially created implicatures to examine the effect of conventionality across implicature types Garcia’s (2004) study

is probably the only existing study that used purely naturalistic dialogues

to examine pragmatic comprehension Using conversations from a corpus, she developed a multiple-choice listening test that measured nonnative English speakers’ ability to identify four types of speech acts: requests, suggestions, corrections, and offers Results found that there were significant differences between lower and higher proficiency participants in their identification of speech acts Follow-up linguistic analyses of individual items revealed four factors that hindered comprehension: unclear agent, disfluency features, unfamiliar use of modal verbs, and mismatch between context and speech acts

Garcia’s work showed that naturally occurring language data can inform the design of pragmatic comprehension measures Various implicatures used in the previous studies can be extracted from naturalistic dialogues and adapted to create an instrument that more closely reflects real life language use More effort is necessary in adapting corpora to develop assessment tools to systematically examine the hierarchy of difficulty and conventionality affecting pragmatic compre-hension gleaned in the previous research Because previous studies showed that learners find it easier and faster to comprehend certain implicatures than others, and accuracy gains precede speed gains (e.g., Taguchi, 2007), this study examined whether these patterns are present when using a test that incorporates naturally occurring implicatures The study adapted implicatures from corpora to develop a listening test to measure pragmatic comprehension Using the instrument, this study examined whether or not accurate and speedy pragmatic comprehen-sion develops over time across implicature types of different degrees of conventionality

METHODOLOGY

Participants

Participants in this study were 48 Japanese EFL (English as a foreign language) learners in a college in Japan There were 16 males and 32 females (average age was 18.33 years) They averaged 6.14 years of formal English education in Japanese Their average TOEFL score was about 460 (SD 5 17.77) None had more than one month of experience living outside Japan In addition to the ESL learners, 25 native English speakers participated in the pilot study and provided baseline data

Instrumentation

The Corpora

Two corpora of face-to-face conversations of American English were selected for this study, representing two registers: family/friends interactions and service encounter interactions Family/friends interac-tions were taken from the Santa Barbara Corpus of Spoken American English (SBC) (Du Bois, Chafe, Meyer, & Thompson, 2000), and the service encounter interactions were taken from the TOEFL 2000 Spoken and Written Academic Language Corpus (T2K-SWALC) (Biber, Conrad, Reppen, Byrd, & Helt, 2002)

Development of Test Items

This study examined EFL learners’ listening ability to comprehend more and less conventional implicatures More conventional implicatures convey conventional meaning and are invariable in a specific situation For instance, the expression ‘‘How can I help you?’’ represents a speech act of offer that takes place within the frame of shopping When one is familiar with conventions of the frame, the expression is understood as formulaic, and comprehension becomes automatic Another example is

a refusal routine of giving an excuse for a refusal (e.g., saying ‘‘I’m busy’’ when refusing someone’s invitation to a party) These more conven-tional implicatures convey indirect meanings by fixed linguistic forms or through predictable discourse patterns

Less conventional implicatures represent indirect meanings that are relatively less conventional and vary according to context, as in the following example:

A: How was your presentation?

B: It’s over, so it’s OK

In this example, an open set of expressions is possible as B’s reply When expressing negative opinions of the presentation, options regarding how

to express them are wide open (endless expressions of liking and disliking) Thus, compared with more conventional implicatures, less conventional implicatures are more idiosyncratic; they require more extensive inferential processing to derive meaning because the listener needs to process a greater number of contextual cues

More and less conventional implicatures were sought for in the SBC and T2K-SWALC corpora Transcriptions of the corpora were read line-by-line, and the target implicatures were hand-coded based on the following criteria More conventional implicatures:

(a) Indirect refusals: refusal responses to invitations, requests, and suggestions with a reason (e.g., saying ‘‘I’m busy’’ when refusing someone’s request for help)

(b) Routines: fixed or semifixed expressions that commonly occur under certain situational conditions and functions (e.g., ‘‘It comes to $2’’

in a service encounter exchange and ‘‘That’s so sweet of you’’ when thanking someone)

Less conventional implicatures: nonliteral comments or opinions that

do not involve conventional linguistic features or language use patterns (e.g., indicating a negative opinion of a dinner by saying ‘‘The dinner was late’’)

The implicatures were taken from the corpora if the conversational context was judged accessible to the beginner EFL learners whom this study targeted This initial process yielded 26 more conventional and 24 less conventional implicatures Then dialogues were adapted to better serve the level of the target learner group and goals of the research Because authentic conversation involved a long stretch of discourse, the conversation was shortened to reduce the effect of short-term memory Proper nouns or vocabulary items that were considered unfamiliar to the target EFL learners were replaced with familiar equivalents (e.g., ‘‘mud pie’’ changed to ‘‘apple pie’’) The dialogues were written as a conversation between a male and female speaker so that the test takers can easily distinguish the voice

Pilot Test One

For the first test, 24 dialogues with more conventional implicatures and 24 dialogues with less conventional implicatures were prepared Although the intended meaning of more conventional implicatures was relatively straightforward, concerning the less conventional implicatures,

it was questionable whether the same intended interpretations arise across different people owing to their variable nature Hence, a pilot study was conducted to confirm the reliability of the interpretations An open-ended survey containing the 24 dialogues was prepared In each dialogue, the target indirect utterance was underlined Participants were instructed to supply the interpretation for the underlined nonliteral utterance as in the following example:

A: Hey Nancy, happy birthday I want you to open this gift It’s from me B: Oh, thank you, Mike This is a pretty package When did you have a chance

to wrap this?

A: At school

B: Look at this A Mickey Mouse watch That’s just what I always wanted The survey was administered to 10 native English speakers at a U.S university, and follow-up individual interviews were conducted For the majority of the items, they showed relatively uniform interpretation However, some items showed a mixed interpretation In the preceding example, five respondents interpreted the target utterance as positive (i.e., Speaker B likes the gift), while four people wrote the opposite (i.e.,

‘‘Speaker B is not thrilled with the gift’’) One person indicated both Follow-up interviews revealed that this utterance could be taken as sarcasm and implies negative opinion about the gift Based on these results, three items that showed inconsistent interpretations were eliminated, resulting in a pool of 21 less conventional implicatures, from which target items were selected

Pilot Test Two

The first version of the listening test (hereafter pragmatic listening test

or PLT) had a total of 42 items: 2 practice items, 32 experimental items, and 8 filler items that tested literal comprehension The experimental items had 16 more and less conventional implicatures Each item had a short dialogue followed by multiple-choice questions with four answer options (see Table 1) The participants listened to the dialogue and selected the statement that is correct based on the content of the conversation In the experimental dialogues, the correct answer was the speaker’s implied intention Correct answers are marked with an asterisk The PLT was computerized using the software Revolution (Runtime Revolution, 2003) and piloted with 25 native speakers of English at a U.S university Several items were revised, and the accuracy rate of the

TABLE 1.

Sample Pragmatic Listening Test Items

More conventional implicatures

Refusal

just opened.

*1 Nancy doesn’t want to go out tonight.

2 Nancy is going out tonight.

3 Nancy got a bill from the restaurant.

4 Nancy is watching TV with Mike.

Routine

Salesclerk: Hi, how can I help you?

Customer: Ah, could I get a small regular coffee, with milk? And a slice of apple pie Salesclerk: For here or to go?

Salesclerk: Here’s a large cup, we don’t have small because we ran out of the small ones.

*2 The man is taking the coffee out.

3 The man is having coffee in the shop.

Less conventional implicatures

but I have to get a better score to get into a medical school.

*1 Mike thinks Nancy’s exam score is good.

2 Mike thinks Nancy’s score is poor.

3 Nancy is a medical school student.

4 Nancy’s medical school exam is next month.

revised items was confirmed with the same native speakers After administering the test to the EFL learners, internal consistency reliability was assessed Cronbach’s alpha was 0.92 for the full test, 0.83 for less conventional implicatures, and 0.84 for more conventional implicatures (0.76 for refusals and 0.72 for routines)

Procedure and Administration

The pragmatic listening test was administered individually to the EFL participants using Windows computers on campus at the beginning of the semester The participants were not informed about the purpose of the research The participants practiced two items and then proceeded

to the 40 test items Immediately following each dialogue, a multiple-choice question with four answer options in English appeared on the screen Participants were told to read each answer option and choose the correct one by pressing the corresponding key from 1 to 4 Response time was measured from the moment the question appeared on the screen and to when the participant pressed the key The procedure was repeated at the end of the semester

RESULTS

Tables 2 and 3 display the descriptive statistics of accuracy scores and response times Not surprisingly, native speakers’ comprehension was almost perfectly accurate and much faster than EFL learners, and their

TABLE 2.

Descriptive Statistics, Accuracy Scores

EFL learners

Time 1

Time 2

Native Speakers

Note SD 5 standard deviation Min 5 minimum Max 5 maximum.

response times were relatively uniform across implicature types For the EFL data (second test), the repeated-measures ANOVA revealed a significant effect of item type on comprehension accuracy, F 5 33.62 (p , 0.05) Significant differences were found between less conventional implicatures and refusals, as well as between refusals and routines, but not between routines and less conventional implicatures

Concerning response times, the repeated-measures ANOVA revealed significant differences across item types, F 5 18.17 (p , 0.05), with significant differences in all contrasts Refusals took the shortest time to comprehend, followed by routines Less conventional implicatures took the longest time to comprehend

Paired-sample t-tests were applied to compare accuracy scores and response times between the first and second tests for the three implicature types There was a significant gain in accuracy scores for the refusal items, t 5 3.50 (p , 0.05), but not for the other two item types: t 5 0.48 (n.s.) for less conventional implicatures and t 5 0.61 (n.s.) for routines Response times remained the same over time; however, response times for refusal items tended to became shorter over time (t 5 22.39, p , 0.05)

DISCUSSION

This study revealed a noticeable interaction between implicature type, comprehension difficulty, and development of pragmatic comprehension

TABLE 3.

Descriptive Statistics, Response Times

EFL learners

Time 1

Time 2

Native Speakers

Note The time refers to the average number of seconds taken to answer each item correctly.

SD 5 standard deviation Min 5 minimum Max 5 maximum.

Comprehension of indirect refusals was the easiest and fastest, and showed

a profound development over a short period of 3 months Because the discourse pattern (i.e., giving a reason for refusal) was shared between L1 Japanese and L2 English, learners were able to apply their L1-based convention to L2 comprehension and made further progress as their general listening ability improved over time

In contrast, comprehension of routines showed very little improve-ment both in accuracy and response times, suggesting different conventionality for refusals and routines Compared with refusals, routines were found to have stronger association with specific situations

or communicative functions For instance, the phrase ‘‘For here or to go?’’ occurs in a service encounter situation of buying food, and the meaning is fixed across similar situations The fixedness and invariant nature of routines typically assists our comprehension because they are processed as a chunk rather than as a series of isolated words and, therefore, meaning is immediately retrievable from long-term memory,

as long as people are aware of the forms and their contextual requirements The low accuracy and little development observed in this study, however, suggest that the EFL learners who participated were unfamiliar with the target routine expressions Owing to the context-dependent, culture-specific nature of the routines, they were probably not able to pick them up naturally in a foreign language environment that is limited to the type of situations in which the routines occur Similar to routines, comprehension of less conventional implicatures were also found to be slow in developing As opposed to conventional implicatures, they were context independent and did not reflect fixed linguistic forms or customized discourse patterns of refusals They required the learner to infer the context of the conversation, beyond understanding the target implicature forms The findings suggest that the ability to comprehend less conventional implicatures does not develop as quickly as conventional ones owing to the extensive inferential bridge speakers use to arrive at correct interpretations Without constant symbolic representations, the comprehension of these implicatures relies on more bottom-up analyses of syntactic and lexical information and contextual cues Using these analyses, learners must understand the literal meaning of the expressions and then work deductively toward the speakers’ implied intentions Multiple levels of processing extend the degree of inferencing that learners must do, thereby increasing difficulty and slowing down processing speed These observations are supported by the comparison of comprehen-sion speed between less conventional implicatures and routines Despite the same accuracy rate, comprehension speed was faster for the routines than for the less conventional implicatures Hence, although the linguistic conventions encoded in routines were unfamiliar and difficult

to access for the EFL learners, once they understood the conventions, they were able to derive meaning quickly by taking advantage of the conventions Less conventional implicatures, on the other hand, did not reveal such facility effect in comprehension speed

IMPLICATIONS FOR PRAGMATIC TESTING AND

TEACHING

Using artificial dialogues, previous studies identified different patterns

of development between accuracy and comprehension speed Learners find it easier and faster to comprehend certain implicatures than others, and accuracy gains precede speed gains (e.g., Taguchi, 2007) This study examined whether these patterns are present when using a test that more closely reflects naturally occurring implicatures The corpus-informed test items used here largely confirmed the hierarchy of demands involved in pragmatic comprehension and its effect on development Hence, this study confirmed that the degree of indirectness and conventionality, as reflected in variation in accuracy and response times, is not merely a property of pragmatic theories It is ubiquitous in everyday conversation, and learners’ comprehension can be assessed as ability to comprehend a variety of implicatures, both conventional and nonconventional, which are present in real-life conversation

The process of test development in this study suggests a number of issues for future research The most important one relates to the degree

of modifications that must be made to the dialogues taken from the corpus to create plausible test items for the target beginner-level learners Owing to those modifications, the dialogues in the listening test were not truly authentic in that they were not presented as they occurred in actuality Although this study purposefully reduced the messiness of naturalistic discourse to control for extraneous variables, future research that targets advanced-level learners should explore ways

to incorporate features of spoken discourse into assessment of pragmatic comprehension

This study offers some implications for the teaching of pragmatics It revealed that comprehension of indirect refusals develops naturally in a short period of time without explicit instruction Hence, instruction could be more profitably targeted at item types that develop more slowly, namely, routines and less conventional implicatures Common situa-tional routines in naturalistic conversations, films, and TV dramas can be brought to the class and taught by drawing students’ attention to situational variables such as setting, purpose of interaction, and speakers

in the exchange In this way, students will be able to draw a clearer connection between routines and their situational requirements, and