How do teachers observe and evaluate elementary school students’ foreign language performance a case study from south korea

How Do Teachers Observe and Evaluate Elementary School Students’ Foreign Language Performance?. A Case Study from South Korea YUKO GOTO BUTLER University of Pennsylvania Philadelph

How Do Teachers Observe and Evaluate Elementary School Students’ Foreign

Language Performance? A Case Study from South Korea

YUKO GOTO BUTLER

University of Pennsylvania

Philadelphia, Pennsylvania, United States

This study investigates how teachers observe and assess elementary school students’ foreign language performance in class and how such assessments vary among teachers Twenty-six elementary school teach-ers and 23 English teachers at secondary schools in South Korea watched videotapes of 6th-grade students’ group activities in English and were asked to assess the students’ performance as if they were in their own classrooms The study found that the teachers varied substantially in their overall evaluations both within and across school levels A discus-sion held among the teachers after the individual assessments were com-pleted showed that the elementary school teachers and secondary school teachers differed with respect to (1) their views toward assess-ment criteria, (2) how to evaluate student confi dence and motivation; and (3) how to gauge students’ potential ability to communicate compe-tently in a foreign language Such differences between the elementary and secondary school teachers appeared to be deeply rooted in their respective teaching contexts Using Davison’s (2004) framework for analyzing teachers’ beliefs and practices in teacher-based assessment, the current study suggests that both groups of teachers need to negoti-ate assessment criteria while paying close attention to the local context and adapting their teaching practices to fi t therein

This study is concerned with how teachers observe and assess young learners’ foreign language performance in class and how such assessments vary among teachers By asking elementary school teachers and secondary school teachers to evaluate 6th-grade students’ English abilities at the end of their elementary school education in South Korea, the current study aims to examine similarities and differences in teach-ers’ observations both among teachers working at the same school level and across different school levels In doing so, it is hoped that this study will help us better understand teacher observation, as one popular type

of teacher-based assessment, and help us enable a smoother transition

in assessment practice from elementary school English to secondary

school English

Teacher-based assessment can be defi ned as “nonstandardized local

assessment carried out by teachers in the classroom” (Leung, 2005, p 871)

A number of countries have begun heavily promoting teacher-based

assess-ments as part of their language-in-education policies in recent times The

degree to which teacher-based assessments have gained in prominence is

particularly evident at the elementary school level We fi nd a strong

empha-sis on the development of communicative competence, especially in the

oral domain, because it has become one of the central goals of English as

a foreign language education at the elementary school level (FLES) The

promotion of teacher-based assessment is also tied to the intentions of

pol-icy makers, who have strived to avoid traditional achievement tests, such as

paper-and-pencil standardized tests, at the elementary school level

Despite the strong promotion of teacher-based assessment in various

educational contexts, including FLES, concerns have been cited with

regard to the validity, reliability, high costs, fairness, and logistical

chal-lenges in developing, administering, and scoring teacher-based

assess-ment (e.g., Gattullo, 2000; Linn, Baker, & Dunbar, 1991; Rea-Dickins &

Gardner, 2000)

Part of the challenge of teacher-based assessment appears to come

from the dilemma between the pedagogic and measurement aspects of

such assessments Namely, at the same time that these assessments are

supposed to help teach students, education policies often ask

teacher-based assessment to fulfi ll an accountability requirement Teachers

world-wide encounter tension in meeting the pedagogical needs of students

while at the same time meeting the accountability requirements that are

often based on prescribed standards and criteria (e.g., Arkoudis &

O’Loughlin, 2004; Brindley, 1998, for a discussion of this tension in

Australia; Davison, 2004, for Australia and Hong Kong; Gardner &

Rea-Dickins, 1999; Teasdale & Leung, 2000, for England)

South Korea is no exception to this trend At the elementary school

level, among the various types of teacher-based assessments, teacher

observation has been promoted as a primary means of assessment

Teachers are encouraged to observe students’ performance

systemati-cally during classroom activities and to use such observations for both

summative and formative purposes. 1 However, in many cases, no specifi c

criteria for conducting observations have been provided, and we know

1 The summative assessment is usually given to students at the end of an instructional sequence

and the results are primarily used for giving students reports about their achievement The

formative assessment is usually undertaken before and/or during an instructional sequence

and is primarily used to help the students identify their strengths and weaknesses, and in

turn, provides teachers with information in order to make instructional decisions

little about how teachers observe and assess their students’ performance during classroom activities At the secondary school level, teacher-based assessment, including teacher observation, has also increasingly been emphasized However, practical and pedagogical challenges including large class sizes and limited class hours leave teachers little time for mak-ing systematic observations for formative purposes Under such condi-tions, parents and students have frequently cited their distrust of teacher observation as a summative assessment (Butler, 2005)

A number of researchers have questioned the application of tional measurement-based concepts of validity and reliability to teacher-based assessment (Brookhart, 2003; McMillan, 2003; Moss, 2003; Smith, 2003) Traditional validity and reliability theories are fundamentally con-cerned with the ability (or lack thereof) to generalize assessment-based inferences, and the consistency of measures irrespective of the context, form, time span, and raters involved in assessment Such concepts are not necessarily relevant or even compatible with teacher-based assessment, which is highly context dependent and primarily formative in nature (McNamara, 2001; Teasdale & Leung, 2000) Teacher-based assessment should not be considered as a collection of miniature summative assess-ments (Rea-Dickins, 2007) Indeed, criteria that are derived from the psy-chometric tradition may not be appropriate for teacher-based assessment (Leung, 2005)

As an alternative approach, Wiliam (2001) proposed construct-referenced

assessment , which is based on “the consensus of the teachers making the

assessment” (p 172) In this approach, there is no predefi ned objective criterion Instead, teachers’ judgments are based on shared understand-ings of what a community of teachers in a given teaching context would consider competency Leung (2005) argues that the concept of construct-referenced assessment is “useful in that it opens the way to an examina-tion of the kind of information teachers seek and the basis of their decision making” (p 880) This approach sheds light on the importance

of understanding teachers’ knowledge about assessment and paying attention to the specifi c context in which the assessment is undertaken

To date, researchers have only a limited understanding of the ing and criteria that teachers use in their teacher-based assessments for young learners in English as a foreign language (EFL) contexts In exam-ining English as a second language (ESL) environments, a number of studies have investigated how teachers understand and work with assess-ment criteria when they perform teacher-based assessments (e.g., Breen,

reason-et al., 1997; Davison, 2004; Leung, 1999; Teasdale & Leung, 2000) In England, for example, Leung (1999) found that teachers do not seem to make judgments simply based on students’ linguistic performance on a given task, but rather that they make holistic judgments while bringing in various external factors such as performance in previous activities

or performance outside of the classroom Much of the previous work,

however, is primarily focused on investigating how teachers interpret

pre-scribed assessment frameworks and how they apply these in their

assess-ment practices

As Rea-Dickins (2007) indicates, there are usually multiple motivations

for implementing FLES Many FLES programs have not only linguistic

targets but also nonlinguistic objectives such as developing positive

atti-tudes toward the target language as well as an appreciation of both

for-eign and domestic cultures One may hypothesize that having so many

varied objectives for FLES programs could make it diffi cult for teachers

to reach a consensus among themselves on what specifi c objectives they

should try to achieve through assessment Moreover, researchers have

observed a lack of consistency in foreign language teaching practices,

including assessment, between the elementary and secondary school

lev-els (e.g., Bolster, Balandier-Brown, & Rea-Dickins, 2004; Butler, 2005)

We also have very little understanding of how secondary school language

teachers understand and evaluate the performance of those incoming

students who have come up through FLES programs

This study therefore investigates how elementary and secondary school

teachers observe and assess 6th-grade students’ foreign language

perfor-mance in class (at the end of their elementary school education) and

how those assessments vary among such teachers The study focuses on

what abilities teachers pay attention to when assessing elementary school

students’ performance and what kinds of criteria and reference points

they use The study attempts to address these topics by focusing on

English FLES in South Korea as a case study As we shall see, South Korean

society has traditionally placed substantial value on measurement in its

educational system, but the government recently began promoting

teacher-based assessment as part of its language-in-education policy

Unlike many of the cases that have been documented thus far, no

pre-scribed assessment framework is available for teachers in South Korea;

rather, teachers are responsible for developing their own assessments as

part of their teaching practice

ENGLISH AS A FOREIGN LANGUAGE EDUCATION

IN SOUTH KOREA

In South Korea, various types of assessment have played a signifi cant

role in education and society as a whole English, as one of the key

demic subjects, has been used as a barometer of students’ general

aca-demic achievement and diligence Grammar translation and vocabulary

exercises long dominated English classrooms at the secondary school

level and beyond; rigorous standardized assessments measuring students’

discrete linguistic knowledge had a substantial impact on students’ future academic and career opportunities

In the 1990s, as part of the South Korean government’s globalization policy, the Ministry of Education shifted their English curriculum from traditional grammar-translation instruction to communicative-based instruction Acquiring communicative competency, and oral communi-cative abilities in particular, became a central goal of English education

In line with the promotion of communicative language teaching (CLT), teachers are now coached to use various types of activities in class and encouraged to use only English in their classrooms A student-centered approach has been strongly promoted in the current policy

Along with this shift in teaching approach, the government introduced

a series of reforms in assessment One such reform was the promotion of teacher-based assessment as part of the assessment requirements The 7th National Curriculum (implemented in 1999) indicated that teachers should assess students’ process of learning as well as the outcome of their learning through ongoing observation and other forms of performance assessment, as opposed to one-shot multiple choice tests It is important that the policy stressed the autonomy of schools and teachers in adminis-tering such assessments Schools were now responsible for deciding the methods, criteria, and frequency of assessments through discussions among their teachers (see, e.g., Chungcheongnamdo Offi ce of Education, 2007)

As part of the effort to enhance the communicative competence of its citizens, the South Korean government introduced English as a compul-sory subject at the elementary school level nationwide in 1997 In addi-tion to the strong emphasis on oral communication as a central goal of FLES, motivating students to learn English was set as another important goal A variety of group activities have been implemented in classrooms based on the uniform national curriculum With respect to assessment, the government (via the Korea Institute of Curriculum and Evaluation,

or KICE) has suggested that teachers should periodically observe their students’ performance in class and keep “observation records” on atti-tudes, oral skill development, and written skill development KICE also created a 5-point scale to serve as an example for these teacher observa-tions (Lee, 2007) However, KICE did not provide precise criteria for each point and domain Individual teachers must decide how to use such assessment information for summative and/or formative purposes Currently, report cards to students and parents at the elementary school level in South Korea are based on verbal descriptions and not on a numeric scale However, parents may request teachers to disclose any of the information on their children’s performance in class that was used as

a basis for their evaluation In practice, many teachers keep records of one type of numeric scale or another, including standardized test scores,

in addition to verbal comments on their students’ performance

Despite the promotion of teacher-based assessment at school, South

Korean society continues to emphasize a measurement-driven

orienta-tion toward assessment Even among elementary school students, a

num-ber of large-scale standardized English tests such as the Test of English

for International Communication (TOEIC) Bridge and the Practical

English Level Test for Elementary English (PELT) are very popular

(Choi, 2008) Many students go to private English institutes where they

can access various types of profi ciency tests including these standardized

tests At the secondary school level, the pressure that students feel about

exams appears to have become even more intense Under recent reforms

in English assessment, a certain portion of the students’ fi nal grades

should now come from teacher-based assessment Final grades, however,

are given to students on a numeric basis Students and parents are greatly

concerned with their English grades at school as well as their scores on

various types of standardized tests of English, especially in relation to

accessing higher education and fulfi lling their career aspirations Under

substantial pressure from parents and students, teachers in South Korea

are expected to promote a learning culture in accordance with

govern-ment policy, but in an excessively focused exam culture (Hamp-Lyons,

2007) Teacher-based assessment thus inevitably becomes part of

account-ability measures, at least to some degree

Finally, it has been reported that elementary school teachers have had

little direct communication with secondary school English teachers

regarding instruction and assessment in South Korea (Butler & Lee,

2005) Teacher training is usually offered to elementary school teachers

and secondary school teachers separately, and teachers typically have few

opportunities to observe English classes at different school levels As

such, assessment practices may differ signifi cantly between elementary

and secondary school teachers

COMMUNICATIVE ABILITIES IN A FOREIGN LANGUAGE

One of the challenges in developing teacher-based assessments in

FLES is the limited understanding of what specifi cally entails having

com-municative abilities in a foreign language Language assessment

con-structs are not yet clearly understood, especially when pedagogical value

is primarily placed on assessment (McNamara, 2001; Rea-Dickins &

Gardner, 2000) In identifying constructs for teacher-based assessment

for young learners, the following three characteristics of existing models

of communicative abilities are especially problematic: (a) the notion that

communicative abilities reside in individuals, (b) the lack of a clear

con-ceptualization of the affective aspects of communicative abilities, and (c)

the lack of a developmental perspective

With regard to the fi rst characteristic, current theories on tive competence in a second or foreign language overemphasize individ-ual performance as opposed to interactive performance (McNamara,

communica-1996, 1997, 2001) McNamara argues that it is dangerous to consider one’s performance on a performance test as being a mere refl ection of one’s individual competence; rather, performance is co-constructed through interactions among various agents such as interlocutors, test materials, raters, and so forth In FLES programs, a frequently empha-sized goal is developing students’ communicative abilities rather than their discrete linguistic knowledge per se, and pair and group activities are widely used One can expect that a student’s performance is infl u-enced by the nature of the activities and interactions with the student’s interlocutors In addition, which activities the teacher chooses to observe and which aspects of the student’s performance the teacher chooses to pay attention to all contribute to the dynamic interactions that infl uence ratings and evaluations However, it is not clear how best to understand students’ communicative competence in such interactions

Second, although affective aspects such as motivation and confi dence are often set as a key objective for FLES programs worldwide, including

in South Korea, current theories on communicative competence do not agree on how best to conceptualize such affective factors in language

assessments (McNamara, 1996) Hymes (1972) distinguished ability for

use from knowledge in his model of communicative competence (which

was originally developed in the context of fi rst language use) Hymes

con-ceptualizes ability for use as one’s potential ability for performance, and it

includes various language-relevant cognitive and noncognitive factors such as motivation However, a model proposed by Canale and Swain (1980) that has become one of the most infl uential models of communi-cative competence in second or foreign language acquisition carefully excludes factors that are relevant to ability for use There have been some attempts to capture the affective dimensions of ability for use in succes-

sive models, such as Bachman’s (1990) strategic competence and Bachman and Palmer’s (1996) affective schemata However, as the term schemata indi-

cates, affective dimensions in their models are conceptualized as tive entities in nature and are primarily considered to be a source of response bias in assessment; the role of affective factors in language use

cogni-is far from clear (McNamara, 1996) In addition, the role of nonverbal behavior in language assessment, such as body movements and facial expressions, has yet to be suffi ciently explored (Young, 2002)

One can also point to a lack of developmental perspectives in the rent leading models of communicative competence Such models iden-tify and classify different constructs of communicative competence but they do not explain how different components interact with each other

cur-or how such interactions may change over the course of individual

development Are abilities in different constructs expected to develop

simultaneously, or are certain constructs more important than others at

some point in development? We do not yet have a comprehensive theory

of the development of communicative competence that teachers can use

when conducting formative assessments for their students

If we are to introduce teacher assessment, therefore, an important

start-ing point would be to understand teachers’ perceptions of what’s

impor-tant in carrying on competent communication in a foreign language

RESEARCH QUESTIONS

The purpose of the study was to examine how teachers observe and

assess elementary school students’ foreign language performance in daily

classroom activities More specifi cally, the study aimed to investigate the

following three questions:

1 How do teachers observe and assess elementary school students’

per-formance while they engage in group activities? How much

consis-tency (or variability) is there in teachers’ assessments of student

performance while their students are interacting with other students?

2 What kinds of selective attention do teachers demonstrate in

assess-ing their students’ performance (includassess-ing both verbal and

nonver-bal aspects)? What criteria or methods do they rely on to observe and

assess their students’ performance? How do they negotiate criteria

among themselves through discussion?

3 Do elementary and secondary school teachers assess 6th-grade

stu-dents differently?

METHOD

Participants

The participating teachers were recruited from an in-service

teach-ers’ training site in central South Korea The local provincial

govern-ment selects groups of English teachers each year from elegovern-mentary and

secondary schools from across the province to receive approximately two

months of professional training at various local sites The elementary and

secondary school teachers receive their training separately The current

study was conducted as part of the in-service training program However,

participation in the current study was on a voluntary basis With the help

of a training organizer, 26 elementary school teachers and 23 second ary

school teachers were recruited at one of the training sites during the

sum-mer of 2007 All the participants came from different schools and their

backgrounds were diverse Table 1 summarizes the teachers’ profi les based

on a background survey that was distributed to the teachers prior to the study Notably, the secondary school English teachers who participated in the current study appeared to be less familiar with the elementary school English curriculum whereas the elementary school teachers who partici-pated were more familiar with the secondary school English curriculum

TABLE 1 Participating Teachers’ Profi les

Elementary school teachers

Secondary school teachers Subjects taught*

Multiple subjects 15 (57.7%) 0 (0%) Educational background

Hours of English taught per week

level they teach (i.e., familiarity with elementary school

curriculum for secondary school teachers and vice-versa)

Experience of observing English classes at a school level

other than the level at which they teach

2 (7.7%) 0 (0%)

Note * As of 2008 in South Korea, English is taught by homeroom teachers who teach multiple

subjects, as well as by teachers who specialize in teaching English only at elementary schools Teachers may change their status on their principals’ requests each year As a result, English teachers may become homeroom teachers and vice-versa At the secondary school level, English teachers are specialized and teach English only ** SD = Standard deviations

Very few of the teachers in either group had observed English classes at the

other school level Although language assessment is consi dered an

impor-tant component of professional development for English teachers in South

Korea, the participating teachers had received relatively little training on

how to conduct teacher-based assessment, and they had not had any

exten-sive discussions on this topic with other teachers prior to this study

Materials and Procedures

The participating teachers watched videotapes of students’ group

activ-ities in English and were asked to assess the students’ communicative

per-formance This activity was conducted separately for the elementary school

teachers and the secondary school teachers The video showed four

6th-grade students engaging in two different activities in a group: One was

a simple jigsaw activity and the other was a more complicated

decision-making activity (Pica, Kanagy, & Falodun, 1993) In the fi rst activity, the

students were asked to complete the weekly schedule for a student named

Minho on a collaborative basis Each of the students had different

infor-mation about the schedule and thus two-way interactions were required to

complete the task However, only a limited set of expressions and

vocabu-lary (e.g., “What does Minho do on Monday afternoon?” and “He plays

baseball”) were needed for the completion of the task The second activity

was an open-ended shopping task wherein one student played a customer

and the rest of the students played shop owners Each shop carried

differ-ent items at differdiffer-ent prices The goals of this task included: (a) to buy a

list of items for a party and leave with as much money as possible (for the

shopper), and (b) to sell many items and make as much money as

pos-sible (for the shop owners) This task required the students to use a

vari-ety of English expressions and vocabulary related to shopping in order to

buy or sell goods and to negotiate prices (all of the necessary expressions

and vocabulary had already been covered in class based on the National

Curriculum) Unlike the fi rst activity, the number of utterances

pro-duced and the content of the discussions among the students varied

sub-stantially in the second activity This latter activity required some simple

math skills to help solve the problems encountered in the task itself Both

activities were commonly used in 6th-grade English classrooms in South

Korea, and each activity lasted for 15 minutes The participating teachers

could see the written notes that the students took during the activities

Among the four students shown in the video, two were boys and two

were girls They used English pseudonyms during the activities: Tom,

John, Jane, and Sally In the video, the four students appeared to differ in

terms of their activeness/shyness and their general English profi ciency

levels The only objective measure of their profi ciency available to the

researcher was their listening scores (raw scores) from the Cambridge Young Learners’ English Test (the Movers’ level) which was administered immediately before the taping of the video 2 The scores were 14 for Tom,

20 for John, 10 for Jane, and 16 for Sally (out of a possible total score

of 25) The activities were videotaped at the end of the school year (shortly before the students graduated from elementary school and moved on to secondary school) These video clips also had been used in past in-service training sessions in Seoul and Pusan However, none of the participating teachers had seen the video prior to the current study

Both groups of teachers individually assessed the students’ mances in two steps The fi rst step was a holistic assessment The teachers were asked to use a 5-point scale as well as to take verbal notes on each of the four students, in accordance with the suggestions of the South Korean government The 5-point grading scale has been used in the South Korean education system for years, and the current study, as suggested by

perfor-KICE, used the following scale: excellent (5), good (4), moderate (3), defi cient (2), and poor (1) The teachers were asked to take notes freely as they usu-

ally do in their own classrooms

The second step tested multiple traits (Hamp-Lyons, 1991) The dents’ performances were evaluated based on three or four specifi c traits

stu-In this study, the traits were self-selected by individual teachers out of 12 prescribed traits These traits were determined based on multiple sources including the national curriculum, teachers’ manuals, and other com-mercial resources that are popular among Korean teachers The traits

were (1) listening comprehension, (2) speaking fl uency, (3) speaking accuracy, (4) pronunciation, (5) range of vocabulary use, (6) content being spoken about

(content), (7) use of appropriate expressions in the given context (pragmatics),

(8) confi dence in talking, (9) motivation, (10) task completion, (11) the ability to

interact effectively with other students (interpersonal), and (12) others These traits

were certainly not exclusive or fi xed traits but they often appeared in ments available to the teachers In this study, these traits were used with the intention of helping the teachers become aware of what they do when they produce a single score (e.g., whether they base the score on an overall judgment or rely on a single or multiple traits) and which abilities they pay attention to when they make holistic judgments The teachers were encour-aged to identify those traits (any number of such traits) that they thought they paid attention to while they observed the students’ performances The teachers were shown the video clips twice for each of the activities

docu-2 Cambridge Young Learners’ English Tests have three levels: Starters, Movers, and Flyers The Movers level and Flyers level correspond to the A1 and A2 levels in the Common European Framework Level, respectively The Flyers level is equivalent to the Key English Test with respect to diffi culty, but its vocabulary and content are designed to be more suit- able for young learners (Cambridge ESOL, n.d.)

After individually assessing the students holistically and identifying the

traits they paid attention to, the teachers then discussed in a small group

how they evaluated each student and which criteria they used for their

assessments All of these activities took place as part of the

aforemen-tioned teachers’ training program and took approximately 2 hours for

each group of teachers All of the teachers’ small group discussions were

transcribed For reporting the results, the transcribed data were

trans-lated into English and back translations were used to ensure accuracy

RESULTS

This study found that both groups of teachers showed substantial

vari-ations in their holistic evaluvari-ations, both within the same group as well as

across groups As we shall see, the most frequently chosen traits by both

groups of teachers were speaking fl uency, confi dence in talking, listening

com-prehension, motivation, and speaking accuracy The results showed similar

tendencies in both groups of teachers However, a qualitative analysis

revealed that the elementary and secondary school teachers interpreted

these traits and arrived at judgments in different ways One could also

observe substantial variability within the same groups of teachers;

differ-ent teachers relied on differdiffer-ent reference points when assessing studdiffer-ents

Moreover, their judgments were frequently infl uenced in nonsystematic

ways by different aspects of the students’ behaviors In this section, I fi rst

review the results of a series of quantitative analyses, followed by a

qualita-tive analysis of the discussions among the teachers In particular, I focus

on the following three issues with respect to the qualitative data: (a) the

nature of the teachers’ observations; (b) the affective constructs

(i.e., confi dence and motivation); and (c) the potential abilities, which

were conceptualized as abilities to acquire high English profi ciency in

the future

Quantitative Analyses of the Teachers’ Judgments

The Variability of Teachers’ Holistic Judgments

As we can see from Figures 1–2 , elementary and secondary school

teach-ers showed substantial variability in their holistic judgments in which they

used the 5-point scale (the same format that has been suggested by KICE)

In the fi rst activity, the students were given an almost equal amount of time

to talk, and they used a limited number of fi xed expressions and

vocabu-lary In this activity the students’ linguistic output was highly controlled,

and yet the teachers’ judgments of the students’ overall performance

showed substantial variability This result may suggest that the teachers paid

FIGURE 1 Teachers’ Holistic Analysis of Activity 1

FIGURE 2 Teachers’ Holistic Analysis of Activity 2