A study of the washback of multiple choice tests on high school english teaching and learning

-MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY NGUYEN THI THANH XU AN A STUDY OF THE WASHBACK OF MULTIPLE CHOICE TESTS ON HIGH SCHOOL ENGLISH TEACHING AND LEARNING SUBMITTED IN PA

-MINISTRY OF EDUCATION AND TRAINING

HANOI UNIVERSITY

NGUYEN THI THANH XU AN

A STUDY OF THE WASHBACK OF MULTIPLE CHOICE TESTS ON HIGH SCHOOL ENGLISH

TEACHING AND LEARNING

SUBMITTED IN PARTIAL FULFILMENT

OF REQUIREMENT OF THE DEGREE OF MASTER IN TESOL

SUPERVISOR: LE VAN CANH, MA

S T A T E M E N T O F A U T H O R S H IP

I certify that all the material

and acknowledged, and that

conferred upon me

in this study which is not my own work has been identified

no material is included for which a degree has already been

Nguyen T h i Thanh Xuan

TABLE OF CONTENTS

STATEM EN T OF A U T H O R S H IP i

T A B LE OF C O N TEN TS ii

A C K N O W LE D G E M E N T S iv

ABSTR AC T V LIST OF A B B R E V IA T IO N S vi

LIS T OF FIGURES, TABLES AND C H A R T S vii

CHAPTER I - IN T R O D U C T IO N 1

1.1 Theoretical Background 1

1.2 Context of the Study 2

1.3 Aims o f the S tudy 5

1.4 Overview o f the S tudy 5

1.5 Scope o f the study 6

1.6 Significance o f the study 6

CHAPTER I I • LIT ER A TU R E R E V IE W 7

2.1 Multiple-choice Tests: Pros and C ons 7

2.1.1 Definition and form o f MCQs 7

2.1.2 Advantages o f M CQs 8

2.1.3 Limitations o f M C Q s 11

2.Z Washback and Its Effect on English Teaching and Learning 15

2.2.1 The role o f testing and assessment in English education 15

2.Z.2 Definitions o f washback and related concepts 16

2.2.3 Negative and positive washback 19

2.2.4 Washback and teachers’ perceptions 21

CHAPTER I I I - RESEARCH M E T H O D O L O G Y 26

3.1 Restating the Research Questions 26

3.2 Participants 26

3.3 Instrument o f Data Collection 27

3.3.1 Document analysis 27

3.3.2 Questionnaire and Interview 33

3.3.3 Procedure o f Data Collection 36

CHAPTER IV FINDINGS AND A N A LY S IS 38

4.1 Questionnaire Data Analysis 38

4.1.1 Teachers’ characteristics 38

4.1.2 Teachers’ perceptions o f the advantages o f MCQs 42

4.1.3 Teachers’ perceptions o f the disadvantages o f M CQs 46

4.1.4 Teachers’ attitudes towards the washback effects of MCQs on their teaching and students’ learning 47

4.2 Interview Data Analysis 53

4.2.1 Teachers’ beliefs o f the current multiple-choice tests 53

4.2.2 Washback effects on the teaching and learning 56

4.3 Discussion 59

4.3.1 Beliefs in the features o f MCQs 59

4.3.2 Teachers’ perceptions o f washback effects 63

CHAPTER V- SUGGESTIONS AND C O NCLUSIO N 69

5.1 Suggestions 70

5.1.1 Combine MCQs with w riting tests 70

5.1.2 Provide teachers with professional training in testing and assessment 72

5.1.3 Create a large item bank 72

5.2 Limitations and Suggestions for Further Research 73

5.3 Conclusion 73

REFERENCES 75

APPENDIX l.Đ ỊN H HƯỚNG K IỂ M TR A K Ế T QUẢ HỌC TẬP P T T H 80

APPENDIX 2.CÁU TRÚC ĐÈ T H I T Ó T N G HIỆP N Ă M 2009 M Ô N TIẾNG A N H 83

APPENDIX 3.ĐẺ T H I TỐT NGHIỆP N Ă M 2008 M Ô N TIÉNG A N H 84

APPENDIX 4 RESULTS OF THE Q UESTIO NNAIRE (QUESTIONS 8-30) 89

APPENDIX 5.TEACHERS’ Q U E S T IO N N A IR E 90

APPENDIX 6 PERSONAL IN F O R M A T IO N ON IN TE R V IEW ED TEACHERS 96

Í would like to express my gratitude to M r Le Van Canh, M.A, for being my thesis supervisor I thank him for his valuable guidance, helpful encouragement and insightful comment fo r the accomplishment o f this study

M y very special thanks go to Ms Nguyen Thai Ha, M A, for her professionalism, dedication and profound knowledge, which have greatly influenced and impressed me

I also wish to express my thankfulness to M r Nguyen Quoc Tuan, M A, from the Institution o f Educational Program and Strategy, Ministry o f Education and Training for his enthusiastic support and constructive suggestion

Finally I wish to acknowledge my family, especially my husband who always motivated

me to write better and cooperated during the process o f data collection

A B S T R A C T

Testing is said to be closely linked with models o f teaching and learning at schools The major purpose o f assessment is and should be to support the teaching and learning process However, it seemed that the operation o f testing is not interrelated with activities o f teaching and learning in Vietnam The present paper investigated the beliefs that upper secondary school teachers have about the adoption o f full multiple-choice tests (MCQs) in English graduation and university/junior college entrance examinations

as well as the impact that these tests can have in the classroom instruction

Questionnaires were collected from 54 teachers who were teaching English at both public and non-public high schools in Hanoi The surveyed was followed by interviews with a small sample o f respondents aimed at providing an insight into the responses made in the questionnaire and to elicit teachers’ perceptions o f the challenges facing them in implementing the test

Three significant findings were drawn from the results: (1) Most o f teachers supported the employment o f MCQs but denied the absolute use o f this technique in the exams It was suggested to combine both MCQs and essay tests so that learners' ability in writing can be assessed; (2) MCQs exerted an impact on the teaching and learning instruction; (3) The constraints that teachers face in designing MCQ items and implementing the teaching to adjust the change in the national testing format

L IS T OF A B B R E V IA T IO N S

Multiple-choice questions: MCQs

M inistry o f Education and Training: МОЕТ

L IS S l OF FIG U RES，TA B LE S A N D C H AR TS

Table 1: English test iteeia types in the graduation exam 2008 (applied to all 7 years, 3 years and subcommitteee programs)

Table 2: Teachers’ percceptions o f MCQs，validity and versatility

Table 3: Teachers’ percceptions o f MCQs，reliability

Table 4: Teachers’ percceptions of MCQs’ practicality

Table 5: Teachers’ percceptions o f the limitations o f MCQs

Table 6: Teachers’ percefptions o f the washback effects o f MCQs on the teaching contentTable 7: Teachers’ percepptions o f the washback effects o f MCQs on the teaching methodsTable 8: Teachers’ percepptions o f the washback effects o f MCQs on the students’ learningTable 9: Teachers’ percepptions o f the washback effects o f MCQs on the teaching materialsTable 10: Teachers' attittudes towardss MCQs

Table 11: Teachers’ positive attitudes towardss MCQs

Table 12: Teachers’ negative attitudes towardss MCQs

Chart 1: Teachers’ gendéer

Chart 2: Teachers’ age

Chart 3: Teachers’ expeirience o f English teaching

Chart 4: Teachers’ types> o f teaching school

Chart 5: Teachers’ qualiification

Chart 6: Teachers' trainiing in MCQs

Chart 7- Teachers’ attitucdes o f the importance o f national exam

C H AP TER I ֊ IN T R O D U C T IO N

Multiple-choice tests have long been the most favorably used in the world and it seems that “ it was the only one way to test" however, in recent years, "this testing technique's limitations have been more generally recognized by professional testers" (Huges, 1989:60) This thesis reports the results o f a study which investigated the high school teachers’ perceived attitudes towards multiple-choice exams and the impact of these tests on the English teaching and learning This introductory chapter provides a description o f the problem, the theoretical background o f the issue under investigation, the context o f the study, the aim and overview o f the study, the significance and the scope o f the study

1.1 Theoretical Background

Public tests are conducted in many countries throughout the world and have been considered to play a significant role in determining what goes on in the classroom in terms o f 4what，and ‘how’ teachers teach and students learn Public exams are often used as instruments to select students as well as a means to control a school system, especially when the educational system is driven by tests or exams (Cheng and Falvey, 2000; Herman, 1992; cited in Chen, 2002)

The influence o f public or national examinations on teaching and learning is

termed as backwash effects by Heaton (1988:170) This phenomenon is also regarded as

test impact (Bachman and Palmer, 1996) These above terms and possible other terms

all refer to different aspects o f the same phenomenon ֊ the influence of testing on teaching and learning Huges (1989) confirmed that backwash can be harmful or beneficial while Spratt (2005) suggested that whether the effects of washback is positive

or negative needs more research to take place into it Alderson and W all (1993) also argued that the relationship between tests and their impact might not be as simple as it is

at first glance and it must be deeply explored in more empirical researches

This paper was carried out to review recent empirical studies of washback of public exams from the point of view o f teachers because they are both the test developers and test makers who determined the success o f the process o f testing reformation Actually, exam drives much o f what teachers do or in other word what the teachers Derceive of the test decides what they w ill do in the classroom Bandura (1986) stated that the beliefs individuals hold are the best indicators o f the decisions they make during the course o f everyday life “ What the teacher thinks, what the teachers believes, what the teacher assumes ֊ all these things have powerful implications for the change process (Hagreaves, 1989: 54) Shohamy (1992:514) confirmed that it is difficult to expect that tests w ill lead to meaningful improvement in learning when the writing o f tests does not involve those who are expected to carry out the change ֊ the teacher Therefore, the study of testing renovation has recently put more concern on the formation and modification o f teacher thinking, their knowledge and beliefs.

1.2 Context o f the Study

In Vietnam, the educational objective is described as being exam-driven, which means that all teachers, students and leaders o f schools try to get the highest scores to maintain their name or prestige Thus, teachers have long been teaching for testing, rather than for learning In fact, the school-leaving and entrance exams do have a great pressure on teachers and learners especially when Vietnam is a society “ where education is for

2007) The importance o f the national examinations cannot be denied: it is a sole determined criterion fo r admission o f students’ future career in pursuing into higher education or getting into job market

So far in Vietnam the quality o f English language testing and assessment has been seen

as a hot topic in educational circles It is said that “ one o f the reasons leading to the poor quality o f English teaching and learning in Vietnam high schools is due to the weakness

insufficiency o f standard framework and guidelines in testing construction procedure as well as the teachers’ backward background knowledge o f this field should be blamed

progress has been made Students can do the test well but they often find itembarrassing to express their ideas in English As a result, a lot of them can not use this foreign language to communicate and miss the chance to get a good job, although they have studied English for at least 7 years in schools.

In recent years, the M inistry o f Education and Training (МОЕТ) has implemented the reformation in English language teaching and learning at the secondary school towards greater emphasis on students’ ability to use English for communicative and academic purposes After years of development, a new English curriculum for all national lower and upper secondary schools was officially approved The new curriculum defines English as “ a compulsory subject” which is “ instrumental to the access o f world science and technology as well as world cultures” (МОЕТ, 2006) Methodologically, the new curriculum adopts a “ learner-centered approach and the communicative approach with

p 12) The aims are to equip students with communicative ability and competence to perform basic language functions receptively and productively, using correct language forms and structures Besides developing the four language skills o f listening, speaking,

reading and writing as well as the accompanying grammar, sound system and

vocabulary, it also adopts an intercultural communication perspective in an attempt to educate students’ into both good national and international citizens who are knowledgeable about the target culture as well as their own national culture (МОЕТ, 2006) This approach is realized in the new textbook structure which entails many activities to develop communicative skills

It is obvious that the final goal o f the testing in the new context should emphasize on the learners’ achievement of both language knowledge and language skills including listening, speaking, reading and writing However, the efforts to attain these expectations seemed to go in vain since the МОЕТ has approved and announced the plan to adopt fu ll MCQs in national examinations in 2006 Three years after the initial experimenting implementation, the use o f 100% MCQs in the school-leaving and university/junior college exams has been considered o f controversial decision

Policy-makers have raised the arguments to support for their modification that MCQs

w ill be very economical and convenient in administering such high-stake exams with

huge number of the candidates In addition, MCQs help in eliminating the cheating and rote-learning and motivating students to learn harder because this technique can measure a large range o f the entire curriculum (Ninh, 2005; Thiep, 2007) Nevertheless,

it is widely suspected that such testing outcomes should be really treated as effective method in improving the quality o f the current testing syllabus as well as contributing to the curriculum innovation in Vietnam Those who are o f the opposite side have pointed out problematic areas o f this testing format

Quite a few o f English researchers and teachers claimed that the present system of examination is mostly based on assessing factual knowledge rather than students’ critical thinking and analytical skills as w ell as their understanding and comprehension Van (2006) maintained that “ The content o f MCQs only focuses on grammar and vocabulary and reading skill; as a result, it deprives the candidates’ creativity and analysis ability The efforts in teaching and learning English towards communicative- oriented approach w ill become nonsense because both teachers and learners w ill pay much attention to practice for the tests and ignore speaking and listening skills” This is echoed by Canh, (2007) that

The current test format only requires the students to answer MCQ questions

on grammar, vocabulary, pronunciation, and reading comprehension plus some controlled writing Neither oral skills nor listening skills are tested In such a test, test-takers’ performance just show the similarities and differences between the language norms o f the examiner and candidate; it tells nothing of the norms which the candidate himself would apply in a use situation, (p 17)What sense do Vietnamese high school teachers, who are the direct testing generators, hold of this testing technique? Do MCQs have any effect on the teaching and learning of English at their schools? I f so, what kind o f effects? There is already a large body o f research done to explore the teachers’ beliefs o f the washback of public exams on teaching and learning in many countries (Chen, 2002; Cheng, 1997; spratt, 2005; Bailey,1999), however, little has been investigated the phenomenon in the context o f Vietnam Obviously, the questions raised so far indicate an urgent need to dig into Vietnamese high school teachers’ attitudes towardss the washback o f MCQs on classroom instruction This study has therefore been implemented to response to that demand

1.3 Aim s o f the Study

The study aims to fu lfill the following objectives:

• To identify Vietnamese teachers’ perceived pedagogical impact that these tests have on their teaching and their students’ learning

as a catalyst for teaching innovation

In order to achieve those objectives, a survey is done on 54 teachers at five high schools located in Hanoi to tap into their perceptions o f the washback o f MCQs in their teaching and students’ learning This paper hypothesizes that MCQs w ill bring about fundamental changes in teaching and learning strategies involving a shift o f emphasis from teaching and learning English for communicative purposes to rote learning of vocabulary and grammar structure

1.4 Overview of the Study

The thesis consists o f five chapters as follows:

Chapter 1- Introduction: provides background to the thesis, the context o f the study, the aims, the significance as well as the scope o f the study The research questions and an outline o f the thesis are also included in this part

Chapter 2- Literature Review: presents the relevant literature This starts with the theoretical background o f MCQs including the definition and significant features o f this technique The second issue to be discussed is the role o f testing and assessment in the English language education The next relevant issue concerns with the terminology of washback and related concepts Finally, the chapter encounters the teachers’ awareness

o f the washback effects o f high-stake examinations in classroom practices

Chapter 3- Research Methodology: concentrates on the methodologies to be exploited, the study procedure and the subjects involved in the study

Chapter 4- Findings and Analysis: discusses in details what have been found from the survey questionnaire and the interview completed by teachers

Chapter 5- Suggestions and Conclusion: provides recommendations using MCQ testsmore properly in the context of Vietnam and implications for further study.

1.5 Scope o f the study

This study limits itself to the survey of English language teachers in five high schools in Hanoi area on theừ perceptions of the impact of multiple-choice tests on their teaching and their students’ learning The instrument used was just a survey questionnaire and face-to- face interviews The questionnaire was aimed to examine teachers’ opinions of how multiple-choice tests influence their teaching and their students’ learning The interview, which was conducted after the analysis of the questionnaire data, was aimed to gain understanding of why teachers thought the way they had done about this approach to testing Generations should, therefore, be made with caution

1.6 Significance o f the study

This study is about to bring insight into MCQs’ advantages and disadvantages perceived

by high school English teachers in Vietnam as well as the effect that these tests can have

on the teaching and learning The findings of this study will be helpful for English

teachers to clarify the strengths and weaknesses o f MCQs and then they can use the tests more appropriately as a catalyst for teaching innovation Also with this study, educational administers realize the teachers’ constraints in designing a MCQ test and implementing it in the classroom so that corresponding measures can be found to make their teaching and testing more effective

CHAPTER I I - L IT E R A T U R E R E V IE W

following with the description o f their pros and cons In the second part, I w ill go todiscuss more the role of testing and assessment in English education, the concepts of washback effects and teachers’ perceptions o f this phenomenon in different investigated contexts

2.1 M ultiple-choice Tests: Pros and Cons

2.1.1 D efinition and form o f MCQs

Multiple-choice question is a form of objective item for which respondents are asked to select one or more o f the choices from a list In other words, when doing a MCQ question, the candidate is required to choose an answer from a number o f given options, only one o f which is correct

According to Huges (1989:59) multiple-choice items take many forms, but their basic structure is described as follows:

problem to be solved, a question asked o f the respondent, or an incomplete statement to

F o r example:

Normally, the number of options ranges from three to five but four is the most popular The distractors should embody misconceptions, partly correct answers and common

errors o f fact or reasoning so that they distract students who are not well prepared forthe test from giving the correct answer.

There are several identified advantages o f this objective technique I f item writers are well trained and items are quality assured this can be a very effective and good test The most noticeable strengths o f MCQs are validity, versatility, reliability and practicality

2.1.2.1 Validity

Validity refers to how well the assessment instrument has, in fact, measured the original objectives o f the test Heaton (1988:159) emphasized “ the validity of a test is the extent

examination using MCQs is high because a large part o f the syllabus can be assessed Typically, an examination composed o f only essay-type questions can assess only part

o f the syllabus A student is, therefore, able to answer many MCQ items in the time it would take to answer a single essay question This feature enables the teacher using MCQ items to test a broader sample o f the course content in a given amount o f testing time As a result, the test scores w ill likely be more representative o f the students’ overall achievement in the course

2.1.2.2 Versatility

Versatility refers to the multiple-function o f MCQs in measuring a wide range of educational objectives The versatile use o f MCQs is appropriate for use in many different subject-matter areas Well-constructed MCQ items can assess a variety o f learning outcomes, from simple recall o f facts to Bloom’ s highest taxonomic level of cognitive skills - evaluation (Osterlind, 1998; Case & Swanson, 2001，cited in McCoubrie, 2004; Clegg & Cashin, 1986) This taxonomy o f the levels within the cognitive domain is presented in the following diagram (Figure 1):

'ầ I 출f ՜ 룹롭^^^^^^확

Figure 1: Bloom ’s taxonom y o f cognitive levels knowledge

Knowledge: simple recognition o r recall o f m aterial

Comprehension: restating o r reorganizing m aterial to show understanding

A pplication: problem solving o r applying ideas in new situations

A nalysh: separating ideas into component parts, examining relationships

Synthesis: combining ideas into a statement o r product new to the learner

E valuation: judging by using self-produced crite ria o r established standards

Doing MCQs, the learners can be evaluated from the the lowest level o f cognition (knowledge) to the highest level (evaluation)

2.1.2.3 Reliability

Reliability is a quality o f test scores and a perfectly reliable score, or measure, would be one which is free from errors o f measurement (Alderson, Clapham and Wall, 1995:88) MCQs are called reliable due to their broad coverage o f knowledge and trustworthy scores

Huges (1989:59) suggested that “ the most obvious advantage o f MCQ is that scoring can be perfectly reliable” In fact, on many assessment, reliability has been shown to improve with large numbers o f items on a test, and with good sampling and care over case specificity, overall test reliability can be further increased Huges added:

Multiple-choice test is likely to make for greater test reliability since the candidate only have to make a mark on the paper to respond, it is possible to include more items than would otherwise be possible in a given o f time.(p 60)Also contributing to the reputation o f MCQs as “ reliable” is that test makers often promote MCQs as objective because there is no human judgment in the marking process, which is usually done by machine Hence, variations in marking due to

commented one good point that MCQs enjoy among professional testers is their

evaluate the result o f a subjective response, the test-marker often with their background knowledge and subjective justification show their opinion of that work The subjective impressions o f scorers to determine learners’ grades does not always give a true measurement o f the students’ ability Whereas, scorers do not need to exercise judgment

in marking responses correct or incorrect in an MCQ test The result o f MCQ exams, therefore, is said to be reliable

2.1.2.4 Practicality

According to Heaton (1988:67) “ a test must be practicable, in other words, it must be fairly straightforward to administers, o ile r (1979:4) agreed that practicality “ must include the preparation, administration, scoring and interpretation o f the test” Brown (1994:253) also said that “ a test ought to be practical within the means o f financial limitations, time constraints, ease o f administration, and scoring and interpretation”

In terms o f practicality, this alternative testing technique is more widespread since it is more economical than subjective tests The expenses for its making and scoring have been decreased The marking work becomes easier and “ they do not necessarily require

an experienced tutor to mark them ，’ (Higgins and Tatham, 2003) Moreover, it is a matter o f concern that due to the difference in opinion presentation to the same problem

o f each candidate, there is no standard key to be set in an essay test Therefore, the score marking o f subjective tests takes more time and effort MCQ is also suitable with exams

o f greater assessment and increasing number o f candidate because expenses for the exam organization w ill be minimized Clegg & Cashin (1986) affirmed that MCQ items require relatively less student time to answer and can be easily and accurately scored by

a person or machine Farthing et al (1998) asserted

MCQs can be marked quickly Typically, marking a script of 10 or 20 MCQs might take about a minute or so Because the marks are reliable, no further time

is spent remarking scripts This advantage is valuable with large classes, and even more so where teams of assessors are needed, since cross-marking is unnecessary

Obviously, theoretically, such good qualities o f a MCQ test w ill make it a very effective testing technique but any kind o f testing contains its own strengths and weaknesses and MCQ is not an exception

2.1.3 Lim itations ofM CQ s

Though MCQs offer many benefits, it should be recognized that the difficulties with thistechnique are varied Huges (1989:60-61) suggested the four following generatedshortcomings

1) The technique tests only recognition knowledge

Arguing that MCQs test only recognition knowledge, Huges wants to mention the question o f construct validity He comments that "A multiple-choice grammar test score, for example, may be a poor indicator of someone's ability to use grammatical structures The person who can identify the correct response in the item above may not

be able to produce the correct form when speaking or writing." (P.60) In fact, i f we want to know how w ell someone can write, there is no way we can get a really accurate measure of their ability by means of a MCQ test Multiple-choice questions can test much more knowledge but they cannot test oral or written skills As a result, the test scores w ill reflect incomplete information o f learners’ ability

His idea seems to be in conflict with the prior authors (Osterlind, 1998; Case & Swanson, 2001 and Clegg & Cashin, 1986) who claims that MCQs can test all level o f cognition Heaton (1988) shared with Huges when he confirmed that testing technique used in objective tests such as MCQ do emphasize more on accuracy, and may not give accurate assessment o f the student’ s performance In addition, since the students select a response from a list o f alternatives rather than supplying or constructing a response, MCQ items are not adaptable to measuring certain learning outcomes For example, MCQ testing is “ appropriate for testing comprehension, application, and content

learning outcomes are better measured by short answer or essay questions, or by performance tests Students may recognize o f known facts or procedures well enough to score high on the test, but not be able to think about the subject or apply knowledge, even though being able to think and apply is essential to knowing any subject

2) Guessing may have a considerable but unknowable effect on test scores

Another pitfall o f MCQs is ambiguity Huges critisized that "the chance o f guessing the correct answer in a three-option multiple-choice item is one in three, or roughly thirty-three percent" It is possible to get MCQ items correct without knowing much or doing any real thinking because all the answers are in front o f the students In addition, MCQ items can be easier than open-ended questions asking the same thing This is because it is harder to recall

an answer than to recognize it It is possible to choose the right option for the wrong reason

or to simply make a lucky guess Higgins & Tatham (2004) called this phenomenon “ lucky

these authors, “ this is no worse than a student who adopts the write-all-you-know approach

to a question for which she can generally expect to pick up marks for the correct points the

might reduce the reliability o f MCQ testing scores

3) The technique severely restricts what can be tested

It is not easy to use MCQs to assess the necessary command o f language because multiple-choice items require distractors, and distractors are not always available Huges took an example o f the distinction in English between the past tense and the present tense and argued that for learners at a certain level o f ability, in a given

linguistic context, there were no other alternatives that were likely to distract As aresult, understanding o f what may be an important structure is not tested.

4) It is very difficu lt to write successful items

As seen from the perspective o f the practicality criteria despite the lower cost for score- marking, one weakness o f MCQs is the difficulty o f construction “ No one w ill argue that while the multiple-choice test is one of the easiest to mark, it is also very time- consuming and difficu lt to create (Davies, 2001) He advised that it is not possible to create a multiple-choice exam at the last minute It may appear to be fairly simple to construct items in the MCQ format However, it is constructing a good MCQ test item that is so d ifficu lt and time-consuming

Huges (1989) also approves that good multiple ֊ choice items are notoriously difficult

to write A great deal o f time and effort has to go into their construction “ MCQ items take time and skill to construct effectively” , Clegg & Cashin (1986:1) Constructing good limited-choice exams takes much longer than open-ended exams because items must contain the pertinent information students need to answer the question, a set of

appropriate distractors, and the correct answer In addition, none of the information

should include clues that point to the correct answer or be written in a way that unnecessarily confuses student reading and interpretation o f the item

Oiler (1979:237) supports this belief when pointing out two challenges in writing items The first is to decide what sort o f items to write The second problem is to write the items with suitable distracters in each set o f alternatives The question o f validity should

be raised when w riting items They also acclaimed that the first principle is to ask i f the material to be included in items in the test is somehow related to what is supported to measure (including skill, construct, curriculum etc.)

5) Backwash may be harmful

Huges (1989:60) assumes that there is a danger that practice for the MCQs w ill have a harmful backwash on learning and teaching As much attention is paid to improving one’s educated guessing as to the content of the items, practice at MCQ items w ill not be the bestway for students to improve thd r command o f a language Some other authors agreed that

.those relating to traditional, multiple-choice, large-scale tests are perceived to have had

1992; Shepard, 1990; cited in Cheng, Watanabe & Curtis, 2004)

Obviously, a major danger with high-stakes MCQ and short answer tests is the only things that are easily measured are taught When schools view MCQs as important, they often narrow their curriculum to cover only what is on the exams For example, to prepare for MCQs, the curriculum in high school may focus on practicing grammar knowledge ignoring speaking and listening skills When narrow tests define important learning,

like the test In this case, students often get no chance to read real books, to ask their own questions, to have discussions, to challenge texts, to explore new ideas ֊ that is to think about and really learn a subject Fuhrman (1996) cautioned that the exclusive use of MCQs could decrease the motivation to teach skills and knowledge that cannot be tested by this method

6) Cheating may be facilitated

Multiple choice tests also have another problem inherent in them in terms of its honest

scores Though the marking is said to be objective one thing should be considered is the cheating can be facilitated (Huges, 1989) It is possible for the candidates to communicate

Limited-choice exams provide more opportunities for cheating than open-ended exams since single letters or numbers are far easier to see or hear than extensive text

In brief, MCQs have both pros and cons but as stated by Farthing et al (1998)，“there is

no commonly accepted notion o f what makes one assessment better than another” What

we can say is that MCQs have strengths when compared with other forms of assessment, and these have to be balanced against the known weakness of non- subjective assessment The significant concern is i f MCQ technique is not applied properly in national examinations, its disadvantages w ill outweigh good points and produce unexpected results for the whole process o f the testing and assessment

2.2 W ashback and Its E ffect on English Teaching and Learning

2.2.1 The role o f testing and assessment in English education

Assessment or testing plays a central role in foreign language teaching and learning It has long been asserted that tests have a powerful influence on language learners who are preparing to take the exams, and on the teachers who try to help them prepare It is testing, not the officia l stated curriculum, that is increasingly determining what is taught, how it is taught, what is learned, and how it is learned (Madaus, 1988:83) Assessment is said to be closely linked with models of learning and impacts on both teaching and learning The major purpose of assessment is and should be to support the

purpose is achieved in reality or not depends on how assessment is implemented

In fact, the most important function o f testing is to serve as an educational tool, not simply as a basic for grading Tests can direct students’ studying and provide important corrective feedback fo r the students Ramsden (1992) emphasized that measuring students’ learning and diagnosing specific misunderstanding w ill help them learn more effectively Huges (1989) also mentioned that testing is appropriate for individuals to be given a statement o f what they have achieved in a second or foreign language and provide information about the achievement o f groups o f learners, without which it is difficu lt to see how rational educational decisions can be made

Tests are also beneficial to teachers o f English as they can help them diagnose their own work, their own efforts and make adjustment in their teaching in order to increase their own effectiveness (Heaton, 1988) Alderson (1986:104) discussed the “ potentially powerful influence o f tests” , and argued for innovations in the language curriculum through innovation in language testing” Besides, Shohamy (1993) stated that the power and authority o f tests enable policy-makers to use them as effective tools for controlling educational systems and prescribing the behaviour o f those who are affected by their

In the foreword to Cheng et al (2004:xi), Alderson further confirmed that “ high-stakes tests

- tests that have important consequences for individuals and institutions - w ill have more

is a criterion for admission into higher education, it supports the processes o f self-evaluation and improvement of educational institutions, and it serves as an instrument and basis for both research work and for granting certain educational benefits.

In sum, the influence o f testing on language teaching and learning is undeniable The validity o f tests crucially relates to not only testers but also teachers Testing w ill become an effective tool in evaluating students’ language achievement i f through it their learning can be enhanced and deepened It is also important for teachers to have a thorough understanding o f the role o f testing in their teaching I f teachers cannot interpret what the test is to achieve, i f they do not understand the item types, the instructions, the criteria for marking, the marking scheme, they w ill not be able to prepare students adequately for the test The better they understand what is being tested, the better they understand how different kinds o f testing techniques are used for different educational purposes and the more likely it is that they change the way they teach towardss intended change or innovation

defined“ washback is the influence of testing on teaching and learning" while Huges

idea when he remarked that “ washback is the effect o f a test has on teaching in the

the test influences language teachers and learners to do things ’’they would not necessarily otherwise do because o f the test’’ Messick (1996:241) affirmed that

•’washback, a concept prominent in applied linguistics, refers to the extent to which the introduction and the use o f a test influences language teachers and learners to do things they would not otherwise do that promote or inhabit language teaching"

Such definitions refer only to washback in terms o f influence that tests might have on the language classroom and the participant roles o f teacher and learner Chen (2001) explored the term washback to a broader meaning as “ the way in which public

"public examinations influence the attitudes, behaviours, and motivation o f teachers, learners and parents, and, because examinations often come at the end of a course, this influence is seen working in a backward direction-hence the term Mwashback" Andrews (2004) also used the term to describe “ the effect o f tests on teaching and learning, the

distinguishes between test impact and test washback in terms of the scope of the effects

individuals, policies or practices, within the classroom, the school, the educational

cited in Cheng, 2000).Though different terms are referred by different researchers, they all refer to the influence o f testing on teaching and learning, therefore the term

washback is also used to refer interchangeably to both impact” and “ backwash” In this

study, washback is the most principally used

Bachman and Palmer (1996:35) noted that washback is a more complex phenomenon than simply the effect o f a test on teaching and learning Instead, they feel the imoact of

a test should be evaluated with reference to the contextual variables of society’ s goals and values, the educational system in which the test is used, and the potential outcome

of its use Shohamy, (1993a:4) summarized four key definitions that are useful in understanding the washback concept:

֊ Systematic validity implies the integration o f tests into the educational system and the need to demonstrate that the introduction o f a new test can improve learning.

Alderson and Wall (1993) distanced themselves from a simplistic assumption about the way a test can influence behaviors Therefore, they developed 15 Washback Hypotheses as follow:

6) A test w ill influence what learners learn

7) A test w ill influence the rate and sequence o f teaching; and

8 ) A test will influence the rate and sequence of learning

9) A test w ill influence the degree and depth o f teaching; and

10) A test w ill influence the degree and depth o f learning

11) A test w ill influence attitudes to the content, method etc of learning /teaching.

12) Tests that have important consequences w ill have no washback: and

conversely,

13) Tests that do not have important consequences w ill have no washback.

14) Tests w ill have washback on all learners and teachers.

15) Tests w ill have washback effect for some learners and some teachers, but

not for others

Which hypotheses w ill be put forward depends on the nature of the test, the educationalcontext, and the nature o f the decisions that are taken on the basis o f the test results.

2.2.3 Negative and positive washback

2.3.3.1 Negative washback

Huges (1989) stated that backwash can have negative effect on teaching and learning

He said " I f a test is regarded as important, then preparation for it can come to dominate all teaching and learning activities And i f the test content and testing techniques are at variance with the objectives o f the course, then there is likely to be harmful backwash", (p 1) Pearson (1988:101) agrees that "washback effect w ill be negative i f it fails to reflect the learning principles, and/or objectives to which they supposedly relate"

W hile the above authors mentioned the gap between the test content and the curriculum might cause unexpected washback, Alderson and W all (1993:120) suggests that a poor test may cause negative or undesirable effect on teaching and learning ’’Poor'’ usually means something that the teacher or learner does not wish to teach or learn” , therefore, the test may fa il to reflect the learning principles and the educational objectives which

they are supposedly related

Vernon (1956:166) asserts that in case o f negative washback teachers tended to ignore the subjects and activities, which did not contribute directly to passing the exam (cited

in Cheng, 2000:10) W all & Alderson (1996) further comment that under negative washback, “ teachers would use whatever methodology they felt most expedient to help them to prepare their students for the examination” (p 200) Some aspects of teaching methods that were thought inefficient for preparing the students for the examination by the teachers might be neglected Smith (1991a:3) shares this point of view that i f teaching is directed to focus on practicing for the exam, it is difficult to assess learners' knowledge accurately He confirms that ’’high-stakes testing affected teachers directly and negatively” and that ”teaching test-taking skills and drilling on multiple-choice worksheets is likely to boost the scores but unlikely to promote general understanding" (cited in Bailey, 1999).2.3.3.1 Positive washback

On the other hand, washback is not always thought to bring unexpected effects Huges indicates that positive washback w ill occur i f the changes in testing can bring better result on the students’ language learning Alderson and Wall (1993:115) suggest that beneficial washback means teachers and learners w ill have a positive attitude towards the test and work w illin g ly towards its objectives.

Further evidence of positive washback is that ’’good teaching-learning tasks w ill be more or less directly usable for testing purposes, even though practical or financial constrains lim it the possibilities" (Pearson, 1988:107)

Bachman and Palmer (1996:33) suggest that one way to promote the potential for positive impact is through involving test-takers in the design and development of the test, as well as collecting information from them about their perceptions o f the test and

perceive tests as more interactive and authentic, and w ill therefore be more motivated, which could lead to enhanced preparation and hence to better performance

Huges (1989:44-46) suggested seven ways for the test designer to promote beneficial washback:

1 rest the abilities whose development you want to encourage

He stressed that “ there is a tendency to test what it is easiest to test rather than what it is

productive language skills (i.e writing and speaking) when subjective marking is required

2 Sample widely and unpredictably

To avoid the content o f a test becoming too predictable, item types should be designed

so that a range o f task is utilized

3 Use direct testing

Huges defined direct testing as “ the testing o f performance skills, with texts and tasks as

4 Make testing criterion-referenced

Criterion-referenced tests are seen to exhibit beneficial washback due to their motivating factors For example, rather than being measured against the performance o f other test-takers, Huges believed that it is better when candidates are clear about exactly what they have to achieve in order to pass a test.

5 Base achievement on objectives

I f achievement tests are based on objectives, rather than on detailed teaching and textbook content, they w ill provide a truer picture o f what has actually been achieved

6 Ensure that the test is known and understood by students and teachers

However good the potential backwash effect o f a test may be, the effect w ill not be fu lly realized i f students and those responsible for teaching do not know and understand what the test demands o f them

ᄀ Where necessary, provide assistance to teachers.

One important reason for introducing the new test may have been to encourage communicative language teaching, but i f the teachers need guidance and possibly training, and these are not given, the test w ill not achieve its intended effect Where new tests are meant to help change teaching, support has to be given to help effect the change

2.2.4 Washback and teachers’ perceptions

There is some evidence to suggest that tests have washback effects on both teaching and learning (Alderson & Wall, 1993) However, Bailey (1999:17) affirmed that “ the most visible participants in program washback are language teachers It is they who are the “ front-line”

available empirical research on washback includes at least some focus on teachers In fact, it

is safe to say that teachers are the most frequently studied of all the participants in the washback process Shohamy (1992) emphasized the central role of teachers when she identified some o f the conditions that can lead to negative washback to programs:

After all, when reliance is on tests to create change: when emphasis is mostly on proficiency and less on the means that lead to it (i.e., what takes place in the classroom as part of the learning process); when tests are introduced as authoritative tools, are judgment, prescriptive, and dictated from above; when the writing of tests does not involve who are expected to carry out the change ֊ the teacher: when the information tests provide is not detailed and specific and does not contain meaningful

feedback and diagnosis that can be used for repair, it is difficult to expect that tests will lead to meaningful improvement in learning, (p.514)

Studies that tapped into teachers’ beliefs o f the washback o f public exams have identified various aspects of classroom which can be affected by washback such as teaching content, teaching materials, teaching methods and learning attitudes (Alderson

& Hamp-Lyons, 1996; Cheng, 1997; W all & Alderson, 1993, Spratt, 2005; Bailey,2000) The paper w ill review the findings for each o f these areas in turn

2.2.4.1 Test W ashback Effects on the Teaching Content

W all & Alderson (1993:28) concluded from their study in Sri Lankan that there existed washback on the content o f teaching and some of this is positive and negative They found that the public tests brought beneficial washback when teachers determined to cover the textbooks On the other hand, due to the test, more attention was paid to reading and less to oral skills than textbook provided for They also added that "teachers skip listening lessons because the exam does not test this skill and long examination preparation period with materials reflecting content o f exam and abandonment of listening and speaking"

Similarly, according to Anderson et al., (1990) in Cheng (1997), when examining the impact o f re-introducing final examinations at Grade 12，teachers reported a narrowing

to the topics the examination was most like ly to include, and that students adopted more

of a memorization approach, with reduced emphasis on critical thinking

In his study of teachers’ beliefs about curricular innovation in Vietnam, Canh (2007:11) reported the same results His interviewed teachers believed that the change o f the testing format to MCQs produced remarkable impact on the content teachers choose to teach in their classroom The lesson objectives stated in the lesson plan are to develop the students’ communicative skills, but actual classroom teaching just emphasizes how to help the students to do well in the multiple-choice tests

In another context, Cheng (1997:50) identified the washback on the teaching content in a way that "the most dramatic change due to the introduction of the 1996 HKCEE lay in the content o f teaching By the time the examination syllabus affected teaching in Hong Kong secondary schools nearly every school had changed theừ textbooks for the students”

2.6.2.2 Test W ashback Effects on the Teaching M ethods

Canh (2007) revealed that the pressure o f preparing the students confront the teachers with their old ways o f teaching For example, one teacher in Canh’ s study stated explicitly in the interview that:

The new textbook requires the teacher to change her methods of teaching and

to be more dynamic However, students have great difficulty participating in communicative activities due to their English deficiency; therefore, they just pay attention to grammar learning Given this fact together with the multiple- choice testing, teaching has to be grammar-focused (p 12)

Examining about the TOEFL (which is also adopted MCQs) and non-TOEFL classes, Alderson & Hamp Lyons (1996;288-289) found that teaching methods do change, as classes become more teacher-dominated, traditional, less student-sensitive, more chalk and talk, there

is less turn taking, and the turns are somewhat longer Following are the major results:

classes;

7) TO EFL classes are somewhat more routinized;

8) There is much more laughter in non-TOEFL classes.

2.6.2.3 Test W ashback effects on the Teaching M a teria ls

classroom to prepare their students for the exams

Examining the use o f materials, Alderson and Hamp Lyons (1996) indicated that at least

in the situation they investigated many teachers independently of their amount o f

experience o f teaching towardss the exam, making heavy use of exam materials Fullilove (1992) when explored the teaching materials designed to prepare students for the public examinations also stated that many such texts are “ little more than clones of past exam papers” (p 139) He goes on to state that “ some students in Hong Kong, particularly the weaker candidates, tend to spend long hours memorising those model answers, rather than actually learning how to answer similar questions” (ibid, p 139, cited in Bailey, 1999).

In terms o f material content, Hamp Lyons (1998:332) has a thorough view on the content

o f TOEFL preparation textbooks and found that "the skills promoted by the textbooks generally consist of test-taking strategies and mastery o f language structure, lexis and discourse semantics that have been observed on previous TOEFLs" and "the books used for this study promote skills that relate quite exactly to the item types and item content found on the actual test rather than to any EFL/ESL curriculum or syllabus or to any model o f language in use"

2.6.2Л. Test W ashback effects on the Learning A ttitudes

There is little empirical evidence available to provide a basis for the exploration of washback effects on English learning strategies Wall (2000:502) has said: “ What is missing are analyses of test results which indicate whether students have learnt more or learned better

learning approaches and performance to be discovered over the past few years

Shohamy, Donista-Schmidt & Ferman (1996) asked their student respondents whether

or how the A S L test and EFL test had promoted learning W ith regard to the high-stakes EFL test, 68% o f the students believed that the test promoted learning and 92% said the goal o f the test was to promote learning

Similarly, Chen (2001) found that that students’ learning attitudes influenced the teachers’ perceptions o f the impact o f the public exam on their curricular planning and instruction It is also reported in this study that students who were highly perceived to

be motivated to succeed in the exam were more likely to expect their teachers to promote their testing skills and cover as much content related to the exam as possible

In this chapter, a literature review on different aspects and issues of language test, including the advantages and disadvantages of MCQs, the role of testing in English education as well as the washback effects of public examinations on the learning and teaching was discussed As indicated in this chapter that high-stakes exams do have impact on what happen in the classroom Unfortunately, this issue has not yet fully researched in the context of Vietnam This study is, therefore, an attempt to provide more insightful picture of this phenonmenon The next chapter w ill present research methodology used for the current study.

3.1 Restating the Research Questions

The general objective o f this study is to find out the perceptions of high school English teachers on the MCQs and the washback effect that these tests have on their daily teaching and students9 learning The focus o f the study, as stated in Chapter One, is to discover:

1 What do high school English teachers perceive o f the advantages and disadvantages

o f the m ultiple-choice tests that have been used to assess the students?

2 What influences do they think these tests have on their teaching and their students' learning?

3.2 P articipants

Participants in this study were from high schools located in Hanoi Due to the limited time and access difficu lty only English teachers teaching at five randomly selected schools in five districts in Hanoi were invited to respond to the survey questionnaire Although the number o f schools was limited to five, these schools were relatively representative in terms o f academic features, students’ characteristics and school types One of them is a foreign languages specializing school while the remaining schools are not One school’ s academic standard is particularly high because o f the increasing number o f students who obtain high scores in the two national exams Two of the schools are considered to have good quality in teaching and learning basing on theresults o f the national exams The rest schools’ prestige is not remarkably high among

the schools in Hanoi, but they certainly are qualified as among the best private schools located in the capital city.

A ll interviewed teachers were purposefully selected from Hanoi’ s high schools and had the following characteristics:

they could provide the needed information related to the research topic

without force

3.3 In stru m e nt o f Data C ollection

Both qualitative and quantitative data were collected for this study to investigate teachers' perceptions Qualitative data included document analysis and teacher interview Document analysis focused on the real test papers for the high school graduation exam and some relevant policy documents Quantitative data were from the questionnaire Although the quantitative data were more important, the qualitative data helped to shed light on some issues which emerged in the process o f analyzing the quantitative data More specifically, the qualitative data was useful because it enriched the study as a whole and, additionally, functioned as a way to crosscheck and validate the data collected through the questionnaire

3.3.1 Document analysis

In the process o f collecting supporting document for this study, it was necessary to start with the official documents issued by the МОЕТ which cleared out the policy of educational renovation, the framework for the English testing and assessment at secondary schools and the adoption o f MCQs in national examinations This information was then compared with the results taken from the teachers’ survey questionnaire and interviews to generate assumptions o f teachers’ attitudes towards MCQs being used in Vietnam

Following is a brief description o f the testing syllabus adopted in secondary schools in the context o f education reformation towards communicative approach and the real test that students took in the year 2008 (For further detail, see Appendix 1，2 and 3)

As it was previously mentioned, it is a necessity that the testing system be adjusted to meet the new demand o f the renovated curriculum The objectives o f testing and assessment at high school level have been therefore directed to consider communicative competences as centre for evaluation Specifically, they focus on the following targets:

skills (listening, reading, speaking and writing) and the basic knowledge (phonetics, vocabulary and grammar) in English

curriculum and textbook

comprehension and application

the tests

(МОЕТ, 2006)

A t high school stage, the assessment o f English language competence required learners

to achieve the first three levels o f Bloom’ s taxonomy of cognition including knowledge,

pass this level, the learners only have to recognize or memorize the phonetics-related aspects such as pronunciation of vowels and consonants or English word stress; vocabulary, simple grammatical structures or basic concepts that they have just acquired Then, learners w ill have to overcome a higher step o f cognition to

information o f the target language After understanding the language, learners w ill have show their ability in applying those acknowledged concepts to produce their own

However, the actual examinations held in Vietnamese high schools do not always go in the same line with the proposed objectives of the testing syllabus Obviously, such 100% MCQ exams w ill fail to evaluate learners' ability to apply their language knowledge productively due to the lack o f speaking and writing skills.

distribution in the 2008 versions o f the test, as well as the type of sub competencies evaluated in types o f task As can be seen from the test sample, out of 50 items o f the test, the content o f the test emphasize on assessing students language knowledge and reading comprehension ability

CONTEXTS OF EVALUATION NUMBER

OF ITEMS

COMPETENCE TO

BE MEASURED

Identification of word stress and pronunciation of consonants

and vow els

Identification of tenses and sequence of tenses, connectives,

syntactical structures, simple communicative functions, and

others etc.

20

Grammatical

General cloze procedures (1 text, ±150-word long) Cloze test

where missing word are verbs, prepositions, connectors,

adjectives, nouns, etc.

5

Textual

com prehension

Table 1: E nglish test item types in the graduation exam 2008 (applied to a ll 7

years，3 years and subcomm ittee program s).

The first five sentences test students’ knowledge o f pronunciation o f the same consonants and vowels in different words Here is an example:

Choose the option A or B，c, D which has the underlined p a rt is differently pronounced from the other words:

Question 46: A soon в food с flood D moon

This type o f task naturally asks students to identify the different sounds in English and partly help them to produce saying word correctly It should be considered an effort of testing makers to bring the oral assessment into the test but it appears to be invalid This exercise should be treated as a kind of grammar task because the speaking skill can only

be evaluated through the ability to use English productively

The next 25 items test exclusively grammatical competence: knowledge of discrete vocabulary items or grammar structure, understanding in tenses, tense agreement, connectives, preposition, conditionals, sentence agreement and other grammatical categories, word forms/classes and word group/phrases It cannot be denied that MCQs

is very useful in evaluating a wide range o f knowledge This testing format requires the candidates to study comprehensively and develop a deep understanding o f fundamental concept to complete this task

Here is an example:

Question 21 checks the students’ knowledge in using English preposition:

- M y sister is very fond eating chocolate candy.

Question 26 tests the candidates’ knowledge in sentence agreement:

- Tom: “ How did you get here?” - John: “ ’’

Л Is it fa r fro m here? в I came here last night

ᄂ The tra in is so crowded D I came here by train

The other 10 items test textual competence or textual coherence The section requires the test-taker to choose one o f the provided alternatives to f ill in each blank in a 100- word passage In order to do this exercise, the students must be aware o f the topic mentioned by looKing through the passage and then use their knowledge in English

students to know the correct verb form or the right preposition in a certain sentence In some cases it also requires understanding o f relationships between adjacent sentences or clauses, asking students to choose the correct conjunction, connector or to interpret a reference This kind o f testing is very useful since it can test both students’ grammar knowledge and reading comprehension skill.

Here is an example:

Today y supermarkets are found in almost every large city in the world But the firs t supermarket (6) opened only fifty years ago It was opened in New York by a man named Michael Cullen A supermarket is different (7) other types o f stores

in several ways In supermarkets, goods are placed on open shelves The (8) choose what they want and take them to the checkout counter This means that few er shop assistants are needed than in other stores.

Ն ա :• A managers в assistants C Customers D sellers

The error correction which is aimed to test the writing skill normally consists of sentences in which four words are underlined, one o f them is incorrect The candidate is required to choose from the options (А, в, с or D) the incorrect one