MULTIPLE CHOICE QUESTIONS Design and Evaluation

Workshop Goals To review the advantages and disadvantages of multiple choice tests  To explore tips and techniques for writing successful multiple choice items  To evaluate and correc

MULTIPLE CHOICE QUESTIONS

DESIGN AND EVALUATION

Helen Huntley Senior English Language Fellow

Ministry of Education and Training

Hanoi, Vietnam

Workshop Goals

 To review the advantages and

disadvantages of multiple choice tests

 To explore tips and techniques for

writing successful multiple choice items

 To evaluate and correct multiple choice questions and answers

Objective Testing

“Tests in which the learning material is

divided into small units, each of which

can be assessed by means of a test item with a single correct answer that can be specified in advance.”

Reed, J (2000) Assessing vocabulary Cambridge University Press

Advantages of Multiple Choice Tests

 good discrimination among levels of ability

 high reliability

 valid indicators of overall language ability

 highly practical: convenient to administer

 well-established procedures to analyze items

 writing ability does not affect performance

 cost-effective format - easy scoring and

grading

 familiarity of students with MCQs

Disadvantages of Multiple

Choice Tests

 tendency to treat words as independent linguistic units

 difficulty in construction

 field-testing, analysis, refinement required

 25% chance of guessing correctly

 items may test knowledge of distractors

 wrong answer may result from lack of understanding

of syntax (English)

 permits only limited sampling of total knowledge

 tests only recognition knowledge

 restricts what can be tested

 washback may be harmful

 cheating may be facilitated

Match the Test Items to the

Blueprint and Task Specifications

 Addresses curriculum objectives in terms of

performance (can do)

 Outline of the test

 Skills to be included

 Item types and tasks

 Topics to be covered

 Elicitation and response formats for items

 Number of items in each section

 Time allocation

 Plan for scoring

Using Item Indices to Accept, Discard, or Revise Items

 Item facility: the extent to which an item is easy or difficult for the proposed group of

test takers

 Item discrimination: the extent to which

an item differentiates between high- and ability test-takers

low- Distractor efficiency: the extent to which the distractors ‘lure’ a sufficient number of

test takers

Multiple Choice Tips I

 Each test item should be designed to measure a specific objective

 The question or task should be clear, testing only one

concept

 The answer should involve interpretation of the material, not just background knowledge

 A context should be provided for the items

 The items should not show an insensitive or unfair bias

 The number of response options should be the same in each question

 Ambiguity in the answers must be avoided; the intended answer should clearly be the only correct one.

 The options should all be similar in length and level of

difficulty

Multiple Choice Tips II

 Items at different levels of difficulty should be

included: start with the easiest first

 Both the stem and options should be stated as simply and directly as possible

 Writing illogical or nonsensical distractors should be avoided

 Extraneous clues should not influence the answer

 The correct answers should appear in all positions (A,

B, C, D)

 Words should not be repeated in the answers

 A negative ‘stem’ should be avoided The questions

should not be dependent on previous items

 Appropriate punctuation should be used

 Grammar reference texts

 High school textbooks

 Flesh-Kincaid Readability Grade (MSW)

Sample MC Questions

Evaluate the following multiple choice

items

 What is the objective of each question?

 Identify problem areas (see Multiple

Choice Tips)

 Correct/improve problem areas

Excuse me, do you know _?

a. where is the post office

b. where the post office is

c. where post office is

 Tuesday and Wednesday come before

Thursday but after _.

The answer to the last question is the

_ day of the week

a. first

b. forth

c. three

d. fifth

Hilary lives in New York City _ works

in the hotel business

a. her

b. he

c. him

d. she

While Susan prepared the for lunch,

her mother watched television

a. kitchen

b. children

c. chicken

d. cartoon

Although the plan has some

disadvantages, they are outweighed

My eyesight has really been deteriorating

lately I wonder if I need glasses I

think I’d better go to the _ to

have my eyes checked

a. pediatrician

b. dermatologist

c. optometrist

d. audiologist

We went to visit the pagodas, _

fascinating

a. Which were beautiful

b. Which were especially

c. Which were holy

d. Which were special

Where did Jane go after the birthday

party last night?

a. Yes, she did

b. Because she was tired

c. To her friend’s house

d. She went home around ten o’clock

Where did George go after the party last

night?

a. Yes, he did

b. Because he was tired

c. To Elaine’s place for another party

d. Around eleven o’clock

Your Turn

Complete the worksheet to find problems in multiple choice

question development.

Multiple Choice Editing

Multiple Choice Editing

Multiple Choice Editing

Questions: Paragraph

Ever since supermarkets first appeared , they

have been take over the world Supermarkets

have changed people’s l ifestyles , yet and at the same time, changes in people’s lifestyles

have encouraged the opening of

supermarkets As a result this , many small

stores have been forced out of business

Moreover , some small stores will be able to survive this unfavorable situation

GapFill/Cloze Multiple Choice

In years to come, zoos

will not (1) _ be

places where animals are

exhibited (2) _ the

public, but repositories

where rare species can

be saved from extinction

Vocabulary Multiple Choice

I disagree with a few of his opinions, but

Vocabulary Multiple Choice

Because fewer people are taking

expensive vacations, the tourist

Item Writing Practice

Further practice in designing multiple

choice test items will be provided in

specific language skill assessment

sessions