Luận văn a study on the construction of a marking scheme for end of semester english oral tests for 10th grade students at cam giang high school in hai duong

VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF LANGUAGES AND EVTERNATIONAL STUDIES FACULTY OF POSTGRADUATE STUDIES NGUYEN TH] OANH ASTUDY ON THE CONSTRUCTION OF A MARKING SCHEME FO

VIETNAM NATIONAL UNIVERSITY, HANOI

UNIVERSITY OF LANGUAGES AND EVTERNATIONAL STUDIES

FACULTY OF POSTGRADUATE STUDIES

NGUYEN TH] OANH

ASTUDY ON THE CONSTRUCTION OF A MARKING SCHEME FOR

END-OF-SEMESTER ENGLISH ORAL TESTS FOR 10 GRADE STUDENTS AT CAM GIANG HIGH SCHOOL IN HAI DUONG

Nghiên cứu xây dựng bảng đánh giá cho bài kiểm tra nói cuỗi kỳ môn

Tiêng Anh cho học sinh lớp 10 trường THPT Cẩm Giang, Hai Duwnp

M.A MINOR PROGRAMME TIIES

Field: English Teaching Methodology

Code: 60140111

Hanoi — 2017

VIETNAM NATIONAL UNIVERSITY, HANOI

UNIVERSITY OF LANGUAGES AND INTERNATIONAL STUDIES

FACULTY OF POSTGRADUATE STUDIES

NGUYEN TIT OANTI

ASTUDY ON THE CONSTRUCTION OF A MARKING SCHEME FOR

END-OF- SEMESTER ENGLISH ORAL TESTS FOR 10TH GRADE

STUDENTS AT CAM GIANG HIGH SCHOOL IN HAI DUONG

Nghiên cứu xây dựng bảng đánh giá cho bài kiểm tra nói cuỗi kỳ môn

'Tiêng Anh cho học sinh lớp 10 trường THPT Cẩm Giảng, Hải Dương

M.A MINOR PROGRAMME THESIS

Field: English Teaching Methodology Code: 60140111

Supervisor: Nguyễn Thị Ngọc Quỳnh, Ph.D

Hanoi - 2017

DECLARATION:

L hereby state that |, Nguyen ‘thi Oanh, declare the thesis entitled “A study on the construction of a marking scheme for the end-of-semester English oral tests for 10th grade students at Cam Giang High School in Hai Duong” is my own research for the Degree of Master of Arts at the Faculty of Post-Graduate Studies, University of

Languages and International Stucies- Vietnam National University, Hanoi This

thesis is the result of ray own research and efforts and it has not been submitted for

any degree at any other university or institution

Hanoi, 2017

Nguyen Thị Oanh

ACKNOWLEDGEMENTS

Hirst and foremost, | owe my deepest gratitude to my supervisor, Dr Nguyen ‘Thi

Ngoc Quynh for her encouragement, support, and guidance and for giving me constructive feedback throughout this thesis She has guided me in searching for relevant theory to my thesis and has also assisted in collecting data Consequently, 1

have learnt a lot about the assessment of English oral tests and construction of a

smarking scheme for oral tests

Secondly, I would like to thank Ms Bui Thien Sao, an expert of the Center for

Language Testing and Assessment of the University of Languages and Iernational

Studies for her invaluable assistance during the research time

Thirdly, this thesis would not have been possible without the enthusiastic

parlicipalion of six English teachers and 150 students at Cam Giang High School

where the research was carried out

Lastly, I would like to thank my family whose love and support help me complete

this thesis.

ABSTRACT

‘This thesis was conducted at Cam Giang Lligh School ‘the participants consisted of

150 tenth-prade students and six English teachers at Cam Giang High School Two experts in the Center for Language Testing and Assessment of the University of Languages and International Studies also participated in this study ‘Ihe research

aims at constructing a marking scheme for the end-of-semester English oral test of

tenth-grade students Constructing a marking scheme for oral tests is a complex process In this paper, a combination of three methods: intuitive, qualitative and

quantitative are employed by the researcher The research slarled with wriling the

dratt of marking scheme Then, the rescarcher cbtained the judgments from experts and other teachers Next, the marking scheme was piloted with 150 tenfh-prade

students Based on analyzing the students’ scores of the oral tesis, the researcher

examined how well the marking scheme works The findings revealed that the marking scheme can be used by the teachers effectively although there is still a need

for further investigation to improve and validate the current marking scheme The anajority of the teachers took a positive attitude toward the marking, scheme They believed in the efficacy of marking scheme in spite of some difficulties at the beginning and recommended the vontinuatiou of using (he marking scheme for lhe next school year Pedagogical implications and suggestions for future studies are drawn out based on the research findings

iii

1.3.2 Approach to construct a marking scheme

1.3.3 Sleps ía construcl a marlting sc]efte

1.3.4 Types of marking schemes

2.4.1 Rationale for using a multiple-rnethod approaeh seo

3.1 The finding from interviews with leachers and expGTfs cuoi 33 3.2 The finding Írorm seotes of studenis" oral t€sfs snsessesieiui7 PART C: CONCLUSION ses sensssesssusiestuseesistenenstneeneenoneeineasiaeieeneneen 3

APPENDIX 4B seovcteintestiseieeesseininstentisieanente sansa IX

U12 08T 1ẽa1a.4 ẻ.ẻốẻ°\A'

LIST OF ABBREVIATIONS

CEER: Common European Framework of Reference

MOET: Ministry of Hducation and ‘lraining

vi

TAST OF FABLES

'Table 1: Pearson Correlatiori oi Gramnar

Table 2: Pearson Correlation on vocabulary

Table 3: Pearson Correlation oi pronunoiation

‘Table 4: Pearson Correlation on fluency and coherence

Table 5: Pearson Correlation on Sum

vii

PART A: INTRODUCTION:

‘This part is offered to introduce the rationale of the study, the problem to be addressed in the study, the aims and objectives of the study, and the research questions to be answered It will also present the scope of the study, significance of the study, an overview of the employed methods and the design of the study

1 Rationale

English is the most popular official language in the world and the primary language

of global trade and commerce It is an intemational means of interaction and

communication in almost all countries Proficiency im English is seen as a desirable

goal for a lot of people in the world In many countries including Vietnam, English

is taught as a compulsory subject at school and it is included in many exams Cf all

four skills: reading, speaking, listening and writing: speaking is generally thought to

‘be the most important due to the fact that a lot of learners have spent years studying

English but they still cannot speak it fluently In order to speak a foreign language,

the learners must mastor the sound system of the language, use appropriaic vocabulary and be able to put words together intelligibly with minimal hesitation Moreover, they also need to understand what is being said to them and respond appropriatcly lo mainlain good-natured relation to achieve communication gual (Luoma, 2004) Although students’ speaking skills are often practiced and developed, it is not extensively assessed Comparing to other skills, speaking is the aost difficult language skill to assess the reliability The student’ speaking ability is usually judged during a face-to-face interaction, m real time between the teacher and the student Besides, the facters such as the nature of the interaction, the kind of

tasks, the questions asked, the topic raised and the opporlumty given students to

speak in Iinglish will all have an impact on the student's performance (Luoma,

2004) On assessing speaking skills, the teacher has to take on the role as an

interviewer and assessor at once, which puls him or her under a Jot of pressure This makes teachers hesitant to assess speaking and focus om assessing, other skills

instead (Rychtarik, 201 4) However, “if you waml to encourage oral ability, len test

oral ability” stated by Hughes (1989) Testing and assessment is a very important part of teaching process, which helps provide necessary information to the students

and leachers aboul the progress made and the work be done If assessment is carried

out aceurately and fairly, it will have a positive impact on both teachers and leamers

It helps learners define the aim of their learning and contributes to improve the

quality of caching

Perceiving the importance of testing oral proficiency, on September 9th, 2014, the

Ministry of Education and Training signed the decision No 5333/BGDT-GDTiII to

promulgate the applying of speaking test in the final term test (account from 20 %

to 30 % of the total score) with the 10th grade students (seven year program and ten-year program) and the 6th grade students (ten-year program) However, there is

no standard marking scheme for the teachers Lo follow Each teacher evaluates

students’ speaking based on the marking scheme he/she individually creates and

uses different criteria to assess The marking schemes produced by teachers consist

amostly or only numbers wilhoul descriplor or with very shorl descriplors such as

very good, good and bad Besides, other factors such as language level, gender,

status of the teacher, the famibarily between the teacher ard the studerits also affect

the evaluation student’s performance and the score that the students got In order to

overcome some of these problems, the researcher is fully aware of the need of

constructing a marking scheme for students’ speaking abilily Thorefore, a lot of

attempts have been made to do a study titled “A study on the construction of a marking scheme for end-of-semester English oral tests for 10th grade students at

Cam Giang High School in Hai Duong” Hopefully, the study, to which I will

devote all my efforts, will make a contribution to the Hnglish teaching and learning

of the teachers and students at Cam Giang Iligh School,

2 Aims and objectives of the study

‘The study is aimed at constructing a marking scheme for end-of-semester English oral tests for 10th grade students at Cam Giang High School in Hai Duong with hope to find an effective lool for assessing studenls’ English speaking compelanec

In order to achieve this aim, the following objectives will be addressed in the study: + ‘fo find out the marking criteria and descriptors of the target marking scheme

+ To investigate the (easibilily of the proposed marking scheme

3 Research question

The study aims at answering the following questions

1 What are the marking criteria of the marking scheme for end-of-semester

English oral test for 10" grade students at Cam Giang High School?

W What are the descriptors for each criterion of the marking scheme for end-of -semester English oral test Cor 10" grade students al Cain Giang High School?

4, Scope of the study

The scope of the research has been made quite clear from the title: “A study on the

construction of a markirys scheme for end-of-scmesicr English oral testis [or TÔ

grade students at Cam Giang Iligh School in [Iai Duong”

Firstly, the study only focuses on oral testing, namely the construction of a marking

scheme To be more specific, iL refers Lo (he process to find out the marking criteria

and descriptors for a marking scheme which helps teachers at Cam Giang High

School assess students? oral test This will help teachers al Cam Giang High School assess students’ speaking more accurately and objectively Secondly, the subject of

the study is restricted to tenth-grade students at Cam Giang Ligh School

5 Significance of the study

‘The study is of great significance to both the 10 grade students and the teachers at

Cam Giang Iligh Schoo! because it may have a great contribution to teaching and

learning speaking For teachers, the study helps them to find a usefid assessment

tool which ensures objectiveness and fairness A marking scheme is an assessment tool that clearly indicates marking criteria which let students know what is expected

of them and how to achieve aim It is hopeful thal the study will give students

opportunity to do self-assessment to reflect on the leaming process

6 Method of the stady

As the aim of the sludy lo construct a marking scheme (or end-of-scinester English

oral tests for 10th grade studonts at Cam Giang High School in Hai Duong, the

study is designed to use a combination of intuitive, qualitative and quantitative

mthods Besides, many sources such as books, newspapors and so

sources om

the internet have been read by the researcher ‘Ihe findings are reported based on the experts’ and the teachers’ responses in the interview and students’ scores on oral test Morcover, constant discussions with the supervisor arc of great signilicarice

7 Design of the study

The study is divided into three main parts:

Part T (introduction) includes the rationale, the auos and objectives, the scope, the

significance, the research questions, the method and the design of the study

Part II (Devetopment) consists of chapters as follows

Chapter 1 (Viterature review) presents the Uhworetical background of [he sludy and the review of the available marking schemes and previous studies

Chapter 2 (Methodology) describes in detail the research methadology which

consists of the corlext of the study, the mformation of the subjects, mstruments of

data collection, procedures of data collection and methods of data analysis

Chapter 3 (Results and Discussion) reporis the statistical results and the analysis of the data

Part [1 (Conclusion) closes the study by summarizing the whole study with ooneluding remarks and olferiry: some Tanitalions and suggestions for further

studies.

PART B: DEVELOPMENT CHAPTER 1: LITERATURE REVIEW

This chapter provides an overview of the literature on the field of communicative competence, assessing speaking and marking scheme It discusses conununicative competence, communicative competence in the CLI'R, definitions of speaking,

assessing speaking, definitions of marking scheme, approaches to construct a

amarking scheme, types of marking scheme and the structure of a marking scheme A review of the available marking schemes and previous studies is also presented

1.1 Communicative competence

During the past few years, the concept of communicative competence has been discussed and redefined by many researchers and authors ‘The term

“communicative competence” was introduced by Hymes (1972) and he emphasized

that language consisted of a wider range of competence while Chomsky (1965) just

concemed about grammatical competence Ilymes stated that language can only be understood if the rules for grammar, speech acts and discourse are analyzed in relation to the speech community and the context (Hymes, 1972) According to Tuoma (2004) communicative competence focuses on language uscr’s means of communicating, whic may affecl the choice of more authentic learting material and communicative tasks as learning material in language classrooms

Canale and Swain (1980) proposed one of the first theoretical models of communivative competence The model distinguishes between communicative

competence and commumicalive performance, as communicative compelence is

kmowledge about grammar, sociolinguistic knowledge, strategic competence, while communicative performance is the actual communication Ilowever, Canale (1983) revised this model and used the term “actual communication” mstead of

“performance” He alsa

sserled that: “Communicative competence refers 10 both

kmawledge and skill in using the knowledge when interacting in actual

communication” (Canale, 1983, p 5)

Bachman and Palmer (1996) developed another model in communicative

competence and language testing: the model of communicative language ability The model focuses more on the inleraction betweon context and langusgc use

(Fulcher and Davidson, 2007) The term “language ability” consists of language

imowledge and strategic competence Language knowledge inclides both organizational knowledge (grammatical and textual knowledge) and pragmatic knowledge (illocutionary competence and sociolinguistic compelence) Strategic

competence involves the ability to assess whether the situation is practicable and to

plan for the next movement (Luoma, 2004} Bachman and Palmer (1996) claimed

that their model can be used as a checklist for developing language tests

LI) Communicative competence in the CEFR

The CLIR is the Common Duropean Framework for language leaming, teaching

and assessment, which focuses on the nature of language use and the language user

and the implications for learning and teaching (Council of Hurope, 2001) ‘The Common European Framework defines levels of proficiency which allow learners’

progress 1o be measured al each slage of learning and provides crileria for assessing

four English skills: reading, speaking, listoning and writing Of all skills, speaking

is very important and within communication the learners can lean and apply various skills The Framework consist of three proficiency levels which are referred

to respectively as Basic User: Al, A2, Independent User: Bl, B2, and Proficiency

User: Cl, C2 (Council of Europe, 2001)

Communicative competence in the narrower sense consists of three components, lmguistic competences, sociolmguistic competence and pragmatic competences

(Council of Europe, 2001, p 108) In the model of communicative competence of Canale and Swain (1980), there are also three components, grarmmnatical competence, sociolinguistic competence and strategic competence Although the CEFR and Canale and Swain label these categories slightly differently, they use similar categories to describe competences In the following paper, the categories from the CLR are particular relevance for the assessment of speaking will be discussed.

Linguistic competence is considered the core of the model of communicative competence, Linguistic competence is divided into lexical competence, grammatical

competence, semantic compelence, phonological competence and orthnepic

competence (Council of Europe, 2001, p.109)

Lexical competence is defined as “knowledge of, and ability to use, the vocabulary

of a language, consisis of lexical clements and grannnatical clements” (Council of

Europe, 2001, p.110) Lexical element inckudes fixed expression and single word forms to enhance language on the different levels of meaning The CEFR also

presents illustrative scale for the range of vocabulary knowledge and the ability to

control that knowledge to specify these competences (Couneil of Llurope, 2001,

pli)

Grammalical competence is deseribad ax “knowledge of, and abilily lo use, the

grammatical resources of a language” (Council of Lurope, 2001, p 112)

Grammatical competence is the ability to understand and express meammg by

producing and recognizing well-formed phrased and sentences (Counvil of Europe,

2001) ‘lo measure grammatical competence, the CEFR has developed an

iflustralive scale demonstrating levels of grammatical accuracy (Comeil of Europe,

2001, p 114) Grammatical competence also includes morphology and syntax

which the learners need to be aware of Morphology deals with the organization of

words and the ways lo modifying words forms and syntax deals with the

organization of words into meaningful sentences (Council of Kurope, 2001)

Semantic competence is “the leamer’s awareness and control of the organization of

meaning” (Council of Europe, 2001, p 115}

Phonological competence involves the knowledge of the sound-units, words stress, sentences stress, sentence rhyme and intonation (Council of Lurape, 2001, p 116)

Onthoepic competence involve knowledge of spelling conventions, abihty to consul

a dictionary, knowledge of the implication of written form for phrasing and intonation and ability to solve ambiguity in various context (Council of Europe,

2001, pp: 117-118)

Sociolinguistic competence is “concerned with the knowledge and skill required to deal with the social dimension of language use” (Council of Hurope, 2001, p 118)

Sociolinguishe compelen

8 elude linguistic markers of social relations,

politeness conventions, expressions of folk-wisdom, register differences, and dialect and accent (Council of Europe, 2001, p 119) Canale and Swain defined

sociolinguistic compelence as “the abilily to communivate appropriately ina variety

of contexts, this includes both verbal and non-verbal comurunication” (Canale and

Swain, 1980)

Linguistic markers of social relations vary frora language to language They include

use of choice of greeting, use and choice of address forms, conventions for tum-taking in conversations and the use of choice of expletive (Council of Europe,

2001, p 119)

Politeness conventions vary from one culture to another and are a frequent source of

inter-ethnic misunderstanding when polite expressions are literally interpreted

(Council of Rurope, 2001, p 119)

Expressions of folk wisdom are fixed formulas about daily life, often used in

newspaper headlines The expressions include proverbs, idioms, and expressions for beliefs, attitudes and values and are often used in graffiti and on T-shut slogans

(Council of Lurope, 2001, p 120)

Register differences refer to “systematic dilTerences between variclios of language

used in different contexts” (Council of Kurope, 2001, p 120) Register differences

express differences in level of formality: frozen, formal, neutral, informal, familiar and intimate (Council of Europe, 2001, p 120)

Dialects and accents perform people’s origin, sociolinguistic competences inckade the ability to distinguish between various social classes, regional provenances, national

origins, ethnicities and occupational groups (Council of Furope, 2001, p 1213

Pragmatic competences include discourse competence and functional competence

“Discourse competence is the ability of a user/ leamer to arrange sentences in sequences so as 1 produce coherent sirciches of language” (Couneil of Europe,

2001.p 123) The CEFR has included illustrative scale with aspects of discourse competence: [lexibility to circumstances, tumn-taking in interactions, thematic

development and coherence and cohesion (Council of Europe, 2001, pp: 123-125) Functional competence is “concemed with the use of spoken discourse and written texts in communication for particular purpose” (Council of Europe, 2001, p 125) Functional compolonce also includes knowledge and ability to use the schema

pattems of social interaction) which underline communication, such as verbal

exchange pattems (Council of Burope, 2001, p 126) The CEFR developed an

Ulusiralive scale for two qualilaiive aspects, Muency and propositional precision

Fluency is “the ability to articulate, to keep going and to cope when one lands in a dead end”, Propositional precision refers to “the ability to formulate thoughts and

propositions so as lo make one’s meamnng clear” (Council of Furope, 2001, pp

Imowledge he or she has about the language therefore it is not easy to be mastered

Bygale (1987) considers learner's speech as a process, speaking is a “real-time”

action because the leamer has to plan, process and produce the language simultaneously The speech process includes planning, selection and production of

speech Planning is an interactive process which requires leamer to have knowledge

about interaction routines to plan the next step of the conversation Selection is the

stage in which the leaner uses knowledge of language and grammar to decide how

1o express oneself Tn the production stage, the leamer uses the knowledge about

pronunciation and communication strategies to produce language (Bygate, 1987)

The CEFR has distinguished clearly between interaction (spontaneous) and prodnelion (prepared) of language Tulcraction activilies arc mairily sponlancous and

are camicd out throughout conversation and more or less informal discussion Production activities are mainly prepared and rehearsed in advance (Council of Furope, 2001, p 178)

1.2.1 Assessing speaking

The categories for oral assessent is enormous and deciding which criteria to use for assessment is a relatively dificult work Assessor should determine the most appropriate criteria that could not only be used to assess students’ speaking, but also

be relevant ta the objective of the course/ lesson, ete (Knight , 1992)

The CBER ch siỗes 12 quallalve calegorlss relevaml lo oral assessment The

CIR has also developed illustrative scales for assessment and each scale describes the level of proficiency There are a number of categories relevant to assess

speaking such as tum-taking stralegics, co-operating stralegics, asking for

clarification, fluency, flexibility, coherence, thematic development, precision,

sociolinguistic competence, general range, vocabulary range, grammatical accuracy,

vovabulary control and phonological control (Counvil of Europe, 2001) However, it

is impossible to assess all criteria simultaneously, The assessers need to make

choices for each assessment silualion and select only several criteria thal are relevant to the particular context Choosing no more than 4 er 5 criteria in cach

testing situation guarantees feasibility as well as reliability of the assessment

(Council of Furope, 2001, pp: 192-193)

1.3 Marking scheme

13.1 What is a marking scheme?

A marking scheme is sometimes referred to as a scoring rubric or a rating scale

which is defined as an explicit set of criteria used for assessing a particular type of work or performance

As Davied, Brown, Elder, Hill, Tuumly, McNamara (1999) defined it

A rating scale is a scale for the description of language

proficiency consisting of a series of constructed levels against which a language learner's performance ix judged Like a test, a

10

proficiency scale provides an operational definition of a

Linguistic construct such as proficiency Bypically such scales

range from sero mastery (hrough to an end-point representing

the well-educated native speaker The levels or bands are commonly characterized in terms of what subjects can do with

the language und their mastery of linguistic features (such as

vocabulary, syntax, fluency and cohesion) Soales are descriptions of groups of typically occurring behaviors, they ave

not in themselves test instruments and need io be used in

conjunction with tests appropriate to the population and test

purpose Raters or judges are normally trained in the use of

proficiency scales so as lo ensure the measure’s reliability (David

etal, 1999, pp: 153-154)

From above definition, it can be seen that a rating scale includes both the fields to

‘be assessed (construct) and the aligmmcul between examner’s performance and the

predetermined levels of behavior descriptions Thus, it is important to consider two

above components when construcling a rating scale Tn addition, for different

purposes the construction of scale can be different, Alderson (1991) and Pellitt and Murray (1996) classify different purposes that rating scales could serve

user-onented, consiructor-oniented and assessor-oriented A user-oriented scale is

“designed to communicate information about typical of likely test taker behaviors at

a given level” (Taylor, 2011, p 190) Constructor-oriented scale “guides test writer

in their choice of tasks to include in a test” (Taylor,

ale mush he proritived to

Tn the present paper, the researcher focuses on constructing an assessor-orienled

1

speaking marking scheme which teachers at Cam Giang high school can utilize to

assess students’ oral tests

1.3.2 Approach to construct a marking scheme

Marking scheme construction is recognized to be a complex process (Brindley,

1998, Fulcher, 2003, North, 2000) Traditionally, the design and construction of

vating scales used a priori approach in which assossment erileria and rating scale

descriptors are developed by “experts” using their own experience and intuitive judgment ( Tulcher, 2003) McNamara (1996) states that marking schemes were

constructed based on the construction of the firsk scale for the Foreign Service

Anstitute’s Oral Proficiency Interview in the 1950s In 1990s, many authors

supported empirically based approach involving analyzing samples of actual

language performanee fo construcl criteria and marking scheme descriptors (Fulcher,

1996; Milanovic, Saville, Pollitt and Cook, 1996, Shohamy, 1990, Upshur and

Tumer, 1995) Fulcher (2003) discusses two basic approaches to rating — scale development: inluitive approaches and cmpirical approaches Intuitive maelhoda

primarily rely on “expert judgment and the principled interpretation of experience”

(Council of Rurope, 2001, p 208) Fulcher also introduced three common subcategorics of an mtuitive method as expert judgment, committee and experiential

‘The expert judgment and committee are alike in which the experts make decisions

upon issues such as the number of levels and the wording af deseriplors based on

existing scales, curriculum, course material and other necessary sources Lixperiential evolves on the basis of the expert judgment and committee by revising

the scale after a further understanding on both the content of existing scale and

sample performance (Fulcher, 2003) Kmpirical methods include data-based, data -driven scale development, empirically derived, binary — choice, boundary

definition scales and s alii descriplors (Fulcher, 2003) More recently, together

with scale development for the Common Luropean Iramework of Reference (CEFR), the mixture of qualitative and quantitative methods makes complementary

coniribuliom to rating scale devclopment, (Counail of Rurope, 2001) Quantitative

methods rely on statistical analyses and careful interpretation of results while qualitative methods involve interpretation of information obtained

Thore are a number of approaches lo marking scheme construction The bosl

methods for marking scheme construction are said to combine all three approaches including intuitive, quantitative and qualitative approaches in “a complementary

and cumulative process” (Council of Europe 2001, 207) Thercfore, in constructing

a marking scheme for assessing student 3° oral test at Cam Giang High School, a combination of intuitive, qualitative and quantitative approaches was used

1.3.3 Steps to construct a marking scheme

According to Mertler (2001) building a marking scheme includes following seven steps: Step 1: Re-examine the learning objectives to be addressed by the task This

helps {o match the teacher’s scoring gurde with objectives and actual

instruction

Step 2: Identify specific observable atwibutes that the students demonstrate in

their product, process, or performance

Step 3: Brainstorm characteristics that describe each attribute Identify ways

to describe above average, average, and below average performance for

cach observable attribute identified in Step 2

Step 4a: Vor holistic rubrics, write thorough narrative descriptions for

excellenl work and poor work incorporating each aliribule into the

description Describe the highest and lowest levels of performance

combining the descriptors for all attributes

Step 4b: For analytic rubrics, write thorough narrative descriptions for

excellent work and poor work for each individual attribute Describe the

highest and lowest levels of performance using the descriptors for each

Step 5b: For analytic rubrics, complete the rubric by describing other levels on the continuum that ranges from excellent to poor work for each attribute Write

deseriptions for all mater mediate levels of performance for cach atlribute

separately

Step 6: Collect samples of student work that exemplify each level These will

help the examiners score in the [uture by serving as benchmarks

Step 7: Revise the rubric, as necessary Be prepared to reflect on the

effectiveness of the rubric and revise it prior to its next implementation

(Mertler, 2001)

Meanwhile, Nakutsuhara (2007) suggests developing a marking scheme

according to four stages:

Stage 1: Reviewing oxisting speaking raling scales outside and iuside Japan Lo

collect marking categories and descriptors to be referenced in the later stage

Stage 2: Examining the course of the study (the guideline of secondary schoal

education) to decide types and levels of marking calegories lor the target

population

Stage 3: Drafling a raling scale based on the existing raling scales, while

obtaiming cxpert Jadsrucnts from cight cxpericnced upper-sccondary school

teachers

Stage 4: Piloting the scale with 42 Japaneso uppor- sccondary students with

two raters, to examine how well the resulting rating scale functions

(Nakutsuhara, 2007)

Taylor (2001) proposes the process of constructing a marking scheme is took place

in three phases as outline below:

Phase 1: Intuitive

The marking scheme is constructed by the researcher The researcher carries onl

according to following steps:

| Review the existing scale in the world and Vietnam, teaching materials,

cumiculum, objectives of the course and relevant source materials

44

+ Propose the eriteria

+ Determine the number of scales

+Develop the descriptors

+Discuss with other teachers and revise

+Trail the scale

+ Stabilize the scale > Version 1

Phase 2: Qualitative

Tixtemal expert reviewing the existing scale

| Rank the descriptors in order of difficulty

+ ‘Trail the scale

| Discuss and revise the scale

+ Stabilize he scale +> Version 2

Phase 3: Quantitative

Standard —setting phase

+ Trail the raters

+ ‘Trail the scale

I Analyze the scares

+ Stabilize the seale -> version 3 (Taylor, 2011, p 195)

In this research, the researcher follows a three - phase process suggested by Taylor

(2011) for some reasons Firstly according to the CEFR (2001, p 207) the besL

snethods for rating scale development are said to take advantage of the strengths of

a range of intuitive, quantitative and qualitative approaches The best scale, the CEFR (2001, p 207) suggests, combine all three approaches in a “complementary and cumulative process” Secondly, the phases are logical and detailed ‘They are clear for the researcher to follow In each phrase, steps are described specifically

therefore iL helps the researcher have an outline to construct a marking scheme

1.3.4 Types of marking schemes

There are many divisions of individual types of marking scheme as there are many

authors dealing wilh this issue According to “Rubric [or assersment” of Norfherm

Illinois University there are several types of rubrics including holistic, analytical,

general, and task specific Limothy Famsworth (2014) of CUNY Llunter College

divided rubric info five types: holistic, analytic, task Fulfillment, performance

decision trees and checklist However, according to Mertler (2001) and Taylor (2011) assessment criteria used to evaluate L2 spoken language performance

gencrally fall into hyo main categories: holistic and analylic Tn this study, the

researcher will focus on describing some advantages and disadvantages of these two

types

A holistic marking scheme requires the raler lo make an “tmpressionistic

assessment” of the quality of students’ oral test using a single marking scheme

(Davies, Brown, Elder, Hill, Lumley and Mc-Namar, 1999, p 75) A holistic

marking scheme is commonly used in assessing 1.2 speaking and it is described as

“general impression marking” in which the overall properties are more important

than particular features of the performance (Association of Language Testers in

Furope, 1998, p 147) Using holistic marking schemes can be relatively quicker

than analytic marking schemes (Nitko, 2001) However, the holistic marking

schemes do not provide detail information on student performance [or each criterion and they are not very useful to help plan instruction because they lack a detail

analysis the strengths and weakness of student’s product (Taylor, 2011, p 178)

An amalylic marking scheme gives separate scores for cach criterion of performance

Each criterion is assessed separately and using different descriptive marking scheme (Taylor, 201], p 179) Analytic marking scheme xesults initially in many scores,

followed by a summary total score representing an assessment on a

snultidensional level (Mertler, 2001) Using an aualytic marking scheme helps to focus rater judgments more narrowly which contribute fo rater agreement and rating

reliability (Weir, 1990) Sconny tends to be more consistent across sludents and

cause the scoring process to be slower because the teacher has to provide scparate assessment for each of a number of aspects of performance Moreover, construction

of an analylie marking scheme can be quite more difficult and Gime-consuming (Martler, 2001)

As mentioned above, compared to a holistic marking scheme, an analytic marking scheme gives bolh teachers as well as studenis a more precise [cedback on wheal students do his helps the students recognize in detail the good points and weak points of their own and find way to improve In the present study, the researcher chooses lo construct an analylic marking scheme which Cam Giang High School teachers can utilize to assess speaking competence of students in their end -of-

semester oral test

1.3.5 Structure of a marking scheme

teri ‘Excellent Good Poor, Criteria 3 2 1 V

‘Bumber ofvources Texto twelve Five tone Ghz to four Fisiorieai accuracy Ni apparent Few inarraracies | Tats af historical

anazcumacies anazewsacies Drganication Can casily rel am | Can tell with Cannot rll oir

which sources difficulty Lom where | which source mfoumaticn was | information came — | anfounstticn care dravat

‘Bibliography Allselevant Biblicgrapay BibGogaphy

‘bibliographic contains mast cartains very litle iufonunticn is relent infomation | innmtien

marking scheme offen meludes four basic componerls: entena, levels of

performance, scores and descriptors (Northern Llinois University, 2014, p 1)

47

Criteria “identify the trait, feature or dunension which is to be measured and include

a definition and example to clarify the meaning of each trait beg assessed”

(Northern Tlhinois University, 2014, p 2) The number of cntena will be decided

depending on tasks, assignments or students’ performance (Northern Illimois

University, 2014, p 2)

Tevels of performances are often adjectives which describe the performance levels For example, the adjectives are often used such as superior, moderate, poor, below average Levels of performance clarify the degree of performance which the

learners need to achieve and provide for consistent and objective assessment, and

better feedback to students (Northern Ulinois University, 2014 p 2)

Score are often combined with levels of performance The teacher should consider how many points are needed to adequately describe the Tange of performance he expects to see in student’s work For example, 0,1,2,3,1 (Northern Illinois

University, 2014, p 2)

Descriptor

derived and what is expected of the students” (Northem Llinois University, 2014, p

"are explicil deseriptions of the performance and show how the score is

2) Descriptors explain clearly each level of performance for each criterion and deseribe in detail the charactenstic of levels of performance, Descriptors also help

the teacher distinguish level of performance of each student’s work “he same

deseripiors can be used for different eniteria within one marking scheme” (Northern

Llinois University, 2014 pp 3-4)

1.3.6 Available speaking marking schemes

There are a number of performance-based rating scales which have been produced

Beginning with the US Foreign Service Institute (SI) scale which was introduced

in the early 1950s and the scale ranged from no proficiency to native speaker ability

The assessment crileria include accent, fluency, comprehension, vocabulary and

grammar, Afterwards, many other language proficiency tests started to apply that xating scale to their scoring such as American Council on the Teaching of Foreign Languages (ACTFL) (ACTFL, 1986) Tr recent year, some prevailing language

18

assessments also witness the cmployment of as IELTS Oral Test and Spoken

linghsh of Test of TOLL In 2001, the Common Luropean lramework of

Releres

Tor Languages: learning, teaching, assessment (CEFR) developed by the Couneil of Europe (COE) rapidly became the most influential scale as the standard reference document for teaching and testing languages in Durope (Council of Burope, 2001, Fulcher, 2004) The fonnewark consisis of six levels: AT (Breakthrough), A2 (Waystage), BI (threshold), B2 (Vantage), Cl (Effective Operational Proficiency) and C2 (Mastery), and each level is associated with a set

of descriptors TELTS and ESOT are examples of the same framework Since the CUR intends to provide a whole set of reference tools that should be selectively employed according to the context, there are a number of categories relevant to oral

agscasmment, such as dan-laking sfraiegies, co-operating strategies, asking for

clarification, fluency, flexibility, sociolinguistic competence, general range, vocabulary range, grammatical accuracy, vocabulary control, phonological control and go on The application of the CEFR is also exlending to outside Europe Although there are few official documents available, there is same evidence about the growmg extension and a great polential of the CEFR throughout the world,

xespectively as Sơ cp (Klementary level = CHFR basic user), Trang cấp

(Intermediate Level = CEFR Independent User) and Cao cấp (Advanced level = CRFR proficiency User) Each level is broken down into two sublevels, making ita 6-level proficiency framework In order to oriented users in the educational system

to some practical purposes, VNELPF provides a list of descriptors of each profivioney level and prasonia detailed stalcments of whal the lenrer can do al wach

49

proficiency level in term of listening, speaking, reading and writing

VSTLP (Vietnamese Standardized Test of Unglish Proficiency) is a test of the universily {0 as: from level 3 to 5 (eqmvalent of CRER

level B1, B2 and C1) according to VNFLPF ‘This is Vietnam’s first test of Hnglish

language competen

proficiency assessment developed basing on principles of test assessment

construction process of ALTE The VSTEP spoaking test is an interview format test

and the VSIKP rating scale for oral assessment has 5 marking categories and subcategories for each category 1) Grammar: range and acouracy, 2) Vocabulary:

yange and control, 3) Pronunciation: individual sounds, stress and intonation, 4)

Fluency: hesitation, extended speech, 5) Discourse management: thematic development, coherence and cohesion with ten levels from 0 to 9 The criteria of valing scale were derived from the performance of VSTEP’s speaking tost The

descriptors seem greatly useful for raters’ understanding of the scale The scale is

particularly remarkable as this is one of the meaningful rating scales in Vietnam

developed lo assess studer’s speaking compeloros Besides, in 2013, the Minister

of Education and ‘Training deployed a test to survey language competence level 2

for students who have Ginished grade 9 participale in Pilot English Curriculum for Victnamese Upper Secondary Schools Speaking section assessment criteria was

developed basing on ILS speaking criteria It was a holistic rating scale including

4 calogories: Flusncy and coherence, lexical resource, grammatical range and

accuracy, and pronunciation

In short, there seem to be well developed rating scale outside and inside Vietnam In

the construction of rating scales, it seems meaningful to selectively use the CEFR as

a reference and modify the descriptors and mumber of levels according to the target context as recommended by Council of Burope (2011) Furthermore, speaking section assessmenl crleria by MOET also serves as a useful basic when constructing a marking scheme for tenth-grade students’ oral test af Cam Giang High School,

1.3.7 Previous studies

There have been some researchers who studied about constructing a matking

scheme for oral tests

Fulcher (1993) cared out his rescarch entitle “The Construction and Validation of

Rating Scales for Oral Tests in English as a Foreign Language” In his research, he investigated the principles upon which rating scale in oral testing are constructed

and used, and the subsequent claims of reliability and validity made for thom He

designed a fluency rating scale based on a larger data-based originally built for research into scale design for a doctoral dissertation

Nakatsubara (2007) conducted the study on “Developing a Rating Scale in Assess

linglish Speaking Skills of Japanese Upper-secondary Students” The study was motivated by the fact that there was no standard scale to rate the speaking of Japanese

upper-sccondary students The work was divided into four stages: studying available

zating scale and their practices and outside Japan, examining relevant levels of analytic

marking categories, drafting a rating scale with expert advice of eight qualified ESL

teachers, and rumming the scale on forty- wo students with two raters

Raza (2008) did the research on “A rating scale to assess English speaking

proficiency of universily students in Pakistan” He proposed! the much needed rating scale to assess English speaking proficiency of students graduating in 124

universities in Pakistan Like Nakatsuhara, he also described the steps to develop a

rating scale 1o asseas English speaking achievement of Pakislan univarsily students

Summary: On the whole, in this chapter, theoretical framework concerning communicative competence, assessing oral test, and marking scheme has been reviewed These theories will serve as the foundation for the author to form and conduct the study according to specific methodology that will be elaborated in the

next chapter.

CHAPTER 2: METHODOLOGY

The research method plays a vilal role in any study sinec il determines the reliability and validity of the study, This study has omployed a combination of three methods including intuitive, qualitative and quantitative to find the answer to the research questions This chapter presents setting of the study, participants of the study and description of the end-of-semester oral test This chapter also discusses the reasons for the choice of methods employed in this study and the procedure of the research The next parts in this chapter are te present the instruments used to collect and analyze data

2.1 Setting of the study

The study was conducted at Cam Giang Iligh School in Ilai Duong The school has

a history of 50 years of development since its foundation in 1966 Up to now, it has

30 classes of livee grades 10, 11 and 12 The average number of studenls in cach

class is 40 students and the classroom is poorly equipped only with a blackboard and 24 small unmovable desks each of which is shared by 2 students ‘here are seven English teachers All of them are enthusiastic and dedicated with work The school is considered one of the center schools of District At school, English is oonsidered as one of the most important subjects in training the studenls and the

students have four lessons of English a weck:

4\t the time this research was conducted, the students were in the second term of the academic year which covered the last 7 units of English text book 10

Material

The current teaching material is “Tieng Anh 10° (seven-year program) published by

Education Editer, Ministry of Education and Traming The English textbook 10

(seven-year program) inchides 16 teaching units and 6 review units called ‘lest yourself ach teaching unit covers a topic and is structured into 5 sections: reading, speaking, listening, writing and language focus The speaking section includes

aclivilies, termed “lasks” relaled la topics of each unit, The Grst and second

activilies provide language input and develop specific language functions such as expressing opinion, agreement and disagreement ‘I'he rest involves short talks on a specific topic with or without prompts l3y the end of the course, students are able to ask and answer about the topics covered They can perform some basic language

tạ 8

functions such as giving instructions, expressing opinions, asking for directions, asking and giving information, etc (General objectives to achieve in Grade 10 for speaking skills (MOLT 2006: 19-25)

2.2 Participants

Participants in this study include students, seven teachers and two experts

Students: 150 tenth grade students at Cam Giang High School were selected to take part in the pilot testing of the marking scheme They were tested for their oral proficiency They were provided with comprehensive instructions about the topic they had to accomplish Their oral tests were tape-recorded for subsequence scoring

Their ages are from 16 lo 17 They have passed an exam in English, Maths and Literature to euler Cam Giang High School All of them are Vicinamese nalive speakers and they started to Jeam English five years ago Their English knowledge was just at the elementary level and some even are at lower level Generally, students at this level can ask and answer about personal preferences, daily routines, common events, and other personal Lopics

Teachers: Seven teachers, including the researcher, who are currently leaching English at Cam Giang High School participate in this study Their teaching experience ranges from 4 years to 21 years All of them have good Lnglish competence and they are regularly involved in student instruction and assessment of

performances All the seven teachers are female Here are all teachers’ profiles

university experience according to

graduation (No of year) | CHFR

Exports: Two cxpeds, who arc working at the Center for Language Testing and Assessment, University of Languages and Intemational Studies participated in the

atiudy They have

<pericnee 1n constructing Loss and marking schemes for many years

2.3 Description of the end-of- semester oral lest

All the students at Cam Giang High School take an oral English test at the end of

gach semester The inpies are constructed based on Ihe same topies as the Lextboak

Mt the examination, the students are required to introduce themselves and then choose randomly one of the prepared topics Each topic is followed by some

guiding questions which students can base on 40 outhne their oral tests They have 5

minutes for preparation and then from 3 to 5 minutes to talk (See appendix for topic cards}

2.4 Research design

2.41 Rationale for using a multiple-method approach

This study is an attempt to construct a marking scheme for the end-of-semester oral test

for tenth grade students af Cam Giang High School Therolore, il is besl assisted by

combining all three methods intuitive, qualitative and quantitative A distinction has

been made between intuitively and empirically developed tating scales Tutuilively ancthods primarily rely on expert judgment and the “principle interpretation of

experience” Empirical methods, by contrast, are data-driven and based on actual

leamer performances Finpirical scale development mathods may be subdivided into

“quantitative” and “qualitative” based on the type of data they draw on

Fulcher (1996) and others (e.g Brindley 1998, Council of Europe 2001; Knoch

2009a, Turner and Upshur 2002) have argued that the intuitive design of many

assessment scales, based exclusively on expert judgment, is compromised by the lack of empirical data on the features of learner speech or writing Recognition of

the limitations of inluilively derived assessment scales has encouraged more

development are said to take advantage of the strengths of a range of intuitive,

quantitative and qualitative approaches

2.4.2 Research procedure

According to ‘Taylor (2011) the process of constructing a marking scheme 1s took

place in three phases: intuitive phase, qualitative phase and quantitative phase In

this study, the researcher follows these above phases However, some minor steps in

each phase are modified to make it appropriate for the specific context of the researcher The intuitive phase 1s the first stage in which the researcher reviews the existing scale in the world and in Vietnam to produce a reference collection of marking criteria and descriptors ‘leaching materials, objectives of the course and relevant source materials are also reviewed to propose the first draft of the marking

scheme Alicr that the researcher obtains judgments from six experienced high

school teachers In qualitative phase, the marking scheme is submitted to the experts

to review and refine the marking scheme, which ensiwes the internal coherence of the markmg scheme The judgments are collected from both the teachers and

experts through interviews Quantitative phase analyzes the scores of students” oral

tesis lo confirm the soundness of the marking schemes, assessment crileria and deseriptors as a whole in live tests

Phase 1: Intuitive phase

Firslly, the researcher determines the evalualion criteria Cor her marking scheme

Based on communicative language ability framework of Canale & Swain (1980) and Bachman and Palmer (1996), the CETR, and rating categories collected from

the Speaking section assessment criteria, it is plausible to choose four analytic

smarking categories for the purpose of this rating scale: 1 Grammar, 2) Vocabulary, 3) Pronunciation, 4) Fluency and coherence The choice of four marking categories

is made because they are based on speaking section a men criteria in the test

ask and answer about the topics covered and perform some basic language functions such as giving instructions, expressing opinions, asking for direction, asking and

giving information, cfc That is equivalent level A2 in the CEFR The CEFR has

developed :llustrative scales for these vategories as follows:

Grammar refers to the accurate and appropriate use of a range of grammatical forms

(Taylor, 2011) Students on proficiency level A2 use some simple structures

courectly, but still systematically makes basic mistakes- for example tend to mix up tenses and forget to mark agreement, nevertheless, it is usually clear what he/she is

trying Lo say” (Council of Furepe, 2001, p 123)

Vocabulary refers to the candidate's ability to use a range of vocabulary to meet task requirement, At level A2 student are expected to “have sufficient vocabulary to

oonduct routine, everyday transaclions involving familiar silualions and topics” and

(Council of Europe, 2001, p 121),

Pronunciation refers to student's ability to produce individual sounds and to link

words together, as well as using stress and intonation Lo communicale meaning

(Thombury, 2005, p 128) Pronunciation in the CEFR is included in an illustrative

scale for phonological control and on profierency level A2 Stiders are expected to

“pronounce clearly enough to be understood despite a noticcable foreign ascent, but

conversational partners will need to ask for repetition from time to time” (Council

of Eurupe, 2001, p 126)

Fluency and coherence refer to student’s ability to talk with nomal levels of continuity, speech rate and effort and to link ideas and language together in coherence, connected

speech and formal structure (MOET, 2014, p 190) A student on proficiency level A2

“can make himself/herself understood im short contributions, even though pauses, false starts and reformulation are very evident” and “can use the most frequently occurring

connectors to lnk sitaple senleuces im order to tell a story or describe something as a

simple list of points” ( Council of Europe, 2001, pp: 134-138)

Secondly, in order to meet the objective of the course, it was decided that all calogories should focus more on success in conveyig a message and

communication rather than language accuracy Therefore, the categories arc not

equally important Some categories account for higher percentage in comparison to

the olhers According to the speaking scclion assessment, criteria, Vocabulary and Fluency and Coherence are more important than Grammar and Promuneiation Therefore, the marking categories were described in the following weighting: Flucney and coherence marked oul of 10 then multiplied by 3 Vocabulary marked out of 10 then multiplied by 3, Grammar marked out of 10 then multiplied by 2 Pronunciation marked out of 10 then multiplied by 2 (MOLT, 2014, p.190)

Thirdly, greal care was taken Lo decide the number of levels, since the number of levels should be adequate to capture and discriminate different ievels of speaking ability, but should not exceed the number of levels that busy school teachers are, wilh limited trainiryg, capable of consistently distinguishing (Nakalsuhera, 2007) The researcher decided to have five levels in each marking categories including poor, bad, average, good and excellent with the score from 0 to 4 respectively The mark which students get will be caloulated according lo following formula

‘The student’ score = (Grammar x 2+ Vocabulary x 3+ Pronunciation x 2+ Kluency and Coherence x 3) / 20

The maximum score the student will get is 2 becanse the oval Iost in the final term

test accounts for 20 % of the total score

Next, the marking scheme was drafied mainly with reference of CEFR The categories were dratted by stretching the CEFR level Al and level A2 into 4 levels bad, average, good and excellent and level 0 was the lowest level describing no English compelencs, The wording for each level was adapled from the CEFR according to the requirement of the task In order to explore the validity of the marking scheme, the draft marking scheme version was distributed to 6 teachers for judgments They were asked to answer the inerviow question which invited their comments on the marking scheme In general, the gulding interview questions focused on some aspects: criteria, levels of performance, descriptors and general look about marking scheme

rey a

Phase 2: Qualitative

After the completion of the first draft, the marking scheme continued to be given to

two experts for review The lwo experts were also invited to give comments on lhe

smarking scheme through the interview ‘he objective of this process is to ensure the internal coherence of the scale, wording them consistently and efficiently to make

them usable for examiners during the process of giving assessment

Phase 3: Quantitative

As the last phase of this study, the resulting marking scheme was pilot-tesied, 150

tenth-grade students participated mm a “mock” speaking test and all thei oral Iesls

were audio recorded ‘The recording took place at Cam Giang Ligh School ‘Iwo experts, three teachers and the researcher participated in the marking The students

were told before the lesl about a given topic Tn the Lesl, students had from 1 lo 2

minutes to introduce themselves and 3 minutes to perform their topic Before

assessing the audio-recorded students’ performance, approximately 1 hour rater

training was admmistored The assessing trairong was conducted as [ollows Firstly, the three teachers and researchers had a short discussion about the rating scale to

establish (he common ground Tl is emphasized thal the level $ in the marking scheme was meant to be the most satisfactory achicvement level tor a Cam Giang,

Ligh Schoo! student, in order not to be confused with the top level described in any

other profigieney tests Secondly, three loachers and rescarcher assessed 10 first audio-recorded and discussed the reasons for scoring When they reached the agreement, they continued assessing 110 audio-recorded individually Finally, the

scores of teachers and experts were compared and discussed

2.5 Data collection instraments

2.5.1 The interview with the teachers and experts

Tu the present study, the inlerview was used The researcher used the interview

questions designed and adapted from rubric checklist LLS 11.00 (NC State

University) The yes-no questions were based on some specific categories regarding

asscasmnent thal all inerviewees were asked Lo comment on to assure the validity in

the research Six English teachors at Cam Giang High School and two experts take part in this research, The interview guide contains 12 questions, and all the questions regarding the marking scheme The questions in the inlerview were

placed into four sections: Section] (from Question 1 to Question 4) four questions

of the first section are designed to ask for teachers’ and experts’ comments about the

criteria of Ihe marking scheme Section 2 (Question 5 and Question 6) find out Ihe

comments of the teachers and experts about levels of performance of the marking

scheme Section 3 (three questions from Question 7 to Question 10) asked for

teachers’ and experLs’ opinion about descriptors Section 4 (three last questions from

Question 11 to Question 13) investigated teachers’ and experts’ general ideas about

the marking scheme and suggestions to make a better marking scheme (see

appoudix for imlerview questions) The Leachers atid experts were informed about

the interview and were given interview questions sheet one week before the

interview Each interview took about 8 to 10 minutes All the interviews were audio

yeeorded by the teachers and experts themselves The teachers and exporis are allowed to speak in Vietnamese so that the writer can elicit more information from

the subjects The data from the interview was analyved after gathering,

2.5.2 Sample oral test

150 students took the pilot-test with the same topic about music which 1s one of the {opie sludenis learnt in the seoond semester The students were infirmed of the topic

by the time of this trail During the test they have 5 minutes for preparation and then from 3 to 5 minutes to talk The students’ responses were audio-recorded After that, each student’s audiotaped was scored following the marking scheme by three

teachers and two experts

The content of the sample oral test is as follows:

Parl 1: Introduction (1 to 2.ininute)

Introduce yourself inchiding name, class, family, hobby and etc

Part 2: Topic (2 t0 3 minutes)

What is your lavounts kind of music?

+ Why do you like it?

+ Who is your favorite musician’ singer?

+ When do you listen to it?

For the researcher, the purpose of the oral test is to collect data on the scores made

by the teachers and experts when using the marking scheme The researcher wishes

to find out whether the marking scheme could be success/ully used by the teachers

In other words, the researcher would like to investigate correlations among, the teachers and experts This success is measured through the quantitative of the scores

nade by teachers and experts

2.5.3 Data collection procedure

The chronological steps of data collection procedure are summarized as follows

- Study the literature review on available marking schemes and write the first version

- Conduct interviews to the teachers and exports to gather their opinions of the marking scheme

- Write the second version of the marking scheme based on teachers’ and

experts’ responses

- Conduct pilot-test with the marking scheme with the participation of 150

tonth-grade sludonls, two teachers, the rescarcher and lwo experts

- Collect ail data for aalysis

- Analyze the data for findings

2.6 Date analysis method

2.61 Descriptive technique

‘The data collected from the interview were possessed and analyzed as it was easier to find out the ideas and comments of the experts and teachers about the marking scheme

2.62 Statistical technique

150 students’ oral tests were marked by two experts, two teachers and the researcher

following the new marking scheme The scores from the students’ oral tests were

computed and analyzed by the statistical soliware SPSS (Pearson’s correlation) to investigate the correlations between the assessing made by each single examiner and by the rest of the examiners

80

2.83 Data analysis procedure

Virstly, the data collected from the imterview were then analyzed ‘he data were processed and analyzed to find oul the teachers’ and experts’ answers to many

questions items

Secondly, the data were collected from the students’ oral tests To see the relationship among icachers, between (cachers and experts, among criteria, a Pearson correlation, which is a test to find out the correlation of ordinal variable, was employed Pearson's correlation is a statistical measure of the strength and

direction of the linear relationship belween pair dala (which are score made by

teachers, experts in this study) and 1t and it is denoted by r of which the value is

such that -l<r<il The closer r is to 11 or -1, the stronger the

relationship While positive values indicale

a relationship such that as values for one increase, values forthe other also increase, negative values indicate a relationship such that as values far one increase, values fer the other decrease Rainer (2008) provided guidelines for interpreting the strength of correlation:

r — 0 indicates no linear relationship or no correlation, 1.6 there is no

tendency for one variable to cither increase or decrease when the other increases

r values from more than 0 to less than +0.3 (from more than -0.3 to less than

0) indicate # weak correlation

r values from 0.3 to less than 0.7 (from more than -0.7 to -0.3) indicate a moderate correlation

r values from 0.7 to less than 1.0 (from more than -1.0 to -0.7) indicate a strong correlation

r= +1 or -l indicates a perfect linear relationship: as one variable increases (decreases) in i's values, the olher variable also increases (decreases) via an exact linear rule

Furthermore, Pearson correlation test in SPSS also provided p-value along the r value This p-value was to decide whether there is my evidences lo suggest thal

SL