(LUẬN văn THẠC sĩ) a study on the construction of a marking scheme for end of semester english oral tests for 10th grade students at cam giang high school in hai duong

VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF LANGUAGES AND INTERNATIONAL STUDIES FACULTY OF POSTGRADUATE STUDIES --- NGUYỄN THỊ OANH A STUDY ON THE CONSTRUCTION OF A MARKING SCHEM

VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF LANGUAGES AND INTERNATIONAL STUDIES

FACULTY OF POSTGRADUATE STUDIES

-

NGUYỄN THỊ OANH

A STUDY ON THE CONSTRUCTION OF A MARKING SCHEME FOR END-OF-SEMESTER ENGLISH ORAL TESTS FOR 10 th GRADE STUDENTS AT CAM GIANG HIGH SCHOOL IN HAI DUONG

Nghiên cứu xây dựng bảng đánh giá cho bài kiểm tra nói cuối kỳ môn Tiếng Anh cho học sinh lớp 10 trường THPT Cẩm Giàng, Hải Dương

M.A MINOR PROGRAMME THESIS

Field: English Teaching Methodology

Code: 60140111

Hanoi – 2017

VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF LANGUAGES AND INTERNATIONAL STUDIES

FACULTY OF POSTGRADUATE STUDIES

-

NGUYỄN THỊ OANH

A STUDY ON THE CONSTRUCTION OF A MARKING SCHEME FOR END-OF-SEMESTER ENGLISH ORAL TESTS FOR 10TH GRADE STUDENTS AT CAM GIANG HIGH SCHOOL IN HAI DUONG

Nghiên cứu xây dựng bảng đánh giá cho bài kiểm tra nói cuối kỳ môn Tiếng Anh cho học sinh lớp 10 trường THPT Cẩm Giàng, Hải Dương

M.A MINOR PROGRAMME THESIS

Field: English Teaching Methodology Code: 60140111

Supervisor: Nguyễn Thị Ngọc Quỳnh, Ph.D

Hanoi - 2017

DECLARATION

I hereby state that I, Nguyen Thi Oanh, declare the thesis entitled “A study on the construction of a marking scheme for the end-of-semester English oral tests for 10th grade students at Cam Giang High School in Hai Duong” is my own research for the Degree of Master of Arts at the Faculty of Post-Graduate Studies, University of Languages and International Studies- Vietnam National University, Hanoi This thesis is the result of my own research and efforts and it has not been submitted for any degree at any other university or institution

Hanoi, 2017

Nguyen Thi Oanh

ACKNOWLEDGEMENTS

First and foremost, I owe my deepest gratitude to my supervisor, Dr Nguyen Thi Ngoc Quynh for her encouragement, support, and guidance and for giving me constructive feedback throughout this thesis She has guided me in searching for relevant theory to my thesis and has also assisted in collecting data Consequently, I have learnt a lot about the assessment of English oral tests and construction of a marking scheme for oral tests

Secondly, I would like to thank Ms Bui Thien Sao, an expert of the Center for Language Testing and Assessment of the University of Languages and International Studies for her invaluable assistance during the research time

Thirdly, this thesis would not have been possible without the enthusiastic participation of six English teachers and 150 students at Cam Giang High School where the research was carried out

Lastly, I would like to thank my family whose love and support help me complete this thesis

ABSTRACT

This thesis was conducted at Cam Giang High School The participants consisted of

150 tenth-grade students and six English teachers at Cam Giang High School Two experts in the Center for Language Testing and Assessment of the University of Languages and International Studies also participated in this study The research aims at constructing a marking scheme for the end-of-semester English oral test of tenth-grade students Constructing a marking scheme for oral tests is a complex process In this paper, a combination of three methods: intuitive, qualitative and quantitative are employed by the researcher The research started with writing the draft of marking scheme Then, the researcher obtained the judgments from experts and other teachers Next, the marking scheme was piloted with 150 tenth-grade students Based on analyzing the students‟ scores of the oral tests, the researcher examined how well the marking scheme works The findings revealed that the marking scheme can be used by the teachers effectively although there is still a need for further investigation to improve and validate the current marking scheme The majority of the teachers took a positive attitude toward the marking scheme They believed in the efficacy of marking scheme in spite of some difficulties at the beginning and recommended the continuation of using the marking scheme for the next school year Pedagogical implications and suggestions for future studies are drawn out based on the research findings

TABLE OF CONTENTS

ơ

DECLARATION i

ACKNOWLEDGEMENTS ii

ABSTRACT iii

TABLE OF CONTENTS iv

LIST OF ABBREVIATIONS vi

LIST OF TABLES vii

PART A: INTRODUCTION 1

1 Rationale 1

2 Aims and objectives of the study 2

3 Research question 3

4 Scope of the study 3

5 Significance of the study 3

6 Method of the study 3

7 Design of the study 4

PART B: DEVELOPMENT 5

CHAPTER 1: LITERATURE REVIEW 5

1.1 Communicative competence 5

1.1.1 Communicative competence in the CEFR 6

1.2 What is speaking? 9

1.2.1 Assessing speaking 10

1.3 Marking scheme 10

1.3.1 What is a marking scheme? 10

1.3.2 Approach to construct a marking scheme 12

1.3.3 Steps to construct a marking scheme 13

1.3.4 Types of marking schemes 15

1.3.5 Structure of a marking scheme 17

1.3.6 Available speaking marking schemes 18

1.3.7 Previous studies 20

CHAPTER 2: METHODOLOGY 22

2.1 Setting of the study 22

2.2 Participants 23

2.3 Description of the end-of- semester oral test 24

2.4 Research design 24

2.4.1 Rationale for using a multiple-method approach 24

2.4.2 Research procedure 25

2.5 Data collection instruments 28

2.5.1 The interview with the teachers 28

2.5.2 Sample oral test 29

2.5.3 Data collection procedure 30

2.6 Data analysis method 30

2.6.1 Descriptive technique 30

2.6.2 Statistical technique 30

2.6.3 Data analysis procedure 31

CHAPTER 3: RESULTS AND DISCUSSION 33

3.1 The finding from interviews with teachers and experts 33

3.2 The finding from scores of students‟ oral tests 37

PART C: CONCLUSION 43

1 Summary of the study 43

2 Pedagogical implications 44

3 Limitation 45

4 Suggestions for further studies 45

REFERENCES 46 APPENDIX 1 I APPENDIX 2 XII APPENDIX 3 XIII APPENDIX 4A XVI

APPENDIX 4B XIX

APPENDIX 5 XXV APPENDIX 6: XXVII

LIST OF ABBREVIATIONS

CEFR: Common European Framework of Reference

MOET: Ministry of Education and Training

LIST OF TABLES

Table 1: Pearson Correlation on Grammar 38

Table 2: Pearson Correlation on vocabulary 39

Table 3: Pearson Correlation on pronunciation 39

Table 4: Pearson Correlation on fluency and coherence 40

Table 5: Pearson Correlation on Sum 40

PART A: INTRODUCTION

This part is offered to introduce the rationale of the study, the problem to be addressed in the study, the aims and objectives of the study, and the research questions to be answered It will also present the scope of the study, significance of the study, an overview of the employed methods and the design of the study

1 Rationale

English is the most popular official language in the world and the primary language

of global trade and commerce It is an international means of interaction and communication in almost all countries Proficiency in English is seen as a desirable goal for a lot of people in the world In many countries including Vietnam, English

is taught as a compulsory subject at school and it is included in many exams Of all four skills: reading, speaking, listening and writing; speaking is generally thought to

be the most important due to the fact that a lot of learners have spent years studying English but they still cannot speak it fluently In order to speak a foreign language, the learners must master the sound system of the language, use appropriate vocabulary and be able to put words together intelligibly with minimal hesitation Moreover, they also need to understand what is being said to them and respond appropriately to maintain good–natured relation to achieve communication goal (Luoma, 2004) Although students‟ speaking skills are often practiced and developed, it is not extensively assessed Comparing to other skills, speaking is the most difficult language skill to assess the reliability The student‟ speaking ability is usually judged during a face-to-face interaction, in real time between the teacher and the student Besides, the factors such as the nature of the interaction, the kind of tasks, the questions asked, the topic raised and the opportunity given students to speak in English will all have an impact on the student‟s performance (Luoma, 2004) On assessing speaking skills, the teacher has to take on the role as an interviewer and assessor at once, which puts him or her under a lot of pressure This makes teachers hesitant to assess speaking and focus on assessing other skills instead (Rychtarik, 2014) However, “if you want to encourage oral ability, then test

oral ability” stated by Hughes (1989) Testing and assessment is a very important part of teaching process, which helps provide necessary information to the students and teachers about the progress made and the work be done If assessment is carried out accurately and fairly, it will have a positive impact on both teachers and learners

It helps learners define the aim of their learning and contributes to improve the quality of teaching

Perceiving the importance of testing oral proficiency, on September 9th, 2014, the Ministry of Education and Training signed the decision No 5333/BGDT-GDTrH to promulgate the applying of speaking test in the final term test (account from 20 %

to 30 % of the total score) with the 10th grade students (seven –year program and ten-year program) and the 6th grade students (ten-year program) However, there is

no standard marking scheme for the teachers to follow Each teacher evaluates students‟ speaking based on the marking scheme he/she individually creates and uses different criteria to assess The marking schemes produced by teachers consist mostly or only numbers without descriptor or with very short descriptors such as very good, good and bad Besides, other factors such as language level, gender, status of the teacher, the familiarity between the teacher and the students also affect the evaluation student‟s performance and the score that the students get In order to overcome some of these problems, the researcher is fully aware of the need of constructing a marking scheme for students‟ speaking ability Therefore, a lot of attempts have been made to do a study titled “A study on the construction of a marking scheme for end-of-semester English oral tests for 10th grade students at Cam Giang High School in Hai Duong” Hopefully, the study, to which I will devote all my efforts, will make a contribution to the English teaching and learning

of the teachers and students at Cam Giang High School

2 Aims and objectives of the study

The study is aimed at constructing a marking scheme for end-of-semester English oral tests for 10th grade students at Cam Giang High School in Hai Duong with hope to find an effective tool for assessing students‟ English speaking competence

In order to achieve this aim, the following objectives will be addressed in the study: + To find out the marking criteria and descriptors of the target marking scheme + To investigate the feasibility of the proposed marking scheme

3 Research question

The study aims at answering the following questions:

1 What are the marking criteria of the marking scheme for end-of-semester English oral test for 10th grade students at Cam Giang High School?

2 What are the descriptors for each criterion of the marking scheme for end-of -semester English oral test for 10th grade students at Cam Giang High School?

4 Scope of the study

The scope of the research has been made quite clear from the title: “A study on the construction of a marking scheme for end-of-semester English oral tests for 10th grade students at Cam Giang High School in Hai Duong”

Firstly, the study only focuses on oral testing, namely the construction of a marking scheme To be more specific, it refers to the process to find out the marking criteria and descriptors for a marking scheme which helps teachers at Cam Giang High School assess students‟ oral test This will help teachers at Cam Giang High School assess students‟ speaking more accurately and objectively Secondly, the subject of the study is restricted to tenth-grade students at Cam Giang High School

5 Significance of the study

The study is of great significance to both the 10th grade students and the teachers at Cam Giang High School because it may have a great contribution to teaching and learning speaking For teachers, the study helps them to find a useful assessment tool which ensures objectiveness and fairness A marking scheme is an assessment tool that clearly indicates marking criteria which let students know what is expected

of them and how to achieve aim It is hopeful that the study will give students opportunity to do self-assessment to reflect on the learning process

6 Method of the study

As the aim of the study to construct a marking scheme for end-of-semester English

oral tests for 10th grade students at Cam Giang High School in Hai Duong, the study is designed to use a combination of intuitive, qualitative and quantitative

methods Besides, many sources such as books, newspapers and some sources on

the internet have been read by the researcher The findings are reported based on the experts‟ and the teachers‟ responses in the interview and students‟ scores on oral test Moreover, constant discussions with the supervisor are of great significance

7 Design of the study

The study is divided into three main parts:

Part I (Introduction) includes the rationale, the aims and objectives, the scope, the significance, the research questions, the method and the design of the study

Part II (Development) consists of chapters as follows

Chapter 1 (Literature review) presents the theoretical background of the study and

the review of the available marking schemes and previous studies

Chapter 2 (Methodology) describes in detail the research methodology which

consists of the context of the study, the information of the subjects, instruments of data collection, procedures of data collection and methods of data analysis

Chapter 3 (Results and Discussion) reports the statistical results and the analysis of

the data

Part III (Conclusion) closes the study by summarizing the whole study with concluding remarks and offering some limitations and suggestions for further

studies

PART B: DEVELOPMENT CHAPTER 1: LITERATURE REVIEW

This chapter provides an overview of the literature on the field of communicative competence, assessing speaking and marking scheme It discusses communicative competence, communicative competence in the CEFR, definitions of speaking, assessing speaking, definitions of marking scheme, approaches to construct a marking scheme, types of marking scheme and the structure of a marking scheme A

review of the available marking schemes and previous studies is also presented

Canale and Swain (1980) proposed one of the first theoretical models of communicative competence The model distinguishes between communicative competence and communicative performance, as communicative competence is knowledge about grammar, sociolinguistic knowledge, strategic competence, while communicative performance is the actual communication However, Canale (1983) revised this model and used the term “actual communication” instead of

“performance” He also asserted that: “Communicative competence refers to both knowledge and skill in using the knowledge when interacting in actual communication” (Canale, 1983, p 5)

Bachman and Palmer (1996) developed another model in communicative

competence and language testing: the model of communicative language ability The model focuses more on the interaction between context and language use (Fulcher and Davidson, 2007) The term “language ability” consists of language knowledge and strategic competence Language knowledge includes both organizational knowledge (grammatical and textual knowledge) and pragmatic knowledge (illocutionary competence and sociolinguistic competence) Strategic competence involves the ability to assess whether the situation is practicable and to plan for the next movement (Luoma, 2004) Bachman and Palmer (1996) claimed that their model can be used as a checklist for developing language tests

1.1.1 Communicative competence in the CEFR

The CEFR is the Common European Framework for language learning, teaching and assessment, which focuses on the nature of language use and the language user and the implications for learning and teaching (Council of Europe, 2001) The Common European Framework defines levels of proficiency which allow learners‟ progress to be measured at each stage of learning and provides criteria for assessing four English skills: reading, speaking, listening and writing Of all skills, speaking

is very important and within communication the learners can learn and apply various skills The Framework consist of three proficiency levels which are referred

to respectively as Basic User: A1, A2, Independent User: B1, B2, and Proficiency User: C1, C2 (Council of Europe, 2001)

Communicative competence in the narrower sense consists of three components, linguistic competences, sociolinguistic competence and pragmatic competences (Council of Europe, 2001, p 108) In the model of communicative competence of Canale and Swain (1980), there are also three components, grammatical competence, sociolinguistic competence and strategic competence Although the CEFR and Canale and Swain label these categories slightly differently, they use similar categories to describe competences In the following paper, the categories from the CEFR are particular relevance for the assessment of speaking will be discussed

Linguistic competence is considered the core of the model of communicative competence Linguistic competence is divided into lexical competence, grammatical competence, semantic competence, phonological competence and orthoepic competence (Council of Europe, 2001, p.109)

Lexical competence is defined as “knowledge of, and ability to use, the vocabulary

of a language, consists of lexical elements and grammatical elements” (Council of Europe, 2001, p.110) Lexical element includes fixed expression and single word forms to enhance language on the different levels of meaning The CEFR also presents illustrative scale for the range of vocabulary knowledge and the ability to control that knowledge to specify these competences (Council of Europe, 2001, p.112)

Grammatical competence is described as “knowledge of, and ability to use, the grammatical resources of a language” (Council of Europe, 2001, p 112) Grammatical competence is the ability to understand and express meaning by producing and recognizing well-formed phrased and sentences (Council of Europe, 2001) To measure grammatical competence, the CEFR has developed an illustrative scale demonstrating levels of grammatical accuracy (Council of Europe,

2001, p 114) Grammatical competence also includes morphology and syntax which the learners need to be aware of Morphology deals with the organization of words and the ways to modifying words forms and syntax deals with the organization of words into meaningful sentences (Council of Europe, 2001)

Semantic competence is “the learner‟s awareness and control of the organization of meaning” (Council of Europe, 2001, p 115 )

Phonological competence involves the knowledge of the sound-units, words stress, sentences stress, sentence rhyme and intonation (Council of Europe, 2001, p 116) Orthoepic competence involve knowledge of spelling conventions, ability to consult

a dictionary, knowledge of the implication of written form for phrasing and intonation and ability to solve ambiguity in various context (Council of Europe,

2001, pp: 117-118)

Sociolinguistic competence is “concerned with the knowledge and skill required to deal with the social dimension of language use” (Council of Europe, 2001, p 118) Sociolinguistic competences include linguistic markers of social relations, politeness conventions, expressions of folk-wisdom, register differences, and dialect and accent (Council of Europe, 2001, p 119) Canale and Swain defined sociolinguistic competence as “the ability to communicate appropriately in a variety

of contexts, this includes both verbal and non-verbal communication” (Canale and Swain, 1980)

Linguistic markers of social relations vary from language to language They include use of choice of greeting, use and choice of address forms, conventions for turn-taking in conversations and the use of choice of expletive (Council of Europe,

2001, p 119)

Politeness conventions vary from one culture to another and are a frequent source of inter-ethnic misunderstanding when polite expressions are literally interpreted (Council of Europe, 2001, p 119)

Expressions of folk wisdom are fixed formulas about daily life, often used in newspaper headlines The expressions include proverbs, idioms, and expressions for beliefs, attitudes and values and are often used in graffiti and on T-shirt slogans (Council of Europe, 2001, p 120)

Register differences refer to “systematic differences between varieties of language used in different contexts” (Council of Europe, 2001, p 120) Register differences express differences in level of formality: frozen, formal, neutral, informal, familiar and intimate (Council of Europe, 2001, p 120)

Dialects and accents perform people‟s origin, sociolinguistic competences include the ability to distinguish between various social classes, regional provenances, national origins, ethnicities and occupational groups (Council of Europe, 2001, p 121)

Pragmatic competences include discourse competence and functional competence

“Discourse competence is the ability of a user/ learner to arrange sentences in sequences so as to produce coherent stretches of language” (Council of Europe,

2001.p 123) The CEFR has included illustrative scale with aspects of discourse competence: Flexibility to circumstances, turn-taking in interactions, thematic development and coherence and cohesion (Council of Europe, 2001, pp: 123-125) Functional competence is “concerned with the use of spoken discourse and written texts in communication for particular purpose” (Council of Europe, 2001, p 125) Functional competence also includes knowledge and ability to use the schema patterns of social interaction) which underline communication, such as verbal exchange patterns (Council of Europe, 2001, p 126) The CEFR developed an illustrative scale for two qualitative aspects, fluency and propositional precision Fluency is “the ability to articulate, to keep going and to cope when one lands in a dead end” Propositional precision refers to “the ability to formulate thoughts and propositions so as to make one‟s meaning clear” (Council of Europe, 2001, pp: 128-129)

1.2 What is speaking?

According to Widdowson (1984), “speaking is an active productive skill” This means that when someone wants to deliver information, he or she needs to activate their background knowledge, choose appropriate words and use correct grammar and pronunciation to gain meaning This process requires someone‟s brain to use all knowledge he or she has about the language therefore it is not easy to be mastered Bygate (1987) considers learner‟s speech as a process, speaking is a “real-time” action because the learner has to plan, process and produce the language simultaneously The speech process includes planning, selection and production of speech Planning is an interactive process which requires learner to have knowledge about interaction routines to plan the next step of the conversation Selection is the stage in which the leaner uses knowledge of language and grammar to decide how

to express oneself In the production stage, the learner uses the knowledge about

pronunciation and communication strategies to produce language (Bygate, 1987)

The CEFR has distinguished clearly between interaction (spontaneous) and production (prepared) of language Interaction activities are mainly spontaneous and

are carried out throughout conversation and more or less informal discussion Production activities are mainly prepared and rehearsed in advance (Council of Europe, 2001, p 178)

1.2.1 Assessing speaking

The categories for oral assessent is enormous and deciding which criteria to use for assessment is a relatively dificult work Assessor should determine the most appropriate criteria that could not only be used to assess students‟ speaking, but also

be relevant to the objective of the course/ lesson, etc (Knight , 1992)

The CEFR classifies 12 qualitative categories relevant to oral assessment The CEFR has also developed illustrative scales for assessment and each scale describes the level of proficiency There are a number of categories relevant to assess speaking such as turn-taking strategies, co-operating strategies, asking for clarification, fluency, flexibility, coherence, thematic development, precision, sociolinguistic competence, general range, vocabulary range, grammatical accuracy, vocabulary control and phonological control (Council of Europe, 2001) However, it

is impossible to assess all criteria simultaneously The assessers need to make choices for each assessment situation and select only several criteria that are relevant to the particular context Choosing no more than 4 or 5 criteria in each testing situation guarantees feasibility as well as reliability of the assessment ( Council of Europe, 2001, pp: 192-193)

1.3 Marking scheme

1.3.1 What is a marking scheme?

A marking scheme is sometimes referred to as a scoring rubric or a rating scale which is defined as an explicit set of criteria used for assessing a particular type of work or performance

As Davied, Brown, Elder, Hill, Lumly, McNamara (1999) defined it:

A rating scale is a scale for the description of language proficiency consisting of a series of constructed levels against which a language learner’s performance is judged Like a test, a

proficiency scale provides an operational definition of a linguistic construct such as proficiency Typically such scales range from zero mastery through to an end-point representing the well-educated native speaker The levels or bands are commonly characterized in terms of what subjects can do with the language and their mastery of linguistic features (such as vocabulary, syntax, fluency and cohesion)…Scales are descriptions of groups of typically occurring behaviors, they are not in themselves test instruments and need to be used in conjunction with tests appropriate to the population and test purpose Raters or judges are normally trained in the use of proficiency scales so as to ensure the measure’s reliability (David

et all, 1999, pp: 153-154)

From above definition, it can be seen that a rating scale includes both the fields to

be assessed (construct) and the alignment between examiner‟s performance and the predetermined levels of behavior descriptions Thus, it is important to consider two above components when constructing a rating scale In addition, for different purposes the construction of scale can be different Alderson (1991) and Pollitt and Murray (1996) classify different purposes that rating scales could serve: user-oriented, constructor-oriented and assessor-oriented A user-oriented scale is

“designed to communicate information about typical of likely test taker behaviors at

a given level” (Taylor, 2011, p 190) Constructor-oriented scale “guides test writer

in their choice of tasks to include in a test” (Taylor, 2011, p 190) Assessor oriented scale “guides the rating process, focusing on the quality of performances expected” (Taylor, 2011, p 190) Because one rating scale is rarely appropriate for all the purposes above, the purpose compatible with the scale must be prioritized to measure the sample language elicited from learner‟s performance in particular testing situation (Nakatsuhara, 2007)

In the present paper, the researcher focuses on constructing an assessor-oriented

speaking marking scheme which teachers at Cam Giang high school can utilize to assess students‟ oral tests

1.3.2 Approach to construct a marking scheme

Marking scheme construction is recognized to be a complex process (Brindley,

1998, Fulcher, 2003, North, 2000) Traditionally, the design and construction of rating scales used a priori approach in which assessment criteria and rating scale descriptors are developed by “experts” using their own experience and intuitive judgment ( Fulcher, 2003) McNamara (1996) states that marking schemes were constructed based on the construction of the first scale for the Foreign Service Institute‟s Oral Proficiency Interview in the 1950s In 1990s, many authors supported empirically based approach involving analyzing samples of actual language performance to construct criteria and marking scheme descriptors (Fulcher, 1996; Milanovic, Saville, Pollitt and Cook, 1996; Shohamy, 1990; Upshur and Turner, 1995) Fulcher (2003) discusses two basic approaches to rating – scale development: intuitive approaches and empirical approaches Intuitive methods primarily rely on “expert judgment and the principled interpretation of experience” (Council of Europe, 2001, p 208) Fulcher also introduced three common subcategories of an intuitive method as expert judgment, committee and experiential The expert judgment and committee are alike in which the experts make decisions upon issues such as the number of levels and the wording of descriptors based on existing scales, curriculum, course material and other necessary sources Experiential evolves on the basis of the expert judgment and committee by revising the scale after a further understanding on both the content of existing scale and sample performance (Fulcher, 2003) Empirical methods include data-based, data -driven scale development, empirically derived, binary – choice, boundary definition scales and scaling descriptors (Fulcher, 2003) More recently, together with scale development for the Common European Framework of Reference (CEFR), the mixture of qualitative and quantitative methods makes complementary contribution to rating scale development (Council of Europe, 2001) Quantitative

methods rely on statistical analyses and careful interpretation of results while qualitative methods involve interpretation of information obtained

There are a number of approaches to marking scheme construction The best methods for marking scheme construction are said to combine all three approaches including intuitive, quantitative and qualitative approaches in “a complementary and cumulative process” (Council of Europe 2001, 207) Therefore, in constructing

a marking scheme for assessing student s‟ oral test at Cam Giang High School, a combination of intuitive, qualitative and quantitative approaches was used

1.3.3 Steps to construct a marking scheme

According to Mertler (2001) building a marking scheme includes following seven steps:

Step 1: Re-examine the learning objectives to be addressed by the task This

helps to match the teacher‟s scoring guide with objectives and actual instruction

Step 2: Identify specific observable attributes that the students demonstrate in

their product, process, or performance

Step 3: Brainstorm characteristics that describe each attribute Identify ways

to describe above average, average, and below average performance for each observable attribute identified in Step 2

Step 4a: For holistic rubrics, write thorough narrative descriptions for

excellent work and poor work incorporating each attribute into the

description Describe the highest and lowest levels of performance

combining the descriptors for all attributes

Step 4b: For analytic rubrics, write thorough narrative descriptions for

excellent work and poor work for each individual attribute Describe the highest and lowest levels of performance using the descriptors for each attribute separately

Step 5a: For holistic rubrics, complete the rubric by describing other levels on

the continuum that ranges from excellent to poor work for the collective attributes Write descriptions for all inter mediate levels of performance

Step 5b: For analytic rubrics, complete the rubric by describing other levels on

the continuum that ranges from excellent to poor work for each attribute Write descriptions for all inter mediate levels of performance for each attribute separately

Step 6: Collect samples of student work that exemplify each level These will

help the examiners score in the future by serving as benchmarks

Step 7: Revise the rubric, as necessary Be prepared to reflect on the

effectiveness of the rubric and revise it prior to its next implementation (Mertler, 2001)

Meanwhile, Nakutsuhara (2007) suggests developing a marking scheme according to four stages:

Stage 1: Reviewing existing speaking rating scales outside and inside Japan to collect marking categories and descriptors to be referenced in the later stage

Stage 2: Examining the course of the study (the guideline of secondary school education) to decide types and levels of marking categories for the target population

Stage 3: Drafting a rating scale based on the existing rating scales, while obtaining expert judgments from eight experienced upper-secondary school teachers

Stage 4: Piloting the scale with 42 Japanese upper- secondary students with two raters, to examine how well the resulting rating scale functions (Nakutsuhara, 2007)

Taylor (2001) proposes the process of constructing a marking scheme is took place

in three phases as outline below:

+ Propose the criteria

+ Determine the number of scales

+Develop the descriptors

+Discuss with other teachers and revise

+Trail the scale

+ Stabilize the scale -> Version 1

Phase 2: Qualitative

External expert reviewing the existing scale

+ Rank the descriptors in order of difficulty

+ Trail the scale

+ Discuss and revise the scale

+ Stabilize the scale -> Version 2

Phase 3: Quantitative

Standard –setting phase

+ Trail the raters

+ Trail the scale

+ Analyze the scores

+ Stabilize the scale -> version 3 (Taylor, 2011, p 195)

In this research, the researcher follows a three - phase process suggested by Taylor (2011) for some reasons Firstly, according to the CEFR (2001, p 207) the best methods for rating scale development are said to take advantage of the strengths of

a range of intuitive, quantitative and qualitative approaches The best scale, the CEFR (2001, p 207) suggests, combine all three approaches in a “complementary and cumulative process” Secondly, the phases are logical and detailed They are clear for the researcher to follow In each phrase, steps are described specifically therefore it helps the researcher have an outline to construct a marking scheme

1.3.4 Types of marking schemes

There are many divisions of individual types of marking scheme as there are many authors dealing with this issue According to “Rubric for assessment” of Northern

Illinois University there are several types of rubrics including holistic, analytical, general, and task –specific Timothy Farnsworth (2014) of CUNY Hunter College divided rubric into five types: holistic, analytic, task Fulfillment, performance decision trees and checklist However, according to Mertler (2001) and Taylor (2011) assessment criteria used to evaluate L2 spoken language performance generally fall into two main categories: holistic and analytic In this study, the researcher will focus on describing some advantages and disadvantages of these two types

A holistic marking scheme requires the rater to make an “impressionistic assessment” of the quality of students‟ oral test using a single marking scheme (Davies, Brown, Elder, Hill, Lumley and Mc-Namar, 1999, p 75) A holistic marking scheme is commonly used in assessing L2 speaking and it is described as

“general impression marking” in which the overall properties are more important than particular features of the performance (Association of Language Testers in Europe, 1998, p 147) Using holistic marking schemes can be relatively quicker than analytic marking schemes (Nitko, 2001) However, the holistic marking schemes do not provide detail information on student performance for each criterion and they are not very useful to help plan instruction because they lack a detail analysis the strengths and weakness of student‟s product (Taylor, 2011, p 178)

An analytic marking scheme gives separate scores for each criterion of performance Each criterion is assessed separately and using different descriptive marking scheme (Taylor, 2011, p 179) Analytic marking scheme results initially in many scores, followed by a summary total score representing an assessment on a multidimensional level (Mertler, 2001) Using an analytic marking scheme helps to focus rater judgments more narrowly which contribute to rater agreement and rating reliability (Weir, 1990) Scoring tends to be more consistent across students and grades It provides students meaningful and specific feedback on area of strength and weakness (Nitko, 2001) Besides these advantages, using analytic marking schemes also has some disadvantages The use of an analytic marking scheme can

cause the scoring process to be slower because the teacher has to provide separate assessment for each of a number of aspects of performance Moreover, construction

of an analytic marking scheme can be quite more difficult and time-consuming (Mertler, 2001)

As mentioned above, compared to a holistic marking scheme, an analytic marking scheme gives both teachers as well as students a more precise feedback on what students do This helps the students recognize in detail the good points and weak points of their own and find way to improve In the present study, the researcher chooses to construct an analytic marking scheme which Cam Giang High School teachers can utilize to assess speaking competence of students in their end -of- semester oral test

1.3.5 Structure of a marking scheme

From above illustration the element of a scoring rubric or a marking scheme, a marking scheme often includes four basic components: criteria, levels of performance, scores and descriptors (Northern Illinois University, 2014, p 1)

Criteria “identify the trait, feature or dimension which is to be measured and include

a definition and example to clarify the meaning of each trait being assessed” (Northern Illinois University, 2014, p 2) The number of criteria will be decided depending on tasks, assignments or students‟ performance (Northern Illinois University, 2014, p 2)

Levels of performance are often adjectives which describe the performance levels For example, the adjectives are often used such as superior, moderate, poor, below average Levels of performance clarify the degree of performance which the learners need to achieve and provide for consistent and objective assessment and better feedback to students (Northern Illinois University, 2014 p 2)

Score are often combined with levels of performance The teacher should consider how many points are needed to adequately describe the range of performance he expects to see in student‟s work For example, 0,1,2,3,4…(Northern Illinois University, 2014, p 2)

Descriptors “are explicit descriptions of the performance and show how the score is derived and what is expected of the students” (Northern Illinois University, 2014, p 2) Descriptors explain clearly each level of performance for each criterion and describe in detail the characteristic of levels of performance Descriptors also help the teacher distinguish level of performance of each student‟s work “The same descriptors can be used for different criteria within one marking scheme” (Northern Illinois University, 2014 pp 3-4)

1.3.6 Available speaking marking schemes

There are a number of performance-based rating scales which have been produced Beginning with the US Foreign Service Institute (FSI) scale which was introduced

in the early 1950s and the scale ranged from no proficiency to native speaker ability The assessment criteria include accent, fluency, comprehension, vocabulary and grammar Afterwards, many other language proficiency tests started to apply that rating scale to their scoring such as American Council on the Teaching of Foreign Languages (ACTFL) (ACTFL, 1986) In recent year, some prevailing language

assessments also witness the employment of as IELTS Oral Test and Spoken English of Test of TOEFL In 2001, the Common European Framework of Reference for Languages: learning, teaching, assessment (CEFR) developed by the Council of Europe (COE) rapidly became the most influential scale as the standard reference document for teaching and testing languages in Europe (Council of

Europe, 2001; Fulcher, 2004) The framework consists of six levels: A1

(Breakthrough), A2 (Waystage), B1 (Threshold), B2 (Vantage), C1 (Effective Operational Proficiency) and C2 (Mastery), and each level is associated with a set

of descriptors IELTS and ESOL are examples of the same framework Since the CEFR intends to provide a whole set of reference tools that should be selectively employed according to the context, there are a number of categories relevant to oral

assessment, such as turn-taking strategies, co-operating strategies, asking for

clarification, fluency, flexibility, sociolinguistic competence, general range, vocabulary range, grammatical accuracy, vocabulary control, phonological control

and so on The application of the CEFR is also extending to outside Europe Although there are few official documents available, there is some evidence about the growing extension and a great potential of the CEFR throughout the world, including Vietnam

On the 24th of January, 2014, the Vietnamese Minister of Education and Training signed Circular No 01/2014/TT-BGDĐT to promulgate Khung năng lực ngoại ngữ

6 bậc dành cho Việt Nam (six-level Foreign Language Proficiency Framework for Vietnam –hereafter referred to as VNFLPF) VNFLPF is based mainly on CEFR This framework consists of three proficiency levels which are referred to respectively as Sơ cấp (Elementary level = CEFR basic user), Trung cấp ( Intermediate Level = CEFR Independent User) and Cao cấp (Advanced level = CEFR proficiency User) Each level is broken down into two sublevels, making it a 6-level proficiency framework In order to oriented users in the educational system

to some practical purposes, VNFLPF provides a list of descriptors of each proficiency level and presents detailed statements of what the learner can do at each

proficiency level in term of listening, speaking, reading and writing

VSTEP (Vietnamese Standardized Test of English Proficiency) is a test of the university to assess language competence from level 3 to 5 (equivalent of CEFR level B1, B2 and C1) according to VNFLPF This is Vietnam‟s first test of English proficiency assessment developed basing on principles of test assessment construction process of ALTE The VSTEP speaking test is an interview format test and the VSTEP rating scale for oral assessment has 5 marking categories and subcategories for each category 1) Grammar: range and accuracy, 2) Vocabulary: range and control, 3) Pronunciation: individual sounds, stress and intonation, 4) Fluency: hesitation, extended speech, 5) Discourse management: thematic development, coherence and cohesion with ten levels from 0 to 9 The criteria of rating scale were derived from the performance of VSTEP‟s speaking test The descriptors seem greatly useful for raters‟ understanding of the scale The scale is particularly remarkable as this is one of the meaningful rating scales in Vietnam developed to assess student‟s speaking competence Besides, in 2013, the Minister

of Education and Training deployed a test to survey language competence level 2 for students who have finished grade 9 participate in Pilot English Curriculum for Vietnamese Upper Secondary Schools Speaking section assessment criteria was developed basing on IELTS speaking criteria It was a holistic rating scale including

4 categories: Fluency and coherence, lexical resource, grammatical range and accuracy, and pronunciation

In short, there seem to be well developed rating scale outside and inside Vietnam In the construction of rating scales, it seems meaningful to selectively use the CEFR as

a reference and modify the descriptors and number of levels according to the target context as recommended by Council of Europe (2011) Furthermore, speaking section assessment criteria by MOET also serves as a useful basic when constructing a marking scheme for tenth-grade students‟ oral test at Cam Giang High School

1.3.7 Previous studies

There have been some researchers who studied about constructing a marking scheme for oral tests

Fulcher (1993) carried out his research entitle “The Construction and Validation of Rating Scales for Oral Tests in English as a Foreign Language” In his research, he investigated the principles upon which rating scale in oral testing are constructed and used, and the subsequent claims of reliability and validity made for them He designed a fluency rating scale based on a larger data-based originally built for research into scale design for a doctoral dissertation

Nakatsuhara (2007) conducted the study on “Developing a Rating Scale to Assess English Speaking Skills of Japanese Upper-secondary Students” The study was motivated by the fact that there was no standard scale to rate the speaking of Japanese upper-secondary students The work was divided into four stages: studying available rating scale and their practices and outside Japan, examining relevant levels of analytic marking categories, drafting a rating scale with expert advice of eight qualified ESL teachers, and running the scale on forty- two students with two raters

Raza (2008) did the research on “A rating scale to assess English speaking

proficiency of university students in Pakistan” He proposed the much needed rating

scale to assess English speaking proficiency of students graduating in 124 universities in Pakistan Like Nakatsuhara, he also described the steps to develop a rating scale to assess English speaking achievement of Pakistan university students

Summary: On the whole, in this chapter, theoretical framework concerning

communicative competence, assessing oral test, and marking scheme has been reviewed These theories will serve as the foundation for the author to form and conduct the study according to specific methodology that will be elaborated in the next chapter

CHAPTER 2: METHODOLOGY

The research method plays a vital role in any study since it determines the reliability and validity of the study This study has employed a combination of three methods including intuitive, qualitative and quantitative to find the answer to the research questions This chapter presents setting of the study, participants of the study and description of the end-of-semester oral test This chapter also discusses the reasons for the choice of methods employed in this study and the procedure of the research The next parts in this chapter are to present the instruments used to collect and analyze data

2.1 Setting of the study

The study was conducted at Cam Giang High School in Hai Duong The school has

a history of 50 years of development since its foundation in 1966 Up to now, it has

30 classes of three grades 10, 11 and 12 The average number of students in each class is 40 students and the classroom is poorly equipped only with a blackboard and 24 small unmovable desks each of which is shared by 2 students There are seven English teachers All of them are enthusiastic and dedicated with work The school is considered one of the center schools of District At school, English is considered as one of the most important subjects in training the students and the students have four lessons of English a week

At the time this research was conducted, the students were in the second term of the academic year which covered the last 7 units of English text book 10

Material

The current teaching material is „Tieng Anh 10‟ (seven-year program) published by Education Editor, Ministry of Education and Training The English textbook 10 (seven-year program) includes 16 teaching units and 6 review units called Test

yourself Each teaching unit covers a topic and is structured into 5 sections: reading,

speaking, listening, writing and language focus The speaking section includes activities, termed “tasks” related to topics of each unit The first and second activities provide language input and develop specific language functions such as expressing opinion, agreement and disagreement The rest involves short talks on a specific topic with or without prompts By the end of the course, students are able to ask and answer about the topics covered They can perform some basic language

functions such as giving instructions, expressing opinions, asking for directions, asking and giving information, etc (General objectives to achieve in Grade 10 for speaking skills (MOET 2006: 19-25))

2.2 Participants

Participants in this study include students, seven teachers and two experts

Students: 150 tenth grade students at Cam Giang High School were selected to take part in the pilot testing of the marking scheme They were tested for their oral proficiency They were provided with comprehensive instructions about the topic they had to accomplish Their oral tests were tape-recorded for subsequence scoring Their ages are from 16 to 17 They have passed an exam in English, Maths and Literature to enter Cam Giang High School All of them are Vietnamese native speakers and they started to learn English five years ago Their English knowledge was just at the elementary level and some even are at lower level Generally, students at this level can ask and answer about personal preferences, daily routines, common events, and other personal topics

Teachers: Seven teachers, including the researcher, who are currently teaching English at Cam Giang High School participate in this study Their teaching experience ranges from 4 years to 21 years All of them have good English competence and they are regularly involved in student instruction and assessment of performances All the seven teachers are female Here are all teachers‟ profiles Teacher Age Year of

university graduation

Teaching experience (No of year)

Level according to CEFR

Experts: Two experts, who are working at the Center for Language Testing and Assessment, University of Languages and International Studies participated in the study They have experience in constructing tests and marking schemes for many years

2.3 Description of the end-of- semester oral test

All the students at Cam Giang High School take an oral English test at the end of each semester The topics are constructed based on the same topics as the textbook

At the examination, the students are required to introduce themselves and then choose randomly one of the prepared topics Each topic is followed by some guiding questions which students can base on to outline their oral tests They have 5 minutes for preparation and then from 3 to 5 minutes to talk (See appendix for topic cards)

2.4 Research design

2.4.1 Rationale for using a multiple-method approach

This study is an attempt to construct a marking scheme for the end-of-semester oral test for tenth grade students at Cam Giang High School Therefore, it is best assisted by combining all three methods intuitive, qualitative and quantitative A distinction has been made between intuitively and empirically developed rating scales Intuitively methods primarily rely on expert judgment and the “principle interpretation of experience” Empirical methods, by contrast, are data-driven and based on actual learner performances Empirical scale development methods may be subdivided into

“quantitative” and “qualitative” based on the type of data they draw on

Fulcher (1996) and others (e.g Brindley 1998; Council of Europe 2001; Knoch 2009a; Turner and Upshur 2002) have argued that the intuitive design of many assessment scales, based exclusively on expert judgment, is compromised by the lack of empirical data on the features of learner speech or writing Recognition of the limitations of intuitively derived assessment scales has encouraged more empirically based methods of scale development However, it is important not to lose sight of the value of expert judgment and the wealth of knowledge which experts can bring to the development process The best methods for rating scale

development are said to take advantage of the strengths of a range of intuitive, quantitative and qualitative approaches

2.4.2 Research procedure

According to Taylor (2011) the process of constructing a marking scheme is took

place in three phases: intuitive phase, qualitative phase and quantitative phase In

this study, the researcher follows these above phases However, some minor steps in each phase are modified to make it appropriate for the specific context of the

researcher The intuitive phase is the first stage in which the researcher reviews the

existing scale in the world and in Vietnam to produce a reference collection of marking criteria and descriptors Teaching materials, objectives of the course and relevant source materials are also reviewed to propose the first draft of the marking scheme After that the researcher obtains judgments from six experienced high school teachers In qualitative phase, the marking scheme is submitted to the experts

to review and refine the marking scheme, which ensures the internal coherence of the marking scheme The judgments are collected from both the teachers and experts through interviews Quantitative phase analyzes the scores of students‟ oral tests to confirm the soundness of the marking schemes, assessment criteria and

descriptors as a whole in live tests

Phase 1: Intuitive phase

Firstly, the researcher determines the evaluation criteria for her marking scheme Based on communicative language ability framework of Cannale & Swain (1980) and Bachman and Palmer (1996), the CEFR, and rating categories collected from the Speaking section assessment criteria, it is plausible to choose four analytic marking categories for the purpose of this rating scale: 1 Grammar, 2) Vocabulary, 3) Pronunciation, 4) Fluency and coherence The choice of four marking categories

is made because they are based on speaking section assessment criteria in the test for surveying language competence level A2 for students, which the present rating scale was also aiming to assess Moreover, these categories are frequently applied criteria for assessing speaking In addition, at the end of the course, the students can

ask and answer about the topics covered and perform some basic language functions such as giving instructions, expressing opinions, asking for direction, asking and giving information, etc That is equivalent level A2 in the CEFR The CEFR has developed illustrative scales for these categories as follows:

Grammar refers to the accurate and appropriate use of a range of grammatical forms (Taylor, 2011) Students on proficiency level A2 use some simple structures correctly, but still systematically makes basic mistakes- for example tend to mix up tenses and forget to mark agreement, nevertheless, it is usually clear what he/she is trying to say” (Council of Europe, 2001, p 123)

Vocabulary refers to the candidate‟s ability to use a range of vocabulary to meet task requirement At level A2 student are expected to “have sufficient vocabulary to conduct routine, everyday transactions involving familiar situations and topics” and (Council of Europe, 2001, p 121)

Pronunciation refers to student‟s ability to produce individual sounds and to link words together, as well as using stress and intonation to communicate meaning (Thornbury, 2005, p 128) Pronunciation in the CEFR is included in an illustrative scale for phonological control and on proficiency level A2 Students are expected to

“pronounce clearly enough to be understood despite a noticeable foreign accent, but conversational partners will need to ask for repetition from time to time” (Council

of Europe, 2001, p 126)

Fluency and coherence refer to student‟s ability to talk with normal levels of continuity, speech rate and effort and to link ideas and language together in coherence, connected speech and formal structure (MOET, 2014, p 190) A student on proficiency level A2

“can make himself/herself understood in short contributions, even though pauses, false starts and reformulation are very evident” and “can use the most frequently occurring connectors to link simple sentences in order to tell a story or describe something as a simple list of points” ( Council of Europe, 2001, pp: 134-138)

Secondly, in order to meet the objective of the course, it was decided that all categories should focus more on success in conveying a message and

communication rather than language accuracy Therefore, the categories are not equally important Some categories account for higher percentage in comparison to the others According to the speaking section assessment criteria, Vocabulary and Fluency and Coherence are more important than Grammar and Pronunciation Therefore, the marking categories were described in the following weighting: Fluency and coherence marked out of 10 then multiplied by 3 Vocabulary marked out of 10 then multiplied by 3, Grammar marked out of 10 then multiplied by 2 Pronunciation marked out of 10 then multiplied by 2 (MOET, 2014, p.190)

Thirdly, great care was taken to decide the number of levels, since the number of levels should be adequate to capture and discriminate different levels of speaking ability, but should not exceed the number of levels that busy school teachers are, with limited training, capable of consistently distinguishing (Nakatsuhara, 2007) The researcher decided to have five levels in each marking categories including poor, bad, average, good and excellent with the score from 0 to 4 respectively The mark which students get will be calculated according to following formula:

The student‟ score = (Grammar x 2+ Vocabulary x 3+ Pronunciation x 2 + Fluency and Coherence x 3) / 20

The maximum score the student will get is 2 because the oral test in the final term test accounts for 20 % of the total score

Next, the marking scheme was drafted mainly with reference of CEFR The categories were drafted by stretching the CEFR level A1 and level A2 into 4 levels bad, average, good and excellent and level 0 was the lowest level describing no English competence The wording for each level was adapted from the CEFR according to the requirement of the task In order to explore the validity of the marking scheme, the draft marking scheme version was distributed to 6 teachers for judgments They were asked to answer the interview question which invited their comments on the marking scheme In general, the guiding interview questions focused on some aspects: criteria, levels of performance, descriptors and general look about marking scheme

Phase 2: Qualitative

After the completion of the first draft, the marking scheme continued to be given to two experts for review The two experts were also invited to give comments on the

marking scheme through the interview The objective of this process is to ensure the

internal coherence of the scale, wording them consistently and efficiently to make

them usable for examiners during the process of giving assessment

Phase 3: Quantitative

As the last phase of this study, the resulting marking scheme was pilot-tested 150 tenth-grade students participated in a “mock” speaking test and all their oral tests were audio recorded The recording took place at Cam Giang High School Two experts, three teachers and the researcher participated in the marking The students were told before the test about a given topic In the test, students had from 1 to 2 minutes to introduce themselves and 3 minutes to perform their topic Before assessing the audio-recorded students‟ performance, approximately 1 hour rater training was administered The assessing training was conducted as follows Firstly, the three teachers and researchers had a short discussion about the rating scale to establish the common ground It is emphasized that the level 5 in the marking scheme was meant to be the most satisfactory achievement level for a Cam Giang High School student, in order not to be confused with the top level described in any other proficiency tests Secondly, three teachers and researcher assessed 10 first audio-recorded and discussed the reasons for scoring When they reached the agreement, they continued assessing 140 audio-recorded individually Finally, the scores of teachers and experts were compared and discussed

2.5 Data collection instruments

2.5.1 The interview with the teachers and experts

In the present study, the interview was used The researcher used the interview questions designed and adapted from rubric checklist ELS 11.00 (NC State University) The yes-no questions were based on some specific categories regarding assessment that all interviewees were asked to comment on to assure the validity in

the research Six English teachers at Cam Giang High School and two experts take part in this research The interview guide contains 12 questions, and all the questions regarding the marking scheme The questions in the interview were placed into four sections: Section1 (from Question 1 to Question 4) four questions

of the first section are designed to ask for teachers‟ and experts‟ comments about the criteria of the marking scheme Section 2 (Question 5 and Question 6) find out the comments of the teachers and experts about levels of performance of the marking scheme Section 3 (three questions from Question 7 to Question 10) asked for teachers‟ and experts‟ opinion about descriptors Section 4 (three last questions from Question 11 to Question 13) investigated teachers‟ and experts‟ general ideas about the marking scheme and suggestions to make a better marking scheme (see appendix for interview questions) The teachers and experts were informed about the interview and were given interview questions sheet one week before the interview Each interview took about 8 to 10 minutes All the interviews were audio recorded by the teachers and experts themselves The teachers and experts are allowed to speak in Vietnamese so that the writer can elicit more information from the subjects The data from the interview was analyzed after gathering

2.5.2 Sample oral test

150 students took the pilot-test with the same topic about music which is one of the topic students learnt in the second semester The students were informed of the topic

by the time of this trail During the test they have 5 minutes for preparation and then

from 3 to 5 minutes to talk The students‟ responses were audio-recorded After that,

each student‟s audiotaped was scored following the marking scheme by three teachers and two experts

The content of the sample oral test is as follows:

Part 1: Introduction (1 to 2 minute)

Introduce yourself including name, class, family, hobby and etc…

Part 2: Topic (2 to 3 minutes)

What is your favourite kind of music?

+ Why do you like it?

+ Who is your favorite musician/ singer?

+ When do you listen to it?

For the researcher, the purpose of the oral test is to collect data on the scores made

by the teachers and experts when using the marking scheme The researcher wishes

to find out whether the marking scheme could be successfully used by the teachers

In other words, the researcher would like to investigate correlations among the teachers and experts This success is measured through the quantitative of the scores made by teachers and experts

2.5.3 Data collection procedure

The chronological steps of data collection procedure are summarized as follows:

- Study the literature review on available marking schemes and write the first version

- Conduct interviews to the teachers and experts to gather their opinions of the marking scheme

- Write the second version of the marking scheme based on teachers‟ and experts‟ responses

- Conduct pilot-test with the marking scheme with the participation of 150 tenth-grade students, two teachers, the researcher and two experts

- Collect all data for analysis

- Analyze the data for findings

2.6 Date analysis method

2.6.3 Data analysis procedure

Firstly, the data collected from the interview were then analyzed The data were processed and analyzed to find out the teachers‟ and experts‟ answers to many questions items

Secondly, the data were collected from the students‟ oral tests To see the relationship among teachers, between teachers and experts, among criteria, a Pearson correlation, which is a test to find out the correlation of ordinal variable, was employed Pearson‟s correlation is a statistical measure of the strength and direction of the linear relationship between pair data (which are score made by teachers, experts in this study) and it and it is denoted by r of which the value is

such that -1 < r < +1 The closer r is to +1 or -1, the stronger the

relationship While positive values indicate a relationship such that as values for one increase, values for the other also increase, negative values indicate a relationship such that as values for one increase, values for the other decrease Ratner (2008) provided guidelines for interpreting the strength of correlation:

r = 0 indicates no linear relationship or no correlation, i.e there is no tendency for one variable to either increase or decrease when the other increases

r values from more than 0 to less than +0.3 (from more than -0.3 to less than 0) indicate a weak correlation

r values from 0.3 to less than 0.7 (from more than -0.7 to -0.3) indicate a moderate correlation

r values from 0.7 to less than 1.0 (from more than -1.0 to -0.7) indicate a strong correlation

r = +1 or -1 indicates a perfect linear relationship: as one variable increases (decreases) in its values, the other variable also increases (decreases) via an exact linear rule

Furthermore, Pearson correlation test in SPSS also provided p-value along the r value This p-value was to decide whether there is any evidence to suggest that