Luận văn an evaluative study on the current final achievement tests for non english majors at quang ninh teacher training college

ABSTRACT The study was intended to give an cvaluation on the current final achievement tests for nen Linglish majors at QNI'TC from perspectives of the teachers and non- English majors

THE CURRENT FINAL ACHIEVEMENT TESTS FOR NON-ENGLISH

MAJORS AT QUANG NINI TEACHER TRALNING COLLEGE

ĐÁNH GIÁ BÀI KIỂM TRA CUÔI KỲ CHO SINH VIÊN

KHONG CHUYEN NGU TRUGNG CAO DANG SU PHAM QUANG NINH

M.A MINOR PROGRAMME THESIS

Kield: English Teaching Methodology Code: 60 140 111

HANOI — 216

MAJORS AT QUANG NINH TEACHER TRALNING COLLEGE

DANTI GIA BAT KIM TRA CUOI K¥ CIO SINII VIEN

KHÔNG CHUYÊN NGỮ TRƯỜNG CAO DANG SU PHAM QUANG BINH

M.A MINOR PROGRAMME TIIESIS

Field: English Teaching Methodelogy

Code: 6) 140 111

Supervisor: Dé Thi Thanh a, Ph.D

HANOI — 206

CANDIDATE?S STATEMENT

he

I, Vu Thanh Iloa, hereby certify that this minor thesis entitled

AN EVALUATIVE STUDY ON THE CURRENT FINAL ACHIEVEMENT TESTS FOR NON — ENGLISI MAJORS AT QUANG NINE TEACHER

Second, I would alsa like to acknowledge my debt of gratitude to the staff members

of the Faculty of Post-Graduate Studies and the lecturers al Universily of Languages

and International Studies, Victnam National University-Hanoi for their valuable lectures, which laid the foundation for this thesis and for their knowledge as well as

their sympathy

Third, a special (hank would also go to the leachers and the non-Fnglist majors at Quang Ninh ‘Teacher ‘Iraining College, who took part in the research Without their participation and cooperation I wauld not be able to complete this research paper

Fourth, I should be grateful to the librarians al ULIS for Licir constant help thanks

to which I was able to access to all materials needed to accomplish the thesis

Finally but importantly, I would like to express my appreciation to my family and

my friends who have been continuously giving me a lot of support and

encouragement for the fulfillment of this challenging work

Hanoi, 2016

ABSTRACT

The study was intended to give an cvaluation on the current final achievement tests for nen Linglish majors at QNI'TC from perspectives of the teachers and non-

English majors at QNTTC In addition, this study also investigated how the current

final achievement tests for non-tinglish majors at QNITC aligned to the CFR The study was carried out by means of two sets of survey questionnaires, and analysis of the current final achievement tests at QNTTC and the CEFR, using

The analysis of the alignment between the current final achievement tests at

QNTTC and the CEFR showed that most of the vocabulary and test items in these tests were in the ranges from levels A2 to B] and some of them were at level AZ and did not reach the target level, Bl Moreover, the current final achievement tesls al QNTTC differed from other international tests (PET) in tens of length ancl

constructs

The study will hopefully contribute to the test making at QNTTC by showing an example of evaluation on the current final achievement tests for non-English majors al QNTTC and the ahgnment between these tests and the CEFR

1.1 Rationale of the study

1.2 Aims of the study -scss sec

1.3 Research questions

1.4 8eope of the study

1.5 Significance of the study

1.6 Methodology

1.7 Outline of the thesis

CHAPTER 2, LEFHRATURE REVIEW

2.1 Basic concepts of testing/ Language testing .cssccsssussssesstesensenstineenees 6

2.2 The role of testing in teaching and learning,

CHAPTER 3 METHODOLOGY

3.1 Setting of the study

3.1.1 English teaching and earning o of non- Bnglish majors at QNTTC

3.1.2 Brief description of the materials used for non-Unglish majors at QNTT

3.1.3 The testing practice at QNITC

3.3 Data collection instruments

3.4 The alignrnent frameworik

3.5 Data collection and data analysis procedur

CHAPTER 4, FINDINGS AND DISCUSSION sccsestssavietsstiniseennenstne 30

41 The current tests øLONTTC,

41.1 Students’ comments on the existing tests

4.1.2 Students’ opinions towards the improvement of the tests

4.1.3 Teachers’ comments on the existing tests

4.1.4 Teachers’ opinions towards the improvement of the tests

4.2 The alignment between the current tests at QNTIC and the tests according to the CHER oececsesssseesseesstestimssieeeieste sie HH H120 tai 36 4.2.1 In terms of their constructs

4.2.2 In terms of contents 4.3 Summary of Chapter 4 `

5.2 Conelading remarks — - 5.3 Limitations and suggestions for further study - - 46 RPERENCDS cice HH tt Hưng nen 48

BĂNG CÂU HÓI ĐIỀU TRA DÀNH CHO SV KHÔNG CHUYỂN NGỮ VI

QUESTIONNAIRE FOR TEACHERS AT QMKTTC óc sec

AFPENDIX 4

t2

„

LIST OF ABBREVIATIONS

QNTTC Quang Ninh Teacher Training College

T1 Teacher training institution

CEFR The Common Furopean Framework of Reference for

Languages: Learning, Teaching, Assessment

TOEFL Testing English as a Foreign Language

TELTS Taternational English Language Testirys System

TOEIC ‘Test of Linglish for International Communication

RMM Pearson Reading Maturity Metric

vil

LIST OF TABLES

‘Table 2.1: Common Lturopean 1rarmework of Reference (CL¿L/)

Table 4.1: Analysis of the depree of difficulty of the test items in phonetics

in Test number 1

Table 4.2: Analysis of the degree of difficulty of the test items in phonetics

in Test number 2

‘Table 4.3: Analysis of the degree of difficulty of the test items

in grammar and vocabulary in Test number I

Table 4.4: Analysis of the degree of difficulty of the test tems

ñ grannmar and voeabulary in Test tuanbef 2 s1 22g12

Table 4.5: Analysis of the degree of difficulty of the reading texts in Test

`

Table 4.8: The comparison between Tests number | and 2

Table 4.9: Comparison of the length between the current tests at QNTTC

and the reading writing tests of FI7T i ni

vill

2 A3

difficulty’ difficulties when doing the test

interests in test items

©oImunienis and susgestions

making the English achievement test altitudes toward the current lest

comments on the current test

CHAPTER 1 INFRODUCTION 1.1 Rationale of the study

Nowadays English has become increasingly important as a means of global communication Tn the process of global imlegration, Vielnam has realized the importance of English Janguage leaming and teaching; Thus, English has been widely used in many fields and it has become a compulsory subject at many

schools and umversilies

Quang Ninh Teacher Trammug College (QNTTC) was established in 1959 and now

it is considered the oldest institution in providing undergraduate teacher education

in Quang Ninh In 1991, the organization was restructured from four provincial teacher training institutions (TTT): Quang Ninh Barly childhood TTY, Quang Ninh

Primary Tl, Quang Ninh Education Management ‘TTI and Quang Ninh Low

Secondary TTI [aving awareness of the importance of English, the college authorities have paid due allention to the matler of improving the quality of teaching

and learning linglish,

Jn the teaching and leaming in general and in the teaching and leaming foreign language process in particular, testing and assessment play a significant role The importance of language testing is recognised by virtually all professionals im the language education Teachers should not be confined to imparting teaching and learning with testing, Testing is of special importance in educational system that is lighly compelilive as Lesting is nol only an indircet slimmulus io earning, but plays a crucial role in determining the success or faihwe of an individual’s career with direct implications for his future career Tu the World Yearbook of education 1969, Lauwerys and Scaton state: “Thus, testing is an important tool in educational research and for programme evaluation, and may even throw light on both the nature of language proficiency and language learning.”

Nea (1997:1) shares the same idea; “Tests are assumed to be powerful determiners

of what happens in classrooms and it is commonly claimed that they affect

teaching and learning activities both dirovily and indirectly”

It cannot be denied that testing is an important part in teaching and leaming

process, but has it been paid enough attention yet? Having taught English for students at a high school and then at QNTTC for 5 years, the author of this study

has designed tests for both English majors and non-Fnglish majors She has also

administered and marked these tests Her teaching experience shows that there still remain some problems that need to be solved such as the test content, the gap between what is tested and what is taught, the reuse of tests from years lo years, from classes to classes As a result, tests may lack validity and reliability, Hughes

(1990:1) also gives another comment on recent language testing: “It cannot be denied that, a great deat of langung

testing is of very poor quality Too ofien language tests have a harmful effect on teaching and learning and too often they fail to measure accurately whatever it is they are intended to measure” Moreover, teachers [requently lack formal training in cducational measurement, techriques

and they tend to be alienated from the testing process,

‘A well designed test is necessary for all language learners even though they have different levels On the ground of the problems already mentioned, it is thought that achicvement tests for the non-English majors at QNTTC should be designed to assure the accuracy and faimess for all students so that they can produce good backwash in the teachers’ teaching and give students satisfaction and encouragement int study Those reasons above encourage me to conduct the surly

“An evaluative study on the current final achicvement tests for non-Knglish

majors at Quang Ninh Teacher Training College”

1.2 Aims of the study

The study ams at evaluating the current final achicvernent tests al QNTTC To

achieve this aim, the following objectives are established:

1 To cvaluatc the curent final achievercut tcsts for non-English majors from

perspectives of the teachers and non-linglish majors at QN1'IC

tạ ‘Yo investigate the alignment of the current final achievement tests at

QNTTC to The Common European Framework af Reference for Languages: Learning, ‘leaching, Assessment (CHER)

1.3 Research questions

In order to achieve the above aims of the study, the following questions will be addressed

1 How do English teachers and non-English majors at QNTTC evaluate the

current, final achievement tests for non-English mayors at QNTTC?

2 How does the current lest alga to the CEFR?

1.4 Scope of the study

As the tile “An evaluative study on (he current final achievement tesis for non-

English majors at Quang Ninh ‘Teacher ‘raining College” suggests, this study is intended to Louch upon some following issues:

- This study 1s only aimed al evaluating the existing testing situations al QNTTC

from two stakcholders, the teachers and the students

- This study is limited to evaluate the final achievement tests for non-English majors

- This study focuses on evaluating the constructs of the final achievement tests at QNTTC and the tests based on the CEFR (PET)

- This study is a detailed survey at ONTTC Therefore, the findings of the study are nol intended to be generalized to other schol contexts Tndeed the findings may rot apply beyond the actual participants in this particular study

1.5 Significance of the study

‘The findings of the thesis serves as a back-up for the improvements of the tests for non-English majors at QNTTC Practically the findings are beneficial for both

3

teachors and learners at QNTTC from the experience of reflection It is also hoped that the thesis will be of contributions towards the development of the testing situation at QNTTC in general and the testing situations for non-Finglish majors al QNTTC im particular

1.6 Methadalogy

‘The above-given aims are to be achieved by means of:

(1) A survey questionnaire carried out on 30 non-Linglish majors at QN'T'TC to

investigate their comments of the existing final achievement tests for non- English majors to gol their evafuation as well as thoir suggestions for

improving the testing situations and language tests at QNITC

(2) A survey questionnaire carried out on 10 teachers of the English Faculty of QNTTC about their comments on the final existing final achievement tests

for non-English majors and (heir suggestions lo improve thơ siinaliơn

(3) Analysis of the contents and constructs of the cient final achievernent tests at QNT'TC to find out the alignment of these tesis to the CHFR

Besides the survey and analysis, more information and data needed for the study were gathered by other methods such as formal and informal discussions with students and teachers as well as critical reading, Moreover, the study employed a combination of qualitative and quantitative methodology that includes cross- tabulation data and statistical analysis of the results of the survey questionnaire and

the analysis of degree of difficulty of the current final achievement lesis al QNTTC

in accordance with the CLFR

1.7 Outline of the thesis

The author divided this study into five chapters

- Chapter 1: Introduction this chapter provides the author’ reasons for choosing, the topic, aims, research questions, svops, significance, methodology and outline

of the study

- Chapter 2: Literature review, this chapter is the most theoretical one, looks at the background knowledge on language testing such as the basic concepts of

language lesling, the role of testing, ly

s of tesls according lo tes! purpose,

criteria of a good test as well as the CHER, target language for non-English

majors

- Chapter 3: Methodology, this chapter discusses on methodology, presents the deep analysis of the setting including English teaching and leaming at QNTTC, brief description of the material used for non-English majors and the current testing situations at QNTTC; the informants; data collection instrument and data collection and dala analysis procedure

- Chapter 4: Findings and discussion, discusses (he major findings of the thesis A brief discussion about the actual English teaching and learning context and the

current tests at QNTTC and the alignment between these tests and the tests

aveording to lic CRFR

- Chapter 5: Conclusion, thc aulhor sheds the mantle of reviewing the sludy and

suggesting further research

CHAPTER 2 LITERATURE REVIEW

2.1 Basic concepts of testing/ Language testing

‘The importance of language testing cannot be denied and is recognized by all

professionals Language tesls are considered us valuable tools in providing

information vonceming language teaching ‘They provide evidence for the results

of learning and instructing the effectivencss of caching as well as information for

‘both teachers and students to make decisions

For these reasons, testing should be part of language teaching and one of the main aspects of methodology Many definitions of testing from different points of view have been given

According to Allon (1974:313), a test is a measuring device which we use when we want to compare an individual with other individuals who belong to the same group Carroll (1968:40) defines: “A psychological and educational lesL is a procedure designed to elicit certain behavior form which one can make inferences about certain characteristics of an individual” Brown (1971:8) has a different point of view to define # test as “a systomatic procedure for measuring an individual’s behavior” Peny Ur (1996:33) provides the following definition of a test: “Lest is

an activity whose main purpose is to convey (usually to the tester) how well the testecs know or can do something” Moore (1992:138) proposes; “evaluation is an essential tool for teachers because it gives them feedback conceming what the students have learned and indicates what should be done next in the leaming process, Hvaluation helps us to understand students better, their abilities, interests, attitudes, and needs in order to better teach and motivate them” Ilowever, Brown (1994a:373) stresses that tests are seen by leamers as dark clouds hanging over their heads, upselting them with thunderous anxiely as they anticipale the lightaing bolls

of questions they do not know and worst of all a flood of disappointed if they do not make the grade Read (1983:3) shares the ideas saying a language test is a sample of linguistic performance or a demonstration of language proficiency Nea (1999.2)

6

also states that “Test most commonly refers to a sct of items or questions designed

to be presented to one or more students under specified conditions” Broughton (1990:1) thinks the word “tes!” is much more complicated with al leash Uhre quite distinct meanings The first meaning refers to a carefully prepared measuring, instrument The second one refers to what is usually “a short quick teacher-devised activily” carried oul in the classroom and used by the leacher as the basis of ant on going, assessment Assessment is the process of documenting knowledge, skills, attitudes and beliefs, usually in measurable terms The poal of assessment is to inake improvements, as opposed to simply being judged In an educational context, assessment is the process of describing, collecting, recording, scoring, and interpreting information about learning It may include a test, but also includes inethods such as ob:

alions, interviews, behavior mumitoring, cle The last one is

that “of an item within a larger test, part of a test battery, or even sometimes what is

often called a question in an examination” Harrison (1983a:1) notices that a natural

extension af classroom work, providing teachers and siudenls with useful information that can serve as a basis for improvement and a test is necessary but unpleasanl imposition from outside the classroom Thal means test is a useful tool

to measure learners’ ability in a certain situation specially in a classroom,

In short, testing is an effective means of measuring and assessing students’ language knowledge and skills The meaning given to the term “testing” is defined differently by test researchers and can be understood as the use of means requiring, students to respond to questions or tesis that are designed to focus on a particular aspects of learning and also perceived rather broadly as a process of assessment, consisting of different stages such as preparation, data collection and evaluation

2.2 The role of testing in teaching and learning

In the past, testing and teaching tended to be separated Many applied linguistic

researchers and professional designers have shared (he idea thal Ianguage testing

~

plays a decision part in language teaching in general and language leaming in particular

Heaton (1988:5 ) states dat “teaching and lesting in some ways are so interwoven and independent that is very difficult to tease apart Both testing and teaching are

so closely interrelated [hal iLis virtually impossible to work im either field without

being constantly concemed with the others”

Heaton (1988:5) also emphasives thal tesls may be constructed primarily as devices to reinforce learning and motivate the students or as a means of assessing the students’ performance in the language In the former case, testing is geared to the leaching, whercas in the latler case, teaching is often geared largely to the testing

However, (esting has both good and bad effects on teaching Hughes (1989-1) shares this point of view: “Backwash can be harmful or beneficial” He states that

if the content of the test is in accordance with the content of teaching and method

of the course being followed, the lesl can be of beneficial effect to the leaching,

process, Otherwise, itis likely to have bad effect,

Ta short, lesling and teaching activities carmoL be separated from each other and from the programme or from the objectives of the course Testing may influence

teaching in either good or bad ways

2.3 Types of tests according to test purpose

Language lusls are developed basing on so many purposes thal there are many types of language tests Since language tests have different purposes and the information obtained from tests is used for different types of decisions, let us

consider a brief description of some types of tests according to test purposes

objectives for purposes of correcting deficiencies “before it is too late”

In addition to it, Brown (1994b:259) gives another comment on this type of tests as follows “A diagnostic Lost is designed lo diagnose a partigular aspect of a particular language.” Moreover, Llarrison (1983b) also states that this kind of tests is used, for example, at the end of a unit in the course-book after a lesson designed to teach

one particular poi

From these delinitions, iis clear thal the mam purpose of diagnostic tests is to identify test-takers’ strengths and weaknesses in the language, as well as to give explanations Lo the problems, aud whal teatment can be assigned to foster

achievement by promoting strengths and climinating weaknesses

2.3.2 Placement tests

According to Hughes (1990:14): “placement tests are intended to provide information which will help to place students at the stage of the teaching program most appropnate to thei abililies Typically, they arc used Lo assign siudonts to

classes at different levels.” In other words, it is used to assign students to classes

according to their abilities so that they can start a course at approximately the same

level as the other students in the class So as a nulc, the results of placement tests

are needed quickly so that teaching may begin (Ilarrison, 1983b :4)

programme (and its objectives teaching and materials) Likewise, a proficiency test looks to the future situation of language use without necessarily any reference

to the previous process of Leaching (McNamara, 2000-7)

Hughes (1990:9) states that “Proficiency tests are designed to measure people's ability in language regardless of any training they may have had in that language.” That is to say the content of a proficiency test is not based on the content or objectives of any language course test takers may have followed It is rather based

on specification of what they have to be able to do in the language to meet the requirement of their future aims

Other test specialists, such as Carroll and Hall (1985), Harrison (1983a) and Henning (1987) share the same view that proficieney test helps both teachers and

learners know whether the learners can be able to follow a particular course or they

have to take some pre-departure training to some other popular tests such as

TOEFL, TELTS, which are used to test sladsnls’ proficiency for their study in some English speaking countries In Vietnam, proficiency tests are of different

A2, BI, B2, Cl and C2 according

to the CEFR or the Vietuam’s English competence Inmnework

levels namely A, B, C in the past and now Al,

2.3.4 Achievement tests

According to Hughes (1990:10), “in contrast to proficiency tests, achievement tests are directly related to Ianguage courses, their purpose being to establish how successful individual students, groups of students or the courses themselves have been in achieving objectives” Achievement tests are commonly used at school of all levels and of great importance in evaluating language knowledge and skills students have acquired during the English teaching learning process

'MeNamara (2000:6) states thal “achievement lest are associated with the process

of instruction, Achievement tests accumulate evidence during, or at the end of a

course of study in order to see whether and where progress has been made in terms

of the goals of learning Achievement, tests should support the tcaching lo which

10

they relate, An achievement test may be sclf-cneloscdl in the sense that it may nơt bear any direct relationship to language use in the world outside the classroorn (ït aay focus on knowledge of particular points of grammar or voesbnlary, for example).” Brown (1994b:259) shares McNamara’s viewpoint, “an achievement

test is related directly to classroom lessons, units or even a total curriculum.”

Achievement tests are divided into two basic types according to the time of administration They are namely progress achievement tests and final achievement tests

(1) Progress achievement tests

Progress achicvement tests (criterion-referenced or objective-refereneed), as the name suggests, are intended to measure the progress that leamers are making Since “progress” in achieving course cljeclives, these {ests should be related to objectives These should make a clear progression towards the final achievement test based on course objectives They are usually carried out to measure the extent which students have mastered from whal has been taught m the classroom

Thanks to the results of the achievement tests, teachers will be able to find out and

diagnose areas not properly mastered by students during the course, which need remedial action Moreover, these tests also provide students with a good chance to stimulate learning and perfonning the largel language they have learnt in a positive and effective manner with confidence ‘this is also considered a preparative step to

make students familiar with the test

(2) Final achievement tests

J'inal achievement tests are given at the end of the course ‘They may be written or administered by ministries of education, offivial examining beards, ar by members

of teaching institutions Clearly, the content of these tests must be related to the

courses with which they are concerned, but the nature of this relationship is still a

matter of disagreement amongst language testers Tt is a good chance for teachers

to judge the degree of success of their teaching and identify students’ weakness

11

THughos (1990:10) điyidod them imto two kinds depending on different approaches used ‘Ihe syllabus - content test is the one in which its content should be based direetly on a detailed course syllabus or on books and olher materials used Whereas the syllabus objective test is used to test objectives so it is good to measure students’ ability to meet course objectives Ilowever, it is bad as they work agains! the leaching because (his approach copes wilh testing problems ralher

than what students have achieved

24 Crileria of a good Lest

As mentioned before, testing may have good or bad effects on teaching so before

making tests, test designers often ask themselves these questions: How do we

design a test that can test all the language skills? Who is it for? Ls it suitable for all

of them? What is it meant to test? How do we know that it is a good one? Does this

test get the target level?

Tn order to construct a good test, teachers have to take into consideration the

various (actors such as the purpose of the esl, the course content and above all students’ background and so on In addition to these factors, good tests must

possess some characteristics namely validity, reliabilily, practicality and

discrimination According to a mumber of leading scholars in testing as Valotte

(1977), Larrison (1983a), Carroll and Lall (1985), Llenning (1987), Lleaton (1988),

Hughes (1990) and Brown (19944) all good losis possess all these four

characteristics ‘hese characteristics will be critically reviewed bellow

2.4.1 Validity

Validity is cerlainly the most important, single characteristic of a tesL TC not valid,

even a reliable test does not worth much Carmen (1995) defines that: “a test is

valid if iL measures what you wanl lo measure” Hughes (1989) also shares the same idoas: “a tost is said to have validity if it measures accurately what it is

intended to be measured”, According to Aik’s opinion (1983:2), “a test is said to

be valid if il is relevant to the aims and purposes of the areas of learning on which

12

it is sct”, In this sense, validity of the test and purposes of the course syllabus arc closely related

‘Yhere are different kinds of validity such as face validity, content validity, criterion-related validity, construct validity, empirical validity, predictive validity, etc but among them content validity, fave validity and criterion-related validity are

the most important

Content validity refers to the correspondence between the content of the test and the content of the materials to be tested OF course a test carmot include all the

clements of the content to be tested Nevertheless, the content of the test should be

a reasonable sample and representative of the total content to be tested In Read’s opinion (1983-6), the most relevanl type of validity for classroom tusling is content

validity, which means that the content of the test should reflect the content and

abjectives of the syllabus that is being followed According to Anastasi (1982-131)

defines confer validity as “

nlially the systematic examination of the test

content to determine whether it covers a representative sample of the behavior

domain fo be measured.” She shows a fact of useful guideline for establishing

cuntent validily:

- The behavior domain to be tested must be systematically analyzed to make

certain that major aspects are covered by the test items with correct proportions;

- ‘the domain under consideration should be fully described in advance, rather than

being defined after the test has been prepared,

- The conient validity depends on the relevance of the individual test relevance of

ite content

From the above concepts, il is obvious thal, lhe contents of a tesls are main concern

in achieving its content validity

that face validity is not validity in the technical senso; it refers, not to what the test actually measures, but to what it appears who take it, the administrative personnel

who decide on iis use and other Lechmically untrained observers Face validily is

supported by the judgment that a test is appealing to laymen students, administrations, etc Tughes (1990) in “testing for Language Teachers” states: “a

tosh is said te have [ace validity if it hooks as if i supposed to mcasure” Tn olher words, tests should be based on the course content and methodological teaching

approaches

Criterion-related validity refers to the correspondence between the results of the

tesL in question and the results obtaiued from an outside criteriou The oulside criterion is usually a measurement device for which the validity is already

established In contrast to face validity and cantent validity, which are determined

subjectively, crilerior-related validity is cslablished quict objectively

Tn short, vahdily is the “must” for testers to take imo consideration when they

construct a language test

2.4.2, Reliability

Reliability is one of the most important characteristics of all tests in general, and language tests in particular Tn fact, an unreliable test is worth nothing TL is of primary importance in the use of proficiency tests for both public achievement and classroom tests An appropriateness of the various factors affecting reliability is

important for the teachers at the very outset, since many teachers tend to regard

tests as infallible measuring instruments and fail to realize that even the best test is

indeed a somewhat imprecise instrument with which to measure skulls

The two things need to be considered about reliability are the consistency of performance from candidates and scoring The former is affected by several factors

such as the number of questions, test administration and test structions This is

defined by Moore (1992:110) that “reliability refers to the consistency with which

4 Measurement device measures some targel behavior or trail, To put il, another

14

way, it means the dependability or trustworthiness of the measurement devico” Likewise, Bachman (1990:24) describes reliability as ‘‘a quality of test scare” Mor instance, a multiple-choice test would probably yield different scores from one administration to another, and would thus be extremely unreliable, Moreover, administration, that is the circumstance under which a test can be taken, affects test results a lol Tt involves in such problems as liming, testing conditions, observalion

of control of testes doing the test, scoring, etc

Finally, it should be noted that a test could be reliable without possessing validity Tlowever, reliability is clearly inadequate by itself if a test does not succeed in

ineasuring what is supposed La measure

2.4.3 Practicality

Tt would be nol good if test constructors arc to scparale testis’ validily and reliability from practicality Practicality refers to facilities available to test developers regarding both administration and scoring procedures of a test In

Hamison’s opinion (1983813): “a vahd and rehable tesl is of hile nse if it docs

not prove te be a practical one A test ought to be practical - in the sense of

financial limitations, time constrains, ease of administration, scoring and

Bachman and Palmer (1996-39) statc that “the relalionship belween the recourse

that will be required in the design, development, and use of the test the resources that will be available for these activities” They link practicality to “the ways in

which the tes will be implemented ma given situation or whether the test will be

used at all.”

In conclusion, a test has practicality if it does not involve much time or money in constructing, implementing and scoring it

15

2.4.4 Discrimination

Another important feature of a test is its capacity to discriminate among the

different candidates and to reflect the differences in the performances of the

individuals in the group Generally speaking, all assessment is based on comparison either between one student and another (norm-reference comparison)

or between the student as he/she is now and as he/she was earlier (Harmson, 1983b) This is truc for both teacher-made tests and standardized tests A good

language test should be able to discriminate between a student and others taking

the same test So if a test is either too easy or too difficult, it cannot realize its

purpose of discrimination between candidates According to Heaton (1988:165),

70% means nothing at all unless all the other scores obtained in the test are known Furthermore, tests on which almost all the candidates scare 70% clearly fail to

discriminate between the various students

Tn order to have the discrimination feature, a lest must have a scale ranging from extremely easy items to extremely difficult items (extremely easy items, very easy items, easy iletms, fairly easy ilems helow average difficult level, items of average difficult level, itoms above average difficult level, fairly difficult itams, difficult items, very difficult items and extremely difficult items)

2.4.4.1 Item difficulty

According to Hai (1.999:26), difficulty level relates to show how easy and difficult

the item is from the point of view of the students who took the test It is important

since test items that are too easy (that all students get right answers) can tell

nothing about differences within the test population On the other hand, Ilenning (1987-49) states thal perhaps the single most imporlanl characteristic of an item lo

be accurately determined is its difficulty

Another argument for including items of difficulty levels is the students’ stimulus motivation It has been assumed that while the difficult items are necessarily

included in the tesL in order to create the motivation among (he good students,

16

“the invlusion of very easy items will encotrage and motivate the poor students”

suspcol nrd may need to be changed (Baker, 1989)

2.5 The CEFR

2.5.1 What is the CEFR?

The Common Huropean Framework of Reference (CHFR) is a framework, published by the Council of Europe (2001), which describes language learners’

alility in terms of speaking, roading, listening and writing al six reference levels

2.5.2 Levels of the CEFR

In November 2001, an European Union Council Resolution recommended using the CUFR to set up systems of validation of language ability ‘The six reference levels (AI, A2, BI, B2, Cl and C2) are becoming widely accepted as the European standard for grading an individual's language proficiency

The CEFR divides learners mto three broad divisions (Basic User, Tndependent User and Proficient User) that can be divided into six mentioned levels For each

level, it describes what a Jeamor is supposed to be able to do in reading, listening, speaking and writing

2.6 Target level for the non-English majors

According to the Decision 1400/QD-TTg dated September 30" 2008, non-English inajors al college and universily must get KNT.NN level 3 (B1) of English to be graduated Based on the CHER assessment (CEFR, 2001), the students get level Bì, that is to say, they:

* Can understand the main points of clear standard input on familiar matters regularly encountered in work, school, leisure, ef:

* Can understand the main point of many radio or TV programmes on current affairs or topics of personal or professional interest when the delivery is relatively slow and clear

* Can understand texts that consist mainly of high frequency everyday or job- related language

* Can understand the description of events, feelings and wishes in personal letters

© Can deal with most situations likely to arise while traveling in an area where

the laygnage is spoken

* Can produce simple connected text on topics that are familiar or of personal TiderosL

© = Can enter unprepared into conversation on topics thal are familiar, of personal interest or pertinent to everyday life (e.g family, hobbies, work, travel and

current events)

* Can connect phrases in a simple way in order to describe experiences and events, my dreams, hopes and ambitions

* Can briefly pive reasons and explanations for opinions and plans I can

narrate a story or relate the plot of a book or film and describe my reactions

® Can describe experiences and everts, dreams, hopes and ambitions and

briefly give reasons and explanations for opinions and plans

18

* Can write personal letters describing experiences and impressions

© Can write simple connected text on topics which are familiar or of personal

interest

The 6 levels of the CEFR aligned with international English tests can be

summarized as follows:

Pass | Score | Score | Score | Score | Score Pass Pass

Table 2.1: Common European Framework of Reference (CEFR)

2.7 Review of related studies

Specifying the relationship between a test product and the CEFR is challenging

because, in order to function as a framework, the CDFR is deliberately

underspecified (Davidkon 4 Fulcher, 2007; Milanovic, 2009, Weir, 2005)

Establishing the relationship is also not a one-off activity, but rather involves the

18

accumulation of evidence over time (¢.g it nocds to be shown that test quality and

standards are maintained)

As a result, so far not many studies have been conducted to find out the alignment

of different kinds of tests to the CEFR In the research memorandum on “The

Association between TOHFL iBT Test Scores and the Common European

Tramework of Reference (CEFR) levels” by Spiros Papagoergiou, Richard I

Tannnebaum, Brent Bridgeman, Yeonsuck Cho (2015), the authors noted the

content alignment of the 'OEFR iB'I' to an extemal language framework such as the

CEFR

2.8 Summary of Chapter 2

Chapter 2 has briefly discussed the basic cancepts of language testing The Chapter

has been concerned with the issues relating 1o different test types according to the test purpose Besides this, the author has introduced the characteristios of a good test Finally, the CEFR, the target level for the non-English majors and the related

studies lo the CEFR have also been introduced

20

3.1 Setting of the study

3.1.1 English leaching and learning of non-English majors at QNTTC

QNY'TC is considered the oldest institution in providing undergraduate teacher education in Quang Ninh Dnglish has been taught here for both Iinglish majors and

non-English mayors since 1982

During the non-Fnglish majors’ course of leaming Bnglish ((hree semeslers with

105 lesson periods or 10 credits in total), they are required to do three tests in accordance with the first, second and third semester respectively After finishing, this course, the students are supposed lo gel evel BT To moet the demand, many books were used First, Ilead Way was chosen as the course book, then it was New Head Way Sometime later, Lifelines replaced New Head Way and now New Culling Flge, Pre-Inlernediale is chosen Besides the materials and teaching and leaming condition, the teaching staff are also improved both in quality and quantity 3.1.2 Brief description of the materials used for non-English majors at QNTTC

‘As mentioned earlier, to get Level Bl they have to take an English course consisting of 3 semesters, the course book “New Cutting Edge - Elementary” and

“New Cutting Hdge - Pre - Intermediate” are now chosen as the main ones According to the authors (Sarah Cunningham, Peter Moor and Jane Comyns Carr),

“New Culling Fdge - Flementary” takes students from Al to A2 level and “New Cutting Edge Pre-Intermediate” takes them from A2 to Bl level of the CEFR including these features: with a task-based learning approach, the main objective is

for students to use the language that they know in order lo achieve a parlicular

21

communication goal These books have a comprehensive syHabus with thorough grammar, vocabulary and skills work, systematic vocabulary building which focuses on high-frequency, useful words and phrases and clearly-structured tasks

to encourage student's fluency and confidence These books are to provide a stepping stone to enable a student to cross from the knowledge of General English toa position where he can handle the sort of IexIbook and instructions be will mec

at college and in their future career

‘the key points of 8 modules (New Cutting Hdge Pre - Intermediate) for third semester are described as in Appendix I

3.1.3 The testing practice at QNTTC

English is a compulsory subject in the whole educational programme therefore, test

actvilics are paid much more considerations At the examinations, the students arc

asked to sit in the alphabetical order and each one is given a different test paper so

‘that they can hardly copy from one another and do the test individually

Besides this, objective tesis such as multiple choice, mistake correcting, senlence building and questions and answers are used in order to get the highest reliability and discrimination among the test takers In general, QN'I'I'C Linglish tests look good and reasonable for students Tn addition to that, to make it casicr for (cachers

to score the examination papers, separate answer sheets are provided for the test

takers to write down the answer and their names in these answer sheets are cut out

to make it more equal for them

The non-English majors at QNTTC have to take 3 tests Test 1 with SO items is given al the end of the first semester when students have learned 13 Modules of the New Cutting Edge — Elementary book in 90 minutes Test 2 is a test with 10 items

which is carried nut in 60 mimules al the end of the second semester afler Ihey have

finished the next 2 Modules of the New Cutting Edge — Elementary and 7 Modules

of the New Cutting Hdge Pre-Imermediate After the last 8 Modules of the New

Cutting Edge — Pre-Intermediate, students have to take the fimal achievement tcst (Test 3) with 40 items in 60 minutes

3.2 Informants

The informants of the study were selected from the student and teacher population

of QNTTC The students were selected on the account thal they were the larget

learners of the course who took the final achievement tests ‘They had taken all the

three achievement tests by the time they were asked to answer the questionnaires

The teachers who were taken as the informants of the study had at least three-year

teaching experience and had ever made final achievement tests for non-Linglish

majors

* The students

The informants of the study are the non-English majors in the college, ranging from

19 to 21 Most of them have been learning English for 7 to 10 years (77%) while

just 3% have studied English from 4 to 6 years and no one has learnt English for

less than 4 years 20% of the students have been learning English for more than 10

years ‘I'his means that they started learning Iinglish sce they were in primary

school and they all come from urban areas Therefore, their English proficiency is

somichow betler than students from rural paris duc to the avalabldy of

opportunities to attend part-time English courses and chances to contact with English speaking people However, they are non-English majors so they are not

really interested in learning English and unmotivaicd ANbough they lend lo regard English as less important than other subjects and they study Hnglish in order to pass the exams, 40% think thal English is quile imporlanl However, only 5 studerts (17%) realize the importance of English and they think that English is national language and they can be used in communication and their lives Up to 37%

consider English not very important or even not important at all (7%) because they

do nol need it in their Future job

23

* The teachers

‘There are 12 teachers of English currently working at QNTTC and 10 of them attended in this study Of whom, three fourths are MA Degree holders or are

studying tn gol MA degrees In general, they are nol very young (rom 27 to 50

‘years old) aud half of them have been working, as a teacher for more than 10 years and only 10% have beer a leacher for 1 lo 3 years 20% had 4 to 6 years? teaching

experience and the same number of teachers spent 7 to 10 yoars teaching English

3.3 Data collection instruments

‘This research is conducted by using surveys, the current final achievement tests for non-English majors at QNTTC, the CEFR and some softwares such as Pearson,

Tim

Tn examining the actual English testing situation at QNTTC, survey questionnaires were used ‘[hey were used as the main instruments for collecting the data in this study because by using questions the researchers can collect information quickly from large numbers of respondents To find out the ahgnment between the current

final achievement tests for non-Unglish majors at QNI'T'

Two sets of survey questiormaires are conducted with the assistance of TÔ teachers:

of the Knglish Faculty and 30 non-English majors

‘Yhe first questionnaire with 12 questions was administered to 30 second-year students of non-English majors at QNTTC:

- The first question was written to find out the students’ time of learning English

- The second question is to investigate whether the students think English is

important ta their future carcer or nol

- The third question aims at finding out how well they complete a test

24

- The fourth question is to cxamine the difficulty/difficultics the students have when they do a test

- Questions 5 - 7 aim at eliciting their interest in some test items (reading and

answering questions, making up sentences and correcting the mistakes)

- Questions 8, 9 and 11 were written to investigate their attitudes toward the current

test

- Question 10 was used lo gel the sludents’ opinions about correcting their work

right after they have done the test

- Question 12 was wriften to find out other opinions of the students about the current test,

The second survey questionnaire was administered to 10 teachers of the English Faoulty at ONTTC including 12 questions:

- Question | is to investigate the teachers’ teaching experience

- Question 2 was wrillen to find out the necessily of an English achiovernent ical at

the end of each semester

- Questions 3 - 5 aim at investigating the teachers’ test making as well as their

reasons for their answers

- Questions 6 - 8 were written to investigate the teachers’ opinion about the content, the marking seale, the time allowance of the current English achievement test [or

non-English mayors with their own reasons,

= Question 9 - 11 were used to get their opinions and reasons for changing the construction of the current test (the content, the marking scale, the time allowance)

- Question 12 is to investigate the teachers’ further comments and suggestions

towards the THIĐTOVGHIGIHL of the current lesh

Those Lwo sols of questienmaire can be seen i Appondixes 2 and 3

> The tests

The two final achicvement tosts uscd for this thesis wore choscn at random and supposod to be the latest and equivalent to other final achievement tests for non-

English majors al QNTTC These losis consists of 4 scclions as follows

Section 1: Phonetics (1.25 points)

- Item format: Multiple choice questions

- Number of items: $

- Scores: 0.25 points for each item

Section 2: Grammar and vocabulary (5 points)

Part 1: Choose the best answer

- Item format: Multiple choice questions

- Number of items: 15

- Scores: 0.25 points for each item

Part 2: Identify the mistakes and correct

- Item format: Q & A

- Number of items: 5

- Scores: 0.4 points for each item

Section 3: Reading comprehension (2.5 points}

~_ 8eorœs: Ô.25 poinfs for cach item

Section 4: Writing (1.25 peints)

Use the set of words and phrases to make meaningful sentences

- Tem formal: Q&A

- Number of items: $

- Seores: 0.25 points for cach item

The chosen tes!

s can he seen in Appendixes 4 and 5

4 The softwares Bnglishprolile, Pearson and Fstin

The software Englishprofile, Pearson and Estim were used to make the evaluation more reliable ‘The author used these softwares as the instruments to evaluate vocabulary, prammar as well as reading and writing items to find out how they are aligned to the CEFR The author searched the vocabulary and grammar ilems iu the software Lnglishprofile to find out the levels they were at With the help of the softwares Pearson and Estim, the reading texts were also checked to see their

degrous of difficulty as well as (he fovels of difficulty of the questions

4 The CEER

To investigale the aligtmnent of the reading Lexts to the CRFR, the author with the help of other English teachers analyzed the test items to see whether they test the skills described at the descriptors of the CETR ar not (see Appendix 6)

3.4 The alipoment framework

The alignment between the cisrent final achievement tests al ONTTC and the

CEFR (tests at Level BL PET) can be evaluated in terms of

- The constructs of the two lests

- The contents of the two tests

- ‘The length of the two tests

- The degree of dititoulty of the final achicvement tests at QNTTC based on the CLR

3.5 Data collection and data analysis procedure

‘Yo accomplish the purpose of the study, the following procedures were pursued:

First, two sets of survey questionnaires were given to 10 teachers of English at the Tinglish Faculty and 30 non-Inglish majors at QNTTC The questionnaire for the teachers was administered al the break time of the English group’s weekly meehing Vor the students, it was administered at the closing of the class At the time they

were asked to answer the questions, the students had taken the current final

avhicvement tests forsion-English majors at QNTTC

Before administrating the instruments, the purposes and the importance of the study were clarified to the participants ‘They also received oral instruction about how to complete the surveys Each survey was gathered after 30 minutes of administration The data oblained from these two surveys were imported into the computer and treated in Hxcel ‘Ihe data were then subjected to some descriptive and inferential

statistics For accurate and effective interpretation of data, the author used

frequencies and sorting to find out the percentage that indicates more emphasis given to each item,

Kinally, to find out the alignment between the final achievement tests to the CLER,

these tests are taken into consideration in terms of their constructs, contents and

length To make the evaluabon more reliable and valid, expert judgments were used Two English teachers who were considered to be good at testing and assessment were asked to help the author These teachers used their experience and the sofiwares Englishprofile, Pearson and Fatim lo evaluate the tesis hy checking the level each item (vocabulary, grammar) was at as well as the degree of difficulty

of the reading text The alignment of the vocabulary and prawmar items were searched using the English profile vocabulary and grammar The author also uscd

28

the softwares Pearson and Estim to find out how the texts of the two current final achievement tests aligned to the CLI'R,

3.6 Summary of chapter 3

‘This Chapter has provided the setting of the study in terms of Knglish teaching and learning of non-English majors at (NTTC, brief description of the material used as well as the current testing situations at QNTTC The informants, the data collection instruments and the alignment framework were mentioned Data collection and data analysis procedure was also introduced

29