(LUẬN văn THẠC sĩ) non english major students’ perceptions and attitudes towards the use of the TOEIC listening and reading test as a graduation requirement

The findings revealed that students generally hold positive attitudes towards the use of the TOEIC scores semi-to determine their graduation eligibility and believe that preparing semi-t

LÊ THỊ NỤ

NON-ENGLISH MAJOR STUDENTS’ PERCEPTIONS AND ATTITUDES TOWARDS THE USE OF THE TOEIC LISTENING AND READING TEST AS A GRADUATION REQUIREMENT

(Nhận thức và thái độ của sinh viên không chuyên tiếng Anh về việc sử dụng bài kiểm tra TOEIC hai kỹ năng nghe và đọc hiểu làm điều kiện tốt nghiệp)

M.A MINOR THESIS (Type I)

Field: English Teaching Methodology

Code: 8140231.01

Hanoi – 2019

LÊ THỊ NỤ

NON-ENGLISH MAJOR STUDENTS’ PERCEPTIONS AND

ATTITUDES TOWARDS THE USE OF THE TOEIC LISTENING

AND READING TEST AS A GRADUATION REQUIREMENT

(Nhận thức và thái độ của sinh viên không chuyên tiếng Anh về việc sử dụng bài kiểm tra TOEIC hai kỹ năng nghe và đọc hiểu làm điều kiện tốt nghiệp)

M.A MINOR THESIS (Type I)

Field : English Teaching Methodology Code : 8140231.01

Supervisor : Prof Dr Nguyễn Hòa

Hanoi – 2019

requirements for the degree of Master of Arts at Faculty of Postgraduate Studies, University of Languages and International Studies, Vietnam National University, Hanoi This thesis has not, wholly or partially, been submitted for an award at any other universities or higher education institutions

Hanoi, 20 July 2019

Le Thi Nu

have been accomplished

My special thanks also go to all the professors and lecturers of Faculty of Postgraduate Studies at University of Languages and International Studies, Vietnam National University, Hanoi for their valuable lectures, which have laid the foundation for this thesis

I am deeply appreciative to the participants of this study for their immense contribution by sharing their experiences and opinions I also thank the director and employees of the center where the research took place

Finally, I am indebted to my friends, my colleagues and my family who have encouraged me in numerous ways to complete this study

collected from 197 participants through survey questionnaires and eight structured interviews at a TOEIC preparation center in Hanoi The findings revealed that students generally hold positive attitudes towards the use of the TOEIC scores

semi-to determine their graduation eligibility and believe that preparing semi-to take the test has a positive impact on their English language proficiency and future employment prospects The TOEIC test scores are also perceived favorably by the respondents as having relatively high levels of reliability and validity; however, the cut-off scores set by some universities might be too low to motivate their students to study English harder The results further indicated that test-takers with better listening and reading skills as measured by the TOEIC tests have a significantly stronger preference for the requirement of an adequate level of speaking and writing proficiency before graduation, provided that tertiary-level English curricula could help students master the four English language skills The study not only contributes to our understanding of students as the key stakeholders who define the context of the test use but also provides empirical evidence to support the use of the TOEIC test as a high-stakes standardized test for university exit requirement in Vietnam

Keywords: Perceptions, attitudes, high-stakes tests, English language

proficiency, TOEIC, test impact, higher education in Vietnam, graduation requirement

TABLE OF CONTENTS

DECLARATION i

ACKNOWLEDGEMENTS ii

ABSTRACT iii

TABLE OF CONTENTS iv

LIST OF ABBREVIATIONS vii

LIST OF TABLES viii

CHAPTER I: INTRODUCTION 1

1 Background of the study 1

1.1 Social context 1

1.2 Educational context 2

2 Rationale of the study 3

3 Aim of the study and research questions 3

4 Methodology and scope of the study 4

5 Significance of the study 4

6 Structure of the thesis 5

CHAPTER II: LITERATURE REVIEW 6

1 High-stakes tests 6

2 Reliability and Validity of standardized tests 7

2.1 Reliability 7

2.2 Validity 7

3 Different reactions to high-stakes standardized testing 8

3.1 Positive reactions 8

3.2 Negative reactions 9

4 Setting standards of high-stakes testing 10

5 Preparing students for high-stakes standardized testing 12

6 Perceptions and Attitudes towards high-stakes standardized testing 14

6.1 Definitions of Perception and Attitude 14

6.2 Importance of test-takers‟ perceptions and attitudes towards high-stakes

standardized testing 15

7 TOEIC as a high-stakes standardized test in Vietnam 16

7.1 What is TOEIC? 16

7.2 Use of the TOEIC test as a graduation requirement in Vietnam‟s HE institutions 17

8 Previous studies on the use of TOEIC as an exit requirement in Vietnam 18

9 Summary 20

CHAPTER III: METHODOLOGY 21

1 Research design 21

2 Research site 22

3 Participants 22

3.1 Survey participants 22

3.2 Interview participants 24

4 Data collection methods 24

4.1 Questionnaires 24

4.2 Interviews 27

5 Data analysis procedures 29

5.1 Quantitative data analysis 29

5.2 Qualitative data analysis 29

CHAPTER IV: RESULTS AND DISCUSSION 31

1 Research Question 1: What are students‟ perceptions of the English requirement policy for graduation? 31

1.1 Results from surveys 31

1.2 Results from interviews 33

2 Research Question 2: What are test-takers‟ perceptions of the appropriateness and effectiveness of the TOEIC as an exit test? 35

2.1 Results from surveys 35

2.2 Results from interviews 39

CHAPTER V: CONCLUSION 44

1 Recapitulation 44

2 Implications of the study 45

3 Limitations of the study 46

REFERENCES 47 APPENDICES I APPENDIX 1: SURVEY QUESTIONNAIRE I APPENDIX 2: INTERVIEW QUESTIONS VII APPENDIX 3: SAMPLE TEST FOR TOEIC L/R IX

LIST OF ABBREVIATIONS

EFL English as a Foreign Language

ELP English Language Proficiency

ETS Educational Testing Service, USA

HE Higher Education

MOET Ministry of Education and Training, Vietnam

TOEIC Test of English for International Communication

LIST OF TABLES

Table 1: Demographic information of participants 23

Table 2: Structure of survey questionnaire 25

Table 3: Students‟ opinions about the exit English requirement policy 32

Table 4: Students‟ opinions about the exit English requirement policy by their ELP 32

Table 5: Interviewees‟ perceptions of the exit English requirement policy 33

Table 6: Survey respondents‟ perceptions of the TOEIC as an exit test 36

Table 7: Survey respondents‟ perceptions of the use of the TOEIC as an exit test by their ELP 38

Table 8: Interviewees‟ perceptions of the TOEIC as an exit requirement 40

CHAPTER I: INTRODUCTION

This chapter introduces the study by outlining the context in which the research was conducted, the rationale of the study, the purpose and research questions, the significance of the study, the methodology adopted, and the structure

In Vietnam, among standardized English language proficiency tests, the Test

of English for International Communication (TOEIC) introduced in 2001 has gradually assumed a greater gate-keeping role Although it may not be as a well-known certificate as TOEFL (Test Of English as a Foreign Language) or IELTS (International English Language Testing System), TOEIC has become a popular English proficiency test not only in Vietnam but also in some other Asian countries like Japan, Korea and Taiwan Being a tool to measure workplace English proficiency for non-native English speakers, TOEIC certificates are required for job application in many business corporations in our country such as Viettel, Vietnam Airlines, LG Electronics, Panasonic, and even in stated-owned financial institutions including BIDV and Agribank, etc

1.2 Educational context

The importance of English as a lingua franca in the context of the fast globalization was early acknowledged by the Vietnamese government, and as a result, English education has been prioritized at all levels of the Vietnamese education system At tertiary levels for example, undergraduates generally took 200 hours of English at university over four academic years (Ton & Pham, 2010) Yet, this instruction was delivered in a manner which did not allow the vast majority of students to achieve sufficient levels of competence to pass the certifications required for recruitment As Ha (2007) indicated, the communicative competence in English of most Vietnamese employees was still very limited, and many freshly graduated students failed to find jobs in foreign companies because of the English language requirements

Vietnamese decision-making bodies were aware that major changes should

be made to upgrade students‟ English proficiency and get them prepared better for global competition at the workplace Accordingly, the Circular No 05/2012/TT-BGDĐT was issued by the Ministry of Education and Training (MOET), effective

as from April 2012, indicating that non-English major graduates need to have English proficiency equivalent to B1 (CEFR), 450 TOEIC, 450 TOEFL ITP, or 4.5 IELTS As a result, the majority of HE institutions in Vietnam gradually have set a requirement for their students to achieve a satisfactory score on one of the recommended English language proficiency tests before graduation

The statistics show that approximately 120 HE institutions in Vietnam, over

30 universities in Hanoi alone, are using TOEIC scores normally ranging from 450

to 550 as an exit requirement for their students For example, the Decision No 148/QĐ-ĐHBK-ĐTĐH dated 5 September 2017 by the Rector of Hanoi University

of Science and Technology states that non-English majors need present the TOEIC certificate of minimum 500 points or equivalent certificates before being assigned a graduation thesis While this university and some HE institutions such as People's Security Academy and University of Transport Technology cooperate with the only

ETS-authorized organization, IIG Vietnam, to set TOEIC-taking examinations for their own students; others request their students to take the tests at IIG Vietnam headquarter by themselves Obviously, the TOEIC test is now one privileged high-stakes standardized test in Vietnam

2 Rationale of the study

The use of the high-stakes standardized test to determine graduation eligibility has a direct impact on students who take the test to meet the exit requirement, and it has important educational and economic implications for students and society at large Language learners or students whose education and language learning are directly influenced by the implementation of the language graduation requirement policy are perhaps the most important stakeholders in this testing situation However, their views on the policy implementation are rarely considered by university authorities

Previous research has suggested that learners‟ attitudes towards and perception of a test and its use can affect their motivation and performance on the test (Bachman & Palmer, 1996) It has been about seven years since the TOEIC test was officially used as an exit English test at some HE institutions in Vietnam, but there is little research examining the appropriateness and effectiveness of the policy, despite its increasing popularity This study is an overall investigation into the test-takers‟ own opinions on the English requirement policy for graduation and the use

of the TOEIC certification as an exit requirement at tertiary education levels Given that the test takers‟ perceptions can have wide-ranging consequences, opinions about the use of the TOEIC tests as an exit test may lead to further investigation to ensure the tests‟ validity

3 Aim of the study and research questions

The study aims mainly to investigate non-English major students‟ perceptions of and attitudes towards the use of TOEIC Listening and Reading test scores to determine their graduation eligibility in some HE institutions However, it

is necessary to examine their perceptions of the English requirement policy for

graduation in general first, and then their opinions about the appropriateness and effectiveness of the TOEIC as an exit test could reveal whether the test use is in line with the intended purposes of the policy In order to achieve this aim, the following objectives were addressed in the study:

- To identify how non-English major students perceive the English requirement policy for graduation

- To investigate the opinions of the students who take the TOEIC Listening and Reading test for graduation about the appropriateness and effectiveness of this exit requirement

With these objectives in mind, the researcher conducted the study to answer these research questions:

1 What are non-English major students’ perceptions of the English requirement policy for graduation?

2 What are their perceptions of the appropriateness and effectiveness of the TOEIC as an exit test?

4 Methodology and scope of the study

This study was carried out at a prestigious TOEIC preparation center in Hanoi, where thousands of students from different universities study to get the target TOEIC scores for graduation The quantitative data were collected by means

of survey questionnaires from 197 non-English major students (n = 197), who were sophomores, juniors, or seniors, of both genders, varied majors and English proficiency The follow-up semi-structured interviews of eight participants were conducted to obtain the qualitative data Then the quantitative data were analyzed using SPSS, and the qualitative results were analyzed with the aid of NVivo to develop themes and codes The triangulation of quantitative data and qualitative themes also increased the reliability of the research findings

5 Significance of the study

As this research is an endeavor to emphasize the importance of listening to the ideas from students – high-stakes standardized test-takers, there are several

significant contributions of this study First, the findings of this study might contribute to raise other stakeholders‟ awareness that test takers‟ voices can provide information about high-stakes standardized English language proficiency tests from different perspectives Second, this research gives the participants chances to share their thoughts and experiences of taking the TOEIC tests as a graduation requirement Given that the test takers‟ perceptions can have wide-ranging consequences, their opinions about the use of the TOEIC tests as an exit test also encourage a further investigation to ensure the tests‟ validity and reliability Finally, investigations into these issues could provide the MOET policymakers and university authorities with important insights into the appropriateness of the English graduation requirements or cut-off scores

6 Structure of the thesis

The thesis contains five chapters including this introduction Chapter 2 reviews the theories of students‟ perceptions of and attitudes towards high-stakes standardized tests and some previous studies on the use of the TOEIC test as an exit requirement in Vietnam It also identifies the research gap that this study should fill

in After that, chapter 3- the methodology presents the way this study was conducted such as the research design, participants, data collection and analysis procedures Next, chapter 4 presents the results and discussion of this study Finally, chapter 5 reports the conclusion of the study and provides information about the limitations of this study and recommendations for future studies as well

CHAPTER II: LITERATURE REVIEW

In order to articulate the research questions and identify necessary research issues, this chapter reviews the literature on main areas: high-stakes tests, reliability and validity of standardized ELP tests, setting standards and preparing students for high-stakes standardized testing, students’ perceptions of and attitudes towards high-stakes standardized tests, and the TOEIC as a high-stakes standardized test in Vietnam Also, some previous studies on the use of the TOEIC test as an exit requirement in Vietnam are reviewed to identify the research gap that this study should fill in

1 High-stakes tests

High-stakes tests usually refer to large-scale standardized tests or public examinations, whose results are used for making important educational and professional decisions about individuals, such as admissions, graduation, employment, or promotion (Madaus, 1988) High-stakes testing policies “represent

a motivational policy” (Ryan & Brown, 2005, p 358) which applies performance contingent rewards and punishments to standardized test scores Performance contingent rewards refer to those given for satisfying or surpassing a certain requirement or standard, whereas performance-contingent punishments are those given for failing it (Deci & Moller, 2005)

Under high-stakes testing policies, schools, teachers, and students are the key stakeholders affected For schools, the received amount of subsidies or grants could

be determined by students‟ test results or performance ratings For teachers, their teaching might be evaluated based on their students‟ test scores For students, if they perform up to certain standards, they are rewarded through praise or through concrete rewards, such as money; in contrast, if they perform badly or fail the test, they could be punished by being held back in school or being unable to graduate (Kohn, 2000) From this perspective, high-stakes testing can create a context where the stakeholders (i.e., schools, teachers, and students) direct their attention to test score results and external contingencies

2 Reliability and Validity of standardized tests 2.1 Reliability

In discussing the same candidate writing the same test at a different time, Hughes (1989) notes, “the more similar the scores would have been, the more reliable the test is said to be” (p 29) While this view of reliability issues being the result of random measurement error is quite common (Lewis, 1999), Bachman (1990) uses a broader definition encompassing not only “how much of an individual‟s test performance is due to measurement error”, but also how much is due “to factors other than the language ability we want to measure” (p 160)

Bachman (1990) discusses three factors that may affect test reliability The first are Test Method Facets, which are broken down into the five categories of testing environment, test rubric (organization), input (format of and nature of), expected response (format of, nature of, and restrictions on), and the relationship between input and expected response (reciprocal, non-reciprocal, adaptive) The second are Personal Attributes and include aspects such as age, gender, cognitive style and, background The third are Random Factors and can include aspects such

as tiredness and a candidate's emotional condition, as well as random differences in the testing environment

Bachman goes on to point out that reliability and validity are not independent entities, but are complementary in that reliability is a necessary condition for

validity (1990:160) Thus, a certain grammar test may be highly reliable, yet not considered a valid test of spoken English, while a poorly assessed spoken test may have low reliability but may otherwise be considered valid for its purpose

3 Different reactions to high-stakes standardized testing 3.1 Positive reactions

The use of high-stakes standardized testing and its impact on educational contexts remain controversial Policymakers intend to use high-stakes standardized tests to make desired educational changes They argue that “the promise of rewards

or the threat of sanctions is needed to ensure change” (National Research Council,

1999, p 35), which assumes that the behaviors that the stakeholders adopt associated with high scores will be reinforced, whereas those with poor test performance will be diminished From this perspective, high-stakes standardized tests have been treated as powerful and “effective tools for controlling educational systems and prescribing the behavior of those who are affected by their results” (Shohamy et al., 1996, p 299)

Many agree on the usefulness of the use of external contingencies to control behaviors Shanker (1993, as cited in Ryan & Brown, 2005) argues that when the achievement outcomes, such as test scores, are significantly attached to the consequences, students will have “the incentive to work hard and achieve because they know something important…is at stake” (p 9) Finn (1991) also holds that high-stakes standardized testing can effectively change student‟s learning behavior when contingent rewards and sanctions exist behind the test results Thomas (2005) argues that some students under high-stakes testing policies will be more motivated

to exert more effort to improve their test scores in the future Stecher (2002) also concludes that students with high-stakes testing are motivated to put in more effort, know better about their own abilities and what to study, and align their own effort with rewards

Some research studies also demonstrate the potential positive effects of stakes standardized testing on students For example, in their qualitative study

high-focusing on the impact of a high-stakes test on 102 low-achieving students in Chicago public schools, Roderick and Engel (2001) found that lower-performing students could hold positive attitudes towards the high-stakes testing policies if the goal is perceived achievable They further suggest that these students would be motivated to study harder if they are provided with “incentives… through goals that provide an opportunity for feedback, a tangible reward, and a way to construct meaning regarding learning” (p 219)

3.2 Negative reactions

The power and controlling nature of high-stakes standardized testing have been subject to criticism for its potential undesirable consequences (Ryan & Brown, 2005) The study conducted by Noble and Smith (1994) shows that when high stakes is placed behind test performance, teachers tend to focus on the activities that could boost test scores, such as practicing test-taking skills or reviewing past examination papers, and their students are thus directed to learn toward the test rather than to enhance long-term knowledge growth

Amrein and Berliner (2002) took a comprehensive look at the consequences

of high-stakes standardized tests by conducting a state-by-state analysis in the U.S

A total of states were included and four separate standardized state tests (i.e., the American College Testing Program (ACT), Scholastic Aptitude Test (SAT), National Assessment of Educational Progress (NAEP), and Advanced Placement (AP) were examined The analyses revealed that the state‟s high-stakes testing policy failed to enhance transfer Amrein and Berliner (2002) found that after the policy was implemented, the students from two-thirds of states performed worse on the ACT; in other words, these students did not perform better on the outside tests, although increased scores were shown on the state‟s high-stakes test Some other studies (Stecher, 2002; Ryan & Brown, 2005) also raise some other concerns other negative impacts of high-stakes testing on students, such as decreasing intrinsic motivation, stimulating test anxiety and frustration, being more competitive, and using more inappropriate test-taking strategies

4 Setting standards of high-stakes testing

Thomas (2005) indicated that five of the more popular criteria used in the U.S to set standards for high-stakes tests are: (a) the ideal student, (b) the attainable level, (c) other students, (d) each student‟s own record, and (e) some combination of (a)–(d) (p 59)

The criterion of “the ideal student” is the standard of a person who is ideally fitted to display all worker-virtues at a high level of efficiency after entering the workforce and able to attain an optimal degree of self-fulfillment Business executives are most impressed by these features of high level of attainment when selecting their employees However, to apply such a standard, education leaders are obligated to explain and specify the numbers of test items and the level of difficulty

in each test, which can increase the level of difficulty in practice As Thomas singled out, the ideal level of this standard will be too high if only a small portion of learners earn acceptable marks, and will be too low if an excessively large proportion of students receive extremely high ratings

The attainable level is the criterion that education leaders (such as policy makers, administrators, test-makers, or teachers) believe that most students can answer correctly if their students have studied diligently The assumption of this standard implied that all participants can master all of the learning objectives For the purpose that all students or nearly all students are to pass, the standard tends to

be set at a level that the least adept can reach This low expectation may “result in the average and above-average students not being challenged to exert their best effort, because they lack the incentive to learn far beyond the limits of such an expectation” (p 60) Failing a test can “motivate some learners to greater effort to score higher in the future” (Thomas, 2005, p 61) However, if the attainable level is mandated to such a high standard that few learners reach it, the large portion of the students who fall below the pass line may hold negative views about their abilities and “view themselves as failures.” Some students, who already have been working diligently and still fail, may become discouraged, give up and drop out

Additionally, having a high proportion of students fail to pass the tests after applying this criterion manifests that “the instruction has been very poor, or that the methods of evaluating student progress have been faulty, or that the achievement standards have been unreasonably strict” (p 60)

The criterion of “other students”, according to Thomas‟s categorical definition, compared students‟ performances to those of their classmates, grade-mates, or age-mates The school administration can avoid being accused of setting

an unreasonably high or low attainment level However, the decisions for the cut-off score (which is subjectively decided by opinion) to distinguish students‟ passing or failing will be challenging to policy makers, administrators, test-makers and teachers, which is subjectively decided (p 61)

The criterion of “each student‟s own record” drew on the judgment of passing or failing a test by focusing on the progress that each student has made for a given period of time This individual-progress approach has been highly eulogized for its promotion of the goal of developing a person‟s potential of being what he or she is capable of However, Thomas explained that the drawbacks of this type of criterion-setting are (a) developing a false sense of a person‟s actual ability if only comparing against his or her past record; (b) failing to provide information about how a learner performs compared with other age-mates or global counterparts of the same age and (c) suffering rude shock in the real world after a learner realizes that the high grade received in school for his or her moderate improvement was unrealistic and devastating

The Hybrid Approach to combining the features of the person, attainable-level, and student-comparisons criteria is the fifth popular achievement standard proposed by Thomas The combination of above-mentioned perspectives is a viable alternative in terms of setting achievement standards The whole learning process is regarded as a system arranged in hierarchical stages combining all these elements The student-comparisons aspect can be viewed as the starting stage for setting their criteria of learning and testing at their own learning

ideally-educated-pace, specifically to the learning disadvantaged student group that should pass the required tests during any given year in order to avoid the failure label The attainable-level element is found in the stage of progress as proposed each year by the school authorities so that eventually every learner can attain the standard set by their school in a certain given period of time The ideally-educated-person element

is obviously represented by the ideal alternate aim to have virtually all students proficient in knowledge and practical skills in the related field after they graduate from schools

In summary, Thomas (2005) denoted that people‟s beliefs about “sufficient competency” and “fair treatment” can lead to their disagreement with the test-score

levels or cut-off points by which to determine the designations success or failure of

a test (p 76) Disagreement in proper standards of tests is bound to continue Expecting complete accord regarding standards-setting is challenging and unrealistic

5 Preparing students for high-stakes standardized testing

Crocker (2005) defined appropriate test preparation as “instruction that is geared not only toward study of content from the domain of knowledge sampled by the test, but also toward practicing the skills that will allow students to demonstrate their knowledge on various types of assessment exercises” (p 161) He explained that teaching for assessment occurs when students are taught “the broader content domain represented by the curricular standards, not simply to that subset of content sampled by the items on a single test form” He used the term “teaching “for” the test” to distinguish the negative connotation of “teaching to the test” When dealing with students‟ urgent needs in high-stakes English exit examinations, test preparation is the primary concern Four essential elements of teaching for the test include: (a) a challenging core curriculum, (b) comprehensive instruction in that curriculum, (c) developing students‟ test-taking skills, and (d) adherence to ethical guidelines regarding preparation of students for assessment (Crocker, 2005)

Approaches to test preparation offered by Smith and Delisi (2001) were useful to classroom instructors in differentiating curriculum alignment from measurement-driven instruction The four stages of test preparation for instructors were as follows: “(a) teach without paying attention to the standardized test and hope that the students‟ abilities will show through on the assessment; (b) spend most of the time in instruction as instructors normally do, but spend some time going over item formats to be found on the assessment so that students will be familiar with these formats; (c) analyze the content of the assessment, make certain that instructors cover the content in the regular instructional program, then work on item format and test taking skills as well; (d) analyze the content of the assessment and restructure the instructional program around that content exclusively” (Crocker,

2005, pp 90-91) The first approach illustrates teaching without test preparation and without curricular alignment The second approach illustrates instruction without curricular alignment, but with some attention to test-taking skills The third approach describes a reasonable balance of instruction with curricular alignment and instruction in test taking skills, especially there is concerted effort to teach subject matter knowledge and test taking skills that will have broad utility to the students beyond this immediate examination situation Two important premises of undertaking the third approach are that (a) the assessment represents a good sample

of the core curriculum, and (b) the core curriculum is worthy and important The fourth approach crosses alignment the line toward measurement-driven instruction

Crocker (2005) used various sources to develop his classroom strategies of proper test preparation (Campanile, 1981) Some of these strategies included: (a) demonstration of a positive attitude toward test preparation with students and parents, (b) building concentration endurance in test-like conditions without becoming fatigued or distracted, (c) practicing various test item formats with instructions and answer sheets as those on the tests, (d) timing practice tests, (e) modeling good problem-solving strategies (how to approach the test items, to

determine the meaning of the question, to provide various ways of posing the same question, and to arrive at the correct answer), (e) practicing working through tests with various difficulty levels, (f) diagnosing any response patterns that needed to be corrected before testing through homework grading or class work, (g) building students‟ test-taking vocabulary, (h) explaining how score rubrics are used to award points for performance assessment items, providing examples of responses generating full, partial, or no credit, and helping students evaluate their responses

Crocker (2005) also suggested four criteria for determining if classroom instruction practice is ethical These four criteria are as follows: (a) Academic ethics: test preparation should be guided by the ethical cannons of the education profession, dealing with cheating, misrepresentation, and respect for intellectual property or work of others (Popham, 1991); (b) Validity: Test preparation should improve validity of test scores by allowing only students who have knowledge or partial knowledge of content being tested in an exercise to display that knowledge; (c) Transferability: Test preparation should provide student with skills that have applicability to a broad range of testing situations (Popham, 1992); (d) Educational value: test preparation that leads to improvement in student scores should simultaneously increase student mastery of the content domain tested (Popham, 1991)

6 Perceptions and Attitudes towards high-stakes standardized testing 6.1 Definitions of Perception and Attitude

According to Campbell (1967), perception is a process where one will form

an impression about someone or something Based on the definition, perception is constructed as a result of individual observation towards certain things or events occurring around them which will produce certain perception The perception then affected their attitude towards certain matters or objects of attitudes

Lindsay and Norman (1977) stated that perception is the process by which organisms interpret and organize sensation to produce a meaningful experience of the world The researchers describe the definition into two categories, which are

sensation and perception Sensation refers to the immediate, relatively unprocessed result of stimulation of sensory receptors in the eyes, ears, nose, tongue, or skin Sensations refer to the human sensory systems which function as the receptor of information regards certain matters or object observed by an individual It is the stimulation process where the input is transferred directly to the human brain On the other hand, perception refers to the one‟s ultimate experience of the world and typically involves further processing of sensory input Perception involved more process of thinking as a result of the information received from the sensory systems as regards certain things or events It is the output process where the judgments or beliefs were produced by

an individual and it influenced the way they think and feel

Gardner (1985, p 9) stated that attitude is an evaluative reaction to some referent or attitude objects, inferred on the basis of the individual‟s beliefs or opinions about the referent Ajzen and Fishbein (2005, p 3) defined that attitude is a disposition to respond favorably or unfavorably to an object, person, institution, or event The definitions of attitudes has been varied as a result of continues studies done by researchers Even though the researchers have their own definition of attitudes, they still share the facts that attitudes guide and direct an individual‟s behavior Attitude is more to how human perform their thinking or their reaction toward certain things or matters Meanwhile, perception did not involve action as it

is more to the way of human thinking Nevertheless, perception and attitudes are related as perception is the root of thinking which invoke certain judgments or beliefs that influence human attitude

6.2 Importance of test-takers’ perceptions and attitudes towards high-stakes standardized testing

Shohamy (2001a) argued that listening to the test-takers‟ voices about tests will lead to a better understanding about tests, especially for high-stakes tests which have powerful effects not only in students‟ educational lives, but also in their social lives That is, while high-stakes test takers are often considered passive participants

in the testing process, their opinions provide information from a different perspective that might not be addressed by other stakeholders She also emphasized the importance of knowing students‟ perceptions towards tests as it can lead into information about the importance, the use, and the meaning of the tests to the students‟ lives In addition, Murray et al (2012) pointed out test-takers‟ attitudes toward test may have significant implications to test validity, which leads to evaluation of the appropriateness, meaningfulness, and usefulness of high-stakes tests According to Tsai & Tsou (2009), test-takers‟ perceptions are significant because they provide evidence of tests‟ effectiveness and one of the aspects in construct validity

Previous research has also suggested that learners‟ attitudes towards and perceptions of a test and its use can affect their motivation and performance on the test (Bachman & Palmer, 1996) Empirical research conducted by Brown and Hirshfeld (2008) demonstrated that meaningful relationships exist between students‟ conceptions of assessment and their academic achievement In higher education, students‟ perceptions of assessment are pivotal because assessment has a critical impact on quality of learning (Entwistle, 1991) Students will be motivated

to prepare for the examination if they perceive it as pivotal; that is, the test-takers‟ perceptions of the examination determine the level of the effort they are willing to take (Wolf & Smith, 1995) In other words, how much effort will be put in the examination is based on the test-taker‟s perception, not merely judged by the stakes

to which the examination may be attached

7 TOEIC as a high-stakes standardized test in Vietnam 7.1 What is TOEIC?

TOEIC (Test of English for International Communication) is the standardized test of workplace English proficiency for non-native English speakers With its widest score range, TOEIC claims it can measure all English levels (from beginning to advanced) which is proper for all working positions, from basic jobs to management thus this is the preferred English language assessment tool to measure

worldwide-the progress of leaners and help worldwide-them to achieve worldwide-their set-goals in his careers The TOEIC test was developed by Educational Testing Service (ETS), the United States

of America in 1979 at the request of the Ministry of International Trade and Industry in Japan

TOEIC consists of two types of tests First is the TOEIC Listening and Reading Test, which consists of different comprehension assessment activities The comprehension is divided into two equal parts of 990 score in total One of these comprehension tests consists of listening comprehension and the other is reading comprehension The candidates obtain score on a scale of 5 to 495, which later adds

to a total score of 990 The second type is the newer TOEIC Speaking and Writing Test, which consists of various parameters to measure the English skills of a candidate It includes pronunciation, vocabulary, grammar, fluency, overall coherence and structure, which make up for a score of 400 in total Candidates get separate scores for speaking and writing They also get an option to appear for the speaking test only, leaving the writing test It approximately takes 20 minutes to complete this test; however, the writing test takes 60 minutes Further, each of these tests comprises of 200 marks, which are then added to a total of 400

7.2 Use of the TOEIC test as a graduation requirement in Vietnam’s HE institutions

TOEIC Listening and Reading Test was introduced to Vietnam in 2001 by IIG Vietnam, the only organization authorized by ETS to provide this test TOEIC Speaking and Writing test was also provided five years later; however, it has not been as popular as the previous type so far In fact, when it is said about TOEIC in Vietnam, it normally refers to TOEIC Listening & Reading test (see Appendix 3 for

a sample test) Also, for the purpose of the current study, TOEIC is short for TOEIC Listening & Reading (LR) test

First, TOEIC is used as a basis for recruitment and promotion in competitive labour markets such as information technology, airline industry and international trade Since 2010, the Ministry of Education and Training (MOET) in Vietnam has

encouraged HE institutions to set English proficiency thresholds for undergraduates and to implement an English requirement policy for graduation The objective is to raise students‟ English language proficiency (ELP) and to better prepare students to cope with global competition and meet the ELP requirements of the workplace Two years later, the MOET has advocated the use of standardized English proficiency tests as exit requirements by issuing the Circular No 05/2012/TT-BGDĐT, indicating that non-English major graduates need to have English proficiency equivalent to B1 (CEFR), 450 TOEIC, 450 TOEFL ITP, or 4.5 IELTS

As a result, the majority of HE institutions in Vietnam gradually have set a requirement for their students to achieve a satisfactory score on one of the recommended English language proficiency tests before graduation

The statistics show that approximately 120 HE institutions nationwide, over

30 universities in Hanoi alone, are using TOEIC scores to determine their students‟ graduation eligibility The cut-off scores set by most universities, such as National University of Civil Engineering, Hanoi Law University, Vietnam Maritime University and Thai Nguyen University, are 450 points Some universities require higher TOEIC scores for graduation For example, the Decision No 148/QĐ-ĐHBK-ĐTĐH dated 5 September 2017 by the Rector of Hanoi University of Science and Technology states that non-English majors need present the TOEIC certificate of minimum 500 points or equivalent certificates before being assigned a graduation thesis While this university and some HE institutions such as People's Security Academy and University of Transport Technology cooperate with the only ETS-authorized organization, IIG Vietnam, to organize TOEIC-taking examinations for their own students; others request their students to take the tests at IIG Vietnam headquarter by themselves These test-takers, however, get the same valid score certificate regardless of either way they take the tests

8 Previous studies on the use of TOEIC as an exit requirement in Vietnam

There have been a couple of studies related to the use of TOEIC as a graduation requirement in HE institutions in Vietnam The first study entitled “The

TOEIC Test as an Exit Requirement in Universities and Colleges in Danang City, Vietnam: Challenges and Impacts” was conducted by Nhan (2013) Conducting this research at the very beginning of introducing the TOEIC certification exit requirement, the researcher found that unsolved challenges had hinted at the unfeasibility of implementing the policy at that moment The limited qualified human resources together with the lack of necessary materials and facilities and the time-constraint curriculum allocated to the teaching and learning of English at HE institutions make it unfeasible for students to achieve the required TOEIC score Besides, a number of students with low levels of English or from poorer backgrounds may be disadvantaged in taking supplementary TOEIC preparation courses Then, it is more practical if HE institutions choose a locally based English test to decide graduation eligibility of currently final-year undergraduates

However, Nhan (2013) argued that in the long run, it must be confirmed that the use of certain English certification, including TOEIC certificates, as an exit requirement should remain an aim It is because companies have been more interested in using an internationally recognized standard or the TOEIC standards for recruitment; and Vietnam‟s HE institutions themselves need to be competitive among new foreign-invested international universities In that case, students need to

be informed right at the start of their tertiary lives so that they allocate enough time and attention to improve English skills and practice for the English exit tests English programs and teacher training should also be modified in a way that could encourage learners‟ production of the target language

The second study entitled “Impact of using TOEIC as an exit requirement at

a public university in Vietnam” was carried out by Tran (2016) The result of the study showed that using TOEIC scores as a graduation requirement exerted strong impact on the English language education at a public university The first impact is

on English language curriculum; accordingly, the teachers reformed the EFL curriculum and created the TOEIC-oriented program to prepare their students for exit exams The second impact is on classroom teaching and learning; that is, the teachers focus on teaching test-taking strategies and pay less attention to interactive and communicative activities

9 Summary

Based on the elaboration in this chapter, it is clear that listening to takers‟ voices about tests will lead to a better understanding about high-stakes standardized tests, as test-takers are able to point out issues that might not be noticed from the view of other stakeholders In the current study, the TOEIC test required by some Vietnam‟s HE institutions is viewed as high-stakes because its result determines granting or denying of the academic degree This policy has been gradually implemented by numerous universities for seven years, and the time is long enough so that its effectiveness can be evaluated comprehensively The limited studies on the TOEIC exit standard in Vietnam, as mentioned above, either have focused only on academic factors which affect teaching and learning for the TOEIC exit test or have not yet given a definite answer to the appropriateness of the policy from the test-taker‟s perspective because one of them was carried out at the pilot implementation stage How crucial students perceive the English exit standard policy and the TOEIC as a graduation requirement remains uncertain, which creates the research gap that the present study should fill in

test-CHAPTER III: METHODOLOGY

This chapter presents the ‘mixed methods’ nature of the research reported here It continues with a description of research participants and then a discussion

of data collection methods used in the research, including the matters of validity and reliability The chapter ends with the process of data management and analysis

1 Research design

This survey research had a mixed methods design, combining both quantitative and qualitative procedures As Creswell (2003) indicated, “with the development and perceived legitimacy of both qualitative and quantitative research in the social and human sciences, mixed methods research, employing the data collection associated with both forms of data, is expanding” (p 208)

In addition, mixed methods of quantitative and qualitative procedures were conducted in order to increase the depth and breadth of the study, which assisted the researcher to see the research matters from different angles, so that

a deeper understanding of the topic could be obtained The triangulation of data that were obtained through different methods also increased the reliability of the findings (Allwright & Bailey, 1991)

Among a variety of mixed methods designs, Explanatory Sequential Strategy in which a qualitative data collection builds directly on the result of the initial quantitative data collection is chosen (Creswell, 2014) Since Explanatory Sequential Strategy allows a thorough understanding of the quantitative results by explaining them with qualitative data, it is believed to provide the researcher with

a satisfying explanation for participants‟ common responses In the present research, a survey questionnaire was used as a primary instrument of quantitative data collection in order to find out how non-English majors perceive the use of TOEIC scores as a graduation requirement; and then the qualitative part employed the method of interview to better understand the reasons for some of the survey participants‟ responses

2 Research site

This study was conducted at one of the oldest and most prestigious TOEIC preparation centers in Hanoi There TOEIC preparation classes are divided into four levels, including basic grammar class, PreTOEIC class for the score target of 350, TOEIC A class for the score target of 450+, and TOEIC B class for the score target

of 600+ This center was chosen as the research site because thousands of students from different universities throughout Hanoi had participated in its evening classes

in order to achieve the TOEIC target scores for graduation requirement Also, the researcher has worked as a part-time English teacher at this center for five years and taught many of the participants of the present study Conducting research in this familiar site brings the researcher considerable advantages in terms of access to the research site, persuading learners to participate, and establishing rapport with participants Such advantages, according to Marshall and Rossman (2011), can contribute the quality of the research data and research findings

3 Participants 3.1 Survey participants

For the purpose of this study, based on the center‟s learner database, the researcher selected participants who met certain criteria First, they were non-English majors from HE institutions where the TOEIC exit requirement policy had been adopted Second, these students had taken the official TOEIC tests at least once for the past six months and had valid TOEIC score reports by the time of data collection If they had taken the official TOEIC tests a longer time before, they might not have remembered their testing experiences exactly In all, 205 eligible students were invited to participate in an email survey; however, a few students did not reply to the invitation, resulting in a final number of 197 participants They were

in academic years at university from the second to the fourth, of both genders, and varied in English proficiency according to their TOEIC scores The researcher would also find out whether these features, as shown at Table 1, probably affected

participants‟ perceptions of the research issues

Table 1: Demographic information of participants

3.2 Interview participants

Processing the survey data, the researcher did not find any illogicality among answer choices of every single questionnaire that is doubtful to be clarified Therefore, researcher decided to select the interviewees first on a voluntary basis

Of the 197 questionnaire respondents, 36 students volunteered to take part in a follow-up interview by providing their contact information in reply emails Due to the researcher‟s time constraint and because of the similar results from several of these questionnaires, eight finally selected students represented different majors and universities, both genders, and various TOEIC scores, including those who do and

do not support the use of the TOEIC as a requirement for graduation However, the representation was not equal across the criteria but was partly dependent on the number of volunteers and the ratio of the responses

4 Data collection methods 4.1 Questionnaires

Questionnaires can be distributed in large numbers and to varied groups of respondents It is more economical to use questionnaires, compared with doing interviews, when the number of subjects is large for instance (Bryman, 2008; Walliman, 2011) Moreover, the respondents usually have an option about whether

to reveal their personal information or not, thus avoiding the risk of exposing their identities Responses in a questionnaire are also easier to quantify and process with the aid of certain software, making it possible for the researcher to see and present results in a quick and conspicuous way In the present study, the researcher emailed the soft-version questionnaires in the attachment to all the participants, which was not only more economical but also more convenient for them to complete the survey than the fact that the hard copies were delivered directly to them

A variety of items may appear in a questionnaire, ranging from closed questions to open questions, from dichotomous to multi-option selection items, and Likertscale items with various numbers of points along the continuum (Brown, 2000) The closed questions or selection items yield data that are easier to quantify,

compare, and classify, since the number of choices is limited and predetermined (Creswell, 2008a), whereas the open questions reserve more space for respondents

to express themselves „in their own terms‟ (Brown, 2000; Punch, 2009) The advantage of using open questions, then, is that the researcher may obtain depth in responses, and the responses may include the ideas or categories that predetermined classifications may have overlooked However, it is also possible that the responses obtained are not as detailed as expected by the researcher When the responses are detailed, they are more difficult to quantify, compare, and classify After these advantages and disadvantages being taken into consideration, the data collection from questionnaires of this study was conducted according to the basic steps as follows:

Designing the questionnaires: Besides demographic information items of

participants in the first part, the researcher designed a 13-statement questionnaire in

Vietnamese with five-point Likert scale items, ranging from totally disagree to disagree, neutral, agree, and totally agree (see Appendix 1) Table 2 gives detailed

explanations for the question statements

Table 2: Structure of survey questionnaire Research Question 1: Perceptions of the English requirement policy for graduation

Purpose of investigation Question statements

Necessity of the policy 1 It is necessary for universities to require their students

to pass a certain score in an ELP test before graduation

Goal of the policy 2 Passing an exit English standard ensures that students

have minimum ELP for their future work

Policy‟s impact on English learning motivation

3 I study English harder when my university adopts the exit English requirement policy

4 I would still be motivated to study English at university even if there were no English graduation requirements Policy‟s impact on ELP 5 I would still be motivated to prepare to take

test-taking motivation standardized English language proficiency tests even if

there were no English graduation requirements

Research Question 2: Perceptions of the appropriateness and effectiveness of the TOEIC as an exit test

Purpose of investigation Question statements

Impact of TOEIC preparation on ELP

6 Preparing for the TOEIC test helps me improve my ELP

Universities preparing their students for TOEIC exit tests

7 English learning curriculum in my university is sufficient for me to take the TOEIC exit test

8 My university often organizes TOEIC preparation classes for students before exit English exams

Appropriateness of off scores

cut-9 Required TOEIC cut-off score is achievable without difficulty

Validity of TOEIC scores 10 TOEIC scores accurately reflect my ELP

Benefit of having a TOEIC certification

11 If I have a high-score TOEIC certificate, I will feel more confident to apply for a job after graduation

Added productive skills testing

12 TOEIC speaking and writing tests should be required for graduation

TOEIC test-taking motivation

13 If there were no English exit requirements, I would still be motivated to prepare for and take the TOEIC test

Piloting the questionnaires: The questionnaires were given directly to 10

students of a TOEIC preparation class at the center, who were not included as participants of the study The researcher was present while students filled in the questionnaires to provide explanations and clarifications should they be required The pilot stage revealed no problems with the comprehension of question statements in the questionnaire and assisted in improving the validation of the methods and findings in this study (Creswell, 2008a)

Conducting the survey: The official questionnaires in the attachment were emailed

to 205 eligible participants, including an appealing invitation to the survey in the text The researcher reminded the students who attached incomplete questionnaires

or did not reply after a week of filling out the survey Finally, 197 valid questionnaires in the form of soft copies (96.1 %) were collected during twelve days They were all numbered from 1 to 197 and saved in a folder for convenient access by the researcher whenever necessary

4.2 Interviews

The interview is a good way to obtain subjects‟ opinions and/or feelings about a certain topic, event or action in a highly personal and detailed level (McDonough & McDonough, 1997; Punch, 2009) The conversational nature of an interview enables the interviewer to adjust the questions, shift the order of questions, omit some questions, or probe more after certain questions right on the spot according to the flow of the interview and the interviewee‟s responses, especially in the less structured forms of interviews (Roulston, 2010) The interviewees can also clarify their points and express themselves more extensively than when completing a questionnaire Moreover, a face-to-face interview provides the interviewer with valuable opportunities to “read between and beyond an interviewee‟s words”: not just what they say, but how they say it, their intonations, facial expressions, or their gestures (Walliman, 2011) By paying proper attention to these signals, an interviewer may learn more than simply what an interviewee is telling them; they may also help determine whether to probe more into certain parts

of the interview to see what the interviewee truly means or actually wants to say In this sense, interviews provide richer and more profound responses than any kinds of questionnaires

For a semi-structured type of interviews employed in this research, the interviewer does not have to follow a number of set questions in a rigid order Instead, a number of core questions or themes are suggested, and are to be covered during the interviews However, the interviewer may provide explanation, elaborate