04051001872 nội dung an investigation into the quality of the final test for second year students at a technical university in hanoi = Đánh giá chất lượng bài kiểm tra cuối kì của sinh viên năm hai tại một trường Đại học kĩ thuật Ở hà nội
INTRODUCTION
Rationale
English is increasingly recognized as essential for success across various sectors in Vietnam Many companies and organizations now list English proficiency as a key requirement for employment However, reports from the Ministry of Labor, the General Statistics Office of Vietnam, and the International Labor Organization in 2014 indicate challenges in meeting this demand.
Since 2010, many university graduates have struggled to find jobs related to their fields, with over half requiring retraining after employment, raising significant concerns about the quality of education, particularly in English skills This has led to increased emphasis on the importance of English language testing among both teachers and learners According to Bachman (1990), testing allows educators to evaluate student achievements, reflect on their teaching methods, and contribute to the language teaching process Additionally, testing enables learners to self-assess their English proficiency, ensuring it aligns with employment requirements and personal goals.
This study examines the quality of final achievement tests, emphasizing their importance in assessing students' language acquisition within a specific program The results significantly influence semester grades, necessitating a design closely aligned with program objectives Systematic achievement tests are essential for the development of a coherent curriculum, reinforcing their critical role in evaluating English courses Key qualities of effective tests include reliability and validity, with reliability pertaining to the consistency of test scores and validity relating to the interpretation and application of those scores (Bachman, 1990).
At IU (pseudonym for the university in which this study takes place), the course
Since its inception in 2009, the "English for Occupational Purposes" (EOP) program has successfully benefited multiple student generations across various majors, including English for Tourism and English for Business, leading to high employer appreciation for IU graduates In 2017, the initiative expanded with the introduction of the "English for Mechanical Engineering" course.
Despite its proven effectiveness, there has been no official assessment of the quality of final achievement tests for students studying "English for Mechanical Engineering" - Book 4, particularly regarding reliability and content validity Consequently, it is crucial to undertake a study to evaluate the final achievement test.
I have chosen the research topic: "An investigation into the quality of the final test for second-year students at a technical university in Hanoi." This study aims to benefit the author, teachers, test-takers, and anyone interested in language testing, specifically focusing on the reliability and validity of an English for Specific Purposes (ESP) achievement test.
Aims and objectives of the study
The primary goal of this study is to assess the reliability and content validity of the final achievement tests currently administered to second-year students at IU The research focuses on specific objectives aimed at enhancing the evaluation process.
This study aims to examine the alignment between the test specification and the content of the final achievement test, serving as evidence for the content validity of the assessment.
- to find out the level of reliability and the statistical evidence for the test‘s content validity from the results of the final achievement test
- to provide some suggestions for improving the test in terms of its reliability and validity.
Research questions
The study was designed to answer the three following questions:
Research questions 1: To what extent is the content (vocabulary, grammar, and reading section) of the final achievement test compatible with the test specification?
Research questions 2: To what extent do the final achievement test results reflect its content validity (in terms of item difficulty and item discrimination)?
Research questions 3: To what extent do the final achievement test results reflect its reliability?
Scope of the study
There are some reasons for the scope of this study as below:
This study is limited by time, conditions, abilities, and the author's interests, focusing exclusively on the evaluation of vocabulary, grammar, and the reading section of the current written final achievement test.
The three evaluated sections utilize dichotomous items, simplifying the author's data collection and analysis process.
The author's experience with assessment practice at IU has highlighted that the vocabulary, grammar, and reading sections are likely to have the greatest impact on the reliability and validity of the study.
This research paper, part of a minor M.A thesis, evaluates the vocabulary, grammar, and reading components of the final achievement test for fourth-semester Mechanical Engineering students at a technical university in Hanoi The analysis focuses on two key qualities of language testing: internal test reliability and content validity.
Significance of the study
This paper provides valuable feedback for English teachers, enabling them to identify the strengths and weaknesses of test content Consequently, this insight will assist them in designing more effective assessments.
Secondly, identifying the ―good‖ and ―not so good‖ points of the test will help improve the English designing tests, test specifications and course books more
4 efficiently Actually, when there is a match between the course books and the tests, the teaching and learning quality will help achieve higher results in their language learning
This resource will provide teachers and designers with valuable insights into assessing the internal reliability and content validity of tests Consequently, it can serve as a reference for researchers focused on creating effective assessments aligned with course objectives and the interests and levels of students.
Method of the study
The study employs a combination of methodologies including qualitative and quantitative methods to achieve the aims mentioned above
The study utilized a quantitative method to gather data on students' test results Following data collection, the author applied the Kuder-Richardson Formula 20 to calculate the internal reliability index and analyze the scores for interpreting item difficulty and item discrimination.
The author employs a qualitative method grounded in a literature review of language testing theories and principles, focusing on the characteristics of effective tests, as well as their reliability and validity This synthesis of existing literature enables the author to establish a framework for evaluating content validity in language assessments.
Structure of the thesis
The thesis is clearly organized into five main chapters:
Chapter 1: Introduction - presents such basic information as the rationale, the aims, the research questions, the significant of the study, the scope and the organization of the study
Chapter 2: Literature review - reviews a theoretical background on evaluating a test which includes these main points: language tests, major characteristics of a good test, language testing in EOP
Chapter 3: Research methodology - describes the methods used in this thesis It includes four main things: the overview of the research contexts and evaluated test,
5 the participants, data collection instruments and procedures and data analysis
Chapter 4: Research Result and discussion - provides the results of the study in response to the research questions
Chapter 5: Conclusion - offers conclusions of the thesis and practical implications for the test improvement In this part, the author also proposes some suggestions for further research on the topic
LITERATURE REVIEW
Language tests
Tests are defined differently depending on the researchers‘ point of view
Bachman (1990, p.20) defines the term ―test‖ as ―a measurement instrument designed to elicit a specific sample of an individual‘s behavior‖ The definition provides the basis and general of tests
According to Brown (2004), a test is a method for assessing an individual's ability, knowledge, or performance in a specific area A well-structured test should include various formats, such as multiple-choice questions with clear correct answers, writing prompts accompanied by scoring rubrics, and oral interviews guided by a scripted questionnaire and a checklist for expected responses Additionally, tests must provide a defined way to report measurements of an individual's capabilities across different formats, such as proficiency or vocabulary tests Despite the seemingly straightforward definition, creating an effective test is a complex process that combines both scientific and artistic elements.
Testing is a fundamental aspect of social life, as highlighted by McNamara (2000), who notes that throughout history, individuals have been tested to demonstrate their abilities and validate their credentials Language tests, in particular, play a significant role in many people's lives, serving as crucial gateways during key transitional phases in education.
7 employment, and in moving from one country to another Besides, they can be devices for the institutional control of individuals, and they should be understood and subjected to scrutiny
This research defines a language test as a collection of questions and problems designed to assess a student's abilities and knowledge in a foreign language they have learned While the focus is on language testing, the definitions provided apply to tests in general.
A specific purpose language test, as defined by Douglas (2000), is designed based on an analysis of a particular target language use situation This ensures that the test tasks and content authentically reflect real-world tasks, facilitating an interaction between the test taker's language skills and their knowledge of the specific purpose content Consequently, such tests enable us to draw conclusions about a test taker's ability to use language effectively within the designated domain.
In the context of IU, where the test is tailored for Mechanical Engineering students, Douglas's conclusion aligns well This study defines an ESP test as one that possesses a certain level of authenticity, facilitating interaction between the test taker's language skills and their specific purpose content knowledge, thereby enabling inferences about their ability to use language within the specific purpose domain.
Language tests may be of various kinds but different scholars hold different views on the types of language tests
Henning (1987), for example, establishes seven types of language test classifications, which are (1) objective vs subjective tests, (2) direct vs indirect tests,
The article discusses various types of tests, including discrete versus integrated tests, aptitude versus achievement versus proficiency tests, and criterion-referenced versus norm-referenced tests It also covers the distinction between speed and power tests, along with other test types A detailed description of these test categories, as outlined by Henning (1987), is presented in Table 1 below.
Table 1: Types of language tests by Henning (1987)
Objective tests - use close-ended questions
- do not need much consideration of the markers Subjective tests - use open-ended questions
- be scored based on the marker‘s judgments or opinions
2 Direct tests - test directly the language pattern
Indirect tests - use other mediums to test
3 Discrete tests - have only one language pattern
Integrated tests - combine different language patterns at one time
Aptitude tests are utilized to identify students suitable for special programs, while achievement tests evaluate students' knowledge in subjects they have already studied Additionally, proficiency tests serve to select students for specific fields of interest.
- compare candidates with an existing authentic version (usually use interviews)
Norm-referenced tests - use criteria or rubrics
Speed tests - consist of easy items
- have inadequate time Power tests - consist of difficult items
Henning's (1987) classification of tests is comprehensive, yet it lacks clarity regarding the specific tests included in type 7 Notably, a single test may be categorized into multiple types depending on the researcher's classification criteria.
According to the contents tested and the purposes of tests, as noted by Hughes
(1989), there are 4 main types of tests which can be illustrated in the following table:
Table 2: Types of tests by Hughes (1990)
- measure how much of a language someone has learnt irrespective of where, when, how and under what conditions language learning has taken place
- (the content) be not based on any course or program
- measure how much of a language someone has learnt with reference to a particular course of study or program of instruction
- (the content) be based on a course or program
- include final achievement tests and progress achievement tests
3 Diagnostic tests 4 Placement tests Final achievement tests
- find out student‘s strengths and problems
- place students at the right level/ class
- be conducted at the end of the course
- measure the student‘s progress during the course
From another perspective, McNamara (2000) divides language tests based on two criteria: test methods and test purposes
Table 3: Types of tests by McNamara (2000)
Traditional paper-and-pencil language tests
Performance tests Achievement tests Proficiency tests
- take the form of the familiar examination question paper
- assess either separate components of language knowledge or receptive understanding
- assess language skills in an act of communication
- most commonly test speaking and writing
- require raters to be well-trained to use an agreed rating procedure
- measure what language the students have learnt as a result of teaching
- look to the future situation of language use without any reference to the previous process of teaching
Traditional paper-and-pencil language tests
Performance tests Achievement tests Proficiency tests comprehension)
- have clear and fixed format and marking scales
(therefore it is easy to rate)
Hughes (1990) and McNamara (2000) offer a simplified classification of tests, both organizing them according to their purposes While Hughes categorizes tests based on their content, McNamara focuses on the methods used in testing.
Different methods of categorizing tests stem from various perspectives and approaches, yet they remain comprehensible and share a common explanatory framework According to the classifications of Henning (1987), Hughes (1989), and McNamara (2000), the assessment administered to the second-year Mechanical Engineering students at IU can be classified as a final achievement test.
To ensure effective testing, it is essential to clearly define the test's purpose and types, while also emphasizing the importance of well-designed test specifications The following section will provide an in-depth discussion of the key concepts related to test specifications.
A test specification serves as a generative blueprint for creating multiple equivalent test items or tasks (Davidson et al., 2001) These specifications can vary in format and design, but they consistently include two key components: sample test tasks and guiding language that instructs how to produce these samples (Fulcher and Davidson, 2007).
According to Fulcher and Davidson (2007), test specifications provide essential guidelines for formulating test items, organizing the test layout, positioning passages, and making various critical decisions during the test preparation process.
11 materials More importantly, they tell us the rationale behind the various choices that we make
Test specifications significantly influence test development by providing a focal point for critical evaluation by both developers and users They ensure equivalence, reliability, and validity, functioning not only as a mechanical blueprint but also as a formal record of essential discussions.
In search from Bachman (1996), test task specifications for English for General Purposes (EGP) include the following information:
Table 4: Components of test specification by Bachman (1996)
1 Describing the purpose of the test
Clarify specific uses, inferences about language ability, and specific decisions of test
2 Identifying and describing tasks in the target language use (TLU) domain
Describe TLU task types Provide a set of detailed descriptions of the TLU task types
3 Describing the characteristics of the language users/ test takers
Make explicit the nature of the population of potential test takers
4 Defining the construct to be measured
Make explicit the precise nature of ability we want to measure
5 Developing a plan for evaluating the qualities of usefulness
Include an initial consideration of the appropriate balance among the six qualities of usefulness and setting minimum acceptable levels for each
6 Identifying resources and developing a plan for their allocation and management
Identify the resources (human, material, time) and how to allocate and manage them
Douglas (2000) outlines essential components for test specifications in ESP tests, including the test's purposes, the TLU situation and tasks, characteristics of the language users or test takers, the construct to be measured, and a plan for evaluating usefulness Additionally, he introduces further components, which are detailed in the accompanying table.
Table 5: Components for test specifications for ESP test proposed by Douglas (2000)
Describe the purpose(s) of the test
Purposes, constraints on the test situation or limitations on equipment, etc
Describe the TLU (target language use) situation and list of TLU tasks
Major characteristics of a good test
2.2.1.1 Qualities of a good English for General Purpose (EGP) test
Everyday English for General Purposes (EGP) is utilized in various common situations (Delgrego, 2009) Scholars have proposed different criteria for assessing the quality of EGP language tests.
Bachman and Palmer (1996) suggests the six criteria as qualities of test usefulness rather than individual factors The ideas of usefulness can be expressed as in Figure
Usefulness = Reliability + Construct validity + Authenticity + Interactiveness +
Figure 1: Test usefulness (Bachman and Palmer, 1996)
In short, reliability is the consistency in test results across testing conditions Meanwhile construct validity is the degree of test scores to be interpreted and
The article discusses five essential qualities of effective assessments: accuracy in measurement, authenticity in relation to target language use, interactiveness to engage test takers, impact on stakeholders such as learners and educators, and practicality in terms of available resources for test development and implementation.
Brown (2004) mentions five cardinal criteria for ―testing a test‖ which will be illustrated in the figure below:
Figure 2: Five cardinal criteria for “testing a test” suggested by Brown (2004)
Leading scholars in testing emphasize the importance of test characteristics, particularly reliability and validity, as essential for interpreting and utilizing measures of language abilities These qualities are crucial in the development and application of tests (Bachman and Palmer, 1996) This study aims to examine these key measurement qualities in a test administered to a large group of second-year mechanical engineering students at IU Sections 2.2.2 and 2.2.3 provide brief discussions on reliability and validity, along with criteria for assessing test practicality, authenticity, and washback.
2.2.1.2 Qualities of a good EOP test
English for Occupational Purposes (EOP) is a subset of English for Specific Purposes (ESP) designed to address the English language needs of learners in their professional environments EOP courses are beneficial in both academic settings, where students prepare for future careers, and in business contexts (Dudley-Evans and St John, 1998).
EOP courses include specialized titles such as English for Mechanical Engineering, English for Nursing, English for Business, and English for Accountants As communication needs grow more complex, these courses must focus on specific areas rather than attempting to address a broad spectrum of job-related requirements.
To evaluate EOP or ESP test, Douglas (2000) mentions six qualities of good testing practice which are stated to be based heavily on the work of Bachman and Palmer
Bachman et al (1991), Davidson and Lynch (1993), and others identify six key qualities of assessment: validity, reliability, situational authenticity, interactional authenticity, impact, and practicality Validity refers to the accurate interpretations of test performance, while reliability emphasizes the consistency and accuracy of measurements Situational authenticity relates to the connection between the target situation and test tasks, and interactional authenticity involves the engagement of test takers' communicative language abilities Impact reflects the influence of the test on learners, teachers, and educational systems, and practicality encompasses constraints such as money, time, personnel, and educational policies.
Douglas (2000) presents qualities that differ from those identified by Bachman and Palmer (1996), as he emphasizes aspects more pertinent to Language for Specific Purposes (LSP) testing Specifically, while Bachman and Palmer view authenticity and interactiveness as distinct qualities of usefulness, Douglas consolidates them into a single quality.
Authenticity in language use can be understood through two key aspects: situational and interactional authenticity Douglas (2000) argues that authenticity should be viewed as a dichotomous concept, emphasizing that it is not merely a characteristic of language-specific purpose (LSP) texts or tasks, but rather exists within the language users as they engage with these texts or tasks In the context of LSP testing, authenticity is contingent upon the integration of target language use (TLU) situational features and the interaction of users' knowledge with the LSP task.
This research emphasizes reliability and content validity, aligning with the test qualities proposed by Douglas (2000), which are deemed more appropriate for the study's context Consequently, the author intends to adopt Douglas's definitions of test qualities, particularly focusing on reliability and content validity.
Different authors have defined reliability differently for many years
According to a 2001 study by the Council of Europe, reliability is a technical term that refers to the degree to which the same rank order of candidates is consistently reproduced in two different administrations of the same assessment, whether real or simulated.
Reliability refers to the consistency of test results across various administrations, forms, items, scorers, and other testing aspects, as highlighted by Bachman and Palmer (1996) and Education & Canti (2000).
In other words, reliability is a quality of test scores, and a perfectly reliable score, or measure, would be one which is free from errors of measurement (American Psychological Association, 1985)
Reliability, as defined by Johnson & Christensen (2019), is typically assessed using a correlation coefficient A coefficient of zero indicates no reliability, while a negative correlation signifies a faulty test Conversely, a reliability coefficient of +1.00 represents perfect reliability Researchers aim for strong, positive reliability coefficients, ideally close to +1.00, as this reflects high reliability in their assessments.
Reliability encompasses various aspects that warrant investigation This article will explore three key issues: the factors influencing language test scores, the different types of reliability, and the methods for examining test reliability.
According to Shohamy (1985), "No measure is perfect," indicating that even experienced raters are influenced by various external factors Bachman (1990) further elaborates on these factors, as depicted in Figure 2.2.
Figure 3: Factors affecting language test scores by Bachman (1990)
Language test scores are primarily influenced by the test taker's language ability Bachman (1990) identifies four key factors affecting these scores, which can be categorized into two types: systematic factors, including test method facets and personal attributes, and unsystematic factors, which are random and unpredictable.
Language testing in EOP
Bachman (1990) identifies language competence, a key aspect of language ability alongside strategic competence, as comprising two main categories: organizational knowledge and pragmatic knowledge These components of language knowledge are detailed in Table 11.
Table 11: Area of language knowledge by Bachman (1990)
(how utterances or sentences and texts are organized)
(how utterances or sentences and texts are related to the communicative goals of the language user and to the features of the language use setting)
(how utterances or sentences and texts are organized)
(how utterance or sentences are organized to form texts)
● Knowledge of rhetorical or conversational
(how utterances or sentences and texts are related to communicative goals of the language user)
(how utterances or sentences and texts are related to features of the language use setting)
● Knowledge of imaginative functions natural or idiomatic expressions
Understanding cultural references and figures of speech is essential in language assessment Douglas (2000) identifies three key characteristics of language for specific purposes tests: specificity, authenticity, and the separability of language and content However, Hamp-Lyons and Lumley (2001) argue that this characterization does not necessarily equate to a comprehensive theory.
Douglas (2000) introduces a model that differentiates between language knowledge, strategic competence, and background knowledge This model adapts Bachman and Palmer's concept of language knowledge alongside a revised version of strategic competence proposed by Chapelle and Douglas (1993) According to the model, language knowledge encompasses grammatical knowledge, which includes vocabulary, morphology, syntax, and phonology; textual knowledge, which involves structuring and organizing language into larger units and marking that organization through cohesion; as well as functional and sociolinguistic knowledge.
Table 12: Communicative language ability from Douglas (2000)
(knowledge of vocabulary, morphology, syntax, and phonology)
Understanding how to effectively structure and organize language into larger units is essential for achieving rhetorical organization and ensuring cohesion This functional knowledge encompasses the ideational, manipulative, heuristic, and imaginative functions of language, allowing for more effective communication and expression.
Sociolinguistic knowledge (sensitivity to dialects, registers, naturalness, and cultural references and figures of speech)
The process of assessment (evaluating the communicative situation and engaging a discourse domain, a cognitive interpretation of the context)
Goal setting (deciding whether and how to respond to the situation)
Planning (deciding what elements of language and background knowledge are required)
Control of execution (organizi ng the required elements to carry out the plan
Reading competence is the ability to understand, use, reflect on, and engage with written texts to achieve personal goals, enhance knowledge, and participate in society (OECD, 2010) This skill is crucial for accessing information and is fundamental not only for academic success but also for personal, professional, and social development (Zayas, 2012).
Reading skills are classified as "receptive skills," presenting a fundamental challenge in task design that effectively exercises these skills while also showcasing their successful application (Hughes, 2003) According to Hughes (2003), this challenge consists of two main issues: the uncertainty surrounding the existence of these skills and the difficulty in accurately measuring their effectiveness.
Hughes (2003) identifies two distinct reading operations based on purpose: expeditious reading, which is quick and efficient, and careful reading, which is slow and meticulous Each type necessitates different reading skills Urquhart and Weir (1998: 123) present a four-part matrix that outlines the skills and strategies for both careful and expeditious reading at global and local levels.
Table 13: Matrix of reading skills and strategies by Urquhart and Weir (1998)
To achieve a clear understanding of the main ideas and supporting details, it is essential to make propositional inferences Quick skimming can help identify the discourse topic, main ideas, text structure, and relevance to specific needs.
Search reading to locate quickly and understand information relevant to predetermined needs
Scanning to locate specific points of information
He suggests a number of parameters which can help specify the texts The following tables will demonstrate in details:
Table 14: Parameters help specify the texts by Urquhart and Weir (1998)
Text types encompass a variety of formats including textbooks, handouts, articles from newspapers, journals, and magazines They also include poems, encyclopedia and dictionary entries, leaflets, letters, forms, diaries, maps, plans, advertisements, postcards, timetables, as well as extracts from novels and short stories Additionally, reviews, manuals, computer help systems, notices, and signs are also considered important text types.
Text forms Description, exposition, argumentation, instruction, narration
Tables, charts, diagrams, cartoons, illustrations
Topics Non-technical, non-specialist, etc
Native speaking science undergraduate students, young native speakers
Length According to the level of candidates and purpose of the test
(expeditious reading or careful reading)
Readability Measure the difficulty of the text (objective but not necessarily very valid)
May be indicated by a complete list of words, by reference either to a word list or to indicate the frequency in a learner‘s dictionary Can be expressed more generally (e.g non-technical)
Maybe a list of structures, or a reference to a those to be found in a course book or (possibly) a grammar of the language
Detailed text specifications enhance test design and improve candidates' reading success This relates to content validity (Hughes, 2003), as it involves comparing test specifications with actual tests regarding coverage, representativeness, and relevance.
Effective time management is crucial in text processing and answering tasks in assessments Test developers must carefully sequence texts and tasks, ensuring adequate time is allocated for all activities Poorly planned time allotments can lead to unpredictable performance If candidates are given excessive time in a reading test or if time is not strictly regulated per section, they may focus too much on reading passages, undermining the intended assessment of their processing abilities.
Expeditious reading, which involves quickly and selectively extracting information, may not always engage these processes effectively When ample time is provided for reading tasks, a more thorough and linear approach to processing information is likely to occur (Weir, 2005) Consequently, it is essential to clearly indicate the time limits on test papers and for invigilators to encourage students to adhere to the given instructions.
Brown (2004) mentions that there are three major genres of reading which ultimately from part of the specifications for assessments of reading ability:
Academic reading encompasses a variety of materials, including general interest articles from magazines and newspapers, technical reports, professional journal articles, reference materials, textbooks, essays, papers, test directions, editorials, and opinion pieces In contrast, job-related reading involves practical documents such as messages, letters, emails, memos, reports, schedules, labels, signs, announcements, forms, applications, questionnaires, financial documents, directories, manuals, and directions.
Personal reading encompasses a wide array of materials, including newspapers, magazines, letters, emails, greeting cards, invitations, messages, notes, and lists It also includes schedules, recipes, menus, maps, calendars, advertisements, novels, short stories, jokes, drama, poetry, financial documents, forms, questionnaires, medical reports, immigration documents, comic strips, and cartoons.
According to Brown (2004), the genre of a text helps readers utilize specific schemata to derive meaningful interpretations He emphasizes that the content validity of an assessment procedure is significantly influenced by the text's genre For instance, in an English for tourism program, if learners focus on customer interactions for bus tours, their assessments should incorporate relevant materials such as guidebooks, maps, transportation schedules, and calendars.
In the same book of Hughes (2003), the author mentions some possible techniques of designing a reading task including: multiple choice, short answer, gap filling and information transfer
