VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF LANGUAGES AND INTERNATIONAL STUDIES FACULTY OF POST-GRADUATE STUDIES -------- ĐỖ MAI CHI AN ANALYSIS OF THE THREE OFFICIAL CAMBRIDG
INTRODUCTION
Background of the study
In the past decade, there has been a notable rise in individuals from non-English speaking countries taking standardized English proficiency tests, such as the IELTS, to enhance their opportunities in a globalized society that values language and intercultural skills These tests, developed by reputable international examination boards, serve as proof of English qualifications for educational institutions and employers The increasing number of non-native test-takers reflects a sociolinguistic trend where English is utilized as a lingua franca (ELF) in the Expanding and Outer Circles (Kachru, 1986) This shift raises concerns about the appropriateness of assessing non-native speakers against native speaker standards, particularly since many non-native test-takers primarily use English for cross-cultural communication with limited interaction with native speakers Critics argue that standardizing tests based on Inner Circle varieties of English may not accurately represent the linguistic realities of a diverse global population (Arik & Arik, 2014).
Jenkins (2014) argues against Lowenberg's (1993) assertion that global English proficiency assessments should follow standardized norms set by native English speakers, emphasizing the need for testing organizations to embrace the diversity of English and its role in intercultural communication The IELTS tests, while viewed as reliable, are criticized for being overly comprehensive and lacking responsiveness in predicting the academic success of specific English language user groups (Leung et al., 2016) Furthermore, Davies, Hamp-Lyons, and Kemp (2003) highlight that these examinations do not account for the variety of English dialects, often penalizing speakers by enforcing native English standards Although there is growing interest from material developers, the integration of English as an International Language (EIL) remains in its infancy and faces numerous challenges.
Despite the paradigm shift in English as an International Language (EIL), significant research gaps remain, particularly the scarcity of empirical studies testing assessment frameworks in diverse educational contexts outside the Inner Circle There is a pressing need for research to explore the practical application of theoretical models in these environments Additionally, investigations into the impact of rapid digital communication on assessment methods are essential Longitudinal and comparative studies are also necessary to evaluate the long-term effectiveness and adaptability of EIL assessments across various contexts.
Research aims and research questions
This research aimed to evaluate assessment frameworks in diverse educational settings, specifically addressing the gap in understanding the representation of English as an International Language across three different contexts.
IELTS Cambridge test books In line with this aim, the following research questions were created:
1 To what extent is English as an International Language represented in IELTS Cambridge 1, IELTS Cambridge 9 , and IELTS Cambridge 18?
2 What are the differences between the three IELTS Cambridge books in terms of how English is represented as an International Language?
Scope of the study
Incorporating English as an International Language (EIL) into preparation materials for international English tests has become a significant topic of discussion among scholars, highlighting the need for comprehensive research in this area This study draws on the works of influential researchers such as Jenkins (2006), Byram (1997), Leung and Jenkins (2003), Bachman and Palmer (1996), and McNamara and Roever (2006) to develop an analytical framework Due to the limitations of this dissertation, the research focused on a selection of three books from the IELTS Cambridge series: IELTS Cambridge 1, IELTS Cambridge 9, and IELTS Cambridge 18 Employing a mixed methods research approach, the study collected and analyzed both qualitative and quantitative data The qualitative analysis involved coding the contents of the selected books according to specific criteria, while the quantitative analysis utilized SPSS to quantify these codes and examine their proportions and differences across the selected books.
Significance of the study
The study holds considerable significance in English language teaching and assessment
English has evolved into an International Language (EIL), serving as a global lingua franca that facilitates communication among individuals from various linguistic and cultural backgrounds This shift highlights the significance of English beyond native speakers, emphasizing its role in connecting diverse communities Analyzing Cambridge IELTS practice tests can provide valuable insights into this phenomenon.
4 of English as an International Language helps us to gain better insights into how English is used and understood in various international contexts beyond its native- speaking domains
The study examines how Cambridge IELTS practice tests address cultural and linguistic diversity among English learners globally By recognizing the varied linguistic backgrounds and cultural nuances that learners bring to their language-learning experience, it is essential to understand how the IELTS assessments ensure fairness and validity in evaluating diverse candidates.
The findings of the study have the potential to greatly influence English language teaching methodologies and practices Educators can leverage the insights from the analysis to customize their teaching approaches, materials, and strategies, ultimately enhancing student readiness for international English language proficiency tests such as IELTS.
Understanding the alignment between Cambridge IELTS practice tests and broader language education goals can provide valuable insights for policymakers and curriculum developers This knowledge can help shape more effective language policies and curricula, ultimately enhancing English language proficiency across diverse international contexts.
Organization
The study comprises five chapters, beginning with an introduction to the research problem and questions The second chapter explores the theoretical framework surrounding the global spread of English, emphasizing the importance of English as an International Language (EIL), assessment principles, and material evaluation criteria It also provides an overview of the IELTS Cambridge Practice tests utilized by candidates for test preparation The third chapter details the research methodology, which integrates qualitative approaches with document analysis and semi-structured interviews to generate results.
The fourth chapter presents a detailed analysis and discussion of findings from five interviews, building on insights from the third chapter The final chapter summarizes key themes identified in the findings and explores their potential pedagogical implications Additionally, it addresses the study's limitations and offers recommendations for future research.
LITERATURE REVIEW
Language test preparation materials
2.1.1 Definition of language test preparation materials
The definition of language test preparation materials has been discussed in numerous studies in the field of language assessment According to Anthony
Language test preparation materials are defined as instructional resources designed to enhance learners' language abilities and test-taking strategies for proficiency exams These materials, which include practice tests, vocabulary lists, grammar exercises, instructional videos, and test-taking strategies, closely align with the structure of the assessments They aim to boost performance through targeted practice and strategy development, while also engaging learners with the specific language skills evaluated in tests, as noted by various scholars.
2.1.2 Types of language test preparation materials
Language test preparation materials are essential for success in proficiency exams and are generally classified into four key types Each type serves a crucial purpose in equipping test-takers with the necessary skills and knowledge to excel in their assessments.
7 materials: practice tests books, study guides and review books, skill-building resources, and strategy guides
Practice test books are essential resources that closely mimic the actual test environment, enabling test-takers to become familiar with the test format, question types, and time constraints This familiarity is vital for reducing anxiety and boosting performance on test day According to research by Alderson and Hamp-Lyons (1996), the washback effect of practice tests demonstrates that repeated exposure to test-like conditions significantly enhances test performance by fostering familiarity and confidence.
Study guides and review books are crucial resources for test preparation, offering a thorough overview of key content areas like grammar, vocabulary, reading comprehension, and writing skills These materials typically feature summaries, explanations, and example questions that enhance understanding of specific topics According to Green (2007), targeted content review through these resources can greatly improve a candidate’s grasp of essential concepts, ultimately boosting their likelihood of success in exams like the IELTS.
Skill-building resources are essential for effective language test preparation, as they enhance key language skills like reading, listening, speaking, and writing According to Cohen (2012), engaging in activities such as extensive reading and listening exercises significantly boosts overall language proficiency, leading to improved test performance When these resources are incorporated into regular language learning routines, they help test-takers establish a strong foundation, ultimately increasing their chances of success in language proficiency tests.
Strategy guides are essential tools for improving test-taking skills, as they provide effective techniques for time management, question analysis, and navigating difficult sections of exams (Xie, 2013).
Strategic preparation for tests enhances performance by equipping test-takers with a clear plan, minimizing common errors Essential techniques like time management, educated guessing, and anxiety reduction are typically included in these preparation guides, making them invaluable resources for achieving success in exams.
This study emphasizes the significance of practice tests as a primary type of language test preparation material These tests are crafted to mimic the actual exam conditions, allowing test-takers to become acquainted with the format, question types, and time limitations they will face on the day of the test Specifically, the IELTS Official Cambridge Practice test books are detailed in Chapter 3, Methodology.
Language materials analysis
Materials analysis in language research is crucial for evaluating instructional resources like textbooks and digital tools, as it helps understand their role in language acquisition and cultural relevance Researchers assess aspects such as linguistic content, authenticity, and learner engagement According to Tomlinson (2012), effective materials should provide accurate linguistic information while also stimulating learners cognitively and contextually Additionally, McGrath (2016) highlights the necessity for adaptable materials that cater to diverse educational settings and the specific needs of learners.
Analyzing language materials requires assessing their effectiveness in fostering key language skills, including reading, writing, speaking, and listening It is crucial to evaluate the balance between skill-building exercises and communicative activities that enhance real-world language use.
Cunningsworth (1995) emphasizes the importance of evaluating language materials for their ability to cohesively integrate essential language skills that reflect authentic usage Furthermore, researchers focus on the cultural content of these materials to ensure they expose learners to diverse perspectives and prevent the reinforcement of stereotypes Harwood (2010) highlights that culturally responsive materials not only improve language competence but also expand learners' global understanding, which is crucial in today's interconnected world.
This study will explore various aspects of selected materials, focusing on their linguistic content, authenticity, and accessibility, while also assessing how effectively they prepare learners for a globalized world.
English as an International Language (EIL) and EIL assessment materials
2.3.1 The spread of English as an International Language
The number of English users has significantly increased, with approximately 380 million native speakers and over a billion people using it as a second language (Crystal, 1997; Michael & Fazard, 2008) McKay (2002) highlights that English serves as an international language, both globally and locally.
Moreover, the global status of a language is maintained when it has a
Every country acknowledges the significant role of language, often recognizing it in two primary ways: as an official language or as a second or foreign language for study.
Kachru (1985) clarified three concentric circles of English using countries
Figure 2 1 Kachru’s three circles of World Englishes (Kachru, 1985)
The Inner Circle refers to countries like Australia, Canada, the United States, and the United Kingdom, where English serves as the primary language This region is significant as it is the birthplace of English language norms and conventions, highlighting its ownership and cultural roots.
(b) The Outer Circle: where English is used as a second language in a multilingual country such as Singapore, India, and the Philippines
(c) The Expanding Circle: where English is widely learned as a foreign language such as China, Japan, and Korea
2.3.2 Features of an International Language
Smith (1976) defines "international language" as a means of communication among individuals from diverse nations (p 38) He elaborates on this concept by highlighting the important relationship between an international language and culture, making several key assertions about their interconnectedness.
+ Learners are not required to internalise the cultural norms of native language speakers
+ The ownership of an international language is no longer tied to any specific nation
+ The educational objective of learning the language is to empower learners to express their ideas and culture to others effectively
These assumptions apply to using English as an International Language (EIL) worldwide, as individuals employ English to interact with individuals from
English as an International Language (EIL) must be tailored to fit regional contexts, particularly in many nations within the Outer Circle, where it serves as a tool for broader communication In these countries, English users are not required to adopt the cultural norms of Inner Circle nations to effectively communicate This adaptability allows for a more inclusive and functional use of English across diverse cultural landscapes.
The development of a global or international language is characterized by four key features It emerges from the establishment of a global eco-cultural system, which includes the formation of an international market and business community, as well as the expansion of a worldwide scientific and cultural sphere Additionally, it coexists with local languages in multilingual settings, where bilingual speakers are common Unlike exclusive lingua francas that serve only the socio-economic elite, this international language is accessible to individuals across various social strata Finally, its spread is facilitated not by the migration of speakers, but through the language acquisition of many individuals.
English has evolved to embody nearly all characteristics of a global language, solidifying its status as the most international language A growing number of individuals are adopting it as a second language, making it essential for economic globalization Additionally, English is becoming the dominant language that drives the dissemination of mass culture worldwide.
2.3.3 An overview of the field of teaching English as an International Language
The teaching of English as an International Language (EIL) has become increasingly significant in academic discussions, with scholars like McKay (2002) and Brown (2014) emphasizing its importance in global English education Brown (2014) argues that English as a global language necessitates that EFL learners understand its implications and make informed decisions regarding their language use.
Teaching and curriculum development should integrate English as an International Language (EIL) within the local context This approach emphasizes four key areas: the content of EIL teaching and learning, effective EIL teaching methodologies, assessment practices for EIL, and the identities of teachers involved in EIL education.
2.3.3.1 Contents of teaching and learning EIL
The English language is widely recognized for its numerous varieties, reflecting its global prominence Major forms like American and British English are spoken by millions, while other localized varieties cater to specific communities The teaching of these diverse English varieties, as discussed by Bieswanger, highlights the importance of understanding regional differences in language use.
(2008), Lee (2018), and Pickering and Huang (2022) asserted that English education should not consist only of a dominant variety, but also consider those varieties relevant to the students
McKay (2002) highlights that English varieties differ in vocabulary, grammar, and phonology, with vocabulary constantly evolving and local terms emerging in multilingual contexts These local variations can lead to grammatical differences that have implications for English as an International Language (EIL) teaching, particularly in language assessment, where certain grammatical features may reflect a nativized variety rather than errors Teachers must recognize the appropriateness of these variations in fostering English proficiency Additionally, phonological differences are linked to intelligibility, with Jenkins (2000) asserting that English pronunciation need not mimic native speakers as long as it remains comprehensible to listeners.
Pragmatics, as described by Thomas (1995), involves creating meaning through interaction, making it essential for understanding cross-cultural encounters This approach emphasizes the collaborative role of both the speaker and listener in the meaning-making process, which is not confined to the words alone Instead, meaning emerges from a dynamic exchange influenced by contextual factors, including the physical environment, social dynamics, and the language used Achieving pragmatic competence involves grasping the illocutionary force of a statement, or the speaker's intention behind it In cross-cultural interactions, it is vital to recognize that the same statement can carry different meanings based on these contextual elements.
The phrase "When are you leaving?" can convey various meanings based on the context, such as a request for a ride or an indication that it's time for someone to depart In cross-cultural interactions, understanding the speaker's true intentions may pose challenges for the listener.
Pragmatic competence involves selecting the most appropriate expression for a specific meaning within a context, which can prevent cross-cultural misunderstandings In the realm of English as an International Language (EIL), two key aspects of pragmatics are crucial: cross-cultural pragmatics, which studies cultural differences in executing speech acts, and interlanguage pragmatics, which examines the communication behaviors of non-native speakers For a deeper insight into current research in interlanguage pragmatics, consulting the work of Kasper and Rose is highly recommended.
(1999) When considering alternatives to using a native speaker model for teaching pragmatic competence, it is beneficial to distinguish between the usage of English
In the context of English as an International Language (EIL), promoting a native speaker model is unnecessary when communicating across diverse nations To enhance cross-cultural communication, it is essential to raise awareness among both native and bilingual English speakers about the significant pragmatic differences that exist Additionally, individuals from various backgrounds can work towards fostering harmony in intercultural interactions Notably, research shows that bilingual English users in Expanding Circle countries view pragmatic violations as less severe than grammatical ones, likely because achieving grammatical accuracy is often deemed more critical for success in qualifying examinations than mastering pragmatic nuances.
Related studies and research gap in assessment materials analysis
Recent studies evaluating English as an International Language (EIL) materials reveal a significant lack of linguistic diversity Research by Yusop & Karakaş (2019) on IELTS and TOEFL tests highlighted that these assessments predominantly feature standardized English, limiting the representation of various English varieties Similarly, Hu & McKay (2014) analyzed a Chinese English textbook and found it inadequately supports multilingual practices, underscoring the necessity for resources that better reflect global linguistic diversity Furthermore, Karakaş (2022) examined international English language tests and concluded that they often overlook diverse English forms, adhering primarily to standard English norms The findings advocate for revisions in assessment criteria by testing boards to enhance the representation of non-native English users and promote inclusivity in English language education.
Some studies also found that EIL tests lack fairness and accessibility Khan
In 2009, a critique of international standardized tests from an English as an International Language (EIL) perspective highlighted significant concerns regarding fairness, accessibility, and global diversity The study revealed that tests like IELTS promote a narrow definition of language proficiency, which may exclude users who do not conform to this limited perspective Additionally, prior research indicated a tendency to prioritize accuracy over communicative competence Building on this, Rose and Syrbe (2018) investigated the alignment of English language assessment practices with the objectives of teaching English as an International Language, seeking to evaluate the effectiveness of current testing methods.
A study evaluating 28 methods for assessing practical English use revealed that standardized tests frequently prioritize linguistic accuracy aligned with native English standards, neglecting global and practical applications Rose and Syrbe (2018) advocate for a shift in testing focus towards measuring strategic language use, intercultural competence, and communication skills.
Despite advancements in understanding EIL representation in English assessments, significant research gaps remain Empirical studies are needed to evaluate assessment frameworks in diverse educational settings, especially in non-Inner Circle contexts, to see how theoretical models operate in real-world situations Additionally, the impact of rapid digital communication on English usage necessitates exploration within current EIL assessment models, focusing on how these changes should be integrated into evaluations Furthermore, the scarcity of longitudinal and comparative studies highlights the need for research that assesses the long-term effectiveness and adaptability of EIL assessment materials across various learner populations and cultural contexts.
This thesis examines the three Official Cambridge IELTS Practice Test Books from EIL Perspectives, which are recognized as reliable resources for IELTS test preparation by both learners and instructors Instead of relying on a single analytical model, the researcher adapts criteria from various evaluators to conduct the analysis, focusing on the experiences of actual users, including IELTS learners and teachers.
Chapter summary
This chapter reviews the literature that underpins the theoretical framework for the research titled "An Evaluation of the Three Official Cambridge IELTS Practice Test Books from the Perspectives of English as an International Language." It explores various definitions and aspects of English as a global language, highlighting its significance in the context of the IELTS preparation materials.
29 international language (EIL), assessment principles in EIL, EIL materials, and EIL materials evaluation have been discussed critically Evaluation work needs a systematic approach with a comprehensive, flexible framework
METHODOLOGY
Research context
3.1.1 The context of the IELTS test in Vietnam
The IELTS test is vital in Vietnam, influencing education and professional growth as a trusted indicator of English proficiency for students pursuing undergraduate or postgraduate studies Nearly all Vietnamese universities require an IELTS score of 4 or higher for admission, while graduation criteria typically range from 4.5 to 7, depending on the faculty This requirement underscores the test's significance in establishing a standardized benchmark for English proficiency across various levels of higher education.
Achieving a specific IELTS score can exempt 12th-grade students from the national English language exit exam, highlighting the test's importance in the Vietnamese education system This exemption reflects the high value placed on IELTS scores, providing students with a notable advantage in meeting their English language requirements.
In Vietnam, the IELTS test plays a crucial role beyond academic settings, particularly in the professional realm Numerous employers, including multinational corporations, investment banks, law firms, accounting firms, and insurance companies, mandate proof of English language proficiency during job applications and promotions Consequently, obtaining an IELTS certification is essential for career advancement and success in the competitive Vietnamese job market.
31 the test underscores its importance in professionally validating individuals' English language skills
A study conducted in Central Vietnam revealed that the IELTS examination positively impacts English language teachers by enhancing their language proficiency, teaching skills, and professional development opportunities It demonstrates that IELTS shapes both teaching content and methodologies, although this influence is affected by various factors, including teachers' autonomy and the diverse goals, proficiency levels, ages, and expectations of learners.
The IELTS test plays a crucial role in Vietnam's educational and professional sectors, acting as a vital benchmark for English proficiency It significantly influences students, educators, and professionals, impacting academic qualifications and career opportunities As such, the IELTS is an essential component of Vietnam's English language learning and usage landscape.
3.1.2 The Official IELTS Cambridge Practice test books
The Cambridge IELTS Practice test books offer comprehensive introductions that clarify the IELTS format and scoring system, ensuring both teachers and students have a solid understanding of the exam Featuring authentic examination papers from Cambridge Assessment English, these books provide an excellent opportunity for students to familiarize themselves with the IELTS and refine their examination techniques Each volume includes detailed explanations of the various modules, answers, additional clarifications, audio for Listening tests, tapescripts, sample Writing responses, and a video of a sample Speaking test, making them perfect for self-study.
Research questions
The present research aims to answer the two following research questions:
1 To what extent is English as an International Language represented in IELTS Cambridge 1, IELTS Cambridge 9, and IELTS Cambridge 18?
2 What are the differences between the three IELTS Cambridge books in terms of how English is represented as an International Language?
Research design
This research utilizes a mixed-method approach to explore the adaptability and effectiveness of educational resources in a global context, combining quantitative statistical analysis with qualitative content analyses Supported by educational research literature, this methodology enhances the reliability and depth of findings by validating quantitative results with qualitative insights The study comprehensively analyzed the EIL orientation in selected IELTS practice books, using qualitative methods to assess the representation of English as an International Language (EIL) and uncovering the real-world implications of the test materials Additionally, quantitative methods compared EIL representation across three books, providing a broad overview of how well these resources align with EIL criteria.
Sampling
The analysis focused on three IELTS Cambridge Practice Test Books: The IELTS Cambridge 1, The IELTS Cambridge 9, and The IELTS Cambridge 18 These books were specifically selected due to their official publication by the University of Cambridge, serving as essential materials for test preparation By examining various editions, the research aimed to identify linguistic changes over time and gain insights into the evolution of the IELTS testing framework.
The IELTS Cambridge 1, one of the earliest editions of the exam, provides valuable historical insights into the foundational principles of the IELTS test This edition features 33 cultural representations within its test materials, which enhances the generalizability of the findings and reflects the diverse contexts in which the IELTS is applied.
The latest editions of the IELTS Cambridge book series, specifically Books 9 and 18, highlight the evolving understanding of English proficiency Book 9 serves as a median reference, allowing for a more accurate observation of trends and changes over time compared to solely analyzing Books 1 and 18 Additionally, the ongoing relevance and global usage of these editions among test-takers enhance their significance in addressing practical concerns in language proficiency assessment A detailed description of these three books can be found in the Research Data section.
Data collection method
Document analysis served as the main data collection method, utilizing qualitative research techniques to interpret documents This approach allows researchers to extract meaningful insights and provide a deeper understanding of the assessment topic (Bowen).
Document analysis serves as a valuable method for gaining comprehensive background information and data coverage, particularly in educational research (Bowen, 2009) This approach is cost-effective and offers access to stable data that remains consistent over time, making it especially relevant for studies examining the representation of English as an International Language (EIL) across different materials (Morgan, 2022) In this study, document analysis was applied to IELTS Cambridge 1, IELTS Cambridge 9, and IELTS Cambridge 18 to assess the integration of EIL in high-stakes test preparation resources and to investigate how test designs and content have evolved in response to the growing significance of EIL The analysis involved coding and categorizing test items from these three publications.
The study utilized a deductive coding process based on 34 predetermined criteria, enabling systematic comparison of various book volumes To ensure the validity and trustworthiness of the research, feedback was obtained from two EIL material developers, which contributed to the creation of a robust framework for document analysis.
The research utilized a data collection method that involved quantifying the results from a document analysis of three official Cambridge IELTS practice test books (editions 1, 9, and 18) ANOVA analyses were conducted to identify statistically significant differences in the evaluation criteria for English as an International Language (EIL) across the various book volumes This analysis offered valuable insights into the alignment of the practice test books with EIL principles and their effectiveness in preparing learners for global communication.
Research data
This study analyzed three Official IELTS Cambridge Practice Test books published by Cambridge University Press, each containing four comprehensive tests that assess all four components of the IELTS exam: listening, reading, speaking, and writing.
Table 3 1 Overview of the test books
No The book Publisher Numbe r of tests
Reading Speaking Writing For the listening component, there were four parts in each test, as follows:
Table 3 2 Description of listening component
Test 1 Part 1: A woman calling a police officer about a missing briefcase
Part 2: A newsreader reporting the headlines for the day
Part 3: A conversation between university lecturer and student
Part 4: A talk given by a university lecturer about the structure of the university
Part 1: A man phoning to enquire about a job vacancy Part 2: A radio programme about the opening of a new local sports shop
Part 3: Two overseas students and their English lecturer discussing how they coped with a university course
Part 4: A lecture about a problem related to the behavior of sea creatures
Part 1: An interview with a woman who is doing a survey on transport Part 2: A woman speaking to a group of people who are interested in becoming volunteers
Part 3: Two students discussing a talk they have just attended related to their course
Part 4: A zoology student giving a presentation Test 2 Part 1: Students being interviewed by a university counsellor
Part 2: A radio talk about buying a bicycle
Part 3: A conversation between two students talking about a tutorial topic
Part 4: A lecture on health issues for new students
Part 1: A student talking to the student accommodation officer at a college
Part 2: A man giving a talk to new members of a wildlife club in the south of England
Part 3: The Director of Studies in an English language center talking to a student
Part 4: A lecture about business cultures
Part 1: A woman from a job agency giving information to a man about work in a chain of restaurants Part 2: A talk to members of the public about plans for a new housing development in a British town Part 3: Two geography students talking about a volcanic eruption Part 4: A student on a fashion design course giving a talk
Test 3 Part 1: Student asking about parking facilities
Part 2: A talk given by a guide showing a group of people around a museum
Part 3: A university student talking to a tutor and another student about a topic he has studied
Part 4: A talk on space management in supermarkets
Part 1: A telephone conversation between a travel company employee and a customer
Part 2: A talk on local radio about a children's theme park
Part 3: A geography student discussing her dissertation with her tutor
Part 4: A lecture about an experimental design for a house
Part 1: A man talking to the coordinator of a photography club Part 2: An expert on picking wild mushrooms talking on the radio
Part 3: Two business students discussing automation and the future of work
Part 4: A part of a lecture for astronomy students
Test 4 Part 1: Two students meeting on campus and talking
Part 2: A talk about banks for students in the UK
Part 3: Two students discussing a project that they are working on
Part 1: A man asking for information about health services in the place where he is living Part 2: A man who owns a holiday home talking on the phone to a woman who is staying there
Part 1: A man who works at an employment agency phoning a woman who is looking for a job
Part 2: A guide at a museum talking to visitors
Part 3: Trainee teachers discussing teaching
Part 4: A university lecture about a sports studies program
Part 3: A conversation between an English teacher and a former student Part 4: A talk about a project on the wildlife found in City Gardens in Britain techniques Part 4: A student giving a talk about a 19th century French writer
For the reading component, there were three parts in each test, as follows:
Table 3 3 Description of reading component
Test 1 Part 1: A spark, a flint: How fire leapt to life
Part 2: Is there anybody out there?
Part 3: The history of the tortoise
Part 1: Urban farming Part 2: Forest management in Pennsylvania Part 3: Conquering Earth's space junk problem
Test 2 Part 1: Right and left- handedness in humans
Part 1: Untitled Part 2: Venus in transit Part 3: A neuroscientist reveals how to think differently
Part 1: Stonehenge Part 2: Living with artificial intelligence
Test 3 Part 1: Spoken corpus comes to life
Part 2: Moles happy as homes go underground
Part 2: Tidal power Part 3: Information theory
Part 1: Materials to take us beyond concrete
Part 2: The steam car Part 3: The case for mixed- ability classes
Test 4 Part 1: Glass: Capturing the dance of light
Part 2: Why some women cross the finish line ahead of men
Part 1: The life and work of Marie Curie
Part 2: Young children's sense of identity
Part 3: The development of museums
Part 1: Green roofs Part 2: The growth mindset Part 3: Alfred Wegener
For the speaking component, there was one part in each test, as follows
Table 3 4 Description of speaking component
Test 1 Asking a representative for information at a university
Numerous questions related to games, markets, and shopping
Numerous questions related to paying bills and cooking
Test 2 Asking a lecturer for an extension on an assignment
Numerous questions related to gift-giving, activities, and learning
Numerous questions related to science and tourism
Test 3 Asking about holiday information
Numerous questions related to telephoning and travel
Numerous questions related to shopping and family life Test 4 Asking for information about an excursion
Numerous questions related to bicycles and community service
Numerous questions related to sleep and friends
For the writing component, there were two parts in each test, as follows:
Table 3 5 Description of writing component
Test 1 Part 1: Report to university lecturer describing chart data related to adult education motivations
Part 1: Summary of island before and after construction of tourism facilities
Part 2: Essay about learning a foreign language at a specific school level
Part 1: Summary of population density in cities across time
Part 2: Essay about the aim of science
Test 2 Part 1: Letter to university lecturer describing a process related to weather forecasting
Part 2: Letter to examiner asking for extension of assignment deadline
Part 1: Summary of phone calls made in UK across a period of time
Part 2: Essay about unpaid community service being part of high school programmes
Part 1: Summary of households in specific income brackets Part 2: Essay about studying for a qualification or learning about several subjects
Test 3 Part 1: Report to university lecturer describing trends in fast food consumption
Part 2: Essay about news delivery
Part 1: Summary of population growth of two countries
Part 2: Essay about improving public health through sports facilities
Part 1: Summary of library plan in the past and present Part 2: Essay about countryside population moving to cities
Test 4 Part 1: Report to university lecturer describing development of a British village
Part 2: Essay about having multiple careers
Part 1: Summary of energy consumption in USA across a period of time Part 2: Essay about language extinction
Part 1: Summary of changes in metal prices Part 2: Essay about benefits and disadvantages of an ageing population
Data collection instrument
In order to perform a qualitative content analysis on the three selected test books, the researcher created a framework to gather, characterize, and analyze the
The framework for evaluating English as an International Language (EIL) was developed based on specific inclusion criteria that emphasize teaching and learning content, as well as EIL evaluation principles After an initial version was created, the researcher collaborated with the study's supervisor to refine it, resulting in a second version that excluded the evaluation focus on 'linguistic norms' to better align with the study's objectives The finalized framework, detailed in Appendix 1, comprises five key criteria that are elaborated upon in subsequent sections.
3.7.2 Description of the framework for in-depth analysis
In this study, a set of criteria was employed to analyze the content of the chosen materials It was divided into five criteria as follows:
The criterion of "Linguistic diversity" assesses the representation of various English varieties in selected materials, crucial for aligning with the principles of English as an International Language and enhancing learners' global communication skills This evaluation investigates whether the books introduce candidates to a range of native and non-native English varieties Native varieties encompass dialects from inner-circle countries, such as American, British, and Australian English, while non-native varieties include those from outer-circle countries, like Japanese English and Indian English This domain is designated as ACC.
The second criterion, “Intercultural Strategic Competence,” assesses how well materials promote the ability to communicate effectively in diverse global contexts This criterion includes two key questions focused on whether the tests encourage intercultural communication among individuals from different cultures and whether they inspire candidates to explore and articulate culturally diverse and appropriate ideas This domain is identified as ICS and CDI.
The third criterion, “Pragmatic and Communicative Competence,” assesses the materials' effectiveness in preparing learners for global communication through four key questions This evaluation emphasizes the importance of both effective and precise communication in speaking and writing tasks, examines various communication patterns (monologue, L1-L1, L1-L2, L2-L2), and identifies opportunities for learners to engage with language relevant to work, study, and socialization This domain is categorized under EC, PC, CP, and EP.
The fourth criterion, labeled as AUTH and REL, focuses on "Authenticity and Contextual Relevance." It includes two key questions aimed at assessing the presence of authentic content within the books and evaluating the academic relevance of that content in context.
The fifth criterion, "Fairness and Bias Reduction," evaluates the representation of diverse cultures and the prevalence of Western perspectives in literary content This assessment includes three key questions aimed at ensuring balanced coverage and minimizing bias, categorized under the domains of Cultural Representation (CR) and Western Dominance (WD).
3.7.3 Applying the framework for analysis
I collaborated with a highly experienced English teacher, who holds a Master of Arts degree and has over ten years of teaching experience, to analyze a comprehensive set of criteria for evaluating IELTS Cambridge Practice Test 1 During our initial meeting, I shared detailed information about the study and the assessment criteria, which she agreed to Together, we examined the test to determine if any adjustments were necessary, ultimately concluding that the framework was suitable for further investigation We decided to conduct our analyses independently and reconvene in two weeks to compare our findings.
In our second meeting, we analyzed the findings and discovered a significant correlation between the two variables, which enhanced the reliability of our framework To indicate the presence of a criterion in the analyzed test, we used a checkmark.
Procedure of data analysis
The test books were analyzed based on five established criteria, with qualitative content being coded and quantified according to frequency This approach allowed the researcher to assess how well the test books represent English as an International Language (EIL) and to identify differences among them, addressing research question 1 Additionally, ANOVA analyses conducted using SPSS provided insights into significant differences between the three books, thereby answering research question 2 One-way ANOVA compares the mean values of two or more groups on a continuous variable, with statistical significance indicated by a sig value below 0.05 The results revealed significant differences in how the three books met the criteria, enabling the researcher to evaluate the importance of changes in EIL representation across the books.
To assess the linguistic diversity of the test books, the researcher analyzed the accents of speakers in each listening section and assigned specific codes accordingly This process involved an iterative approach, beginning with the researcher familiarizing themselves with a variety of accents through extensive exposure to audio and audiovisual materials, including YouTube videos.
The researcher utilized various online educational resources to understand accent differences and subsequently analyzed audio materials from test books by coding the identified accents Through multiple reviews for consistency, the researcher ensured the accuracy of the codes, which were then quantified for analysis To assess the representation of English as an International Language (EIL) within linguistic diversity, the researcher compared the percentage values of each accent Additionally, ANOVA analysis was performed to evaluate frequency differences among accents in the three books, with a significance threshold set at 0.05 to determine statistical significance (Pallant, 2020).
The researcher evaluated the intercultural strategic competence of test books by analyzing speaking and writing prompts for evidence of intercultural communication In the speaking context, non-native speakers responded to questions from native English examiners, and instances of intercultural communication were documented and quantified The study also identified opportunities for exploring and expressing culturally diverse ideas, with frequencies of such exploration calculated ANOVA analysis was performed to assess significant differences in cultural idea exploration among the three test books, as detailed in Appendix 3.
The researcher assessed the pragmatic and communicative competence of test books by analyzing preferences for effective communication versus grammatical precision, focusing on marking criteria for writing and speaking tests Each section was coded to reflect these preferences, and frequencies were calculated to gauge their proportions Additionally, communication patterns in the listening and speaking tests were identified using a specific coding system (refer to Appendix 2: Codebook) The researcher also evaluated the purpose of English in the tasks to see if candidates were encouraged to engage with language relevant to their lives To analyze these aspects, ANOVA analyses were conducted using SPSS, revealing any statistically significant differences among the three test books regarding communication marking criteria and patterns (see Appendix 3: ANOVA analysis data).
To assess the authenticity of the test books, the researcher coded authentic sections and compared their ratio to non-authentic ones Contextual relevance was evaluated by identifying tasks similar to academic English, which were also coded for comparison The study then analyzed the proportion of contextually relevant tasks against non-relevant ones Furthermore, ANOVA analyses were performed to detect significant differences in authenticity and contextual relevance across the three books, focusing on the quantity of both contextually relevant and authentic sections.
The researcher evaluated the fairness and bias reduction of test books by analyzing the representation of various cultural topics Each section's main topics were coded according to their cultural association, allowing for the calculation of cultural proportions Additionally, the study assessed the prevalence of Western dominance in the representation of people, places, and events by categorizing them as either Western or non-Western ANOVA analyses were performed to identify significant differences in cultural representation and Western dominance across the three examined books.
Following is an overview of the data analysis procedure for each of the criteria
Table 3 6 Overview of data analysis procedure
Linguistic diversity Accent diversity Identify accents of each section Assign code Calculate frequencies and percentages of accents ANOVA analysis Intercultural strategic competence
Identify prompts encouraging intercultural communication Calculate frequencies ANOVA analysis
Culturally diverse ideas exploration and expression
Identify instances where diverse ideas are explored and expressed Calculate frequencies and percentages ANOVA analysis
Effective communication over grammatical precision
Identify preference for either EC or GP Calculate frequencies ANOVA analysis Communication patterns Identify communication patterns Assign code Calculate frequencies ANOVA analysis
Relevance of English purpose Identify purpose of English Assign code
Calculate frequencies ANOVA analysis Authenticity and contextual
Authenticity Identify authentic (AUTH) sections
Calculate frequencies and compare to
44 relevance non-authentic sections ANOVA analysis
Contextual relevance Identify contextually relevant sections
(REL) Calculate frequencies and compare to non-relevant sections.
Reliability of qualitative data analysis and quantitative analysis with
Cultural representation Identify particular culture represented
Assign code Calculate frequencies ANOVA analysis
Western dominance Identify people and places associated with the West Calculate frequencies ANOVA analysis
3.9 Reliability of qualitative data analysis and quantitative analysis with ANOVA
To enhance the reliability of qualitative analysis, the researcher collaborated with a colleague for coding verification and adhered to Creswell and Creswell’s (2018) guidelines, emphasizing the importance of consistent code usage to avoid conflicting meanings This process involved creating a qualitative codebook, which serves as a dynamic tool that evolves as new insights are gained during data analysis Consequently, the researcher initially coded the data using a preliminary codebook but made necessary adjustments to the codes as relevant information emerged.
To ensure reliable ANOVA analysis, it is crucial that observations are independent, as highlighted by Pallant (2020) This means that measurements must not be affected by other observations to prevent significant violations of this principle In this study, each section of the books was analyzed independently, ensuring that the analysis of one section did not influence another.
Chapter Summary
Chapter 3 describes the methodology behind this research It first described the research context It also restated the research questions and described in detail
45 the framework of this research based on principles for EIL assessment suggested by
In the context of English as an International Language (EIL), Hu (2012) explores the significance of second language (L2) proficiency, drawing on Iyldyz's (2007) insights and incorporating elements from the analytic framework developed by Hu & McKay (2014) The article details the data collection process and outlines the research data, followed by an explanation of the data analysis process and the measures taken by the researcher to ensure reliability.