A pilot study on pet and aptis tests for english non – major students at hoa sen university

Aptis, non-majors, PET The English program at English Zone abbreviated to EZ, HSU, has been designed to equip the non-majors with the four basic language skills in English, namely readin

A PILOT STUDY ON PET AND APTIS TESTS FORENGLISH

NON-MAJORSTUDENTS AT HOA SEN UNIVERSITY

QUACH THI TO NU, VUONG NGỌC TIEN, HONG NGUYEN THANH TAM

Hoa Sen University;

tonuquachthi@gmail.com, tien.vuongngoc@hoasen.edu.vn, tam.hongnguyenthanh@hoasen.edu.vn

Abstract This study aims to investigate the appropriateness of PET and Aptis for students whose major

is not English (called non-majors) at Hoa Sen University (abbreviated to HSU), to decide on the more suitability of either tests and to give some recommendations for teaching English to the non-majors Both quantitative and qualitative research methods were employed in this study, which was mainly based on the Test Results of 39 non-majors Questionnaires were used to collect extra-information on those tests The research findings suggest that PET and Aptis should be suitable to HSU non-majors, that Aptis could

be more suitable than PET, and that the currently applied curriculum should be changed a bit to make the non-majors familiar with composing Email responses and Interactive writing in PET and Aptis

Key words Aptis, non-majors, PET

The English program at English Zone (abbreviated to EZ), HSU, has been designed to equip the non-majors with the four basic language skills in English, namely reading, writing, listening and speaking, to succeed academically and to be able to communicate fluently in the demanding working environment In order to graduate, the non-majors can submit one of those three certificates: TOEIC 550, TOEFL iBT 64,

or IELTS 5.0 These requirements are relevant to the Vietnamese Six-Level Language Competence Framework (VLLCF) 2020 Accordingly, Vietnamese graduates have to complete 3/6 in VLLCF (equivalent to B1 of CEFR) Students can choose a certificate of their own interest and orientation HSU has been cooperating with IIG to provide a free TOEIC exam to students who joined the whole English program at EZ It can be inferred that TOEIC is not the only exit certificate for the non-majors However, most of them choose it because EZ students can take a free exam once and non-EZ students do not have

to pay a lot of money for this two-skilled test

After years of organizing TOEIC exams to the non-majors, we have found out some drawbacks: Firstly, TOEIC no longer matches the English program objectives as the currently-used TOEIC test does not test all the four basic language skills in English Secondly, TOEIC is irrelevant to CEFR and VLLCF used for building EZ curriculum, choosing course books, teaching and assessing TOEIC, therefore, cannot be integrated into the curriculum’s testing and assessment Thirdly, most of the non-majors do not take the TOEIC exam right after finishing the English Program but wait until their graduation time, which results in knowledge loss and skill weakening Fourthly, EZ has to provide a TOEIC preparation course

as the TOEIC exam is far different from EZ’s English program Last but not least, only 58% of the non-majors get the required scores This is a waste of HSU investment and students’ time Hence, we need to find a substitute test (1) which is relevant to the training aims, HSU’s English program, (2) which tests all four skills, (3) which is relevant to CEFR and the VLCF, (4) which can be integrated into the teaching program to assess students during and after the program, (5) which can increase the percentage of successful English learners and guarantee the teaching quality, and (6) which is affordable

Those new conditions lead us to consider PET and Aptis to assess the non-majors competence after completing the EZ English program These new certificates are provided by prestigious organizations (Cambridge Examinations or British Council) whose assessment systems can be compared to CEFR and MoET In comparison to TOEIC, their prestige, confidentiality, validity and reliability of language assessment are equivalent Upon providing new test, HSU keeps recognizing TOEIC 550, TOEFL iBT 64 and IELTS 5.0 as English graduation benchmark There is a need for us to find out whether these two tests actually meet our hypothesis

2 LITERATURE REVIEW

2.1 What is PET

A Cambridge English: Preliminary (PET) qualification shows that you have mastered the basics of

English and now have practical language skills for everyday use

Results and score scale

Cambridge English: Preliminary (PET) results are reported on the Cambridge English Scale[1].You will

receive a separate score for each of the four basic language skills and use of English, giving you a clear understanding of your performance These five scores are averaged to give you an overall result for the exam You will also be given a grade and Common European Framework of Reference for Languages (CEFR) level

Cambridge English: Preliminary is targeted at CEFR Level B1, but also provides reliable assessment at the level above B1 (Level B2) and the level below (Level A2).The candidate’s overall score is averaged from the individual scores for each of the four skills[2]

Aptis is a business-to-business product whose clients are organizations and institutions using Aptis

to benchmark the English language levels of their employees, potential employees, students or teachers (British Council, English Language Assessment Research, 2014, p.7)[3]

The history and aim of Aptis development

Based on the main or general version of Aptis launched in August 2012 and Aptis development project conducted by an English Language Assessment Research Expert Group at the British Council, Aptis is designed and developed in order to reflect best practice in the area of language testing and fit with the British Council’s ambitions in the area of assessment literacy The British Council’s ambitions are to aim

at offering world-class advice and consultancy to many governments, institutions, education ministries, recruitment agencies, and corporation across the world The language teaching and assessment of the Aptis test is mainly linked to the referenced document − the Common European Framework of Reference for Languages (abbreviated to CEFR) [4] Consequently, the testing English levels from A1-C on CEFR, Aptis is an English test for adults (16 years old or older), which can be used to assess ability in all the four language skills.(Tucker, 2014: 11) [5]

Scoring

The report score of each candidate will be reported either on a numerical scale (0-50) or reported as a CEFR level The numerical scale score includes the overall scores of the four language skills except grammar and vocabulary while the CEFR skill profile will be in form of bar chart along with the CEFR skill descriptors The descriptors of a candidate’s ability which comprise the CEFR indicate the

progression to mastery and are presented on a rising six-level scale (A1, A2, B1, B2, C1, and C2) [6]

Aptis Cut Score Table [7] should be referenced for clarification

3 SCOPE AND PROCEDURE OF RESEARCH

The study done for and reported in this paper aims to explore the suitability of HSU non-majors with either PET or Aptis tests So the study addresses the following research questions:

1 What are HSU non-majors’ perceptions to Aptis?

2 What are HSU non-majors’ perceptions to PET?

3 What are HSU non-majors’ performances on the two tests?

Polit and Hungler (1999:37)[8] refer to the population as an aggregate or totality of all the objects, subjects or members that conform to a set of specifications In this study, the population was 40 upper-intermediate HSU non-majors

Elmes et al (2012:56)[9] stated that random selection is related to sampling Therefore, it is most related the external validity of the results In this research, random selection has been employed through these processes

First, more than 200 EGC4 students aged from 19 to 21 and coming from different majors are informed to register for the new tests as an extra benefits different from TOEIC Second, only 80 students getting 5.6 to 10 score (in which their average score for each skill must be at least 5) in EGC4 are qualified They are put into apreparation class equipping them with knowledge and strategies for both tests Nearly half of them have been learning English for six to ten years, and up to 37% of the students have a prolonging experience with English when they have been exposed to English since the age of six Nearly 50% of the students started at EZ at the Intermediate levels (EGC1 or EGC2) It can be inferred that their Entry levels were relatively high Third, after the mock test (a mixed test of PET and APTIS question types) at the end of the preparation time, the shortlisted 40 students are divided into two equivalent groups of 20 for PET and 20 for Aptis Each group comprises average, fair and good scores The students do not know which real test they will take until 7 days before the exam

EGC 4 scores

The final exam scores of EGC4 students in semester 15.A2 at HSU were used to select participants whose scores range from 5.5 to 8.5

Mock Test

The mock test, which consists of four parts, is a mix of the two tests: PET and Aptis [10, 11, 12, 13] The two-folded purpose is to familiarize the students with both of the tests’ question types and to divide them into two equivalent groups for the real tests later

Part 1 is a 30-question Listening pack divided into five sessions focusing on ticking the correct picture, MCQs, gap fill, YES/NO, short conversations Part 2 is a 18-question Grammar and Vocabulary focusing on matching definition and MCQs Part 3 is a 28-question Reading pack divided into four parts focusing on putting the story events in the correct order, matching the heading, matching the people with the correct interest, gap fill Part 4is a three-part Writing pack focusing on sentence completion, stating feelings on a club, formal and informal emails

Pre-Test Questionnaire (See Appendix B.)

The students are first requested to fill in some background information including their full name, gender, age, major, years of learning English, and English levels taken at EZ Then they are asked to give their opinions on the five following parts Part 1 is on test-taking skills: the students evaluate their own knowledge and familiarity to PET and APTIS question types Part 2 is on test psychology: the students express their feelings, aims, seriousness, motivation and confidence on the test Part 3 is on the Mock test evaluation: the students evaluate the clarity of the test instruction, their time management skill, their familiarity with test question types, and the speed and sound quality of the test audio Part 4 is on the relevance of EGC program to the Mock Test: the students identify the overlapping question types between EGC program and PET/APTIS test Part 5 is on the review session: the students evaluate the usefulness of the review days on their test-taking skills and knowledge

Post-Test Questionnaire (See Appendix C)

The students are requested to fill in some background information including their full name, gender, age, major, years of learning English, and English levels taken at EZ Then they are asked to give their opinions on the six following parts Part 1 is on the overview feelings on the exam: the students evaluate their feelings on doing the test on computers, exam instructions, their familiarity with question types, the time allotted, their results on paper versions, the review time, and the review session content Part 2 is on the Reading Test: the students evaluate their Reading skill, Reading question types, and reading obstacles Part 3 is on the Listening Test: the students evaluate their listening skill, listening question types, and listening obstacles Part 4 is on the Writing Test: the students evaluate their Writing skill, Writing question types, and writing obstacles Part 5 is on the Speaking Test: the students evaluate their Speaking

skill, Speaking question types, and Speaking obstacles Part 6 is extra information on their attendance frequency in the review session, their preference on those tests, their self-study review time, and their further suggestions for the teachers and university

PET/Aptis results

Since the two real tests are confidential, we cannot get them from the test providers So in this part, we just describe the general format of the two tests

PET test [14] is made up of three papers: Reading and Writing (1 hour 30 minutes), Listening

(36 minutes, including 6 minutes' transfer time), Speaking (10-12 minutes per pair of candidates)

Aptis test [15]is described as follows: Core (25 minutes), Reading (30 minutes), Listening (25-50

minutes), Writing (50 minutes), Speaking (12 minutes)

In this study, SPSS was used to analyze the collected data Independent sample T-test analysis was used to find whether there were any differences among EGC4 average scores of each skill, between PET and Aptis mock test and real test results

4 RESULT OF RESEARCH

Mock test results comparison

Figure 1 Mock test results of PET and Aptis groups

Figure 1shows that individual scores in two groups PET and Aptis ranged from 40 to 90 Aptis group’s average score is 65.16 while that number for PET is 65.93 It can be concluded that two groups have equivalent band scores, peaking at 70

- Independent samples T-test

Table 1.Mock test results comparison between PET/Aptis

N Mean

Std

Deviation

Std Error Mean T Df

Sig (2-tailed)

Mean Difference

Std Error Difference Listening Aptis 19 72,807 12,8266 2,9426

,111 37 ,912 ,474 4,274 PET 20 72,333 13,8116 3,0884

Reading Aptis 19 70,677 17,6407 4,0471

,172 37 ,865 ,855 4,984 PET 20 69,821 13,2862 2,9709

Writing Aptis 19 67,434 11,7073 2,6858 ,434 36 ,667 1,645 3,792

PET 19 65,789 11,6682 2,6769

Grammar Aptis 19 49,708 20,7435 4,7589

-,374 37 ,710 -2,515 6,722 PET 20 52,222 21,2048 4,7415

Mock

Test

Aptis 19 65,156 11,8153 2,7106

-,214 36 ,832 -,776 3,633 PET 19 65,932 10,5451 2,4192

An independent sample t-test was run to see if there is a significant difference between two groups in individual scores It can be seen from table 1 that sig level of the Mock Test and all individual skills are bigger than 0.05 (p>0.05), resulting in no differences in individual and overall scores between two groups In other words, two groups performed similarly to each other in both overall and individual scores

EGC4 Scores Comparison

Figure 2 Average scores of EGC4 between two groups PET and Aptis

As can be seen from Figure 2, the students in two groups have average scores ranging from 5.5 to 8.5 Aptis group’s average score is 67.32, and PET’s is 67.55 The figure also shows that two groups have similar average scores peaking from 6.5 to 7.5

- Independent samples T-test

Table 2.The comparison of EGC4 average scores between PET and Aptis

N Mean

Std

Deviation

Std Error Mean T df

Sig (2-tailed)

Mean Difference

Std Error Difference Listening Aptis 19 71,579 14,9120 3,4211

-,086 37 ,932 -,421 4,891 PET 20 72,000 15,5935 3,4868

Reading Aptis 19 65,000 17,3606 3,9828

-,076 37 ,940 -,375 4,952 PET 20 65,375 13,4084 2,9982

Writing Aptis 19 62,368 16,7803 3,8497

-2,723 31 ,011 -12,382 4,547 PET 20 74,750 10,8185 2,4191

Speaking Aptis 19 73,070 11,0730 2,5403

,168 36 ,867 ,614 3,647 PET 19 72,456 11,4040 2,6163

Vocabulary Aptis 19 58,947 20,2470 4,6449

-,096 37 ,924 -,553 5,739 PET 20 59,500 15,3811 3,4393

Grammar Aptis 19 60,526 17,4717 4,0083

,331 37 ,743 2,026 6,131 PET 20 58,5 20,5899 4,6040

EGC4

AVERAGE

SCORES

Aptis 19 67,315 6,91257 1,5858

-,097 37 ,924 -,234 2,423 PET

20 67,55 8,13358 1,8187

EGC4 AVERAGE SCORES EGC4 AVERAGE SCORES

Table 2 shows the comparison of EGC4 average scores and detailed scores between Aptis and PET groups through T-tests The sig level values of average scores and detailed scores comprising of Listening, Reading, Speaking, Vocabulary, Grammar are all greater than 0,05 (p>0.05), it can be concluded that there is no significant difference between two groups In other words, two groups have equivalent average scores and detailed scores As for Writing scores, with t-test results are t=2,723, df=31, p=0,011 As p<0.05, we can conclude that two groups have a significant difference in Writing Skill In brief, PET group has better Writing score than Aptis group

Pre-Test Questionnaire Comparison

Table 3 Self-evaluation of necessary skills and knowledge for the exam

I have sufficient vocabulary and grammar for the tests

I have adequate listening skills for the tests

I have adequate reading skills for the tests

I have adequate writing skills for the tests Aptis PET Aptis PET Aptis PET Aptis PET Strongly

disagree

% within Groups 0,0% 5,3% 0,0% 5,3% 0,0% 5,3% 0,0% 5,3% Disagree Count 1 1 1 1 0 0 0 0

% within Groups 5,6% 5,3% 5,6% 0 0,0% 0,0% 0,0% 0,0% Neutral Count 3 5 1 2 1 3 4 2

% within Groups 16,7% 26,3% 5,6% 10,5% 5,6% 15,8% 22,2% 10,5% Agree Count 8 11 11 13 14 13 9 14

% within Groups 44,4% 57,9% 61,1% 68,4% 77,8% 68,4% 50,0% 73,7%

Strongly

agree

% within Groups 33,3% 5,3% 27,8% 10,5% 16,7% 10,5% 27,8% 10,5%

After the revision time, the students claimed that they had been well-equipped with Vocabulary, Grammar, Listening, Writing and Reading When compared, Aptis groups thought that they had better Vocabulary, Grammar competence, Listening, and Reading Meanwhile, PET group thought they had better Writing skill Although two groups self-evaluated differently, Mann-Whitney U test revealed that such difference is not statistically significant

Table 4 listed students’ feedback on their attitude and psychology In general, two groups of students are quite confident Most of them had motivation and serious attitude to the exam They, therefore, are confident in their test and time management skills Despite the varied answer percentage, Mann-Whitney

U test results show that such difference is not statistically significant

Table 4.Students’ attitude and psychology before the exam

I am confident before the test

I have clear objectives for the test

I take the test seriously

I am motivated to take the test

I am confident with both tests

I have the skills to complete the tasks in both tests

I know the structure of the tests to use time effectively Aptis PET Aptis PET Aptis PET Aptis PET Aptis PET Aptis PET Aptis PET

Strongly

disagree

%

within

Groups

0.0% 5.3% 0.0% 5.3% 0.0% 5.3% 0.0% 5.3% 0.0% 5.3% 0.0% 0.0% 0.0% 0.0%

%

within

Groups

0.0% 5.3% 0.0% 5.3% 0.0% 0.0% 0.0% 0.0% 5.6% 10.5% 0.0% 5.3% 0.0% 5.3%

%

within

Groups

44.4% 15.8% 5.6% 0.0% 0.0% 0.0% 16.7% 15.8% 44.4% 47.4% 16.7% 5.3% 27.8% 26.3%

Agree Count 8 11 14 12 12 13 13 10 8 5 14 16 13 12

%

within

Groups

44.4% 57.9% 77.8% 63.2% 66.7% 68.4% 72.2% 52.6% 44.4% 26.3% 77.8% 84.2% 72.2% 63.2%

Strongly

agree

%

within

Groups

11.1% 15.8% 16.7% 26.3% 33.3% 26.3% 11.1% 26.3% 5.6% 10.5% 5.6% 5.3% 0.0% 5.3%

PET and Aptis test results comparison

- Skill-based estimation

Table 5.The skill-based result comparison of the two groups −Aptis and PET

Listening Reading Speaking Writing Aptis PET Aptis PET Aptis PET Aptis PET Number of students 19 20 19 20 19 20 19 20

Average score (%) 66,53 45,7 76,00 48,7 68,63 53,1 78,74 55,2

B1 21% 40% 26% 50% 68% 80% 26% 85% B2 58% 20% 21% 10% 16% 10% 68% 5%

It is clearly shown from Table 5 that, in all four skills, Aptis test-takers have achieved higher average scores When contrasted to the CEFR standard, the percentage of students in the Aptis group gaining a B2 and above is comparatively higher than that of the PET group in terms of three of the four skills including Listening, Reading and Writing However, the PET group’s writing score in EGC4 outweighed its counterpart

- Overall result estimation

Figure 3.Aptis and PET test result comparison Students’ test-taking skills for both tests are revealed in Figure 3 Notably, a majority of Aptis test- takers got scores of higher than 60% or above while half of PET test-takers scored 40%-60%, together with 75% of those receiving lower than 60%

Table 6 CEFR standard-based APTIS and PET test result comparison

Number of students 19 20

Average score (%) 72.47 50.6

B1 Lower 0% 55%

B1 Upper 16% 10%

A percentage of 95% students gained from Upper B1 to higher standard which meets HSU graduation requirements. Besides, 63% of the students joining Aptis test reached B2 and 16% reached C

On the contrary, there was an equally minimal number of 10% of Pet group getting B1 and B2, and none

of them reached level C

- Independent samples T-test

Table 7 General and skill-based Aptis and PET test result comparison

Group Statistics T-test for Equality of Means

N Mean

Std

Deviation

Std

Error

Sig (2-tailed)

Mean Difference

Std Error Difference Listening Aptis 19 66,526 11,7159 2,6878

2,694 24,499 ,013 20,126 7,470 PET 20 46,400 31,1708 6,9700

Reading Aptis 19 76,000 20,1329 4,6188

3,685 37 ,001 27,300 7,408 PET 20 48,700 25,6394 5,7331

Speaking Aptis 19 68,632 13,7768 3,1606

3,262 37 ,002 15,532 4,761 PET 20 53,100 15,8210 3,5377

Writing Aptis 19 78,737 9,5268 2,1856

6,091 33,301 ,000 23,537 3,864 PET 20 55,200 14,2519 3,1868

Test

Result

Aptis 19 72,473 11,1821 2,5653

4,534 37 ,000 21,799 4,808 PET 20 50,675 17,8931 4,0010

Table 7 describes the general Aptis and Pet test result comparison along with the single skill results

by using the T-test application Basing on the value of sig level of Test Result and the four components, Listening, Reading, Speaking and Writing lower than 0,05 ( p˂0.05), it can be concluded that the discrepancy of the general result of Aptis and PET as a whole, as well as the specific skill result is statistically significant

Post-Test Questionnaire Comparison

After the real exams, the students were requested to feedback on various aspects of the exams In general, the majority of the students had positive feedback As for computer-based exam, Aptis and PET groups claimed that they experienced no difficulty performing their exam Most of the students stated they understood the exam instructions clearly, mastered question types, and had sufficient time for the exam Although most of them had positive feedback on computer-based exam, about 40% of Aptis group, and 50% of PET group thought they would have better results if doing the test on paper, while the percentage of disagreement were 27% and 15% for Aptis and PET, respectively Despite the students’ varied percentage in answers, Mann-Whitney U test results show that such difference is not statistically significant

Table 8 Students’ post-test feedback

I had no difficulty doing the test on computer

I understood all instructions in the test

I was familiar with all task types

I had enough time to finish the test

I would have had better result

if had done the test on paper

Strongly

disagree

% within Groups

% within Groups

5,6% 10,5%

% within Groups

Agree Count 6 10 9 7 10 13 8 11 5 4

% within Groups

% within Groups

5 CONCLUSION AND RECOMMENDATIONS

The findings of the study are presented in the order of the research questions The analysis of EGC4 results and Mock Test results revealed that the two groups of students are equivalent in English level The data collected from the questionnaires showed that students had similar feedback Both groups were quite confident in their own knowledge and test-taking skills before the tests Likewise, two groups were positive in their test performance regarding their comfort in doing computer-based tests as well as the test content

However, significant differences were found in tests results between two groups Aptis groups excelled in the total scores and band scores As a result, Aptis students were ranged from Upper B1 or up

in CEFR Meanwhile, PET students received considerably lower scores The analysed data revealed that both PET and Aptis tests are suitable to HSU non-majors’ skills and knowledge acquired from the English program at EZ Both tests are held by the international prestigious organizers and in reference of CEFR They, therefore, have equivalent reliability However, their different scoring systems may lead to lower results in PET group

Considering the test reliability, the correlation to EGC program, the students’ feedback, exam fees, test results-informing time, the convenience in exam organizing, we suggest using Aptis test as a Foreign Language Graduation Exam at HSU

Basing on the interview with students and questionnaires, some recommendations are put forward as follows:

As for skills, first, it is highly suggested that HSU and its test provider − British Council should provide more opportunities for test-takers to frequently practice computer-based oral skill so that students will gain more confidence having real speaking exams Additionally, seating arrangement in the speaking test room should be reconsidered because, despite the supply of headphone, students were distracted by the noise caused from too close seating Second, a more challenging vocabulary source and practice are necessary for students as, although the Aptis group got better result, they still found the vocabulary section difficult Third, more time for regular writing practice ought to be added in the review sessions so that students can be able to be familiar with Aptis writing format Fourth, British Council should supply more practice tests for HSU to be used as review materials

As for population, a suggestion for further study on this issue is to have frequent reminders for EZ learners who volunteer to join full review sessions and try not to miss the real test so as to guarantee the population

The present study was conducted with limited number of students due to the budget Hence, similar studies should be conducted with bigger population to have a larger picture of the studied issue

APPENDIX

Appendix A: Terms and Acronyms

EZ English Zone, a language center teaching English to students whose major is not Englishat

Hoa Sen University HSU Hoa Sen University

PET Preliminary English Test, provided by Cambridge Examination

Aptis A modern and flexible English language proficiency test provided by British Council TOEIC Test of English for International Communication, provided by IIG

CEFR The Common European Framework of Reference for Languages

EZ students HSU students who study English at EZ

non-EZ

students

HSU students who do not study English at EZ

EGC4 English for Global Communication level 4- using Upper intermediate Solutions books by

Oxford University