3.3 Research instruments and materials
3.3.1 The pretest and the posttest
There were two tests assigned to participants. These two tests, namely the pretest and posttest were designed by virtues of following reasons.
Regarding RQ1, the two tests included the pretest (Appendix B) and the posttest (Appendix C). Keys of these two tests were displayed in Appendix D.
In terms of the format, there was a similarity among the two tests, each of which consiseds of four parts (Part 1, Part 2, Part 3, and Part 4). The format of these tests were identical although their contents changes relatively. Formats were similar in order to avoid participant’s estrangement which leads to invalidity of test scores.
If participants had been surprised by alien formats, scores of the tests would have been affected. Additionally, this resemblance could give rise to homogeneity or
parallelism among tests. Alternatively, format identicalness could result in reliable stability via a test-retest process, which benefited consistency of the assessment.
First, Part 1 consisted of mutilple choices in order to assess receptive knowledge of collocation. In this Part 1, there were fifteen multiple choices (Questions 1 to 15). Participants had to choose only one correct answer out of four options, three of which were distractors. The answer was the right verb suitable for the noun given in the sentence to form a right collocation. The right collocations were target collocations chosen because of high frequencies in BNC (British National Corpus). In contrast, wrong collocations were wrong word combinations or wrong meaning. Sometimes, the verbs of distractors were synonyms of the correct option;
however, it was untrue to use these verbs. This was because in special situations there was only one verb that could collocate with the given noun.
Another reason for a correct option in tests was that the collocation in the correct option was a fixed expression that was idiomatic. As explained in chapter 2 (part 2.1), idioms were formed because of native habitual uses. This is irreplaceable, and cannot be altered by other verbs.
This part 1 of the pretest and the posttest was in multiple choices because multiple choices are popular among previous studies on collocation (Gyllstad, 2009;
Gyllstad, 2017; Nguyen and Webb; 2016, and many other authors). Chan and Liou (2005) used synonyms of verbs to make this multiple choice format of their collocation tests. An example of the pretest and the posttest is
Example: Noise can ______ stress even when a person is sleeping.
A. cause B. affect C. make D. bring Answer: A. Cause
The nouns that were key words of target collocations were in reference to Lei and Liu (2018), Gyllstad (2007; 2009), Barfield and Gyllstad (2009), Ackermann
and Chen (2013), Chan and Liou (2005), Barfield and William (2006), Wang(2016), Chi, Wong and Wong (1994), Nesselhaulf (2003), and Peng (2016).
Part 2 of the pretest and the posttest comprised five questions (Questions 16 to 20). This part 2 was error recognition, or also known as the proofreading task.
This was designed in reference to Chang and Sun (2009, p.286). Chang and Sun (2009, p.286) stated that proofreading could give rise to writing skills and writing competence.
From a paragraph having five errors of collocations, part 2 of the pretest and the posttest required learners to select unsuitable word combinations. To do this task, learners had to recognise collocations among combinations in a paragraph with many sentences. An example can be seen in the part 2 of the pretest (Appendix A).
Besides the correct collocations, there were distractors of word combinations for test-takers to consider.
Part 3 had five questions (Questions 21 to 25); this part let test-takers fill collocations in gaps. These collocations were given with only initial letters of the verbs and Vietnamese meaning in a bracket. For example, “s_______ an exam” (trãi qua một kỳ thi) (Answer: sit an exam). This format of this part 3 was in reference to Webb and Kagimoto (2009), Szudarki (2012), Nataka (2006). The purpose of this test was to highlight differences between the collocations and their translated Vietnamese meaning. This design aimed to adopt the contrastive approach in learning collocations and to focus on major problems facing learners as for non-congruent collocations.
Sometimes, there were one or two technical collocations being embedded. This was compatible with Nesselhauf's (2003) types of collocations and Nesselhauf (2003)'s suggestions to teaching collocations.
Example: She showed these children a series of pictures and got them to t_____ what they had seen .( kể chuyện)
Answer: tell a story
Part 4 of the pretest and the posttest had five questions (Question 26 to 30).
Participants made sentences from a given noun by thinking of a suitable verb for the noun. If a sentence made by participants did not contain target collocations, it would be illegitimate and unmarked. This aimed at assessing productive collocation knowledge which had been used in many studies such as Daskalovska (2015).
Example: (DIARY)...
Answer: He has kept a diary since the age of thirteen.
To sum up, there had four parts in three tests given. Each part required students to various tasks. The three tests were designed in a wide range of test tasks such as multiple choices, spoting errors, making sentences from hints and making a whole new sentence from given words.
Despite the pretest and the posttest having had similar formats, they nonetheless included different collocations. Some collocations were located at later weeks for their idiomacity which could embarrass those who learn with corpus the first time. Besides, there were some collocations overlapped between the pretest and the posttest. Reasons for this are next discussed.
There were some compelling reasons why there are ten collocations repeated in the pretest and posttest. Firstly, those who had not known collocations in the pretest were tested again in the posttest to show if there would be any improvement.
Secondly, the rate of repeated collocations was one third of targeted collocations, which could produce minor influence on participants’ assessment as the whole.
Thirdly, presence of such repetition guarantees the two tests' consistency.
Had much repetition been given, much repetition would have produced errors in data collection. In fact, if all collocations had been repeated in both tests, unexpected effects such as memorization would have influenced results gained. In contrast, if the pretest and the posttest had no repeated collocations, it would have been impossible to observe participants’ improvement.
Another reason to avoid much repetition of target collocations in the pretest and the posttest is to hinder the teach-for-test effect which many researchers disagree with. These researchers took issues with teaching methods in which teachers only teach students some knowledge for exams, and then design a test containing this repetitive question items. It is recommended that the knowledge taught to students should be more than the knowledge in the tests.
Last but not least, differences of the pretest and the posttest promote an objective evaluation in a wide range of collocation types and task effectiveness on teaching each type. This means the number of collocation types in the pretest and that in the posttest are diverse.
Validity, reliability and practicality of the pretest and the posttest are also important. The two tests were in line with theories and previous studies as thereupon discussed.
According to Coombe, Davidson, O'Sullivan & Stoynoff (2012, p.37), a good test adopts three benchmarks: validity, reliability, and practicality. On this account, a good test encompasses principles: validity, reliability, practicality, authenticity and washback as marshalled in Brown (2004, p.19).
With regard to validity definitions, Coombe et al (2012, p.38) explained validity is when "a test measures what is supposed to measured". To classify, validity types are face validity, content validity, criterion-related validity.
These validity types can be explained in Coomber et al. (2012, p.38). On the one hand, criterion-related validity is simply understood when the test is similar to another good valid test. On the other hand, content validity refers to content of the test. This test’s content is valid if the proportion of knowledge in the test is equivalent to the proportion of knowledge in the class. Finally, face validity represents the test's physical appearance to suit expectation.
With regard to reliability, tasks chosen for the pretest and posttest were concurrently mentioned in numerous studies, in particular Aijmer (2009, p.70) and
other studies. These task tests were synthesized from different studies, especially for teaching collocations.
Considering reviews, the pretest and posttest items had been reviewed many times before being brought into use, which undertook drastic changes in selected collocations and the part of recognizing errors in a paragraph whose forerunner had been error-recognizing separated sentences. The pretest and the posttest were rewritten many times in line with criteria of the assessment production cycle. Indeed, Green (2014, p.42) stated that item writing and review should experience an assessment production cycle where items are repeatedly considered through many stages. These stages include specifications, item writing, item review, piloting, piloting review, operational assessment, assessement review.
Bachman and Palmer (2010, p.66-68) describe the elements of test specification including setting, rubric, input, response, and relationships between input and response. These concepts can be explained as follows.
First, rubrics are guidelines to score the tests. The rubrics of scoring the final part of the pretest and posttest can be seen in appendices. One important note is the pretest and the posttest were to assess collocation knowledge regardless of grammar.
Therefore, rubrics focused on collocations.
Secondly, Davidson and Lynch (2002, p.14), Green (2014, p.37) stated that prompt attributes are needed to instruct test-takers to do test tasks. If these instructions are not clear, testtakers can misunderstand and mistake the tests.