Using the DASH Method to Measure Reading Comprehension

To help moderate the effect of these variables, an innovative experimental approach, the duplicated answer sheets DASH method, was designed with the purpose of investigating the influenc

Using the DASH Method to Measure Reading Comprehension

WENYUH SHIEH

Minghsin University of Science and Technology

Hsinchu, Taiwan, Republic of China

MARK R FREIERMUTH

Gunma Prefectural Women’s University

Gunma, Japan

Vocabulary knowledge has been recognized by researchers as a critical component of reading comprehension, not only as a means to facilitate first language acquisition but also as an integral element in the learning

of English as a second or foreign language To reduce the effect of insufficient vocabulary knowledge during reading, a compensatory strategy for many English as a foreign language (EFL) readers is dictionary consultation Several studies have investigated the effect of dictionary use on second language reading comprehension; however, findings have been inconclusive It is our contention that the reasons behind the inconsistent research findings stem from different manipulations of the related variables influencing reading comprehen-sion To help moderate the effect of these variables, an innovative experimental approach, the duplicated answer sheets (DASH) method, was designed with the purpose of investigating the influence of dictionary use on text comprehension using a timed-interval assess-ment With this in mind, the DASH was given to five proficiency-leveled groups of EFL students at a university in Taiwan The groups were further divided into participants who used dictionaries and those who did not The results indicate that dictionary use has little effect during the early stages of testing However, during subsequent test intervals dictionary use seems to benefit learners’ text comprehension, regard-less of proficiency level This research provides strong evidence that second language learners, given enough time, will benefit from using dictionaries as a tool to assist in text comprehension

doi: 10.5054/tq.2010.217676

R eading comprehension has always been a core issue in second language teaching research, and, as such, it continues to warrant a significant amount of investigation One of the key factors affecting a reader’s capability to comprehend a text is vocabulary knowledge Studies on text readability indicate that difficulties comprehending

vocabulary can account for up to 80% of predicted reading variance (Alderson, 2000) Moreover, researchers have posited that a learner’s vocabulary knowledge can be correlated predictably to performance for reading comprehension tasks (see Crow, 1986; Yorio, 1971) Anderson and Freebody’s (1981) research supports this notion; in their study, they found that a causal relationship exists between vocabulary knowledge and reading comprehension As they point out, learners’ vocabulary knowledge is an important indicator of their language proficiency and

an independent predictor of students’ text comprehension ability (cf Perfetti, 1985) Coady (1993) concurs, claiming that the more vocabulary knowledge one has the better reading comprehension ability one will have

A common compensation strategy practiced by language learners lacking vocabulary knowledge is dictionary consultation Although this practice has been discussed in reading research, the studies aimed at finding the benefits or drawbacks of using dictionaries have produced a mixed bag of results The purpose of this study, then, is to investigate further the relationship between dictionary use and reading compre-hension

LITERATURE REVIEW

Dictionary Use

A review of research studies investigating the relationship between dictionary use and reading comprehension reveals a mix of advice from researchers Bensoussan, Sim, and Weiss (1984) question the benefit of using dictionaries in text comprehension In their study, three groups of advanced English as a foreign language (EFL) learners (i.e., mono-lingual dictionary group, bimono-lingual dictionary group, and without dictionary group) took a battery of reading comprehension tests The authors concluded that dictionary use had little effect on comprehen-sion and tended to slow learners down Participants who relied on dictionaries tended to be students with lower proficiencies; participants with higher reading proficiency scores tended to feel comfortable without using dictionaries Additionally, the researchers concluded that language learners, especially at lower proficiency levels, may not be particularly adept at using dictionaries They also suggest that students with lower reading proficiency levels feel a strong need to depend on dictionaries for compensation during test-taking situations, because they may not have the capability to apply contextual guessing strategies or they might not possess adequate vocabulary to draw inferences In light

of these conclusions, Bensoussan et al suggest that teachers provide opportunities for students to improve their awareness of strategies, such

as identifying important words in a text and identifying the role of particular words in a text (i.e., the parts of speech)

Partially replicating their study, Nesi and Meara (1991) draw a similar conclusion, namely, that dictionary consultation does not aid in text comprehension during reading tasks Hulstijn’s (1993) findings lend support to Nesi and Meara’s conclusions Hulstijn (1993) investigated intermediate EFL learners’ dictionary use and look-up behavior and concluded that there is no qualitative difference in reading comprehen-sion performance on vocabulary tests By using multimedia annotations (text, picture + text, and video + text) for individual vocabulary items in second language reading, Chun and Plass (1996) also concluded that foreign language learners who look up more vocabulary items still do not demonstrate better reading comprehension; however, a moderate correlation was found between vocabulary knowledge and reading comprehension

Nevertheless, the waters remain clouded, because there are ample counterarguments extolling the benefits of using dictionaries as a compensatory reading aid strategy For example, Luppescu and Day (1993) point out that, when learners are not able to infer meaning from the context, dictionary consultation helps learners disambiguate word meaning (see also Fraser, 1999) Grabe and Stoller’s (1997) support stems from a five-month case study based on a Portuguese learner who used a bilingual dictionary for extensive reading Their conclusion is that dictionary use is extremely beneficial in learning a second language They argue that consulting a dictionary while reading not only provides the learner with a ‘‘psychologically valuable accuracy anchor’’ (Coady & Huckin, 1997, p 54) but also facilitates text comprehension Rumizen’s (1994) findings also support the notion that dictionary consultation appears to offer an advantage It seems particularly useful for decoding texts and as an effective tool for lexical access when using a bilingual dictionary during global-reading comprehension tests Also, Gonzales (1999) concluded from student interviews that dictionary use, though laborious, was still necessary Dictionary consultation ‘‘provides fast and reliable support for learners who have vocabulary and language limitations’’ (p 265)

Reader and Reading Variables

Before testing the effects of dictionary use as it relates to this study, it needs to be pointed out that the elements that comprise reading comprehension are rather complex (Chun & Plass, 1996) Though vocabulary knowledge plays a crucial role in reading comprehension, researchers have uncovered other factors, which might influence text

comprehension as well Alderson (2000) categorizes these variables into two main types: reader variables and text variables

Reader variables include those factors related to the readers themselves They are a reader’s (1) language proficiency level; (2) knowledge of the topic and the world; (3) purpose and motivation toward the reading task; (4) individual reading strategies; (5) affective state while reading; and (6) relatively stable characteristics, such as age, personality, gender, and social or educational background

Text variables refer to the genre and the type of texts, such as the topic, content, and linguistic-related variables, such as lexical, syntac-tical, semantic difficulties, and text organization Research has also identified other factors that can have a profound effect on reading performance as well, and these include such elements as the reading environment, medium of text presentation (Kerr & Symons, 2006), typographical features (Anderson, 2003; Herman, Anderson, Pearson, & Nagy, 1987), and time available for reading Of these, the element of time has not been satisfactorily examined This is unfortunate, because the time duration of a reading test, as well as the overall length of the text are two relatively weighty matters, especially when one considers the reality that many university students must read extensively in the second language, outside of the classroom setting In light of this, it is of paramount importance to understand the effect duration has on a reading task, as it concerns a learner’s reading ability and text comprehension

When examining previous research related to dictionary use, one can see that learners’ proficiency levels as well as time variables play crucial roles Some studies controlled for these variables; others did not Knight (1994) investigated the effect of dictionary access on a reading comprehension test, taking into consideration the learners’ different verbal abilities The results favor the benefits of dictionary use while reading—the low verbal ability group benefited significantly more from dictionary use while reading than did their high verbal ability counter-parts Learners with dictionary access in each proficiency group spent more time in completing the test than their counterparts with no dictionary access In Luppescu and Day’s (1993) study, the researchers noted that, despite the benefits associated with dictionary use, there was

a decrease in overall reading speed Nesi and Meara (1991) also found that dictionary users took longer, and even those who had the dictionaries but did not use them took longer—as long as those who actually used their dictionaries Those without dictionaries completed the reading task significantly more quickly Nesi and Meara concluded, however, that the dictionary users, though slower, did not gain any benefit toward greater comprehension, despite the additional time (see also Bensoussan et al., 1984)

Without conclusive evidence either for or against dictionary use by second language learners, the issue requires additional analysis from another perspective Specifically, there is a need to know the effects of dictionary use that considers reading comprehension on a variety of texts over an extended period of time Our purpose, then, is to examine reading comprehension in timed increments over an extended time period as a way of gauging whether reading with the help of dictionaries

is more or less effective than reading without using them By acknowl-edging that text comprehension is affected by factors such as learners’ language proficiency level; text difficulty; learners’ relevant textual background knowledge; individual ability in inferencing, predicting, and guessing from the context; the available time for reading; syntactical complexity; and the vocabulary knowledge of the learners, we attempt here to reduce the effect of these variables on the research results Our approach is to analyze comprehension of reading texts via an innovative research design called the duplicated answer sheets (DASH) method The DASH method is distinctive, inasmuch as it allows researchers to test student performance (reading comprehension) using dictionaries in chunks of time, while still controlling for other factors The manner in which the DASH method controls these factors is discussed in the Materials section In addition to analyzing reading comprehension in timed increments, we intend to address the following research questions:

1 Does dictionary use enhance reading comprehension?

2 What is the actual effect of time on reading comprehension?

3 What is the effect of time, if any, on differences in reading proficiencies?

A side benefit of this study is that we consider a learner’s reading habits outside of the classroom, where, generally, most extensive reading

in foreign languages takes place If it can be shown that the use of dictionaries is progressively more beneficial toward reading comprehen-sion over time, dictionary use can be recommended to students as a tool, rather than discouraged

METHODOLOGY

Participants

The participants in this study were 55 university sophomores from the same department in a Taiwanese university (an EFL context) All of the students had had at least 7 years of English language learning experience Using their freshman English term grades as a placement determinant (based mainly on the results of quizzes, midterms, and final examinations),

the participants were divided into two groups with similar proficiency levels, that is, the with-dictionary group (Y group with 25 participants) and the without-dictionary group (N group with 30 participants) These groups were then subdivided into smaller groups, based again on their freshman English term grades (Y1, Y2, Y3, Y4, Y5; and N1, N2, N3, N4, N5) As can be seen from Table 1, participants whose scores placed them in Y1 and N1 were the most proficient students, and students whose scores placed them

in Y5 and N5 were the least proficient Table 1 shows the divisions of the groups and subgroups based on their scores

Materials

To investigate the effect of dictionary use on reading among learners of different proficiency levels, we used a reading comprehension test with a multiple-choice format developed by a local test service center The format

of the test was patterned after a nationwide General English Proficiency test used in measuring reading comprehension, with questions consisting

of one correct item and three distractors Although there are some concerns when using a multiple-choice format in vocabulary tests and reading comprehension (see Nation, 1990; Peirce, 1992), it was used here for its overriding convenience of administration and relative ease of analysis (Wesche & Paribakht, 1996) The 50-item exam tested reading comprehension for the sentence level (connections and meaning within a sentence) and for longer passages (connections and meaning at the discourse level), in the form of vocabulary choice, cloze, and reading passages of general topics (see Appendix) The test scores were used to indicate the effect of dictionary use on the participants’ overall reading comprehension The DASH method was used to score the test, because it allows for incremental scoring over extended periods of time

DASH Method as a Tool of Investigation

The DASH method consists of multiple carbon-copy style answer sheets, which can be peeled off as a way to record student scores in timed

TABLE 1

Means of Term Scores of Y Group (With-Dictionary) and N Group (Without-Dictionary) Dictionary group

(Number of participants)

Y1/N1 (5)/(6)

Y2/N2 (5)/(6)

Y3/N3 (5)/(6)

Y4/N4 (5)/(6)

Y5/N5 (5)/(6) Mean of term scores

Y/N

86.2/85.8 78.1/77.6 71.8/71.3 65.6/66.4 45.0/51.3 Standard deviation

Y/N

5.23/9.1 1.56/2.6 1.48/1.4 2.10/1.4 15.26/14.6

increments In our case, participants’ score sheets consisted of six different answer sheets, so that the scores could be collected every 15 min during the 90-min test (at the 15-, 30-, 45-, 60-, 75-, and 90-min marks) In other words, each answer sheet peeled off and collected represented the test score that a subject gained during the allotted test time Since participants were able to change their answers on previously answered questions, each test score was treated as a score independent

of the other five that were collected.1

Regarding the rationale behind using the DASH method, as previously alluded to, the components affecting reading comprehension are multiple and complex (see Chun & Plass, 1996) In addition to learners’ language proficiency levels, there are still components such as syntactic structures, background knowledge, depth of vocabulary knowl-edge, and available reading time We employed the DASH method to effectively control for these variables In short, the benefits of employing the innovative DASH method as a measurement device are as follows

No Textual Factors Involved; Clean Time Effect Data

Variables such as textual knowledge and ease or difficulty of a text can affect readers’ performance on comprehension tests Investigating the time effect of dictionary use on learners of different proficiency levels using a traditional test method would require the researchers to give the participants six different tests (one requiring 15 min to complete, another requiring 30 min, another requiring 45 min, etc) In addition, even with an extensive effort to modify such tests with the aim of achieving test similarity between the six versions, factors such as text difficulty and topical knowledge would likely have a considerably negative effect on the results It might be argued that, by employing identical tests repetitively, the text difficulty variance could be addressed However, researchers would then be faced with the practice effect caused by readers’ repeated exposure to the same texts and questions, not to mention the fact that this is not remotely similar to how reading is done or tested in more natural settings Additionally, if learners are indeed learning, they logically should score higher on subsequent tests And, regarding topical knowledge, although there are particular readability formulas, such as the Lix formula (Knight, 1994), there is always the problem posed by using different test texts, because different learners inevitably have different strengths as related to content

1 Observation of students revealed that occasionally students did change their answers Although the frequency of changed answers was not tabulated, it would make for an interesting study in its own right to see how often second language learners do change answers on reading tests.

(background) knowledge The DASH method allowed us to analyze test scores using the same texts in relation to the time effect By obtaining the mean scores of the participants in each collection, the factor of individual textual knowledge or background knowledge could thus be averaged out, leaving the scores and making for a clean time effect

Time Efficiency and Practicality

In addition to the benefit of the clean time effect associated with using the DASH method, the traditional method of giving six different tests at 15, 30, 45, 60, 75, and 90 min, respectively, would consist of testing participants for a total of 315 min to complete such a study Irrespective of any variability in text difficulties, the diminishing returns

on reliability and validity because of participants’ affective conditions (such as sheer exhaustion and boredom) and environmental stability in the over-5-hr, nonstop testathon would be significant By incorporating the DASH method in this study, only one reading assessment test was needed, because the other variables could be easily controlled (leaving the variable of time and learners’ proficiency levels to be investigated with dictionary use while reading) Besides the time-saving effect of the DASH method, it was also economical, because it uses duplicate carbon copies

Procedure

The participants were provided with the same testing material during the 90-min test Prior to the test administration, the participants were informed of the purpose of the study and reassured that their test scores would not be used as a determining factor in their English class grade This was done as a means to lower students’ reading anxiety while taking the test (Saito, Garza, & Horwitz, 1999), with the hope that they would, nevertheless, try their best to answer the questions

The same 90-min test was given to both groups of students using the DASH method as a scoring tool (Figure 1) The test sections were collected as intended at the 15-, 30-, 45-, 60-, 75-, and 90-min marks The participants with dictionary access were allowed to choose the kind of dictionary they were familiar using and were asked on the answer sheet to indicate the type of dictionary they had used during the test: monolingual (English only), bilingual (English with equivalent Chinese translations), or a dual-bilingual dictionary that contains English entries with English definitions and their literal Chinese translations The participants were also told and encouraged to look up the unfamiliar vocabulary they encountered, to fulfill the aims of this study, and they

were required to underline the words they looked up Through the researchers’ on-site observation, informal posttest interviews with the participants in the dictionary group, and the underlined words on the question sheets, it was clear that the with-dictionary groups had consulted dictionaries extensively throughout the test as a means to identify vocabulary items with which they were unfamiliar

RESULTS

Table 2 shows the average reading performance of the two groups at each time allotment The mean test scores of group Y (the with-dictionary group) and group N (the without-with-dictionary group) in relation to the amount of time spent are recorded in Table 2 and plotted

in Figure 2, with the x-axis representing the amount of time spent and the y-axis representing the effect of dictionary use as determined by the test scores The table indicates that without-dictionary users scored

TABLE 2

Mean Scores of With-Dictionary Group Y and Without-Dictionary Group N in Relation to Time Spent

Time spent

(min)

15 min

Mean (SD)

30 min Mean (SD)

45 min Mean (SD)

60 min Mean (SD)

75 min Mean (SD)

90 min Mean (SD) Mean scores

of Y

18.2 (6.3) 33.7 (11.4) 43.2 (12.6) 49.3 (13.6) 54.0 (14.6) 57.0 (14.8) Mean scores

of N

21.3 (11.3) 36.1 (15.0) 42.1 (15.5) 46.2 (13.4) 47.8 (14.1) 48.1 (13.8)

Note SD 5 standard deviation.

FIGURE 1 The DASH method

higher during the early stages; however, at about 45 min into the test, scores leveled; from that point forward, dictionary users performed increasingly better As seen in Figure 2, a sharp rise in test scores is seen

in the first 60 min of the test However, a much slower rate of gain was observed after the fifth collection, at the 75-min mark Toward the end

of the 90-min test, a much slower rate in mean score increases was evident, especially in the without-dictionary group, where the difference between the mean scores recorded at 75 min (47.8) and 90 min (48.1) was, in practical terms, not significant During the first 15 min, the average score of the dictionary-use group proved lower than that of the without-dictionary group Nevertheless, as Figure 2 illustrates, dictionary users recorded progressively higher scores on the reading test over time The crossover point circled in Figure 2 indicates the approximate time at which dictionary use started to become more effective toward comprehending the text

Table 3 shows the mean scores of the five subgroups of the with-dictionary (Y1 to Y5) and the without-with-dictionary groups (N1 to N5); these are plotted in Figure 3 and Figure 4, respectively Except for the lowest proficiency groups Y5 and N5, the without-dictionary groups N1

to N4 seemed to outperform their dictionary counterparts Y1 to Y4 during the initial stages During the latter stages, however, dictionary use proved to be more effective on reading performance, as participants began tallying higher reading comprehension scores compared with the without-dictionary groups Moreover, the lowest proficiency group with dictionary use (Y5) consistently performed better than the without-dictionary counterparts (N5) throughout the 90-minute test (Figure 5)

FIGURE 2 Average reading performance