Database Selection Guidelines for Meta-Analysis in | Applied Linguistics |

3 Ideally, all applied linguistics journals that report research must be included in order to conduct a more precise examination of the databases that were used in previous meta-analyses

Trachtenberg, S (1979) Joke telling as a tool in ESL English Teaching Forum, 21(4), 8–12.

Vega, G (1989) Humor competence: The fifth component Unpublished master’s thesis, Purdue University, West Lafayette, IN.

Database Selection Guidelines for Meta-Analysis in

Applied Linguistics

YO IN’NAMI

Toyohashi University of Technology

Aichi, Japan

RIE KOIZUMI

Tokiwa University

Ibaraki, Japan

doi: 10.5054/tq.2010.215253

&The massive accumulation of knowledge and studies on certain topics within the academic domain heightens the need for more research syntheses, and this is also true for the field of applied linguistics A

researchers attempt to quantitatively summarize a set of empirical data across studies in order to identify consistencies and explain variabilities through this method (e.g., Cooper & Hedges, 1994; Norris & Ortega,

2006, 2007) Although the concept of meta-analysis is a century old, Glass (1976) and Norris and Ortega (2000) initiated a widespread modern interest in this method in education and applied linguistics research

According to Cooper (1982), the meta-analysis process can be classified into five stages: (a) the problem formulation stage, in which research questions that are to be addressed in the meta-analysis are specified and formulated; (b) the data collection stage, in which the literature on relevant studies is researched; (c) the data evaluation stage,

in which information that will help answer the research questions is coded; (d) the analysis and interpretation stage, in which the coded information is statistically integrated; and (e) the public presentation stage, in which the findings from the synthesis are disseminated among the audience It must be noted that individual studies provide data included in a meta-analysis, which is in contrast to the traditional

1 There are also nonstatistical approaches to synthesizing research (see discussion in Light & Pillemer, 1984; Norris & Ortega, 2006).

approach to conducting a study where individual participants provide data Thus, as emphasized by White (1994), discovering previous studies for synthesis is a major concern for meta-analysts because all the synthesized findings depend on these studies

How many studies are necessary for meta-analysis? Although as few as two studies can be synthesized, most researchers argue that, in order to represent the domain of research as thoroughly as possible, all existing published and unpublished studies must be comprehensively covered

comprehensive approach is particularly useful for examining the file-drawer problem, where studies are less likely to be published when they report statistically nonsignificant results, and for examining the stability

of findings across published and unpublished studies through mod-erator variable analyses (e.g., Lipsey & Wilson, 2001)

Because a comprehensive inclusion of previous studies is one of the key aspects in meta-analytic research, researchers track down as many relevant studies as possible through such means as using databases, scanning books and journals manually, and soliciting information from researchers who are familiar with the topic Among these search strategies, the current article focuses on using databases We believe database usage is becoming increasingly more important because databases are developing steadily as technology provides enhanced opportunities to create them, and there are certainly many more available today as compared with, say, ten years ago Nevertheless, we re-emphasize the point that a meta-analysis must encompass the entire domain of available research, and that databases provide a useful initial step in the analytic process (but they do not assure comprehensiveness

in encompassing the entire domain of work) For the meta-analysis process in applied linguistics, the Education Resources Information Center (ERIC) has been widely used (e.g., Russell & Spada, 2006; Saito, 2008), because it provides a rich source of published and unpublished materials However, in the fields of education and psychology, Glass, McGaw, and Smith (1981) showed a slight overlap between ERIC and other databases (i.e., Council for Exceptional Children Abstracts, Dissertation Abstracts, and Psychological Abstracts) This finding suggests that ERIC is not a sufficient resource and that a comprehensive literature search requires the use of multiple databases and must not be based on a single database

Accordingly, what databases should be used when conducting a meta-analysis in applied linguistics, and in what manner? In order to address these issues, we examine three research questions: (a) Which databases

2 One can also argue that when the number of collected studies is small, such a domain is not perhaps ready for meta-analysis (see Ellis, 2006).

are used in meta-analyses in applied linguistics? (b) What combination

of databases is used in applied linguistics? (c) Which databases provide a comprehensive coverage of journals in applied linguistics? Reed and Baxter (1994) noted the importance of caution in selecting databases that are relevant to one’s needs However, the existence of numerous types of databases makes it difficult to appropriately select which databases should be used and in what combination Because, to our knowledge, there is no summary of database sources tailored for applied linguistics, we compare databases in this article in order to present practical guidelines for selection of databases for meta-analytic purposes

We do so by addressing each of the three questions posed earlier

Research Question 1: Which Databases Are Used in Meta-Analyses in Applied Linguistics?

One method for identifying appropriate, useful, and relevant databases is to examine what databases were used in previous meta-analyses For this purpose, previous meta-analyses in applied linguistics were searched manually in March 2009 Norris and Ortega’s (2006) well-known book and 24 representative journals were reviewed Norris and Ortega’s edited volume was reviewed because it is a collection of meta-analytic studies in applied linguistics The 24 journals were selected because we believe that they are often read by applied linguists and are

journals that were reviewed include the following: Annual Review of Applied Linguistics (ARAL), Applied Language Learning (ALL), Applied Linguistics (AL), Applied Psycholinguistics (AP), Assessing Writing (AW), Canadian Modern Language Review (CMLR), ELT Journal (ELTJ), Foreign Language Annals (FLA), International Journal of Applied Linguistics (IJAL), International Review of Applied Linguistics in Language Teaching (IRAL), JALT Journal (JALTJ), Language Assessment Quarterly (LAQ), Language Learning (LL), Language Learning & Technology (LLT), Language Teaching (LTea), Language Teaching Research (LTR), Language Testing (LTes), Modern Language Journal (MLJ), Reading Research Quarterly (RRQ), RELC Journal (RELCJ), Second Language Research (SLR), Studies in Second Language Acquisition (SSLA), System, and TESOL Quarterly (TESOLQ) All issues of each journal were reviewed Fifteen meta-analytic studies were identified (marked with asterisks in the references), of which 12 specified the databases that were used When necessary, every effort was made to contact the authors to request further details about the processes they employed

3 Ideally, all applied linguistics journals that report research must be included in order to conduct a more precise examination of the databases that were used in previous meta-analyses.

As shown in Table 1, 2006 had the highest frequency (40%) of meta-analytic studies, and these studies appeared as book chapters in the previously cited Norris & Ortega (2006) collection For the remaining years, a small but consistent number of meta-analytic studies (one or two) was conducted, thereby suggesting continuing interest in meta-analysis among applied linguists

Table 2 summarizes the databases used in the 12 studies ERIC was by far the most widely used database, and it was used in all the studies (N 5 12) This frequency was followed by Linguistics and Language Behavior Abstracts (LLBA; n 5 7), PsycINFO (n 5 7), and ProQuest Dissertations and Theses (ProQuest D&T; n 5 5)

The characteristics of these four most frequently used databases and those of the other databases are briefly described on the basis of their homepage information First, ERIC provides access to more than 1.2 million bibliographic records of journal articles and other education-related materials free of charge ERIC’s main strength lies in its indexing

of a wide range of education-related materials (e.g., conference papers and proceedings, dissertations, and theses) Full texts are often available

in the form of ERIC microfiches at local libraries, and full texts for most

TABLE 1

Year of Publishing and Frequency of Meta-Analyses

Year 97 98 99 00 01 02 03 04 05 06 07 08 09 Frequency of

published articles

TABLE 2

Databases Used in Previous Meta-Analyses

Comprehensive Dissertation Abstracts 1

ERIC (Educational Resources Information Center) 12

LLBA (Linguistics and Language Behavior Abstracts) 7 MLA (Modern Language Association) International Bibliography 1 OCLC (Online Computer Library Center) Proceedings First 1 ProQuest Dissertations and Theses 5

PsycINFO (including PsycLit, which merged with PsycINFO in 2000) 7

SSCIa(Social Sciences Citation Index) 1 Note.aAvailable only through the Web of Science platform The total frequencies exceed the total number of studies (12) because the use of multiple databases in a single study was coded separately.

materials from the 1990s onward are available in PDF format Identifying and permitting easy access to these fugitive materials is a great advantage

of using ERIC for meta-analysis Second, LLBA, as indicated by its name,

is a database that encompasses documents in linguistics and related disciplines LLBA provides more than 370,000 abstracts of journal articles, books, book chapters, dissertations, and conference papers, drawn from more than 1,500 publications Documents from 1973 or later are included; however, full texts are not available Third, PsycINFO provides abstracts of scholarly publications in behavioral and social sciences from the 1800s to the present The database encompasses more than 2,150 journals (comprising 78% of the database), English-language books (11%), and dissertations (11%) Full texts are not available in the database Fourth, ProQuest D&T includes more than 2.3 million dissertations and theses, mainly those published in North America One strength of this database is that it provides comprehensive coverage

of documents dating from 1637 onward and relates to a wide range of academic disciplines Further, the full texts of the dissertations and theses dating from 1861 onward are available This extremely long-term coverage of documents across numerous fields and the availability of full texts are truly unique features of ProQuest D&T Dissertations and theses can also be searched for in LLBA; however, LLBA appears limited, for example, in terms of fields included (linguistics and related disciplines only), coverage period (from 1973 to the present day), and unavailability

of full texts Thus, a meta-analyst who is planning to collect unpublished studies, particularly dissertations and theses, is strongly advised to access ProQuest D&T

Brief descriptions of the characteristics of the remaining nine databases are provided as follows First, Academic Search Premier provides citations and abstracts for more than 8,300 journals encompassing numerous subject areas Full-text access to more than 4,500 of those titles is available Second, the Comprehensive Dissertation Abstracts database

is a collection of dissertations and theses that are mainly published in North America, including more than 1.6 million records dating back to

1861 To the best of our knowledge, this database is no longer available, and users are referred to ProQuest D&T for dissertation and thesis searches Third, Current Contents or its electronic version, Current Contents Search, provides access to tables of contents, abstracts, bibliographic information, and abstracts from more than 7,600 scholarly journals and 2,000 books Current Contents Search also includes bibliographic informa-tion from certain electronic journals before they are published Fourth, FirstSearch provides access to more than 90 million records, with full-text access to more than 10 million articles Unlike other stand-alone databases, FirstSearch permits searching of documents across databases, such as ArticleFirst for all academic areas, Arts & Humanities Search for

humanities, and WilsonSelectPlus for business and management Fifth, MLA International Bibliography indexes more than 66,000 books and articles on literature and linguistics, and it has been compiled by the Modern Language Association of America since 1926 The database archives more than 2 million records and provides full-text links to

ProceedingsFirst contains tables of contents of papers presented at conferences worldwide and received by the British Library Document Supply Centre since 1993 Seventh, PsycARTICLES is a database of full-text articles that are mainly from journals published by the American Psychological Association It encompasses 66 psychology journals published since 1894, totaling more than 140,000 articles Eighth, ScienceDirect contains abstracts of more than 2,500 journals or 8.5 million articles mainly on science, technology, and medicine However, full-text availability varies depending on subscription status Ninth, although the Social Science Citation Index (SSCI) is infrequently used, we believe it is indispensable, becaue it reveals the history of each article by displaying who has cited it and the number of times it has been referred to Thus, articles that cite relevant studies can be easily traced In addition, SSCI enables searches across disciplines for all articles that have common cited references These are invaluable features of SSCI, because meta-analytic findings may be biased if they only include articles that report clear, impressive results that are more likely to be published and cited

SSCI to locate and include both popular and less popular articles while conducting a meta-analysis

Research Question 2: What Combination of Databases Is Used in Applied Linguistics?

Thus far, we have examined which databases are widely used for meta-analyses in applied linguistics In order to reveal combination patterns of the databases used in previous studies, the 15 studies that we examined for Research Question 1 were rearranged according to the number of databases used for each study (see Table 3) It was found that the number of databases used had two distribution peaks around 1 and 5, thereby suggesting that previous meta-analytic studies were often conducted with either one or five databases Although using only one database may not allow a comprehensive search of the existing studies, the researchers in those studies might have chosen to trust one database only rather than to use multiple databases Of particular interest was the

4 It is difficult to identify all research reports and publications in a meta-analysis However, statistical procedures can be employed to tackle the file-drawer problem See Hsu (2002), Orwin (1983), and Rosenthal (1991).

combination of databases The presence of ERIC, LLBA, ProQuest D&T, and PsycINFO was conspicuous: ERIC and LLBA were used together in seven studies; ERIC, LLBA, and PsycINFO in six; ERIC, LLBA, and ProQuest D&T in four; and all four databases in three Thus, based on the frequency of use (Research Question 1) and combinations (Research Question 2) of databases used for meta-analytic studies, we recommend combined use of ERIC, LLBA, ProQuest D&T, and/or PsycINFO

Research Question 3: Which Databases Provide a Comprehensive Coverage of Journals in Applied Linguistics?

Although the answer to Research Question 2 revealed combination patterns of the databases used in meta-analyses in applied linguistics, the obvious question at this point is related to uniqueness or redundancy features between these databases In order to answer this research question, we conducted another analysis by focusing on two character-istics of databases—journal coverage rates and periods of coverage across databases A corresponding list of databases and journals was con-structed Because such databases abound and our focus is on applied linguistics, the homepages of the 24 representative journals were examined in March 2009 in order to determine which databases included the 24 journals The identified databases were also accessed in order to cross-check whether they, in actuality, included the journals that the journal homepages claimed they included Although we found

98 databases, only 10 of them are presented in Table 4: nine databases that provided the most comprehensive coverage of applied linguistics journals, and one that was frequently employed in applied linguistics meta-analyses (i.e., PsycINFO; see Table 2) Two databases that provide journal information (e.g., advertising rates, submissions, peer-review

TABLE 3

Combination of Databases Used in Meta-Analyses

Number of databases used Frequency

2 1 (Comprehensive Dissertation Abstracts, ERIC)

3 2 (ERIC, LLBA, PsycINFO; ERIC, LLBA, ProQuest D&T)

4 2 (ERIC, LLBA, ProQuest D&T, PsycINFO; Academic Search

Premier, ERIC, LLBA, PsycINFO)

5 3 (Current Contents, ERIC, MLA International Bibliography,

ProQuest D&T, PsycINFO; ERIC, LLBA, OCLC Proceedings First, ProQuest D&T, PsycINFO; ERIC, LLBA, ProQuest D&T, PsycARTICLES, PsycINFO)

6 1 (ERIC, FirstSearch, LLBA, PsycINFO, ScienceDirect, SSCI)

89–07 [100

80–82; 96–

94–00; 02–

96–01; 05–

ed Jour

E Rese

a ML

b Fo

c Former

d The

e The

f 100%,

g over

h less

i The

status) rather than journal coverage rates and periods (i.e., MLA Directory

of Periodicals; European Reference Index for the Humanities) were excluded ProQuest D&T was often used in applied linguistics (see Table 2) but not included because it only covers dissertations and theses, not journals Table 4 presents journal coverage, and a blank indicates that the journal was not included

The total number of journals covered by each database ranged from 9

to 24, with a mean of 16.60 and standard deviation of 5.23 (see Table 4) This finding indicates that the journals listed here are, as a whole, well-covered across the databases However, only LLBA well-covered all 24 journals ERIC and MLA International Bibliography (MLA) covered 22 journals, Linguistics Abstracts covered 20, and Scopus covered 18 These five data-bases conspicuously covered a wider range of journals than other datadata-bases Because four of the databases (i.e., ERIC, MLA, Linguistics Abstracts, and Scopus) did not contain certain journals, these databases must be used complementarily in terms of journal coverage For example, ERIC and MLA do not include Language Assessment Quarterly and Language Teaching In contrast, Linguistics Abstracts includes both journals but lacks Applied Language Learning, Foreign Language Annals, JALT Journal, and TESOL Quarterly, all of which are indexed in ERIC and MLA Thus, when

we select from among the three databases (ERIC, MLA, and Linguistics Abstracts), a combination of either ERIC and Linguistics Abstracts or MLA and Linguistics Abstracts must be used, and extra attention must be paid

in conducting manual searches of journals that are missing from these databases

Along with journal coverage, also noteworthy are the journal coverage periods It was found that ERIC, Linguistics Abstracts, LLBA, MLA, and Scopus were more comprehensive as compared with other databases For example, the above-mentioned five databases cover Applied Linguistics since 1987 (ERIC), 1986 (Linguistic Abstracts, with occasional breaks),

1980 (LLBA), 1980 (MLA), and 1996 (Scopus) In contrast, ProQuest Education Journals, for example, covers Applied Linguistics only since 1999 and PsycINFO only since 2001

However, there are two caveats First, even the five databases that excel in journal coverage rates and periods (ERIC, Linguistics Abstracts, LLBA, MLA, and Scopus) have numerous gaps in the years that are covered, and careful selection is necessary in order to ensure that all periods that researchers intend to include are indeed included in the search An example of this problem is ERIC, which does not cover certain issues of a journal, although it specifies any breaks in journal coverage on its homepage For example, ERIC has been covering Language Learning since 1970 but did not cover it in 1973 and between

1980 and 1982