Quantitative methods for second language research carsten roever, aek phakiti, routledge, 2018 scan

QUANTITATIVE METHODS FOR SECOND LANGUAGE RESEARCH Quantitative Methods for Second Language Research introduces approaches to and techniques for quantitative data analysis in second langu

QUANTITATIVE METHODS FOR

SECOND LANGUAGE RESEARCH

Quantitative Methods for Second Language Research introduces approaches to and techniques for quantitative data analysis in second language research, with a primary focus on second language learning and assessment research It takes a conceptual, problem- solving approach by emphasizing the understanding of sta-tistical theory and its application to research problems while paying less attention

to the mathematical side of statistical analysis The text discusses a range of mon statistical analysis techniques, presented and illustrated through applications

com-of the IBM Statistical Package for Social Sciences (SPSS) program These include tools for descriptive analysis (e.g., means and percentages) as well as inferential

analysis (e.g., correlational analysis, t- tests, and analysis of variance [ANOVA])

The text provides conceptual explanations of quantitative methods through the use of examples, cases, and published studies in the fi eld In addition, a companion website to the book hosts slides, review exercises, and answer keys for each chapter

as well as SPSS fi les Practical and lucid, this book is the ideal resource for data analysis for graduate students and researchers in applied linguistics

Carsten Roever is Associate Professor in Applied Linguistics in the School of

Languages and Linguistics at the University of Melbourne, Australia

Aek Phakiti is Associate Professor in TESOL in the Sydney School of Education

and Social Work at the University of Sydney, Australia

QUANTITATIVE

METHODS FOR

SECOND LANGUAGE RESEARCH

A Problem- Solving Approach

Carsten Roever and Aek Phakiti

by Routledge

711 Third Avenue, New York, NY 10017

and by Routledge

2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN

Routledge is an imprint of the Taylor & Francis Group, an informa business

© 2018 Taylor & Francis

The right of Carsten Roever and Aek Phakiti to be identifi ed as authors

of this work has been asserted by them in accordance with sections 77 and

78 of the Copyright, Designs and Patents Act 1988.

All rights reserved No part of this book may be reprinted or reproduced

or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording,

or in any information storage or retrieval system, without permission in writing from the publishers.

Trademark notice: Product or corporate names may be trademarks or

registered trademarks, and are used only for identifi cation and explanation without intent to infringe.

Every effort has been made to contact copyright- holders Please advise the publisher of any errors or omissions, and these will be corrected in subsequent editions.

Library of Congress Cataloging- in- Publication Data

A catalog record for this book has been requested

List of Illustrations vii Foreword xv Preface xvii Acknowledgments xxii

CONTENTS

12 Two- Way Mixed- Design ANOVA 166

2.9 Accessing Case Summaries in the SPSS menus 20

2.11 SPSS output based on the variables set in the Summarize

2.14 Illustrated example of an Excel data fi le to be imported into SPSS 24 2.15 SPSS dialog when opening an Excel data source 24 2.16 The personal factor questionnaire on demographic information 25 2.17 SPSS spreadsheet that shows the demographic data of

3.2 A pie chart based on a 10- point score range 34 3.3 A bar chart based on a 10- point score range 35 3.4 An example of questionnaire items using a Likert- type scale 40 3.5 The positively skewed distribution of length of residence 41 3.6 The negatively skewed distribution of speech act scores 42 3.7 The low skewed distribution of implicature scores 42

4.4 Defi ning selfrate (self- rating of profi ciency) in the Value Labels dialog 47

4.6 SPSS menu for computing descriptive statistics 49

4.10 A histogram of the self- rating of profi ciency variable with a normal curve 53

5.2 A scatterplot displaying the values of two variables with a

5.3 A scatterplot displaying the values of two variables with a

5.4 A scatterplot displaying the values of two variables with a perfect

5.5 A scatterplot displaying the values of two variables with a low

5.6 SPSS output displaying the Pearson product moment correlation

5.10 A scatterplot displaying the values of the listening and

5.12 A scatterplot displaying the values of the listening and

grammar scores with a line of best fi t added 77

Illustrations ix

7.1 Accessing the SPSS menu to perform the

independent- samples t- test 98

7.2 SPSS dialog for the independent- samples t- test 99

7.4 Accessing the SPSS menu to perform the

paired- samples t- test 102

8.1 SPSS menu to perform the Mann- Whitney U test 109 8.2 SPSS dialog to perform the Mann- Whitney U test 110 8.3 SPSS menu to perform the Wilcoxon Signed- rank test 113 8.4 SPSS dialog to perform the Wilcoxon Signed- rank test 113

9.5 SPSS menu to launch the Kruskal- Wallis test 129

9.7 Variable entry for the Kruskal- Wallis test 131 9.8 Analysis settings for the Kruskal- Wallis test 131

9.10 Model Viewer window for the Kruskal- Wallis test 132

9.12 Pairwise comparisons in the Kruskal- Wallis test 13310.1 Accessing the SPSS menu to launch the Compute

10.3 Checking ANCOVA assumption of independence of

10.4 Accessing the SPSS menu to select Cases for analysis 143

10.9 Univariate dialog for choosing a model to examine an

10.10 Univariate: Model dialog for defi ning the interaction term to

10.11 Changing the analysis setup back to the original setup 149

11.1 A pretest, posttest, and delayed posttest design 15411.2 Accessing the SPSS menu to launch a repeated- measures ANOVA 159

11.3 Repeated Measures Defi ne Factors dialog 159

12.1 Diagram of a pretest- posttest control- group design 16712.2 Changes across time points among the fi ve groups 169

12.5 Repeated Measures: Profi le Plots dialog with colres∗section shown 17212.6 Repeated Measures: Post Hoc Multiple Comparisons for Observed

13.1 Accessing the SPSS menu to launch the two- dimensional

13.5 VassarStats website’s chi- square calculator

(http://vassarstats.net/newcs.html) 19613.6 Contingency table for two rows and two columns 19713.7 Contingency table for two rows and two columns with data

14.1 A scatterplot of the relationship between chocolate consumption

14.2 Accessing the SPSS menu to launch multiple regression 207

14.6 Linear Regression dialog for a hierarchical regression (Block 1 of 1) 21314.7 Linear Regression dialog for a hierarchical regression (Block 2 of 2) 21414.8 Linear Regression dialog for a hierarchical regression (Block 3 of 3) 21415.1 Accessing the SPSS menu to launch Cronbach’s alpha analysis 22415.2 Reliability Analysis dialog for Cronbach’s alpha analysis 224

15.4 A selection from Ch15analyticrater.sav (Data View) 228

15.6 Accessing the SPSS menu to launch Reliability Analysis 23215.7 Reliability Analysis dialog for the split- half analysis 233

15.9 Accessing the SPSS menu to launch Crosstabs for kappa analysis 236

15.11 Crosstabs: Statistics dialog for choosing kappa 237

Illustrations xi

15.12 Reliability Analysis dialog for raters’ totals as selected variables 24015.13 Reliability Analysis: Statistics dialog for intraclass correlation analysis 241

Tables

1.2 An example of learners’ scores converted into percentages 4

1.4 How learners are scored on the basis of performance descriptors 6 1.5 How learners are scored on a different set of performance descriptors 6

1.8 The three placement levels taught at three different locations 9 1.9 The students’ test scores, placement levels, and campuses 9 1.10 The students’ placement levels and campuses 10

3.1 IDs, gender, self- rated profi ciency, and test score of the fi rst 50

participants 29

3.3 Frequency counts based on test takers’ self- assessment of

3.4 Frequency counts based on test takers’ test scores 32 3.5 Frequency counts based on test takers’ test score ranges 32

3.7 Imaginary test taker sample with an outlier 36

4.3 SPSS frequency table for the selfrate variable

4.4 Taxonomy of the questionnaire and Cronbach’s alpha (N = 51) 59 4.5 Example of item- level descriptive statistics (N = 51) 59 5.1 Descriptive statistics of the listening, grammar, vocabulary, and

5.2 Pearson product moment correlation between the listening

5.3 Spearman correlation between the listening scores and

7.1 Mean and standard deviation of error counts for generation

7.2 Mean and standard deviations of ratios of error- free clauses in the cartoon description task for both modalities 95 7.3 Means and standard deviations of the two groups 99

7.6 Means and standard deviations of the two means 104 7.7 Correlation coeffi cient between the two means 104

8.3 Mean ranks in the Mann- Whitney U test (N = 46) 110

10.2 Post hoc tests for independence of covariate and independent

10.3 Post hoc tests for the independence of covariate and independent variable 14210.4 Output of homogeneity of regression slopes check

10.5 Descriptive statistics of the routines scores between the two

11.1 Six different tests with 10 vocabulary items 155

11.6 Results from tests of within- subjects effects 163

Illustrations xiii

12.1 Descriptive statistics of the percentage scores for correct use

for the fi ve treatment conditions by three tasks 168

12.6 Results from tests of within- subjects effects 176

12.10 Pairwise comparisons on collapsed residence 177

13.1 Frequency of phrasal verb use in fi ve registers 18313.2 Chi- square observed and expected counts and residuals 18413.3 Frequency counts of language- related episodes (LREs) by

13.4 Marginal totals, expected frequencies, and residuals for recall by

13.6 Marginal totals, expected frequencies, and residuals for

13.7 SPSS summary of the two- dimensional chi- square analysis 19413.8 Cross- tabulation output based on gender and collapsed residence 19413.9 Outputs of the two- dimensional chi- square test 19513.10 Symmetric measures for the two- dimensional chi- square test 195

14.3 Correlations among the outcome and predictor variables 209

14.7 Model coeffi cients output: Unstandardized and standardized Beta

14.13 Excluded variables 21715.1 A simple (simulated) data matrix for a course feedback

15.2 The reliability for the 12- item implicature section of the TEP 22215.3 Item- total statistics of the 12- item implicature section of the TEP 22315.4 The case processing summary for items ‘imp1sc’ to ‘imp12sc’ 226

15.11 Cross- tabulation of pass- fail ratings for 25 ESL learners 23415.12 Cross- tabulation of pass- fail ratings by raters 1 and 2 23815.13 Case processing summary for raters 1 and 2 238

15.15 Simulated data set for two raters (rater 1 and rater 2) 239

There is a certain degree of confi dence or credibility that often accompanies statistical evidence “The numbers don’t lie”, we often hear in casual conversa-tion As consumers of information, whether in the news or in published second language (L2) research, we tend to associate statistical evidence with objectivity and, consequently, truth The road that leads to statistical evidence, however, is often long, winding, and full of decisions (even detours!) that the researcher has taken In the case of L2 research, examples of such choices might include deciding (a) whether to collect speech samples using a more open- ended versus a controlled task, (b) whether certain items in a questionnaire—or individuals in a sample—should be removed from analysis based on aberrant observations, and (c) how to score learner production that is only partially correct Each of these choices may infl uence a study’s outcomes in one direction or another, and it is critical that we recognize the centrality of researcher judgment in all that we read and produce As Huff (1954) stated in his now- classic introduction to pitfalls that both researchers

and consumers succumb to, How to Lie With Statistics, “Statistics is as much an art

as it is a science” (p 120)

A second point I offer as you enter into the wonders of quantitative research

is that nearly all of the objects we measure and quantify are actually qualitative

in nature It may seem odd to point this out in the forward of a text like this, but it is true! And although quantitative techniques are valuable in helping us to organize data and to conduct the many systematic and insight- producing analyses described throughout this book, they almost necessarily involve abstractions from our initial interests Imagine, for example, a study of the effects of two instruc-tional treatments on learners’ ability to speak accurately and fl uently En route to addressing that issue we would likely transcribe participants’ speech samples and then code or score them for a given set of features Next, we would summarize

FOREWORD

those scores across the sample, the results of which would be subject to one or more statistical tests for subsequent interpretation In each of these procedures, we have made abstractions, tiny steps away from learner knowledge.

I realize these comments might make me appear skeptical of quantitative research

Of course I am! Likewise, we should all approach the task of conducting, ing, and understanding empirical research with a critical eye And thankfully, that

report-is precreport-isely what threport-is very timely and well- crafted book will enable you to do, thereby advancing our collective ability both to conduct and evaluate research The text, in my view, manages to balance on the one hand a conceptual grounding that enlightens without overwhelming and, on the other, the need for a hands-

on tutorial—in other words, precisely the knowledge and skills needed to make and justify your own decisions throughout the process of producing rigorous and meaningful studies I look forward to reading them!

Luke Plonsky Georgetown University

In the fi eld of L2 research, the quantitative approach is one of the predominant methodologies (see e.g., Norris, Ross & Schoonen, 2015; Plonsky, 2013, 2014; Plonsky & Gass, 2011; Purpura, 2011) Quantitative research uses numbers, quan-tifi cation, and statistics to answer research questions It involves the measurement and quantifi cation of language and language- related features of interest, such as language profi ciency, language skills, aptitudes, and motivation The data collected are then analyzed using statistical tools, the results of which are used to produce research fi ndings In practice, however, the use of statistical tools and the way that the results of quantitative research are reported leaves much to be desired.

In 2013, Plonsky conducted a systematic review of 606 second language sition (SLA) studies in regard to study design, analysis, and reporting practices Several weaknesses in those practices were found, including a lack of basic statisti-cal information, such as mean, standard deviations, and probability values Plonsky and Gass (2011), and Plonsky (2013, 2014) call for a reform of the data analysis and report practices used in L2 research According to Plonsky and Gass (2011), these shortcomings could be a refl ection of inadequate methodological and statis-tical concept training, as well as insuffi cient coverage in research methods courses

acqui-in graduate programs of how researchers should report statistical fi ndacqui-ings

The dearth of adequate training in quantitative research has potentially ous repercussions for the fi eld Certain areas in L2 research cannot be adequately addressed if there is a lack of appropriate training in statistical methods and if suf-

seri-fi cient resources are inaccessible to new researchers and experienced researchers new to quantitative methods Quantitative methods, particularly inferential statis-tics, can be technical and diffi cult to learn because they require an understanding

of not only the logic underpinning the statistical approaches taken, but also the technical procedures that need to be followed to produce statistical outcomes In

PREFACE

addition, researchers need to be able to interpret outcomes of statistical analyses and draw conclusions from them to answer research questions.

Since researchers in applied linguistics frequently come from an arts, ties, education, and/or social sciences background, they often have little familiarity with mathematical and statistical concepts and procedures, and perceive statistics

humani-as a foreign language, feeling apprehensive at the prospect of grappling with titative concepts and developing statistical skills This may lead them to choose a qualitative research approach, despite a quantitative one being more suitable to answer a particular research question

quan-Not only can a lack of familiarity with quantitative procedures close off major avenues of research to students, but it can also prevent new researchers from under-standing and critically evaluating existing studies that use quantitative methods: if readers do not understand the use of statistics in a paper, they are forced to take the author’s interpretation of statistical outcomes on faith, rather than being able

to critically evaluate it In the current market, there are a number of books that deal with quantitative methods (e.g., Bachman, 2004; Bachman & Kunnan, 2005; Larson- Hall, 2010, 2016), but these can be highly technical, mathematical, and lengthy in their statistical treatments, as such books are often written for a particular audience (e.g., advanced doctoral students, or experienced researchers) By contrast, the current book assumes no prior experience in quantitative research and is writ-ten for students and researchers new to quantitative methods

The Aims and Scope of This Book

This book aims to introduce approaches to and techniques for quantitative data analysis in L2 research, with a primary focus on L2 learning and assessment research It takes a conceptual, problem- solving approach, emphasizing the under-standing of statistical theory and its application to research problems and pays less attention to the mathematical side of statistical analysis

This book is, therefore, intended as a practical academic resource and a starting point for new researchers in their quest to learn about data analysis It provides con-ceptual explanations of quantitative methods through the use of examples, cases, and published studies in the fi eld Statistical analysis is presented and illustrated through applications of the IBM Statistical Package for Social Sciences (SPSS) program.Formulae that can easily be computed manually will be presented in this book More involved statistical formulae associated with complex statistical procedures being introduced will not be presented for several reasons First, this book is intended to nurture a conceptual understanding of statistical tests at an intro-ductory level Second, applied linguistics researchers rarely calculate inferential statistics such as those presented in this book manually because there are numerous statistical programs and online tools that are able to perform the required compu-tations Finally, there are many books on statistics that present statistical formulae that the reader can consult if desired

Preface xix

In this book, a range of common statistical analysis techniques that can be employed in L2 research are presented and discussed These include tools for descriptive analysis, such as means and percentages, as well as inferential analy-

sis, such as correlational analysis, t- tests, and analysis of variance (ANOVA) An

understanding of statistics for L2 research at this level will lay the foundation on which readers can further their learning of more complex statistics not covered

in this book (e.g., factor analysis, multivariate analysis of variance, Rasch analysis, generalizability theory, multilevel modeling, and structural equation modeling)

Overview of the Book

The book begins with the basics of the quantifi cation process, then moves on to more sophisticated statistical tools The book comprises a preface, 15 chapters, an epilogue, references, key research terms in quantitative methods, and an index However, readers may choose to skip some chapters and focus on those chapters relevant to their particular interest or research need The chapters in this book include specifi c examples and cases in quantitative research in language acquisition and assessment, as well as analysis of unpublished data collected by the authors Most chapters illustrate how to use SPSS to perform the statistical analysis related

to the focus of the chapter

Chapter 1 (Quantifi cation) introduces the concept of quantifi cation and

dis-cusses its benefi ts and limitations, and how data that are not initially quantitative may become quantitative through coding and frequency counts It also introduces different scales of measurement (interval/ratio, ordinal, and nominal scales)

Chapter 2 (Introduction to SPSS ) presents the interface of the SPSS program,

the appearance of an SPSS data sheet, and preparing a data fi le for quantitative data entry

Chapter 3 (Descriptive Statistics) describes ways of representing data sets,

including graphical displays, frequency counts, and descriptive statistics It also foreshadows some of the statistical conditions that must be met to use some of the statistical tests described later in the book

Chapter 4 (Descriptive Statistics in SPSS ) shows how to compute descriptive

statistics in SPSS, and how to create simple graphs or displays of data

Chapter 5 (Correlational Analysis) introduces the fi rst two types of inferential

statistics, Pearson and Spearman correlations The rationale behind correlations and how to interpret a correlation coeffi cient are discussed

Chapter 6 (Basics of Inferential Statistics) discusses the distinction between a

population and a sample, the logic of hypothesis testing, the normal distribution, and the concept of probability The concept of signifi cance is also discussed The relationships among signifi cance level, effect size, and sample size are highlighted

Chapter 7 ( T- Tests) presents inferential statistics for detecting differences between groups (the independent- samples t- test), and between repeated measure- ment instances from the same group of participants (the paired- samples t- test).

Chapter 8 (Mann- Whitney U and Wilcoxon Signed- Rank Tests) presents the two

nonparametric versions of the t- tests presented in Chapter 7 These two tests are

useful for the analysis of nonnormally distributed and ordinal data

Chapter 9 (One- Way Analysis of Variance [ANOVA]) extends between- group

comparisons as performed in the independent- samples t- test to three or more groups

It discusses the principles of the one- way ANOVA and effect size considerations

Chapter 10 (One- Way Analysis of Covariance [ANCOVA] ) presents an extended

version of the one- way ANOVA that is used when there are preexisting ences between groups, which can distort outcomes

differ-Chapter 11 (Repeated- Measures ANOVA) is an extension of the independent-

samples t- test to more than two groups The repeated- measures ANOVA can

analyze whether there are differences among several measures of the same group This chapter covers the procedures that must be followed when using the repeated- measures ANOVA, and discusses the types of research questions for which this procedure is useful

Chapter 12 (Two- Way Mixed- Design ANOVA) presents an inferential statistic

that combines a repeated- measures ANOVA (Chapter 11) with a between- groups ANOVA (Chapter 9) Such a combination has the advantage of not only evaluat-ing whether group differences affect performance outcomes, but also of being able

to simultaneously analyze the infl uences of time or task factors on performance outcomes

Chapter 13 (Chi- Square Test) demonstrates the use of the chi- square test in

L2 research and compares it with the use of Pearson and Spearman correlations

Chapter 14 (Multiple Regression) presents simple regression and multiple

regres-sion analyses, which are used for assessing the relative impact of language learning and test performance variables Multiple regression allows researchers to examine the relative contributions of predictor variables on an outcome variable

Chapter 15 (Reliability Analysis) demonstrates an extension and application of

correlational analysis to examine the reliability of research instruments

The Epilogue at the end of the book suggests resources for further reading in

quantitative methods

Quantitative Research Abilities

At the end of this book, readers will have developed the following abilities:

• to understand and use suitable quantitative research analyses and approaches

in a specific research area and context;

• to critically read and evaluate quantitative research reports (e.g., journal cles, theses, or dissertations), including the claims made by researchers;

arti-• to apply statistical concepts to their own research contexts This ability goes beyond understanding the specific research examples and statistical proce-

In preparing and writing this book, we have benefi tted greatly from the support of many friends, colleagues, and students First and foremost, we wish to acknowledge

Tim McNamara, whose brilliant pedagogical design of the course Quantitative Methods in Language Studies at the University of Melbourne inspired us to write an

introductory statistical methods book that focuses on conceptual understanding rather than mathematical intricacies In addition, several colleagues, mentors, and friends have helped us shape the book structure and content through invaluable feedback and engaging discussion: Mike Baynham, Janette Bobis, Andrew Cohen, Talia Isaacs, Antony Kunnan, Susy Macqueen, Lourdes Ortega, Brian Paltridge, Luke Plonsky, Jim Purpura, and Jack Richards We would like to thank Guy Middleton for his exceptional work on editing the book chapter drafts We also greatly appreciate the feedback from Master of Arts (Applied Linguistics) students

at the University of Melbourne and Master of Education (TESOL) students at the University of Sydney on an early draft We would like to thank the staff at Routledge for their assistance during this book project: Kathrene Binag, Rebecca Novack, and the copy editors

The support of our institutions and departments has allowed us time to centrate on completing this book The School of Languages and Linguistics at the University of Melbourne supported Carsten with a sabbatical semester, which he spent in the stimulating environment of the Teachers College, Columbia Uni-versity The Sydney School of Education and Social Work (formerly the Faculty

con-of Education and Social Work) supported Aek with a sabbatical semester at the University of Bristol to complete this book project Finally, Kevin Yang and Damir Jambrek deserve our gratitude for their unfl agging support while we worked on this project over several years

ACKNOWLEDGMENTS

Quantifi cation is the use of numbers to represent facts about the world It is used to

inform the decision- making process in countless situations For example, a doctor might prescribe some form of treatment if a patient’s blood pressure is too high Similarly, a university may accept the application of a student who has attained the minimum required grades In both these cases, numbers are used to inform deci-sions In L2 research, quantifi cation is also used For example,

• researchers in SLA might investigate the effect of feedback on students’ ing by comparing the writing scores of a group of students that received feedback with the scores of a group that did not They may then draw con-clusions regarding the effect of that feedback;

writ-• researchers in cross- cultural pragmatics might code requests made by people from different cultures as direct or indirect and then use the codings to com-pare those cultures; and

• researchers may be interested in the effect of a study- abroad program on dents’ language proficiency level In this case, they may administer a language proficiency test prior to the program, and another following the program Analysis of the test scores can then be carried out to determine whether it is worthwhile for students to attend such programs

stu-This chapter introduces fundamental concepts related to quantitative research, such as the nature of variables, measurement scales, and research topics in L2 research that can be addressed through quantitative methods

1

QUANTIFICATION

Quantitative Research

Quantitative researchers aim to draw conclusions from their research that can be generalized beyond the sample participants used in their research To do this, they must generate theories that describe and explain their research results When a theory is in the process of being tested, several aspects of the theory are referred to

as hypotheses This testing process involves analyzing data collected from, for

exam-ple, research participants or databases In language assessment research, researchers may be interested in the interrelationships among test performances across various language skills (e.g., reading, listening, speaking, and writing) Researchers may hypothesize that there are positive relationships among these skills because there are common linguistic aspects underlying each skill (e.g., vocabulary and syntac-tic knowledge) To test this hypothesis, researchers may ask participants to take a test for each of the skills They may then perform statistical analysis to investigate whether their hypothesis is supported by the collected data

Variables, Constructs, and Data

In quantitative research, the term variable is used to describe a feature that can

vary in degree, value, or quantity Values of a variable may be obtained directly from research participants with a high degree of certainty (e.g., their ages or fi rst language), or may have to be inferred from data collected using observation or

measurements of behavior In quantitative research, the term construct is used to

refer to a feature of interest that is not apparent to the naked eye Often constructs are internal to individuals, for example, L2 constructs include language profi -ciency, motivation, anxiety, and beliefs Researchers may use a research instrument (e.g., a language profi ciency test or questionnaire) to collect data regarding these constructs For example, if researchers are interested in the vocabulary knowledge

of a group of students, then vocabulary knowledge is the construct of interest Researchers can ask students to demonstrate their knowledge by taking a vocab-ulary test Here, students’ performance on the test is treated as a variable that represents their vocabulary knowledge The test scores are the data, which will

enable researchers to infer the students’ vocabulary knowledge The term data is used to refer to the values that a variable may take on The term data is, therefore,

used as a plural noun (e.g., ‘data are’ and ‘data were analyzed’)

Issues in Quantifi cation

For the results of a piece of quantitative research to be believable, a minimum number

of research participants is required, which will depend on the research question under

analysis, and, in particular, the expected effect size (to be discussed in Chapter 6).

Quantifi cation 3

In most cases, researchers need to use some type of instrument (e.g., a guage test, a rating scale, or a Likert- type scale questionnaire) to help them quantify a construct that cannot be directly seen or observed (e.g., writing abil-ity, reading skills, motivation, and anxiety) When researchers try to quantify how well a student can write, it is not a matter of simply counting Rather, it involves the conversion of observations into numbers, for example, by applying a scoring rubric that contains criteria which allow researchers to assign an overall score to a piece of writing That score then becomes the data used for further analyses

lan-Measurement Scales

Different types of data contain different levels of information These differences

are refl ected in the concept of measurement scales What is measured and how it is

measured determines the kind of data that results Raw data may be interpreted differently on different measurement scales For example, suppose Heather and Tom took the same language test The results of the test may be interpreted in different ways according to the measurement scale adopted It may be said that Heather got three more items correct than Tom, or that Heather performed better than Tom Alternatively, it may simply be said that their performances were not identical The amount of information in these statements about the relative abili-ties of Heather and Tom is quite different and affects what kinds of conclusion can

be drawn about their abilities The three statements about Heather and Tom relate directly to the three types of quantitative data that are introduced in this chapter:

interval, ordinal, and nomina/categorical data.

Interval and Ratio Data

Interval data allows the difference between data values to be calculated Test scores are a typical kind of interval data For example, if Heather scored 19 points on

a test, and Tom scored 16 points, it is clear that Heather got three points more than Tom A ratio scale is an interval scale with the additional property that it has a well- defi ned true zero, which an interval scale does not Examples of ratio data include age, period of time, height, and weight In practice, interval data and ratio data are treated exactly the same way, so the difference between them has no statistical consequences, and researchers generally just refer to “interval data” or sometimes “interval/ratio data”

It is the precision and information richness of interval data that makes it the preferred type of data for statistical analyses For example, consider the test that Heather and Tom (and some other students) took Suppose that the test was com-posed of 20 questions The full results of the test appear in Table 1.1

According to Table 1.1, it can be said that:

• Heather got more questions right than Tom, and also that she got three more right than Tom did;

• Tom got twice as many questions right as the lowest scorer, Mary; and,

• the difference between Heather and Jack’s scores was the same as the ence between Tom and Mary’s scores, namely eight points in each case.Interval data contain a large amount of detailed information and they tell us exactly how large the interval is between individual learners’ scores They therefore lend them-selves to conversion to percentages Table 1.2 shows the learners’ scores in percentages.Percentages allow researchers to compare results from tests with different maxi-mum scores (via a transformation to a common scale) For example, if the next test consists of only 15 items, and Tom gets 11 of them right, his percentage score will have declined (as 11 out of 15 is 73%), even though in both cases he got four questions wrong In addition to allowing conversion to percentages, interval data can also be used for a wide range of statistical computations (e.g., calculating means) and analyses

differ-Typical real- world examples of interval data include age, annual income, weekly expenditure, and the time it takes to run a marathon In L2 research, interval data include age, number of years learning the target language, and raw scores on lan-guage tests Scaled test scores on a language profi ciency test, such as the Test of English as a Foreign Language (TOEFL), International English Language Testing System (IELTS), and Test of English for International Communication (TOEIC) are also normally considered interval data

TABLE 1.1 Examples of learners and their scores

TABLE 1.2 An example of learners’ scores converted into percentages

Quantifi cation 5

Ordinal Data

For statistical purposes, ratio and interval data are normally considered desirable because they are rich in information Nonetheless, not all data can be classifi ed as interval data, and some data contain less precise information Ordinal data contains information about relative ranking but not about the precise size of a difference

If the data in Tables 1.1 and 1.2 regarding students’ test scores were expressed as ordinal data (i.e., they were on an ordinal scale of measurement), they would tell the researchers that Heather performed better than Tom, but they would not indi-cate by how much Heather outperformed Tom Ordinal data are obtained when participants are rated or ranked according to their test performances or levels of some trait For example, when language testers score learners’ written production holistically using a scoring rubric that describes characteristics of performance, they are assigning ratings to texts such as ‘excellent’, ‘good’, ‘adequate’, ‘support needed’, or ‘major support needed’ Table 1.3 is an example of how the learners discussed earlier are rated and ranked

According to Table 1.3, it can be said that

• Heather scored better than all of the other students;

• Phil and Tom scored the same, and each scored more highly than Jack and Mary; and

• Mary scored the lowest of all the students

While ordinal data contain useful information about the relative standings of test takers, they do not show precisely how large the differences between test tak-ers are Phil and Tom performed better than Mary did, but it is unknown how much better than her they performed Consequently, with the data in Table 1.3,

it is impossible to see that Phil and Tom scored twice as high as Mary Although

it could be said that Phil and Tom are two score levels above Mary, that is rather vague

Ordinal data can be used to put learners in order of ability, but they do little beyond establishing that order In other words, they do not give researchers as much information about the extent of the differences between individual learn-ers as interval data do Ratings of students’ writing or speaking performance are

TABLE 1.3 How learners are rated and ranked

often expressed numerically; however, that does not mean that they are interval data For example, numerical values can be assigned to descriptors as follows: Excellent (5), Good (4), Adequate (3), Support Needed (2); Major Support Needed (1) Table 1.4 presents how the learners are rated on the basis of perfor-mance descriptors.

The numerical scores in Table 1.4 may look like interval data, but they are not They are only numbers that represent the descriptor, so it would not make sense

to say that Tom scored twice as high as Mary did It makes sense to say only that his score is two levels higher than Mary’s This becomes even clearer if the rating scales are changed as follows: excellent (8), good (6), adequate (4), support needed (2), and Major support (0) That would give the information in Table 1.5

As can been seen in Tables 1.4 and 1.5, the descriptors do not change, but the numerical scores do Tom and Phil’s scores are still two levels higher than Mary’s, but now their numerical scores are three times as high as Mary’s score This illustration makes it clear that numerical representations of descriptors are only symbols that say nothing about the size of the intervals between adjacent levels They indicate that Heather is a better writer than Tom, but since they are not based on counts, they cannot indicate precisely how much of a better writer Heather is than Tom

In L2 research, rating scale data are an example of ordinal data These are commonly collected in relation to productive tasks (e.g., writing and speaking) Whenever there are band levels, such as A1, A2, and B1, as in the Common Euro-pean Reference Framework for Languages (see Council of Europe, 2001), or bands

TABLE 1.4 How learners are scored on the basis of performance descriptors

TABLE 1.5 How learners are scored on a different set of performance descriptors

fi nal data type often used in L2 research (i.e., nominal or categorical data) does not contain information about the strengths of learners, but rather about their attributes.

Nominal or Categorical Data

Nominal data (i.e., named data, also called categorical data) are concerned only

with sameness or difference, rather than size or strength Gender, native language, country of origin, experimental treatment group, and test version taken are typical examples of nominal data (i.e., data on a nominal scale of measurement) In the example of Heather, Tom, Phil, Jack, and Mary, the nominal variable of gender has two levels (male and female), and there are two males and three females In research,

nominal variables are often used as independent variables; in other words, variables

that are expected to affect an outcome Independent variables, such as teaching methods and types of corrective feedback on performance, can be hypothesized to

affect learning outcomes or behaviors, which are then treated as dependent variables,

as they depend on the independent variables It should be noted that dependent and independent variables are related to research design The nominal variable

‘study- abroad experience’, with the levels ‘has studied abroad’ (Yes = coded 1) or

‘has not studied abroad’ (No = coded 0), can be used to split a sample of ers into two groups in order to compare the scores of learners with study- abroad experience with the scores of learners without study- abroad experience

learn-Nominal data are often coded numerically to facilitate the use of spreadsheets Table 1.6 presents an example of how nominal data can be coded numerically

As can be seen in Table 1.6, it does not matter which numbers are assigned to the nominal data because the idea that one number is better than another is meaningless

in this case Also, the numerical codes do not have a mathematical value in the way that ratio, interval and ordinal data do For example, it cannot be said that females are better than males merely because the code assigned to females is 2 and the code for males is 1 However, frequency counts of nominal variables can be made, which

do have mathematical values For instance, for the variable ‘gender’, there are three males and two females (i.e., 40% of the participants are female and 60% are male in the data set)

Nominal data are sometimes called categorical data because objects of

inter-est can be sorted into categories (e.g., men versus women; Form A versus Form

B versus Form C) When a variable can only have two possible values (pass/fail; international student/domestic student, correct/incorrect), this type of data

is sometimes called dichotomous data For example, students may be asked to

com-plete a free writing task in which they are limited to three types of essays: personal experience (coded 1), argumentative essay (coded 2), and description of a process (coded 3) Table 1.7 shows which student chose which type

The data in the Type column do not provide any information about one learner being more capable than another It only shows which learners chose which essay type, from which frequency counts can be made That is, the process description and personal experience types were chosen two times each, and the argumenta-tive essay was chosen once How nominal data are used in statistical analysis for research purposes will be addressed in the next few chapters

Transforming Data in a Real- Life Context

In a real- life situation, raw data need to be transformed for a variety of reasons Take the common situation in which new students entering a language program

TABLE 1.6 Nominal data and their numerical codes

Native or nonnative speaker Native (coded 1), nonnative (coded 2)

Pass or fail Pass (coded 1), fail (coded 0)

Test form Form A (coded 1), Form B (coded 2), Form C (coded 3) Nationality American (coded 1), Canadian (coded 2),

British (coded 3), Singaporean (coded 4), Australian (coded 5), and New Zealander (coded 6)

First language English (coded 1), Mandarin (coded 2),

Spanish (coded 3), French (coded 4), Japanese (coded 5) Experimental groups Treatment A group (coded 1), Treatment B group

(coded 2), Control group (coded 3) Profi ciency level groups Beginner (coded 1), Intermediate (coded 2),

High Intermediate (coded 3), Advanced (coded 4)

TABLE 1.7 Essay types chosen by students

Quantifi cation 9

take a placement test consisting of, say, 60 multiple- choice questions assessing their

listening, reading, and grammar skills Based on the test scores, the students are

placed in one of three levels: beginner, intermediate, or advanced In addition, the

three levels are taught at three different locations, as presented in Table 1.8

Table 1.9 presents the scores and placements of the fi ve students introduced earlier.

The test scores are measured on an interval measurement scale that is based on

the count of correct answers in the placement test and provides detailed

informa-tion It can be said that:

• Heather’s score is in the advanced range since her score is 11 points above the

cut- off, and her score is much higher than Tom’s, whose score was 23 points

lower than hers;

• Tom’s score is in the intermediate range, but it is close to the cut- off for the

advanced range, missing it by just three points;

• Tom’s score is far higher than Phil’s, with a difference of 17 points, yet both

scores are in the intermediate range;

• Phil’s score is just one point above the cut- off for the intermediate level, and

is only four points higher than Jack’s score Despite the small difference in

their scores, Jack was placed in the beginner level and Phil was placed in the

intermediate level; and,

• Mary’s score is in the middle of the beginner level

Because the information is detailed, the placement test can be evaluated

criti-cally For example, Phil and Tom’s scores are 17 points apart whereas Phil and

Jack’s are only four points apart Phil’s profi ciency level is arguably closer to Jack’s

than to Tom’s Yet, Phil and Tom are both classifi ed as intermediate, but Jack is

classifi ed in the beginner level This is known as the contiguity problem, and it is

TABLE 1.8 The three placement levels taught at three different locations

TABLE 1.9 The students’ test scores, placement levels, and campuses

common whenever cut- off points are set arbitrarily: students close to each other but on different sides of the cut- off point can be more similar to each other than

to people further away from each other but on the same side of the cut- off point.Now imagine that there are no interval- level test- score data, but instead just the ordinal- level placement levels data and the campus data, as in Table 1.10

As can be seen in Table 1.10, the differences between Tom and Phil and the problematic nature of the classifi cation that were so apparent before are no longer visible The information about the size of the differences between learners has been lost and all that can be deduced now is that some students are more profi -cient than others Tom and Phil have the same level of profi ciency and Jack is clearly different from both of them This demonstrates why ordinal data are not as precise as interval data Information is lost, and the differences between the learn-ers seen earlier are no longer as clear

Highly informative interval data are often transformed into less informative ordinal data to reduce the number of categories the data must be split into No language program can run with classes at 60 different profi ciency levels; moreover, some small differences are not meaningful, so it does not make sense to group learners into such a large number of levels However, setting the cut- off points is often a problematic issue in practice

While the ordinal profi ciency level data are less informative than the interval test- score data, they can be scaled down even further, namely to the nominal cam-pus data (see Table 1.11)

If this is all that can be seen, it is impossible to know how campus assignment

is related to profi ciency level However, it can be said that:

• Tom and Phil are on the same campus;

• Mary and Jack are on the same campus; and

• Heather is the only one at the Ocean campus

This information does not indicate who is more profi cient since nominal data

do not contain information about the size or direction of differences They cate only whether differences exist or not

indi-Transformation of types of data can happen downwards only, rather than upwards, in the sense that interval data can be transformed into ordinal data and

TABLE 1.10 The students’ placement levels and campuses

Quantifi cation 11

ordinal data can be transformed into nominal data (e.g., by using test scores to place learners in classes based on profi ciency levels and then by assigning classes to campus locations) Table 1.12 illustrates the downward transformation of scales.Transformation does not work the other way around That is, if it is known which campus a learner studies at, it is impossible to predict that learner’s profi -ciency level Similarly, if a learner’s profi ciency level is known, it is impossible to predict that learner’s exact test score

Topics in L2 Research

It is useful to introduce some of the key topics in L2 research that can be examined using a quantitative research methodology Here, areas of research interests in SLA, and language testing and assessment (LTA) research are presented

SLA Research

There is a wide range of topics in SLA research that can be investigated using quantitative methods, although the nature of SLA itself is qualitative SLA research aims to examine the nature of language learning and interlanguage processes (e.g., sequences of language acquisition; the order of morpheme acquisition; charac-teristics of language errors and their sources; language use avoidance; cognitive processes; and language accuracy, fl uency, and complexity) SLA research also aims to understand the factors that affect language learning and success Such factors may be internal or individual factors (e.g., age, fi rst language or cross- linguistic infl uences, language aptitude, motivation, anxiety, and self- regulation), or external or social factors (e.g., language exposure and interactions, language and

TABLE 1.11 The students’ campuses

TABLE 1.12 Downward transformation of scales

socialization, language community attitude, feedback, and scaffolding) There are several texts that provide further details of the scope of SLA research (e.g., Ellis, 2015; Gass with Behney & Plonsky, 2013; Lightbown & Spada, 2013; Macaro, 2010; Ortega, 2009; Pawlak & Aronin, 2014).

Topics in LTA Research

LTA research primarily focuses on the quality and usefulness of language tests and assessments, and issues surrounding test development and use (e.g., test validity, impact, use and fairness; see Purpura, 2016, or Read, 2015, for an overview) Like SLA research, LTA research focuses on the measurement of language skills and communicative abilities in a variety of contexts (e.g., academic language purposes such as achievement tests, profi ciency tests, and screening tests, and occupational purposes such as tests for medical professions, aviation, or tourist guides) The

term assessment is used to cover more than the use of tests to elicit language

perfor-mance For example, assessment may be informally carried out by teachers in the classroom There are several books on LTA that consider the key issues: Bachman and Palmer, 2010; Carr, 2011; Coombe, Davidson, O’Sullivan and Stoynoff, 2012; Douglas, 2010; Fulcher, 2010; Green, 2014; Kunnan, 2014; Weir, 2003 While there has been an increase in qualitative and mixed methods approaches in LTA, quantitative methods remain predominant in LTA research This is mainly because tests and assessments involve the measurement and evaluation of language ability Like SLA researchers, LTA researchers are interested in understanding the internal factors (e.g., language knowledge, cognitive processes, and affective factors), and external factors (e.g., characteristics of test tasks such as text characteristics, test techniques, and the task demands and roles of raters) that affect test performance variation SLA and LTA research are related to each other in that SLA research focuses on developing an understanding of the processes of language learning, whereas LTA research measures the products of language learning processes

Quantifi cation 13

this study High profi ciency learners were assumed to have greater target language competence than low profi ciency learners had, but the degree of the difference was not relevant The researcher was interested only in comparing the issues that high and low profi ciency learners struggled with Khang’s other measures were interval variables (e.g., averaged syllable duration, number of corrections per min-ute, and number of silent pauses per minute, which can all be precisely quantifi ed)

Summary

It is essential that quantitative researchers consider the types of data and levels of measurement that they use (i.e., the nature of the numbers used to measure the variables) In this chapter, issues of quantifi cation and measurement in L2 research, particularly the types of data and scales associated with them, have been discussed The next chapter will turn to a practical concern: how to manage quantitative data with the help of a statistical analysis program, namely the IBM Statistical Package for Social Sciences (SPSS) The concept of measurement scales will be revisited through SPSS in the next chapter

Review Exercises

To download review questions and SPSS exercises for this chapter, visit the panion Website: www.routledge.com/cw/roever

There are a number of statistical programs that can be used for statistical analysis

in L2 research, for example, SPSS (www.01.ibm.com/software/au/analytics/spss/), SAS ( Statistical Analysis Software; www.sas.com/en_us/software/analytics/stat.html), Minitab (www.minitab.com/en- us/), R (www.r- project.org/), and PSPP (www.gnu.org/software/pspp/)

In this book, SPSS is used as part of a problem- solving approach to tive data analysis IBM is the current owner of SPSS, and SPSS is available in both

quantita-PC and MacOS formats SPSS is widely used by L2 researchers, partly because its interface is designed to be user friendly: users can use the point- and- click options

to perform statistical analysis There are both professional and student versions of SPSS At the time of writing, SPSS uses a licensing system under which the user has to pay to renew his/her license every year It is advised that readers check whether their academic institution holds an institutional license, under which SPSS can be freely accessed by staff and students Alternatively, readers could con-sider PSPP, a freeware program modeled on SPSS

Preparing Data for SPSS

In Chapter 1, the nature of quantifi cation in L2 research was discussed In this ter, four basic steps required to prepare the data for analysis using SPSS are outlined

chap-Step 1: Checking and Organizing Data

Once the data have been collected, the researchers check whether the data are plete or whether there are missing data or responses from participants Missing data will reduce the sample size The data should then be organized by assigning identity

com-2

INTRODUCTION TO SPSS

Introduction to SPSS 15

numbers (IDs) to each participant’s data IDs are important in that they allow the data

in SPSS to be checked against the actual data If the research instrument requires ing (e.g., a test), the scoring will need to be completed and checked before the data can be entered into SPSS The data should be stored in a secure place

scor-Step 2: Coding Data

As discussed in Chapter 1, quantitative data can be categorized into ratio, interval, ordinal, and nominal data Researchers need to know the type of data that they have obtained so that they can code the data appropriately for analysis In some cases, data may already be numerical and hence will not require coding (e.g., age, test scores, grade point average, number of years of study, ratings or responses in Likert- type scale questionnaires) These data can be used directly for data entry Researchers simply need to know the numerical data type to be able to analyze them appropriately Other types of data, especially nominal data, require coding For example, the numbers 1 and 2 can be assigned to male and female participants respectively Also, country codes for each participant can be assigned if the partici-pants are from different countries (e.g., 1 = China, 2 = United States, 3 = Spain, etc.) Once the data have been coded, they need to be entered into SPSS

Step 3: Entering Data Into SPSS

After the data have been organized into quantitative form, they can be typed into SPSS or imported from a Microsoft Excel fi le into an SPSS spreadsheet (see the

“Importing Data From Excel” section) Data types need to be defi ned in SPSS, so that they can be properly analyzed

Step 4: Screening and Cleaning Data

Once data entry has been completed, the accuracy of the data entry needs to be carefully checked The issue of missing data also needs to be addressed The data

in the SPSS fi le can be compared with the actual data one point at a time, or by randomly checking a sample of the data Screening can also be achieved through

an application of descriptive statistics The number of items in the data set and the minimum and maximum values in the data set may easily be found and compared with those of the actual data For example, if the maximum score for a question-naire item is 5, but the maximum score detected in the SPSS fi le is 55, it is clear that there was a mistake in data entry

Important Notes on SPSS

First, SPSS deals with quantitative data There is limited scope to enter words into SPSS, and this should be avoided For example, while it is possible to enter the word ‘male’ in an SPSS spreadsheet for a learner’s value under the nominal vari-able ‘gender’, it is more effective to enter ‘1’ to represent ‘male’, and ‘2’ to represent

‘female’, for example This chapter will illustrate how to code data in SPSS

Second, SPSS can produce a statistical analysis output as per researchers’ tions, but the output can be ‘meaningful’ or ‘meaningless’ depending on the types of data used and how well the characteristics of the scales discussed in Chapter 1 are understood For example, SPSS will quite readily compute the average of two nomi-nal data codes, such as gender coded as ‘1’ for male and ‘2’ for female However, it does not make sense to talk about ‘average gender’ SPSS will not stop researchers from performing such meaningless computations, so knowledgeable quantitative research-ers need to be aware of what computations will produce meaningful, useful results.

instruc-Creating a Spreadsheet in SPSS

When using SPSS, the fi rst thing to do is to create a spreadsheet into which data can be entered Data collected on the fi ve learners in Chapter 1 ( Tom, Mary, Heather, Jack, and Phil) will be used to illustrate how to perform analysis in SPSS

SPSS Instructions: Creating a Spreadsheet

Open SPSS.

FIGURE 2.1 New SPSS spreadsheet

Cancel the dialog offering to open an existing spreadsheet A new, blank spreadsheet will open (as shown in Figure 2.1).

There are two tabs at the bottom left- hand side of the spreadsheet (Data View and Variable View) When a new fi le in SPSS is created, you will automatically be in

Introduction to SPSS 17

Data View, and the data can be entered using this view However, it is best to defi ne the variables that will be used fi rst To do this, click on the Variable View tab

To illustrate how to defi ne variables, the data from the fi ve students in Table 1.9

in Chapter 1 will be used There are four variables, namely student name, test score, placement level, and campus When the word ‘student’ is typed into a cell in the Name Column, SPSS automatically populates the rest of the row with default values (see Figure 2.2)

SPSS does not allow spaces in names For example, ‘First Language’ (with a space between the two words) cannot be typed into the Name Column, but

‘FirstLanguage’ (without a space) can If a space is present in a variable name, SPSS will indicate that the ‘variable name contains an illegal character’ Further information on how to name variables can be found at: www.ibm.com/support/knowledgecenter/SSLVMB_20.0.0/com.ibm.spss.statistics.help/syn_variables_variable_names.htm)

In the second column (Figure 2.3), Type is automatically set to Numeric, which means that only numbers can be entered into the spreadsheet for that variable If researchers wish to enter the names of the research participants, they need to be able to enter words (SPSS calls variables that take on values containing characters other than numbers ‘string’ variables) To do this, click on and then on

the blue square with ‘ .’ that appears next to Numeric When the Variable Type

dialog opens, choose ‘String’ and then click on the OK button (see Figure 2.4).

The variable type is now set to be a string variable The column width in SPSS

is set to a default of eight characters, but this can be increased Another column that is optional but useful to fi ll in is Label (see Figure 2.5) Labels are useful when abbreviations or acronyms are used as variables (e.g., L1 = fi rst language; EFL = English as a foreign language)

FIGURE 2.2 SPSS Variable View