ĐÁNH GIÁ VIỆC THIẾT kế BẢNG câu hỏi THU THẬP dữ LIỆUTRONG các NGHIÊN cứu KHẢO sát ở KHOA sư PHẠM TIẾNG ANH

The study, as its name reveals, has been conducted with the hope to review and evaluate the validity and reliability of the employed questionnaires in ELT survey research at FELTE, ULIS, VNU. With the design of evaluation research, the study aims to present critical comments on what student practitioners have already applied successfully from their research methodology background, and what they need to improve in order for the research to be more successful. Through the reviews of eight questionnaires employed in six survey graduation papers from 2007 to 2012 at FELTE, ULIS, VNU, the limited validity and reliability of those data collection instruments were revealed. The quality criteria of the source, pilot procedure, main parts and the layouts of the questionnaires, as well as the consideration of the questionnaire items should have received more deliberation for the insurance of the quality of questionnaires. From these findings, some practical implications were proposed with the hope to suggest some solutions to the problem of limited validity and reliability of the questionnaires, thus contributing to the value enhancement of the research projects on ELT Methodology at FELTE, ULIS.

The study, as its name reveals, has been conducted with the hope to review andevaluate the validity and reliability of the employed questionnaires in ELT surveyresearch at FELTE, ULIS, VNU With the design of evaluation research, the studyaims to present critical comments on what student practitioners have already appliedsuccessfully from their research methodology background, and what they need toimprove in order for the research to be more successful

Through the reviews of eight questionnaires employed in six survey graduationpapers from 2007 to 2012 at FELTE, ULIS, VNU, the limited validity andreliability of those data collection instruments were revealed The quality criteria ofthe source, pilot procedure, main parts and the layouts of the questionnaires, as well

as the consideration of the questionnaire items should have received moredeliberation for the insurance of the quality of questionnaires

From these findings, some practical implications were proposed with the hope tosuggest some solutions to the problem of limited validity and reliability of thequestionnaires, thus contributing to the value enhancement of the research projects

on ELT Methodology at FELTE, ULIS

TABLE OF CONTENTS

Page

Acknowledgements i

Abstract ii

Table of Contents iii

List of Abbreviations vi

List of Tables and Figures vii

CHAPTER 1: INTRODUCTION 1.1 Statement of the Problem and Rationale for the Study 1

1.2 Aims of the Study and Research Question 2

1.3 Scope of the Study 2

1.4 Significance of the Study 3

1.5 Organization of the Study 3

CHAPTER 2: LITERATURE REVIEW 2.1 English Language Teaching Research 4

2.1.1 Introduction to English Language Teaching (ELT) 4

2.1.2 Research in ELT 4

2.2 Survey Research in ELT 6

2.2.1 Definitions of Survey Research 6

2.2.2 Values of Survey Research 7

2.2.3 Survey Research Procedure 10

2.2.4 Data Collection Instruments in Survey Research 13

2.3 Questionnaires in Survey Research 13

2.3.1 Definitions of a Questionnaire 13

2.3.2 Types of Data Collected from Questionnaires 14

2.3.3 Advantages and Disadvantages of Using Questionnaires 15

2.4 Questionnaire Construction 17

2.4.1 Questionnaire Construction Procedure 17

2.4.2 Quality Criteria for Constructing a Questionnaire 19

2.4.2.1 Quality Criteria for the Main Parts of the Questionnaire 19

2.4.2.2 Quality Criteria for the Content of the Questionnaire 20

2.4.3 Criteria for a Well-Constructed Questionnaire 21

2.5 Research Gaps in Questionnaire Construction Evaluation at FELTE, ULIS 23

2.6 Summary 23

CHAPTER 3: METHODOLOGY 3.1 Research Context 24

3.2 Research Question 25

3.3 Research Approach: Evaluation Research 25

3.3.1 Evaluation Research 25

3.3.2 Evaluation Research Procedure 26

3.4 Sampling 28

3.4.1 Evaluands 28

3.4.2 Sampling Strategy: Criterion Sampling 28

3.4.3 Sampling Procedure and Results 30

3.4.4 Description of the Subjects of the Study 31

3.5 Data Collection Instrument 34

3.6 Data Collection Procedure 35

3.7 Data Analysis Procedure 35

3.8 Summary 36

CHAPTER 4: RESULTS AND DISCUSSIONS 4.1 Evaluation of the Reliability 37

4.1.1 Analysis of the Sources and the Pilot Procedure 37

4.1.2 Analysis of the Main Parts 39

4.1.3 Analysis of the Layouts 41

4.2 Evaluation of the Validity 43

4.2.1 Analysis of Questionnaire A 43

4.2.2 Analysis of Questionnaire B 47

4.2.3 Analysis of Questionnaires C1 and C2 49

4.2.4 Analysis of Questionnaires D1 and D2 50

4.2.5 Analysis of Questionnaire E 53

4.2.6 Analysis of Questionnaire F 55

4.3 The Extent to which the Student-Researchers Paid Attention to the Quality Criteria of Questionnaire Construction 61

4.4 Summary 63

CHAPTER 5: CONCLUSION 5.1 Summary of the Major Discussions 64

5.2 Pedagogical Implications 65

5.3 Limitations 66

5.4 Suggestions for Further Studies 67

REFERENCES 68

APPENDICES 70

LIST OF ABBREVIATIONS

ELT: English Language Teaching

FELTE: Faculty of English Language Teacher Education

ULIS: University of Languages and International Studies

VNU: Vietnam National University

LIST OF TABLES AND FIGURES LIST OF FIGURES

Figure 1: Broad Categories of Research 5

Figure 2: Stages in the Planning of a Survey 11

LIST OF TABLES Table 2.1: Overview of Research Styles 7

Table 2.2: Steps of Conducting a Survey Research Study 10

Table 2.3: Equivalence between Survey Research Procedures provided by Cohen et al., (2007) and Mertens (2005) 12

Table 2.4: Advantages and Disadvantages of Using Questionnaires 15

Table 2.5: The Process of Constructing a Good Questionnaire 18

Table 2.6: The Main Parts of a Questionnaire 19

Table 2.7: Types of Validity 21

Table 2.8: Criteria to Evaluate the Reliability of a Questionnaire 22

Table 2.9: Criteria to Evaluate the Validity of a Questionnaire 23

Table 3.1: Evaluation Research Procedure 27

Table 3.2: Criteria for Sampling 29

Table 3.3: The Sampling Procedure and Results 30

Table 3.4: Subjects of the Study 31

Table 3.5: Data Collection Instrument: An Evaluation Framework for Questionnaire Construction 34

Table 4.1: Sample Questionnaire Sources and Pilot Procedures 38

Table 4.2: Main Parts of the Sample Questionnaires 40

Table 4.3: Layouts of the Sample Questionnaires 42

Table 4.4: Analysis of Questionnaire A 44

Table 4.5: Analysis of Questionnaire B 48

Table 4.6: Analysis of Questionnaire D1 51

Table 4.7: Analysis of Questionnaire D2 52

Table 4.8: Analysis of Questionnaire E 54

Table 4.9: Analysis of Questionnaire F 56

CHAPTER 1: INTRODUCTION1.1 Statement of the Problem and Rationale for the Study

Research is one major activity in the teaching and learning process at Faculty ofEnglish Teacher Education (FELTE), University of Languages and InternationalStudies (ULIS), Vietnam National University, Hanoi (VNU) The student researchactivity is conducted annually, calling for a large number of projects in variousfields, especially in English Language Teaching Methodology (ELT) Moreimportantly, research is used as an activity for assessment of senior students, in theform of graduation paper Annually, dozens of projects are conducted by students,many of whom choose the survey approach and employ questionnaires as theirresearch instrument In fact, it is evident that the questionnaire is generallyconsidered one of the most popular research methods utilized by the studentresearchers

Nonetheless, to which extent do the student practitioners pay attention toquestionnaire construction is questionable, especially in terms of validity andreliability The help of computers and the Internet have led to the ease of

questionnaire design, as Dornyei (2003, p.3) says: “it is possible to draw up something that looks respectable in a few hours”, but the quality of data is another

issue As Gillham (2000, p.1) states, “good research cannot be built on poorlycollected data”, and well constructed questionnaires will eliminate this risk In otherwords, “developing a questionnaire that will yield worthwhile data” (Gillham, 2000,p.1) is the prerequisite for valuable findings in research in general and ELT research

in particular

Meanwhile, the scarcity of research projects on the issues of questionnaireconstruction at FELTE, ULIS is recognized In the University Library, there hasbeen little or even no evidence of the reviewing and evaluating the current situation

of questionnaire construction Those studies, if conducted may serve as a source ofreference for the enhancement in the quality of the questionnaires employed in theirresearches as well as the quality of the students’ projects as a whole

Therefore, the research “Questionnaire Construction in ELT Survey Research at FELTE, ULIS: An evaluation study” was conducted as regards the graduation

theses at FELTE, ULIS, VNU with the close investigation of the questionnaireconstruction in those survey research projects.

1.2 Aims of the Study and Research Question

The study was conducted with the hope to develop an Evaluation Framework forquestionnaire construction, to analyze the previously employed questionnaires and

to evaluate their validity and reliability Through the investigation into the utilizedquestionnaires, which are the products of the questionnaire construction process,this project aimed to present critical evaluation on the questionnaire construction ofthe previous theses, especially in regards to its relevance to the criteria for a well-constructed questionnaire

To be specific, the research project aimed at answering the following question:

To what extent do the questionnaires utilized in the ELT survey research projects at FELTE, ULIS satisfy the quality criteria of questionnaire construction?

1.3 Scope of the Study

From the aims of the study and the research questions, the scope of the study isdetermined The study concerns with only the graduation papers in the field of ELTMethodology at FELTE, ULIS As the research topic is questionnaire construction,the graduation papers investigated involves attitudinal issues for the most typicalfeatures of the questionnaires to be utilized Of those research projects on attitudinalissues, the projects which employ questionnaires as their main or only datacollection instrument are selected

The researcher limits the investigated period of time to only six years from 2007 to

2012 The reasons for this decision is that 2006 was the year when the Course ofResearch Methodology was first introduced to the third year students at FELTE,ULIS These students conducted their graduation papers in 2007; consequently, thestarting year for investigation is decided to be 2007 to ensure that the studentresearchers had been some theoretical backgrounds of research methodology

To sum up, the research Questionnaire Construction in ELT Survey Research at FELTE, ULIS: An evaluation study concerns with only the survey graduation

papers which employ questionnaires as their main or only data collection instrumentfrom 2007 to 2012.

1.4 Significance of the Study

This research project, focusing on questionnaire construction, aims to provide aclearer view of the questionnaire construction quality in ELT research at FELTE,ULIS The investigation into the situation of questionnaire construction, especiallyregarding the validity and reliability of the employed questionnaires, is hoped topresent critical evaluation on questionnaire construction – what practitioners havealready applied successfully from their research methodology background, and whatthey need to improve in order for the research to be more successful

Furthermore, through the reviews of related studies on questionnaire, as well as theevaluation tool presented in this study, the author hopes to raise the researchpractitioners’ awareness of the “standardized questionnaire” as well as provide them

a suggestion of questionnaire self-evaluation The study is expected to serve as auseful source of reference for junior research practitioners who want to employquestionnaire as a tool for their survey research, especially in ELT field

1.5 Organization of the Study

Chapter 1 – Introduction - presents an overview of this evaluation research.

Chapter 2 is the review of relevant literature, setting the theoretical foundation for

data collection and analysis

Chapter 3 describes the design of this study, including the approach, sampling

strategy, data collection instrument and procedure, as well as framework for dataanalysis

Chapter 4 presents results of the research and discussions from the analysis.

Finally, chapter 5 – Conclusion – is the summary of the findings as well as the

pedagogical implications and limitations of this evaluation study

CHAPTER 2: LITERATURE REVIEW

In Chapter 2, the relevant literature of Survey Research on English Language Teaching (ELT) Methodology is reviewed with the emphasis on the construction procedure of questionnaires as the data collection instrument of the survey research Particularly, the quality criteria of a well-constructed questionnaire is thoroughly investigated and presented in this chapter.

2.1 English Language Teaching Research

2.1.1 Introduction to English Language Teaching

English Language Teaching (ELT) is an applied linguistics branch of study of greatattention This branch started to receive much interest from researchers in the latesixteenth century with the arrival of the first textbooks “designed solely to teachEnglish as a foreign language” (Howatt, 1984, p 6) Through the appearance anddevelopment of various theories in English Language Teaching, this field hasbecome a profession worldwide since 1900 Today, ELT has become the topic ofinterest of thousands of linguists around the world and also a major in numerousuniversity programs, usually under the name of TESOL – Teaching English toSpeakers of Other Languages, or Language Teacher Education

2.1.2 Research in ELT

Research in ELT is generally understood as the systematic investigation in ELT InBrown and Rogers (2002, p 12), this term is defined as “any systematic andprincipled inquiry in language learning and teaching” It means that Englishlanguage learning and teaching process is the overall objective of ELT research The importance of research in ELT is emphasized through the article “Language

Teacher Education” by Jourdenais (in The Handbook of Language Teaching, edited

by Long and Doughty, 2009, pp 647-58) In this article, Jourdenais highlights therole of research in ELT:

It is not enough that teachers be successful users of the target language; nor is it sufficient that they understand how the language itself works They must have a level of awareness of language that enable them to assess, analyze, and present

it to learners in ways that will enhance acquisition (p 652)

Approaches and methods in ELT research quite varies and many authors presentthese approaches and methods clearly in their books, some prominent names amongwhom are Brown (2001), Brown and Rodgers (2002), Mertens (2005) andDenscombe (2010) Among them, the best summary of approaches and methods isprobably the figure “Broad Categories of Research” by Brown (2001, p 1) asfollows:

Figure 1: Broad Categories of Research (Brown, 2001, p.1)

Figure 1 demonstrates Brown’s idea to classify different types of research in ELT.However, Brown (2001) uses this figure to illustrate the position of questionnairesamong various approaches and methods in ELT research, for it is the topic of the

book Using Surveys in Language Program Brown and Rogers (2002), in their book Doing Second Language Research, also use this chart, but this time a more detailed

discussion of research categories is presented As can be seen clearly in the chart,

“research traditions available to second language scholars” (in their words, p 10), is

classified into primary and secondary research, the former involves the findings

from original data, while the latter is based on secondary sources such as “otherresearchers’ books and articles” Each sub-type is divided into smaller ones, withdifferent instruments and designing techniques Brown and Rogers also emphasize

the conclusion that the initial steps of any research project is “to decide what type,

or types, of research you are doing” (p 12)

2.2 Survey Research in ELT

2.2.1 Definitions of Survey Research

Survey Research is a genre of social research The term “Survey Research” is oftenreferred to as an approach or a strategy, different from the word “survey” only,which means the tool or the method to collect data The distinction between

“strategy” and “method” is quite clear, especially through books by Mertens (2005,p.167) and Denscombe (2010, pp 3-8) For example, Denscombe (2010) clarifiesthat “research methods are the tools for data collection – things like questionnaires,interviews, observation and documents” (p 4), while he lists surveys (in plural form– which refers to the design, rather than the tool) as one among the eight mainresearch strategies (pp 5-6) He also presents three main characteristics of the

survey approach, namely Wide and inclusive coverage, at a specific point of time and empirical research (pp 11-12)

Brown (2001, p 1) has similar ideas through his clarification of the term “surveyresearch” He highlights the attributes of survey research, the most distinctive ofwhich are data-based and using interviews and questionnaires to collectinformation In the latter part of Chapter 1, Brown (2001) explores survey research

in linguistics field of study, which he calls “language surveys” In his words,

“Language surveys are any survey research studies that gather data on thecharacteristics and views of informants about the nature of language or languagelearning through the use of oral interviews or written questionnaires” (p 2)

Based on the definition of “language surveys” by Brown (2001), it can beunderstood that survey research’s most prominent characteristic is the datacollection process to find out “characteristics and views” of the participants aboutthe interested topic of study, using interviews or questionnaires It can also beinferred from his definition that survey research is favourable thanks to its ability of

generalization, for survey research involves numerous participants to discover andgeneralize those collected “characteristics and views”

2.2.2 Values of Survey Research

Different types of research serve different purposes and it is the aim of the researchstudy that researchers choose the appropriate design or research styles It should benoted that each type has its own advantage and it is suggested that researchers have

a good overview of those types to make appropriate decision of their researchdesign

There have been numerous books and journals about research methods, but it seems

that the summary of Cohen et al (2007) organizes the information and key terms in

the most efficient way This summary provides researchers an overview of researchapproaches and designs with prominent characteristics of each type The summary

is cited in Table 2.1:

Table 2.1: Overview of Research Styles

(Adapted from Box 3.2: Elements of Research Styles by Cohen et al., 2007, pp.84-6)

Survey Gathering large-scale

data in order to make

Describes and explains Represents wide population Gathers numerical data Much use of questionnaires and assessment / test data

Experiment Comparing under

Control and experimental groups

Treats situations like a laboratory

Causes due to experimental intervention

Does not judge worth Simplistic

continued

Table 2.1: Overview of Research Styles (continued)

Ethnography Portrayal of events in

subject’s terms Subjective and reporting of multiple perspectives Description, understanding and explanation of specific situation

Perceptions and views of participants Issues as they emerge over time

Context specific Formative and emergent Responsive to emerging features

Allows room for judgements and multiple perspectives Wide database gathered over a long period of time

Time-consuming to process data

Action

research

To plan, implement, review and evaluate an intervention designed to improve

practice / solve local problem

To empower participants through research

involvement and ideology critique

To develop reflective practice

To promote equality democracy

To link practice and research

To promote collaborative research

Everyday practices Outcomes of interventions Participant empowerment Reflective practice Social democracy and equality Decision-making

Context-specific Participants as researchers Reflection on practice Interventionist – leading to solution of ‘real’ problems and meeting ‘real’ needs

Empowering for participants Collaborative

Promoting praxis and equality Stakeholder research

Case study To portray, analyse and

interpret the uniqueness of real individuals and situations through accessible accounts

To catch the complexity and situatedness of behaviour

To contribute to action and intervention

To present and represent reality – to give a sense of

‘being there’

Individuals and local situations Unique instances

A single case Bounded phenomena and systems:

Holistic treatment of phenomena

What can be learned from the particular case

continued

Table 2.1: Overview of Research Styles (continued)

Testing and

assessment

To measure achievement and potential

To diagnose strengths and weaknesses

To assess performance and abilities

Academic and academic, cognitive, affective and psychomotor domains – low-order

non-to high-order Performance, achievement, potential, abilities Personality characteristics

Materials designed to provide scores that can be aggregated Enables individuals and groups to be compared In-depth diagnosis Measures performance

It can be concluded from Table 1 that survey research tends to be the mostadvantageous in circumstances of factual information gathering with the closerelationship with large-scale population

This conclusion agrees with Denscombe (2010, p 12) that the data most frequentlyand most effectively collected from survey research are often mass-data related,concerned with “specific issues” within “groups or categories of people” and

“relatively uncomplicated”

In particular, language surveys serve best for “curriculum development” and for

“research” (Brown, 2001, p 12) Survey research benefits needs analysis andprogram evaluation to improve an academic course, or explores opinions andattitudes of targeted objectives thanks to the ability of “describing, exploring orexplaining physical characteristics, phenomena, behaviors, attitudes, and so forth”(Brown, 2001, pp 12-16)

However, the values of a particular survey study depend greatly on the quality ofinstruments employed, because “good research cannot be built on poorly collecteddata” (Gillham, 2000, p.1) As presented in 2.2.3, interviews and questionnaires are

two instruments used in survey research This study will focus on analyzingquestionnaire construction in ELT survey research as an importance factorcontributing to the values of the studies.

2.2.3 Survey Research Procedure

Survey research procedure is discussed thoroughly in many books and projects on

research methodology, such as Brown (2000), Mertens (2005) and Cohen et al., (2007) Among them, Mertens (2005) and Cohen et al., (2007) seem to organize and

express ideas in the easiest way for junior practitioner in education research tounderstand

In this part, a comparison between two explanations of those two books is presented

to draw out the prominent steps of conducting a survey study

Mertens (2005, pp 166-212) explains thoroughly survey research procedure

Chapter 6 in the book Research and Evaluation in Education and Psychology Integrating diversity with Quantitative, Qualitative and Mixed Methods presents

four steps of conducting a survey research study, which are briefly illustrated inTable 2.2 as follows:

Table 2.2: Steps of Conducting a Survey Research Study

(based on Mertens, 2005, pp 166-212)

1 Design phase Determination of purposes, objectives and types of information

needed for the surveyConsideration of design options for the survey

2 Sampling plan Identification of the respondents, population definition and

Also in the discussion of survey research procedure, Cohen et al., (2007, p 210)

present a detailed diagram illustrating the stages in the planning of a survey:

Figure 2: Stages in the Planning of a Survey (cited in Cohen et al., 2007, p 210)

In comparison between the steps summarized by Mertens (2005) and Cohen et al.,

(2007), it can also be noted that the summary by Mertens (2005) is much simpler

with fewer steps than that of Cohen et al., (2007) However, in fact, the procedure

provided by Mertens (2005) is not simple One explanation is that Mertens (2005)groups numerous smaller steps into a bigger one with detailed explanation in latter

parts of the chapter and the book, which is similar to the explanation by Cohen et al., (2007) Therefore, it can be concluded that two books share similar ideas about

the procedure of conducting a survey study

The equivalence between two ways of description is summarized as follows:

Table 2.3: Equivalence between Survey Research Procedures

provided by Cohen et al., (2007) and Mertens (2005)

Design phase

Conducting the survey

Samplingplan

Designing thequestionnaire

2.2.4 Data Collection Instruments in Survey Research

As illustrated by Brown and Rodgers (2002, p.11), data collection instruments insurvey research are interviews and questionnaires There are also other instrumentsused in survey research, which are stated clearly in the 5th edition of Research Methods in Education by Cohen et al., (2000, p 171) Other instruments are

standardized tests and attitude scales However, interviews and questionnairesusually receive more attention in survey research, especially in the context ofFaculty of English Language Teacher Education, University of Languages andInternational Studies

Brown (2001, pp 5-7) explains in more details the characteristics and the roles ofeach type According to this author, interviews are useful in collecting oral dataabout the interested topic, but this instrument can be time-consuming and demandsthe tactics of questioning to handle effectively However, questionnaires prove theiredge over interviews in the issues requiring a large number of informants

2.3 Questionnaires in Survey Research

2.3.1 Definitions of a Questionnaire

“Questionnaire” is not an uncommon term in the modern world today It exists innumerous situations and in most social fields of study Questionnaires have beenconsidered an effective information gathering tool for researchers to serve theirwork

Researchers have pointed out some definitions of this term In Questionnaires in Second Language Research – Construction, Administration, and Processing,

Dornyei (2003, p.5) presents again the clarification of Aiken (1997) on different

names of “Questionnaire” This term is sometimes called “inventories, forms, opinionnaires, tests, batteries, checklists, scales, surveys, schedules, studies, profiles, indexes/indicators” or “sheets” Despite various names questionnaires

might be called, Dornyei focuses on the sense of “employed as research instrumentsfor measurement purposes to collect reliable and valid data” (p.6)

A “questionnaire”, as defined in Longman Dictionary of Language Teaching and Applied Linguistics (Richards et al., 1992, p 303), is “a set of questions on a topic

or group of topics designed to be answered by a respondent” The authors of thisbook also emphasize that validity, reliability and unambiguity should be paid muchintention in constructing questionnaires Other linguists such as Seliger andShohamy (1989, p 172), Brown (2001, p 6) and Denscombe (2010, pp 155-6)

have similar ideas For example, in Using Surveys in Language Programs, Brown

(2001) explains this term as “any written instruments that present respondents with

a series of questions or statements to which they are to react either by writing outtheir answers or selecting from among existing answers” (p 6)

Although there might be some other ways of defining the term, all authors agree ontwo characteristics of a questionnaire Firstly, a questionnaire consists of variousquestions on a particular interested field Secondly, questions, whichever type theybelong to, are systematically designed to serve the research purpose

2.3.2 Types of Data Collected from Questionnaires

Although questionnaires are effective and advantageous, it is not possible forquestionnaires to be utilized in every circumstance According to Dornyei (2003, p.8) questionnaires can collect three types of information, namely factual, behavioraland attitudinal information

The first type of data, factual information, reveals background information ofrespondents “that may be relevant to interpreting the findings of the survey”.Typical examples of factual information in second language research are “learners’language learning history, amount of time spent in an L2 environment, level ofparents L2 proficiency, or the L2 coursebook used.” The second type is behavioralinformation, aiming to reveal “actions, life-styles, habits and personal history”

Lastly, attitudinal information is about respondents’ thoughts, including “attitudes,opinions, beliefs, interests and values”.

2.3.3 Advantages and Disadvantages of Using Questionnaires

The popularity of the questionnaire in research leads to a multitude books andjournals discussing the instrument’s characteristics, especially with regards to thepositive and negative attributes Prominent authors discussing this issue are Dornyei(2003), Gillham (2000) and Denscombe (2010)

Dornyei (2003, pp 9-14) summarizes the advantages and disadvantages of usingquestionnaires, which is presented briefly in Table 2.4 as follows:

Table 2.4: Advantages and Disadvantages of Using Questionnaires

(Dornyei, 2003, pp 9-14)

Ability of generalization while still:

- Saving researchers’ time

- Saving researchers’ effort

- Reducing the cost of research

- Simplicity and superficiality of answers

- Unreliable and unmotivated respondents

- Little or no opportunity to correct the respondents’ mistakes

- Social desirability

- Self-deception

- Acquiescence bias

- Halo effectAccording to Dornyei (2003, pp 9-14), questionnaires prove their efficiency in

three main aspects, namely researcher time, researcher effort and financial resources Perhaps these three aspects are the main reasons for the wide use of

questionnaires in student research activity and in graduation papers at FELTE,ULIS

However, Dornyei also points out nine serious limitations of questionnaires Thosenine limitations, which are presented in Table 2.4, refer to three main groups ofdrawbacks The first one concerns the depth of investigation, which is also the firstamong those nine limitations Questionnaires may limit “the depth of theinvestigation” (p 10) due to the short amount of time respondents are willing tospend on questionnaires Three other drawbacks concern the quality of answersaffected by questionnaire design Respondents may misunderstand the real meaning

of the questionnaire items; therefore, they make mistakes However, there is little or

no opportunity to correct these mistakes and the quality of the research is affected.The rest four problems concern with respondents’ psychology factors, namely

unreliable and unmotivated respondents, social desirability bias, self deception, acquiescence bias and halo effect In short, respondents, due to psychology reasons,

may not provide true information during the completion of questionnaires

The question follows Table 2.4 is why questionnaires still remain to be one of themost popular data collection instruments when the number of disadvantages exceedits advantages

In fact, it is easy to misunderstand the point of Dornyei (2003) and careless readersmay think of the questionnaire popularity as a contradiction if they only look intothe number of advantages and disadvantages listed by Dornyei (2003, pp 9-14).However, it should be noted that the three advantages refer to the powerful ability

of generalization of this data collection instrument Hardly can any other instrumentdeal with large-scale research population as effectively as questionnaires.Furthermore, those disadvantages stated above can be restricted and eliminatedfrom the high-quality construction of a particular questionnaire

The advantages of questionnaires are confirmed by numerous other research studiesand books Questionnaires are hightlighted as “a suitable method when it is intended

to collect some specific information from a large number of people” (Manual forEducational Media Researchers, 2006) The powerful ability of generalization is

also emphasized by Gillham (2000) and Denscombe (2010) Noticeably, Gillham

(2000, p 14) suggests that the questionnaire disadvantages can be seen as “a list of pointers to producing a good questionnaire” and “as with many other things in life,

a successful outcome is a lot to do with trouble taken in preparation”

Therefore, it is essential to dig into questionnaire construction carefully to make themost use of this data collection instrument

2.4 Questionnaire Construction

2.4.1 Questionnaire Construction Procedure

Questionnaire construction, used in this study, is the term to describe the design ofquestionnaire form for it to serve as the main (or only) method in ELT projects at

ULIS As in Survey Questionnaire Construction, Martin (2006, p 3) defines that

“[C]onstructing a questionnaire involves many decisions about the wording andordering of questions, selection and wording of response categories, formatting andmode of administration of the questionnaire, and introducing and explaining thesurvey.”

Questionnaire construction can be complicated as regards the elimination of thedisadvantages and enhancement of the advantages of the data collection instrument.The quality criteria of a well constructed questionnaire will be discussed in the laterpart of this chapter Initially, the steps of constructing a good questionnaire should

be understood carefully as they are the prerequisite for a questionnaire to be aneffective data collection instrument

A questionnaire can be conducted by adopting and adapting from the questionnaires

by other researchers, or by self-development of the author (Mertens, 2005, p 179)

As for questionnaires of self-development, Dornyei (2003, pp 16-7) discussesthoroughly the steps the process of constructing a good questionnaire, which can besummarized in Table 2.5 as follows:

Table 2.5: The Process of Constructing a Good Questionnaire

(Dornyei, 2003, pp 16-7)

1 General features’ decision

making

Deciding on the general features of thequestionnaire, such as the length, the format,and the main parts

2 Items writing Writing effective items/questions and drawing

up an item pool

3 Items selecting and

organizing

Selecting and sequencing the items

4 Instructions writing Writing appropriate instructions and examples

5 Piloting Piloting the questionnaire and conducting

item analysisWhile the steps 1 to 4 are quite easy to understand, step 5 might be misunderstood

by the junior research practitioners The pilot procedure involves the response ofsome participants but it should not be confused with the questionnaireadministration procedure Piloting in its basic meaning is the process of “trial runs”

to allow the researcher to “collect feedback about how the instrument works andwhether it performs the job it has been designed for” (Dornyei, 2003, p 63)

The pilot procedure can be conducted at the point of item pool completion and atthe point which the almost-final-version of the questionnaire has been prepared(Dornyei, 2003, pp 63-8) The participants of the questionnaire pilot procedure can

be divided into three groups The first group should not be “specialists” in the field,who can help the questionnaire constructor with the jargon misunderstandingelimination, the second one involves the professionals in survey research while thethird one knows “the target population well” (Dornyei, 2003, p 66)

2.4.2 Quality Criteria for Constructing a Questionnaire

The term “Quality criteria” is referred to as one among “basic research issues” thatcan influence researcher’s “methodological decisions” (Dornyei, 2007, p 48).Dornyei even ranks this issue the first in the order of importance, because “we can

only claim that our investigation is indeed a disciplined inquiry if we can set

explicit quality standards to achieve” (p 48)

In the investigation of questionnaire construction, quality criteria often involve thequality of the layout and quality of the content of the examined questionnaire

2.4.2.1 Quality Criteria for the Main Parts of the Questionnaire

Mechanically speaking, a questionnaire should include five main parts which arepresented in the following table according to Dornyei (2003, pp 25-30)

Table 2.6: The Main Parts of a Questionnaire (Dornyei, 2003, pp 25-30)

1 Title Be informative in terms of the domain of the investigation, the initial

orientation

Be able to activate various content schemata

2 Instructions

2.1 General

instruction Be able to cover those following points:1 What the study is about and why it is important

2 The organization responsible for conducting the study

3 Emphasizing that there are no right or wrong answer;requesting honest answers and trying to elicit integrity ingeneral

2.4.2.2 Quality Criteria for the Content of the Questionnaire

Quality criteria for the content of the questionnaire refers to its reliability and validity In fact, these two criteria are often listed out and discussed thoroughly in

numerous books for research methodogy, without under the classification of

“quality criteria”, such as in Brown (2000), Cohen et al., (2007) and Denscombe

(2010) However, reliability and validity always appear when discussing about thevalues of a research project or a means of measurement Therefore, it is accepted toexamine questionnaires’ validity and reliability in the process of evaluatingquestionnaire construction

As stated above, validity, reliability and unambiguity should be paid much intention

in constructing questionnaires (Richards et al., 1992, p 303) These three issues are

also considered the quality criteria for constructing a questionnaire However,unambiguity, which refers to the appropriate wordings of items, can be merged into

the meaning of “reliability”; therefore, only two criteria are considered: reliability and validity

Those two quality criteria for questionnaire construction in particular and forresearch design in general are explained quite thoroughly in books such as inFowler (1995), Dornyei (2007) and Bordens and Abbott (2011) Among those three,Bordens and Abbott (2011) seem to explain reliability and validity of aquestionnaire in the easiest way for junior research practitioners to understand

Bordens and Abbott (2011, pp 274-7) define reliability as “the ability of a measure

to produce the same or highly similar results on repeated administrations” (p 274).Therefore, a questionnaire is reliable if the responses are similar on testing andretesting In other words, reliability refers to the consistency of the results collectedfrom the questionnaire

Reliability of a questionnaire can be enhanced through the number of items, throughthe standardized administration procedures and through the layouts of thequestionnaire (Bordens and Abbott, 2011, p 276)

Validity of a measure in general, and of a questionnaire in particular, is provedthrough its usefulness, specifically speaking, “it must measure what it is intended tomeasure” (Bordens and Abbott, 2011, p 276)

There have been numerous kinds of validity, but for the questionnaire, the kinds of

validity often discussed are content validity and construct validity The table below

depicts the characteristics of each type applied to according to Bordens and Abbott(2011, pp 276-7)

Table 2.7: Types of Validity

Types of validity Application to the Questionnaire

Content validity The questionnaire should include items relevant to all the

major issues relating to the topic of the research

Construct validity The questionnaire’s results agree with the predictions based

on theory

From those above descriptions, the differences between reliability and validity can

be recognized easily In short, reliability refers to the consistency of the results andvalidity concerns with whether the data collected can answer the research questions

or not

2.4.3 Criteria for a Well-Constructed Questionnaire

As stated above, reliability and validity are two main quality criteria for the content

of a questionnaire Nonetheless, Bordens and Abbott (2011, p 276) states that thelayouts of the questionnaire can also enhance its reliability Therefore, in theestablishment of the systematic criteria of a well-constructed questionnaire, thecriterion of “layout” is merged into the criterion of “reliability”

In short, with the focus on the construction process of a questionnaire, the quality of

a well-constructed questionnaire is mainly assessed through reliability and validity.

Part 2.4.2 already presents the characteristics of reliability and validity of aquestionnaire Briefly speaking, reliability can be assessed through the number ofitems, the administration procedure and the layouts (Bordens and Abbott, 2010) andthrough the source of the questionnaire and its pilot procedure (Dornyei, 2003).However, the number of items is rather vague and somehow relates to the criteria oflayouts Furthermore, the evaluation of administration procedure can be rathercomplicated especially in regards to its quite simple description in the surveyprojects of this research context Thus, only the source, the pilot procedure and thelayouts are used in the establishment of quality criteria for the reliability of aquestionnaire in this research.

In terms of validity, the quality criteria focus on content and construct validity of aquestionnaire according to Bordens and Abbott (2011, pp 276-7)

To be specific, Table 2.8 and Table 2.9 below present the criteria of a constructed questionnaire adapted from Borden and Abbott (2011) and Dornyei(2007)

well-Table 2.8: Criteria to Evaluate the Reliability of a Questionnaire

Source - How is the questionnaire designed?

Questionnaire

Layout

- Five main parts with appropriate quality of each part

- Length: no more than 4 pages long

- Booklet format

- Economical use of space with full but not overcrowded pages

- An orderly layout that utilizes various typefaces andhighlighting options, and appropriate sequence marking

- Order of the items in the questionnaire: Personal/ factualquestions about the respondent should go to the end

- Instructions: Clear for respondents

Pilot procedure - Is the questionnaire piloted in a systematic manner?

Table 2.9: Criteria to Evaluate the Validity of a Questionnaire

Content validity Does the questionnaire include items relevant to all the major

issues relating to the topic of the research?

Construct validity Can data collected from questionnaires answer the research

Therefore, the research “Questionnaire Construction in ELT Survey Research at FELTE, ULIS: An evaluation study” was conducted as regards the graduation

theses at FELTE, ULIS, VNU with the close investigation of the questionnaireconstruction in those survey research projects

2.6 Summary

Chapter 2 presents the review of relevant literature, including survey research inELT and the construction procedure of questionnaires as the data collectioninstrument of survey research Particularly, this chapter investigates thoroughly thequality criteria of a well constructed questionnaire, setting a firm base for datacollection and analysis in later parts of this evaluation research

CHAPTER 3: METHODOLOGY

Chapter 3 is the description of the research context and the design of this evaluation project The detailed explanation of the sampling strategy, sampling procedure, data collection procedure and data analysis are also provided in this chapter.

3.1 Research Context

Research is one major activity in the teaching and learning process at Faculty ofEnglish Teacher Education (FELTE), University of Languages and InternationalStudies (ULIS), Vietnam National University (VNU), Hanoi The student researchactivity is held annually, calling for a large number of projects in various fields,especially in English Language Teaching (ELT) More importantly, research is used

as an activity for assessment of senior students, in the form of thesis Annually,dozens of projects are conducted by students, many of whom choose the surveyapproach and employ questionnaires as their research instrument In fact, it isevident that the questionnaire is generally considered one of the most popularresearch methods utilized by the student researchers

Nonetheless, it seems that a considerable number of student practitioners do not payenough attention to questionnaire construction, especially in terms of validity andreliability The help of computers and the Internet have led to the ease of

questionnaire design, as Dornyei (2003, p.3) says: “it is possible to draw up something that looks respectable in a few hours” However, “good research cannot

be built on poorly collected data” (Gillham, 2000, p.1), and well constructedquestionnaires will eliminate this risk In other words, “developing a questionnairethat will yield worthwhile data” (Gillham, 2000, p.1) is the prerequisite for valuablefindings in research

Meanwhile, the scarcity of research projects on the issues of questionnaireconstruction at FELTE, ULIS is evident Student researchers tend to experience ademand for the studies reviewing and evaluating the current situation of

questionnaire construction Those studies may serve as a source of reference for theenhancement in the quality of the questionnaires employed in their researches aswell as the quality of their projects as a whole.

Therefore, the research “Questionnaire Construction in ELT Survey Research at FELTE, ULIS: An evaluation study” was conducted as regards the graduation

theses at FELTE, ULIS, VNU with the close investigation of the questionnaireconstruction in those survey research projects

3.2 Research Question

The study was conducted with the hope to analyze the questionnaire construction ofthe questionnaires in ELT research and evaluate their validity and reliability.Through the investigation into the utilized questionnaires, which are the products ofthe questionnaire construction process, this project aimed to present criticalevaluation on the questionnaire construction of the previous theses, especially inregards to its relevance to the criteria for a well-constructed questionnaire

To be specific, the research project aimed at answering the following question:

To what extent do the questionnaires utilized in the ELT survey research projects at FELTE, ULIS satisfy the quality criteria of questionnaire construction?

3.3 Research Approach: Evaluation Research

3.3.1 Evaluation Research

The research was conducted with the aim to review and evaluate the questionnaireconstruction in the previous ELT theses of FELTE, ULIS, VNU Basing on thataim, the researcher chose the design of evaluation research to conduct this project.The evaluation approach is often preferable with regards to assessing and evaluating

a certain process or product According to Scriven (1991) (as cited in Mertens,

2005, p 47), the overall idea of evaluation is “determining the merit, worth or value

of something, or the product of that process” To achieve that aims, evaluation,especially in an educational context, “should be systematic and undertakenaccording to certain guiding principles using carefully defined criteria” (Rea-Dickins and Germaine, 1992, p 5)

In general, the evaluation approach is most commonly used for education programassessment and development It should be noted that the education program dependssignificantly on research as valuable research projects on a particular educationprogram can lead to the appropriate adjustment and development of that program.Also, the quality of the research projects result mainly from the quality of theresearch instruments and the data collected from those instruments Therefore,evaluating the construction of those instruments is one important and inevitable part

of evaluating an education program

As regards the research context, as mentioned in part 3.1, questionnaires are used innumerous research studies at FELTE, ULIS and the data collected from thosequestionnaires are used as the main source to investigate the current attitudes andsituations of the Faculty’s language learning and teaching process

Thus, evaluation approach is suitable for the author to develop a framework forquality criteria of questionnaire construction and examine their effectiveness in thecontext of FELTE, ULIS

3.3.2 Evaluation Research Procedure

The evaluation steps are discussed in details by Mertens (2005, pp 60-73) with thesummary and critical review of other works such as Madison (1992), Mertens and

McLaughlin (2004) and Shadish et al., (1991)

Overall, there are three main steps for conducting an evaluation study (Merterns,

2005, pp 60-73), namely Focussing the evaluation, Planning the evaluation and

Implementing the evaluation Following those steps, the evaluation research on the questionnaire construction in ELT survey research at FELTE, ULIS, VNU was

conducted and the detailed step description was presented in Table 3.1 as follows:

Table 3.1: Evaluation Research Procedure

- The purpose of the evaluation

- The stakeholders in the evaluation

- Constraints affecting the evaluation

- The evaluation questions

- Selection of an evaluation model

- Making the decision on the evaluands, thepurposes and the stakeholders of the study

- Analysing the research context at FELTE, ULIS toidentify the constraints and design the detailedplan to conduct the research

- Identifying the evaluation question: To what extent do the questionnaires in the ELT survey research at FELTE, ULIS satisfy the quality criteria of questionnaire construction?

- Establishing the criteria for evaluation

2 Planning the

evaluation

- Data collection specification, analysis,interpretation and used strategies

- Management of the evaluation

- Making decisions on the sampling strategies, thesampling plan and the data collection procedure

- Collecting data and analyzing data basing on theevaluation criteria

3.4 Sampling

3.4.1 Evaluands

The target population, or the evaluands, of this evaluation research was thequestionnaires in the previous ELT survey research From examining theconstruction of those questionnaires, the researcher wanted to draw conclusions onthose authors’ concerns when they developed questionnaires to collect data for theirresearch projects

It should be noted that ELT survey research investigated in this evaluation project

mainly involved the studies of attitudinal issues, because those projects require the

main employment of the questionnaire as their data collection instrument to yieldvalid and reliable data

3.4.2 Sampling strategy: Criterion Sampling

Criterion sampling is a subtype of “Purposeful Sampling” in the categories provided

by Mertens (2005, pp 307-24) This strategy means the process of establishing a set

of criteria and then identifying cases that meet those criteria

This strategy is suitable for this evaluation study as it enables the researcher tochoose samples purposefully in the way that fits the aims the study

Criteria used to choose samples for analysis in this study are presented in Table 10

as follows:

Table 3.2: Criteria for Sampling

1 Field of

Study:

ELT

In the University Library, these theses are under the category of

“GHP”, which means that the appropriate graduation papers will have codes in the form of “GHP + number”.

Only research projects about issues relating toEnglish language learning and teaching will beconsidered in this evaluation research

2 Type of

Research:

Survey

Research

2.1 Scope of the studies

The research projects taken into consideration of analysismust have the following characteristics:

- Concerning with issues of respondents’ thoughts,

including attitudes, opinions, beliefs, interests and values.

- Containing such words as “survey, opinions, beliefs, interests, preferences, motivation, attitudes, difficulties, perceptions” in their titles.

2.1 Scope of the studies

Criterion 2.1 was employed to choose sample on thebasis of a research project’ title

2.3 Research instruments

The graduation papers investigated in this evaluationresearch must use questionnaires as the main data collectioninstrument

Methodology was first introduced to the third yearstudents at FELTE, ULIS and those studentsconducted their graduation papers in 2007

3.4.3 Sampling Procedure and Results

The sampling procedure was conducted according to the chosen sampling strategystated above with the relevance to the criteria presented in part 3.4.2 The detailedprocedure is presented in Table 3.3 as follows:

Table 3.3: The Sampling Procedure and Results

Steps Number of Satisfied Research Projects

to exist here in the overall criterion sampling strategy

3.4.4 Description of the subjects of the study.

Table 3.4 below presents the information of seven chosen projects from 2007 to 2012

Table 3.4: Subjects of the Study

Instruments

1 A study of students’

attitudes towards oral

errors and error correction

in speaking class

1 What are students’ attitudes towards oral errors?

2 What are student’s attitudes towards teacher

correction and peer correction?

52 students - Questionnaire A for

52 students

2 A research on the

difficulties in learning

listening and speaking

skills by second year

1 What are the difficulties of second year student in

learning English listening skills?

2 What problems do they have in learning English

speaking skills?

3 What should be done to help second year students

to overcome their problems in learning listeningand speaking skills?

40 students - Questionnaire B for

80 students

- Interview with 5 students

- Observation of 4 classes

continued

Table 3.4: Subjects of the Study (continued)

Instruments

3 Beliefs about language

learning held by first year

students and their teachers at

English department, HULIS

– VNU

1 What are the beliefs about language learning held by

first year students at English Department, HULIS –VNU

2 What are the beliefs about language learning held by

these students’ instructors?

3 What are the major differences between language

learning beliefs of the two groups?

teachers and students at

FELTE, HULIS, VNU in

exploiting the Internet to

teach and learn English

1 What are the difficulties experienced by teachers and

students at FELTE, HULIS, VNU in exploiting theInternet to teach and learn English?

2 What are solutions used or suggested by the teachers

and students at FELTE, HULIS, VNU to solve thosedifficulties?

Table 3.4: Subjects of the Study (continued)

Instruments

5 Students’ perception factors

affecting English learning

2 How has the motivation of the students changed after

a period of time studying English at ULIS, VNU?

3 As perceived by students, what are the factors

affecting their motivation that lead to those changes?

148 students - Questionnaire E for

148 students

- Semi-structured interviews with 10 students

6 Teachers’ difficulties in

using information-gap

activities to teach speaking

skill for the first year

students at ULIS – VNU

1 What are the factors affecting the choices of

information-gap activities to teach speaking skill as perceived by the teachers who are teaching speaking

skill for first year students?

2 What are the difficulties in selecting and conducting information-gap activities to teach speaking skill as perceived by the teachers who are teaching speaking

skill for first year students?

3 What are the solutions to the problems of the teachers

in conducting IGA to teach speaking skill as perceived by the teachers who are teaching speaking

skill for first year students?

15 teachers - Questionnaire F for

15 teachers

- Interview with 2 teachers

- Classroom observation (1 class)