an evaluation of learners attitudes towards the effectiveness of the online b1 level english learning program on website www tienganhb1 com

DECLARATION I certify my authority of the study submitted entitled AN EVALUATION OF LEARNERS’ ATTITUDES TOWARDS THE EFFECTIVENESS OF THE ONLINE B1 LEVEL ENGLISH LEARNING PROGRAM ON WEBS

VIETNAM NATIONAL UNIVERSITY, HA NOI UNIVERSITY OF LANGUAGES AND INTERNATIONAL STUDIES

FACULTY OF POST - GRADUATE STUDIES

M.A MINOR PROGRAMME THESIS

Field: English Teaching Methodology Code: 60140111

Ha Noi, 2014

VIETNAM NATIONAL UNIVERSITY, HA NOI UNIVERSITY OF LANGUAGES AND INTERNATIONAL STUDIES

FACULTY OF POST - GRADUATE STUDIES

M.A MINOR PROGRAMME THESIS

Field: English Teaching Methodology Code: 60140111

Supervisor: Đỗ Tuấn Minh, Ph.D

DECLARATION

I certify my authority of the study submitted entitled

AN EVALUATION OF LEARNERS’ ATTITUDES TOWARDS THE EFFECTIVENESS OF THE ONLINE B1 LEVEL ENGLISH LEARNING

PROGRAM ON WEBSITE WWW.TIENGANHB1.COM

in fulfillment of the partial requirements for the Degree of Master of Arts at University of Languages and International Studies, Vietnam National University- Hanoi

Hanoi, 2014

Vũ Thị Lê Vy

ACKNOWLEDGEMENTS

To complete this paper, I am deeply indebted to many people for their

assistance and encouragement during the conduct of my study

First and foremost, I would like to express my deepest thanks and gratitude

to my supervisor – Dr Do Tuan Minh – for his continual support, enlightening guidance, valuable advice and considerable encouragement, which are decisive factors in the completion of this paper

Besides, I owe a special debt of gratitude to Assoc Prof Le Hung Tien and M.A Nguyen Quynh Trang for their valuable feedback on my portfolios for research methodology subject, which guided me and laid the foundation for my study

I deeply thank all VIP members on the website http://tienganhb1.com/ for their enthusiasm and helpfulness in participating in my study

My sincere thanks also go to my family and friends who gave me unconditional support and encouragement without which my thesis would not have been accomplished

Last but not least, I am especially indebted to the authors whose materials have been used in the study

ABSTRACT

The concept of online learning or e-learning has become increasingly popular However, studies on the evaluation of online learning still remain in infancy, especially in Vietnam Among the very few studies on this topic in Vietnam, this thesis investigates the effectiveness of the online B1 level English learning program on website http://tienganhb1.com/ (the OLP) from the perspectives of VIP members of the website who are preparing for their B1 test in post-graduate format Specifically, the study seeks to measure the level of satisfaction of the targeted participants and their suggestions for improving the OLP In order to achieve these aims, a model of evaluation was designed based on the combination of Brill & Park’s (2011) and Bui Thi Thu Giang’s (2012) models The data were collected by means of questionnaires surveying participants’ attitudes towards the OLP regarding the instructional design, user-friendliness, support and cost-effectiveness; their difficulties encountered while taking the course and suggestions for change Interviews were conducted after that in order to obtain insightful information concerning learners’ difficulties and recommendations Results show that most learners were satisfied with the design of content, level of support provided by the website administrators and teachers and that the website is generally user-friendly and cost-effective Suggestions were given to improve all four skills and all areas of instructional design, user-friendliness and level of support Among these, improving the design of content mostly involved adding up new speaking topics or full B1 tests or making the OLP more motivating with videos or supplementing translation in speaking and writing to improve comprehensibility

TABLE OF CONTENTS

Certification of authority of the study i

Acknowledgements ii

Abstract iii

List of tables and abbreviations vi

PART A: INTRODUCTION 1

1 Rationale of the study 1

2 Aims and research questions of the study 2

3 Scope of the study 2

4 Methods of the study 3

5 Significance of the study 3

6 Overview of the rest of the study 3

PART B: DEVELOPMENT 5

CHAPTER 1: THEORETICAL BACKGROUND 5

1.1 Introduction 5

1.2 B1 level of English 5

1.3 Online learning 5

1.3.1 Definitions of online learning 5

1.3.2 Reasons for online learning 7

1.3.3 Challenges in online learning 8

1.4 Evaluation of online learning program 8

1.4.1 The importance of evaluation 8

1.4.2 Types of evaluation 9

1.4.3 Models of evaluation 9

1.5 Previous studies on online learning in Vietnamese context 14

2.2 Description of the context and participants 16

2.2.1 The context 16

2.2.2 The participants 18

2.3 Data collection method 19

2.3.1 Data collection instruments 19

2.3.2 Data collection procedure 21

2.4 Data analysis method and procedure 22

2.5 Summary 22

CHAPTER 3: RESULTS AND DISCUSSION 23

3.1 Introduction 23

3.2 Research question 1: To what extent are the learners satisfied with the online B1 level English learning program on website http://tienganhb1.com/? 23

3.2 Research question 2: What are the learners’ suggestions to maximize the effectiveness of the online B1 level English learning program on website http://tienganhb1.com/? 28

3.3 Summary 37

PART C: CONCLUSION 38

1 Summary of the results 38

2 Limitations and suggestions for further studies 40

BIBLIOGRAPHY 41

APPENDIX 1 I

APPENDIX 2 III

APPENDIX 3 VI

APPENDIX 4 VII

LIST OF TABLES Page

Figure 1: Spectrum of evaluation perspectives 10

Figure 2: Conceptual framework applied to analysis of PBJ tutorial evaluation

13

Figure 3: Homepage of the website http://tienganhb1.com/ 17

Figure 4: The OLP on http://tienganhb1.com/ 17

Figure 5: The questionnaire uploaded on the website http://tienganhb1.com/ 20

Figure 6: Sample respondent’s details for the interviews 21

Table 1: Perspectives on evaluation for e-learning 11

Table 2: Criteria for the evaluation of the OLP 15

Table 3: Learner satisfaction in terms of instructional design 23

Table 4: Learner satisfaction in terms of user-friendliness 24

Table 5: Learner satisfaction in terms of support and cost-effectiveness 26

Chart 1: The importance of the OLP 27

Chart 2: Overall evaluation of the OLP 28

PART A: INTRODUCTION

1 Rationale of the study

Enormous advances in information and technology and the increasing demand for new methods of knowledge delivery have led to the rapid growth in the application

of web-based training or e-learning Online learning is considered as “an excellent solution for the old problem of mass education, beyond that of an impractical apprenticeship method, since there are far too many knowledge seekers and not enough knowledge providers” (Penalvo, 2008) Penalvo also emphasizes that e-learning is possible anytime and anywhere Additionally, online courses are at lower costs than face to face ones (Petrova and Sinclair, as cited in Penalvo, 2008) Yang & Yuen (2010) stress that the use of computers helps to “support teaching and learning for educators and improve quality and excellence in education” (p 2) Due to numerous benefits of online learning, many educators have applied it in their teaching E-learning has been widespread in recent years However, studies in the evaluation of e-learning are still limited in number So far very little research has dealt with e-learning evaluation in Vietnam; therefore, it leaves many gaps for those who are interested in this topic to investigate This study aims at filling this gap

Since the Circular No 05 /2012/TT- BGDĐT dated 15 February, 2012 by

MOET came into effect, there has been a very high demand for learning resources to prepare graduate learners for B1 English test There have been several English websites worldwide offering preparation courses for the B1 test known as PET such as http://www.examenglish.com/PET/, http://www.englishaula.com/, http://www.english-online.org.uk/, www.Tienganhb1.com Most of these websites have guidelines for each part of these exams including practice tests However, none of these provide extensive courses for B1 tests designed for Vietnamese learners, except for www.Tienganhb1.com The website www.Tienganhb1.com was designed and officially launched in June, 2013 in order to meet the increasing need for B1 learning materials in Vietnamese context It has a wide range of online materials for B1 test in Vietnam Since its introduction, there have been a large number of members

registering for the website Many of them applied to be VIP members with full access

to the online B1 level English learning program on website http://tienganhb1.com/ (the OLP) However, there has been no research conducted in order to evaluate how effective this program is to the learners Therefore, the researcher is particularly interested in seeking whether the program is effective to its users and there are any suggestions to improve it

In short, key motivation for the researcher to conduct this study is the need to evaluate available online resources for B1 test preparation and to examine unanswered questions left in the field

2 Aims and research questions of the study

The first objective set out in this study is to investigate the effectiveness of the OLP according to the level of satisfaction of VIP members, mainly using the theories proposed by Brill & Park (2011) and Bui Thi Thu Giang (2012) as methods of evaluation Next, the study purports to explore learners’ suggestions to improve the educational outcomes of the OLP

In short, the study aims to answer the following research questions:

1 To what extent are the learners satisfied with the online B1 level English learning program on website http://tienganhb1.com/?

2 What are the learners’ suggestions to maximize the effectiveness

of the online B1 level English learning program on website http://tienganhb1.com/?

3 Scope of the study

The study focuses on evaluating the effectiveness of the OLP from the learners’

This work also stresses the evaluation provided by participants who are going to

take B1 test in the post graduate format as specified in the Circular No 05 /2012/TT-

BGDĐT dated 15 February, 2012 by MOET This means that participants are limited

to VIP members as MA or prospective PhD students who are going to take the B1 test under post graduate format pursuant to the above Circular All other VIP members who are undergraduate students or English teachers are beyond the scope of this study

4 Methods of the study

The study employed both qualitative and quantitative methods The data were gathered by using questionnaires and interviews First, questionnaires were administered to selected participants online after they logged into the system Survey questionnaires were designed based on the conceptual framework Survey questionnaires consisted of close-ended question to draw findings to the first questions concerning learners’ satisfaction of the OLP and open-ended questions to capture respondents’ difficulties encountered while taking the course and suggestions to improve it Next, semi-structured interviews were adopted to address the second research question for deeper information

5 Significance of the study

As one of the very few studies conducted on the evaluation of e-learning in Vietnam, this research is supposed to shed some light on the field, especially on suggesting criteria for online course evaluation as stated in the rationale Accordingly, the study is hoped to encourage more researchers to investigate the topic Also, the study is expected to explore learners’ feedbacks and suggestions for the OLP Therefore, with the findings of the study, the researcher wishes to propose changes to maximize the effectiveness of the OLP

6 Overview of the rest of the study

The rest of the study is comprised of four chapters which are as follows

Chapter two, Theoretical Background, reviews prominent literature directly relevant to the topic being discussed

Chapter three, Methodology, presents the methods used for data collection and data analysis as well as justification for these methods

Chapter three is followed by chapter four, Results and Discussion, in which the findings of the study are discussed to answer the research questions

Finally in chapter five, recapitulation of the whole study is presented before suggestions for further studies and a conclusion are given

PART B: DEVELOPMENT CHAPTER 1: THEORETICAL BACKGROUND 1.1 Introduction

This chapter provides definitions and background of the theories and concepts related to the topic investigated in this research paper Specifically, the researcher will discuss the concepts of B1 level of English, online learning and evaluation of an online learning program

the Circular No 05 /2012/TT- BGDĐT dated 15 February, 2012 by MOET for those

who are completing MA courses or submitting doctorate proposals (seen in the Appendix 4)

1.3 Online learning

1.3.1 Definitions of online learning

“Online learning” is defined as the application of Web-based tools for learning purposes (Penalvo, 2008) Other synonymous terms could be virtual learning, distance learning or e-learning The following definition of online learning is provided by Petrova and Sinclair (as cited in Penalvo, 2008):

E-learning is used as a comprehensive term to identify the use of a variety of information and communication technologies to enhance and support learning, often

blending their use Online learning can be defined as an implementation of e-learning using Web-based technologies (Petrova, 2007)

Petrova and Sinclair (as cited in Penalvo, 2008, p 113)

Online learning can also be referred to as computer-assisted language learning (CALL) Ken Beatty (2010) gives a broad definition of CALL as “any process in which a learner uses a computer, and as a result, improves his or her language” It covers a range of activities known as a reward for better learners or a remedial aid for weaker students Materials may comprise purpose-made ones for language learning or those which are adapted from existing computer-based materials, videos Delivery methods might involve online instruction through the WWW, distance and networked learning using emails, blogs, online social networking and other interactive sites and services (Beatty, 2010)

Horton (2011) shares the similar definition of e learning with Penalvo (2008) and Beatty (2010) as “the use of electronic technologies to create learning experiences” Several types of e-learning are standalone courses, learning games and simulations, mobile learning, social learning, virtual classroom courses, the first of which are courses taken by individual learners without interaction with teachers and classmates

According to Bakia et al (2012), “online learning” refers to the Internet-based instructional environment in which instructions are delivered to students through the Internet or interactions are facilitated between teachers and students and among students in some cases “Online learning” can be either fully online or blended learning The former is considered as a form of distance education with all instruction

In short, a variety of terms are used to define online learning, for example, learning, virtual learning, distance learning, web-based learning, and computer-assisted language learning All of these terms share the same meaning, so they will be used interchangeably in this research In this study, online learning includes a range of activities classified into four skills uploaded on the website or web-based activities allowing learners to practice Online learning program is a fully online or standalone course taken by individuals without interaction among classmates

e-1.3.2 Reasons for online learning

With the increasing popularity of the Internet, online learning has many advantages

Firstly, it is flexible It is true that for e-learning, users can access whenever and wherever they like As a result, geographical location is not an issue It might be difficult to have meetings in different time zones (Hockly & Clandfield , 2010, p 10) Bakia et al (2012) also emphasizes that the popularity of online learning is caused by more flexible access and instruction It helps to increase the learning experiences for those unable to attend face-to-face lessons

Secondly, online courses are more cost-saving than traditional lessons (Hockly

& Clandfield , 2010) also consider e-learning courses to be cheap for learners Accordingly, the ratios of students and teachers are raised but learning outcomes remain equal to those of traditional classroom instruction

Finally, online learning provides a more efficient dissemination of instructional content Thanks to the use of multimedia, learners tend to be more interested and motivated (Ravichandran, 2000, p 82 as cited in Mariott & Torres, 2009) Also, the use of Internet caters for learners’ different styles of learning

1.3.3 Challenges in online learning

Although benefits of online learning are numerous, its drawbacks are unavoidable Web-based learning provides learners with flexibility but learners have to

be self-disciplined and self-motivated Loading speed is another problem when bandwidth may cause difficulty in viewing some multimedia Some learners may find working online alienating, or spend too much time already online at work

lower-The literature of online learning so far has addressed the following concerns: the quality of e-learning with learning outcomes, student persistence and student satisfaction There are controversies over the quality of e-learning Some studies state that there are no differences between the learning outcomes of traditional classrooms and e-learning (Russell, 1999; Tran Thanh Dien & Esichaikul, 2007) Another challenge is student persistence in online learning with high dropout rates Feelings of anxiety, isolation and confusion are blamed for dropout rates among students (King, 2002)

1.4 Evaluation of online learning program

1.4.1 The importance of evaluation

Evaluation refers to “assigning a value to something” (Horton, 2001, p 1) It gives the answers to the questions about the value of an e-learning course or project Horton also emphasizes that only evaluation can measure the success of a firm Horton (2001) gives some particular reasons for an evaluation of e-learning as follows:

- Justify investments in training

- Make better decisions about training

- Encourage learning

Various aspects of training can be measured in terms of quality and effectiveness through evaluation such as materials, instructors, facilities, and presentation techniques Horton (2001) stresses that a good evaluation process should

be flexible, reliable, economical and easy to implement

Brill & Park (2011) also emphasizes that “evaluation plays an essential role in almost all instructional development (ID) models and training cycles” In fact, evaluation is central to the quality of training courses Another reason for evaluation lies in the decision making process toward the effectiveness and cost-effectiveness of learning sources Evaluation for development purpose has now been more popular than judgmental purpose

The evaluation of the OLP in this study was conducted to enhance the quality of the OLP so that learning is more motivational and promoted

1.4.2 Types of evaluation

There are two types of evaluation: formative and summative evaluation The former focuses information on program improvement The latter, on the other hand, uses information to make decisions among options or to certify the effectiveness of a program Most evaluations are both formative and summative (Baker & Alkin, 1973; Scriven, 1967 as cited in Yang & Yuen, 2000) The evaluation of the OLP on the researched website adopted both of them The outcomes of the evaluation are expected

to make changes to the existing system for the better At the same time, judgments are made towards the effectiveness of the program

1.4.3 Models of evaluation

Horton (2001) claims evaluation should start with the scope of evaluation from micro to macro breadth of view The micro view concerns with specific results of a particular learning objective whereas the macro one deals with broader results

Figure 1: Spectrum of evaluation perspectives

He suggests that evaluators should consider the scope of evaluation, the economic role in e-learning, timing, internal or external evaluation, the results achieved once the evaluation finishes

Perspective Postitions for

Your evaluation

Why evaluate From this position? Breadth of View

At what scope – micro or

macro – do you evaluate?

□ Before training is developed

□ For an existing training package

□ After training has been conducted

Internal versus External

What characteristics do you

□ Level 4: Business results

Table 1: Perspectives on evaluation for e-learning (Horton, 2001, p 11)

Son (2005) on evaluating websites uses a likert scale to evaluate the selected websites in terms of the purpose, the accuracy of the content, currency or updating of the website, authority, loading speed, usefulness, organization, navigation, reliability, authenticity, interactivity, feedback, multimedia, communication, integration

Torstein Rekkedal (2006) mentions criteria for evaluating the quality of learning programs including the credibility of the institution delivering the online course, the quality assurance or quality management system, pre-enrolment information and guidance on whether or not information is provided sufficiently for learners to decide to take the course, course costs, subject-related, social and technical support for e-learners, individual preferences (whether the e-learning course cater for learners’ needs in terms of structure, communication and learning styles)

e-Hadjerrouit (2010) suggests usability as a criterion for evaluation Usability comprises technical usability and pedagogical usability The former concerns evaluating content design, page design and site design The latter deals with assessment of understandability, added value, goal-orientation, time, interactivity, multimedia, motivation, differentiation, flexibility, autonomy, collaboration, variation

Brill & Park (2011) applied a model of evaluation including three elements, namely the characteristics of the evaluand, the dimensions of evaluation, and the application of results

Figure 2: Conceptual framework applied to analysis of PBJ tutorial evaluation

(Brill & Park, 2011)

Concerning the dimensions of evaluation, Brill & Park (2011) created their framework of three levels (reaction, learning, and transfer) in light of Kirkpatrick’s model Criteria comprise evaluating learning conditions concerning design of content and user-friendliness and evaluating learning

It can be seen that there has been no consensus in the literature regarding criteria for evaluating an online course Among these, however, the model of Brill and Park (2011) seems to be the most comprehensible with clear criteria and guidelines for evaluation

1.5 Previous studies on online learning in Vietnamese context

As has been mentioned, very few studies have been done on evaluating an online program in Vietnamese context One notable work attempting to deal with this phenomenon exhaustively is that by Bui Thi Thu Giang (2012) on the effectiveness of the online learning program blended with the traditional course to second-year non-English major students at Hanoi University of Industry By using such methods as survey questionnaires, interviews, pretests and posttests and online learning reports, she found that both students and teachers appreciated the online learning program Also, the learning reports and the findings from pretests and posttests showed that the online learning program is effective to the students

The other on evaluating an online program conducted in Vietnam that the researcher could find is the “Effective evaluation of online learning environment: a case study of Can Tho University” by Tran Thanh Dien & Esichaikul (2007) The researchers used questionnaires to students and instructors and hypothesis testing By conducting this study, they wanted to compare traditional (face-to-face) and online learning program and the statistics showed that there is a difference in students’ learning achievements However, the comparative evaluation is not even because the results were measured on the same learners on two different subjects Moreover, the research is a case study; therefore, the results cannot be generalized to a larger population

In short, except for the work by Bui Thi Thu Giang (2012) that holds major significance in laying the theoretical foundation for other studies on evaluating an online program in Vietnamese context, papers on this topic in Vietnam have been very limited in number This reality, as clarified in the introduction, has constituted a good reason for the conduct of this study

In short, following criteria will be applied in evaluating the OLP in this study

Cost-Is the cost of the course in accordance with expected results and benefits for the learners?

Table 2: Criteria for the evaluation of the OLP

In the next chapter, details on how this study was conducted will be presented

CHAPTER 2: METHODOLOGY 2.1 Introduction

This chapter gives an overall account of the participants from which the data were drawn, and presents the methods of data collection and data analysis

This chapter consists of three sections In the first section, a general description

of the context and participants is given The second section presents the methods of gathering data and the rationale for adopting these data collection methods, followed

by procedure of data collection Finally, data analysis methods will be described

2.2 Description of the context and participants

2.2.1 The context

www.TiengAnhB1.com comprises B1 test related information, for example, a full description of B1 test for graduate students and a comparison of this and other types of B1 tests, authorized institutions organizing this kind of test and most importantly, an online learning program The test preparation program consists of tests

at all four skills Listening, Speaking, Reading and Writing categorized according to

the post-graduate format specified in the Circular No 05 /2012/TT- BGDĐT dated 15

February, 2012 by MOET (seen in the Appendix 4) required for graduate students before thesis defense and doctorate dissertation submission Materials are mostly compiled from PET resources such as past examinations by Cambridge University Press and other books preparing students for the test Besides, test-like items with sample answers are provided for speaking part following topics regulated by the MOET For Listening and Reading including mostly multiple choice items, learners take the test online and check answers Listening is provided with transcripts to facilitate low-level learners to check their answers The content of the online program

Figure 3: Homepage of the website http://tienganhb1.com/

Figure 4: The OLP on http://tienganhb1.com/

2.2.2 The participants

In a survey research, sampling is of great importance because it has a significant influence on the results of the research Bearing this in mind, the researcher has carefully chosen the participants for the study

Questionnaire respondents

The population of the study was 151 people, filtered from 752 VIP members thanks to the help of the technical team They were active VIPs who needed B1 certificate to complete their MA courses or to enroll for doctorate proposals Simple random sampling was then employed to select 100 participants for the questionnaires including 77 MA students and 23 prospective PHD students All of these lists were completed with the help of the technical team The questionnaire respondents come from different regions in Vietnam and were going to take B1 tests in various institutions in Vietnam, for example, National University Hanoi, Da Nang University, Thai Nguyen University, University of Science and Technology, Hue University, National University of Civil Engineering, Ho Chi Minh University of Education They are in contact with teachers and administrators through website live chat, emails

or phone

Interview respondents

10 VIP members who logged onto the system most were chosen from the population to be participants for phone interviews Respondents come from different institutions in Hanoi such as Hanoi University of Science, University of Social Sciences and Humanities, Electric Power University, Hanoi Law University, University of Forestry and they were going to take B1 test in different authorized institutions to organize B1 test They were selected as interview participants because

The reasons for adopting the methods of data collection including questionnaires and interviews are indicated in the following section

2.3 Data collection method

2.3.1 Data collection instruments

In order to investigate the effectiveness of the OLP, two data collection instruments were adopted namely questionnaires and semi-structure interviews

2.3.1.1 Questionnaires

Questionnaires aim at asking about student attitude, ease of use of the applications, and suggestions for improvement from those who participated in the technology (Eva L Baker, as cited in Heineke, W F., & Blasi, L (2001) It can also be observed that questionnaires were used in many previous studies on evaluating an online learning program (Brill J M., Park Y, 2011; Bui Thi Thu Giang, 2012) In this study, the questionnaires were employed to discover the learners’ satisfaction of the OLP in terms of instructional design, user-friendliness and other criteria of support and cost-effectiveness, the difficulties encountered by the participants and their suggestions to improve the OLP Questionnaires were used as they were suggested in the literature Additionally, this technique allows collecting a large number of data in a short time The questionnaire was translated into Vietnamese so as to avoid any misunderstanding during data collection without the researcher’s presence

Figure 5: The questionnaire uploaded on the website http://tienganhb1.com/

in-In this study, interviews were conducted at convenient time of the day from 9am to

Figure 6: Sample respondent’s details for the interviews

2.3.2 Data collection procedure

The process of data collection consisted of four main steps

Step 1: Designing and piloting the questionnaires

The questionnaire was designed with a view to collecting participants’ attitudes towards the OLP It was created in Google documents on Google drive

The questionnaires were then delivered to 10 VIP members After the information was collected, a brief data analysis was implemented, followed by any amendment of the questionnaire

Step 2: Delivering the complete questionnaires to the participants

After being piloted, the complete questionnaires were uploaded onto the website The selected participants could see the questionnaires once they logged onto their accounts In the first attempt, only 62 participants returned In order to improve

the response rate, the researcher sent an email to all unanswered participants with the link for evaluation to remind them The result was 16 people participated in the questionnaires; increasing the total number to 78 respondents

Step 3: Designing and piloting interviews

After data was collected from the questionnaire, it was briefly analyzed before interviews were designed Interviews were created to find missing and in-depth information regarding the difficulties encountered by the users and their suggestions to improve the OLP After that, the interviews were piloted to two participants and changes were made to complete them

Step 4: Conducting interviews and collecting data from interviews

Interviews were carried out at convenient time of the day

2.4 Data analysis method and procedure

To analyze data, both quantitative and qualitative methods were employed Descriptive analyses were applied to the close-ended survey items (Johnson, pg.116) Descriptive statistics were reported in percentages Data from close-ended questions were displayed in charts and tables in percentages Information collected from open-ended questions in the questionnaires and interviews were analyzed and coded according to themes (McDonough, 2011, p.186)

2.5 Summary

This chapter has elaborated on how data for the study were collected and analyzed in fulfilling the research objectives In the next chapter, findings and discussion of results of the research will be presented

CHAPTER 3: RESULTS AND DISCUSSION 3.1 Introduction

The chapter will be divided into five main sections Apart from the chapter introduction, the second and third sections serve to present the findings of data analysis to find the answers to the two research questions The fourth section provides

a thorough discussion of the results Finally, a brief chapter summary will be given

3.2 Research question 1: To what extent are the learners satisfied with the online B1 level English learning program on website http://tienganhb1.com/?

Leaner satisfaction was evaluated by learner’s reaction to the learning program

in respect of instructional design, user-friendliness and other factors (support and effectiveness) Data was collected under Likert scale from 1 (strongly disagree) to 5 (strongly agree)

cost-Learner satisfaction in terms of instructional design

Concerning instructional design, learner satisfaction is measured through objectives, content, practice and feedback

1 The objectives meet my learning goals 0 1.3 20.5 38.5 38.5

2 The program covers content relevant to my learning

needs 1.3 2.6 15.4 43.6 37.2

3 The instructions are clear and easy to understand 2.6 3.8 21.8 34.6 37.2

4 I have the opportunity to practice the lessons (all four

skills) 1.3 1.3 16.7 47.4 29.5

5 I get feedback for my practice from the teachers 6.4 7.7 32.1 35.9 17.9

Table 3: Learner satisfaction in terms of instructional design

Overall, the table demonstrates that the respondents are satisfied with the design

of content This is shown by the majority of the participants claiming their coincidence between their learning and the objectives of the OLP, the suitability of the content and the space for practice and feedback However, feedback gets the least votes (53.8%) compared to such criteria as objectives, content; practice with 77%, 80.8%, 76.9% respectively Those who show their disagreement are very limited in number with only 1.3% for objectives, 3.9% for content, 2.6% for practice

Most participants suppose that the objectives of the OLP meet their goals whereas only 1.3% of them show their negative attitudes toward this item Similarly, the content of the OLP is favoured by a large number of respondents with 37.2% strongly agreeing and 43.6% agreeing Item 3 was also designed to ask for information about the content of the OLP A large majority of people surveyed agreed that the instructions of the OLP are clear and understandable In terms of practice, more than two-thirds of the participants claimed that they have opportunity to rehearse the lessons Nearly half of the people asked said that they receive feedback from teachers but the number of respondents who are against this item (14.1%) is highest of all criteria Hardly anyone gives comments for these items above except one claiming that

he has self studied and has never contacted with any administrators or teachers

Learner satisfaction in terms of user-friendliness

Questions from 6 to 9 were designed to get information about how learners are contented with the OLP in respect of user-friendliness This category is indicated by accessibility, navigation and control, media and delivery method The 6th item deals with accessibility, the 7th item with navigation and control, the 8th with media and the last one in this category with delivery method

USER-FRIENDLINESS 1 2 3 4 5

6 I find it easy to access the online learning program 7.7 5.1 15.4 34.6 37.2

7 The content is organized in an easy way to locate 3.8 6.4 12.8 46.2 30.8

8

I find the multimedia (videos, pictures…) helpful in

learning rather than merely entertaining or possible

distracting me

2.6 1.3 21.8 47.4 26.9

9 Texts are written in an understandable level 3.8 2.6 16.7 51.3 25.6

Table 4: Learner satisfaction in terms of user-friendliness

It can clearly be seen from the table that those who are in favour of the interface, the ease of navigation of the website, the use of multimedia and the delivery method outnumber those who are against Specifically, 71.8% of the people surveyed could log into the system without difficulty and found the multimedia supportive in their learning The number of respondents claimed that they could locate the needed information easily was slightly higher at 77% This is almost as equal as the figure showing effective delivery method of the OLP A minority of the respondents (12.8%) have trouble accessing the OLP or (10.2%) moving around the website Even very few

of them find multimedia unhelpful in their learning In conclusion, proponents of the OLP concerning accessibility, navigation and control, media and delivery method were limited in percentages at 12.8%, 10.2%, 3.9% and 6.4% respectively

Learner satisfaction in terms of support and cost-effectiveness

Items 10 – 12 were designed to search for data related to support from providers and teachers and the productivity of the OLP

OTHERS 1 2 3 4 5

10 I always get technical support when I need 3.8 1.3 26.9 39.7 28.2

11 I always get academic support when I need 2.6 2.6 24.4 42.3 26.9

12 I think the cost of the course is worth what I gain

from the OLP 3.8 5.1 15.4 46.2 29.5

Table 5: Learner satisfaction in terms of support and cost-effectiveness

In general, almost all of the participants claimed to receive both technical and academic support (94.8% and 93.6% respectively) Also, the majority of the people asked agrees or strongly agrees that the course is cost-effective Item 10 was expected

to survey whether learners are assisted by providers with all technical problems These may concern with the inability to access the website or account, forgetting passwords, inability to access one or more parts of the OLP, activation or reactivation of VIP accounts and so on Most of the users said that the providers were helpful in all technical issues and only a very limited number (5.1%) did not receive support Item

11 deals with content support from providers This includes replying to request to add new content items to the OLP, correcting any wrong answers if any or clarifying the components of the OLP and so on A large proportion of the participants get academic assistance with 26.9% strongly agreeing, 42.3% agreeing while only 5.2% state that they do not receive any content-related support The last item in this part concerns the productivity of the course This is meant to survey if the cost of the course is affordable and reasonable to the users Similarly to the two above items, most of the respondents with 29.5% strongly agree and 46.2% agree with the tuition fee of the course and only 8.9% disapprove of it This signals the providers to keep the cost unchanged

In order to decide the extent to which the OLP satisfy its users, item 14 was designed to address the importance of the OLP to the learners’ preparation for the B1 exam

Chart 1: The importance of the OLP

This is the first time an evaluation has been conducted for the website and it has received very positive feedback from its users Most of them (84%) consider the OLP

to be important; particularly those who emphasize the contribution of the website to their learning remain largest in number On the contrary, nobody thinks that this program is of no significance These figures can prove that the OLP really helps learners in preparing for the B1 level tests

Chart 2: Overall evaluation of the OLP