EVALUATION OF UNICEF-SUPPORTED MOET’S INITIATIVE OF MOTHER TONGUE BASED BILINGUAL EDUCATION IN VIET NAM 2006 - 2014

v ABBREVIATIONS AND ACRONYMS AR MTBBE Action Research on Mother Tongue-Based Bilingual Education BOET Bureau of Education and Training Department MOET´s district level CEMA Committee fo

i

EVALUATION OF UNICEF-SUPPORTED MOET’S INITIATIVE

OF MOTHER TONGUE BASED BILINGUAL EDUCATION

IN VIET NAM 2006 - 2014

-

Final Evaluation Report

December 2015

4.1.6 Role in improving educational policy regarding the ethnic minority people 28

4.4.5 Was the programme or project implemented in the most efficient way compared to

5.2 What could or should be done differently in future replications and/or scaling up? 78

Annex 5 - Additional tables and graphs regarding the effects of the Action Research MTBBE on

Annex 8 – Systematised list of benefits of the Action Research MTBBE 160

Annex 10 – Summary of findings, conclusions and recommendations 169

iv

LIST OF TABLES

Table 1 Number of key informant interviews and focus groups, planned and actual 13 Table 2 Brief general information of the target provinces 22 Table 4 SWOT analysis of implied factors on the sustainability of MTBBE scaling up 73

LIST OF FIGURES

Figure 1 Percentage of MTBBE first cohort students according to grade and general scores 45 Figure 2 Percent of students by language dimensions and scores in MOTHER TONGUE School year 2012-

2013, 1 st and 2 nd cohorts 45 Figure 3 Percent of students by language dimensions and scores in Vietnamese School year 2012-2013, 1 st

and 2 nd cohorts 46 Figure 4 Percent of fourth graders (First cohort) according to provinces and MTBBE / Non-MTBBE

students, by scores in Vietnamese 48 Figure 5 Per cent of fourth graders (First cohort) by provinces and MTBBE / Non-MTBBE students, by

scores in Mathematics 48 Figure 6 MTBBE Costs by component 58 Figure 7 MTBBE Costs 2008-2014 60 Figure 8 Schematic interpretation of the main intervening factors in the development of the AR MTBBE in the three participating provinces 70 Figure 9 Path of factors involved in the replica and scaling up of the MTBBE in areas with characteristics similar to existing project sites 76

v

ABBREVIATIONS AND ACRONYMS

AR MTBBE Action Research on Mother Tongue-Based Bilingual Education

BOET Bureau of Education and Training Department (MOET´s district

level) CEMA Committee for Ethnic Minorities

DOET Provincial Education and Training Department (MOET´s provincial

level) EAPRO UNICEF´s East Asia and Pacific Regional Office

ECED Early Childhood Education Department of MOET

EMED Ethnic Minority Education Department of MOET

EML Ethnic Minority Language (s)

NIESAC National Institute of Educational Strategy and Curriculum (under MOET)

MoU Memorandum of Understanding between the Ministry of

Education and Training and United Nations Children Fund Viet Nam of 15 August 2007, which defines the common

understanding and operating modalities between the parties

MOET Ministry of Education and Training

MTBBE Mother Tongue-Based Bilingual Education

NIL National Institute of Linguistics

vi

PED Primary Education Department of MOET

RCEME Research Centre for Ethic Minority Education, VIES

SIL Summer Institute of Linguistics

SWOT Strengths, Weaknesses, Opportunities and Threats Analysis

UNESCO United Nations for Educational, Scientific and Cultural

Organization UNEG United Nations Evaluation Group

UNICEF United Nations Children's Fund

VIES Viet Nam Institute for Educational Sciences1

1 This is the new name of the National Institute for Education Strategies and Curriculum since 2009

1

ACKNOWLEDGEMENTS

We thank to all relevant Government agencies, local and international organizations, experts, specialist, managers, who actively engaged in the preparation, field work and analysis of the evaluation We express great appreciation to Viet Nam Ministry of Education and Training, Departments of Education and Training in Lao Cai, Gia Lai and Tra Vinh, Ethnic Council of National Assembly/, UNICEF Viet Nam and UNICEF Regional Office for East Asia and the Pacific region for valuable guidance, inputs, technical supports, expertise and financial support for this evaluation In particular, our sincere gratitude goes

to MOET Vice Minister Nguyễn Vinh Hiển, Trần Thị Thắm- Deputy Director of Primary Education Department, Prof Nguyễn Văn Kha- Former General Director of VIES and his team, Hà Đức Đà, RCEME director and his team, other entities within the MOET, managers

of DOETs, BOETs; Madam Trieu Thi Nai, Vice-Chairman of the Ethnic Council of the National Assembly, and Nguyen Manh Quynh, Vice Director of the Ethnic Minority Department of the National Assembly Office; Mr.Youssouf Abdel-Jelil, UNICEF Viet Nam Representative, Jesper Moller-Deputy Representative Thanks to the close technical and management support by Joyce Patricia Bheeka- Education Chief, Vu Manh Hong-Chief, Monitoring and Evaluation and Dinh Phuong Thao and Nguyen Thi Thanh Nga, Inclusive Education Specialists, Nguyen Minh Nhat, Education Officer

The Evaluation Reference Group was a valuable source of help with their comments on the Inception Report and the drafts of the Evaluation Report In the same vein we recognize the UNICEF Regional Office for East Asia and the Pacific whose valuable feedback improved this Evaluation Report

Teachers, parents, community representatives, interpreters of the local languages deserve

a special acknowledgement for their warm reception and readiness to answer our questions Without them we could not have done this work School children became the best part of our mission because their spontaneity, contagious joy and dreams tell us how much the MTBBE is transforming their lives as well as what to expect for the future of Viet Nam

2

EXECUTIVE SUMMARY

Viet Nam has 54 officially recognised ethnic groups, many of which have their own distinct language and live in remote and economically disadvantaged areas2 By 2009 the total ethnic minority (EM) population was about 12.5 million, or 14 percent of the total population3 The official language of instruction in schools is Vietnamese, which is used to teach all children, regardless of their level of knowledge of this language This has created

a “language barrier” for many EM children who have limited, or in some cases no, understanding of and/or proficiency in Vietnamese In this context, two key issues facing education in Viet Nam are: 1) how to cope with the needs of many different language groups, and 2) how to fulfil the right to education for all children in order that they learn and develop to their full potential

The Action Research4 on Mother Tongue-Based Bilingual Education (AR MTBBE) has been

an important initiative of UNICEF´s Education Programme during two Country Programmes

of Cooperation 2006-2011 and 2012-2016 The AR has been implemented in the provinces

of Lào Cai, Gia Lai and Trà Vinh, with the participation of students from three EM groups, i.e Mong, J´rai and Khmer The programme covered six years from pre-primary (one year)

to primary education (five years) levels This is a cooperation project between the Ministry

of Education and Training (MOET) and UNICEF since school year 2008 with two cohorts of

EM children By the end of school year 2013-2014 (May 2014), the first cohort completed a six-year learning programme, followed by the second cohort by the end of the school year 2014-2015 in May 2015

The main objectives of the AR MTBBE are as follows: (i) To implement a valid and feasible design of bilingual education (BE) in ethnic minority languages (EML) and Vietnamese in pre-primary and primary schools; and (ii) To contribute to the development of policies and practices, including legal frameworks, that will promote the use and development of EML

as a means to improve access, quality and equity of education and other social services

The aim of this final evaluation, which covers the period 2007-2014, is to make an objective assessment of the MTBBE approach implemented through the AR MTBBE to date and document evidence and lessons learned for replication and scaling-up in the coming years This evaluation will serve MOET and UNICEF in fulfilling their mandate to

“Successfully implement the AR MTBBE to inform the subsequent development of an appropriate and sustainable policy for BE in Viet Nam.” Other important users of the evaluation report are relevant policy-makers and decision-makers in MOET, CEMA,

2 The World Bank, 2009, page 51

3 2009 Housing and Population Census of Viet Nam

4 Action research is defined as observation, reflection, and planning activities which allow for improvement,

correction, documentation and evaluation of project activities, in this case, what is happening in the selected schools.

The methodology was grounded in the Terms of Reference (ToR) and the approved

Inception Report, where the latter included a detailed Evaluation Design Matrix In

accordance with UNEG Standard 3.10, paragraph 21, the approved Inception Report constitutes the extent of the mutually agreed upon work programme The Evaluation follows a mixed methods approach, which created opportunities for the stakeholders’ participation in providing insights about pertinent issues This methodology used multiple lines of evidence that yielded robust information for the analysis

The evaluation started in October 2014 and ended in August 2015 Fieldwork for data collection took place during the second half of November 2014, including a validation

4

Effectiveness

The programme has been effective in developing and implementing a context-appropriate MTBBE model that has created conditions to improve the learning outcomes of participating EM children and the professional capacity of teachers and education managers

By offering the programme from pre-school to the end of primary education and adopting

a flexible approach, the curriculum design allowed the development of both the students’ mother tongues and Vietnamese right from the first day of schooling as well as offered optimal conditions for the transfer of knowledge from one language to the other

Nevertheless, there were reports of misapplication of flexible teaching methodologies, as well as some hesitation about the teaching of Vietnamese as a second language, in particular in earlier stages of the programme

Impact

The MTBBE has produced factual and perceived impacts on its intended beneficiaries According to different internal/official assessments and evaluations, children’s learning outcomes have been at a high level in all schools of the three provinces, as compared with national standards In addition, children displayed confidence, self-esteem and good communication skills Teachers, school management, board members and all levels of education managers generally supported the programme and recognized its effect on improving their competences to successfully apply the MTBBE design and contributed to develop policies and practices promoting the use of EMLs Parents showed awareness and support to the MTBBE and recognized its benefits in the life of their children

Sustainability

The AR MTBBE has created favourable basic conditions for future replication and scaling

up of the bilingual programme In addition the socio-political climate is also supportive of the scaling up of the MTBBE programme to new schools and new ethnic minority groups in Viet Nam

5

However, for sustainable scaling up of the MTBBE programme, following adjustment of the relevant laws, including Education Law and sub-laws is needed: EMLs can be used as medium of teaching and learning, together with supportive policies and permanent dialogue between central and provincial governments

Gender and Equity

The MTBBE approach has also proved its relevance to UNICEF’s renewed focus on equity, particularly considering its contribution to promote inclusive policies on education and other social affairs By helping ethnic minority children overcome the language barrier in school, MTBBE was regarded as one of the platforms that can be used to improve the academic performance of these children and set the ground for official use of EML in other formal settings, including in the health and legal sectors As a part of the equity efforts, the MTBBE has contributed to good participation of girl students The evidence indicates girls had as good academic performance as boys

Overall Conclusion

The main conclusion of this evaluation is that the AR MTBBE has proved to be a valid and feasible education approach for the context of Viet Nam, as evidence showed relevance, effectiveness, impact, efficiency, sustainability and as a contribution to achieve education equity Evidence gathered from the three pilot sites indicated that the programme had a substantial positive effect on students, teachers, education managers, parents and beneficiary communities, thus contributing to improve the learning outcomes of the participating ethnic minority children The success of the AR MTBBE suggested that EM groups in Viet Nam may be better served and included in social life if EMLs are officially used in other formal arenas, including in the health and legal sectors

Main Recommendations

The recommendations are based on the findings and conclusions of this evaluation These recommendations are intended to inform the design and implementation of the on-going and future MTBBE programmes in Viet Nam They are chiefly directed to MOET and its relevant departments, A few recommendations are also directed to UNICEF, as the promotor and the main supporter of the implementation and management of the AR MTBBE, and to other development partners The recommendations were developed in close consultation with key stakeholders, including UNICEF, MOET and local level education authorities They are not listed in any priority order, but thematically organized based on the evaluation criteria ,

1 Develop a Project Initiation Document (PID), including a theory of change, for the next

programme delivery, clearly defining performance indicators to be used in the management and assessment of its overall success This should be part of any programme/project initiative

2 Review the existing Vietnamese legal and policy frameworks with the view of strengthening them clearly towards MTBBE

6

3 Show strong political commitments to introduce mother tongue learning by establishing

a specific policy on how the MTBBE approach should be implemented in Viet Nam

4 In the current context of curriculum and general education reform in Viet Nam, incorporate MTBBE as one of the solutions to improve quality and equity for ethnic minority children in this country

5 Conduct careful sociolinguistic studies before implementing the MTBBE approach in new

EM groups in Viet Nam as this may give indications about the relevance and the likelihood

of success of the programme in those settings

6 Develop a strategic advocacy/communication plan to ensure that relevant institutions issue policies/guidance on the use of EML in other official arenas, including in basic service delivery sectors such as health, child protection and legal

7 In the scaling up phase, replicate the design and the flexible approach adopted during the pilot stage, as they yielded a strong form of bilingual education, which is recommended for the socio-cultural context of Viet Nam This design should form part of the future policy on MTBBE in Viet Nam

8 As part of the preparation of the scaling up phase, organize a seminar/workshop aimed

at analyzing and reviewing the teaching-learning methodologies used in the MTBBE to recommend adjustments to make them more responsive and appropriately applied

9 Adopt multilingual-oriented second language methodologies in the teaching and learning of Vietnamese as part of the MTBBE approach This may include the involvement

of NGOs promoting the teaching and learning of Vietnamese as a second language in Viet Nam (e.g Plan International) in training workshops and/or continuing professional development programmes for MTBBE teachers

10 Conduct (longitudinal) studies of former MTBBE students’ performance at secondary school in order to assess the educational impact of MTBBE beyond primary education level The results of those studies will provide further context-specific evidence about the feasibility and validity of the MTBBE for ethnic minority students in Viet Nam

11 Develop a Behaviour Change Communication Strategy (campaign) to raise awareness

of parents and communities about the purpose and benefits of the MTBBE and encourage their support and participation, as well as to mobilize collective action in support of the programme

12 Scale up the MTBBE programme, involving more schools and EM students, thus maximizing the academic and sociocultural benefits of the approach and reducing its costs

by capitalizing on economies of scale

7

13 Consolidate all bilingual reference materials (i.e teaching guides, grammars, dictionaries, additional reading and reference books, and visual aids) in the list of basic materials to be produced and supplied to MTBBE schools

14 Mobilize independent writers and publishers to produce low-cost and/or no-cost reference materials in EML for MTBBE schools; this may include award offers and promotion of contests and competitions culminating in the publication of the best manuscripts, including on-line publication

15 Develop a strategy for a gradual expansion and scaling up of the MTBBE programme

This should have two goals: (a) the continuity and dissemination of MTBBE in the three pilot provinces and (b) the design and implementation of new MTBBE interventions in provinces with characteristics similar to those of the provinces involved in the pilot stage

or in other provinces fitting MOET’s criteria

16 Continue to provide financial support for teacher training and development of teaching and learning materials to the experimental provinces for three to five years while they will continue to find other donors to sustain the programme

17 Provide support to teachers for them to effectively deliver a multi-language policy by ensuring pre- and in-service training which lead to the improvement of their own proficiency in the language of instruction and in teaching skills in ethnic minority languages, including early literacy development in these languages

18 Establish and maintain a dialogue with relevant provincial authorities in order to clarify responsibilities and find solutions for the financial needs of local governments to sustain current MTBBE efforts and results and set the bases for the expansion and scaling up of the programme in those provinces

19 Bring language of instruction to the forefront of dialogue on education sector funding

with the government partners, particularly in relation to both quality and equity concerns, ensuring the coverage of MTBBE is highlighted as international indicator of education quality

20 Mobilize resources to support the introduction of MTBBE and offer technical assistant

to develop flexible competency targets, literacy materials and an assessment framework in multiple languages

MOET and UNICEF were key institutional stakeholders of the AR MTBBE Within MOET, the pilot involves different specialized departments, i.e Viet Nam Institute for Educational Sciences (VIES), Research Centre for Ethnic Minority Education (RCEME which was part of VIES and assigned as focal point of the pilot implementation), Early Childhood Education Department (ECED), Primary Education Department (PED) and Ethnic Minority Education Department (EMED) MOET´s Vice-Minister, UNICEF´s Deputy Representative and the Directors of MOET´s departments form the Steering Committee

In addition, the Advisory Group included representatives of the ECE, PED, EMED, RCEME,

NIESAC, CEMA and NIL, as well as senior staff of UNICEF, UNESCO and The World Bank

At the local level MOET is represented by Provincial Research Management Boards and District Action Research Management Boards, both adding the representation of provincial and district education managers, school principals and teacher associations

During the implementation, the AR MTBBE continuously received technical support from well known international experts such as Carol Benson, Margaret Franken, Andy Smart,

Susan and Denis Malone and other experts from the Summer Institute of Linguistics (SIL),

as well as the best national experts of the Viet Nam Linguistic Institute for the design,

curriculum and materials development, training of master teachers´, Action Research planning and operation, learning outcome assessment, and Primary Classroom Language Mapping

The Ethnic Council of the National Assembly, was another important partner in promoting

the MTBBE approach and policy Also CEMA was recognised as having an important role in policy development

9 For a more detailed project overview, See Annex 2

9

The specific responsibilities of UNICEF in the AR MTBBE have been:

 To provide support to MOET in the overall implementation and management of the Action Research, through technical assistance and/or contracting of international specialists, as per the Research Overall Design

 To ensure that the initial and on-going design, technical aspects, and overall quality of the AR are based on sound practices according to inputs from international specialists working with national counterparts

 To liaise with and report to interested donors to support the raising of financial resources for the implementation of the AR activities throughout the entire Research cycle

 To provide support to RCEME in ensuring participation of relevant institutes and organizations such as NIL, CEMA, Teachers´ Colleges and Pedagogical departments and universities, and in relevant AR activities

2.0 EVALUATION, PURPOSE, OBJECTIVE AND SCOPE

2.1 Purpose10

The main purpose of the evaluation is to generate substantive evidence and knowledge to inform national policies on mother tongue bilingual education and the subsequent reduction of education inequities in Viet Nam as well as strengthen the sub-national commitment in implementing the MTBBE approach

The main users of the evaluation are relevant policy-makers and decision-makers in the MOET, CEMA, National Assembly, subnational authorities, education managers, teachers,

UN agencies and various development partners

The first MTBBE student cohort entered pre-primary school in 2008 By May 2015 the second cohort finished the five grades of the primary level, which marked the end of the

AR MTBBE UNICEF- MOET ´s Education Programme Annual Work plan 2014 anticipated an

independent Evaluation This evaluation was designed to make an objective assessment of the MTBBE approach implemented through the AR MTBBE to date and document evidence and lessons learned for replication and scaling-up in the coming years This evaluation would also serve MOET and UNICEF to inform the subsequent development of an appropriate and sustainable policy for bilingual education in Viet Nam By supporting organizational learning and accountability, this evaluation was expected to help related agencies continually to improve their performance and results In addition to that this evaluation would support planning and decision-making, and provide a basis for informing relevant education policies that promote the well-being of all children In sum, this evaluation served to improve results and stakeholder satisfaction

10 ToR, 2013, p.2-3

10

2.2 Evaluation Objectives

The Terms of Reference (ToR) stated the following objectives:

 To assess the extent to which the MTBBE approach has been relevant, effective, and efficient and has contributed to changes at the outcome and impact levels, including the contribution to the better learning outcomes of the participating children (first cohort and second cohort) The evaluation will also seek to determine whether the MTBBE approach is sustainable

 To analyse whether the results obtained through the overall strategy of implementing the MTBBE approach at both the national and sub-national levels have contributed to the reduction of education inequities and to the fulfilment of children´s rights to education in Viet Nam

 To identify lessons learnt on design and implementation and to provide recommendations for on-going and future MTBBE in Viet Nam

 To recommend actions to advocate for supportive policies on the use of mother tongues in bilingual education for the improvement of quality of education in ethnic minority areas

 To identify good practices, innovative interventions and shortcomings (in terms of implementation of MTBBE including inclusiveness and effectiveness, child friendly learning environment, participation of students, parents and the community people), positive effect

on individual children, teachers, parents, community leaders and local authorities, etc

The criterion on “sustainability”, particularly on the question of scaling up in areas with characteristics similar to existing project sites, became particularly challenging because the team had to cope with incomplete or contradictory information on key issues, such as the

11 UNICEF, 2013, page 10

11

number and the geographical distribution of different ethnic populations, the extent to which they are alone or mixed with others, and what linguistic situations emerge from this distribution

In particular, the methodology was grounded in the Terms of Reference (TOR) and the approved Inception Report, where the latter included a detailed Evaluation Design Matrix

(see Annex 2) In accordance with UNEG Standard 3.10, paragraph 2116, the approved Inception Report constitutes the extent of the mutually agreed upon work program The evaluation was structured in hierarchical sequence by the evaluation criteria (i.e relevance, effectiveness, impact, efficiency, and sustainability), and within these by the corresponding evaluation questions and sub-questions These “were then analysed by the types of evaluation question (descriptive, normative, cause-effect), quantitative and qualitative measures or indicators, the type of evaluation design (experimental; quasi-experimental and non-experimental), the multiple line of evidence, also referred to as data sources (documents, interviews, focus groups, site visits, etc.), approaches to data collection (e.g sample or census), data collection instruments (e.g questionnaires), and notes on approaches to data analysis

The original questions and the sub-questions were included in the Annexes The key questions are as follows:

 To what extent is the MTBBE approach relevant to the socioeconomic, legal

and policy contexts of Viet Nam and UNICEF´s renewed focus on equity?

16 UNEG Standard 3.10 (paragraph 21) states: “The responsibilities of the parties who agree to conduct an

evaluation (specifying what, how, by whom and when what is to be done) should be set forth in a written agreement in order to obligate the contracting parties to fulfill all the agreed upon conditions, or if not, to renegotiate the agreement Agreements, such as the Terms of Reference, should be established at least in the follow areas: financing, time frame, persons involved, reports to be produced or published, content,

methodology, and procedures to be followed Such an agreement reduces the likelihood that

misunderstandings will arise between the contracting parties and makes it easier to resolve them if they arise Providing an inception report at the start of the evaluation is a useful way of formalizing such an agreement and ensuring proper interpretation of the Terms of Reference.”

12

 How effective is the MTBBE approach to improving both the learning

outcomes of participating children, and the capacity of teachers and education managers?

 What impacts did the MTBBE approach have on four stakeholders groups: (A)

MTBBE students; (B) Teachers, school managements, board members; (C) Parents and members of the community; and (D) Managers from grassroots to central levels?

 Does the actual implementation of the approach use resources in the most

efficient (economical) manner to achieve expected results?

 To what extent should the benefits of the MTTBE approach be continued after the donor funding ceases?

3.1 Data Collection Methods and Sources

To meet the challenge of the evaluation questions it was necessary to gather reliable and valid quantitative and qualitative data To this end the evaluation team followed a mixed methods approach that created opportunities for the stakeholder participation in providing insights about pertinent issues This methodology used multiple lines of evidence that yielded robust evidence

The evaluation started in October 2014 and ended in August 2015 Fieldwork for data collection took place during the second half of November 2014, including a validation workshop with the Steering Committee members UNICEF Viet Nam staff assisted the evaluation team with travel logistic and arranging interview appointments in the three provinces where the AR MTBBE schools are located, as well as interviews and other

research activities in Ha Noi

The evaluators reviewed 277 documents of background information previously sent by UNICEF Viet Nam This information was complemented by other documents collected in the field The other methods of data collection were face-to face in-depth interviews with key informants and focus groups with children, some provincial and district educational authorities and school managers, as well as teachers and parents The evaluation team interviewed 62 key informants at the central, provincial and district levels, and conducted

22 focus groups in which 163 teachers, parents and children participated.17

The sample of key informant covered the entire range of informants at the central level (education authorities, other high level governmental authorities and donors) At the local level these interviews covered educational authorities of the three provinces, seven districts, and seven schools participating in the AR MTBBE Participants in the focus groups included the majority of currently active MTBBE teachers, and a moderate number of parents and students The sample for the focus groups had a gender-balanced representation See Table 1 and Annex 4

17 The original planned number of focus groups participants was 112

13

Table 1 Number of key informant interviews and focus groups, planned and actual

Key-informant interviewee category Estimated # in

Target Population

Actual # of Interviews

Key informant interviews with MOET, other

central level managers and representatives of

the donor community

Key informant interviews with DOET and other

provincial and local authorities

For evidence regarding the Impact section (Section 4.3), the evaluation team used secondary data provided by MOET, which had been used in the preparation of the

“Programme Brief” and the Annual Student Assessments; both publications had been produced by MOET with technical support from UNICEF

3.2 Data Analysis

Evidence from documents, key informant interviews and focus groups was entered into

NVivo for reviewing and coding in accordance with the questions contained in the Evaluation Design Matrix

To ensure the quality of the analysis, triangulation and validation of findings were undertaken from all sources including the documents, interviews and site visits No conclusions were based on only one opinion An initial internal peer review process involved UNICEF Viet Nam and UNICEF´s East Asia and Pacific Regional Office, and later Central Level stakeholders

3.3 Data Gaps and Limitations

14

3.3.1 Data quality assurance and sound analysis

The Inception Report warned about different challenges that the evaluators might find in assuring data quality (reliability and validity) and sound analysis To meet such challenges particular measures were taken as described below:

 Data were collected from representatives of all key stakeholders

 Multiple lines of evidence were used including secondary sources (files, documents, databases, specialized literature) and primary sources (individual and group interviews, focus groups)

 The questions contained in the interview questionnaires and guides addressed

all evaluation criteria, questions and indicators as presented in the Evaluation Design Matrix Each line of evidence included program staff, recipients, and

stakeholders

 Information was carefully handled by intensive use of NVivo software for systematization/coding and analysis Facts were carefully distinguished from opinions or perceptions

 All evaluators participated in entering the data and, in turn, validating the analysis by means of collegial oversight/review of the information

 UNICEF Viet Nam, UNICEF´s EAPRO and the Evaluation Reference Group also participated in reviewing different drafts of the evaluation report

3.3.2 Data gaps and limitations

A weak base for generating performance indicators The AR MTBBE was undertaken on the basis of the MoU and the AR Implementation Framework There was no proper Project Initiation Document (PID), which usually defines the project to form the basis for its

management and the assessment of overall success The PID normally contains the program goal, purposes, justification, management and roles, inputs, and an execution schedule The PID is key to develop an adequate and precise monitoring system based on performance indicators Without a PID, performance indicators were hard to identify, thus evaluators had to review the MoU and its Annexes for this purpose These two documents offered clues for generating performance indicators but they lacked precision in quantifying most of the specified activities A project monitoring system, which could be a simple Excel control table, easily shows the key performance indicators throughout the execution period Usually evaluators receive this information to assess how close each indicator is from the target Evaluators also verified this information in the field by observing, for instance, whether the textbooks were supplied to a school in the quantity specified in the monitoring system In the case of this evaluation, the Research Team did not find evidence that such a monitoring system was operating; therefore, they concluded that the monitoring of the AR MTBBE was focused on learning indicators rather than other performance indicators (e.g number of training workshops; developing, printing and

15

delivery of teaching and learning material), where the latter were probably registered as a part of MOET´s general inventory control system rather than as a separate one

Secondary data used for key assessments on impact The key data of this evaluation were

those related to learning outcomes In this regard, the evaluation team and its UNICEF counterpart agreed that primary data collection regarding students´ learning outcomes would be unnecessary because MOET already had data resulting from previous annual assessments The main results of these evaluations had been already published by MOET, with the support of UNICEF, in the "Programme Brief" and “Annual Assessment Reports”

It was expected that the evaluation team should consider this information to be available for use as the basis of Section 4.3.3 ("What impact did the MTBBE approach have on participant learning outcomes of students?") This information was considered to be primary information in the case of MOET´s assessment reports, but it would be managed

as secondary information for the case of the present evaluation

Because MOET had twice hired two different agencies to collect and analyse data related

to student learning outcomes, some inconsistencies in the assessment methodology could

be expected MOET provided a copy of Excel and some SPSS (2013-2014) databases This set of information did not allow the evaluation team to rebuild a complete matrix or matrices of data regarding all students, all indicators and all years by means of which to process whatever cross-tabulation was needed In some cases the information was already grouped by schools or provinces; in other cases some indicators were available for some years but not for all In short, most of the information available for the evaluation team had been previously processed and analysed On this basis the evaluation team prepared Section 4.3.3

As is common with secondary information, the evaluation team first assumed the data were reliable and valid since there was evidence that (the data) had been collected and analysed with appropriate methodological rigor18 The evaluation team did not review the methodology used in the collection of data and organization of the database, although it remained attentive to detect any inconsistency In fact, the evaluation team found virtually

no evidence of inconsistencies in the data up to becoming an obstacle to (i) assess general academic performance of each of the two student cohorts during the entire primary school period; or (ii) comparing these results by province The evaluation team only encountered a problem trying to compare the performance of students participating in the program with non-participating students throughout the period (primary school years) The data analysis and preparation of the evaluation report was scheduled to take place

18 MOET had standards for the assessment of student learning outcomes (testing criteria, methods and protocols for the different subjects) UNITEC also hired international consultants to prepare manuals about the oral and written versions of Vietnamese Placement Test for primary school The results of these tests were monitored by the consultant Steve Walter to guarantee that regional variation in the use of Vietnamese was properly considered

Time was a constraint for more in-depth observation of the learning environment The

evaluation team visited three participating provinces according to an agenda of interviews, which was more intense than anticipated This generated an opportunity to interview authorities, teachers, children and parents However, time was very short to go deeply into the different issues For example, it might have been interesting to observe the development of some classes to get a first-hand impression of the teaching-learning process As is known, this is a time-consuming activity that was not planned because of its cost implications

The selection of participating children without discrimination and bias was important; however, this was out of reach in the case of this evaluation Two baseline surveys had

been conducted at the beginning of the school year 2006-2007 and at the beginning of

2007 to ensure selected criteria were well communicated to local authorities, schools, teachers and parents The evaluation team conducted discussions with parents and members of local communities in their EM language to ensure active participation Annex

2 of the MoU presents the criteria for selecting the target pre-primary and primary schools Those criteria expressed preference for small schools; pre-primary schools should

be included in or located close to a primary school; schools should be linguistically homogenous, i.e., close to 100% students should speak the target EM language; the schools should have competent and qualified teachers in the EM language as well as in Vietnamese (preferable), among other similar criteria It was also important to procure classes where the number of boys and girls was almost the same Besides these criteria no others had been expressed, for instance criteria to prevent subtle or open discrimination against EM children in a particular physical or psychological situation The evaluators were not informed by local interviewees about episodes of discrimination According to the evaluators´ experience with similar programmes in other developing countries, discrimination against special students, when detected, represented only isolated cases The detection of this problem becomes difficult with the type of data collection tools usually used in evaluations where the detection of discrimination and bias against participant children is not a priority, given the time and cost that most appropriate research tools would imply

3.4 Ethical Standards

17

UNICEF Ethical Principles and Premises of the Evaluation19, UNICEF Ethical and Principle Guidelines for the Reporting on children and young people under 18 years old20, as well as UNEG Norms and standards for the protection of confidentiality, dignity, rights, respect of the values of the visited communities, and child welfare (friendly environment and talks) were applied throughout the entire processes of primary data gathering, analysing and report writing In particular these involved those on Gender and Human Rights, including child rights Interviews were held on an informed and voluntary basis In most cases, interviewees permitted the recording of the sessions to facilitate the note-taking process Testimonial material had been used in a confidential manner to protect the anonymity of respondents

Ethnic minority children in selected schools, including MTBBE and non-MTBBE students, participated in six focus group sessions, which were previously authorized by the respective school authorities, thanks to UNICEF Viet Nam and MOET recommendations In preparation for these meetings the evaluation team participated in a special session in Hanoi where UNICEF´s staff provided information on protocols to be followed to respect child rights

3.5 The Theory of Change

This section addresses the following evaluation question: What was the theory of change

of the intervention, as summarized or indicated in a reconstructed results chain through which inputs are translated into successive higher levels of results?

The theory of change is summarized in the results chain shown below The results chain displays the relation of the inputs to expected outputs, outcomes and the long term results

The displayed results chain is an ex-post interpretation undertaken by the evaluation team; it was designed after a careful review of the MoU and the AR implementation framework 2007-2014 It represents the reconstructed rationale of the intervention, which

is implicit rather than explicit Draft versions of the results chain were modified according

to the comments received from reviewers of this Report

The AR implementation framework shows results-oriented planning, but without the guide

of an explicit Theory of Change Evaluators also noticed the lack of a PID for the MTBBE, which would have mainly described the goal and purposes, justification, management and roles, inputs, and execution schedule This explained why the MoU between MOET and UNICEF dated 15 August 2007 became very important for grasping the project´s rationale The lack of a PID became an obstacle to the evaluability of the project given the value

19 See ToR, Annex 5

20 See ToR, Annex 6

As illustrated by the reconstructed results chain shown below, the final expected result of the AR MTBBE would be the contribution to the development of an applicable and sustainable policy on bilingual education This policy had to be based on both the existing legal framework and evidence on the advantages of the MTBBE approach For this general purpose it was necessary to implement the AR MTBBE in some carefully chosen provinces and schools, where two cohorts of EM students had to attain learning outcomes according-to-national-standards learning This would require (i) trained educational managers and teachers, (ii) all-level (local, provincial, national) stakeholders´ support, and (iii) attention from MOET and policy-makers to adopt the MTBBE approach as one solution

to improve the quality of EME

That final result and outcomes would be attained after overcoming several challenges, such as the lack of a specific MTBBE curriculum according to MOET´s standards, appropriate learning and teaching materials, training opportunities for teachers and education managers, and orientation for policy-makers Also, monitoring, evaluation and reporting activities, based on action-research had to be organised to achieve quality and effectiveness control Inputs in the form of both funding (UNICEF, donors, MOET) and inter-institutional close coordination and management had to be guaranteed as triggering condition of the proposed programme

21 Revision 2 20141031, page 38

19

• It captures congruently planners´ vision and beneficiaries´ perception of their needs;

• It responds to changing and emerging development priorities and needs in a responsive manner;

• It gets acceptance and capacity to operate within the local context

In evaluating relevance, the ToR included questions to assess the extent the AR MTBBE had considered national priorities as emerging from the socioeconomic, legal and policy contexts of Viet Nam It also included the extent to which the initiative aligns with UNICEF´s renewed focus on equity

4.1.2 Summary regarding Relevance

The general question on relevance was: To what extent is the MTBBE approach relevant to the socioeconomic, legal and policy contexts of Viet Nam and UNICEF´s renewed focus on equity?

Finding No.1 The MTBBE approach is relevant and responsive to the socioeconomic, legal and policy context of Viet Nam and UNICEF´s focus on equity

MTBBE has proved to be relevant and responsive to the Vietnamese context at both national and subnational levels MTBBE translates the Vietnamese legal and policy framework for a more inclusive and equitable educational system, as expressed in the Constitution, Education Law, Education Strategic Development Plan (2011-2020), among other relevant laws and policy frameworks From the cultural point

of view, the relevance of the programme is closely linked to its potential to contribute to the preservation

and development of ethnic minority languages and cultural practices and beliefs

Despite the attested enabling implementing environment, a review of Vietnamese key legal and policy instruments suggests the need of revisiting/ strengthening the existing policies and laws clearly towards

MTB-MLE

The MTBBE approach has proven to be relevant to UNICEF’s renewed focus on equity, particularly considering its contribution to promote inclusive policies on education and other social affairs By helping ethnic minority children to overcome the language barrier in school, MTBBE can be regarded as one of the platforms that can be used to improve the academic performance of these children and lay the ground for

official use of EML in other formal settings, including in the health and legal sectors

22 OECD Network on Development Evaluation, n.d., page 13

21

A relevant

approach?

To the socioeconomic context?

To international treaties and commitments?

To legal and policy frameworks?

To UNICEF´s renewed focus on equity?

There is a need to strengthen legal and policy frameworks clearly towards MTBBE



Sections 4.1.3 to 4.1.5 responded to each of the sub-questions on relevance by providing the summary evidence that substantiates Finding No.1

4.1.3 Relevance to the Vietnamese - context

The first sub-question regarding relevance was: To what extent is the MTBBE approach relevant and responsive to the Vietnamese context and needs at both national and sub-national level?

Language and poverty are key factors in explaining the educational gap between the Kinh majority and children from ethnic minorities Language is also a barrier for the social inclusion of the ethnic minority children As soon as MTBBE is a solution to overcome this language barrier and its negative effects on equity, social capital and social cohesion, it becomes relevant in the context of Viet Nam

Several structural barriers hamper upward social mobility of ethnic minority groups In the realm of education, poverty and language23 have been identified as paramount problems Most school aged children from ethnic minorities are put at a disadvantage when Vietnamese is the only medium of instruction in schools As abundant scientific evidence has shown, students who do not learn their mother tongue proficiently have greater difficulty learning another language well.24

Within this context, MTBBE becomes relevant because it offered one promising solution for overcoming the language barrier that affects most ethnic minority children The goal of recent educational reforms is to increase the learning potential of all children, which is very difficult to attain when poverty, languages and other major obstacles exist Viet Nam needs a better-educated population to sustain and revitalize its rapid economic growth

includes children with a poor command of English because it is not their native tongue)”

24 Dutcher, 2004, p 10

22

At the subnational (provincial, local) level, the greater relative importance of some ethnic groups must be considered For instance, in Lào Cai province ethnic minority groups represent about two thirds (64%) of the population, in Gia Lai 61% and in Trà Vinh 32% The AR MTBBE´s participant ethnic groups form the bulk of the ethnic minority groups in those provinces, i.e J´rai 34%, Khmer 30% and Mong 22% of the total population [See Table 2] Hence, by creating conditions for improving access to basic quality education, MTBBE might greatly empower provinces with an important share of ethnic minority people to be better positioned in contributing to and taking advantage of economic growth as well as the relatively increasing offer of social services This comment considers that MTBBE becomes relevant in communities with relatively homogenous EM populations s This means that one EM language is really predominant as the home language of almost all school-age children Despite the fact that there is no disaggregated data on the % of each ethnic groups in the province

Table 2 Brief general information of the target provinces

Ho, Nhang, Thai, Muong

26 including Kinh

36%, Mong 22%, Tay

16%, Dao 14%, Giay 5%, others: Phu La, San Chay, Na Nhi and the La Chi

4.1.4 Relevance to international treaties and commitments

The second sub-question on relevance was: To what extent is the MTBBE approach relevant to international treaties and commitments to which Viet Nam already signed up?

23

International agreements encourage governments to observe cultural and linguistic dimensions as crucial components of human rights The approach is relevant because its design considers and builds on these agreements by fostering MTBBE as one of the best practices which many countries have adopted to express deference for cultural diversity and recognition of its pedagogical value to develop the full potential of children from ethnic minorities

Viet Nam appears in the list of countries that have signed international ratification, accession or succession treaties and conventions, such as the UNESCO Convention against

Discrimination in Education (1960), the International Convention on the Elimination of all

Forms of Racial Discrimination (1965), the International Convention on Economic, Social and Cultural Rights of 1966, the United Nations Convention on the Rights of the Child of

198925

This body of international agreements encourages governments to observe cultural and linguistic dimensions as crucial components of human rights This way, governments become encouraged to avoid discrimination against other cultures, e.g ethnic minority groups Education is a field in which open or discreet (indirect) discrimination may act against ethnic minorities In this case international legislation has been clear regarding the right of all children to access compulsory and free primary education and has opened the door to allow that “children living in exceptionally difficult situations” have special, positive treatment, such as provided in affirmative action policies A difficult situation is not only to live in a conflict area or have physical impairments; there are countries in which children of ethnic minorities live also under structurally difficult conditions because

of poverty and social exclusion Recognizing this fact, an increasing number of countries have been introducing MTBBE By doing so these countries show respect to cultural diversity while making use of tested pedagogical methods that develop the full potential of children from ethnic minorities and minimize school absenteeism and dropout

The MTBBE approach is relevant regarding this matter because its design considers and builds on these international laws

The UNESCO Convention against Discrimination in Education (CADE) and The Convention

on the Rights of the Child (CRC) are two examples of pertinent international laws to which Viet Nam has subscribed

The CADE, which entered into force in 1962, recalls the Universal Declaration of Human Rights, particularly the principle of non-discrimination and the proclamation that every person has the right to education

25 Minority Rights Group and UNICEF, 2009, pages 222-233; Human Rights Library of the University of Minnesota

24

The CRC entered into force in 1990 The Convention recognized that, “in all countries in the world, there are children living in exceptionally difficult conditions, and that such children need special consideration” In those States in which ethnic, religious or linguistic minorities or persons of indigenous origin exist, a child belonging to such a minority or who is indigenous shall not be denied the right, in community with other members of his

or her group, to enjoy his or her own culture, to profess and practice his or her own religion, or to use his or her own language (Article 30)

25

4.1.5 Relevance to the Vietnamese legal and policy frameworks

The third sub-question on relevance was: To what extent is the MTBBE approach relevant

to Constitution, Education Law, and Law on Child Care, Protection and Education, national policies, and others?

“MTBBE operationalizes the body of legislation and educational policies addressed to benefit ethnic minority children; it shows a path to go beyond words”.26 The Government

of Viet Nam has tested several approaches in the past to guarantee ethnic minorities´ access to education while respecting their rights to use their own language and preserve their cultures However, the results have not met the expectations The failure greatly responds to the fact that mother tongue has not been in the role of language of learning,

a practice supported by some articles of the educational legislation MTBBE challenges the assumptions behind this legislation, which leaves little room for the scaling up of this programme

Relevance to the Constitution

The AR MTBBE was highly relevant to the Constitution because it shows a way to honour cultural diversity in schools where this diversity is a problem (instead of an opportunity) for learning The MTBBE approach promised that a second language can be learned well when a child first becomes literate and fluent in his/her own language

Article 5 of the new Constitution of Viet Nam (2013) clearly expresses that although Vietnamese is the national language:

 every nationality has the right to use its own language and system of writing, to preserve its national identity, and to promote its fine customs, habits, traditions and culture;

 the State implements a policy of comprehensive development, and provides conditions for the national minorities to promote their internal abilities and to develop together with the nation

Relevance to legislation on education

The Vietnamese constitution, education laws and many education policy documents have strongly supported the use of ethnic minority languages (EML) in education These documents gave equal support to the learning of Vietnamese by ethnic minority people, thus emphasizing bilingual education as a good strategy for ethnic minority learners However, they are not explicit about the use of EML as languages/media of teaching and learning in formal education

26 High level official from MOET

26

Article 7 of the Education Law (Law No.38/2005/QH11) points out, that Vietnamese is the official language to be used in schools and other educational institutions It also prescribes that “The State shall enable ethnic minority people to learn their spoken and written languages in order to preserve and develop their ethnic cultural identity, helping pupils from ethnic minorities easily to absorb knowledge when they study in schools and other educational institutions The teaching and learning of these languages shall be conducted

in accordance with the Government regulations” The revised Education Law 200927

maintained the same idea, thus repeating that the teaching and learning of EML should be

in line with Government´s regulations

Governmental Decree 82 of July 2010 (82/2010/ND-CP) regulates “the teaching and learning of EMLs including the conditions, content, methods and forms of teaching and learning: teacher training programmes, policies for teachers and learners for the teaching and learning of EMLs” Article 6 clearly indicates that EMLs are to be taught as a subject at all education levels and categories (paragraph 1)

In sum, EMLs were taught in school only as subjects So far MTBBE was implemented only

on an experimental basis For expansion, MTBBE would require either a revision of legal framework in the education or authorities tolerating de facto situations, where the second

is not a recommendable alternative Legislators have a legitimate concern with the importance that Vietnamese has for people´s full social inclusion and socioeconomic mobility The Education Law was based on the assumption that learning Vietnamese at early age is beneficial for ethnic minority children However, the clear learning gaps between the Kinh majority and most ethnic minority (EM) children suggests that this assumption is partially inaccurate EM children are not learning Vietnamese sufficiently well as to guarantee the expected benefits at school and in society as a whole As shown

by international experience, and by the evaluation of the AR MTBBE, Vietnamese can be learned better if EM children first develop appropriate levels of oral competency and literacy in their own languages

Relevance to the Vietnamese policy framework

The policy reference for MTBBE comes mainly from central and sectorial plans ratified by the Government The most important policies, valid for the period of the AR MTBBE, are the National Education for All (EFA) Action Plan 2003-2015, the Socio-Economic Development Strategy (2006-2010 and 2011-2020), the Education Strategic Development Plan (2006-2010 and 2011-2020), the Resolution No.29 (November 2013) of the Central Steering Committee of the Communist Party of Viet Nam on comprehensive reform with socialist orientation and global integration, and the Prime Minister´s Decision on Approval

of the School Education Textbook and Curriculum Innovation Scheme (27 March 2015)

27 Law Amending and supplementing a number of articles of the Education Law The National Assembly No.44/2009/QH12, 25 November 2009

27

The EFA Action Plan 2003-2015, in its updated version of 2012, recommends the design of

an inclusive development strategy that merges all relevant short-term measures, such as the extension of the role of special schools, expansion of training in inclusive teaching methods and the provision of additional materials and resources for disadvantaged children (EFA, 2012, page 8) Under the framework of this strategy, the EFA Action Plan highlights the relevance of MTBBE and other approaches28 For EFA, MTBBE is a

“promising” option to, for example,

improve learning outcomes of ethnic minority pupils, and as a condition for the success of this effort, the promotion of teachers´ training and support and offer a preventive strategy against school repetition and dropout.29

Other documents include those promoting inclusive education measures addressed to disadvantaged populations For instance, (i) The Socio-Economic Development Strategy (2011-2020) and the Education Strategic Development Plan (2011-2020); (ii) The Socio-economic Development Strategy, dated 10/29/2012 and approved during the 6th Conference of the Communist Party; and, (iii) The Resolution No.29-NQ/TW, dated 11/04/2013, on “fundamental and comprehensive innovation in education serving industrialization and modernization in a socialist-oriented market economy during international integration”, ratified in the 8th session of the Central Steering Committee of the Communist Party of Viet Nam

The most recent Prime Minister´s Decision on Approval of the School Education Textbook and Curriculum Innovation Scheme (No.404/QD-TTg, 27 March 2015) operationalizes the aforementioned Resolution No.29 (November 2013) This way the new scheme proposes, for the entire education system (Grades 1 to 12), curriculum and textbook development aligned with the education reform The new curriculum combines knowledge and practice

as well as competences and student-centred teaching methodologies, including special treatment to meet the needs of the most vulnerable children Budget provisions allocate public resources for developing and piloting the curriculum and compiling and piloting a set of textbooks, including bilingual textbooks (Vietnamese and EMLs)

For all these documents, education plays a leading role for which it has to experience a comprehensive renovation In this sense, it is necessary to consolidate the universalization

of primary education and pursue the universalization of secondary education and the school A part of these efforts is improving the quality of education and the development

pre-of policies focused on promoting education in the most disadvantaged regions to reduce inequality in education access

28 The source mentions five approaches, i.e PEDC or Vietnamese reading model, JICA´s “Lesson Study”, Teacher Action Research which is a Belgian Project, Escuela Nueva model in Viet Nam supported by the World Bank, and finally, a group integrated by different projects supported by Oxfam UK, Enfant et

Développement, Save the Children and UNICEF “which provide a more professional approach” (EFA

Programme, 2012, page 32) Escuela Nueva, Save the Children and UNICEF projects use mother tongue

in the learning process

29 Education for All Action Plan, 2012, pages 35-36

28

In sum, the Vietnamese policy framework sees education in a prominent role for development and creates a favourable environment for reform and focus on the most disadvantaged regions This creates good will and guarantees resources for new curricula and textbook approaches However, it remains a challenge to legislate on the use of EMLs

as language of learning beyond an experimental basis of the MTBBE approach

4.1.6 Role in improving educational policy regarding the ethnic minority people

A fourth sub-question on relevance was: What role has MTBBE played in terms of improving educational policy with respect to the ethnic minority people of the country?

MTBBE has played an outstanding role in improving educational policy on bilingual education for ethnic minority people It has implemented an MTBBE design to show the advantages of this approach, and it has become a benchmark practice inspiring innovation

in educational policy MTBBE has gained the attention and receptivity of educational authorities and policy makers toward bilingual education, which is key to enlarge the legal space this approach needs for its expansion

MTBBE was organized to play a key role for improving educational policy related to ethnic minority people of Viet Nam First, the AR MTBBE has implemented a valid and feasible design of bilingual education in EMLs and Vietnamese in pre-school and primary school The design was a reference experience to show the advantages of the approach in terms

of learning outcomes The experience included training and advocacy addressed to policymakers, education managers, principals, teachers, students and communities to attain their understanding and support to MTBEE Simultaneously great efforts were made

on developing teaching, learning, reference and advocacy materials in EMLs Second, the

AR MTBBE contributed to the development of policies and practices for promoting the use and development of EMLs The next sections on effectiveness and impact will examine evidence in this regard

Some outstanding results of this commitment were the following:

• In 2010 the provincial authorities of Lào Cai decided to scale up the MTBBE approach It was a provincial initiative with the potential to include other ethnic minority communities (see Annex 4, Box 2) It showed an open-minded attitude of education authorities to the new approach

• The AR MTBBE gained momentum and the attention of the National Assembly Office (see details in Section 4.2.6)

• In 2012 a Primary Classroom Language Mapping exercise was implemented in Lào Cai This was an important information tool for policy makers and sustainability

of MTBBE that MOET and UNICEF agreed to produce30 upon the request of the Lào

30 2011 Annual Work Plan

29

Cai Department of Education and Training and Primary Education Department

(MOET) The initiative focused on generating more accurate information on language use and identifying locations where the MTBBE can be replicated Beyond this focus, the study was considered useful for supporting the development of a standard Language Mapping tool for Viet Nam and for contributing to the development of the education system, making it more relevant and matching the multilingual nature of Viet Nam.31 Lào Cai has been utilizing the findings from this tool for education sector planning The province of Dien Bien also utilised this tool

to conduct their language mapping In addition The World Bank is now utilizing this tool for its supported project in East Timor

• The 2012 updated version of the EFA Action Plan 2003-2015 recognized the potential of the MTBBE approach to respond to educational needs of ethnic minority groups, and recommends that strategies, such as SEDS 2011-2020 and ESDP 2011-2020 for the period 2011 to 2020, consider the implementation of an inclusive development strategy for the benefit of disadvantaged children.32 A component of this strategy would be the delivering of mother tongue-based bilingual or multilingual education and its scaling up to other provinces.33 For some testimonies on this issue see Annex 6, Box 3 and Box 8

4.1.7 Relevance to UNICEF´s renewed focus on equity

The fifth sub-question on relevance is: To what extent is the MTBBE approach relevant to UNICEF´s renewed focus on equity?

The MTBBE approach is specifically focused on ethnic minority children, which are the most deprived and vulnerable children in Viet Nam MTBBE contributes to remove the language barrier that is considered one of the main causes of low learning outcomes, school absenteeism and dropout During adult life this educational disadvantage turns into other social disadvantages in the labour market, in the access to and use of services, and others Given this commitment, the MTBBE approach is strongly relevant to UNICEF´s renewed focus on equity, by which it tries both to reach the most deprived and vulnerable children and remove barriers to service provisions, particularly in the case of the poor and marginalised34

Since ethnic minority children face a language barrier to learning at school in Viet Nam and

in other spheres (labour market, business, social services), actions to overcome this barrier become relevant to achieve equity For UNICEF´s focus on equity, AR MTBBE is a key project, particularly as a vehicle to promote inclusive policies on education and other social spheres The following is a testimony on this perspective:

30

For UNICEF equity is a global level issue, not country specific Viet Nam has a different context In this country there is a good investment in education The Government has developed a universal primary education However, about 4% of the target population is outside the education system These are children from ethnic minority groups, children with disabilities, children who are out of school due to socio-economic reasons and children of immigrants, very poor people This is why there is a focus on inclusive education with broad meaning of a functional education system that would include all children regardless of their background, ethnicity or development status The MTBBE approach was part of this effort together with inclusive education for children with disabilities A direct work with poor children or migrant children will be the next step of the UNICEF education programme in the future 35

In sum, The MTBBE approach offered a pertinent solution to overcome the language barrier in an alignment to international, national and sectorial legal frameworks as well as policy and development strategies However, the existing policies and laws in Viet Nam are lacking clear and explicit statements about the use of EML as media of teaching and learning, which would facilitate the expansion of MTBBE

4.2 Effectiveness

4.2.1 Defining effectiveness

The OECD defines effectiveness as “a measure of the extent to which an aid activity attains its objectives” Two questions are typical in this regard; these are:

 To what extent were the objectives achieved or likely to be achieved?

 What were the major factors influencing the achievement or non-achievement of the objectives? 36

4.2.2 Summary on effectiveness

The general question on effectiveness was: How effective was the MTBBE approach to improving both the learning outcomes of participating children and the capacity of teachers and education managers? Were contextual factors taken into account in the approach design/ implementation of interventions?

Finding No.2 The Programme has been effective in developing and implementing a context–appropriate MTBBE model that has created conditions to improve the learning outcomes of participating children and the capacity of teachers and education managers

The effectiveness criterion makes reference to the attainment of the following Programme objectives: (i) implementing BE models in EML at pre-school and primary levels, and (ii) contributing to policy making capacity in education at local and central levels Turning those objectives into reality has required inputs and actions regarding curriculum design, teaching and learning materials development, and capacity/ improvement of teachers and education management, to mention the most important activities National standards were met in the case of curriculum and educational materials By offering the programme from

35 UNICEF Official from the Education Section

36 OECD Network on Development Evaluation, n.d page 13

31

pre-school to the end of primary education and adopting a flexible approach, the curriculum design allowed the development of both the students’ mother tongues and Vietnamese right from the first day of schooling

as well as offered optimal conditions for transfer of knowledge from one language to the other

The in-service training of teachers has developed/ improved mastery of innovative BE teaching methods and self-learning skills, as well as action research competences Educational managers were involved in activities oriented to improve their EML knowledge, comprehension and support, and in this way, increase their potential for policy making at different levels

The AR MTBBE considered contextual factors at both stages, i.e design and implementation In meeting challenges from structural factors the MTBBE had an effective advocacy component Right from the preparation phase and throughout the pilot phase, there were awareness campaigns and lobbying initiatives chiefly targeting parents, communities, politicians, policy makers, donor agencies and NGOs As a result of these advocacy measures, key stakeholders got acquainted with the basic principles of MTBBE and/or understood the relevance of this approach to the Vietnamese context The attention and support devoted to the programme by participating communities, key policy makers and other members of the public can be regarded, at least in part, as a result of the effectiveness of the advocacy component of the AR MTBBE

Effective?

To improving learning outcomes?

To improving capacities of teachers and education managers

To consider contextual factors

The MTBBE model was implemented mainly by the following actions:

 Designing of the MTBBE pilot Action Research

 Curriculum development according to MOET standards

 Developing supporting material according to MOET standards

There are other actions oriented to the implementation of the MTBBE models, e.g training

of teachers and education managers, which are being considered under the second question on effectiveness (Section 4.2.4)

sub-MTBBE models: Have they been successfully implemented and assessed?

32

There was no question that the AR MTBBE model, with its three variants, was implemented and there was also good evidence about actions on monitoring and assessment

The MTTBE went through two stages, a first one on designing and a second one on implementation The designing stage started in 2006 while the implementation stage commenced in 2008 in selected pre-primary and in 2009 in primary schools in Gia Lai, Trà Vinh and Lào Cai provinces with three EM languages: J´rai, Khmer and Mong.37

In preparation of the implementation stage, several actions on curriculum design and teaching and learning material development were carried out As planned, the first cohort

of students entered pre-school in September 2008 in eight selected schools of the above mentioned provinces During the next school year the second cohort of students entered pre-school while the first was ready to start First Grade

By June 2014, the first student cohort was finishing primary school (the Sixth Grade), this way completing the period designed for the AR MTBBE programme As expected, the second student cohort also finished Sixth Grade in 2015

The AR MTBBE anticipated actions to guarantee the models would be under permanent monitoring and assessment These actions were the responsibility of technical staff and education managers at central, provincial and district levels

Students´ progress was assessed annually, which provided valuable data about the impact

of the programme on learning outcomes.38 The first student assessment was carried out in September 2008 collecting baseline data for the first cohort of pre-primary students The assessment focused on five development components in both the mother tongue and Vietnamese39 A second assessment, carried out in May 2009 replicated the preceding study to find that “significant progress in all five components” had occurred Teachers and learning material were also objects of this last assessment The assessment also extended

to a sample of non-MTBBE students since 2011 As a result, valuable information was collected and used to improve textbooks, provide insights for teacher training, and strengthening the teaching-learning process.40This participative monitoring process turned into a routine activity of the Primary Education Department in collaboration with the RCEME These two entities were aware of the importance the annual assessments had as a

37 MoU 15 August 2007, page 1

38 See for instance, UNICEF-MOET The Programme Brief 3 of September 2012, page 12

39 The following were the five assessment components in mother tongues: (a) listening comprehension of spoken language; (b) understanding simple concepts; (c) understanding antonyms; (d) arranging pictures and story-telling according to pictures; and (e) recognizing mother tongue alphabets The five components

in Vietnamese were: (a) listening comprehension of spoken language; (b) knowing the names and

functions of parts of the body, objects, trees and animals; (c) implementing commands; (d) naming

people/animals and their activities; and (e) arranging pictures in sequence (MOET-UNICEF, 2009, pages 4-5)

40 MOET-UNICEF, 2009, pages 6-7

33

tool to demonstrate the importance of transitioning from mother tongue to Vietnamese as well as to control the quality and effectiveness of the programme Starting with the Assessment Report 2012-2013, the objective of setting up a database for the project became manifest41 As observed in the different assessment reports, each indicator was constructed according to methodological procedures commonly used by education specialists around the world

Programme actions attained the purpose of having the AR MTBBE models operating as planned How successful these models were in improving MTBBE students´ learning outcomes is a question to be discussed in Section 4.3, which corresponds to the impact criterion

Curriculum development: Did it follow MOET standards?

The MTBBE programme was based on a curriculum design, developed by national and international experts and in line with national standards MOET´s adoption of this design and the fact that it yielded students with appropriate competences for their grade level indicate its success in terms of meeting national standards By offering the programme from pre-school to the end of primary education and adopting a flexible approach, the curriculum design allowed the development of both the students´ mother tongues and Vietnamese right from the first day of schooling as well as offering optimal conditions for a smooth transfer of knowledge from one language to the other

Curriculum development was one key initial activity of the AR MTBBE To lead this process

a Bilingual Curriculum Design Team was formed during the last quarter of 2007, which comprised key national technical experts42 In supporting this team, UNICEF contracted an international specialist to work directly with the Viet Nam counterpart and Carol Benson, the main international advisor in the design of the Action Research MTBBE

The core task of the Bilingual Curriculum Design Team was to develop a bilingual curriculum framework based on the national standards for the EML That is, at the end of grade 5 students must meet the national standards In fact, the current Vietnamese standards would be the EML standards, and the Vietnamese standards should represent appropriate levels for second language learners.43

The Bilingual Curriculum was approved in April 2008 The Curriculum Framework identified eight (8) “Strands” and their respective “sub-strands” for the Mother Tongue and Vietnamese Each strand was then detailed for pre-primary and each grade of the primary

41 Report of the Survey Results of Grade 3 and 4 on Learning Outcomes of MTBBE Students Academic Year 2012-2013 Hanoi, 2014, page 3

42 The team was led by the Research Centre for Ethnic Minority Education and benefitted from the

participation of about 30 persons at central and provincial levels

43 MoU 15 August 2007, Annex 3: Research Design, page 4

Teaching and learning materials: Do they meet MOET standards?

Developing teaching and learning materials followed the MOET standards since its

Research Centre for Ethnic Minority Education (RCEME) led this process directly UNICEF

contracted Andy Smart, international expert in bilingual material development, to support the national team at the beginning, including the training and development of a material development strategy The strategy for the development of teaching and learning materials was flexible, experimental and grounded in the curriculum framework

In the development of the supporting material, language aspects were considered, such as: (i) Narrative and imaginative, (ii) conceptual, (iii) phonological, and (iv) social and/or communicative

Based on those aspects RCEME proposed a minimum requirement of materials to be supplied by the project.46 By May 2013, 543 copies had been printed; almost 60 copies for preschool students and almost 20 copies to train teachers About 2,900 copies of textbooks for all primary grades in subjects like Mother Tongue (textbook and a workbook for writing), Vietnamese, Mathematics, Social Sciences, Natural Sciences, History, and Geography were printed and delivered.47

RCEME instructed the developers of these materials to base their work on general principles, such as, culturally adapted, close to children´s experience, consistently integrating different learning levels, and without complicated concepts

44 M Franken (July) 2008 Curriculum Framework (July 5)

45 ToR UNICEF Consultancy Notice Edu 2015-01NOTICE EDU 2011-01

46 These materials consisted of “big books” (10 textbooks series per language), picture books (10 textbook series per language), wall charts (10-12 textbook series per language), teacher´s methodology guides (one series per language), oral literature resource books (one series only in mother tongues), Jigsaw puzzles, and letter cards and picture cards (flash cards)

47 MOET Annual Progress Report 2013