Improving Learning in South African Schools pdf

LIST OF FIGURES AND TABLESFigures Figure E: Grade 9 mathematics classroom-functionality levels in 2004 and change Figure I: National mean mathematics scores for QLP and control schools F

Improving Learning in South African Schools:

The Quality Learning Project (QLP) Summative Evaluation (2000 to 2004)

A Kanjee & CH PrinslooAugust 2005

HUMAN SCIENCES RESEARCH COUNCIL

Compiled by the Assessment Technology and Education Evaluation Research Programme, Human Sciences Research Council

Funded by the Business Trust

Intervention Programme Managed by JET Education Services

Evaluation by the Human Sciences Research Council

in writing from the publishers.

ISBN 0-7969-2145-8

Cover Design and Layout: Vinesh Naidoo

Production: Shereno Printers

Distributed in Africa by Blue Weaver

PO Box 30370, Tokai, Cape Town, 7966, South Africa

Tel: +27 (0) 21 701 4477

Fax: +27 (0) 21 701 7302

email: orders@blueweaver.co.za

www.oneworldbooks.com

Distributed in Europe and the United Kingdom by Eurospan Distribution Services (EDS)

3 Henrietta Street, Covent Garden, London, WC2E 8LU, United Kingdom

Tel: +44 (0) 20 7240 0856

Fax: +44 (0) 20 7379 0609

email: orders@edspubs.co.uk

www.eurospanonline.com

Distributed in North America by Independent Publishers Group (IPG)

Order Department, 814 North Franklin Street, Chicago, IL 60610, USA

CONTENTS

LIST OF FIGURES AND TABLES

Figures

Figure E: Grade 9 mathematics classroom-functionality levels in 2004 and change

Figure I: National mean mathematics scores for QLP and control schools

Figure J: National mean language scores for QLP and control schools

Figure M: Index scores for teacher-intervention coverage and quality by year,

Tables

Table I: Indicators at Grade 12 level of the success of the QLP (from 2000 to 2004) xiii

Table C: Change in Grade 12 learner performance between 2000 and 2004

Table D: Indicators at Grade 12 level of the success of the QLP (from 2000 to 2004) 9

The hard work and commitment of many dedicated individuals and organisations have over five years made itpossible to bring the formal QLP evaluation to a close by publishing the final, summative report All contributionsare appreciated and are hereby acknowledged

The Business Trust is acknowledged for providing the funding for this project

Acknowledgement is made of JET Education Services and the Business Trust for managing the QLP and forproviding valuable support and assistance through the project Steering Committee and other mechanisms NickTaylor, Anele Davids, Hemant Waghmarae, Jackie Moyana, and Leigh-May Moses, all from the QLP programmemanagement team of JET Education Services and Charles Barnard, Brian Whittaker, Mdu Ndhlovu and NomfundoMqadi, with Theuns Eloff at the outset, from Business Trust, all deserve special mention They contributedcontinued guidance throughout the project, with considerable effort and time spent on many rounds of comment

to draft versions of instruments and documents of many kinds, and especially the baseline, mid-term and currentsummative reports

The Department of Education is acknowledged for its continually increasing role in direct communications andmeetings between the HSRC and participants at national and provincial level (at the Director General’s office andthrough strategic planning sessions, respectively)

Consortium members and service providers are thanked for valuable inputs at various stages and throughfeedback sessions, Partners Forum meetings, and otherwise

Acknowledgement is made of the fieldworkers and observers, especially through contracts with AC Nielsen,assisted by Mictert in 2002, and the many qualified teachers, as well as the data-capturing team of Datanet underthe guidance of Pio Combrink

Professor Johann Mouton is thanked for his comments and ongoing advice, mainly on methodology, during thefirst half of the study

Special mention has to be made of the contributions by district managers, regional or circuit managers(institutional development officials), learning area specialists, school principals, teachers, learners and theircaregivers, for allowing researchers into their institutions, offices, classrooms and lives, and for makingthemselves available for observations, interviews, the completion of questionnaires, and for making learnersavailable for sitting for performance tests

The role of and contributions by the official QLP co-ordinators in the office of JET Education Services are alsoacknowledged The QLP co-ordinators became an institutionalised channel through which certain business wasconducted just so much more efficiently Access to the districts and schools, and district-level and interventioninformation collection are two cases in point [The co-ordinators were: Alfred Mabina (Gauteng); Kedibone Boka(Mpumalanga); Darwin Solomon (Northern Cape); Noel Daniels (Western Cape); Samuel Nkosi (KwaZulu-Natal,Inanda and Ixopo districts); Thulani Dlamini (KwaZulu-Natal, Ubombo district); Rose Machobane (North WestProvince); Nosipho Nxiweni (Free State); Vuyani Mrwetyana (Eastern Cape); and Maxwell Malatji (Limpopo).]Marcel Croon, professor at Tilburg University in the Netherlands, is thanked for providing invaluable assistanceand training in 2002/3, and again in 2004/5, with the data modelling and analysis, and with related software

Shereno Printers must be mentioned for printing and packing research instruments within very tight schedulesduring the three evaluation years, and also for producing the mid-term technical report, this summative report,and its technical companion report.

Wordsmiths English Consultancy is acknowledged for language editing, formatting and laying out the manuscript

of the mid-term technical report in 2003, and for language editing this summative report and its technicalcompanion report in 2005

Professor H.S Bhola, Professor Brahm Fleisch, and Hersheela Narsee are thanked for reviewing the finalmanuscript and for their helpful comments and assistance in improving it

The following HSRC team members (listed alphabetically) are also noted with gratitude for their respective roles

as part of the evaluation team at different points in time during the final evaluation phase following 2003:1BrutusMalada, Carla Pheiffer, Elsie Venter, Gerda Diedericks, Godwin Khosa, Heidi Paterson, Hendrik de Kock, LeratoMashego, Lolita Winnaar, Makola Phurutse, Matthews Makgamatha, Natalie le Roux, Nicolaas Claassen, SannieReyneke, Sophie Strydom, Vijay Reddy, Xola Mati and Zinhle Kgobe

This report marks the end of a unique, long-term and extensive teaching and learning intervention programme,the Quality Learning Project (QLP) In concluding the evaluation activities of the QLP, a reflection on theevaluation processes and findings is desirable For this reason, some of the most important functions and roles ofthe QLP and its evaluation are placed in perspective In doing so, the report emphasises the crucial nature andfunction of evaluation for teaching and learning, more so in view of the transformation context of the SouthAfrican education system

At the most apparent and immediate level, this summative report provides a conclusive account to the sponsors

of the QLP of how successfully the funds of the project have been spent

Additional value also lies in reflecting more deeply on the complexities inherent in large-scale and lengthyendeavours such as the QLP These reflections carry a positive verdict about the methods and models selected forthe QLP intervention programme and its evaluation Finally, the reflections allow affirmation of the policydecision implicit in undertaking the QLP work at the outset In this regard, professional and policy experts canfind justification in the soundness, replicability and sustainability of the road travelled by the QLP

This report also provides background information on the interventions as well as on the evaluation design andmethodology of the QLP, its findings, and the conclusions and recommendations derived from the findings

ABBREVIATIONS USED FOR THE QLP DISTRICTS

JSM = Johannesburg South Mega

WCME = Western Cape Metropole East

Map showing location of Quality Learning Project (QLP) districts*

* District labels appear next to markers indicating the centroid of the particular district It has to

be noted that for certain districts, such as Zeerust and Karoo, actual district areas may be quite large

E S

EXECUTIVE SUMMARY

The summative report of the Quality Learning Project (QLP) encapsulates the evaluation work of the projectspanning the past five years In addition, it focuses on the successes and findings of the QLP, and the implicationsand recommendations flowing from the evaluation study The details underpinning the summative report can befound in the technical companion report

The success of the QLP

The QLP adopted a specific theoretical model for interventions designed to improve learning and teaching inschools, and for evaluating the success of these interventions As such, the hierarchical levels of the system(districts, schools and classrooms) were taken into account Observations were made at three points in time tostudy trends and causal patterns Comparisons were also made between project and control schools

Performance targets for the QLP were set at the outset QLP schools were to show an improvement, measured byoverall learner performance, against a comparable sample, by the end of 2004 What was required was:

• A 10% improvement in mean overall matriculation pass rate;

• A 10% improvement in mean mathematics pass rate; and

• A 10% improvement in mean English Second Language pass rate

However, pass rates, when used as sole indicators, have certain weaknesses For example, small increases fromlow baselines (previous poor matriculation results) appear as large improvements Moreover, schools are able toartificially inflate Grade 12 pass rates by holding back potentially unsuccessful Grade 11 learners or by requiringlearners to take subjects at the Standard Grade (SG) These targets were therefore refined after the 2002 mid-termevaluation, using categories that more reliably reflected school-performance outcomes These categories were:

(a) The increase in the absolute number of learners passing, as an indication of the quantitative improvement

of learner results;

(b) The increase in the number of learners passing with university exemption, and with mathematics at

Higher Grade (HG), rather than Standard Grade (SG), as an indication of an improvement in quality of

the learner results; and

(c) The increase in matriculation pass rate, as an indication of improved efficiency in learner results.

Table I compares the performance of QLP evaluation schools with that of control schools in terms of thefinal evaluation criteria adopted It shows that the matriculation results of QLP schools consistentlyimproved more than those of control schools with respect to quantity, quality and efficiency

Table I: Indicators at Grade 12 level of the success of the QLP (from 2000 to 2004)

Selected indicators at school level (Grade 12) Percentage points by which improvement

in QLP schools is higher than in control schools

QUANTITY OF OUTPUT

QUALITY OF OUTPUT

EFFICIENCY OF OUTPUT

# The very low baseline of 6 in QLP schools increased by 55, resulting in this high percentage point increase For control schools, 133 was reduced by 10 to show a decline of 7.52 percentage points.

* QLP schools were discouraged from having an increase in the number of learners in this category; hence, the change in quality of output was not significantly higher than that obtained in the baseline study.

Trends and causal patterns

The QLP evaluation focused on identifying the impact of the intervention programmes on district, school andclassroom functioning as well as on learner performance

Functionality within QLP districts improved at each level for the period from 2002 to 2004, as is evident from

the findings outlined below:

• Overall district functioning improved over time, but did not surpass moderate functioning levels The

strongest improvements (by more than 10% over the duration of the project) were in the design and use

of job descriptions, financial management, within-district planning, support planning, and support implementation

school-QLP schools fared much better than control schools in:

• Overall school functioning, with the greatest contributions coming from the supply and use of resources

and facilities, learning support materials (LSMs), curriculum leadership, school management, and schooladministration;

• Aspects of mathematics teacher and classroom functioning, including teacher competency (mainly

experience and qualifications) (Grade 9), curriculum coverage (Grade 11), lesson pedagogy (Grade 9),access to and use of LSMs (Grade 9), classroom practices (Grades 9 and 11), and homework practices(Grade 11);

• Aspects of language teacher and classroom functioning (with a focus on reading and writing), including

teacher competency (mainly experience and qualifications) (Grade 9), curriculum coverage (Grades 9 and11), pitching lessons at the appropriate level for learners (Grade 9), lesson pedagogy (Grade 9), andclassroom practices (Grades 9 and 11); and

• Overall teacher and classroom functioning, at the level of Grade 9 mathematics only (Language

classroom and teacher functioning, at the levels of Grades 9 and 11, also improved over time, whilemathematics classroom and teacher functioning remained stable, but at a high level of functionality.)

With regard to learners, two observations are warranted:

• Learner perceptions and attitudes fluctuated and even sometimes declined over the period of the project.

• In terms of learner performance at the national level, a marked improvement was observed only in the

case of writing skills (language) for Grade 11 learners Learners fared better in selecting correct informationfor mathematics and for reading and writing than they did in constructing responses, especially wheredemands were more abstract and challenging Learners who took the test in their home language (English

or Afrikaans) obtained higher scores Similar trends were noted in the control schools, although theirscores were always higher given that the control schools were not part of the QLP sample (and thus weredeemed not to require any interventions)

The outcomes of the causal modelling and analyses of the programme’s effects, between 2002 and 2004, are

indicated below

• There has been consistency over time with regard to interventions, functioning and learner performance

across all levels, subjects and grades, indicating that critical mass and impetus, once achieved, can besustained

• There are many indications that service providers targeted interventions dynamically and interactively in

areas that needed them most

• Interventions improved functioning in areas targeted by the QLP This is evidenced in improved school

functioning driven by good classroom and teacher interventions District interventions also played a role

in improving school functioning

• Improved functioning within the QLP led to better learner performance in many instances This applied

especially in cases of school and teacher/classroom functioning

• QLP interventions led to improved learner performance in some cases The positive effects of district

interventions on Grade 11 mathematics performance during 2003/4, and of language-teacherinterventions on overall matriculation pass rates in 2003/4 are especially significant (bearing in mind thatlanguage interventions focused on language across the curriculum)

• The dosage and quality of QLP interventions were subject to the risk of fatigue effects over time, making

improvements harder to sustain District coverage and Grade 9 language-teacher interventions were theexceptions

The complete technical report and the summative report contain detailed recommendations emerging from thestudy A selection of the most significant recommendations is provided below

• Policy makers need to sustain and enhance the programme’s benefits This can best be done by adopting

logical, structured, integrated and comprehensive approaches to school development, based on soundtheoretical principles For example, greater attention needs to be paid to the role of language in learning,given that reading and writing skills, abstract thinking, and producing meaning are central learningobjectives In addition, resource allocation should be prioritised according to need It is especiallyimportant to monitor and evaluate all policy effects

• Education planners need to provide integrated and coherent intervention plans to improve teaching and

learning Such plans should earn the support and commitment of participants through appropriateengagement, and should adhere to sound frameworks and models Interventions should target the earlierstages of school life rather than focus on matriculation learners

• Education managers in districts and provincial offices should sustain their efforts to manage their

school-support and monitoring roles Visionary supervision, mentorship and leadership are required, with dueregard for capacity, infrastructure and process

• Curriculum developers should support teaching and learning by producing relevant, practical, and

high-standard learning content underpinned by solid foundational knowledge

• School management teams (SMTs) should nurture the professional development of their teachers through

good mentoring and motivation Sound management, discipline, and curriculum leadership (the heartbeat

of the school) are crucial to this process

• The provision of adequate numbers of excellent teacher trainers and mentors (including learning area

specialists (LASs) or subject advisors) can no longer be neglected These are needed to support teachersthrough mentoring, motivation, and technical (learning area or subject) assistance

• Classroom teachers should not compromise on teaching time and curriculum coverage The dignity of

teacher-learner relationships and interactions, discipline, and the provision of sufficient facilities andlearning materials all have to be maintained Lack of subject expertise should not be allowed to kill theinherent curiosity of learners and the joy and fun of learning Commitment and passion characterise goodrole models

• Parents and learners should pursue every opportunity for reading, and should value all learning.

• Funding agencies should support large-scale, long-term and complex interventions and evaluations similar

to those undertaken by the QLP, given the sheer scale of the challenges of the education system

• Evaluators should pursue sophisticated methodologies, designs, methods, models, data management and

statistical analyses, in keeping with complex programme characteristics Secondary analysis, capacitydevelopment, and evaluation should be supported

• Book publishers should seek to disseminate findings from and information on interventions and studies

such as this one

In order to ensure that schools obtained effective support and monitoring from districts and that the goodpractices gained from the project would be institutionalised, the programme also focused on the development ofdistrict systems and officials.

In improving the quality of learning outcomes, the QLP adopted a systemic approach, which entailed improving:

• Learning outcomes in the languages of instruction and mathematics in Grades 8 to 12 in 524 schools;

• The teaching of mathematics, and reading and writing skills in 524 schools;

• The effectiveness of governance and management in 524 schools; and

• The effective management of 17 district offices in the nine provinces

The QLP aimed to achieve the above by developing management capacity at district and school levels, and byimproving the classroom skills of teachers to enhance learner performance

During the first two years of implementation, the key outcomes of the QLP were streamlined so that:

each provincial cohort of the QLP schools would, by the end of 2004, show an improvement in school

performance measured by overall learner performance with special emphasis on:

• A 10% improvement in mean overall matriculation pass rate;

• A 10% improvement in mean mathematics pass rate; and

• A 10% improvement in mean English Second Language pass rate,

against a comparable sample of control schools drawn for the province (JET QLP proposal)

Framework for the Evaluation Study

The final evaluation framework for the formative and summative evaluation studies was derived from the baselineevaluation model applied to the QLP Before the end of 2002, amendments were made to the original evaluationmodel, and integrated into the theoretical position that underpinned the intervention programme and evaluationproject The original hierarchical process model for evaluation was improved through these amendments Thisled to the addition of control schools, the extension of observational data to all sites, the concomitant reduction

in evaluation sample size, as well as various improvements and amendments to the instruments used for datacollection (In addition, some factors associated with the availability and coverage of intervention data for theperiod 2000 to 2002 determined that the effect of interventions would only be modelled for the purposes of thesummative evaluation.) The QLP therefore was a theory-driven intervention, and was underpinned and informed

by the following model:

IF the demands [to perform better] on the school and teacher are increased AND we enable the district

to provide high quality support to the schools AND we train the school governing bodies and schoolmanagement teams (SGBs, SMTs, etc.) to manage their schools more effectively AND we train theteachers to teach mathematics and the languages better, THEN we should get improved TEACHINGQUALITY IN THE CLASSROOMS which WILL LEAD TO IMPROVED LEARNER PERFORMANCE

F E S

The interventions made by the programme at various levels of the education system (district, school andteacher/classroom) were aimed at building institutional capacity to manage, support and monitor educationalactivities between the district and the school, as well as between the school and the teacher

Figure A demonstrates the articulation and operationalisation of each of the three levels of the QLP It indicatesthe primary expected outcomes at each level, and how these outcomes were defined operationally The figurealso shows which indicators were identified as relevant and measurable, and finally points out the sources andmethods of evidence that had to be utilised at a given level

The theoretical model provided conceptual coherence and integrity for the QLP interventions, as well asanalytical guidance for its evaluation

Figure A: The QLP model at district, school and teacher level

EFFECTIVE FUNCTIONING OF DISTRICT OFFICE

• Effective organisationaldevelopment, planning andmanagement

• Effective HR management

• Effective financial management

Effective school support

Effective school monitoring

Effective school development planning

EFFECTIVE FUNCTIONING OF SCHOOL

• Effective school management

• Effective human resourceperformance monitoring

• Effective school administration(tracking of learners)

OUTCOMES

• Improved learnerparticipation in class

• Improved learnerperformance

EFFECTIVE CURRICULUM MANAGEMENT

• Monitoring delivery ofcurriculum

• Improved assessment practices

• More effective use of LearningSupport Material (LSM)

E M R D

Evaluation Methodology and Research Design

The QLP Steering Committee, comprising staff from JET Education Services, the Business Trust, the Department

of Education (DoE), and the HSRC (through representatives from the evaluation team), accepted the improvedlogic model for the QLP, which made explicit the details at the different levels, focused the evaluation, andintegrated the evaluation with the intervention programme

A core feature of this project was the establishment of an explicit and common framework to ensure that allintervention and evaluation activities would be aligned over time For the evaluation study, a disadvantage wasthat some continuity was sacrificed between the initial evaluation model and the new evaluation framework Theformer had already served as the basis for the HSRC’s instrument development and data collection for baselinepurposes The longer-term benefits of these changes, though, outweighed the short-term delays anddiscontinuities related to the baseline study

A first implication of the change from the baseline study was that many of the instruments had to be amended.Although strong efforts were made to keep core design aspects as consistent as possible, the changes did implythat specific questions were improved upon The changes also meant that certain sources of data or the ways inwhich data was collected were adjusted to modes that were considered more likely to render reliable and validfindings (for example, a shift was made from questionnaire responses to observations)

A second change to the earlier evaluation studies was the evaluation sample The reduction in the number ofexperimental schools to 70 in 2002 from 102 in the original 2000 baseline sample implied a certain loss in datacomparability over time, but brought the benefit of added commonality (and with it the opportunity fortriangulation) between the data sources underpinning the self-reporting and performance measurementinstruments, and the case studies and observations

Thirdly, the revised evaluation model included 16 control schools from across four of the QLP districts Thisfeature had not added value during the mid-term evaluation, but became important for the trend analyses from

2002 to 2004 The inclusion of control schools in the evaluation assisted in the evaluation of the performance ofthe experimental schools undergoing interventions against the control schools, which did not receive anyinterventions As a result, the initial disadvantages of the loss of continuity in monitoring programme impact atthe mid-point of the programme were turned into advantages for the final summative evaluation phase in 2004.Final 2002 and 2004 evaluation sample figures are shown in Figure B and Tables A and B

Figure B: Number of schools in the evaluation survey and case-study samples

Total QLP schools - 524

Experimental schools

70

Control schools 16

E M R D

Table A: Total sample obtained for mid-term and summative evaluations

-Table B: Number of schools sampled per district 2

E M R D

A useful component of the summative evaluation was the availability of sound, useful and complete interventiondata However, the conversion of information for quality control and monitoring purposes into intervention datafor the mid-term evaluation and report early in 2003 proved to be an insurmountable task As a result, it wasdecided not to incorporate correlation analyses in the mid-term report, but rather to re-design the format in whichthe intervention information would be collected This was also done with a view to converting the informationinto indices and indicators in the same way as the other QLP evaluation data had been converted

Against this background, it comes as no surprise that the lessons learnt as the programme interventions andevaluation processes unfolded were many One could not expect much different from a first venture of this kind,

in terms of content, rigour and scale Thus, the expectation that the summative report would provide manyvaluable findings has not been unfulfilled

The characterisation of the HSRC’s role as both “independent” and “formative” evaluator proved to be a useful,but sometimes contradictory, requirement In the formative sense, participation often occurred to share earlyinsights, which were used to direct subsequent interventions and their implementation This participation tookplace immediately after the baseline and mid-term reports, during many feedback forums, strategic planningsessions, programme management and steering committee meetings, and service provider discussions Suchcontact also had the benefit of sensitising the evaluation team to many of the dynamics of the QLP programme.This increased the understanding of the evaluation team about many of the issues at stake, and of how to collectreliable and objective data on such issues It also conveyed much insight into the process and benefits of ascientific evaluation to future beneficiaries and other stakeholders However, some contamination of objectivitymay have occurred, as respondents often formulated ideas, some realistic and others not, about desirableoutcomes anticipated by the evaluators

The evaluation team had to surrender some control of the approach followed in the collection of data Forexample, intervention data was obtained from the QLP management team, while district-level information wascollected by QLP provincial co-ordinators This was in some ways a useful occurrence, as liabilities associatedwith strict independence, such as alienation and the non-credibility of the evaluator, could have led to losses interms of access to, and collaboration with, the programme participants

All said, a reasonable compromise on the processes for the summative evaluation was achieved The complexityand magnitude of both the intervention programme management as well as the evaluation process should not beunderestimated The instrument development, data management, and analytical skills required were extensive,

as were the skills required for research design, sampling, and methodology in general Continuity and criticalmass in terms of human resource expertise were therefore central to the success of the project In addition, thelogic model that drove both the interventions as well as the evaluation provided a unifying approach to theproject that secured coherence and integrity for the whole venture

E C

Evaluation Criteria

One could at the outset, in addition to the path analysis that the HSRC undertook, compare the QLP(experimental) and control schools in terms of available information, although this comparison falls outside theoriginal sampling and design decisions taken at the beginning of the evaluation project Reasonably completematriculation statistics are available for this purpose As a result, and subsequent to the preliminary discussions

and findings presented in December 2003 as a separate report (Grade 12 results of QLP schools: Supplement to

were to determine:

(a) The increase in the absolute number of learners passing their matriculation examinations as well as

English (HG) as an indication of the quantitative improvement of learner results;

(b) The increase in the number of learners passing with university exemption, and with mathematics at

Higher Grade, rather than Standard Grade, as an indication of an improvement in quality of the learner

results; and

(c) The increase in matriculation pass rate, as an indication of improved efficiency in learner results.

One advantage of this approach is that the control and QLP schools could be compared more directly In four ofthe provinces, the initial allocation of schools, and early changes to this allocation, resulted in a small number

of schools falling outside the QLP eventually These schools formed the control-school sample In these cases, itcan be assumed that district contextual and impact factors had been quite similar for the QLP and controlschools, which only or mainly, again by assumption, differed in terms of receiving QLP interventions, or not As

a result, differences in learner performance changes over time can be ascribed to the QLP interventions,especially at school and teacher or classroom levels

For further consistency in terms of the approach followed thus far in the evaluation, it was decided to stick to thesample of 70 QLP and 16 control schools Given that the control schools only existed in four provinces, two sets

of comparisons were made First, direct comparisons were made between QLP and control schools for the fourprovinces in which control schools were available Second, broader comparisons were made between QLPschools in all nine provinces and control schools located in the four provinces

Such analyses would also be in the spirit of initial intentions to keep a close watch on the improvement in thematriculation results of the schools participating in the QLP as one of the criteria set for evaluating the project’ssuccess Another basis for such a comparison is the initial criteria set for evaluating the key outcomes of the QLP4,already cited verbatim in the background to this report

S Q L P

Success of the Quality Learning Project

The report focuses on two questions: did the QLP model and intervention programme make a difference toteaching and learning in terms of both processes and outcomes? (more bluntly, was the QLP worth the effort?)and, secondly, how can the lessons learnt from the QLP be made applicable to other similar programmes?

Table C portrays the findings based on the refined criteria used in the supplementary report released in December

2003 Data for the control and QLP schools is directly comparable for only four provinces However,comparisons were also made between the data of these control schools and the QLP schools from all nineprovinces

The increase in the numbers of learners that had passed was calculated for all QLP schools, then aggregated byQLP district, and totalled for all QLP schools in the nine provinces, and for the QLP schools in the four provinces,where control schools were also present The percentage change from 2000 to 2004 was then calculated in eachinstance The same procedure was followed with regard to the numbers of learners passing Grade 12 with orwithout university exemption (or endorsement), mathematics (HG and SG), and English (HG, Second Language),

as well as the overall matriculation pass rate Towards the bottom of Table C, comparisons are reported between:(1) the QLP schools in all nine provinces and the control schools in the four provinces; and (2) the control schools

in the four provinces and the corresponding QLP schools in these provinces

Table C provides information on the change in Grade 12 learner performance, aggregated by province, from

2000 to 2004 in terms of:

• The number of learners passing their matriculation examinations (e.g., 53 or 10.3% more learners in theEastern Cape passed their matriculation examinations in 2004 compared to 2000);

• The number of learners passing their matriculation examinations with exemption;

• The number of learners passing English HG;

• The number of learners passing mathematics HG and mathematics SG; and

• The overall pass rate

A summary of these results is presented in Table D in terms of the quantity, quality and efficiency indicatorsdeveloped to determine the success of the QLP programme

S Q L P

Table C: Change in Grade 12 learner performance between 2000 and 2004 across QLP and control schools by province

It is clear from Tables C and D that the initial project objectives were met almost without exception (It is useful

to point out that JET Education Services made similar comparisons for purposes of quality control and programmemanagement on information about all 524 schools, taking provincial and national figures and trends from allnon-QLP schools as the comparative basis These analyses also overwhelmingly reflect the outcomes listedabove.)

S Q L P

Table D: Indicators at Grade 12 level of the success of the QLP (from 2000 to 2004)

Selected indicators at school level (Grade 12) Percentage points by which improvement

in QLP schools is higher than in control schools

QUANTITY OF OUTPUT

Number of learners passing matriculation examinations 16.84

Number of learners passing English Second Language (HG) 36.03

QUALITY OF OUTPUT

Number of learners passing with endorsement (exemption) 61.79

EFFICIENCY OF OUTPUT

# The very low baseline of 6 in QLP schools increased by 55, resulting in this high percentage point increase For control schools, 133 was reduced by 10 to show a decline of 7.52 percentage points.

* QLP schools were discouraged from having an increase in the number of learners in this category; hence, the change in quality of output was not significantly higher than that obtained in the baseline study.

In terms of quantity improvements (that is, the overall number of learners passing matriculation examinations),pass rates in QLP schools were approximately 17 percentage points higher than those in control schools Forlearners passing English (HG, Second Language), the difference, when compared across the four provinces thatcomprised both QLP and control schools, was 36 percentage points in favour of the QLP schools However, whencomparing QLP schools in all nine provinces to the control schools from the four provinces, the increase wasslightly lower at 24 percentage points

In terms of quality improvements (that is, the number of learners passing their matriculation examinations withendorsement, and with mathematics at HG), QLP schools managed to achieve increases in matriculationexemptions that were 60 to 62 percentage points higher than those of the control schools The growth trendtowards producing candidates with a pass in mathematics at HG level is also markedly more dramatic in QLPschools, at 332 percentage points (for QLP schools from all nine provinces against control schools in fourprovinces) and at 924 percentage points (direct comparison across four provinces) higher than control schools.However, very low baselines contributed to some inconsistency and to the seemingly large changes The QLPand control schools were rather similar in terms of the increases in the number of candidates passingmathematics at SG level This finding can be seen as positive, in that QLP schools were deliberately encouraged

to prepare teachers and learners to consider offering mathematics at a higher level only

Improvement in efficiency (as based on the overall increase in matriculation pass rate) was higher by eight to 13percentage points in QLP schools above control schools Given that only potentially successful candidates couldhave been allowed through to Grade 12 from Grade 11, this figure is not as meaningful as the figures indicatingquality and quantity improvements Nevertheless, the comparative picture is informative

Pass-rate changes (as shown in the technical report) were also compared for Grade 12 English (HG) andmathematics (SG) In the case of English, improvements in QLP schools were greater than those in control schools

by 19 percentage points (34% improvement in QLP schools above the 15% in control schools) when directcomparisons were made in the four provinces The figure dropped to 11 percentage points when the full QLPgroup in all nine provinces was considered (26% improvement in QLP schools above the 15% in control schools)

In the case of Grade 12 mathematics (SG), improvements in QLP schools exceeded those in control schools by

R QLP E

just over seven percentage points (94% improvement in QLP schools above the 87% in control schools) in directcomparisons made in the four provinces The figure changed to three percentage points in favour of the controlschools when the full QLP group in all nine provinces was considered (the 84% improvement in QLP schools isbelow the 87% in the control schools)

Results of the QLP Evaluation

Analysis was conducted to determine the effect of the QLP interventions on the functioning of districts, schoolsand classrooms as well as on the performance of learners

District-level Functioning

Of all the levels in the education system, the district level seemed to be most directly affected by earlyrestructuring processes in the Department of Education (DoE) As late as 2004, 13 districts indicated that theyhad undergone restructuring, with some indicating that such events had occurred five times It is reasonable toassume that provincial restructuring exercises would have undermined the QLP model and impacted negatively

on key elements for improving the quality of schooling

Figure C shows the change in district-functioning levels from 2002 to 2004 (see scale on the left-hand side) Thesechanges are indicated by the bars in the figure For district functionality, change scores ranged from -6 to +6 Amoderate improvement is observed when all the district scores are combined While district functioning inGauteng, KwaZulu-Natal and Limpopo provinces deteriorated, it showed a marked improvement in the Free Stateand Northern Cape provinces and in the district of Zeerust

Markers (black dots) indicate the level of district functioning in 2004, and represent the index scores (indicated

on the right-hand side in the figure), which range on a scale from 0 to 13 Score values below 4.5 can beconsidered low, while those above 9.5 are high Scores in between this range are considered moderate Mostdistricts still function only at moderate levels, with low functionality noted in the Limpopo province andSedibeng West district (See Tables 3.14 and 3.15 in the technical report for additional details.)

Figure C: District-functionality levels in 2004 and change from 2002 to 2004

R QLP E

existence and use of organograms, which fell in the high category, and appropriate financial managementpractices, which fell in the low category A noteworthy improvement from the 2002 functioning levels is evidentboth in terms of individual indices and the overall district-functionality index

Results of the statistical modelling reveal that QLP interventions affected district functioning (especially during2003/4), as well as functioning at other levels of the system; namely, school and classroom levels Evidence alsoindicates that the programme targeted interventions at those points where the needs were greatest

The QLP model posited that districts drive the improvement of learner performance, mainly through schoolsupport and monitoring The latter specifically covers the quality of school practices pertaining to schoolmanagement and development, curriculum management and development, and teacher-learner interactions inthe classroom

Functioning levels are still moderate for the majority of indicators, with the financial management indicator ofgreatest concern However, the fact that most of the individual indices and the overall functionality indeximproved from 2002 to 2004 is one of the most exciting outcomes of this evaluation

Modelling suggests that these increases can partly be attributed to the interventions (district and other), especially

to the principle of efficiency of resource usage, which was applied by targeting interventions where the need wasgreatest

The “distance” between districts and classrooms/learners is perhaps still quite large, and the implementation of

a district-based school-development model is still in its infancy This situation, paired with continued instabilityand restructuring, and large numbers of dysfunctional districts in some provinces, could explain the absence oflarger or more immediate effects of interventions on the improvement of districts and learner performance

It is recommended that the rationale, model and logic of district-based school improvement be accepted

as sound However, lack of resources and capacity, poor infrastructure, and related issues still pose a vast challenge Despite these challenges, this project indicates that good order, discipline and the will to succeed can make the district level a strong force in facilitating desired change in schools.

Many of the comments made in the mid-term report remain valid, and it is worth being reminded about these:

• Instability, inappropriate structures and lack of resources at the district level of the education system impactdramatically on the transition between the general and further education and training bands

• The lack of parity between the ranks and roles ascribed to various officials in districts across the provincescreates uncertainty, and issues related to reporting lines, hierarchies, and authority need to be clarified so thatinterference with school (teacher) support and monitoring can be avoided

• More critical comparisons are needed between the school-effectiveness and school-improvement approaches(more recently referred to as the outside-in and inside-out approaches to school improvement), especiallywith a view to interrogating existing practices and policy making within the national education system

School-level Functioning

Figure D shows the change in overall school functionality from 2002 to 2004 by district, for QLP and controlschools Change is indicated by the bars in the figure, against the scale on the left-hand side For school-levelfunctionality, change scores ranged from -3 to +8 A moderate improvement in school-level functioning scores is