Learner Performance in South Africa pdf

List of figures and tables iv Executive summary v Acknowledgements ix Abbreviations x 1 Introduction 1 1.1 Structure of this report 2 2 Features of the Quality Learning Project 5 3 Data

Free download from www.hsrcpress.ac.za

Compiled by the Research Programme on Human Resources Development,

Human Sciences Research Council

Commissioned by JET Education Services and funded by the Business Trust

Published by HSRC Press

Private Bag X9182, Cape Town, 8000, South Africa

www.hsrcpress.ac.za

© 2005 Human Sciences Research Council

First published 2005

All rights reserved No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, including photocopying and recording, or in any information storage or retrieval system, without permission

in writing from the publishers.

ISBN 0-7969-2041-9

Cover by Flame Design

Produced by comPress

Distributed in Africa by Blue Weaver Marketing and Distribution,

P.O Box 30370, Tokai, Cape Town, South Africa, 7966, South Africa.

Tel +27 +21 701-4477

Fax: +27 +21 701-7302

email: booksales@hsrc.ac.za

Distributed worldwide, except Africa, by Independent Publishers Group,

814 North Franklin Street, Chicago, IL 60610, USA.

www.ipgbook.com

To order, call toll-free: 1-800-888-4741

All other enquiries, Tel: +1 +312-337-0747

Fax: +1 +312-337-5985

email: Frontdesk@ipgbook.com

List of figures and tables iv

Executive summary v

Acknowledgements ix

Abbreviations x

1 Introduction 1

1.1 Structure of this report 2

2 Features of the Quality Learning Project 5

3 Data preparation, statistical procedures

and methodology 7 3.1 Introduction to the statistical analysis 7 3.2 Data preparation 7

3.3 Data reliability problems 9 3.4 Analytical strategy 13 3.5 School effects 15 3.6 Household effect variables 19 3.7 Removing the household effects from the school environment

and language and mathematics interest and experience variables 21 3.8 The set of explanatory variables 24

3.9 The language and mathematics test scores 24 3.10 Do richer communities have better schools? 27 3.11 Regression of language and mathematics scores on the

explanatory variables 28

4 Commentary on the findings 33

Appendices

Appendix 1 Districts participating in the QLP 35 Appendix 2 The QLP Model 36

Appendix 3 The programmes of the QLP 39 Appendix 4 Sampling methodology 41 Appendix 5 Developing and administering the instruments 46 Appendix 6 Conversion of data from the learner questionnaire into the

list of variables used in this study 52 Appendix 7 Learner background questionnaire 55 Appendix 8 List of schools ordered by general factor, Grade 11 factor

and Senior Certificate 73 References 76

Figures

Figure 1: Survey sample 5 Figure 2: Education of mothers 11 Figure 3: Education of fathers 11 Figure 4: Household wealth vs household income 12 Figure 5: General school factor on household wealth 27 Figure 6: School test factor on household wealth 28 Figure A3.1: QLP programmes 40

Tables

Table 1: Total sample obtained for the baseline fieldwork in 2000 6 Table 2: Language most often spoken at home, by population group 9 Table 3: Education of mother and father, by grade 10

Table 4: Expected level of the household wealth score at different levels of income 13 Table 5: Analysis of variance of school environment, interest and learning experience

scores 16 Table 6: Correlation matrix and factor analysis of the regression coefficients of

school-level variables 17 Table 7: Regression coefficients for Senior Certificate pass rates on general and Grade

11 factors, 1999 and 2000 18 Table 8: Distribution of the use of the language of instruction at home 19 Table 9: Distribution of household scores 20

Table 10: Regression of study aids, meals, parental support, time use and home

reading scores on whether an African language is spoken at home, logarithm

of household wealth, parental support score (time use and home reading only) and a Grade 11 dummy variable 21

Table 11: Regression of interest and experience residuals on home effects variables 22 Table 12: Correlation matrix and factor analysis of the regression coefficients of

school-level variables after the school and household effects have been removed 23 Table 13: Distribution of language and mathematics marks 25

Table 14: Factor analysis of test scores and Senior Certificate results 26 Table 15: Percentage test scores regressed on explanatory variables 29 Table 16: Language scores by quintile 31

Table 17: Mathematics scores by quintile 32 Table A 2.1: Outcomes and covariates at district level 37 Table A 4.1: Number of schools sampled per district 42 Table A 4.2: Number of learners assessed by school district 44 Table A 4.3: Total sample obtained for the baseline fieldwork in 2000 45 Table A 5.1: List of instruments 46

Table A 5.2: Test topics – mathematics, Grades 9 and 11 48 Table A 5.3: Reading and writing skills tested, Grades 9 and 11 49 Table A 5.4: Testing duration of the assessment instruments in minutes 50

iv

©HSRC 2005

List of figures and tables

v

Despite all the difficulties associated with the expansion of its educational system during

the twentieth century, South Africa has done well in systematically lengthening the

average education of each successive age cohort By the 1990s, more than 90 per cent of

the 7–16 age group was enrolled in school, although not every learner was putting in a

full day’s attendance But the quality of the output from the school system has left much

to be desired For many years, close to half of the Senior Certificate candidates have

failed the examination outright And international comparisons, such as the Third

International Mathematics and Science Study of 1995, have given no comfort South Africa

was bottom of both the mathematics and science tables, even though a number of similar,

middle-income countries were included in the study

In seeking to identify the reasons for this situation, it is important to relate educational

outputs (competencies, as measured for instance by Senior Certificate examinations or

standardised tests) to inputs The most obvious input is that of the school itself – the

quality of teachers, facilities and management Another input is that of the household –

the education of the parents, household income and wealth, and support for learners

Education is a joint product of school and home, and learners who are backed by strong

household resources have an advantage The abilities and proclivities of individual

learners is a third input; learners from the same household and the same school can end

up with very different profiles of achievement

Determining the relative contributions of these inputs to educational outputs is not

straightforward Partly this is a data problem The information necessary to carry out a

comprehensive analysis is extensive and usually not fully available Partly it is a statistical

problem: many of the explanatory variables are themselves related in complex ways, so

identifying the true drivers of the situation under analysis is difficult Moreover, in South

Africa very little educational production function analysis has been undertaken, so there

are few landmark results from which one can take one’s bearings It is no exaggeration to

say that educational production function analysis in South Africa is in a preliminary

exploration phase The results of this study must be interpreted in that light

Up until the Quality Learning Project’s (QLP) baseline study in 2000, no South African

data set had ever included test results, school characteristics and information on the

household circumstances of individual learners Before that one could, as in the analysis

of Senior Certificate results, relate results to schools and schools to the communities

within which they were located But one could not relate individual learners to the

households from which they came The QLP data set therefore offers a new analytical

opportunity The research question posed for this study was: what are the effects of

socio-economic variables on educational outcomes in the QLP schools?

Limitations of the study

Before describing the methods and the findings of the analysis, it is important to note two

limitations of this study The first is that, within the time and resources available, it was

not possible to use the full QLP data set This study, therefore, concentrates on the

learner background questionnaire (which elicited information on households) and the

learner achievement questionnaire It did not use (with one small exception) the data

v

©HSRC 2005

Executive summary

©HSRC 2005

from the school or educator questionnaires Of course, the schools attended by learners were identified in the background and achievement questionnaires, so that school effects could be identified But they were not correlated with a wider set of characteristics of schools Thus schools appear in this study through the eyes of learners, rather than through those of teachers or principals

The second limitation of this study is the reliability of the information about households The information collected covered:

• Education of parents;

• An indicator of household wealth, as evidenced by items within the home;

• Language mostly spoken at home;

• How often the language of instruction is spoken at home;

• Frequency of meals;

• Reading at home;

• Parental practices supporting education;

• Time use

Unfortunately, the data on the education of parents turned out to be unusable Preliminary regression results produced incoherent results and checking of average educational levels in the immediate environment of the school from the 1996 Census suggested that educational levels had been exaggerated, particularly by Grade 9 pupils The exaggeration was probably worse at the bottom than at the top It was also not systematic, so that there was no way of working back from reported to actual education The variable had to be discarded – a serious blow, since it is a very important one Its absence makes the findings of this study less certain than they would otherwise be There may also have been some (but less fatal) exaggeration of items contained within the home

The conclusion from this experience is that household variables really need to be collected from adults within the households rather than learners at schools The reporting

of educational levels in population censuses since 1960 has been reasonably accurate and

is certainly much more accurate than the QLP data

The explanatory variables

A great many questions in the learner background questionnaire dealt with the school environment, and the interest and experience of learners in the language of instruction and mathematics in both Grade 9 and Grade 11 These data were combined into nine indices (one for school environment and eight covering interest and experience by grade and subject) Each index measured positive or negative orientation of the learner towards the topic in question

The first question becomes: how much of the variance in indices could be explained by the schools themselves? The answer turned out to be between 10 and 30 per cent The correlations between indices were such that one could identify the most generally favoured schools and relate them to performance in the 1999 and 2000 Senior Certificate examinations Schools approved by pupils generally did better in the Senior Certificate examinations, though the correlation was not perfect So one can conclude that the indices, in part, measured something about the school But the 70 per cent or more of unexplained variance meant that they measured other things as well Some of the remaining variance was explained by differences between households The rest can be

Learner Performance in South Africa

ascribed to individual learner motivation and morale The statistical analysis enables one

to separate out which portion of the indices should be ascribed to which levels

Turning now to the household – the main entity of interest in this study – it quite rapidly

emerged that language use affected test results If the language of instruction was used a

lot in the home, test results rose This really was an issue for households in which an

African language was mostly spoken at home For others, the language mostly used at

home and the language of instruction was likely to be the same The other fundamental

household variable is household wealth One could then start to relate other household

variables to these two Thus:

• The study-aid score rises with household wealth as does the meal score (though at a

given level of household wealth, this is lower for African-language households);

• Interestingly, parental support for study is stronger among African-language

households and rises with household wealth It drops from Grade 9 to Grade 11;

• Time-use patterns conducive to study are likely to be better if an African language is

spoken at home, better as household wealth and parental support rises, but worse

in Grade 11 than in Grade 9 Reading at home rises with household wealth and parental support, but is less extensive in African-language households

By using the household explanatory variables one could remove the effects of households

as well as schools on the school environment and interest/experience indices The

correlation among the purged indices threw up three patterns:

• A general orientation to school;

• An interest in mathematics;

• A positive interest in subjects, but negative experience of them

The analytical approach enabled one to separate out the data into individual-level,

household-level and school-level information, in such a way as to reduce the correlations

between the explanatory variables Other variables were added, namely: gender and age

for grade in the case of learners and urban or rural location in the case of schools

The test results

The test results had the following characteristics:

• They were higher for the language of instruction than mathematics;

• There was a considerable deterioration in language scores between Grade 9 and

Grade 11 This finding is significant because experts, at grade appropriate levels, set the tests The achieved improvement (off a low base: the median mark in Grade 9 was 38 per cent) is not equal to the expected improvement;

• The level of mathematics performance in Grades 9 and 11 was approximately the

same and extremely poor (median in Grade 9 was 7 per cent and in Grade 11 was

8 per cent) Eighty per cent of pupils scored less than 15 per cent in Grade 9 and

16 per cent in Grade 11 Even at the 95th percentile, pupils achieved a mark of less than 30 per cent in both grades

There is a significant but not perfect correlation between school achievement in the tests

and its achievement in the Senior Certificate examinations of 1999 and 2000 Tests and

Senior Certificate results are measuring approximately the same thing There are also

significant correlations between:

vii

©HSRC 2005

Executive summary

• The school general quality factor perceived by pupils and the mean of the logarithm

of household wealth derived from pupils at the school;

• The school achievement quality factor (derived from test results and Senior Certificate results) and the mean of the logarithm of household wealth

Not surprisingly, richer communities tend to have better schools

The relation between test results and the explanatory variables

Language of instruction

Individual level variables:the ones which matter are general positive orientation (which has a positive effect) and the over-age for grade variable (which has a negative effect) The effects are stronger at Grade 9 than Grade 11 The gender coefficient is not significantly different from zero at the 5 per cent level

Household level variables:African home language is a marked disadvantage at the Grade

11 level Whether the language of instruction is spoken at home often is a marked advantage Household wealth exerts a positive effect Study aids and meals exert a positive influence over and above the expected level

School level variables:The school general factor exerts a positive influence Other things being equal, rural schools do better than urban schools

Mathematics

Because the average result is so poor, school and household variables explain considerably less of the mathematics score; individual ability and proclivity is the main cause of such variation in marks This is confirmed by the result when one adds in the language mark residual (after school and home effects have been removed)

Conclusions

The findings of this study can be generalised only for QLP schools and not the national school system Suburban (as opposed to township) schools are entirely unrepresented in the QLP universe QLP schools are drawn from the bottom 70–80 per cent of schools, but they have not been randomly drawn Inferences to the wider school system would be risky Nonetheless, the general finding in this study – that social and economic variables do not play an enormous role in determining performance at the individual level, with the exception of language variables – deserves further investigation Moreover, included in the set of social and economic variables are behavioural variables that can change at existing levels of household incomes and wealth The rules to be followed are simple:

• Feed learners as well as possible;

• Equip them with a full range of inexpensive study aids;

• Talk to them often in the language of instruction

By contrast, household wealth does not give much of an edge in school performance And competence in the language of instruction is valuable not only in itself, but as a means to improved mathematics performance

viii

©HSRC 2005

Learner Performance in South Africa

Nick Taylor and Mark Orkin played a pivotal role in conceiving this research project in

discussion at a meeting between the HSRC and Joint Education Trust (now JET Education

Services) staff in early 2001

We thank the following people for their various contributions to the development of this

report, in alphabetical order:

• Nicolaas Claassen and Godwin Khoza for their inputs;

• Helen Perry for her useful comments and suggestions on the draft and for providing

an introduction to production functions;

• Jacques Pieterse for his advice on statistical aspects;

• Phumudzo Singo for administrative assistance;

• Elsie Venter for data cleaning, preparation and for running the procedures;

• Penny Vingevold for her support and critical input;

• Mariette Visser for preparing the Senior Certificate data;

• All those attending the presentation of the first full draft of this paper, including

Carol Deliwe

ix

©HSRC 2005

FRC Free reponse question JET Joint Education Trust MCQ Multiple choice question NBI National Business Initiative NGO Non-governmental organisation QLP Quality Learning Project SCE Senior Certificate Examination

x

©HSRC 2005