To avoid making structural assumptions about the link between inputs, performance, and outcomes, we do not suggest that outcomes should be part of the Service Delivery Indicators survey. However, it may make sense to report separately on outcomes when the various sub-indicators and the potential aggregate index are presented. In health, there are measures for many countries at the national level, such as under-five mortality rates, but no indicator that can be linked directly to the service quality of individual facilities. Quantity outcomes in education are also available (various measures of flows and stock of schooling) for a large subset of countries. However, on quality there are no comparable data available, at least not for multiple countries. Thus, student learning achievement has been collected as part of the survey in education.
Available evidence indicates that the level of learning tends to be very low in Africa. For instance, assessments of the reading capacity among grade 6 students in 12 eastern and Southern African countries indicates that less than 25 percent of the children in 10 of the 12 countries tested reached the desirable level of reading literacy (SACMEQ, 2000-2002). As part of this survey, learning outcomes were measured by student scores on a mathematics and language test.
Table 25: Average score on student test
Sample All Rural Urban
Language
Senegal 0.54 0.53 0.62
(0.01) (0.01) (0.02)
Tanzania 0.43 0.41 0.52
(0.02) (0.02) (0.03)
Mathematics
Senegal 0.45 0.44 0.48
(0.01) (0.01) (0.02)
Tanzania 0.39 0.38 0.48
(0.02) (0.02) (0.03)
Note: Weighted mean with standard errors adjusted for weighting and clustering in parenthesis. 1787 observations from 180 schools in Tanzania, of which 449 (45 schools) are from urban areas. 1485
observations from 151 schools in Senegal, of which 610 (61 schools) are from urban schools. Test scores are averaged at the school level.
We test younger cohorts partly because there is very little data on their achievement, partly because SACMEQ already tests students in higher grades, partly because the sample of children in school becomes more and more self-selective as we go higher up due to high drop-out rates, and partly because we know that cognitive ability is most malleable at younger ages (see Heckman and Cunha, 2007).
For the pilots, the student test consisted of two parts: language (English and French, respectively, in Tanzania and Senegal), and mathematics. Students in fourth grade were tested on material for grades 1, 2, 3 and 4. The test was designed as a one-on-one test with enumerators reading out instructions to students in their mother tongue. This was done so as to build up a differentiated picture of students’
cognitive skills. Results of the grade 4 student test are presented in Table 25.
The average score on the test was just over 50 percent in Senegal and just over 40 percent in Tanzania, for the language section and 45% and 39% for the mathematics section. Senegalese students performed significantly better than Tanzanian students on both sections of the test, and the difference is significant.20
Table 26: Language: Percentage of student who can read a sentence (in French/English)
The difference on the language test is at least partly due to the fact that teaching takes place in French from grade 1 onwards in Senegal, while English is only introduced as the medium of instruction in grade 3 in Tanzania. As expected, rural schools score significantly worse than urban schools.
Sample All Rural Urban
Senegal 0.33 0.28 0.53
(0.02) (0.03) (0.04)
Tanzania 0.06 0.06 0.10
(0.01) (0.01) (0.03)
Note: Weighted mean with standard errors adjusted for weighting and clustering in parenthesis. 1787 observations from 180 schools in Tanzania, of which 449 (45 schools) are from urban areas. 1484
observations from 151 schools in Senegal, of which 610 (61 schools) are from urban schools. Test scores are averaged at the school level.
While the mean score is an important statistic, it is also an estimate that by itself is not easy to interpret. Table 26 depicts a breakdown of the results. As is evident, reading ability is low. In fact, only 33 percent of students in Senegal are able to read a sentence and only 6 percent of students in Tanzania are able to complete this task.21
20 The test consisted of a number of different tasks ranging from a simple task testing knowledge of the alphabet (involving 3 questions) to a more challenging reading comprehension test (involving 3 questions) in languages and from adding 2 single digits (1 question) to solving a more difficult sequence problem (1 question) in mathematics. Just as for the teacher test, the average test scores are calculated by first calculating the score on each task (given a score between 0-100%) and then reporting the mean of the score on all tasks in the language section and in the mathematics section respectively. Since more complex tasks in the language section tended to involve more questions, this way of aggregation gives a higher score than simply adding up the score on each question and dividing by the total possible score. Following this latter method of aggregation would lead to a roughly 8-10% lower score in the language section in both countries. In the mathematics section the simpler tasks involved more questions, therefore aggregating by task gives a slightly lower score than simply adding up the score on all the questions (roughly 5 %).
This difference is significant. In mathematics, 86% of Senegalese students and 83% of Tanzanian students can add two single digits. Again, as expected, rural schools perform significantly worse than urban ones. For a more detailed description of performance on various tasks, see the technical appendix.
21 The reading task consisted of reading a sentence with 7 words/11 words in Senegal and Tanzania respectively.
We have defined the percentage of students who can read a sentence correctly as those who can read all words correctly. With a somewhat more lenient definition of being able to read all but one word, the numbers rise to 48% and 11%.
Table 27: Mathematics: Percentage of student who can add two single digits
Sample All Rural Urban
Senegal 0.86 0.85 0.90
(0.01) (0.02) (0.02)
Tanzania 0.83 0.81 0.93
(0.02) (0.03) (0.02)
Note: Weighted mean with standard errors adjusted for weighting and clustering in parenthesis. 1787 observations from 180 schools in Tanzania, of which 449 (45 schools) are from urban areas. 1484
observations from 151 schools in Senegal, of which 610 (61 schools) are from urban schools. Test scores are averaged at the school level.
The Service Delivery Indicators are a measure of inputs (including effort), not of final outcomes.
Nevertheless, in the final instance, we should be interested in inputs not in and of themselves, but only in as far as they deliver the outcomes we care about. Given that we have collected outcome data in education, we can also check whether our input measures are in some way related to outcomes. Of course, these are mere correlations that cannot be interpreted causally, but we still believe that it is interesting to examine how our Indicators correlate with educational achievement. Figure 21 depicts unconditional correlations between student achievement and the education indicators, where the data from each country is pooled. Interestingly – and across the board – there are fairly strong relationships between the indicators and student knowledge, with all the correlations having the expected sign.22 Figure 21: Relationship between student performance and the education Indicators
22 Results are similar when running a regression of student test score separately on each indicator, a country dummy and a rural/urban dummy.
0.2.4.6Test Score
0 1
Infrastructure
.2.4.6.8Test Score
0 20 40 60 80 100
Pupil Teacher Ratio Student Test Score Fitted values
Pupil Teacher Ratio
.2.4.6.8Test Score
0 2 4 6 8 10
Books per Student Student Test Score Fitted values
Books per Student
.2.4.6.8Test Score
0 .2 .4 .6 .8 1
Absent from Classroom Student Test Score Fitted values
Absenteeism
.2.4.6.8Test Score
0 100 200 300 400
Time spent teaching Student Test Score Fitted values
Time spent teaching
.2.4.6.8Test Score
.2 .4 .6 .8 1
Teacher Test Score Student Test Score Fitted values
Teacher Test Score
Relationship between Student Performance and the DSI indicators