GoKawiilData
In each country, we have two waves of data: baseline and endline. In Nepal, we conducted a baseline survey with a random 50% of the sample, and in Kenya, we relied on administrative data instead of baseline survey data. The endline surveys took approximately 30 min to administer and included approximately 20 questions. These questions included a learning assessment, child wellbeing, parental engagement in educational activities, and parental perceptions of their child’s learning. A portion of the survey was conducted with the parent, and learning outcomes were collected by directly assessing the child over the phone. Endline surveys were conducted a few months after the programme ended. A set of common core questions in the baseline and endline surveys are included in the Supplementary Information.
The learning assessment was adapted from the ASER (Annual Status of Education Report) test, which has been used frequently in the literature to measure learning outcomes14,15 and is used routinely across 14 countries. The test consists of multiple numeracy items, including two-digit addition, subtraction, multiplication and division problems. In addition, we asked students to solve a word problem and a fraction problem to capture learning outcomes beyond a core set of mathematical operations. Results are presented in several main results figures and tables. Distributions of raw baseline and endline learning levels are shown in Supplementary Fig. A3.
To maximize the reliability of the phone-based assessment, we introduced a series of quality-assurance measures. To minimize family members in the household assisting the child, students had a time cap of 2 minutes per question and we asked each child to explain their work. We only marked a problem correct if the child correctly explained how they solved the problem and enumerators were confident parents were not assisting their child. We also conducted a battery of validity checks to ensure the reliability of the learning outcomes. These results contribute new evidence on the robustness of remote learning assessment data across five countries. Phone assessment has emerged as a common strategy for large-scale household surveys such as the World Bank Living Standards Measurement Survey (LSMS). A growing literature has started to explore the validity of phone-based assessments to measure learning outcomes20,56,57,58, with emerging evidence suggesting phone assessment can capture meaningful information at high frequency and low cost.
Extended Data Table 5 shows the results of five checks we conducted on the validity of our main learning outcomes of learning assessments via phone. Column 1 shows the first robustness check where we compare in-person to phone-based assessment for a representative sample of the same students in Kenya. An additional test included back-checks, with a random subset of students tested twice on the same competencies. We find a strong relationship as expected, with large positive coefficients and t-statistics ranging from 5 to more than 20. We further randomize students to receive different problems of the same proficiency (for example, four different questions to measure two-digit addition with carryover). Finally, column 5 shows results from a real-effort question to disentangle effects of the intervention on effort on the test versus cognitive skills. Students were asked to answer several effort tasks, for example figuring out the day of the week or counting zeros and ones. Altogether, results suggest measuring learning by phone can be robust, similar to findings from the World Bank LSMS for household measures of consumption.
The survey also included questions on caregiver engagement in their child’s education and beliefs. We measured engagement by asking caregivers how often they spent helping their child with their schoolwork over the previous weeks. We also included a measure of a caregiver’s confidence in their child having made progress in learning over the previous months, and their perception of their child’s numeracy level. Additional questions included information on whether the caregiver has returned to work. We also asked about parents’ demand for remote learning services in the future, and whether they would be willing to pay for such a programme. For students, we asked about child’s mental wellbeing, how much they enjoy school, and the child’s own belief about what mathematics problems they will be able to answer. We also measured non-cognitive skills, such as perseverance and ambition. Results are shown in Supplementary Tables A2 and A3. Finally, we included demographic questions, recording the child’s age, grade and gender.
The overall sample size, pooling all sites, is 16,936 households. The flow of participants from enrolment through randomization to endline follow-up is summarized in a CONSORT diagram (Supplementary Fig. A5). For endline surveys, we randomly sampled households from the full sample to interview for a total sample of 12,331. This was due to time and cost constraints. In Supplementary Table A8, we show that those randomly selected for endline interview are statistically equivalent to the full sample at baseline along a series of indicators such as gender, student grade and baseline learning level.
Supplementary Table A9 presents the response rate to the endline surveys and an analysis of survey attrition for those randomly selected to be part of the endline sample. The follow-up rate was around 76% of respondents at endline. Supplementary Table A9 also presents a test of whether response rates differed by treatment assignment. This provides evidence that our sample has a high and unbiased response rate.
Finally, we also include a survey for teachers to assess their beliefs and instructional practices. These questions include their desire to be a teacher, and teachers’ view that the phone call tutorial programme was helpful for student learning. Questions also include instructional practices, such as involving parents in education further and better targeting feedback to students’ actual learning level. These questions can capture potentially persistent effects on educational systems through teachers changing their beliefs and behaviours beyond the lifecycle of the programme.
Experimental design
... continue reading