The Sierra Leone Education Innovation Challenge (SLEIC) was a three year outcomes contract, managed by the Education Outcomes Fund (EOF) and Sierra Leone’s Ministry of Basic and Secondary School Education (MBSSE). Five organisations, of which Save the Children was one, delivered programmes in schools, with payments being based on the extent to which learning outcomes improved as compared to control schools.
In previous blogs we have spoken about our results, and our reflections on design and implementation factors which influenced our results. In this blog we take a step back to look at what we learned about Outcomes Based Financing more broadly.
As an organisation, Save the Children has been interested in exploring whether OBF represents an opportunity to improve programme impact and ways of working. While there are concerns about whether payment by outcomes could drive narrowed focus and avoidance of more complex, harder to measure outcomes we were eager to engage and learn. Building on the ambitions set out by EOF we created our own expanded framework for our expectations of what OBF could do for us. At the level of individual programmes, we hope that OBF could lead to 1) more flexible, impact driven programming, 2) more credible evidence of what works, 3) improved organisational ways of working, and 4) systemic change through institutionalisation and scaling of cost-effective, evidence-based interventions. Beyond individual programmes (and therefore out of scope for our reflections on SLEIC) we hoped that OBF could lead to the opening of new funding streams in education, through increased evidence and credibility for our work.
Figure 1 - Our Learning Framework
The rest of this blog reflects on whether we saw these shifts. It summarises our more detailed working paper, available here for those looking for more detail.
Ambition 1: OBF will lead to improved data-driven performance management, increased outcomes-oriented mindsets, and shifts in organisational ways of working
Looking at our own analysis, we would say this is partially true. Internally we recognised that the OBF mechanism pushed new ways of thinking, and while we were slow to develop the mindset, as a team we felt that by the end of the programme we were more focused on outcomes than we considered to be "business as usual" for us. This was not without its downsides. It highlighted a need for us to work on learning culture as an organisation, ensuring that accountability for outcomes doesn't lead to teams feeling that they will be blamed if outcomes are not achieved.
In terms of data driven performance management, over time we developed systems and mindsets, but we can only say this was partially achieved. While structures for making decisions based on data, and the systems for collecting data were in place, we never arrived at a point where the data was available in the right form at the right time for decision making. Much of the deep analysis of monitoring data was only done retrospectively at the end of the programme (see learning report).
How was this influenced by the OBF mechanism? Our conclusion was that the systems unique to the outcomes contract - i.e., the yearly evaluation, and the opportunity to exchange learning with other providers - did not make a fundamental difference to how we approached adaptive management. The former was too aggregate and too late to inform yearly planning, and in the case of peer learning, the quarterly peer learning sessions were useful, but never fed meaningfully into our planning. It is interesting to note that so far on the Rwanda outcomes contract, peer exchange has been much more active, being more practical and more provider let. This is potentially due to lessons learned and relationships created through SLEIC, with both Bridges Outcomes Partnerships and Save the Children being common to both.
Will it last? There is some hope that some changes to ways of working, particularly in the financial management of outcomes contracts within SCUK may translate into permanent change. However, at the project level, one project is not going to be enough to change organisational structures and ways of working - though dissemination and internal learning continues.
Ambition 2: OBF will deliver better outcomes at a lower cost than through traditional grant funded programmes.
We have widely shared that our results fell below expectations. This isn't to say they weren't significant (statistically) or important (actually), but we did not reach the targets set out in our outcomes contract. Other providers got much better results, and so to consider this question we should consider the overall pooled learning gains, as well as our own. Using the intended cost per child we can see below that, when compared to a range of other programmatic approaches (as taken from the immensely valuable work done by Angrist et al (2024) (1). From the data shown in Figure 2, we can see that SLEIC is slightly underwhelmingly below the median LAYS per hundred dollars spent. This isn't to say the results aren't good, they just aren't world beating, and it would be hard to say that OBF has facilitated uniquely cost-effective education programming.
Figure 2 - Comparing SLEIC with other aggregated cost-effectiveness data for other interventions. In this figure “Pooled” refers to both English and Maths gains, and "All" refers to all providers.
Using just our learning gains we can be more nuanced in this. While our learning gains are fixed, the total costs and the total number of children reached can be interpreted in different ways.
For budget we have compared a) our budget costs, and b) our total cost, including time spent on the design phase, and additional unbudgeted time spent by technical teams in Sierra Leone and the UK.
For total children reached, we have used a) the actual recorded head counts from the evaluation, and b) the adjusted figures used for the payment calculations, which account for significant decreases in attendance in years 2 and 3, which were hypothesised to be due to the timing of the assessment, rather than actual decreases in attendance. For both attendance figures we have applied an adjustment to calculate the total unique children reached including new grade 1 enrolments in each year, who would naturally have benefited from any learning gains in the years they were part of the programme.
As can be seen in Figure 3, combinations of these options give us a range of figures ranging from below the median to a long way above it.
Figure 3 - Alternative calculations. "Standard cost" refers to LAYS/$100 for our maths gains using the $36/child figure.
This doesn't change our conclusion that, in this case, results were good, but not uniquely good. It is also important to remember that the other studies quoted as comparators are not good counterfactuals. Fundamentally we can't say whether a grant funded programme would have been less (or more) cost effective in delivering learning gains.
Further, more rigorous analysis being conducted by the What Works Hub will add more nuance to this, looking at whether specific programme components can be linked to learning gains, and what their costs were.
Ambition 3: OBF will deliver scalable (and scaled) solutions and systemic change
The MBSSE has been a powerful advocate for SLEIC and for Outcomes Based Financing in general, as well as a leading voice in advocating for more financing for education, and pushing it to the top of the domestic development agenda.
However scaling models from SLEIC relies on (among many other things); resources to support scaling (financial and human), and a clear vision of what it is that can be rolled out nationally or integrated into policy.
In the case of the latter need, SLEIC presents five different models, each of which contains multiples components. The evaluation tells us which providers demonstrated greater learning gains, but what it doesn't tell us is a) which components of those providers' programmes were responsible (or is the whole package necessary), and b) were the differences between providers due to quality of design, or quality of delivery? Without knowing these nuances it is difficult for the MBSSE to set out a clear vision of what (if anything) from SLEIC they see as being scalable.
The former need is where we can loop back to how cost-effective is cost-effective enough. Taking a figure of 36 dollars per child, we can say that, given the current government spending on education of approximately 341 dollars per child in primary school, scaling SLEIC interventions to all children in Sierra Leone would necessitate up to a ten percent increase in the education budget (2). Considering that Sierra Leone already spends more of its budget on education than all comparable countries, it is easy to see how this will be challenging.
Taken individually most "scalable" parts of the interventions are those that rely on existing systems, for example, we saw good results from our work in supporting expanded pedagogical remits for both headteachers and School Quality Assurance Officers (SQAOs). While both are in short supply in the system, the numbers are growing, there is significant appetite within these cohorts to have an increased role in supporting teachers (anecdotally demonstrated by the fact that our SQAO WhatsApp groups have remained active long after programme closure).
While this may not seem like an optimistic view of the potential of SLEIC to change systems, there is cause for hope.
So is it viable? Is it optimal?
Our conclusion as a team was that the experience of SLEIC was positive. We felt the push to focus on outcomes and to ground our work in data was helpful. These are not new concepts. All our programmes aim for this, and many achieve it. However, having it baked into the design of the fund gives it extra leverage, and removes many barriers to being adaptive and outcome focused. However, we don't necessarily see the current model of OBF used by EOF as the only way to achieve this. For example, we also see the value in exploring less punitive versions of OBF, in which performance unlocks additional budget, rather than reducing payments (more like GPE's multiplier for example).
Throughout SLEIC EOF was a good partner, being open in communication and recognising that flexibility is needed throughout an outcomes contract to ensure that targets and payment calculations remain fair for all parties considering changing circumstances. Being EOF's first outcomes contract in basic education SLEIC was a learning experience both for us and for EOF.
For us we are learning that for OBF to be a useful approach to us, we need to become more flexible, and to focus on developing a learning culture that can support rapid cycles or learning and adaptation. For EOF, the learning should be to focus on how the investments in evidencecan create clear pathways for scale, and clear guidance that give policy makers a "menu" of evidence-based approaches to integrate into their sector planning. As a sector we can learn that while OBF may not (in our opinion) represent a silver bullet for strengthening education systems, it is a welcome push towards greater accountability for outcomes, and greater engagement of governments in setting the parameters for what they want to see, and what they want to learn from donor funded programmes. This value will be greatly solidified if, across programmes, OBF can use the transparency and "de-risking" to crowd in new donors and sources of funding to the education sector. We remain, an enthusiastic, but professionally sceptical participant in the world of outcomes based financing (I refuse to use the term "critical friend").
(1) Angrist, Noam, David K. Evans, Deon Filmer, Rachel Glennerster, Halsey Rogers, and Shwetlena Sabarwal. "How to improve education outcomes most efficiently? A review of the evidence using a unified metric." Journal of Development Economics (2024): 103382
(2) This is an approximate comparison as UIS figures are in PPP$ and ours in current US$
