If like me, you work in the international development sector, then you must care about making a difference. Donors, UN organisations, NGOs, and our partner governments – we were all created to make that difference. If this is so, I would ask then, why do we have comparatively limited accountability as a sector in regards to demonstrating that we have actually made that fundamental difference in the lives of the people we all exist to serve
A glimpse into Sierra Leone’s Education Innovation Challenge
I’ve just come back from a short trip to Sierra Leone, where I spent some time visiting schools, speaking with children, parents, teachers, headteachers and other education officials, who are all stakeholders in an education initiative called the Sierra Leone Education Innovation Challenge (SLEIC). Myself, I work for Save the Children, one of the partners supporting the SLEIC initiative. I presume that many of the issues I’ve encountered are not unique to my organisation, and so, in the spirt of transparency and learning, I wanted to share a few personal reflections.
SLEIC is an incredibly ambitious initiative, championed by David Sengeh, former Minister of Education in Sierra Leone, now Chief Minister in the Government of Sierra Leone (GoSL), and long-time innovation pioneer. It is supported by the UK Government and several other donor partners who, along with Minister Sengeh, are challenging this ‘accountability paradigm’. Led by the Education Outcomes Fund, it adopts a Development Impact Bond (DIB) model – where investors take a financial risk on behalf of the delivery partners – and where they only recoup their investment should the project achieve agreed increases in numeracy and literacy progress amongst P1-P6 school children compared to a control group. It is being evaluated by an independent organisation to ensure that there is no ‘marking your own homework’. Please click here for more information.
The DIB model
In the whole scheme of things, the SLEIC project is relatively small, compared to many of the projects that the GoSL, the contributing donors, and even my own organisation are involved in. And whilst the DIB model created an initial financial risk to my organisation (we only secured investors after we had signed the contract), it raised many questions internally as to whether this was a sensible thing to be involved in. Given we would only be paid if we reached the ambitious learning targets set, there were real concerns about whether we could make that difference, particularly with a budget per child limit much lower than our traditional projects. Many of our peer international non-governmental organisations (INGOs) dropped out of the process due to this lack of confidence in achieving the target outcomes.
Rethinking risk and return
This question on risk is linked to how an organisation judges Return on Investment (or risk and return in different words). Sadly, I see ‘return’ far too often being confused with ‘$$’, rather than the potential for impact. This is something that is a very live issue in my organisation, where we have recently consulted on what we call our ‘Impact Vision’, which aims to describe the types of programmes and partnerships that should be valued and prioritised. This impact vision talks to well worn phrases in the sector such as ‘sustainability’, ‘replicability’, ‘partnership’ and ‘inclusion’, but also seeks to move past the rhetoric to meaningfully design for impact. For me, the SLEIC project ticks several of these boxes – a programme designed in partnership with government, at a price point per pupil that the government says is affordable and scale-able. Hand on heart, I don’t think I could argue that we have the same potential in some of our other projects. But given the commitment to walking the talk on our impact vision, it is an incredibly intensive programme, requiring a huge lift to maximise the opportunity for success (more on this below). Coupled with the financial risk and the fact that it has a comparatively small overall budget and reach, I continue to hear remarks within my own organisation of ‘is it worth the effort?’
Balancing affordability and scale-ability
How far can we compromise on cost is another question that I believe is very valid. Like many other organisations, we adhere to sector-wide standards on risk management and like to think we have high technical quality standards, underpinned by a strong evidence base of what works. Yet logically, the more requirements or activities you build into a project, the more the costs will increase.
SLEIC really challenges us to think differently as an organisation. With the input of the GoSL, SLEIC has set a maximum budget of $36 per child per year– one that I understand from my education colleagues is significantly lower than the cost than we would normally include in an education programme. The team explained how they have therefore had to really focus on stripping back the activities to a point that is affordable, and yet still enables us to make a meaningful difference. For example, we would normally implement 4 to 6 training days of Teacher Professional Development across an academic year with significant school-based support and coaching along the way. In SLEIC, however, this has been reduced to just one programme year, with limited additional days in subsequent years and less intensive school-based support overall. Similarly, our learning circles for remedial learning are also much shorter and less intense than we have previously implemented. We are trusting that the learning data being shared with District Officers will mitigate this through their in-school coaching, and that the usual staff to school ratios that we had often assumed as ‘best practice’, can be upended.
‘Is this realistic?’, ‘Are we trying to do the impossible?’, ‘surely this won’t work?’ remain legitimate questions. I saw first-hand how the compromises on the quality of activities and contact time with teachers and schools were impacting progress. But not to try, nor strive, for the potential for replicability and scale-ability would in my opinion be the greater risk.
Internal lessons
And yet, juxtapose this against a comment that really blew my mind the other day. It was made by one of our most experienced education technical advisors, who has been involved in many different education projects across many different countries over the years. On discussing the myriad of challenges in the project, he paused and reflected saying….. ‘but where else have we ever had learning outcome data 4 months into a project?’ Isn’t that crazy – that in this project that has one of the most restrictive of budgets, we have a MEAL system in place, that is regularly collecting literacy and numeracy data down to individual school-level. Where there is a will, there is a way, I think the phrase goes. And we’ve been forced to think differently, challenge our own assumptions, and invest in being across the detail of outcome level data like never before.
It was also fascinating to observe the tension and challenges that managing a project like this brought to our Country Office. Like everyone else, many projects we deliver are based on the same linear approach – design, deliver, evaluate, report – generally with baselines, mid-lines and/or end-lines but not always. Outcomes may only be measured 2 or 3 times during a 5-year project. Our organisational Key Performance Indicators (KPIs) are based largely on outputs – have we delivered the activities agreed with the donor on time and on budget. We frequently request project and budget revisions with donors, but this can be a laborious process, that is often viewed as problematic rather than something to be embraced. And yet, despite everyone knowing that the SLEIC project is different, and is based on adjusting and re-adjusting every week, our internal systems have little empathy, and continue to throw up red flags on SLEIC team left, right and centre. Budget not being spent according to plan; procurement plans out of date; and even our fancy new organisational MEAL system – which has taken years to put in place – needs adaptation itself, if we are truly to be an agile, adaptive, results driven organisation.
One thing I hadn’t really anticipated was the management style changes that may occur in our teams because of this new style of project. We have a great team in Sierra Leone, with hundreds of years of development project management experience between them. However, before last year, no experience of managing this new approach to programming demanded by SLEIC. And having spent time with many different project teams over the years, what really struck me was the energy of the SLEIC team, their attention to detail, and the way they engaged the headteachers and teachers in a supportive yet accountable manner. I’m not sure whether the framework of the SLEIC project is the principle driver, but I was also struck by the ‘hand in glove’ relationship with the GoSL, particularly at the district level. Has trust been strengthened by the inter-dependency demanded by SLEIC, and the fact that we’ve put our neck on the line (financially so to speak)?
A model for the future?
So why then are projects like SLEIC the exception rather than the rule? I’m not saying that every type of project or context is suitable for an outcomes funding model, nor do I think the complexity and cost of a DIB necessarily presents value for money. We have delivered incredible results through transformational projects that have not had a results-based financing model. And World Bank and FCDO funded work like ‘Best Buys’ and ‘Smart Buys’ and the What Works Hub are helping to clarify the most impactful, scalable and value-for-money interventions in education. But why is there not a greater push from recipient governments and donor partners for contracting models that can help governments take these to scale? And why is there not a greater demand for accountability from organisations like my own – both externally, but also from within – to put our money where our mouth is on achieving change that is sustainable and does have a credible pathway to scale?
That said, I’m also very mindful of the privilege my own organisation enjoys – one of the big players in the sector – where doors open easily, and where the organisation has the capacity to manage and absorb risk. We would have absolutely failed as a sector if outcome funding models simply reinforced the power imbalances in the sector – and there is a risk that the top-down nature of these types of contracts could do.
Consequently, how then can projects like SLEIC be a force for good to accelerate shifts in the sector, through models where investors or donors carry the risk, yet pass on the flexibility and any potential for financial gains from achieving results to local and national NGOs. Interestingly in SLEIC, Save the Children is one of the 2 INGOs, with the other 3 partners all national Sierra Leonean organisations – each enabled to participate in the project through support from an investor.
Betting on results
So, we’ve just had the year 1 results in – deep breath. And whilst it’s still very early days, they do point to some initial gains in numeracy outcomes in Save the Children intervention schools compared to the control schools. And our own internal evaluations that we now carry out 2 to 3 times a year, correlate with the findings of the independent evaluator on numeracy, but also show some small literacy gains too. There is still a fair chance we may fail to achieve the target improvements – I’d probably call it as 50:50. But I sincerely hope that my organisation embraces the learning from SLEIC and even if we do fall short, that we don’t simply consign it to the ‘too difficult bucket’, but learn from our experience and use this as one stepping stone to truly become an organisation that lives and breathes results.
