Could paying for results (finally) help us learn?

MonitoringWaterfall

Source: Scott Chaplowe, International Federation Red Cross and Red Crescent Societies

I came across this really interesting blog post recently (shout-out to Instiglian Michael Eddy for sharing) by a disgruntled M&E consultant.

S/he asks a simple question: why are NGOs so slow to use monitoring data that could improve the effectiveness of their work?

The answers were spot-on and made me wonder whether – and to what extent – paying for results (often referred to as “results-based financing“) helps address some (if any) of these challenges.

I’ll just go down the list:

Reason 1: Most of the data that NGOs collect is useless, usually geared towards measuring a small/narrow number of quantitiative indicators that have little to do with the actual program. 

As someone who believes in and promotes RBF, I tend to be a big fan of outcome metrics, indicators and measurement. Having said that, I completely agree that the types of metrics and indicators used by most NGOs have little to do with their actual programs and theories of change. But I think it’s important to understand the underlying reason for that: most NGOs’ M&E frameworks (and therefore the data they collect) is driven by donors agendas. Indeed, rare is the donor who disburses “unrestricted” funds; instead, most donors fund a specific project that advances their narrow strategic priorities and impose their own methods and indicators for measuring success. So, dozens of donors –> dozens of M&E frameworks –> useless data for the NGO –> few opportunities (or incentives) for actual learning.

In an RBF contract, the donor is not paying for projects per se. Instead, s/he is purchasing an outcome (i.e. a 10% reduction in the incidence of malaria in a given community) regardless of which programs – or combination of programs – it took to get them there. This should, in theory, promote the use of metrics and indicators more closely aligned with NGOs’ theories of change. A major assumption, however, is that donors are both willing and able to wash their hands clean of any programatic decision-making once an outcome has been agreed…this is where third-party intermediary organizations can add value in terms of negotiating contracts and setting the record straight.

Reason 2: Pressure to spend quickly means there is little incentive (or need) for learning.

The pervasive disconnect between funding disbursements and programmatic realities on the ground is a huge problem in development…and usually comes from the top (i.e. the donors). Again, there’s a reason for that: if U.S. government agencies don’t spend all the money they’re allocated in a given year, their agencies will get less money the following year. For foundations, there are serious tax/legal ramifications for not disbursing a certain percentage of their endowment each year. Yes, the system’s messed up. Unfortunately, RBF alone won’t provide a solution. While it’s true that paying for results in one lump sum at the end should in theory remove the perverse incentive to spend, spend, spend, if donors are unable (for legal reasons or otherwise) to set aside a pool of money that just sits there for a couple of years and does nothing, RBF won’t stand a chance. Donors need to figure out a way around this ASAP.

Reason 3: Short-term projects often leave little time for staff to really learn from M&E. 

It’s true that most donors fund projects on a short-term basis…too short to enable any real focus on outcomes or learning. In an RBF contract, because outcomes are precisely what donors are paying for, they should be committing funds over longer time horizons. However, this is assuming that the donor has the legal authority/political will to actually commit – and hold on to – funding over multiple years in the first place (see previous grumblings).

Reason 4: Learning from monitoring data requires some kind of process to allow organizations to feed this learning back into performance. Donors don’t typically make this easy (em, DFID) by imposing rigid logframes that are impossible and/or take forever to amend. 

Don’t get me started on logframes. Suffice it to say that in an RBF contract, the idea is that once an outcome has been agreed and the donor has committed to paying a specified amount for that outcome, donors wash themselves completely of any programmatic decision-making. That includes changes to our beloved logframes.

Reason 5: NGOs genuinely don’t have a clue about how to actually improve programs. That’s because development is complex and programs typically aim to do ridiculously ambitious things.

I agree that development programs are complex, which is why a hands-off, more flexible approach like RBF might be preferable to a rigid, linear funding model. I also agree that many NGO programs aim to do ridiculously ambitious things. In some ways, because RBF requires both donors and NGOs to focus on things they can actually measure (ironically, this is a common criticism of RBF), perhaps RBF can help bring things down a notch…and in a good way. Start small, think big.

Standard

5 thoughts on “Could paying for results (finally) help us learn?

  1. Thanks for your comment – really helpful. It looks like we roughly agree on results based finance to governments, so I’ll focus on implementers in this post.

    I think you accepted the challenges I posed, and offered a variety of ways to mitigate them. I’m going to go through them one by one, and add some thoughts on which seem most convincing.

    1) Implementers could share risks with donors.
    In some conditions, this might work. However, the risks you mentioned (like weather and macro-economic conditions) are hard to specify in advance, negotiate over, and put into contracts. Most big risks are going to be unforeseen, and foreseeable risks are challenging to quantify and adjust payments accordingly. This could end up with payments dependent on the whim of the donor (or the implementer.)

    2) Implementers could share risks with the private sector.
    This is an exciting possible usage. However, I’m not convinced that public and private sectors are inherently more or less risk seeking – I think they both react to the incentives set by the funder. Sharing risks with the private sector would increase access to finance – which is good in itself – but you still face problems with measurability, gaming and perverse incentives. (Arguably more, since the private sector can be more sensitive to these incentives)

    3) You could have mixed contracts, with some results based and some not results based finance.
    Does this not lose the benefits of results based financing? If the donor is giving grant money, then they would still track the activities and outputs of the grantee – and so you’d lose the benefit of flexible funding.

    4) Intermediary outcomes
    This is a really interesting point, as I think this is what happens to results based financing contracts that I’ve seen in practice. The trouble here is that intermediary outcomes end up looking suspiciously like outputs – and so you end up with an inflexible contract again. Potentially even more inflexible, since the outputs are now crucial for payment, and so it’s much more sensitive to change them.

    5) Development impact bonds.
    An interesting idea and worth trying! I have to admit that I find it difficult to imagine what it would look like in practice, but would be interested to see pilot results. As with other results based financing, I appreciate the idea but am not sure how scalable it is likely to be.

  2. Pingback: Why Don’t We Ever Learn? | AID LEAP

  3. Firstly, thanks for your thoughtful response to our post, which I greatly enjoyed reading! I think payment by results are one part of a solution; but I also have quite a few questions about it, and maybe now is a good time to put them down more clearly.

    I think results based financing faces significant challenges, when applied to a typical NGO or private sector contractor. By pushing large amounts of risk onto the implementing partner, it encourages (1) the use of larger implementing partners (who can take the risk), (2) risk-averse approaches rather than experimentation, and (3) gaming of outcomes; for example adopting unsustainable, short-term approaches rather than real solutions. In your example above, this could be giving everyone free bednets, rather than trying to change attitudes or behaviours. (See http://www.bond.org.uk/data/files/publications/Payment-by-results.pdf for a useful description of how this works in practice.)

    There are a number of trade-offs here. As you move from paying for outputs to outcomes, you reduce the risk of gaming (assuming your measurement is good enough) – but increase the risk to the implementer, and so reduce their risk-aversion. In particular, the outcome quickly becomes so far outside the control of the implementer that it doesn’t really make sense to pay them according to it. A 10% reduction in malaria is a good example; this is dependent on so many stakeholders (health workers, NGOs, community groups, the overall medical system) that it’s difficult to imagine one implementer being able to have much of an impact on it. If the implementer can’t have a significant effect on the outcome, then it probably make sense for them to sit back and do nothing – they still have a good chance of getting paid!

    One obvious solution to the above is to move away from seeing results based financing as something to use with implementers, and instead use it with governments. Governments have a much higher risk appetite, and have a vastly greater ability to affect the final outcomes. So they would seem to overcome a lot of the problems seen above. This is all fine, but actually donors already have a great tool for flexible aid to governments. It’s called general or sector budget support, and has been used for decades. While results based financing has some advantages (perhaps it aligns incentives better, perhaps it’s more easily sellable to a cynical donor public) I don’t see why flexibility is one of them.

    Obviously results based financing is still a valuable tool. In some cases, you’ll find the perfect indicator; something that can’t easily be gamed, that is easily measurable, that is high enough in the logic chain to be useful, and that implementers or governments have control over. But I’m not sure that will often be the case, and so I’m afraid I suspect the answer to your question is generally no!

    • Elina Sarkisova says:

      Thanks for much for your response, Aid Leap! I think many of the issues you mention are things that the RBF community struggles with every day. Top among them are picking an outcome metric that is (a) measurable but avoids gaming and (b) is as close as possible to what you actually want to achieve but not so “out there” that it dilutes incentives. For this (and other) reasons, I think that there are some problems that RBF will be able to address better than others. Having said that, the RBF toolbox is growing and evolving to respond to a wider variety of realities and needs…and is not always as black and white as people make it out to be.

      Let me explain:

      (1) On on your point about risk transfer (i.e. by pushing large amounts of risk onto the implementing partner, it encourages use of larger implementing partners): There are lots of different ways for NGOs to mitigate and/or share risk in an RBF contract. For example, NGOs could share risk with the donor (i.e. risks associated with factors outside of NGOs’ control like weather or macroeconomic conditions wouldn’t typically be transferred to the NGO) or private investors (see Development Impact Bonds). Moreover, rare is the RBF contract that pays 100% at the end and where outcomes are binary (i.e. success/failure). In other words, there’s usually some intermediary outcomes that trigger payments throughout the course of the contract and “success” is typically incremental so there’s incentives to keep improving (even after you’ve achieved the minimum threshold for success). Otherwise, if it becomes apparent to the NGO that outcomes aren’t going to be achieved, they’ll just throw up their hands mid-project and give up (since they know they won’t get paid anyway). That wouldn’t be a good outcome.

      (2) While you’re right that focusing on “real” outcomes (i.e. reductions in malaria) is hard because there’s usually lots of different variables and stakeholders involved, a Development Impact Bond is one instrument (albeit still new and untested) that could overcome this problem. By allowing a number of different actors/stakeholders to work together – and get paid together – DIBs enable funders to take a more “systems” approach. It builds on this broader movement towards “collective impact” but provides an actual accountability mechanism. The biggest downside to DIBs is, of course, the high transactions costs associated with putting it together. But that’s the case for any new thing – the more we do it, the less it will cost in the future. We just need to give it a try.

      On your last point about results-based aid (i.e. aid to governments), I disagree that governments have vastly greater ability to influence outcomes…or that they have higher risk appetite. Yes, governments can theoretically “absorb” more of a financial loss (i.e. not getting paid by a donor will not make or break any government or country), they are much more risk-averse than private actors, not least because they’re actually accountable to someone….their constituents. I’m also skeptical that the only reason governments fail to deliver is because they’re not sufficiently incentivized by outside donors. First, aid makes up a minuscule amount of developing countries’ overall budgets (perhaps with the exception of the “least developed countries,” which have their own unique problems of course). I think government failure has much more to do with the internal politics of each individual country (see Why Nations Fail) and whether or not governments are held accountable by their people.

      Feel free to push back!

      • Thanks for your comment – really helpful. It looks like we roughly agree on results based finance to governments, so I’ll focus on implementers in this post.

        I think you accepted the challenges I posed, and offered a variety of ways to mitigate them. I’m going to go through them one by one, and add some thoughts on which seem most convincing.

        1) Implementers could share risks with donors.
        In some conditions, this might work. However, the risks you mentioned (like weather and macro-economic conditions) are hard to specify in advance, negotiate over, and put into contracts. Most big risks are going to be unforeseen, and foreseeable risks are challenging to quantify and adjust payments accordingly. This could end up with payments dependent on the whim of the donor (or the implementer.)

        2) Implementers could share risks with the private sector.
        This is an exciting possible usage. However, I’m not convinced that public and private sectors are inherently more or less risk seeking – I think they both react to the incentives set by the funder. Sharing risks with the private sector would increase access to finance – which is good in itself – but you still face problems with measurability, gaming and perverse incentives. (Arguably more, since the private sector can be more sensitive to these incentives)

        3) You could have mixed contracts, with some results based and some not results based finance.
        Does this not lose the benefits of results based financing? If the donor is giving grant money, then they would still track the activities and outputs of the grantee – and so you’d lose the benefit of flexible funding.

        4) Intermediary outcomes
        This is a really interesting point, as I think this is what happens to results based financing contracts that I’ve seen in practice. The trouble here is that intermediary outcomes end up looking suspiciously like outputs – and so you end up with an inflexible contract again. Potentially even more inflexible, since the outputs are now crucial for payment, and so it’s much more sensitive to change them.

        5) Development impact bonds.
        An interesting idea and worth trying! I have to admit that I find it difficult to imagine what it would look like in practice, but would be interested to see pilot results. As with other results based financing, I appreciate the idea but am not sure how scalable it is likely to be.

Leave a Reply

Your email address will not be published. Required fields are marked *