This was the challenge set by the Australian Embassy, Indonesia at the start of phase 3 of the Australia Indonesia Partnership for Justice (https://aipj.or.id) – a long-standing and successful programme supporting reform in the justice and security sectors.
To their credit, the Embassy, DT Global (the programme managers) and the AIPJ3 team have taken on an innovative approach in response: Outcome Likelihoods and Causal Analysis (OLCA). Most programme monitoring does a good job describing what has been done and the challenges encountered. But it struggles to say whether any of that is actually moving the needle on the change the programme exists to support. That gap is what OLCA is designed to close.
The starting point is a simple theory of change (TOC) developed with each programme team. Participants are then asked to consider each step in the TOC in turn, what factors are enabling and constraining it and assess the likelihood of it being achieved under different scenarios. The really smart thing is that all those joint probabilities are easily compiled – using Bayes’ theorem – to provide overall estimates of the likelihood of success, to allow ‘what if’ analyses, and to examine the influence of different factors (including the programme itself). Think a combination of Strategy Testing and Contribution Analysis but supercharged.
We’ve just finished the baseline assessments. These will be repeated every six months, to incorporate new evidence and learning, allowing partners to update their (expert) beliefs in the light of new knowledge. Using OLCA during the course of implementation, we will track in real-time how these changes are affecting the likelihood of achieving the programme’s objectives.
If you’re interested in the OLCA approach, check out the resources available on https://www.quantqual.net and/or get in touch with me with any questions.