The Intel

The new aid policy expects to elevate aid in foreign affairs, but there is a risk that the effectiveness of Australia’s aid program, which has steadily declined over the past few years, will not keep up. DFAT is quite good at counting things in terms of program delivery, but less good at measuring whether the longer-term changes they were hoping for are being achieved. DFAT also tends to overestimate how well their interventions are going so more independent oversight is needed. Good quality evaluation and monitoring need specialist skills to inform action that needs to be undertaken earlier and to support genuine learning across DFAT and partner countries if it is to help improve effectiveness.

So what’s an alternative? In my view, DFAT should engage an expert Chief Evaluator to elevate effectiveness in the implementation of the new aid policy. More independence and longevity would be assured if this was a statutory office. The Chief Evaluator would advise the Secretary and Minister on issues of effectiveness in the aid program and lead a small team of specialised staff to provide expert intervention at an earlier stage in programs. This could include:

• assessing the likely effectiveness of new program designs
• advising on the suitability of monitoring systems, especially their:
  - ability to track expected changes and outcomes
  - usefulness in informing management decisions
  - usability by local partners

• undertaking regular country program evaluations and synthesising multiple evaluations to understand why different types of interventions work or don’t work in different contexts
• sharing approaches that work across similar contexts

By elevating the evaluation position, effectiveness also has a better chance of gaining traction across the foreign affairs portfolio and potentially the development efforts of other government departments in the implementation of the new aid policy.

While we don’t know what form an ODE-successor might take, there are three approaches any alternative should strive for:  

Give it teeth. Independence and autonomy are the bedrock of credibility, and enhance the likelihood that evaluations are conducted comprehensively and objectively. When real or perceived conflicts of interest are at play, the toughest questions don’t get asked, and programs are deprived of the opportunity to reach their full potential. All evaluation teams should be staffed (at least majoritively) by externals, and be afforded the power to publish and follow up on recommendations without the prior clearance of DFAT staff.

‍Don’t let the perfect get in the way of the good. Randomised control trials are every evaluator’s dream, but they’re not always the right tool for the job. Too often, robustness is prized at the expense of relevancy. This results in large spends on polished data sets that can’t offer meaningful insight into what’s actually happening, or how things can be improved. Evaluators of Australia’s aid program need the expertise to choose the right evaluation methods with confidence, and enough autonomy to go against the grain when it matters most.

‍Prioritise learning. To ensure evaluation findings can be harnessed for improvement, learning needs to form a central part of any assessment. This means ensuring that evaluation processes are participatory, and that they also facilitate opportunities for safe and meaningful reflection and discussion. In practice, it also requires promoting a “fail fast” approach that reframes setbacks as opportunities to enhance program performance, adaptability, and innovation. 

Why shouldn’t ODE be returned? Fortunately, the Minister seems to want something better than ODE: a starting point, not something to be disregarded. Most experts and the current Minister lamented its demise, which has also corresponded to a significant decline in technical and economic capacity in and around the aid program. DFAT’s own reporting (p.66) shows that 38 per cent of its completed projects in 2021-22 were rated as unsatisfactory against effectiveness and efficiency criteria. Understanding effectiveness requires impact evaluation. Understanding cost-effectiveness and efficiency requires serious economics and measurement chops. While costs are usually straightforward to calculate, the benefits side of the ledger requires credible estimates of development impacts attributable to aid. 

What should a new, better-than-ODE approach involve? What ODE lacked, or did not focus on enough, is an obvious starting point. First, genuine independence in evaluations and protection from political interference. I am agnostic as to whether this is an in-house government function, which may be more feasible, or a separate, transparent, and public-facing office. What matters most is longevity and informed decision-making. Second, careful measurement – not in M&E matrices, but of development indicators to assess progress and guide programming, based on need (e.g., poverty). Third, a focus on asking the right questions with rigorous impact evaluation – not strictly experimental, but with a commitment to credible counterfactual-based analysis able to inform a serious value-for-money discussion.  

These three basic ingredients may require a major injection of technical expertise but are essential to meet the current government’s aspirations, help us know what works, and do more of that and less of other things.