The Intel

Development can be complex and difficult. Evaluation helps us find out what works and what needs improvement in Australia’s $4.5 billion spend on aid. Good evaluation gives voice to directly affected people about what they need and value. Together with its sister discipline, monitoring, evaluation is a central component of good management.

That’s why the enthusiasm for evaluation in Canberra is welcome. But let’s ensure we do it right. Here are four ideas.

1. It’s critical that senior DFAT leaders have the incentives to value and demand robust evidence about investment performance. Without this the rest is largely meaningless.  

2. DFAT needs more monitoring and evaluation (M&E) specialists on staff – to explain and champion the approach. To ensure timely inputs to budget decisions, designs and investments. To ensure rigour in program M&E. And to work in situations where required security clearances rule out most external consultants.  

3. M&E should be proportionate to an investment’s scale and purpose. All investments over $3M would benefit from an independent evaluation. But those which are significantly larger, higher risk or of extra strategic importance need more. This could mean an early review of: 1) a program’s design and contracting arrangements; 2) how well the M&E system is being implemented and; 3) whether data is being used in management and decision-making.

4. Resource meaningful learning across a portfolio of programs related by theme or geography – during their implementation. For example, in DFAT’s work in gender equality and women’s empowerment in the Pacific, or in areas of historic low performance.

And while we’re at it – let’s set up a knowledge management system within DFAT, so that we can access and make use of our growing body of information about good development, and how to evaluate it!

Australian government appetite for evaluation comes in waves. In a recent trough, DFAT relaxed requirements for development programs to be evaluated and leaders could nominate a select few to be published. Demand for investment evaluations may be on the rise with more rigorous evaluation policies expected, but it doesn’t follow that all leaders will jump on board – some fear media or senate estimates scrutiny, others see evaluations as bringing a low return for scarce resources.

What can proponents of evaluation say to DFAT leaders to encourage them to embrace evaluation of their investments?

1. Findings are not personal – we’re learning

Development is complex and demonstrating results can be tough. You need evaluations to tell you what’s working and what’s not. They give substance to claims about effectiveness and explain the context (including to senate estimates or the media) when progress towards results isn’t as expected.

2. Contestability will improve performance

Incentivise your team to be effective investment managers, including planning for quality evaluations. You have control over the timing, terms of reference, selecting diverse team members, providing quality data based on a properly resourced monitoring, evaluation and learning system, and setting the context for program partners so they know what to expect. But don’t try to control the findings or recommendations. Stay at arm’s length, project an open mind, and invite contestability.

3. Demonstrate accountability through actioning responses

Digest and respond to evaluation findings and recommendations. You don’t need to agree with everything. Publish the evaluation and your management response and, for ongoing investments, hold your team and implementing partners accountable for actioning your response. Annual investment check-ins are a good time to ask your teams to explain their progress.

Building a performance culture needs leadership effort. Improved evaluation policy can’t do it alone.

The increased attention on evaluation is welcome and overdue. But it’s important to remember that the real agenda sitting behind this shouldn’t just be a navel-gazing “let’s improve how we think about ourselves” moment, but a more transformative one where we’re asking, “if Australia’s serious about being a high impact development partner, what will it take to drive lasting reform?”

The risk of a strategic misstep lies in any assumption that doing ‘more’ or ‘better’ evaluation alone will lead to improved performance, because ultimately our development program is managed by busy humans making complex decisions. Perhaps they’re the key to transforming our impact? At a recent Situation Room, we heard the views of 16 of Australia’s top evaluators, and this notion of starting at the human level to drive reform had a lot of traction. Here are two of the key themes that resonated with me the most:

Capacity development across the board will bolster supply of good quality evaluation… It was important to a number of the experts that DFAT embeds evaluation skills into performance and promotion incentives at all levels, while at the same time just get more resources in to make sure that already time-constrained individuals can have the space to do it well. And since external contractors do a huge amount of evaluation work, perhaps it makes sense to send some APS6s out to experience it with them.

… but accountability in senior leadership is the best way to actually transform performance. More than anything, attendees wanted to see changes to the way that Heads of Mission and Secretaries are incentivised to pursue better development outcomes. While some advocated for a return of the Office of Development Effectivess or the appointment of a Chief Evaluator, others felt it was time to look to external best practice by increasing DFAT’s healthy competition with other central agencies that might be ahead in using evaluation outcomes to drive impact.