The Department for Foreign Affairs and Trade (DFAT) commissioned Itad to undertake a review of 2012 operational evaluations to address a need to improve the quality of its evidence in measuring the effectiveness of its aid.
The Review has two objectives: to promote good quality independent evaluations; and to inform the Minister, public, partners and aid program staff of overarching lessons emerging from the findings of independent evaluations.
Itad assembled a small team of 3 consultants with skills in synthesis evaluations, performance assessment and quality assurance. The assignment was delivered in two stages: 1) a review of the quality of all independent evaluations commissioned by DFAT in 2012. This involved developing a quality assurance template which was used to review all 87 operational evaluations and identify key strengths and weaknesses. For a sample of evaluation we then interviewed both the DFAT evaluation mangers and the evaluation teams to understand drivers and barriers to evaluation quality; 2) a synthesis evaluation of a sample of reports to draw out key lessons for DFAT. 64 evaluations which were considered of sufficient robustness in the quality review were taken through into a synthesis which sought to identify the main lessons for DFAT. Based on an initial review of the evaluations a series of hypothesis were developed, these were then further refined and revised based on further reviews of the report by the team. The strength of the evidence supporting the hypothesis was then assessed and the best evidenced hypothesis were taken through into the final synthesis.
Final report on the quality of Australian Aid operational evaluation can be found here: http://www.ode.dfat.gov.au/publications/documents/quality-of-aust-aid-op-evaluations.pdf
Final synthesis report on learning from Australian Aid operational evaluations can be found here: http://www.ode.dfat.gov.au/publications/documents/learning-from-aust-aid-op-evaluations.pdf