This report presents the findings of a study undertaken by Itad for DFID to critically review the methods used to evaluate M4P programmes and thereby help guide the design and implementation of future evaluations.
Scope of this review
The review compiled information about the scope and purpose of evaluations of M4P programmes and analysed the trends, strengths, and weaknesses of these. Based on this analysis, it provided recommendations for best practice in M4P evaluation.
The reviewers analysed 32 M4P programme reviews and evaluations, monitoring and evaluation (M&E) frameworks and programme reports, including 14 evaluation reports, and consulted with 14 individuals and organisations in the field.
Key findings
M4P programmes are defined as playing a facilitative, adaptive role in order to contribute to systemic, large scale and sustainable market changes that positively affect the poor. The nature of the M4P approach and the complexity of the markets within which it operates present a number of challenges for evaluation. Evaluation approaches that address these challenges are needed to avoid inaccurate estimations of impact.
A number of general weaknesses with M4P evaluations were identified, particularly relating to:
- consideration of systemic, sustainable changes in market systems;
- data quality (small sample sizes with little consideration of sampling frames, statistical significance or bias);
- triangulation practices (particularly with regard to qualitative data collection);
- the use of theories of change (those used were often linear, not externally vetted, with assumptions not adequately tested);
- consistency in units for facilitating accurate aggregation; and
- consideration of unintended negative effects.
The report includes recommendations for best practice in relation to M4P evaluation methods (how), timing (when), responsibilities (who), and level of analysis (what).
Download the report to read more.
