Mixing, matching and modifying methods in realist evaluation: should we be purists or pragmatists?

At Itad, we see choosing an evaluation approach as an art as much as a science: there’s always a lot to consider. As a result, our evaluation designs are often hybrids, drawing on a range of different approaches in order to tick all the boxes.

https://svgsilh.com/tag/pieces-1.html

Realist evaluation can be used alongside other evaluation approaches.

In fact, deciding which approach to use means considering the interplay between the evaluation questions, the approaches that might be feasible given the nature of the programme and its context, the various other interests and preferences of the commissioner, and of course what is possible given the resources available (see Elliot Stern’s and Barbara Befani’s evaluation design triangles). Yet, how do these considerations influence realist evaluations?

This second blog from the Itad Realist Evaluation Learning Group asks: how far can realist evaluation be modified, watered down, or ‘mixed and matched’ with other evaluation approaches? Or should we aim to be ‘purists’ in order to retain the integrity of the approach? We came up with four insights – what do you think? Let us know in the comments.

  1.      Be careful about saying you’re doing a ‘realist-inspired’ evaluation…

We all agreed that realist evaluation is more than an approach: it’s a way of thinking. More precisely, it’s a specific way of thinking about causality and how to test causal relationships, based on a realist understanding of reality and how we can know about it (see Gill Westhorp’s great introductory paper for a primer).  Applying it means ‘thinking like a realist’ at all stages of an evaluation, from design, through to sampling and data collection, and then undertaking analysis in a very specific way. We felt that approaches that just adopt ‘elements’ of realist evaluation are likely to miss this, especially when implemented by evaluators who haven’t applied the approach before. There’s a risk of falling into common traps like conflating mechanisms with interventions, or failing to grasp the nuances of context-mechanism-outcome configurations and how to develop and test them. There’s a real possibility that a ‘realist inspired’ evaluation has all the jargon of a realist evaluation but none of the benefits we highlighted in our first blog.

  1.       …but realist ways of thinking can be very helpful when used informally

On the other hand, we’ve found that the realist evaluation ‘way of thinking’ can be beneficial even when the application of the approach isn’t perfect. For example, in the BRACED evaluation the Itad team introduced realist ideas to project implementing partners, who have been reporting their results in terms of contexts, mechanisms and outcomes – and although not all of them became expert realist evaluators overnight, many partners said this encouraged them to really think through how and why they expected their programmes to work.  Some members of our learning group also feel that we take the realist way of thinking into other evaluations, now that we’ve internalised the approach. The practice of digging beneath the surface to find underlying causal mechanisms (and the contexts that enable or constrain them) can be especially helpful when getting to grips with what a programme is hoping to do, developing theories of change, or conducting interviews.

  1.      Other approaches might be needed to answer questions about contribution

In both the BRACED and BCURE evaluations, commissioners wanted to know whether the programmes had made a meaningful contribution to observed outcomes, and how important that contribution was in a context where many other factors are influencing change. We found this question was difficult to answer using realist evaluation alone. Realist evaluation will give you a tested theory in the form of a set of confirmed CMO configurations which explain what worked, for whom, in what circumstances. But these don’t necessarily add up to a clear picture of how important the programme’s contribution was to achieving any given outcome, in comparison to other factors. (*We’ve put a more techy breakdown of how we understand realist evaluation approach to contribution at the bottom of this blog, if you’re interested!) However, contribution analysis is specifically designed to answer the question ‘how far did my programme contribute to change’. Can the two approaches be combined?

One important challenge to this is that realist evaluation and contribution analysis seek to demonstrate causality in quite different ways, even though they are both use a generative model of causation. Contribution analysis works through establishing a plausible contribution claim – looking to increase our confidence that a programme contributed to change through weighing up the relative importance of the programme vs other factors, in a fairly pragmatic way. Realist evaluation establishes causality through a more formal process of developing and testing more nuanced, granular theories (CMOs), about how and why change happened.  How can you combine these two approaches in light of these differences?

  1.       It might be better to talk about ‘sequencing’ rather than ‘combining’ approaches.

In order to avoid muddying the theoretical waters, we felt it makes more sense to talk about ‘sequencing’ approaches rather than ‘combining’ or ‘merging’ them. This is the approach we took in BCURE and BRACED – applying a ‘contribution analysis’ lens and a ‘realist lens’ separately in a three step analytical process, that involved focussing in on a specific outcome and then sequentially ‘digging deeper’…

  1. First, establishing that an outcome did (or didn’t) happen.
  2. Then looking for evidence that the programme did (or didn’t) contribute, and the influence of other (non-programme) factors: the contribution analysis lens, in a rather top line way.
  3. Finally, digging deeper using the realist lens, to unpack how and why the programme did (or didn’t) contribute.

This is described further in the BCURE final evaluation report and annexes. While this solution may not be a pure application of either realist evaluation or contribution analysis, we found that it helped respond to the pragmatic needs of evaluation commissioners and users by providing a clear contribution judgement – while still generating the detailed, granular ‘how and why’ explanations that realist evaluation excels at.

These four insights are just scratching the surface. There are many other interesting conversations to be had about how realist evaluation can be combined with other theory based approaches including process tracing and QCA – and that’s not even getting into the heated debate currently ongoing in the realist evaluation community about whether it makes sense to do ‘realist RCTs.’ We’re looking forward to continuing these techy conversations over the coming months!

 

* More detailed musings about how realist evaluation deals with contribution:

Realist evaluation establishes causality through developing and testing granular theories (CMOs) about how and why change happened. While realist evaluation doesn’t explicitly talk about ‘programme contribution’, the programme is part of some theories, and not part of others. Through testing the theories, the evaluation aims to confirm that an outcome in x context emerged through y mechanism – and how the resources introduced by the programme gave rise to this (or how factors other than the programme gave rise to mechanisms that led to the outcomes).  Therefore you have attributed the change to a particular causal explanation which may or may not include the programme. In this sense realist evaluation is more binary than contribution analysis – the programme is either part of a tested, confirmed CMO or it is not (in x context).

  1. Edward Appollis

    This article has assisted me to see that I have used a hybrid approach and do not need to abandon my research because it is not a purist evaluation product. I am now embarking n my CMO configuration. Thanks for your help in this blog.

    1. Mel Punton

      Hi Edward – great to hear that the blog has helped your thinking. Yes, our experience definitely suggests that there’s no need to get too hung up on being ‘purists’ – as long as the core elements of the realist approach are internalised and applied, as as long as we’re clear about what we’re modifying, mixing or dropping. Good luck!

  2. Rick Davies

    Re “…testing of granular theories”, in the lasts para my query is what constitutes an adequate _test_ of a CMO? How do we know when it has failed?

    My second query is how you aggregate evidence relating to the presence of a particular causal mechanisms, relating to a given CMO

    1. Mel Punton

      Hi Rick,

      Great questions that we’ve been grappling with for some time.

      In my understanding, an adequate test of a CMO requires establishing a) that the C, M and O are all present, and b) the outcome is a result of this mechanism operating in that context (i.e. C+M = O). What the test looks like will depend on the nature of the C, M and O; the types of data that are required to establish their presence; and the methods more appropriate to collect this data. Establishing that C+M=O is the tricky bit – requiring the researcher to weave together various data sources and construct a plausible argument that the outcome occurred or did not occur as a result of x mechanism, operating in y context. Bayesian logic offers a lot of potential here, and we used it implicitly in BCURE. Rather than asking ‘is this CMO right or wrong’, we asked: ‘how confident are we that this CMO is a valid description of reality; how does the evidence help us update our confidence; what is the likelihood we would find this evidence if the CMO was valid / not valid?’ Ideally, you should also test rival CMOs (similar to the ‘alternative causal explanations’ in contribution analysis), to help avoid confirmation bias.

      In terms of how you aggregate evidence – this is where realist synthesis comes in. It is less about ‘aggregating’ and more about theory building – juxtaposing insights, reconciling contradictions, adjudicating between contradictory findings, consolidating different results into multi-faceted explanations, etc.

      Some of our thinking on this is contained in the BCURE annexes here (sections 3.7 and 3.8): http://itad.com/wp-content/uploads/2018/02/BCURE-Final-Evaluation-Report-Annexes-25-Jan-2018.pdf

Comments are closed.