Skip to content

Blog

Learning in real time: building evidence that supports delivery

Our Test and Learn blog series explores what it takes to learn and adapt while programmes are being delivered. Here we look at how real-time learning can connect timely evidence, contextual knowledge and live decisions, drawing on Itad’s Somalia experience.

In our previous blog, we explored how a theory of change can identify the assumptions that matter most and translate them into a focused learning agenda. The next challenge is to turn that agenda into a real-time learning capability: generating timely, proportionate evidence, combining it with contextual and operational expertise, and using it to decide what to test next while there is still time to adapt.

Test and Learn depends on evidence that is timely, proportionate and connected to live decisions. People also need regular opportunities to interpret it alongside their contextual and operational knowledge, reflect on its implications, and decide what to test next.

Real-time learning is therefore not a product. It is a capability built around delivery.

Why real-time learning is valuable for Test and Learn

The value of real-time learning lies in its connection to live delivery decisions. In complex programmes, assumptions are challenged, new questions emerge and choices often need to be made before comprehensive evidence is available.

The aim is not simply to produce more evidence, or to produce it faster. It is to ensure that the right evidence and learning are available while programmes can still act on them.

From evidence generation to learning loops

Real-time learning does not mean evaluating everything more quickly or replacing robust evaluation with rapid, less systematic evidence. It means shortening the distance between:

  • identifying an important uncertainty;
  • gathering evidence that is proportionate to the decision;
  • interpreting it alongside contextual and operational knowledge;
  • adapting delivery or refining the programme’s understanding; and
  • deciding what to test next.

Evidence can come from many sources, including monitoring data, partner reporting, targeted research, third-party monitoring, stakeholder feedback, contextual analysis, reviews and evaluation. What counts as appropriate evidence depends on the question and the decision it needs to inform.

The important thing is to connect these different forms of knowledge through an iterative learning loop in which you test what matters, make sense of what you learn, adapt, and test again.

What this looks like in practice

Itad, in partnership with Altai Consulting, delivered a monitoring, evaluation and learning partnership for a complex UK government portfolio in Somalia. Spanning a decade and multiple delivery phases, it illustrates what real-time learning can involve in practice.

Monitoring, targeted reviews, evidence synthesis and facilitated reflection helped build a stronger, more connected evidence base. Each exercise served a different learning need. Third-party monitoring (TPM) generated evidence on implementation and emerging results. Evidence synthesis brought together learning across the portfolio to inform programme redesign. Strategic reviews examined broader questions about how different parts of the portfolio worked together and adapted to a changing context. Combined, these sources linked learning about day-to-day delivery with wider strategic questions.

The strongest learning came not from any single method, but from how evidence was connected to decisions, interpreted collectively and retained over time.

Three lessons stand out.

Focus evidence on live decisions

Real-time learning starts by asking what the programme most needs to understand now.

Longer-term monitoring and evaluation remain important, but they can be complemented by focused, iterative exercises that examine emerging signals, implementation challenges or assumptions that have become especially important to a forthcoming decision.

In Somalia, targeted TPM looked beyond whether activities had taken place to examine early progress, implementation challenges, emerging outputs and lessons. The approach was designed to remain responsive to where activities and learning needs were most relevant, so evidence could inform decision while delivery was under way.

Do not collect evidence simply because it is available or routinely required. Start with the uncertainty or decision, then choose the depth and form of evidence it needs.

Connect evidence with contextual knowledge

Formal evidence is only one part of real-time learning. In fragile and volatile settings, delivery teams, local partners and advisers hold important knowledge about changing conditions, emerging risks, and how activities are working in practice.

Our Somalia experience demonstrates the value of bringing these sources together. Monitoring reviews, structured points to pause and reflect, regular exchanges and facilitated discussions enabled programme teams and partners to compare experience, test interpretations, and identify implications for delivery.

Evidence rarely explains its own significance. A monitoring finding may show that progress differs between locations, but local and operational knowledge is needed to understand why, assess whether the difference matters and decide how to respond.

This is why collective sensemaking is central to real-time learning.

Evidence becomes more useful when people can interpret it together in relation to the decisions they face.

Turn informal learning into organisational learning

Trusted relationships and informal networks can help complex programmes exchange information quickly and navigate uncertainty. But when learning depends on particular people and relationships, valuable knowledge can be lost as people move on.

There is also a risk of creating ‘learning gatekeepers’. Knowledge needs to move in both directions across the delivery chain, with commissioners, implementing partners, subcontractors and local partners able to contribute to and benefit from learning.

The answer is not heavier reporting. It is to support trusted relationships with a light but deliberate learning structure. This may include:

  • regular opportunities for teams and partners to reflect together;
  • concise records of decisions, assumptions and emerging lessons;
  • periodic synthesis across monitoring, research and programme experience;
  • purposeful handovers that preserve contextual and relational knowledge; and
  • revisiting the learning agenda as new questions emerge.

Our Somalia experience points to both sides of real-time learning: trusted relationships make rapid learning possible, while more systematic learning mechanisms help make it durable.

Capture enough of the learning to make it an organisational asset, without turning reflection into another reporting burden.

Keep adapting in the learning loop

A learning loop does not end when evidence answers the original question.

Evidence may challenge an assumption, reveal a new uncertainty, or show that the programme is asking the wrong question. Changes in context or strategic priorities can also make an earlier learning priority less important and bring another to the fore.

Programmes therefore need to revisit not only what they are doing, but what they are trying to learn and how they are learning it.

In Somalia, successive monitoring, synthesis and review exercises provided different perspectives at different stages of the programme. Together, they built a cumulative evidence base that supported ongoing delivery as well as key moments of reflection and redesign. This helped the programme move towards a more integrated delivery model, with greater visibility of opportunities to improve value for money and generate catalytic effects.

This is what distinguishes a learning system from a series of disconnected studies: each exercise deepens the programme’s evolving understanding, informs action and shapes the next question to test.

The takeaway

Learning in real time turns Test and Learn from a series of discrete exercises into a continuous way of working. It means:

  • focusing evidence on live decisions, rather than trying to learn everything at once;
  • connecting formal evidence with contextual and operational knowledge through collective sensemaking;
  • capture important learning so it does not remain within informal networks or individual relationships; and
  • adapt the learning agenda as understanding, decisions, and circumstances evolve.

The Somalia experience shows that no single method creates real-time learning. It comes from an interconnected system of targeted evidence, contextual intelligence, trusted relationships, information sharing and regular opportunities to reflect and adapt.

But evidence and learning loops are not enough on their own. Teams also need the incentives, authority, and organisational conditions to respond honestly to what they learn. The next blog in our Test and Learn series explores how programmes can create those conditions for adaptive delivery.

To learn more about how Test and Learn approaches can support adaptive delivery and evidence-informed decision-making, contact us or explore the other articles in our Test and Learn series