How Real-Time Evaluation Can Enhance Crisis Response

Although there is not one agreed definition of real-time evaluation, there is consensus that the approach should have one or more features, such as the collection of real-time data, real-time reporting of evaluation data, multiple timings of evaluative activity, multiple learning loops, and the rapid provision of information to decision makers to enable planning and course correction.

In practice, real-time evaluation is a formative evaluation of intermediary results. It is a dynamic tool used to provide real-time learning to enable rapid adjustments and course corrections. It bridges the gap between monitoring and evaluation in an emergency. For this reason, many see the role of a real-time evaluator as a facilitator, enabling stakeholders to make sense of the response, find solutions to remove blockages, and take corrective action while this can still make a difference. In this context, real-time evaluation is both a process and a tool to improve the quality and performance of response policies and programs.

Real-time evaluation is best used in situations where its potential benefits justify the costs of conducting it. This tool was first used by the international development community during the 1990s to examine humanitarian interventions in Africa and the Middle East. The first one was undertaken by the United Nations High Commissioner for Refugees (UNHCR) in 1991 in Iraq to examine the humanitarian impacts of the war. Over time, the approach has evolved to include the examination of a range of other crises, such as the impacts of civil conflict, natural disasters, and a global financial crisis. For example, the World Bank and the Asian Development Bank (ADB) both conducted real-time evaluations of their support to address the global financial crisis of 2008–2009. 

A successful real-time evaluation requires clarity and consensus among stakeholders about the purpose and scope of the evaluation and what would constitute a good report (speed verse rigor) and ensuring that the team undertaking the evaluation have the skills and resources to meet the standards and expectations.

Real-time evaluation must be carried out near to the start of a crisis or recovery response, not at the end or after completion. The duration of the evaluation can be short or iterative over a longer period to inform and support ongoing interventions and sometimes to garner lessons for future operations. It should be undertaken as a form of self-assessment or independent evaluation and not just for accountability and compliance purposes.

Data collection methods, such as, field visits, interviews, and big data, may be used. Findings must be reported immediately to intended users who are expected to use the report to make decisions and undertake course corrections (if needed).

In 2020, ADB’s Independent Evaluation Department conducted a real-time evaluation of the bank’s COVID-19 response. The team initially analyzed if there were sufficient will and absorptive capacity to adapt operations and policy amid the crisis. It may be best to wait until after the crisis to evaluate if an organization is not well placed to be agile or adjust its programs or policies in real-time.

Once convinced of the need and likely benefits, the evaluation team proceeded with postulating a clear theory of change, establishing a common understanding that the focus of the evaluation was on learning and not accountability, dealing with the risk of providing interim evidence that does not offer a full perspective of the situation, managing rapidly changing information and data, and overcoming logistical issues like travel restrictions.

The real-time evaluation approach used during the COVID-19 response will remain relevant and useful for development actors in the future. When applied effectively, this tool has tremendous potential to serve organizations, especially the decision makers and staff working under immense pressure with limited capacity to take on yet more work but with a desire and need to learn and adapt rapidly.