Introduction What you need to know Engagement of key stakeholder groups in operations financed by the Asian Development Bank (ADB) promotes good governance, transparency, innovation, responsiveness, and development effectiveness. Effective engagement of stakeholder groups, including civil society, project beneficiaries, and project-affected people, requires the understanding and effective use of participatory tools throughout the project cycle. However, while one participatory tool may work well in one context, it may not be appropriate in another. This series of explainers provides a range of tools from which practitioners can pick and choose, according to different phases of the ADB project cycle, context, and available time/resources. Some tools may be specific to particular phases in the ADB project cycle, such as monitoring and evaluation tools, while others may be used throughout the project cycle, such as participatory assessment tools. This piece focuses on Tools for Monitoring and Evaluation. Tools for Monitoring and Evaluation Participatory monitoring and evaluation is the process of engaging key stakeholders in the evaluation of a project or program. Stakeholder participation can occur at any phase of the project monitoring and evaluation cycle. Stakeholders can participate in the evaluation design, including forming the evaluation questions and choosing the evaluation approach, collection of data, analysis of data, preparing recommendations and how results are used. Participatory engagement of stakeholders provides for improved development results and is fundamental to good governance and transparency. Appreciative Inquiry What Appreciative Inquiry is a method for monitoring, evaluation, and organizational change that focuses on strengths and successes rather than problems and weaknesses. It draws on a strengths-based approach as opposed to a needs-based or problem-based approach. It asks organizations to “first to discover what is working particularly well and then to envision what it might be like if ‘the best of what is’ occurred more frequently.”* It focuses on what is the best in an organization or development initiative. It asks what is working well, what inspires people, what makes people proud. Appreciative Inquiry generates opportunities for people to dream about what the best situation would look like and encourages and enables people to be positive. *T. T. Catsambas and H.S. Preskill. 2006. Reframing evaluation through appreciative inquiry. Thousand Oaks, Calif: SAGE Publications. Why It builds stakeholder engagement and focuses on what is working well. It does not ignore problems; rather it focuses on what is working well and how this can be replicated. It is particularly useful where there is discord or hostility among or within stakeholders, previous evaluation efforts have failed or there is fear or scepticism about evaluation. When It can be used as a monitoring or evaluation tool and therefore can be used during project implementation (to monitor progress) or in a post-project evaluation. Also, appreciative questioning can be used at the design and preparatory phase of a project. Who This can be undertaken by an ADB social development specialist, a local or international CSO, or by implementing agency staff. All will require training in using the Appreciative Inquiry approach if they have not been exposed to it before. How Central to the understanding of Appreciative Inquiry is the use of language and appreciative questions. There are different models for Appreciative Inquiry, including the 4-I model (Inquire, Imagine, Innovate, Implement) and the 4-D model of (Discover, Dream, Design, Delivery). The former is discussed below: Inquire. In this phase, most Appreciative Inquiry processes will include some variation on four foundational questions. These questions ask participants to describe a ‘peak experience’ of the organization or program under question, what they value, and their ‘three wishes’ for the effective functioning of the program or organization. Generic examples of these four questions are: Describe a high-point experience in your organization—a time when you were most alive and engaged. Without being modest, what is it that you most value about yourself, your work, and your organization? What are the core factors that give life to your organization, without which the organization would cease to exist? What three wishes do you have to enhance the health and vitality of your organization?* *A. T. Coghlan, H. Preskill & T.T. Catsambas. 2003. An overview of appreciative inquiry in evaluation. New Directions for Evaluation, 2003(100), pp 5-22. Imagine. Participants are then asked to ‘imagine’ or ‘dream’ of a positive future for their organization, project, or program. A direction to participants of an Appreciative Inquiry process at this stage may be: Imagine that it is 2-3 years from now and you are preparing for an awards ceremony to celebrate the program’s excellence. The ADB wishes to write an article on this exceptional program. You are so proud to be part of this program. What is happening to make you proud? What are people saying? What is happening internally in the program or organization? What changes or events made this success possible? Innovate. Participants are asked to develop “provocative propositions (also called design statements, opportunity or possibility statements)." The aim is to build on past successes to develop actionable possibilities for the program or organization’s future. This is the most difficult phase of Appreciative Inquiry and may stretch over months. Implement. The final phase is where the organization implements the visions previously described and agreed. It also includes developing plans for monitoring progress, evaluating results and supporting/celebrating success. References and Further Reading ADB. 2008. Appreciative Inquiry. Manila. Appreciative Inquiry Commons. D. Cooperrider, D. Whitney and J. Stavros. 2007. Appreciative Inquiry Handbook. San Francisco: Berrett-Koehler H. Preskill, & T.T. Catsambas. 2006. Reframing Evaluation through Appreciative Inquiry. Thousand Oaks, California. Sage Publications A. T. Coghlan, H. Preskill & T.T. Catsambas. 2003. An overview of appreciative inquiry in evaluation. New Directions for Evaluation, 2003(100), pp 5-22. Most Significant Change What Most Significant Change is a participatory monitoring and evaluation tool developed in Bangladesh in the 1990s by Jess Dart and Rick Davies. It asks simple questions about what is the most significant thing that has changed in a development initiative and why. It involves generating and analyzing personal accounts of change and deciding which of these stories is the most significant. It can be a stand-alone tool or can be used across different stakeholder groups to assess priorities for the different groups. Why This tool works well in storytelling societies, relevant to many of the cultures in the Pacific. Stories of change told by beneficiaries encourage dialogue and consensus building about significant perceived impacts. It is helpful in explaining ‘how’ change comes about (processes and causal mechanisms) and ‘when’ (in what situations and contexts). When This is applied during the project implementation phase to enable lessons learned to be incorporated and used while implementation is ongoing and at the end of project evaluation. Who It can be undertaken by CSOs, government officials, or ADB staff. It must involve field staff and beneficiaries from a project area as storytellers. How Dart and Davies provide 10 steps in implementing this tool: 1. How to start and raise interest 2. Defining the domains of change 3. Defining the reporting period 4. Collecting significant change stories 5. Selecting the most significant of the stories 6. Feeding back the results of the selection process 7. Verification of stories 8. Quantification 9. Secondary analysis and meta-monitoring 10. Revising the system. However, this tool is a flexible system and not all steps may be relevant to all projects. Dart and Davies identify three of these 10 steps as fundamental to Most Significant Change: these are steps 4 (collecting significant change stories), 5 (selecting the most significant of these stories) and 6 (feeding back to the project stakeholders the results). A typical step-by-step guide for an ADB project may be: 1. Decide what will be the focus or ‘domains of change’. This might be the changes in a project on the quality of people’s lives, the changes on their livelihoods; or the changes in relationships between key stakeholder groups. 2. Decide how significant change stories will be collected. The different methods for collection include: Fieldworkers write down unsolicited stories they have heard. Interviews or focus groups with key beneficiaries or implementers. Through paired interviews in a workshop setting. The beneficiary writes the story of change themselves. For an ADB-assisted project, a local CSO may be engaged to interview beneficiaries, or the ADB CSO anchor could interview implementing agency staff and other key stakeholders. 3. Collect the stories. Depending on the method used to collect the stories Information should be recorded on who collected the story, what the story is, and the significance of the story (to the storyteller). Stories collected should be written in the words of the storytellers, to document the story as it was told. Stories should be collected from groups of key stakeholders/beneficiaries (street, village, district) affected by the project or policy. The writer would ask open questions within the selected domain such as "Looking back over the last month, what do you think was the most significant change in the quality of people’s lives in this community?" The writer probes why the story is significant to the storyteller, using a question like “Why is this significant to you?” 4. Select the most significant change stories. A group of stakeholders from the project area read aloud their individual significant stories of change. An in-depth discussion is held with participants on the value of change and then one story from each street/village/district is selected in a participatory manner of the most significant change. These stories of most significant change from different streets/villages/districts are then fed through a hierarchical process to the project team. The project team considers each selected story, then in turn holds an in-depth discussion and selects only one story that describes the most significant change story. 5. Feed back the results. Feedback is then given to the community over which stories were selected and why. Feedback is important as it completes a communication loop and provides information on what was the selection process for the most significant change stories. Sample Flow Source: R. Davies and J. Dart. 2005. The Most Significant Change (‘MSC’) Technique – A Guide to Its Use. p29. Case Study Evaluating a Pacific educational leadership master’s program using Most Significant Change The Masters of Education in Educational Leadership course at the Queensland University of Technology in Australia is part of a national capacity building program for 18 Pacific Islanders. The university evaluated the course using the most significant change technique to determine the course’s impact on the individual students. The evaluation complemented the standard evaluation process each semester. The evaluators selected three methods to collect most significant change stories: individual interviews with the students focus groups, and reflective essays submitted by each student The individual interviews focused on the following three questions: What positive or negative changes have you observed in yourself and in your working life during the time that you have been studying for the Master of Education degree? Which of these changes do you consider to be the most significant? Why do you consider this change to be the most significant? The team recorded and transcribed the interviews. Then they analyzed for themes around the significant changes. Two student focus groups verified the findings from the interviews. The focus group format also allowed free-flowing discussion and debate around the significant changes. Lastly, the team analyzed the reflective essays for themes. The findings indicated that the key significant changes were: an increase in self-confidence change as a person, and recognition and respect by others. While most of the significant changes were positive, there were some significant changes that were negative: envy from others, an increased workload, and lack of recognition as ‘new leaders’ by supervisors. The university was therefore able to adapt the program based on the most significant change evaluation results. Source: S. Choy and J. Lidstone. 2013. Evaluating leadership development using the Most Significant Change Technique. Studies in Educational Evaluation. 39 (4), pp. 218-224. References and Further Reading ADB. 2008. The Most Significant Change Technique. Manila. Better Evaluation. Most Significant Change. R. Davies and J. Dart. 2005. The 'Most Significant Change' Technique - A Guide to Its Use. Overseas Development Insittute. Tools for Knowlede and Learning: Most Significant Change (MSC). S. Choy and J. Lidstone. 2013. Evaluating leadership development using the Most Significant Change Technique. Studies in Educational Evaluation. 39 (4), pp. 218-224. Social Audits What Social audits are accountability systems where citizens monitor and audit government policies and initiatives to make sure they are working for the needs of the people. They foster accountability in the use of public resources. Citizens monitor, track, analyze and evaluate government performance and the use of government funds. Why Social audits increase transparency and accountability on the use of public funds. It also encourages citizen engagement with policy-makers. Social audits enhance service delivery by promoting dialogue between citizens and government, improved access to information and creating incentives for government initiatives to be effective and efficient. They engage citizens as active participants in development. When Social auditing occurs at all stages of the project or policy development cycle: at the design stage, during implementation, and after a government policy or program has been put in place. Who Citizens, NGOs, CSOs, communities, the media, parliamentarians, and the private sector can conduct activities under social audits. How There are many different tools and approaches in social audits, which are each quite detailed. To learn more, visit these external resources on citizen report cards, public expenditure tracking surveys, community scorecards, gender audit, and child rights audit. Case Study Government of Timor-Leste supports social auditing The Government of Timor-Leste has promoted civil society involvement in social audits. In the collaborative reviews, the beneficiaries of public funds join government officials in evaluating projects, policies, and their implementation. The government expects civil society to play a key role in the social auditing process, which encompasses the health, agriculture, infrastructure, and education sectors. To formalize this cooperation, the government established a social audit unit and signed a Memorandum of Understanding with Timor-Leste’s NGO peak body, FONGTIL, on social auditing. The European Union has financed the preparation of a social auditing handbook and the Asia Foundation, with support from the Government of Australia, has worked with FONGTIL to establish the National Network for Social Auditing. Over 2014-2015, the National Directorate for Water Supply and WaterAid, working with the network of organizations and NGOs involved in rural water supply in Timor-Leste, BESI-TL, conducted a social audit of rural water supply services. The aim was improving service effectiveness and efficiency through a participatory social audit approach. A community scorecard tool was used to rate the quality of water services in eight communities in Timor-Leste. The community scorecard approach engaged with 246 adults across eight communities. The results indicated that the services did not completely satisfy the users and did not meet all the national/international standards for water quality. The community scorecard tool itself was found to be an effective tool for engaging communities and monitoring service levels. Sources: Emma Walters with additional reporting from The Asia Foundation’s Can Social Auditing Drive More Inclusive Development in Timor-Leste? Social Audit for Rural Water Supply Services in Timor-Leste Report References and Further Reading G. Berthin. 2001. A Practical Guide to Social Audit as a Participatory Tool to Strengthen Democratic Governance, Transparency and Accountability. UNDP: Panama. Centre for Good Governance. 2005. Social Audit: A Toolkit A Guide for Performance Improvement and Outcome Measurement. CGG. Hyderabad, India. FAO. Training module on social audit. The Asia Foundation. 2017. Can Social Auditing Drive More Inclusive Development in Timor-Leste? Some Do’s and Don’ts Do Plan for participatory evaluation at the design stage. Engage CSOs and communities early in the process and provide communities with opportunities for input into the selection of participatory evaluation techniques. Be clear on the purpose of the evaluation and the role that CSOs and communities are expected to play. Articulate the benefits that using a participatory approach will bring to the evaluation and to the stakeholders involved. Adequately resource the participatory evaluation effort. Inform CSOs and communities of the outcome of the participatory evaluation. Use the learnings from the evaluation in the design of future projects. Ensure that the evaluation utilizes the expertise and experience of women. Use simple evaluation tools. Don’t Have unrealistic expectations of the knowledge, skills and time availability of CSOs and communities – most will require significant training and support in using even the simplest M&E tools. This should be planned for in the evaluation design. Expect that all stakeholders will want to participate in the evaluation. Use overly complicated evaluation methodologies. Limit CSO and community engagement to data collection efforts. Give up! Participatory evaluation can empower communities and help future project design. Related Links Participation Tools for the Pacific - Part 1: Engaging Pacific Civil Society Organizations Participation Tools for the Pacific - Part 2: Stakeholder Analysis Participation Tools for the Pacific - Part 3: Design and Monitoring Framework Participation Tools for the Pacific - Part 4: Assessment Participation Tools for the Pacific - Part 5: Implementation Downloadable Copy Participation Tools for the Pacific - All Parts Participation Tools for the Pacific - Part 6 Ask the Experts Lainie C. Thomas Unit Head, Climate Change and Sustainable Development Department, Asian Development Bank Lainie Thomas heads the Fragile and Conflict-Affected Situations (FCAS) and Small Island Developing States (SIDS) unit. Prior to her current assignment, she supported civil society participation in ADB’s operations through its NGO and Civil Society Center and led health and education projects in Southeast Asia. Before joining ADB, she worked for a range of international and local nongovernment organizations. With more than 25 years of development experience, she has also worked in the field in Kenya, South Sudan, Azerbaijan, Somalia, Cambodia, and The Gambia. Emma Walters Management and Participatory Engagement Consultant, Asian Development Bank Emma has 20 years’ experience working with Australian and international organizations. Over the past eight years, she has worked on projects designed to increase civil society participation in ADB work, with a focus on the Pacific. Her expertise includes facilitation, training, and partnerships for international development. Since 2012, she has also worked with the University of Sydney as a soft-skills trainer on Australian-government funded short course programs on agricultural research in Africa. Suzanne M. Nazal Senior Social Development Consultant (Civil Society and Participation), Sustainable Development and Climate Change Department, Asian Development Bank Suzanne Nazal organizes knowledge events, develops knowledge products, and provides technical support related to strengthening civil society participation and engagement in ADB’s operations. She also co-manages the social media platforms for ADB’s NGO and Civil Society Center. Suzanne worked with several NGOs in the Philippines before joining ADB. She holds a master's degree in urban and regional planning from the University of the Philippines. Ninebeth S. Carandang Principal Social Development Specialist, Human and Social Development Sector Office, Sectors Group Ninebeth Carandang is in charge of ensuring safeguards compliance of ADB-funded projects in the Pacific region, supporting the engagement of civil society organizations at various stages of the project, and managing the resolution of project grievances including concerns raised by civil society. She has over a decade of work experience in the field of social development, poverty, and gender. Asian Development Bank (ADB) The Asian Development Bank is committed to achieving a prosperous, inclusive, resilient, and sustainable Asia and the Pacific, while sustaining its efforts to eradicate extreme poverty. Established in 1966, it is owned by 69 members—49 from the region. Its main instruments for helping its developing member countries are policy dialogue, loans, equity investments, guarantees, grants, and technical assistance. Follow Asian Development Bank (ADB) on Leave your question or comment in the section below: View the discussion thread.