Participation Tools for the Pacific - Part 6: Monitoring and Evaluation
Appreciative Inquiry, Most Significant Change, Social Audit are activities that can be used to engage stakeholders in project evaluation.
What you need to know
Engagement of key stakeholder groups in operations financed by the Asian Development Bank (ADB) promotes good governance, transparency, innovation, responsiveness, and development effectiveness. Effective engagement of stakeholder groups, including civil society, project beneficiaries, and project-affected people, requires the understanding and effective use of participatory tools throughout the project cycle. However, while one participatory tool may work well in one context, it may not be appropriate in another. This series of explainers provides a range of tools from which practitioners can pick and choose, according to different phases of the ADB project cycle, context, and available time/resources. Some tools may be specific to particular phases in the ADB project cycle, such as monitoring and evaluation tools, while others may be used throughout the project cycle, such as participatory assessment tools.
This piece focuses on Tools for Monitoring and Evaluation.
Tools for Monitoring and Evaluation
Participatory monitoring and evaluation is the process of engaging key stakeholders in the evaluation of a project or program. Stakeholder participation can occur at any phase of the project monitoring and evaluation cycle. Stakeholders can participate in the evaluation design, including forming the evaluation questions and choosing the evaluation approach, collection of data, analysis of data, preparing recommendations and how results are used. Participatory engagement of stakeholders provides for improved development results and is fundamental to good governance and transparency.
Appreciative Inquiry is a method for monitoring, evaluation, and organizational change that focuses on strengths and successes rather than problems and weaknesses. It draws on a strengths-based approach as opposed to a needs-based or problem-based approach. It asks organizations to “first to discover what is working particularly well and then to envision what it might be like if ‘the best of what is’ occurred more frequently.”* It focuses on what is the best in an organization or development initiative. It asks what is working well, what inspires people, what makes people proud. Appreciative Inquiry generates opportunities for people to dream about what the best situation would look like and encourages and enables people to be positive.
Central to the understanding of Appreciative Inquiry is the use of language and appreciative questions. There are different models for Appreciative Inquiry, including the 4-I model (Inquire, Imagine, Innovate, Implement) and the 4-D model of (Discover, Dream, Design, Delivery). The former is discussed below:
- Inquire. In this phase, most Appreciative Inquiry processes will include some variation on four foundational questions. These questions ask participants to describe a ‘peak experience’ of the organization or program under question, what they value, and their ‘three wishes’ for the effective functioning of the program or organization. Generic examples of these four questions are:
- Describe a high-point experience in your organization—a time when you were most alive and engaged.
- Without being modest, what is it that you most value about yourself, your work, and your organization?
- What are the core factors that give life to your organization, without which the organization would cease to exist?
- What three wishes do you have to enhance the health and vitality of your organization?*
*A. T. Coghlan, H. Preskill & T.T. Catsambas. 2003. An overview of appreciative inquiry in evaluation. New Directions for Evaluation, 2003(100), pp 5-22.
- Imagine. Participants are then asked to ‘imagine’ or ‘dream’ of a positive future for their organization, project, or program. A direction to participants of an Appreciative Inquiry process at this stage may be: Imagine that it is 2-3 years from now and you are preparing for an awards ceremony to celebrate the program’s excellence. The ADB wishes to write an article on this exceptional program. You are so proud to be part of this program.
- What is happening to make you proud?
- What are people saying?
- What is happening internally in the program or organization?
- What changes or events made this success possible?
- Innovate. Participants are asked to develop “provocative propositions (also called design statements, opportunity or possibility statements)." The aim is to build on past successes to develop actionable possibilities for the program or organization’s future. This is the most difficult phase of Appreciative Inquiry and may stretch over months.
- Implement. The final phase is where the organization implements the visions previously described and agreed. It also includes developing plans for monitoring progress, evaluating results and supporting/celebrating success.
ADB. 2008. Appreciative Inquiry. Manila.
D. Cooperrider, D. Whitney and J. Stavros. 2007. Appreciative Inquiry Handbook. San Francisco: Berrett-Koehler
H. Preskill, & T.T. Catsambas. 2006. Reframing Evaluation through Appreciative Inquiry. Thousand Oaks, California. Sage Publications
A. T. Coghlan, H. Preskill & T.T. Catsambas. 2003. An overview of appreciative inquiry in evaluation. New Directions for Evaluation, 2003(100), pp 5-22.
Most Significant Change
Dart and Davies provide 10 steps in implementing this tool:
1. How to start and raise interest
2. Defining the domains of change
3. Defining the reporting period
4. Collecting significant change stories
5. Selecting the most significant of the stories
6. Feeding back the results of the selection process
7. Verification of stories
9. Secondary analysis and meta-monitoring
10. Revising the system.
However, this tool is a flexible system and not all steps may be relevant to all projects. Dart and Davies identify three of these 10 steps as fundamental to Most Significant Change: these are steps 4 (collecting significant change stories), 5 (selecting the most significant of these stories) and 6 (feeding back to the project stakeholders the results).
A typical step-by-step guide for an ADB project may be:
1. Decide what will be the focus or ‘domains of change’. This might be the changes in a project on the quality of people’s lives, the changes on their livelihoods; or the changes in relationships between key stakeholder groups.
2. Decide how significant change stories will be collected. The different methods for collection include:
- Fieldworkers write down unsolicited stories they have heard.
- Interviews or focus groups with key beneficiaries or implementers.
- Through paired interviews in a workshop setting.
- The beneficiary writes the story of change themselves.
For an ADB-assisted project, a local CSO may be engaged to interview beneficiaries, or the ADB CSO anchor could interview implementing agency staff and other key stakeholders.
3. Collect the stories.
- Depending on the method used to collect the stories Information should be recorded on who collected the story, what the story is, and the significance of the story (to the storyteller).
- Stories collected should be written in the words of the storytellers, to document the story as it was told.
- Stories should be collected from groups of key stakeholders/beneficiaries (street, village, district) affected by the project or policy.
- The writer would ask open questions within the selected domain such as "Looking back over the last month, what do you think was the most significant change in the quality of people’s lives in this community?" The writer probes why the story is significant to the storyteller, using a question like “Why is this significant to you?”
4. Select the most significant change stories.
- A group of stakeholders from the project area read aloud their individual significant stories of change.
- An in-depth discussion is held with participants on the value of change and then one story from each street/village/district is selected in a participatory manner of the most significant change.
- These stories of most significant change from different streets/villages/districts are then fed through a hierarchical process to the project team.
- The project team considers each selected story, then in turn holds an in-depth discussion and selects only one story that describes the most significant change story.
5. Feed back the results. Feedback is then given to the community over which stories were selected and why. Feedback is important as it completes a communication loop and provides information on what was the selection process for the most significant change stories.
Evaluating a Pacific educational leadership master’s program using Most Significant Change
The Masters of Education in Educational Leadership course at the Queensland University of Technology in Australia is part of a national capacity building program for 18 Pacific Islanders. The university evaluated the course using the most significant change technique to determine the course’s impact on the individual students. The evaluation complemented the standard evaluation process each semester.
The evaluators selected three methods to collect most significant change stories:
- individual interviews with the students
- focus groups, and
- reflective essays submitted by each student
The individual interviews focused on the following three questions:
- What positive or negative changes have you observed in yourself and in your working life during the time that you have been studying for the Master of Education degree?
- Which of these changes do you consider to be the most significant?
- Why do you consider this change to be the most significant?
The team recorded and transcribed the interviews. Then they analyzed for themes around the significant changes. Two student focus groups verified the findings from the interviews. The focus group format also allowed free-flowing discussion and debate around the significant changes. Lastly, the team analyzed the reflective essays for themes.
The findings indicated that the key significant changes were:
- an increase in self-confidence
- change as a person, and
- recognition and respect by others.
While most of the significant changes were positive, there were some significant changes that were negative:
- envy from others,
- an increased workload, and
- lack of recognition as ‘new leaders’ by supervisors.
The university was therefore able to adapt the program based on the most significant change evaluation results.
ADB. 2008. The Most Significant Change Technique. Manila.
Better Evaluation. Most Significant Change.
R. Davies and J. Dart. 2005. The 'Most Significant Change' Technique - A Guide to Its Use.
Overseas Development Insittute. Tools for Knowlede and Learning: Most Significant Change (MSC).
S. Choy and J. Lidstone. 2013. Evaluating leadership development using the Most Significant Change Technique. Studies in Educational Evaluation. 39 (4), pp. 218-224.
Social audits are accountability systems where citizens monitor and audit government policies and initiatives to make sure they are working for the needs of the people. They foster accountability in the use of public resources. Citizens monitor, track, analyze and evaluate government performance and the use of government funds.
Government of Timor-Leste supports social auditing
The Government of Timor-Leste has promoted civil society involvement in social audits. In the collaborative reviews, the beneficiaries of public funds join government officials in evaluating projects, policies, and their implementation. The government expects civil society to play a key role in the social auditing process, which encompasses the health, agriculture, infrastructure, and education sectors.
To formalize this cooperation, the government established a social audit unit and signed a Memorandum of Understanding with Timor-Leste’s NGO peak body, FONGTIL, on social auditing. The European Union has financed the preparation of a social auditing handbook and the Asia Foundation, with support from the Government of Australia, has worked with FONGTIL to establish the National Network for Social Auditing.
Over 2014-2015, the National Directorate for Water Supply and WaterAid, working with the network of organizations and NGOs involved in rural water supply in Timor-Leste, BESI-TL, conducted a social audit of rural water supply services. The aim was improving service effectiveness and efficiency through a participatory social audit approach. A community scorecard tool was used to rate the quality of water services in eight communities in Timor-Leste. The community scorecard approach engaged with 246 adults across eight communities. The results indicated that the services did not completely satisfy the users and did not meet all the national/international standards for water quality. The community scorecard tool itself was found to be an effective tool for engaging communities and monitoring service levels.
G. Berthin. 2001. A Practical Guide to Social Audit as a Participatory Tool to Strengthen Democratic Governance, Transparency and Accountability. UNDP: Panama.
Centre for Good Governance. 2005. Social Audit: A Toolkit A Guide for Performance Improvement and Outcome Measurement. CGG. Hyderabad, India.
The Asia Foundation. 2017. Can Social Auditing Drive More Inclusive Development in Timor-Leste?
Some Do’s and Don’ts
- Plan for participatory evaluation at the design stage.
- Engage CSOs and communities early in the process and provide communities with opportunities for input into the selection of participatory evaluation techniques.
- Be clear on the purpose of the evaluation and the role that CSOs and communities are expected to play.
- Articulate the benefits that using a participatory approach will bring to the evaluation and to the stakeholders involved.
- Adequately resource the participatory evaluation effort.
- Inform CSOs and communities of the outcome of the participatory evaluation.
- Use the learnings from the evaluation in the design of future projects.
- Ensure that the evaluation utilizes the expertise and experience of women.
- Use simple evaluation tools.
- Have unrealistic expectations of the knowledge, skills and time availability of CSOs and communities – most will require significant training and support in using even the simplest M&E tools. This should be planned for in the evaluation design.
- Expect that all stakeholders will want to participate in the evaluation.
- Use overly complicated evaluation methodologies.
- Limit CSO and community engagement to data collection efforts.
- Give up! Participatory evaluation can empower communities and help future project design.
Leave your question or comment in the section below:
YOU MIGHT ALSO LIKE
The views expressed on this website are those of the authors and do not necessarily reflect the views and policies of the Asian Development Bank (ADB) or its Board of Governors or the governments they represent. ADB does not guarantee the accuracy of the data included in this publication and accepts no responsibility for any consequence of their use. By making any designation of or reference to a particular territory or geographic area, or by using the term “country” in this document, ADB does not intend to make any judgments as to the legal or other status of any territory or area.