Measuring impact: a marriage of numbers and stories
At Oxfam we're committed to monitoring, evaluation, accountability and learning to make sure we are having the most positive impact we can on the lives of people living in poverty. As part of this approach we need to find new and better ways of measuring our impact. Here, Anne Oudes, monitoring evaluation and learning advisor introduces the World Citizens Panel, an impact measurement method developed by Oxfam Novib.
We want the people we work for to tell us how we are doing so we can learn from this and improve. That's why Oxfam Novib has developed the World Citizens Panel (WCP) as an evaluation method. Key characteristics of the approach are, first of all, that we are using both qualitative and quantitative techniques (stories of change and surveys) to measure the impact of our programmes on people living in poverty and suffering from injustice. Secondly, the communities we work with provide the information that is used for evaluation themselves instead of external experts. Lastly, we put a lot of emphasis on learning by actively involving partners in the evaluation.
Since WCP's inception in 2012, impact measurements have been conducted in seven countries. We recently published the outcomes from Somalia and Cambodia. Besides the reports, major findings are visualised in infographics that show the relation between quantitative and qualitative information and present some of the key findings of the impact studies. In 2015, we are measuring impact in a wide range of programmes, including Mozambique, Mali, Niger, Nigeria, Sudan, Tunisia, Myanmar, Vietnam, Uganda, Pakistan and the Great Lakes region, among others.
Why is impact measurement so important?
Through the WCP people in poverty have a central voice in our monitoring, evaluation and learning system.
We measure impact to be accountable to the people that participate in our programme and to our donors. We want to be open and transparent about what works - and what doesn't. Through the World Citizens Panel people in poverty have a central voice in our monitoring, evaluation and learning system. Through active participation in the research process, our partner organisations are better equipped to measure impact themselves and to reflect on the results. Learning about what works is also crucial for us at Oxfam. By testing our theories of change we learn how to make change happen and improve the effectiveness of our work.
A big impact picture: surveys and stories
Changes are measured by comparing survey data over time (baseline and endline) and between project participants and non-participants (a comparison group). The surveys provide useful information about the scale of the impact achieved. Qualitative research then provides in-depth insight in how change came about. For this, we collect stories of change by asking project participants what has changed in their lives in recent years and how that change has come about.
Involvement
...we use a user-friendly smart phone app to collect survey data.
Our research methods are simple yet reliable. This enables our partner organisations to take an active part in the impact measurement. For instance, we use a user-friendly smart phone app to collect survey data. The data is received in our online database, which immediately presents preliminary analyses. Partner organisations can use the research methods easily, understand the outcomes quickly and use these for joint reflection and learning. At the end of each data collection period, we discuss the findings with key stakeholders and reflect together on how we can improve our work in order to achieve more impact.
Results
The impact reports from our work in Cambodia and Somalia show some outstanding results as well as areas where less impact was achieved than expected. For example, the study in Somalia showed that participation in the projects of Oxfam's partners had a positive impact on school enrollment, but there is clearly still much to do with respect to the quality of education. In Cambodia, the number of land concessions granted to external investors has increased despite the work that Oxfam and partners have done in this area. Awareness on land rights has grown, people have more frequently been able to file their complaints, but little follow up action (such as compensation) has been observed from the government.
Below: Oxfam's impact on land rights in Cambodia - key findings and lessons learned
Based on the outcomes of the assessment, Oxfam and partner organisations in the country have revised their strategies to become even more effective. For example, on the topic of land rights in Cambodia, Oxfam and partners have concluded that awareness raising and community mobilization are crucial steps in ensuring participants get and keep the rights to their land. They are now placing more emphasis on facilitating networking among communities and among organizations to increase the chance of success in securing land rights.
Look out for more World Citizens Panel findings in the next year. For more information on our approach, have a look at our website or contact us directly at worldcitizenspanel@oxfamnovib.nl. We would love to hear from you!
