In my previous blog posting yesterday, I wrote about a recent trip I made to visit the WorldFish-lead KSinR Pilot Project which has been tying out Outcome Mapping as a knowledge sharing approach to enhance a research project on fish culture activities in Vietnam.

But why Outcome Mapping? How does this serve as knowledge sharing? and what does it offer to improving the research process?

Outcome Mapping is relatively new approach, developed by IDRC to planning and M&E. The theory behind Outcome Mapping is that it focuses on one specific type of result: Outcomes as behavioural change. “Outcomes are defined as changes in the behaviour, relationships, activities, or actions of the people, groups and organizations with whom a program works directly.”

For more information on the Outcome Mapping approach- see:

So Outcome Mapping offers a new way of thinking about achievements within a project–a different type of knowledge–which is valuable for a project such as the WorldFish-run CPWF 35 project.

This project found that monitoring technical aspects alone such as fish numbers and types, snail population, etc was not enough to understand what was going on in the fish culture activities to provide support to the next season or to other groups. What was missing was a method which would also allow the project to understand what various groups were doing during the season, how did activities and behaviours change, what relationships formed and how did they work–all important knowledge to use in strengthening fish culture activities for those doing it and providing lessons for the project on the management of this collective activity.

Nets are used to create distinct ponds for fish culture activities in flooded areas. These are then managed collectively by a group--an activity which needs to be monitored and evaluated to generate lessons for further activities.
Nets are used to create distinct ponds for fish culture activities in flooded areas. These are then managed collectively by a group–an activity which needs to be monitored and evaluated to generate lessons for further activities.

Outcome Mapping apart from looking at a different type of knowledge also offers a different way of generating and sharing knowledge. Outcome Mapping is designed to be a participatory form of planning and M&E, taking into account the perspectives of various stakeholders both in the (intentional) design stage of Outcome Mapping for a project as well as in the monitoring and evaluation activities too.

Meeting with members of the farming club doing fish culture to discuss their vision, activities, experiences and lessons as part of using Outcome Mapping for M&E and learning

Outcome Mapping is meant to be interactive and based on the sharing of knowledge between a project and many of its stakeholders. The knowledge generated from this M&E approach is done in such a way that both two-way communication as well as learning are promoted. This is different from more traditional styles of M&E which consist of surveys or formal observation visits by project personnel (or consultants) who extract information from stakeholders, project sites and activities and then it resides in reports and databases. This offers little chance for others to learn from the process of M&E.

Since the WorldFish- run CPWF 35 project has been designed as an adaptive management approach, it is vital that lessons and experiences from one season of fish culture activities be fed back into the process for the groups of people who are carrying out and supporting such activities to learn from past seasons and readjust activities for the next season. A previous survey method used by the project was very complex, time-consuming, and data heavy. The results of the survey took a long time to collect, a long time to enter into a special database- leaving little time for analysis and with little prospect or mechanism for getting ‘results’ back to the people undertaking fish culture.

It was necessary to find a better way to carry out M&E within the project, in a way that could both better capture what was going on through including perspectives of those involved, as well as allowing an opportunity for learning by those involved in the fish culture activities themselves.

Thus Outcome Mapping was chosen as an approach to try…and we are following the efforts of this Project to use Outcome Mapping and making it available on the KS blog and website.

Advertisements

The Project Leader of the Knowledge Sharing in Research (KSinR) project recently visited the WorldFish-lead Pilot project in Vietnam.

The objectives of the trip were:

  • To observe the use of Outcome Mapping in the project
  • Provide support in the use of chosen KS tools
  • Capture lessons from the project
  • Collect materials and documents and take photographs (see trip photos on KS Flickr)
  • Discuss and conduct interviews with various stakeholders involved in the project
  • Plan for future project activities

So what is this project all about…

The WorldFish Centre submitted a proposal to the Knowledge Sharing in Research component of the KS project with the intention of “applying KS tools to impact monitoring and project M&E” to the CPWF 35 project on ‘Community-based fish culture in seasonal floodplains and irrigation systems’ that it is running. The rational was that as the overall project aims to develop appropriate technologies through an adaptive management approach it requires both national partners and direct beneficiaries at the community-level to evaluate fish culture activities each year and modify the following year’s approach based on the results. As such more participatory approaches to impact monitoring would provide a more appropriate and accurate picture of fish culture activities, including the perspectives of those involved, and make the learning directly accessible to those who are working in fish culture.

According to Natasja Sheriff, Project Leader of CPWF 35 and the KSinR Pilot project component:

“It is necessary to use approaches which adopt a more participatory approach to impact monitoring to ensure we are building a complete and accurate picture of project impacts and which allow project beneficiaries to more openly share their experience of project impact. Through more emphasis on sharing knowledge- we can all learn alot”

Through the Knowledge Sharing in Research project others can learn about the experience of this WorldFish Pilot project in using more knowledge sharing-oriented approaches to learning and M&E.

The trip–what was happening?

On this trip the Pilot Project Team was having to re-introduce Outcome Mapping to a new project site due to the previous site having decided not continue doing fish culture activities any longer. The activities of the week included:

*Presentation of and training in Outcome Mapping approach to local research partner- RIA2 again.

*Development of intentional design of Outcome Mapping by team, with inputs of VISIONS, BOUNDARY PARTNERS and PROGRESS MARKERS from various stakeholders such as WorldFish, RIA2, Local Authority, Fisheries Department, as well as the farmers themselves.

Keep tuned into this blog for a series of posts on the trip to Vietnam…

Natasja Sheriff, Project Leader of the WorldFish KSinR Pilot Project, together with Tonya Schuetz (IWMI)-who helped facilitate the WorldFish Pilot’s training workshop, wrote and submitted a paper for the workshop ‘Rethinking impact: Understanding the Complexity of Poverty and Change‘ which was convened in Cali, Colombia 26-29 March 2008.

The paper, entitled “Monitoring for change, assessing for impact: the WorldFish center experience” can be viewed from the following link- paper. This paper was based on the experience gained from the initial introduction to and use by the Project of Outcome Mapping and Most Significnat Change-the workshop which was described in a previous bog post.

According to the paper “like many CG centers, a traditional emphasis on the development and dissemination of new technolgies has shaped impact assessment within the WorldFish center” and ” assessing the impact of projects undertaken…has largely been quantitative in nature, applying economic models to assess productivity, welfare and technological efficiency for example”.

This paper contends that “in comparison to ex post impact assessment activities, less attention has been given to monitoring and evaluation, and to the process of learning and adaptation, during project implementation”.

This paper outlines the new trend of research towards a broader approach to addressing poverty alleviation and the move towards development and application of methods which increase the impact of agricultural research on poverty and which facilitate learning and change.

The paper posits that “there is a lack of appropriate, effective tools for participatory monitoring and evaluation for application in a natural resource management context” and “simultaneously there has been a trend towards increased partner collaboration and impact-oriented research which requires a more responsive and adaptive approach to impact assessment and M&E than has been previously applied”.

This was the driver behind the proposal of the CP35 project at the WorldFish Centre to the Knowledge Sharing in Research call for proposals, to pilot new M&E methods to compliment existing quantitative M&E tools, and to support a more open and responsive approach to change occurring in communities involved in the project. This paper outlines the initial experiences of the project in piloting new approaches to M&E and impact assessment mainly in the form of Outcome Mapping and Most Significant Change methods.

The paper provides a rationale for the choice of particular approaches, examining what potential fit and benefit they would have with the research project into which they would be integrated.

Next a clear description of the activities undertaken to introduce and initiate such methods was provided.

The real ‘meat’ of the paper comes in the strong review and analysis of the piloting of these kind of methods which is presented in the form of annotated lists of ‘benefits’ and ‘issues hindering effectiveness’ of each of the methods being employed. While there were some initial positive signs of benefits that would/could be derived from using such methods, the challenges, issues and concerns raised were more of a highlight.

Some benefits include:

  • Creating a longer term vision for sustainability and impact
  • Identifying unanticipated problems and constraints to project success
  • Revealing outcome and impact priorities held by project participants and stakeholders
  • Creating a sense of ownership and responsibility for project success
  • Clarifying roles and responsibilities
  • Articulating where change is needed and monitoring progress towards required change

Some disadvantages identifed were:

  • The potential for unequal power relationships amongst stakeholders (and even team) to influence the process/method
  • Relative complexity of the approach
  • Difficulty in communicating terminologies and processes in various languages
  • Substantial time investment of project team and stakeholders to work through OM
  • Potential for misinterpretation and inappropriate application of the concept of ‘behavioural change’

Although some negative consequences were described, these were proposed to be valuable learning experiences from which specific attention could be paid to relevant modifications and adaptations which could be made for future use in the project-in its other country sites.

The authors concluded, therefore that “there is a need to carefully evaluate alternative methodologies available to research scientists and to put forward appropriate tools for impact assessment and M&E that can be readily taken up and applied in R4D, particularly in the natural resource management context”.

On our last day and before lunch we hold a session based on the needs and knowledge gaps that we had identified the day before, and that participants considered as important for their projects. We used the session to introduce the Open Space Technology, even if we used it only to a certain extent. Participants could add session topics to those that we had previously identified. They were then asked to sign up for a maximum of two sessions. A short review of the numbers of sessions and the sign-ups, the group decided to go for two rounds of parallel sessions on 4 different topics:

  • Monitoring and evaluation of KS
  • How to choose a KS tool or method
  • How to identify user needs
  • Learning more about RSS, CMS and Yellow Pages

Here is a summary of the M&E discussion:

Two types of M&E have been discussed among Selam, Pete, Gauri, Petr, Maria Grazia, Nadia, Grace, and Sandra:
1) M&E of Web resources
2) M&E of KS

  1. On the first topic Pete Shelton pointed us to a blog he wrote on that topic where he lists a few potential indicators for measuring impact on the web while commenting their overall value and utility: Page views/visitors, downloads, citations, mentions in the media, RSS feeds, Search Engine rankings. Pete would like to identify IFPRI’s different audiences and perhaps use a planning, and M&E method to reach them more effectively.
  2. On the second topic we mentioned and briefly introduced some possible methods, like the Impact Pathways, Outcome Mapping, Innovation Histories or Most Significant Change. Those methods have in common that they provide ways to look at behavioral changes through SNA, storytelling, interviews, and more.

Gauri mentioned an effort at FAO to measure the usefulness of Communities of Practice at FAO. The number of members of s CoP (they are run on Dgroups) is not an indicator of success. Gauri asked for ways to find out more about their impact. The group suggested doing a survey combined with a set of interviews. Nadia raised the issue of the need to always combine different methods in order to cover different aspects of M&E.