Download Chapter
DFID/International Development Research Centre/Thomas Omondi

Chapter 18.2 Monitoring and evaluation

Planning and operation

Photo: DFID/International Development Research Centre/Thomas Omondi

M&E must be planned carefully, bearing in mind that no two projects are the same. Many agencies have developed generic evaluation criteria, which can be helpful. For example, the evaluation criteria set out by the Organisation for Economic Cooperation and Development’s Development Assistance Committee (OECD-DAC) are widely used in development projects, and often in humanitarian actions and DRR as well (see Box 18.1: OECD-DAC evaluation criteria). However, such frameworks and approaches should not be adopted thoughtlessly: they cannot be applied to every situation. They can be used to start discussions about what to evaluate and how to go about it, but the evaluation should be designed with the specific project in mind.

Box 18.1 OECD-DAC evaluation criteria

Development

Relevance: The extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor.

  • To what extent are the objectives of the programme still valid?
  • Are the activities and outputs of the programme consistent with the overall goal and the attainment of its objectives?
  • Are the activities and outputs of the programme consistent with the intended impacts and effects?

Effectiveness: A measure of the extent to which an aid activity attains its objectives.

  • To what extent were the objectives achieved/are likely to be achieved?
  • What were the major factors influencing the achievement or non-achievement of the objectives?

Efficiency: Measuring the outputs in relation to the inputs.

  • Were activities cost-efficient?
  • Were objectives achieved on time?
  • Was the programme or project implemented in the most efficient way compared to alternatives?

Impact: The positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended.

  • What has happened as a result of the programme or project?
  • What real difference has the activity made to the beneficiaries?
  • How many people have been affected?

Sustainability: Measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn.

  • To what extent did the benefits of a programme or project continue after donor funding ceased?
  • What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project?

Humanitarian action

Relevance/appropriateness:

  • Relevance: Assessing whether the project is in line with local needs and priorities (as well as donor policy).
  • Appropriateness: Tailoring humanitarian activities to local needs, increasing ownership, accountability and cost-effectiveness accordingly.

Connectedness: The need to ensure that activities of a short-term emergency nature are carried out in a context that takes longer-term and interconnected problems into account.

Coherence: The need to assess security, developmental, trade and military policies as well as humanitarian policies, to ensure that there is consistency and, in particular, that all policies take into account humanitarian and human rights considerations.

Coverage: The need to reach major population groups facing life-threatening suffering wherever they are.

Efficiency: Measuring the outputs – qualitative and quantitative – achieved as a result of inputs. This generally requires comparing alternative approaches to achieving an output, to see whether the most efficient approach has been used.

Effectiveness: The extent to which an activity achieves its purpose, or whether this can be expected to happen on the basis of the outputs. Implicit within the criterion of effectiveness is timeliness.

Impact: the wider effects of the project (social, economic, technical, environmental) on individuals, gender and age groups, communities and institutions. Impacts can be intended and unintended, positive and negative, macro (sector) and micro (household).

OECD-DAC, DAC Criteria for Evaluating Development Assistance (Paris: OECD-DAC, 2000), http://www.oecd.org/development/evaluation/49756382.pdf; ALNAP, Evaluating Humanitarian Action Using the OECD-DAC Criteria: An ALNAP Guide for Humanitarian Agencies (London: ALNAP, 2006), http://www.alnap.org/material/78.aspx.

Evaluations can take many forms, including real-time evaluations, after-action reviews with communities, internal or self-evaluations by project staff and partners, and formal, externally-led evaluations. Evaluation guidelines sometimes divide them into three main kinds: summative (judging the merits and achievements of a project or programme), formative (to enhance project/programme learning, by understanding what worked and why), and developmental (to introduce new learning and ideas to organisations or the sector as a whole).+M. Buchanan-Smith and J. Cosgrave, Evaluation of Humanitarian Action: Pilot Guide (London: ALNAP, 2013), pp. 33–34. Mid-term evaluations are usually undertaken in longer projects. Impact evaluations long after the conclusion of the project are also valuable, but are rare.

The evaluation process should begin at the project design stage, when goals and objectives are set and logical or other results-based frameworks developed. Ideally, there should be a series of evaluations during and after the project, to permit longitudinal analysis, although this rarely happens. Evaluations should be scheduled at those points in the project where they can be most useful, principally at key moments for decision-making.

The purpose and methods of any monitoring exercise, review or evaluation should be clearly defined and agreed. Since it is almost never possible to assess everything, there must be some focus to the assessment, and its objectives must be realistic in relation to the resources that go into it. Thought should be given to such issues as:

  • Indicators (see Section 18.7) – this is very important.
  • Units of assessment. M&E can take place at individual, household, group, community, institutional, district and national levels. Even in a large project, it is important to get as close to the grassroots as possible: data can be collated subsequently.
  • Sampling: sample size and sampling methods.
  • Scope. This is conditioned by the project’s coverage in terms of geographical area, hazards and risks addressed, and the number and types of vulnerable people assisted, as well as by factors such as remoteness, difficulty of access and security.
  • Existing information sources. Most evaluations draw on external sources (e.g. government and other agencies’ data sets and surveys) and internal sources (e.g. project documents) as well as field surveys. The quality and accessibility of external data is likely to vary. Agencies need to have knowledge management systems in place to identify and obtain relevant internal documents.
  • Who should be involved in collecting, providing and discussing evidence (see also Section 18.4: Accountability and participation). The size, composition and skills of evaluation teams are important considerations.
  • Scheduling. Reviews and evaluations should be scheduled at an appropriate point in the project’s lifetime, or after it has ended, and at suitable times of year (i.e. not at periods when communities are very busy, such as harvest time, or when weather conditions are difficult). When in the field, researchers must find appropriate times of the week or day for talking to the different beneficiaries.
  • Tools and methods. In the field, these may comprise formal surveys, structured or semi-structured individual and group interviews, group discussions such as focus groups and workshops, direct observation, community mapping, seasonal calendars, timelines, problem/solution trees, other participatory learning and action methods and case studies. Each method brings its own advantages and drawbacks. Project evaluations generally use several methods. The methods adopted must be appropriate to what is being assessed and the resources available to carry out the assessment.
  • Matching inputs and outputs. The evaluators must have enough time and resources to carry out the proposed activities and achieve the outputs required. Effective M&E also requires organisational capacity to support it: systems, financial resources and specialists in data management, analysis, reporting and M&E training.
  • How the findings will be reported back to all the stakeholders concerned, and how they will be acted upon. This is often neglected.

Clear terms of reference are vital. Many problems with evaluations stem from a failure to achieve this clarity and reach agreement on it. Sufficient time should be set aside for this. Even the best plans can break down when confronted with reality in the field, so flexibility is essential. Good planning should allow for this.

Often, evaluation teams are not given the time or resources to do their work thoroughly. Overcrowded schedules are common. This limits time for preparation and in the field, forcing evaluators to place too much reliance on what may be very selective field evidence, on agency documents that may be incomplete or unavailable and on interviews in head offices. In consequence, many evaluations are little better than snapshots of an initiative, coloured by chance encounters and personal views. Experienced evaluators can compensate for this to some extent by drawing on their skills in identifying and gathering key data and their knowledge of similar initiatives elsewhere, but if they rely too much on their general knowledge they may miss features that are distinctive to the programme or project in question. However, snapshots can be useful. Small-scale or rapid assessments do provide valuable insights in some cases, especially when focused on a distinct aspect of risk reduction (as in Case Study 18.1: Reporting on disaster response).

Case Study 18.1 Reporting on disaster response

On 11–12 November 2002 a cyclone warning was issued along the coast of the Bay of Bengal. A relatively weak cyclone struck, with high winds and heavy rain in several places. The Orissa State Branch of the Indian Red Cross used the event to assess the effectiveness of its disaster preparedness work. The initial assessment was based on telephone calls from local voluntary coordinators and emergency team members in eight locations. These conversations focused on the following:

  • When the cyclone warning was received, and from which source(s).
  • Actions taken by local disaster preparedness teams.
  • Actions taken by villagers.
  • Details of the event (wind speed, condition of the sea, rainfall) and its impact.

The phone calls provided plenty of local detail. Using this, it was possible to build up a picture of the situation on the ground and actions taken almost as they happened, the effectiveness of warning and response mechanisms and factors affecting them, and variations between the locations. The phone call method was not seen as a substitute for field surveys, but it would not have been possible to carry out such surveys immediately after the event.

Orissa State Branch, Indian Red Cross Society, ‘Actions by 8 Red Cross Cyclone Shelter Communities in Orissa during Cyclone Warning (Nov. 11 to 12, 2002)’, mimeo, 2002.