M&E must be planned carefully, bearing in mind that no two projects are the same. Many agencies have developed generic evaluation criteria, which can be helpful. For example, the evaluation criteria set out by the Organisation for Economic Cooperation and Development’s Development Assistance Committee (OECD-DAC) are widely used in development projects, and often in humanitarian actions and DRR as well (see Box 18.1: OECD-DAC evaluation criteria). However, such frameworks and approaches should not be adopted thoughtlessly: they cannot be applied to every situation. They can be used to start discussions about what to evaluate and how to go about it, but the evaluation should be designed with the specific project in mind.
Relevance: The extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor.
Effectiveness: A measure of the extent to which an aid activity attains its objectives.
Efficiency: Measuring the outputs in relation to the inputs.
Impact: The positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended.
Sustainability: Measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn.
Connectedness: The need to ensure that activities of a short-term emergency nature are carried out in a context that takes longer-term and interconnected problems into account.
Coherence: The need to assess security, developmental, trade and military policies as well as humanitarian policies, to ensure that there is consistency and, in particular, that all policies take into account humanitarian and human rights considerations.
Coverage: The need to reach major population groups facing life-threatening suffering wherever they are.
Efficiency: Measuring the outputs – qualitative and quantitative – achieved as a result of inputs. This generally requires comparing alternative approaches to achieving an output, to see whether the most efficient approach has been used.
Effectiveness: The extent to which an activity achieves its purpose, or whether this can be expected to happen on the basis of the outputs. Implicit within the criterion of effectiveness is timeliness.
Impact: the wider effects of the project (social, economic, technical, environmental) on individuals, gender and age groups, communities and institutions. Impacts can be intended and unintended, positive and negative, macro (sector) and micro (household).
OECD-DAC, DAC Criteria for Evaluating Development Assistance (Paris: OECD-DAC, 2000), http://www.oecd.org/development/evaluation/49756382.pdf; ALNAP, Evaluating Humanitarian Action Using the OECD-DAC Criteria: An ALNAP Guide for Humanitarian Agencies (London: ALNAP, 2006), http://www.alnap.org/material/78.aspx.
Evaluations can take many forms, including real-time evaluations, after-action reviews with communities, internal or self-evaluations by project staff and partners, and formal, externally-led evaluations. Evaluation guidelines sometimes divide them into three main kinds: summative (judging the merits and achievements of a project or programme), formative (to enhance project/programme learning, by understanding what worked and why), and developmental (to introduce new learning and ideas to organisations or the sector as a whole).+M. Buchanan-Smith and J. Cosgrave, Evaluation of Humanitarian Action: Pilot Guide (London: ALNAP, 2013), pp. 33–34. Mid-term evaluations are usually undertaken in longer projects. Impact evaluations long after the conclusion of the project are also valuable, but are rare.
The evaluation process should begin at the project design stage, when goals and objectives are set and logical or other results-based frameworks developed. Ideally, there should be a series of evaluations during and after the project, to permit longitudinal analysis, although this rarely happens. Evaluations should be scheduled at those points in the project where they can be most useful, principally at key moments for decision-making.
The purpose and methods of any monitoring exercise, review or evaluation should be clearly defined and agreed. Since it is almost never possible to assess everything, there must be some focus to the assessment, and its objectives must be realistic in relation to the resources that go into it. Thought should be given to such issues as:
Clear terms of reference are vital. Many problems with evaluations stem from a failure to achieve this clarity and reach agreement on it. Sufficient time should be set aside for this. Even the best plans can break down when confronted with reality in the field, so flexibility is essential. Good planning should allow for this.
Often, evaluation teams are not given the time or resources to do their work thoroughly. Overcrowded schedules are common. This limits time for preparation and in the field, forcing evaluators to place too much reliance on what may be very selective field evidence, on agency documents that may be incomplete or unavailable and on interviews in head offices. In consequence, many evaluations are little better than snapshots of an initiative, coloured by chance encounters and personal views. Experienced evaluators can compensate for this to some extent by drawing on their skills in identifying and gathering key data and their knowledge of similar initiatives elsewhere, but if they rely too much on their general knowledge they may miss features that are distinctive to the programme or project in question. However, snapshots can be useful. Small-scale or rapid assessments do provide valuable insights in some cases, especially when focused on a distinct aspect of risk reduction (as in Case Study 18.1: Reporting on disaster response).
On 11–12 November 2002 a cyclone warning was issued along the coast of the Bay of Bengal. A relatively weak cyclone struck, with high winds and heavy rain in several places. The Orissa State Branch of the Indian Red Cross used the event to assess the effectiveness of its disaster preparedness work. The initial assessment was based on telephone calls from local voluntary coordinators and emergency team members in eight locations. These conversations focused on the following:
The phone calls provided plenty of local detail. Using this, it was possible to build up a picture of the situation on the ground and actions taken almost as they happened, the effectiveness of warning and response mechanisms and factors affecting them, and variations between the locations. The phone call method was not seen as a substitute for field surveys, but it would not have been possible to carry out such surveys immediately after the event.
Orissa State Branch, Indian Red Cross Society, ‘Actions by 8 Red Cross Cyclone Shelter Communities in Orissa during Cyclone Warning (Nov. 11 to 12, 2002)’, mimeo, 2002.