Evaluation Handbook

I. C. Types of Evaluation, Part II

 

3. In-depth Evaluations: Evaluation managed and undertaken by the Independent Evaluation Unit

An In-depth Evaluation is led and managed by IEU in collaboration with independent external evaluator(s) or specialised companies

In-depth Evaluations are usually large-scale strategic evaluations, such as country, regional, thematic programme or corporate-level evaluations, or cluster evaluations, which are planned in the IEU Biennial Work Plan.

The Executive Committee endorses IEU Biennial Work Plan, which is aligned to UNODC Strategic Framework and details the evaluation priorities for the two upcoming years. On an annual basis, IEU updates and revises the Work Plan based on consultations with UNODC management and Member States.

a) Planning

Undertaking In-depth Evaluations differs from undertaking other types of UNODC evaluations because of their relative size, geographical scope, the longer time frame covered and the greater complexity of methodological evaluation instruments used.

In-depth Evaluations usually require extensive consultation, take longer to complete, command significantly larger budgets, produce reports shared with a wider audience and are complex processes to manage. It is therefore recommended to plan In-depth Evaluations at least 8 months in advance.

b) Roles and Responsibilities

In-depth evaluations are undertaken by external consultants or specialised companies alongside IEU staff.

IEU also ensures that the internal and external stakeholders that will use the findings of these In-depth Evaluations are involved throughout the whole evaluation process as Core Learning Partners. Specifically, Core Learning Partners are involved in the evaluation process at the following stages: preparation of the terms of reference, selection of the evaluators, briefing of and interviews with the evaluation team, presentation of preliminary findings after the field mission, and providing comments on the draft evaluation report before finalization.

For further information on roles and responsibilities please see Chapter II, Section F.

Programme Evaluations:

Regional and Country Programme evaluations could be managed and led by IEU as any other In-depth evaluation. The roles and responsibilities for these Programme Evaluations are therefore identical to the In-depth Evaluations.

In line with the "integrated approach", ideally a multi-disciplinary evaluation team undertakes the programme evaluations. The team could therefore be composed of one expert per each sub-programme and headed by a lead evaluator tasked with the coordination of the evaluation.

An Evaluation Approach Paper is a reference document providing preliminary information on the evaluation process, and roles and responsibilities of the main evaluation stakeholders, including the Core Learning Partnership (CLP), within this process.

An Evaluation Approach Paper is drafted by IEU in order to guide the evaluation stakeholders in the preliminary steps to take for an evaluation. This ensures common understanding of an evaluation objectives, tasks and processes. It is complemented by the Terms of Reference. Please see the template for an Evaluation Approach Paper in the Chapter Tools.

The table of contents for an Evaluation Approach Paper is presented in the table below.

Table of Contents for an Evaluation Approach Paper

I. Rationale
II. Background
III. Evaluation purpose
IV. The Core Learning Partnership
V. The Evaluation team
VI. Communication and dissemination
VII. Proposed timeframe
VIII. References

c) Criteria

The range of possible In-depth Evaluations is wide, depending on the focus, purpose or complexity of the analysis required and on the human and financial resources available to conduct it. IEU usually conducts between two and four In-depth Evaluations each year depending on the resources available

Possible focus/subject of In-depth Evaluations


WHAT will be looked at in the evaluation?
  • Overall UNODC strategy and mandate
  • Specific theme (thematic programme) or result area within the UNODC strategy and mandate
  • Regional or country programme: geographical area (country or region), where one or several themes would be evaluated
  • Topic proposed by UNODC staff or donors as an area of particular interest
  • Global project or programme
  • Cluster of projects
  • Specific project or programme (for example, evaluation of a large project for which the Unit supports project and programme managers more intensively than during an Independent Project Evaluation)

Programme Evaluations

To support the move towards integrated programming and results orientation, IEU undertakes In-depth Evaluations at the integrated programming levels contingent on UNODC strategic priorities. IEU approach therefore adjusts to UNODC programming structure.

In line with UNODC strategic shift, IEU has developed an approach to the evaluation of regional and country programmes which is detailed in the green boxes. As Thematic Programmes undergo internal reviews they are excluded.

The depth and timing of the programme evaluations is decided on a case by case basis depending on the below criteria:

-    The maturity of the programme: whether the critical mass of activities has already been implemented and whether the building blocks of the Regional Programme (i.e. Country Programmes, Sub-Programmes and/or Global Projects) and of the Country Programme (i.e. projects, Sub-Programmes and/or Global Projects) have already been evaluated. Up-to-date Regional and Country Evaluation Plans are therefore instrumental to assess the maturity of the programme to be evaluated. For further information on Evaluability Assessments please see Chapter II, Section A, paragraph 1;
- The utility of the evaluation: what are the information needs and the ''return on investment'';
- The evaluability of the programme: an Evaluability Assessment could be carried out to identify the limitations (e.g. data constraints such as missing baseline and monitoring data) to the programme evaluations. It should be noted that such limitations could either affect the scope and methodology of the evaluation or prevent the evaluation from taking place. For further information on Evaluation Plans please see Chapter II, Section B.

d) Purpose

Possible purpose of In-depth Evaluations

 

WHY is this evaluation being conducte

  • To learn lessons and determine best practices that can be used for the design of new projects and programmes
  • To increase accountability towards donors
  • To share knowledge within UNODC and with key stakeholders
  • To make decisions about whether to extend or roll out a project or programme

Programme Evaluations

The purpose of Programme Evaluations is to:

-     Contribute to organizational learning by identifying the strengths and weaknesses of UNODC in the region or in the country and under each thematic area;
- Contribute to accountability by assessing the achievements of UNODC in the region or in the country;
- Contribute to decision-making in relation to UNODC strategic orientation in the region or in the country, and in thematic areas.

e) Scope

Programme Evaluations

Notwithstanding the adherence to the DAC Evaluation Criteria (Relevance, Effectiveness, Efficiency, Sustainability, and Impact), additional evaluation criteria such as programme funding (mobilization and utilization), synergy (internal and external - including partnerships and cooperation), design, institutional learning, and other cross-cutting issues such as gender and human rights are considered.

Programme evaluations assess the achievements of the programmes against their key objectives, including re-examination of the relevance of the objectives and appropriateness of the design. It also identifies factors that have facilitated or impeded the achievement of the objectives.

The scope of Regional Programme Evaluations includes assessment of:

-     The contribution to the objectives of the UNODC Medium-Term Strategy and of the UNODC Strategic Frameworks
- The contribution to the Millennium Development Goals (MDGs)
- UNODC regional/country offices' role in support of the Regional Programme
- The linkages between the Regional Programme and its building blocks (Country Programmes, Global Projects and projects)
- The integration and synergies between the various programming instruments (thematic, regional, country programmes)
- The phasing out of existing projects into programming
- The coexistence of in UNODC portfolio of programmes and national projects
- UNODC comparative advantage in the thematic area
- UNODC partnerships
- The resource mobilization
- The administrative and oversight processes
- The reporting mechanism(s) related to the Regional Programme
- The design of the Regional Programme

Regional Programme Evaluations aim in particular at answering, among others, the following questions:

-

To what extent does the Regional Programme contribute to the objectives of the UNODC Strategy and of the UNODC Strategic Frameworks?
-     To what extent does the Regional Programme provide a framework for Country Programmes?
- To what extent is the Regional Programme integrated? Are the building blocks of the Regional Programme (e.g. umbrella projects, Country Programmes, Global Projects and projects) reinforcing each other in a coherent and cohesive manner to achieve the common objectives stated in the Regional Programme?
- Are UNODC regional services and products relevant?
- Are UNODC mandates translated adequately into projects and programmes?
- How does the Regional Programme relate to the Thematic and Country Programmes?
- Are UNODC partnerships in the region efficient and effective?
- How efficient are the reporting mechanisms related to the Regional Programme?

The scope of Country Programme Evaluations includes assessment of:

-     The contribution to UN-wide mechanisms, such as UNDAFs and One UN Programmes. The issue of system-wide coherence of the UN has received particular attention since the launch of the "Delivering as One" initiative in eight pilot countries in 2006. Country programme evaluations therefore assess  UNODC involvement in UN collaboration mechanisms.
- The positioning of UNODC vis-à-vis other agencies
- The linkages/synergies that exist between the projects and the coordination, cooperation and integration processes
- The linkages between country and regional activities. Since regional projects often constitute a major part of UNODC activities in a country, these are also assessed in the country programme evaluation.
- The phasing out of existing projects into programming
- The pipeline projects

f) Methodology

In terms of methodology, In-depth Evaluations take the form of any other evaluation: sampling strategy, self-assesmment, field mission(s), questionnaire(s), or desk review(s) depending on how much has already been evaluated at the sub-levels of the subject evaluated (Project, Sub-Programme and Global Project levels). This will be decided on a case by case basis.

Methodology used and complexity of the analysis for In-Depth Evaluations

 

HOW detailed will the methodology be, keeping in mind the purpose of the evaluation and the time and resources available?

  • Level 1: desk review of relevant documents and evaluations achieved (utilization of self- and independent project evaluation reports)
  • Level 2: desk review of documents in addition to interviews over the phone and questionnaires sent by e-mail (without a need for travel)
  • Level 3: desk review of documents in addition to field visits, face-to-face interviews at Headquarters and in Field Offices and, possibly, specific data collection exercises (e-mail questionnaires, workshops, telephone interviews etc.)

In-depth Evaluations

Programme evaluations are carried out through analyses of various sources of information including desk analysis, survey data, focused group discussions and interviews with main stakeholders, field visits, on-site observation, etc. Special attention is paid to an unbiased and objective approach and the triangulation of sources, methods, data, and theories.

While maintaining independence, in-depth evaluations are carried out based on a participatory approach which seeks the views and assessments of all parties.

Evaluations of Projects, Sub-Programmes and Global Projects are the building blocks of programme evaluations; hence IEU carries out a meta-analysis of related project, Sub-Programme and Global Project evaluations before undertaking an In-Depth Evaluation at the Programme level.

In-depth Evaluations at the programme level focus on two interrelated dimensions: (i) the inter-linkages between programme components; (ii) the assessment of the overarching objectives of the programme concerned.

Sampling Strategy

In-depth Evaluations at the programme level do not consider all projects that fall under the time coverage of the programme considered, but only projects matching the below sample criteria (e.g. projects of a certain scale, considered particularly interesting or strategically important in relation to the purpose of the evaluation).

The evaluation should cover a portfolio that (i) is representative of what is done under a specific theme or in a particular region or country and (ii) allows the evaluation to answer the questions identified in the ToR.

The reasons for selection or exclusion of certain projects need to be explained in the ToR, if possible, or in the Evaluation Approach Paper.

The sampling criteria are as follows:

-     thematic areas addressed by the Regional Programme or the Country Programme
- budgetary execution of the projects (small and large projects)
- substantive execution of the projects (the critical mass of activities is delivered)
- types of partnerships: government, civil society and the private sector
- type of projects (pilots, innovative projects that were successful or that did not function properly and stood out for that reason)
- ongoing projects
- pipeline projects
- projects' performance: projects that performed well or that faced problems during execution

Methodology for the components of the Regional Programme:

The first step in defining the evaluation methodology is a mapping and grouping of the components of the Regional Programme. The following is a possible guideline for this process.

Building blocks (Country Programmes, Global Projects, individual projects) of the Regional Programmes that are due for an In-depth Evaluation [11] within the same timeframe as the regional programme evaluation:

The evaluation team may want to use these as ''case studies'' and include them as stand alone parts of the regional programme evaluation report.

Building blocks (Country Programmes, Global Projects, individual projects) of the Regional Programmes that do not formally require an In-Depth Evaluation within the same timeframe as the regional programme evaluation:

A desk review of the available documentation is carried out to assess the relevance and cohesiveness with the Regional Programme. In addition, the available progress information is validated through interviews with key stakeholders.

Building blocks (Country Programmes, Global Projects, individual projects) that have been evaluated within two years before the Regional Programme Evaluation:

The corresponding evaluations findings are used as inputs to the evaluation

Building blocks (Country Programmes, Global Projects, individual projects) that have been evaluated more than two years before the Regional Programme Evaluation:

A meta-evaluation and a review of implementation of the evaluation recommendations is carried out.

Specific issues or themes:

The regional programme evaluation can focus on one or several key issues or themes that are of interest to the stakeholders in the region (e.g. review of corruption, review of implementation modalities, review of the effectiveness of resource mobilisation, etc.). The components related to the issue or theme should be clustered for the evaluation.

The evaluation team may want to use these as case studies and be stand alone parts of the regional programme evaluation report.

Pipeline projects:

A desk review of the available documentation is carried out to validate the foreseen intervention logic/design with a special focus on the relevance and cohesiveness with the Regional Programme.

Methodology for the Regional or Country Programme:

Once the components of a Regional or Country Programme are mapped and grouped, the following methodology is applied:

-   

Self-assessment, in form of an on-line survey,  of UNODC strategic positioning is based on the logical framework, and, possibly, on:

-                global relevance
- response capacity
- value added and comparative advantages
- partnerships
- UNODC contribution to the coordination of the UN system
- internal cohesion

This self-assessment is carried out by the Regional or Country Programme Coordinator, and involves the Field Representative, government counterparts, the relevant HQ based staff, and the implementing partners and other relevant stakeholders.

-    Desk review, including the review of the evaluation reports from OIOS and JIU, and the review of the audits reports (Secondary Data)
- UNODC staff interviews
- Quantitative and Qualitative Questionnaires. Surveys and interviews.
- Reconstruction of before and after data,  treatment and control groups
- ''Case studies'' to generate specific lessons, e.g. thematic or country programme case studies
- Assessment of the Regional Programme's results against planned objectives using the sampling strategy mentioned above.

4. Joint Evaluations

A joint evaluation is an evaluation to which different agencies and/or partners participate. Note: There are various degrees of "jointness" depending on the extent to which individual partners cooperate in the evaluation process, merge their evaluation resources and combine their evaluation reporting. Joint evaluations can help overcome attribution problems in assessing the effectiveness of programmes and strategies, the complementarity of efforts supported by different partners, the quality of aid coordination, etc [12].

"The joint evaluation approach became popular in the 1990s with the promotion of the approach through the DAC Principles for Evaluation of Development Assistance, which stated, "Joint donor evaluation should be promoted in order to improve understanding of each others' procedures and approaches and to reduce the administrative burden on the recipient." [13]. The Paris Declaration also reinforced the joint evaluation approach through the commitment made by development agencies and partner countries to find more effective ways of working together. [14]."

Any evaluation can be conducted as a joint evaluation in the case of jointly funded or implemented programmes/projects with Member States or partners, such as other United Nations and multilateral organizations.

UNODC undertakes Joint Evaluations with the objectives of:

(i)      increasing harmonization and cohesion of international aid;
(ii)  reducing the burden on recipient countries by reducing the overall number of evaluations undertaken;
(iii) building ownership by increasing the acceptance and legitimacy of findings and recommendations;
(iv) sharing the burden of work involved;
(v) building mutual capacity and learning.

As regards to the evaluation process, the level of cooperation between UNODC and its partners can vary. However, Joint Evaluations in UNODC often involve a close cooperation during the planning and implementation phases. Partners jointly:

-     fund the evaluation;
- find the evaluator;
- draft the TOR;
- select or establish selection criteria for qualified, professional and independent evaluators;
- undertake field visits;
- draft the evaluation report;
- disseminate lessons.

As regards to the evaluation resources, partners' evaluation funds are not merged with UNODC evaluation funds. Partners individually finance their evaluation activities, including payment of their evaluators. Therefore undertaking joint evaluations do not preclude Project Managers from setting aside evaluation funds.

"The main disincentive to participate in a joint evaluation is the perceived cost. There is no doubt that joint evaluations are expensive. The direct costs for a large and complex joint evaluation, especially if it includes a number of case studies in developing countries, can easily reach over one million Euros.'' [15]

At the country level, one of the most obvious examples of a Joint Evaluation is the UNDAF evaluation, in which a number of UN organizations and the government participate. UNDAF evaluations are jointly commissioned and managed by the heads of UN organizations and national governments. They are conducted by external consultants selected by mutual agreement between the United Nations and the government through a transparent and thorough selection process. The 2007 CCA and UNDAF Guidelines could be consulted for more information.

For further guidance on how to plan, manage and undertake a Joint Evaluation, see Chapters II, III and IV.

TIP

Please refer to UNIDO-UNODC Joint Evaluation: Independent External Evaluation Post-Opium Surpass Poverty Project Oudomxay Province, Lao PDR.

http://www.unodc.org/documents/evaluation/ProEvals-2009/ProEvals-2010/final_report_psp_lao_rev.pdf

_________________________________
[11] For which an evaluation is mandatory according to the evaluation provisions in the programme or project document.
[12] OECD-DAC definition

[13] OECD, 'DAC Principles for Evaluation of Development Assistance', Development Assistance Committee (DAC), Paris, France, 1991, p.8. Available at: http://www.oecd.org/dataoecd/21/32/41029845.pdf.

[14] OECD, 'DAC Guidance for Managing Joint Evaluations', Development Assistance Committee

(DAC), Paris, France, 2006. Available at: http://www.oecd.org/dataoecd/28/14/37484787.pdf.

[15] OECD/DAC, Joint Evaluations: Recent experiences, lessons learned and options for the future, 2005


 
IEU Home
Evaluation Handbook Home
Table of Contents
Acronyms
Foreword
Introduction
Chapter I: Defining Core Concepts
Chapter I Tools:
Chapter II: Planning an Evaluation at the Design Stage
Chapter III: Managing an Independent Project Evaluation
Chapter IV: Undertaking an In-depth Evaluation
Chapter V: Undertaking a Participatory Self-Evaluation
Chapter VI: Using the Evaluation
Annexes
Annex I: Evaluation Glossary
Annex II: UNEG Norms
Annex III: UNEG Standards
Download as pdf
Chapter I: Defining Core Concepts
Feedback
Feedback