United Nations Office on Drugs and Crime

Evaluation Handbook

III. C. Evaluation Terms of Reference (ToR)

Terms of Reference (ToR) are a written document presenting the purpose and scope of the evaluation, the indicative methods to be used, the issues to be addressed, and the resources, schedule and reporting requirements.

1. Roles and Responsibilities

The ToR of the evaluation are developed by the Project Manager.

IEU offers guidance during the drafting process, and reviews, comments on and clears the ToR. The Project Manager then informs the CLP about the evaluation, their role and asks them to comment on the evaluation questions.

Once the ToR are cleared, the Project Manager proceeds with the identification, selection and recruitment of the evaluation team, following the guidelines and clearance process set by IEU.

2. Structure

Most of the previous Sections (Please see Chapter II Sections C, D, E and Chapter III, Section A and B) of the present evaluation handbook (on purpose, scope, timing, budgeting etc.) actually come together in the various parts of the ToR, as described below.

The ToR of the evaluation reflect the Project Manager's understanding of the overall framework of the evaluation.

The ToR form an integral part of the agreement between UNODC and the independent evaluators. Well-considered and well-written ToR, with sufficient clarity and details, are therefore the foundation of a good evaluation and must contain the below content. Please see the Guidelines for Evaluation ToR in the Chapter 3 tools.

ToR Table of Contents

I.	Background and context
II.	Disbursement history
III.	Purpose of the evaluation
IV.	Scope of the evaluation
V.	Evaluation criteria and key evaluation questions
VI.	Evaluation methodology;
VII.	Timeframe and deliverables
VIII.	Evaluation team composition
IX.	Management of evaluation process
X.	Payment modalities
XI.	Annexes:
a.	Job descriptions of evaluators
b.	List of background documents for the desk review
c.	List of CLP members
d.	UNODC standard format and guidelines for evaluation reports

3. Evaluation Methodology

The evaluation methodology is the approach used to collect and analyze data on the programme or project being evaluated.

The Project Manager suggests an evaluation methodology; the evaluation team further develops the evaluation methodology, including the methodological tools, in the Inception Report. (Please see Chapter IV, Section C for further information on Inception Reports)

The evaluation methodology provides the basis for the credibility of the evaluation results.

The present paragraph provides the Project Manager with basic knowledge about evaluation methodology. This information should enable him/her to (i) draft the evaluation methodology paragraph in the ToR and (ii) judge the quality of the Inception Report provided by the evaluation team at a later stage of the evaluation. For further details on the evaluation methodology, please consult Chapter IV, Section C, about Inception Reports.

Project Managers should ask the evaluator to develop an evaluation methodology that:

-	Allows to answer the evaluation questions specified in the ToR (scope paragraph);
-	Selects adequate tools to gather the data required to answer these questions.

The methodology paragraph in the ToR could therefore include:

-	Possible identification of the methods for data collection (optional)
-	Identification of the sources for data collection (mandatory)
-	Triangulation of the methods and sources (mandatory)

a) Data Collection Methods

The Project Manager suggests the data collection methods to be used by the evaluation team to answer the evaluation questions. The data collection methods in the ToR should be indicative only, as the external evaluators will have to develop the concrete sampling strategy and data collection tools.

Methods include, but are not limited to, the following:

-	desk review
-	questionnaires
-	surveys
-	interviews
-	focus group
-	workshops
-	field visits
-	observations
-	case study

Focus group: qualitative evaluation methodology in which small groups of people are brought together to discuss specific topics under the guidance of a moderator

The data collection methods should ensure validity, reliability and credibility of the evidence gathered.

Validity: The extent to which the data collection strategies and instruments measure what they purport to measure.

Reliability: Consistency or dependability of data and evaluation judgements, with reference to the quality of the instruments, procedures and analyses used to collect and interpret evaluation data.

Credibility: The extent to which the evaluation evidence and the results are perceived to be valid, reliable and impartial by the stakeholders, particularly the users of the evaluation results.

To improve reliability and validity of the data collected:

-	Improve the quality of sampling by ensuring that the sample is representative of the population (e.g. clarify the characteristics of the sample, how it is selected, the rationale for the selection, and the limitations of the sample for interpreting the evaluation results).
-	Improve the quality of data gathering by (i) training data collectors to consistent data gathering, (ii) defining key words used in questionnaires or interviews, (iii) considering the characteristics of interviewers.
-	Use mixed methods of collecting data and building in strategies (triangulation) to verify or cross-check data using several pieces of evidence rather than relying only on one.

b) Data Sources

Data sources include, but are not limited to, the following:

-	documents
-	monitoring reports
-	evaluation reports
-	research papers or publications
-	national records
-	UNODC staff
-	Project/programme beneficiaries
-	Key stakeholders
-	Ministries

c) Triangulation

Data must be collected from various sources and through various methods to enhance reliability and validity of the evaluation findings. This means that the Project Manager should ensure that at least 3 different methods and 3 different sources are used in the collection of the data.

Triangulation is the use of three or more theories, sources or types of information, or types of analysis to verify and substantiate an assessment. By combining multiple data sources, methods, analyses or theories, evaluators seek to overcome the bias that comes from single informants, single methods, single observer or single theory studies. The purpose of triangulation is to enhance reliability of evaluation findings.

Triangulation of methods: the evaluator should have identified preferably three different methods for the purpose of revealing different aspects of the same reality and therefore for more reliability and credibility

Triangulation of sources: the evaluator should analyse when and why there are differences in the results from three different sources.

4. Planning for an Impact Evaluation in the ToR

The present paragraph is addressed to Project Managers willing to measure the impact produced by their projects/programmes.

Should Project Managers want to evaluate impact of their projects/programmes, they should stress in the ToR that impact evaluation is a requirement.

Therefore the evaluation team should reflect on evaluation methods to measure impact depending on the data that is already available, e.g.:

(i)	large scale sample surveys, in which the treatment and control groups are compared before and after, and possibly at several points during the project implementation; or
(ii)	small-scale rapid assessments and participatory appraisals, where estimates of impact are obtained from combining group interviews, key informants, case studies and available secondary data.

For further information on how to assess impact, please consult Chapter IV, Section C.

5. Declaration of Interest

Before undertaking an evaluation with UNODC, Project Managers should ensure that each evaluator completes a declaration of interest form (please see Chapter III Tools).

The declaration of interest form should be attached to the Terms of Reference of the evaluation.

Please see Chapter III, Section D. for more information on conflict of interest.

The full guidelines for UNODC Terms of Reference of evaluation are set out in the Chapter III Tools.

6. The Case of Joint Evaluation

a) Drafting the ToR

It is generally practical for one party to take the lead in drafting the ToR. After a draft is produced, it should be discussed and agreed upon by the CLP, as well as the partner organisation. It is important to satisfy the interests of all parties concerned in the ToR to the extent possible. Consideration should be given to creating a common agenda reflecting priorities that balance ownership with what is feasible.

b) Determining whose Procedures will be used

Since different organizations take different approaches to evaluation, it is important to allow flexibility to adapt and additional time to accommodate delays due to such differences. There are two common approaches to managing this issue: to agree that the evaluation will be managed based on the system and procedures of one of the partner organizations, or to split the evaluation into components and agree whose systems will be used to manage which components[1]. The ToR should reflect any decision as regards the procedures to be used.

_________________________________

[1] UNDP Evaluation Handbook

________________________________________________________

IEU Home

Evaluation Handbook Home

Table of Contents

Acronyms

Foreword

Introduction

Chapter I: Defining Core Concepts

Chapter II: Planning an Evaluation at the Design Stage

Chapter III: Managing an Independent Project Evaluation