Evaluation Handbook

III. D. Evaluation Team

 

1. Selection of Evaluators

In order to promote transparency and ensure the maximum objectivity of evaluations, UNODC relies on external evaluators selected on the basis of their competence, independence and integrity.

Evaluators are selected by means of a transparent process by the Project Manager.

Upon request, IEU may provide recommendations and curricula vitae from its own database of experts; other stakeholders may also have valuable suggestions which might help to identify potential evaluators. The relevant technical units should also be consulted for advice on prospective evaluators.

Please refer to the Chapter III tool: Guidelines for the selection of evaluators.

In the case of In-depth Evaluations, evaluators are identified by IEU in consultation with the CLP, often through a bidding process.

The Project Manager and the Core Learning Partners (including UNODC, the beneficiary Government and donors) can also be invited to propose evaluators, participate in the selection of the evaluation team members, and nominate observers to be assigned to the evaluation team.

The cost of the participation of observers is borne by the organization they represent.


TIP

Process
- Project Manager is responsible for selection & recruitment processes
Important Considerations
- Candidates are external and independent from the project
- Candidates MUST have experience in evaluation
- IEU recommends recruitment of an Evaluation Team to provide complementary skills (e.g. evaluation expert and subject matter expert)
- Gender balance in the evaluation team should be considered

2. Competencies

Competencies are identified in the ToR by the Project Manager.

To avoid conflict of interest and undue pressure, evaluations ought to be conducted by evaluators who have had no prior involvement in the design or implementation of the project or programme to be evaluated. These are usually independent consultants, external to UNODC.

Evaluators must have no vested interest and have the full freedom to conduct impartially their evaluative work, without potential negative effects on their career development. They must be able to express their opinion in a free manner.

3. Conflict of Interest

When selecting the evaluation team, the Project Manager must ensure that there is no conflict of interest. A Declaration of interest (see Chapter III tools) should be signed by the Evaluator(s).

A conflict of interest in consultant recruitment could be defined as a situation in which, because of a person's work history or possibilities for future contracts, the consultant may find himself/herself in a position to provide a subjective analysis in order to obtain undue benefits for himself/herself or affiliates, with a potential or actual bias against the interests of the employer [2].

The following potential sources of conflicts of interest could be identified:

a) Conflict of interest due to past engagement

Consultants should not be assigned to the evaluation of projects or programmes in which they have had prior involvement in the design, implementation, decision-making or financing stages. Typical examples of prior involvement include the inception, formulation, appraisal, supervision, support mission, or any other design or support activity for projects, programmes, corporate processes or policies to be examined by the evaluation.

b) Conflict of interest due to potential future involvement

There is a ceiling to the percentage of work that a consultant can perform for UNODC. In general, consultants with an UNODC work history that exceeds 25 per cent of their total work history will not be recruited. In addition, when consultants are recruited through a firm (reimbursable loan), a ceiling of 35 per cent of the overall total work history will be applied to the firm or institution in question. Further restrictions apply according to the task to be performed and are explained below.

Work history refers to professional experience, including consultancies, employment by UNODC as a staff member, as temporary staff or the equivalent.

c) Conflict of interest due to involvement in multiple assignments

Some restrictions are also placed on concurrent and future employment of consultants: they are to have no parallel assignments within UNODC during the contract period, and they should agree not to work with the division or department concerned by the evaluation for a period of six months after the expiration of the contract.

Other potential sources of conflict of interest that are not covered by the above provisions should be assessed on a case-by-case basis by the evaluation officers concerned in consultation with their supervisor [3].

If a conflict of interest is uncovered or arises during the evaluation, the organization should determine whether the evaluator should be dismissed or the evaluation terminated.

4. Composition

The composition of the evaluation team is identified in the ToR by the Project Manager.

The composition of the evaluation team should be gender balanced, geographically diverse and include professionals from the countries or regions concerned [4].

Getting the gender balance right in an evaluation team is important as female evaluators may have easier access to women and can encourage them to participate in discussions and express their views. However, even more important than the sex of the evaluators is their knowledge and commitment to gender issues. It is better to have on the team a gender-sensitive man who has received training on gender issues than a gender-insensitive woman.

An evaluation team consists of a team leader, and depending on the size and complexity of the project evaluated, of inter-disciplinary national and international team members with evaluation expertise. When needed, a translator could be part of the evaluation team.

For small-scale projects, one evaluator supported by the Project Manager might be able to undertake the whole evaluation on his or her own.

IEU generally recommends at least two evaluators per evaluation, one international and one national where they have complementary skills and abilities. However, this cannot always be accommodated. Given the scope of most Independent Project Evaluations, the evaluation can only be conducted by one evaluator with both technical and evaluation expertise. In this case, although technical expertise as regards the subject evaluated is important, priority should be given to a consultant with evaluation expertise. In addition, there should be distinction on whether the evaluator is a national or international recruit.

There are many benefits to hiring a national versus an international and vice versa. The rationale for hiring a national evaluator is that s/he allows for competencies that perhaps an international evaluator may not have (e.g. knowledge of local culture, fluency in local dialects, understanding of social norms, etc). Furthermore, involving national evaluators helps to strengthen the national evaluation capacity. Conversely, the rationale for hiring an international evaluator is that s/he may have more experience in specific evaluation tools and methods which would be of great benefit to any evaluation allowing for perhaps increased reliability of the findings and better learning, etc.

In addition to external evaluators, evaluations managed by IEU ( In-depth Evaluations) should always include in the evaluation team an evaluator from IEU, appointed by the Chief of IEU (in some cases, acting as the team leader). This staff member could also fulfil the function of IEU Evaluation Manager.

5. Roles and Responsibilities

The evaluation team leader acts as the primary liaison with the Project Manager in the respective Units and Sections at Headquarters and Field Offices. The evaluation team leader bears the primary responsibility for the evaluation, coordinates the input provided by the various team members and ensures the timely undertaking of the evaluation and a smooth evaluation process. This requires technical expertise, evaluation skills and experience, as well as good interpersonal, management, facilitation, writing and presentation skills. The specific tasks of the team leader ought to be reflected in the Terms of Reference.

Example

The following text provides an example of how a description of the team leader's tasks might read:

- Works closely with the evaluation manager and manages the team throughout the process to ensure that all aspects of the Terms of Reference are fulfilled;
- Undertakes a desk study and on this basis oversees the finalization of the evaluation methodology, in compliance with the evaluation norms and standards of the United Nations Evaluation Group (UNEG);
- Produces an Inception Report based on UNODC Evaluation guidelines;
- Undertakes relevant field missions and receives appropriate briefings;
- Presents and discusses preliminary findings in the field and at Headquarters, and considers received feedback and comments;
- Facilitates discussion among the team members on the findings, conclusions and recommendations;
- Drafts the evaluation report, with the inputs received from the different team members, to be circulated for comments and factual validation to evaluation stakeholders;
- Duly considers comments received from evaluation stakeholders, in particular comments as regards factual verification;
- Produces the final report;
- Presents the final evaluation report at a stakeholders workshop (if required).

Each team member is responsible for a certain part of the evaluation exercise, including the report writing. Usually the following tasks are shared: conducting some of the data collection and drafting parts of the evaluation report, including findings, conclusions and recommendations. The ToR ought to provide a detailed description of the qualifications required for each team member and the deliverables expected.

6. Qualifications and Skills

The qualifications and skills of the evaluation team are identified in the ToR by the Project Manager.

Independent evaluation consultants are selected for their evaluation and technical skills and experience and their local or country knowledge, depending on the specificity of each project. They must also have an understanding and experience of evaluation concepts, techniques and ethics, and be able to work as part of an international team. Whenever possible, local consultants should also be included since they are familiar with the local context and speak the local language. The involvement of local consultants also contributes towards building local evaluation capacity.

When a team of several consultants is needed, the team leader plays a key role in getting the team to work together in a smooth and efficient way. Besides having relevant evaluation and technical skills the team leader should therefore have effective management, interpersonal, facilitation, writing and presentation skills, as the team leader's working style may also affect the acceptance of evaluation results.

7. Joint Evaluation

There are several ways to approach the selection of evaluators for a Joint Evaluation.

- One option is to task one of the partners with recruiting the evaluation team, in consultation with the other partners;
- Another option is for each of the partners to contribute their own evaluators.

In some cases, the approach taken to the selection of evaluators may need to correspond to the funding modality.

- For example, if parallel financing is used, each partner might need to bring its own evaluator to the team;
- In cases where the contribution of another funding agency exceeds that of UNODC, the evaluation team leader may be nominated by that agency;
- In cases where each party brings its own evaluators to the team, evaluators may have difficulty in reporting to one actor while serving as a member of a joint team. To resolve this issue, all of the institutions involved should agree on the identity of the team leader at the onset, or delegate a particular agency to recruit the team leader and make clear to evaluators that the independence of the team will be respected and expected [5].

_________________________________

[2] IFAD Evaluation Manual

[3] IFAD Evaluation Manual

[4] UNEG Norms and Standards, 2005

[5] UNDP Evaluation Handbook

________________________________________________________

 

 
IEU Home
Evaluation Handbook Home
Table of Contents
Acronyms
Foreword
Introduction
Chapter I: Defining Core Concepts

Chapter II: Planning an Evaluation at the Design Stage

Chapter III: Managing an Independent Project Evaluation

Chapter III Tools:
Chapter IV: Undertaking an In-depth Evaluation
Chapter V: Undertaking a Participatory Self-Evaluation
Chapter VI: Using the Evaluation
Annexes
Annex I: Evaluation Glossary
Annex II: UNEG Norms
Annex III: UNEG Standards
Download as pdf
Chapter III: Managing an  Independent Project Evaluation 
Feedback
Feedback