Evaluation and COVID-19

EVALUATION DURING COVID-19 CRISIS

The COVID-19 pandemic is increasing risks and changing environments related to UNODC's mandated areas of work, most notably in corruption and organized crime. This means that the need for effective evaluation of UNODC's programmes and projects is as important as ever. IES has responded to the crisis by investing in resources to guide Evaluators and Programme/Project Managers, testing innovative approaches to data collection, and, sharing lessons learned with evaluation functions across the UN system. Read more in IES Web Story: Evaluation in the Time of COVID-19.

When the pandemic hit, IES had a large number of evaluations underway. But the field work required for those was no longer an option. It was clear that new procedures, protocols, methods and systems would be needed to ensure continued evaluation work, most importantly the collection of data. As part of IES' immediate response, a 1-page Brief (in English and Spanish) was prepared together with a dedicated Guidance Note for Programme/Project Managers and Evaluators. Several alternative evaluative activities were initiated and UNODC staff have appreciated the various new options for collecting evaluative evidence. 

The Guidance Note contains a decision matrix for specific actions based on the stage of the evaluation that shows Managers on how to adjust to COVID-19 in ongoing and planned evaluations. Moreover, there is guidance for evaluators on how to tailor evaluation methodologies, ensuring that no one is left behind, considering that certain data collection methods will not be feasible due to travel restrictions. Read more below and in the Guidance Note.

Apart from developing guidance resources for evaluators, IES has invested heavily in enhancing remote data collection. IES is further engaging in an even closer communication with Programme/Project Managers at HQ and in the field to ensure that evaluations will be finalized and the accountability needs of Member States are met. This has included ensuring better coordination with field offices to arrange e.g. phone interviews, further increasing using on-line interview for data collection and investing in new technology to ensure robust methodology continues to be implemented in all UNODC evaluations.

In June, IES also held two webinars with Q&A sessions for Field Offices and HQ on "Planning and undertaking evaluations in UNODC during the COVID-19 crisis" to give UNODC staff the chance to discuss the Guidance Note, ask questions and share ideas. 67 people attended the webinars, including Programme/Project Managers, as well as focal points involved with UNODC evaluations that are on-going or that are planned to start in the second half of 2020. 

The webinars were chaired by Ms. Katharina Kayser, Chief, IES, and Ms. Katherine Aston, Deputy Chief, IES, with IES Evaluation Officer, Mr. Carlos Asenjo Ruiz, as moderator. In addition, Mr. Jim Newkirk, Senior Evaluation Advisor at I4DI and development practitioner for over 35 years, currently leading the evaluation of UNODC work in West and Central Asia, shared his experience in adapting an evaluation in collaboration with UNODC field offices.

GUIDANCE FOR MANAGERS: PLANNING AN EVALUATION

When planning for an evaluation during the COVID-19 and other crises, all evaluations should:

a)  ensure a do no harm approach; b) adhere to UNEG Ethical Guidelines for Evaluationc) avoid overburdening and/or off-loading work onto field offices; d) safeguard the quality standards of evaluative work including representation of all stakeholder groups, leaving no one behind; e) only be undertaken when there is a clear plan for utility and the needs for the evaluation results outweigh the risks and costs associated. 

Read more and see the Evaluation Planning Decision Tree (p. 3) in the Guidance Note 

GUIDANCE FOR MANAGERS: ON-GOING EVALUATIONS

For an on-going evaluation during the COVID-19 and other crises, Programme/Project Managers should:

a) communicate on the information needs and any changes from the original ToR based on COVID-19; b) share any relevant data on COVID-19 responses and or impact to the programme; c) remain open to innovative data collection tools and methods; d) actively engage with IES and the evaluation team; e) ensure an evaluation design and final report that meet the evidence needs; f) ensure adequate representation, leaving no one behind. 

Read more and see the Evaluation Management Decision Tree (p. 5) in the Guidance Note 

GUIDANCE FOR EVALUATORS: RISKS FOR EVALUATIONS

IES has identified potential risks when conducting evaluations in the context of the current pandemic, requiring potential measures to mitigate:

a) bias against under-represented groups; b) lower quantity and quality of data collected; c) inability to conduct on-site-data collection; d) limited access to field sites ; e) limited availability of national evaluators; f) limited availability of field office staff and counterparts; g) health-related risks; h) hesitance to novel methods

Read more and see Table 1 on Risks and Mitigation Measures (p. 7) in the Guidance Note 

GUIDANCE FOR EVALUATORS: DATA COLLECTION OPTIONS

The current situation provides a window of opportunity to diversify the portfolio of data collection and analysis methods. IES has identified some options for remote data collection and analysis methods, as well as their limitations and specific requirements:

a) surveys via mobile phone, email, online tools; b) on-line discussion platforms; c) video calls; d) remote observation via on-line feed or video footage; e) web scapping; e) web search data analysis; f) crowd sourcing.

Read more and see Table 2 on Evaluation data Collection Options (p. 9) in the Guidance Note