Evaluation of pilot programmes of substitution therapy of opioid dependence: Lessons learnt from WHO study

David Newcombe

World Health Collaborating Centre for Research in the Treatment of Drug and Alcohol Problems

Adelaide, South Australia

Overview

•      WHO Study: Manual for evaluation of substitution programme

–   Outcome evaluation

–   Process evaluation

•      What we did that helped in evaluation of pilot and new programmes

–   Planning

–   Training

–   Ongoing effective project management

 

Manual for outcome and process evaluation of substitution treatment

Outcome Evaluation

•      Outcome evaluation assesses treatment effectiveness – that is whether clients in a treatment service have changed over time and if this can be attributed to the care received.

•      Outcome areas explored:

–    Individual well being and health status.

–    Community/social benefits

–    Programmes performance

 

 

Process evaluation

•      Process evaluation is concerned with how a treatment programme operates and how resources are used to produce outputs. It also tries to identify where improvements could be made. 

 

•      The main aims of process evaluation are:

–    To describe the implementation of the programme

–    To describe service performance and service quality

–    To describe the acceptance of the programme by clients, staff, and community.

 

 

Evaluation Protocol

•      Clients assessed at baseline, 3- and 6- month

•      Outcome Evaluation

–    Social/Demographic details

–    Drug use and treatment history

–    Current Drug use

–    Criminal involvement

–    Physical and psychological health

–    Social functioning

–    HIV risk practices

–    Biological samples results

 

Evaluation Protocol cont.

•      Process evaluation

–    Programme implementation

–    Service description (maintenance treatment & HIV/Hepatitis prevention & management)

–    Service performance (maintenance treatment & HIV/Hepatitis prevention & management)

–    Service quality (staff)

–    Treatment perception (client)

–    Focus groups (staff, clients, professionals)

 

WHO manual for process and outcome evaluation of substitution therapy of opioid dependence

•      Proposed manual

–    Guidelines for the process and outcome evaluation

–    General procedures

–    Questionnaire battery

–    Data entry modules

–    Data entry and coding modules

What we did that helped in evaluation of pilot and new programmes

Planning

•             Sites to be involved in evaluation should be identified early (to allow training)

•             Adopt a clear and comprehensive protocol (WHO Manual)

•             Clearly identify aims of evaluation, eg:

•          To examine changes in quantity and frequency of opioid use

•          To examine changes HIV risk behaviour

•          To examine changes in health status and others…..

•             Ensure procedures are clear and available to all staff involved in study:

•          Procedure for using and coding questionnaires

•          Procedures for interviewing

•          Procedures for collecting biological samples

 

 

 

Planning cont

•      Staff

–    Appoint principal investigator early on in study (senior academic/clinician).

–    If multi-site study

•    appoint study manager to coordinate everyday activities

•    appoint data entry operator

•    appoint data analyst

–    At each site identify evaluation team and clearly define roles

•   Study coordinator

•   Interviewers

 

Training

•      To facilitate training the principal investigator/ or delegate should visit each site before the start of the study to:

–    carry out training with the interview staff., including;

•    how to use and code questionnaires

•    training in effective interview technique.

–    witness staff interviewing clients

 

•      Principal investigator visited sites more than once to solve problems and/or to provide more training

 

 

Project Management

•       The study manager communicated with study sites on a regular basis in order to;

–    provide regular and frequent updates,

–    to rapidly convey changes to protocol,

–    to remind sites of key dates, and,

–    to request key outputs.

•       It is important to remain flexible., eg.

–    Minor protocol modifications may be required to suit local conditions

–    study sites may commence evaluation at different stages.

•       Study manager must be easily accessible to study coordinators from other sites (e-mail) , to encourage sites to communicate problems early.

–    For example, we encountered many technical problems with data entry program.  Effective communication permitted effective troubleshooting.

 

Data Collection Strategy/ quality assurance

•      Provide sites with clear timeline for data collection [frequently remind sites]

•      We required each study site to undertake a pilot study of the protocol.

–    This involved 5 participants at each site

–    Data collected was checked by coordinating site for data irregularities that might reflect problems with administering/coding questionnaires

–    Report on pilot study results by site also provided

–    If required further training was supplied.

–    Data discarded

Thankyou