Evaluation designs for adequacy, plausibility and probabilit

Both complex and simple evaluations, however, should be equally rigorous, whether they assess the adequacy... plausibility... or probability..." (p.17)

Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact

JP Habicht, CG Victora and JP Vaughan

Charlotte Allen

Monday 12th May

Once you understand the "Why" you can move to the "How"...

Classification Axes

“Define the target audience for the evaluation results, since the responsibilities and expertise of the decision makers will affect what questions will be asked” (p.11)

What do you want to measure?

How sure do you want to be?

What do you want to measure?

An example table...

Provision

- The services must be provided

Utilization

- People must be using them

Coverage

- The interface between the delivery and population

Impact

- The achieved outcomes

"Any important shortcomings in the early stages of this chain will result in failures to the later achievements" (p.11)

Adequacy

- Did the expected changes occur?

Plausibility

- Did the program seem to have an effect above and beyond other external influences?

Probability

- Did the program have an effect?

"how confident decision makers need to be that any observed effects are in fact due to the project or programme" (p.11)

Dealing with confounding (external) influences requires the measurement of probable confounding factors

Probability effect: did the program have an effect? (P < x%)?

Plausibility assessment: Did the programme seem to have an effect above and beyond external influences?

Adequacy: Did the expected changes occur?

Adequacy assessment does not require a control group, as it's measured against set criteria i.e absolute value, or absolute incremental value in the case of a longitudinal study.

Under special circumstances a lack of change may be a positive as it would represent an effective safety net.

Adequacy assessments are unable to link changes causally to the programme.

"Adequacy evaluations are limited to describing weather or not the expected changes have taken place" (p.12)

Probability evaluations aim at ensuring that there is only a small known probability that the difference between programme and control areas were due to confounding, bias or to chance.

They require

Random control groups

the evaluator must be present at the very early stages of the program to ensure authentic randomization.

They are the most expensive and intensive of all effect evaluations

You don't need to know the confounding factors when conducting Probability assessments, they provide you with the effect size regardless.

Plausibility goes beyond an adequacy assessment, they attempt to rule out confounding factors which might have caused the effect.

Requires a control group, either:

Historical control group: the same target population, a change before and after the program

Internal control group: individuals or communities that should have received the full intervention but have not received full intervention. Can employ a dose-response relation between intensity of the program and level of impact. (This allows for a stronger inference of plausibility than all or nothing groups)

External control group: Individual communities without the programme.

"To reach the highest level of plausibility, one must formally discard all other likely explanations." (p.14)

"Plausibility is often markedly improved if they are used in combination." (p.13)

Choosing the Evaluation Design

Features of the evaluations

A cross section table...

"The objective of an evaluation is to influence decision makers, how complex and precise the evaluation must be depends on who the decision maker is and on what types of decisions will be taken as a consequence of the findings" (p.17)

Factors when making your selection

What to think about

Sector of your program

The level of certainty required must be specified, this will inform whether you must select adequacy, plausibility or probability evaluating

Timing and timliness

For all types of evaluation programs it is best practice for the evaluator to be present at the planning stages of the program.

• In order to conduct probability testing the evaluator must be present in the planning stages, however Adequacy and Probability can be undertaken after the program is underway.
• How long you have to conduct your evaluation is also a factor

Cost

Cost is often a major factor in selecting your type of evaluation. The following issues should always be discussed

• Is a full evaluation required?
• Is there a need for collecting new data?
• Does the design include an intervention-control or a before-and-after comparison?
• How rare is the event to be measured and how small is the difference to be detected?
• How complex will the data analysis be?

Questions on the Text

In what context would Adequacy, Probability or Plausibility evaluations be required?

How would you go about conducting an Adequacy assessment of Impact?