Prezi

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in the manual

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Make your likes visible on Facebook?

Connect your Facebook account to Prezi and let your likes appear on your timeline.
You can change this under Settings & Account at any time.

No, thanks

Program Evaluation: Theory and Methods

No description
by Ariel Nash on 30 January 2013

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of Program Evaluation: Theory and Methods

Bamberger et al “Consists of an explicit theory or model of how the program causes the intended or observed outcomes and an evaluation that is at least partly guided by this model”---(Rogers Petrosino 2000:5)
It “identifies program resources, program resources, program activities and immediate outcomes and ultimate program goals”---(Wholey 1987:78)
It is popular due to the recognition that a program’s success or failure can be assessed only with a clear understanding of the problem it was intended to address, the rationale for choosing a particular approach, and how the program was expected to operate.
Theory and Analysis Need to select or develop explicit criteria and standards to evaluate a program or product
Clear criteria and standards allow for an agreed upon the value of the program or product

What will be measured?
What indicates successful implementation and desirable outcomes? Using Program theory to develop measurable criteria Disagreement over purpose of program can occur
Due to implicit assumption about problem causes and best solutions
Difficult to agree on measurable criteria for program effectiveness, to conduct evaluation, or improve intervention without an explicit theory of development for an intervention or a conceptual framework that links intervention to project outcomes Creating a Program Theory People respond differently to programs
People need to envision carrying out the desirable behaviour before the program starts
Describing processes of “doable” behavioural changes can lead to program effectiveness Theory identifies intermediate results not just outcomes There are 3 important restrictions on the selection of criteria
Evaluation budget
Time available for the project - Political climate, services available and needs can change
Criteria that is credible to stakeholders
Limitations in selecting evaluation criteria Evaluators do not make theory plan for staff
Provide info for staff who then can accept and adapt research into their program
To know what to observe evaluators must then learn what is to happen to project completers, changes to be made, experiences to lead to those changes and what activities the program needs to have Talk with staff to identify how staff believe program activities will lead to desired intermediate and final outcomes
Go over research literature on same/similar programs
Similar programs may describe program theories ie. using peer leadership for youth programs
Basic research can show what is effective and what isn’t Making the theory: steps for evaluators Utilizing evaluation results to foster learning:

Instrumental: whether a program should continue, or which strategies to pursue

Conceptual: affecting how people think about an issue or program

Ritual / Symbolic: confirm already-held beliefs about the program by program managers or stakeholders Reflective Peacebuilding Monitoring-as-Learning Reflective Peacebuilding Bamberger et all Continued Logical Framework Analysis Program Impact Model Purpose of program theory evaluation is to help explain why excepted impacts were or were not achieve by assessing the following:
oThe strengths and weakness of logic of the program theory underlying the project or program design
oThe strengths and weaknesses of how the project or program was implemented
oHow contextual factors contributed to, or militated against, the achievement of intended impacts
oHow the project program affects, and is affected by different groups.
Formative Evaluation: Assess how any impact was achieved
Summative Evaluation: Also recommend whether, where and how the program could be scaled up or applied in other settings.
How to Construct Basic Logic Model (187)
oStep 1: Definition of the theory of change underlying the model
oStep 2: Articulation of the theory of change in a program theory model
oStep 3: Graphical representation of the program theory (logic Model)
oFurther Refinements
oStep 4: A result Chain
oStep 5: Contextual analysis
oStep 6: Process Analysis
oStep 7: Refinements to the initial logic model
Contextual Variables
Mediators
Program Design and Inputs
Implementation Strategy
Outputs
Outcomes and Impacts
Mediators
Step 1: Conducting a readiness assessment
Step 2: Agreeing on a set of outcomes of monitor and evaluate
Step 3: Selecting key indicators
Step 4: Generating baseline data on indicators
Step 5: Planning for improvements: selecting results targets
Step 6: Monitoring for results
Step 7: Using evaluation to support a Results- Based Management system: designing impact evaluation studies to assess attribution and causality
Step 8: Reporting findings
Step 9: Using Finding
Step 10: Sustaining the M&E system within the organization Using Program Theory to Help explain causality in Mixed Method Evaluations
oStep 1: Construct an impact model
oStep 2: Identify some of the alternative hypotheses/ explanations concerning the causes of change and the expected outcomes/impacts
oStep 3: Define operationally measurable input, process and outcomes indicators
oStep 4: Define the time period over which outcomes and impacts are expected to occur and the intensity of inputs required to achieve outcomes
oStep 5: Using PRA (Participatory Appraisal Methods) and other qualitative methods to identify causality
oStep 6: Define and combine all available evidence for inferring causality
“A local health care centre participated in a program to teach volunteer community residents how to use computers to find online health information. Some stakeholders postulated that this effort would result in fewer cases of low birth weight babies in the community. Is this a plausible result of such a program? What might be more plausible outcomes from such a program?” p. 66. Exam Question Create a
Learning
Document!!! Evaluation-as-Learning
See the full transcript