Create your own polished presentation in minutes. Just describe what you need.
Free to try, no credit card requiredPractising Developmental Evaluation
Conventional Evaluation Process
Getting to know the program
Relationship-building
What can be accomplished by an effective DE
Without effective relationship-building, no evaluation can succeed!
(c) CSO Network Japan
Created by Naoki Chiba, Minako Nakatani, Katsuji Imata and all the DE training participants in 2017, with the help of Michael Quinn Patton and Kate McKegg.
Your attitude matters
Listening skills
Program improvements and innovaton
Benefit to you
Building an effective relationship
Gaining a comprehensive understanding of the program
It is the organisation, not you, who ends up owning the evaluation process and product.
The organisation trusts you so that it won't be hard for you to access a variety of organisational information.
①Triangulation but more
② Your repertoire matters
The "three questions" are useful in this process, too
You have skills to let people open up - use them!
Show pictures, play music, play chess, sing and dance! - so that people want you to succeed.
★ Pay attention to context - not only "what is being said." (text)
★ Interview different people on the same question.
★ Fit for purpose (are you creating an environment in which the interviewee is comfortable to speak?)
★ Be empathetic.
★ Right question at the right moment.
Sum is greater than its parts
Making the world a better place through social innovation
③ Seize the moment
④ Understand what you're faced with as a system
There are forks on the road, but usually you do not know there was a fork until after you chose one path over the other. But if you pay attention, you can seize the moment and work with innovators to consciously decide which path to take.
Realise that the world is complex and use theoretical innovations to understand the world as such.
★ Be patient and wait for the moment to emerge. Grasp it.
★ Sometimes you just need to do nothing, sometimes you need to shake up deliberately.
System thinking
Complexity theory
Collective impact
Basics of Evaluation
Organisational learning and growth
Program Evaluation
What is Evaluation?
The systematic assessment of the operation and/or outcomes of a program or policy, compared to a set of explicit or implicit standards as a means of contributing to the improvement of the program or policy.
(Carol Weiss)
Evaluation is about determining merit and worth of things being evaluated
Role of DE Evaluators
But first, let's start with the basics.
Evaluation can take many shapes and forms depending what you want to do with it
And, at the end, we will reach a new horizon.
Facilitating social change
Internal or external evaluator - either will work
Support real-time, data-based decision-making by providing rapid feedback
Knowledge
generation
Satisfying accountability needs
Evaluation can be used for different purposes
And there are different types of program evaluation.
Improving programs and activities
Organisational learning, Decision-making support
Bring to the forefront the adaptation and innovation process
Become a part of the social innovation team
Building an effective relationship (long-term) is a key
What is the difference between evaluation and ...?？
What is Developmental Evaluation (DE)?
Evaluation and Monitoring
Evaluation and Social Science Research
Evaluation has ethics and standards you need to uphold
Monitoring checks progress against the plan.
Evaluation is a systemic review of the plan, process, outputs and outcomes.
Evaluation makes a value judgment.
Research works on causality and correlations among phenomena.
Developmental evaluation (DE) informs and supports innovative and adaptive development in complex dynamic environments. DE brings to innovation and adaptation the processes of asking evaluative questions, applying evaluation logic, and gathering and reporting evaluative data to support project, program, product, and/or organizational development with timely feedback.
(Michael Quinn Patton)
Three types of program evaluation
Evaluation and Management Support
Preskill & Beer (2012) "Evaluating Social Innovation," FSG & Center for Evaluation Innovation
What should DE evaluators do in response to complex situations?
Key is "accompanying" social innovators throughout the evaluation process
It is not a method but a way of doing an evaluation
Management support provides roadmaps, strategies, guidance, etc.
Evaluation (DE in particular) provides decision-making support (that can include roadmaps and strategies) based on facts.
Developmental Evaluation
Summative Evaluation
Formative Evaluation
Initiative is stabilizing and well-established
Initiative is forming and under refinement
Initiative is innovating and in development
And don't forget...
Stages of program development
Now you are ready to go into nuts and bolts - but let's recap first
Right brain
Left brain
Intuition, pattern recognition, holistic thinking
Language, logic, analysis
DE is particularly useful for:
Other typologies exist, too:
Making good use of standard evaluation process
Use it effectively for:
Vision of success
DE Readiness
Logic model
People have different skills and strengths - working in an evaluation team would be ideal
Not all organisations and projects are DE-ready. Here is a sample checklist to see if you are ready for a DE.
tea break
Three questions
Rich picture
DE evaluators need to use her/his left and right brains
Who is ready to do DE?
Situational Analysis
DE Toolbox
Accompaniement record chart
Values exercise
*Original tool created by CSO Network Japan
Different Situations
DE Basics
Before starting a DE, you need a good situational analysis.
Raising a child
Baking a cake
Paraire Huata (Te Ngaru Learning
Systems, 1995)
Simple
Sending a rocket to the moon
Utilisation-
Focus
Raising a child
"These are the keys!"
(Michael Quinn Patton)
Complicated
Complex
System mapping
A definition of
"Social Innovation"
A novel solution to a social problem that is more effective, efficient, or sustainable than existing solutions and for which the value created accrues primarily to society as a whole rather than private individuals.
(Kriss Deiglmeier)
Photo voice
Now let's go into nuts and bolts, but remember, these are conventional ingredients of evaluation steps and you might not follow them precisely in a DE.
Principles
Three Key Pillars of DE
They are not all in the menu. Make use of a variety of tools as appropriate.
Social Innovation Support
肥満に関するシステム・マップ
Government Office for Science, UK Foresight Tackling Obesities: Future Choices - Building the Obesity System Map October 2007より引用
An effectiveness principle is a statement that provides guidance about how to think or behave toward some desired result.
Complexities and System Thinking
When the situation is complex...
8 Principles of DE
Three Questions
Repeat asking Three Questions
Use principles as a guide for your inquiry in Three Questions by asking, "what is important for the organisation?"
What are the methods and tools for DE?
It is hard to predict what will happen in complex and emergent situations. Stay open and be adaptable by asking three questions
We worked with Michael to come up with these.
1. Developmental purpose
2. Evaluation rigor
3. Utilization focus
4. Innovation niche
5. Complexity perspective
6. Systems thinking
7. Co-creation
8. Timely feedback
By doing so, you can:
・Constantly on a lookout
to understand the situation
・Interpret the situation as
it unfolds
・React to the changing
situation
DE provides a real-time feedback to the social innovators so that they can learn, make decisions and innovate. Managing the three questions and making an evolving loop by on-going questioning is at the core of conducting a good DE.
Collect real-time data, understand and be guided by unfolding facts.
Kate McKegg
Vision of Success
Stay on track by keeping in mind what will happen if the program succeeds?
Forecast from where you are
Backcast from your vision
Decide on evaluation purpose
Setting evaluation questions and indicators
Collect data, analyse and summarise evaluation results
Write and present evaluation report
Provide evaluation feedback
What is it that you want to accomplish in the evaluation - The scope needs to be adequate (not too broad, not too narrow).
Decide on what data you need to gather and how.
Present your analysis in an easy-to-understand fashion
DE is about giving innovators "a-ha!" moments. To promote this process, you need to show tables and graphs, tell stories, and facilitate learning.
Very important to provide feedback constantly, but summarise them at the end. Feedback should not be one-way - innovation is co-creation!
Evaluation questions need to be crafted in partnership with the organisation and people you work with. Never impose your questions on them.
Reporting comes at last, but be mindful that many evaluation reports end up on a shelf, having never read fully. To avoid it:
① Make it available to a wide audience.
② The recommendation part should be created by engaging key stakeholders.
③ Make is reader-friendly.
What does the organisation and the primary users want to know through the evaluation work? They may not have the answer immediately. Work with them so that it comes to them.
Understand and trust that necessary action based on the evaluation will arise from the organisation at the right moment.
Copyright 2018 CSO Network Japan All Rights Reserved.
Again, this should not be used as a cookie cutter in a DE. Be adaptable!
After all, evaluation is for use.
(Michael Quinn Patton)
Listen, listen and listen!
Listen about the issues that might fall outside the scope of the evaluation at hand.
coffee break
Ready for a DE venture?
Create your own polished presentation in minutes. Just describe what you need.
Free to try, no credit card required