Practising DE

Practising Developmental Evaluation

Conventional Evaluation Process

• Show respect to the work of organisation and its people, as well as values they uphold.

• Show interest in what the organisation does. Spend time with the staff.

• Make the most of "off" time.

• Throw away your pre-conceived ideas about their strengths and weaknesses of the organisation.

• Show your true colors.

• Do not present yourself as someone who came to judge.

Getting to know the program

Relationship-building

What can be accomplished by an effective DE

Without effective relationship-building, no evaluation can succeed!

(c) CSO Network Japan

Created by Naoki Chiba, Minako Nakatani, Katsuji Imata and all the DE training participants in 2017, with the help of Michael Quinn Patton and Kate McKegg.

Your attitude matters

Listening skills

Program improvements and innovaton

Benefit to you

Building an effective relationship

Gaining a comprehensive understanding of the program

It is the organisation, not you, who ends up owning the evaluation process and product.

The organisation trusts you so that it won't be hard for you to access a variety of organisational information.

①Triangulation but more

② Your repertoire matters

The "three questions" are useful in this process, too

You have skills to let people open up - use them!

Show pictures, play music, play chess, sing and dance! - so that people want you to succeed.

★ Pay attention to context - not only "what is being said." (text)

★ Interview different people on the same question.

★ Fit for purpose (are you creating an environment in which the interviewee is comfortable to speak?)

★ Be empathetic.

★ Right question at the right moment.

Sum is greater than its parts

Making the world a better place through social innovation

③ Seize the moment

④ Understand what you're faced with as a system

There are forks on the road, but usually you do not know there was a fork until after you chose one path over the other. But if you pay attention, you can seize the moment and work with innovators to consciously decide which path to take.

Realise that the world is complex and use theoretical innovations to understand the world as such.

★ Be patient and wait for the moment to emerge. Grasp it.

★ Sometimes you just need to do nothing, sometimes you need to shake up deliberately.

System thinking

Complexity theory

Collective impact

Basics of Evaluation

Organisational learning and growth

Program Evaluation

What is Evaluation?

The systematic assessment of the operation and/or outcomes of a program or policy, compared to a set of explicit or implicit standards as a means of contributing to the improvement of the program or policy.

(Carol Weiss)

Evaluation is about determining merit and worth of things being evaluated

Role of DE Evaluators

But first, let's start with the basics.

Evaluation can take many shapes and forms depending what you want to do with it

And, at the end, we will reach a new horizon.

Facilitating social change

Internal or external evaluator - either will work

Support real-time, data-based decision-making by providing rapid feedback

Knowledge

generation

Satisfying accountability needs

Evaluation can be used for different purposes

And there are different types of program evaluation.

Improving programs and activities

Organisational learning, Decision-making support

Bring to the forefront the adaptation and innovation process

Become a part of the social innovation team

Building an effective relationship (long-term) is a key

What is the difference between evaluation and ...?？

What is Developmental Evaluation (DE)?

Evaluation and Monitoring

Evaluation and Social Science Research

Evaluation has ethics and standards you need to uphold

Monitoring checks progress against the plan.

Evaluation is a systemic review of the plan, process, outputs and outcomes.

Evaluation makes a value judgment.

Research works on causality and correlations among phenomena.

Developmental evaluation (DE) informs and supports innovative and adaptive development in complex dynamic environments. DE brings to innovation and adaptation the processes of asking evaluative questions, applying evaluation logic, and gathering and reporting evaluative data to support project, program, product, and/or organizational development with timely feedback.

(Michael Quinn Patton)

Three types of program evaluation

Evaluation and Management Support

Preskill & Beer (2012) "Evaluating Social Innovation," FSG & Center for Evaluation Innovation

What should DE evaluators do in response to complex situations?

Key is "accompanying" social innovators throughout the evaluation process

It is not a method but a way of doing an evaluation

Management support provides roadmaps, strategies, guidance, etc.

Evaluation (DE in particular) provides decision-making support (that can include roadmaps and strategies) based on facts.

Developmental Evaluation

Summative Evaluation

Formative Evaluation

Initiative is stabilizing and well-established

• Established
• Mature
• Predictable

Initiative is forming and under refinement

• Improving
• Enhancing
• Standardising

Initiative is innovating and in development

• Exploring
• Creating
• Emerging

And don't forget...

Stages of program development

Now you are ready to go into nuts and bolts - but let's recap first

Right brain

Left brain

Intuition, pattern recognition, holistic thinking

Language, logic, analysis

DE is particularly useful for:

• Art projects for which their value is hard to measure;
• Rapid change in external environment is accompanied such as disaster recovery;
• Work in a network or coalition such as policy advocacy;
• Evaluating organisational strategy;
• Collective impact work with multiple stakeholders.

Other typologies exist, too:

• Based on timing (mid-term, final, etc)
• Where the evaluator sits (self, internal, external)

Making good use of standard evaluation process

Use it effectively for:

• relationship-building
• sense-making
• process work

Vision of success

DE Readiness

Logic model

People have different skills and strengths - working in an evaluation team would be ideal

Not all organisations and projects are DE-ready. Here is a sample checklist to see if you are ready for a DE.

tea break

Three questions

Rich picture

DE evaluators need to use her/his left and right brains

Who is ready to do DE?

Situational Analysis

DE Toolbox

Accompaniement record chart

Values exercise

*Original tool created by CSO Network Japan

Different Situations

DE Basics

Before starting a DE, you need a good situational analysis.

Raising a child

Baking a cake

Paraire Huata (Te Ngaru Learning

Systems, 1995)

Simple

Sending a rocket to the moon

Utilisation-

Focus

Raising a child

• Raising one child gives you an experience but there is no guarantee your next attempt will succeed.
• Expert knowledge is helpful but it is not almighty.
• There can be no perfect manual.
• Situations matter.

"These are the keys!"

(Michael Quinn Patton)

Complicated

Complex

System mapping

A definition of

"Social Innovation"

A novel solution to a social problem that is more effective, efficient, or sustainable than existing solutions and for which the value created accrues primarily to society as a whole rather than private individuals.

(Kriss Deiglmeier)

Photo voice

Now let's go into nuts and bolts, but remember, these are conventional ingredients of evaluation steps and you might not follow them precisely in a DE.

Principles

Three Key Pillars of DE

They are not all in the menu. Make use of a variety of tools as appropriate.

Social Innovation Support

肥満に関するシステム・マップ

Government Office for Science, UK Foresight Tackling Obesities: Future Choices - Building the Obesity System Map October 2007より引用

An effectiveness principle is a statement that provides guidance about how to think or behave toward some desired result.

Complexities and System Thinking

When the situation is complex...

8 Principles of DE

Three Questions

Repeat asking Three Questions

Use principles as a guide for your inquiry in Three Questions by asking, "what is important for the organisation?"

What are the methods and tools for DE?

It is hard to predict what will happen in complex and emergent situations. Stay open and be adaptable by asking three questions

We worked with Michael to come up with these.

1. Developmental purpose

2. Evaluation rigor

3. Utilization focus

4. Innovation niche

5. Complexity perspective

6. Systems thinking

7. Co-creation

8. Timely feedback

By doing so, you can:

・Constantly on a lookout

to understand the situation

・Interpret the situation as

it unfolds

・React to the changing

situation

DE provides a real-time feedback to the social innovators so that they can learn, make decisions and innovate. Managing the three questions and making an evolving loop by on-going questioning is at the core of conducting a good DE.

Collect real-time data, understand and be guided by unfolding facts.

Kate McKegg

Vision of Success

Stay on track by keeping in mind what will happen if the program succeeds?

Forecast from where you are

Backcast from your vision

Decide on evaluation purpose

Setting evaluation questions and indicators

Collect data, analyse and summarise evaluation results

Write and present evaluation report

Provide evaluation feedback

What is it that you want to accomplish in the evaluation - The scope needs to be adequate (not too broad, not too narrow).

Decide on what data you need to gather and how.

Present your analysis in an easy-to-understand fashion

DE is about giving innovators "a-ha!" moments. To promote this process, you need to show tables and graphs, tell stories, and facilitate learning.

Very important to provide feedback constantly, but summarise them at the end. Feedback should not be one-way - innovation is co-creation!

Evaluation questions need to be crafted in partnership with the organisation and people you work with. Never impose your questions on them.

Reporting comes at last, but be mindful that many evaluation reports end up on a shelf, having never read fully. To avoid it:

① Make it available to a wide audience.

② The recommendation part should be created by engaging key stakeholders.

③ Make is reader-friendly.

What does the organisation and the primary users want to know through the evaluation work? They may not have the answer immediately. Work with them so that it comes to them.

Understand and trust that necessary action based on the evaluation will arise from the organisation at the right moment.

Copyright 2018　CSO Network Japan All Rights Reserved.

Again, this should not be used as a cookie cutter in a DE. Be adaptable!

After all, evaluation is for use.

(Michael Quinn Patton)