Deterministic counterfactual - NCA Evaluation Conference, Prague Nov 22, 2017

Counterfactual in deterministic ontology : methodology developments for evaluation of ESF projects aiming at criminal recidivism prevention

Vladimír Kváča, Ph.D.

Evaluation Conference

Prague, Nov 22, 2017

Děkuji za pozornost

vladimir.kvaca@gmail.com

ESF - OP Employment, Social innovation

RESTART II

Polytechna Consulting, a.s.

Prisons in Jiřice, Kynšperk nad Ohří, Pardubice

Oct 2016 - Sep 2018

Change is possible -

I want it

A-GIGA s. r. o.

Prison in Vinařice

Aug 2016 - Jul 2018

Evaluation status: finetuning details of design,

ongoing collecting of data

+ Similar project by Rubikon Centrum starting in July 2018.

Data available

Selection process

Features of the interventions limiting the evaluation design

Desistence

Limited size of the treatment group (N = 144 / 140), severe attrition rate after release from prison (significant part of TG stops all contacts

with the project).

Modest comparison group (N ≈ 30) available (hopefully).

?

Age

Employment

Housing

Non-criminal networks

Positive relations

Non-criminal identity

Motivation, hope and self-confidence

Social skills

Problem solving skills

Addictions

Debts

...

ESI

Project specific data collection tool.

14 areas (like housing, addictions, employment, relations)...

Rated 0 - 10, self-assessment + assessment by the project team member.

ESI III

Additional question on 3 areas (how things were in last month - housing, employment, relations)

No chance to directly measure recidivism within

the 2 years long project.

Recidivism

SARPO

Tool for Assessment of Offenders' Criminogenic Risk and Needs

Routine data available for (most) prisoners.

Age, criminal history, education and maritial status.

Dynamic risk factors (housing, employment, finance, family and social contacts, education, addictions, personality) - score 0-10 in each category

Total dynamic risk, total static risk - score 0-100

PICTS-cz

Psychological Inventory of Criminal Thinking Styles

(Vinařice only)

Pre-test done independently on project,

post-test data to be done.

Evaluation questions

EO1: Does the project's theory of change work?

EO2: Which combination of factors at the level of individual members of target group allows (not) for a positive change?

Complex causality

QCA

Set-theoretic methods

Qualitative Comparative Analysis

Set theory serves well in unraveling complex pattern of causality as equifinality, conjunctural causation and asymmetry.

• Equifinality: alternative factors can produce the same results.
• Conjunctural causation: single conditions may not display an effect on their own, but only in combination.
• Asymmetry: explanation for the non-occurrence of the result cannot automatically be derived from the explanation for the occurrence of the result.

Membership scores of cases in sets

Set of European countries

QCA is a set-theoretic method.

Set-theoretic methods in general share three concepts, they:

• work with membership scores of cases in sets
• perceive relations between social phenomena as set relations
• interpret set relations in terms of sufficiency and necessity

Is Latvia an European country? YES!

Membership score for Latvia is 1.

Is Japan an European country? NO!

Membership score for Japan is 0.

What about Turkey?

Crisp set – dichotomies

(either in or out)

Fuzzy set – allows partial membership in a set (most social science concepts are not clearly dichotomous)

Is Turkey an European country? Partly!

Membership score for Turkey could be 0.03 (based on area).

Relations between social phenomena as set relations

Is a synthetic comparative method, as is share features of both

• case-oriented methods (keeps relations within the case) and
• variable-oriented methods (works with medium to large N and allows for good generalization).

“All EU members are democracies”

Set of democracies is a superset of the set of EU members.

Set of EU members is a subset of set of democracies.

Interpret set relations in terms of sufficiency and necessity

Being a democracy is a necessary, but not sufficient condition of being an EU member.

Four groups to understand counterfactual

Always takers

Compliers

Experiments (RCT)

Quasiexperiments

(PSM, RD, IV...)

counterfactuals

N = large

QCA (logic of elimination)

N = medium

MSSD/MDSD (Mill's method of Difference/Similarity), cross-case inference using logic of elimination

N = small

Congruence analysis (Bayesian updating on multiple theories)

N = 1

Defiers

Never takers

N/A

Fuzzy sets

Not in our case

Process tracing

N = 1

Bayesian updating on unbroken causal chain

Equifinality

Assymetry

By probabilistic methods we can estimate their numbers.

Conjuctural

causation

In fuzzy sets, the membership score is not dichotomous (0 or 1) but anywhere between 0 and 1. However, fuzziness is not based on lack of precision in empirical measurement. Fuzziness is due to not sharp conceptual boundaries.

By QCA we can investigate their profile.

≠

Fuzzy scales have three qualitative anchors:

- Complete presence in the set (1)

- Point of indifference (0.5)

- Complete absence in the set (0)

Verbal descriptions can be easily used for explanation of scores:

- 0.9 for “almost in”

- 0.55 for “little bit more in than out”

Identifying compliers

Case selection for Process tracing

Interaction of

QCA and PT