Poverty Stoplight:

Frames Poverty by Proposing 50 Indicators

Country-adapted Poverty Stoplight is validated by local clients

Validation process enough to ensure definitions are locally relevant?

We will find out in the focus groups.

• Idea of an "(in)correct" answer instead of true individual expression?
• Visibility bias: “perceptions of increasing poverty incidence may simply be due to its increasing visibility” (Saffer 2013:41)
• Recall and nostalgia bias.

“... locally meaningful dimensions of poverty were identified through dialogical processes and subsequently standardized and included in a household survey administered to a probabilistically sampled population. Such studies succeeded in bolstering claims of external validity but require assumptions of a relative population homogeneity to meet requirements of ‘basket’ and ‘levels’ consistency” (Shaffer 2013:48)

“locally generated dimensions of poverty are used... to identify meaningful dimensions of poverty and to facilitate comparisons of 'like with like' with a view to address 'basket' consistency. The main difference is such dimensions are 'mapped' onto existing household survey data” (Shaffer 2013:48).

"the main contribution...has been to address 'basket' consistency by basing interpersonal comparisons on correlates of poverty" (Shaffer 2013:49)

• "Rather than relying on correlates of poverty, the focus is on poverty rankings adjusted to take into account site-specific differences which could be driving results" (Shaffer 2013:51)

• Example:
• Step 1: respondents asked to locate themselves on 6 step ladder where 1 is poorest and 6 richest.
• Step 2: four hypothetical families are presented showing different problems (frequency of meat consumption, ability to heat home, secondary education etc.)
• Step 3: these hypothetical families are also ranked.
• Purpose: Hypothetical families give a background for the interpretation of self-assessment of poverty.

• 'basket' consistency is not addressed. Only perception on poverty.

• As it stands, dimensions are created in a top-down fashion.
• Focus groups and interviews will identify if top-down created indicators and level-definitions are locally relevant.
• Before starting, in my opinion, I believe that we could, in the future, look at ways of strengthening the manner in which bottom-up indicators are created or at least we shoud...

• ...weight the indicators according to locally generated valuations.

• Currently, our options of weighting indicators are:

• Flat indicators (all indicators are equally relevant)
• Flat dimensions (all dimensions are equally relevant, but indicators are of different relevance according to the amount of indicators inside each dimension)
• Statistically generated weights (PCA or CFA is used to generate the weights according to the correlations between indicators)

• The downside of these weighting methods is that they are, again, top-down. Top-down definitions + top-down weights are not ideal.

• When the results are consumed at the individual level, each indicator has its own "conceptual threshold" (i.e. if you are yellow or red in an indicator you are poor in that indicator)
• However, when creating an index, it will be necessary to know how many reds/yellows do you need in order to be defined as poor? Or what index number is the cutoff between poor and non-poor?
• Without this cutoff the Poverty Stoplight can't identify who is poor and who isn't. Although it might be able to show who is poorer than who (as people could be ordered on a continuum).
• For institutional purposes, knowing who is poor and who isn't is important.

Stoplight adaptation process

Top-down definitions:

Fundación Paraguaya

Realist perspective

Measurement/construct validity

• Are indicators locally relevant?

• Are level definitions locally relevant?

• Are interpretations of level definitions equal among all clients? (comparing like with like)

Realist perspective

Measurement/construct validity

Realist perspective

Local MFI parters

levels consistency/external validity

Adapt

Poverty Stoplight Paraguay

• "If the dimensions differ over the range of comparisons one is not comparing 'like with like' (Shaffer 2013:44)
• Necessary for generalizability

Definitions

Levels

(sometimes drop/add indicators but preferably not)

Poverty Stoplight Application

Asesora

"concepts such as poverty should bear a close relationship to local categories of social differentiation. Otherwise, 'we' are imposing analytical categories with little local relevance (Shaffer 2013:44)

Authority figure?

• (dialogical epistemology) “relies on a consensus theory of truth that rests on the premise that truth is the property of a statement which has been argumentatively, or discursively, validated” (Shaffer 2013:26)
• Necessary that arguments be held on an equal footing, and that one doesn't trump the other based on extortion or authority.

Clients/Local populations

• Clients are asked to "validate" definitions. But in the strictest sense, indicator and level definitions are not locally generated.
• Indicators are first framed by the Fundación, and later accepted/modified by poverty experts.
• Only after this dual-process of framing, adaptation and re-framing, are the indicators exposed to clients, who have to decide upon/judge the twice-framed definitions.
• Although definitions might have local relevance (we would find out in the focus groups), they are NOT locally generated.

Frame responses

• Understand the language? (locally relevant meanings)
• Do colors create biases?
• Order in which level definitions are exposed to clients (first red, then yellow and then green; not simultaneous).
• Fear of using the tablet?

Negotiated answers

Tablet

• Are all responses a reflection of dialogical approaches?
• Or are asesora interventions a form of consumption verification?
• How we choose to answer this is relevant because it makes us have to justify different forms of validity and reliability

Reported through

Client

Results

Where we stand (according to what I think)

Theoretical Interlude

• Up to this point the results were used to empower clients into taking action in eliminating poverty in their lives.
• However, in order to measure poverty, we will have to be able to turn these indicators into an index, and we have to think about problems of aggregation and generalizability.

The Poverty Stoplight:

Time for a New Metric for Microfinance?

Operationalizing dimensions

Practicality

3 Methods

Measurement

Statistical Adjustments and Vignette

Including Locally Meaningful Definitions

Poverty Correlate/Dimension Mapping

Interviewees rank themselves

Locally-grounded perceptions of poverty result from comparison of both

Interviewees rank hypothetical situations of poverty

• PRA
• FG
• INTERVIEWS

Locally meaningful poverty dimensions

Practicality is independent from the Poverty Stoplight's measurement capabilities, and has more to do with what carrying it out requires than what the tool does.

Locally meaningful poverty dimensions

Mapped onto existing household survey data

PRA/FG/INTERVIEWS

At the core of the reliability and validity discussion are the competing purposes of the Poverty Stoplight of producing: (1) dashboard results that are consumed by the individual in order to empower this individual, and (2) an index that produces a generalizable measurement about a population which can be used for institutions to identify poverty, and identify the characteristics of these poor people, incorporating locally relevant definitions of poverty (Q-squared methods).

Individual Consumption

Institutional Consumption

Questions of validity:

Q-squared: generalizability vs. incorporation of local meanings

Weighting Dimensions

• Are the dimensions locally relevant?
• Are the levels locally relevant?
• Do aspects of the visual survey induce biases?
• Does the relationship between asesora and client induce biases?

• Weights, aggregation, representativeness and generalizability are not relevant for this level of information consumption.

Operationalizing Dimensions of poverty

Establishing a poverty threshold

Weighing dimensions of poverty

Questions of reliability:

Locally-meaningful weights

Arbitrary/Normative weighting

Statistical Procedures

• Top-down
• Not locally generated
• Created according to the analysts' predilections

• Shaffer clearly prefers this.

• "'locally meaningful' weighting schemes based on people's perceptions of the relative importance of different dimensions of poverty" (Shaffer 2013:53)

• Weights based on correlation structure between dimensions

• Procedures usually used:
• Principal Components Analysis
• Factor Analysis

• Again these don't take into account the local relevance of dimension weights.

Indirect approach

Direct approach

(Shaffer 2013:53)

"In the direct approach to eliciting local weights, people are simply asked to rank timensions of poverty or well-being in tems of their relative importance" (Shaffer 2013:54)

• "'back-out' weights through econometric analysis" (Shaffer 2013:53)
• Attempt to integrate tatistical analysis of conditional relationships between variables with locally meaningul definitions of poverty as reflected in wealth-ranking results (Shaffer 2013:54)

(Shaffer 2013:53)

• Are interpretation of level definitions the same or different among different clients (different interpretations could lead to unreliability)
• Because at this level aggregation is non-existent, It's hard to see how reliability would be much of an issue.
• Reliability would mainly be transparency. Transparency of how clients are interviewed, the role of the asesora, and justifications for why we made decisions about the survey design (methodological trustworthiness; realist paradigm; Baumgarten 2010:4).

• Test-retest
• internal reliability (statistical tests)
• External reliability (potential for generalizability)
• 'basket' reliability
• 'levels' reliability
• Discriminatory power (PWR and PS application to non-poor people)

Setting Thresholds

Consumption Adequacy Question

Data Discontinuities

Conceptual Thresholds

• "respondents are asked if their level of consumption... is more than, less than or just adequate to meet family needs" (Shaffer 2013:57)

"analysis of locally generated data on poverty correlates or characteristics to determine if there are natural breaks which distinguish population groups" (Shaffer 2013:56)

• 'built-in' conceptual thresholds.
• definitions derived from PRA.

(Shaffer 2013:45)

Poverty Stoplight:

Reliability and Validity throughout the application process