Assessing rural transformations: piloting a qualitative ...
Attributing development impact: the QuIP quest for better impact evaluation James Copestake [email protected] IDS 20 March 2018 1. Motivation: starting premises Public investors should be able to demonstrate social impact credibly and cost-effectively. Intended beneficiaries should have a strong and credible voice in providing such feedback. Evidence should be timely enough to influence future actions, building on what is known already. It should reflect the complexity and diversity of context, interventions and outcomes. It could be based on better methodological standards. Filling an evidence gap? Donors and investors Other knowledge communities Contractual relationship Development Management Hierarchical relationship Implementing staff Service
relationship Commissioned evaluator Performance assessment (short feedback loop) Independent researcher Impact evaluation (intermediate feedback loop) Applied research (long feedback loop) Intended beneficiaries PROJECT THEORY MID-RANGE THEORY GENERAL THEORY Intermediate methodological options Rely on
But 1. Operational data and management feedback Mostly change data not impact Cognitive biases and vested interests 2. Participatory approaches 3. Quantitative counter-factual approaches 4. Qualitative social research methods Limited generalisability; Limited credibility to non-participants Costly and indivisible Narrowly framed (spurious precision) Akerlofs lemon problem The QuIP - story so far 2012-2015 Design and piloting Assessing Rural Transformations - ESRC/DFID project: goal to design and test a new approach (QuIP). Joint QuIP design with Farm Africa, Self Help Africa and Universities in Malawi, Ethiopia and UK. 8 pilot studies (2 countries x 2 projects x 2 years) 2016-2017 Proof of concept
Set up BSDR Ltd as a social enterprise. Twenty further QuIP across eleven countries. BSDR QuIP studies 2016-17 Activities Countries Burkina Faso Mexico Child nutrition Ethiopia Tanzania Uganda Climate change adaptation Ghana Kenya UK Community mobilisation India Zambia Malawi Drought response Factory working conditions Commissioned by Housing improvement Acumen Habitat for Humanity Bristol City Council Medical training C&A Foundation Oxfam Microfinance Concern Worldwide Save the Children Diageo Ltd Seed Global Health Rural livelihoods Gorta Self Help
Tearfund Africa TreeAid Value chain improvement www.bathsdr.org QuIP design features 1. A standard to be jointly adapted with the commissioner for each new assignment. 2. Sample from operational, monitoring data. 3. Benchmark of 24 semi-structured narrative interviews + 4 focus groups collected in a week. 4. Conducted by independent field researchers without knowledge of the project (blindfolding). 5. Data entry from field notes using pre-formatted Excel sheets to facilitate coding and analysis. continued 6. Deductive coding of attribution claims as explicit, implicit or incidental (confirmatory). 7. Additional inductive thematic coding of drivers, outcomes and drivers of change (exploratory) 8. Semi-automated generation of tables and charts 9. Use of summary report as the starting point for joint sensemaking 10. Transparency: drill down from summary evidence to raw data for QA and deeper learning Example of thematic coding Full Answer There have been negative changes in the food we eat during the last six-months. This is related to both increase in price of food items and loss of income sources. Contrary to this, the
clean water we drink has improved during this period as *** drilled a bore hole in our village which made it possible to survive here. Broken Answer There have been negative changes in the food we eat during the last six-months. This is related to both increase in price of food items and loss of income sources. There have been negative changes in the food we eat during the last six-months. This is related to both increase in price of food items and loss of income sources. the clean water we drink has improved during this period as *** drilled a bore hole in our village which made it possible to survive here. N1 N3 P8 Loss of income source Increase in food prices rehabilitated water sources https://www.youtube.com/ watch? v=3VPzj1KaCXs&feature=em- Driver of Primary Secondary Attribution Change Outcomes Outcomes (1-9) (P/N) (OP/ON) (OP/ON) N1
ON1 N3 ON3 P8 OP2 ON3 6 6 OP3 1 ON1 No or decreased income ON3 Decreased food OP2 Clean drinking water OP3 Improved living conditions How can a QuIP be used? As a stand-alone exploratory approach for needs assessment and programme design As a stand-alone confirmatory evaluation approach (better alongside quantitative monitoring data, but doesnt require a baseline or control group). Mixed method combinations 1. Before other studies: scoping 2. In parallel: triangulation 3. Follow-up: drill down into specific issues 10
Illustrative case study Irish Aid Local Area Development Programme in Luwingu and Mbala districts. Implemented by Self Help Africa (2013-17). Goals: improved livelihood, health, food and nutrition security for 16,000 households. Evaluation combined one QuIP in each district with livelihood zoning, a baseline survey, followup nutrition surveys and a farm livelihood survey. Example of closed question responses Income from farming Home food production Quantity of food consumed Variety of food consumed Children's diet Gender Age Family health CH 13 F
39 + = = = = + CH 6 F 40 = = - = - - CH 7 F 60
= = - = - - CH1 F 46 - = = = = = CH10 F 30 =
= - - - = CH11 F 41 + = - = + + CH14 M 67 = -
- = = = CH15 M 21 - - = - = = CH3 M 53 + - +
+ = = CH4 M 40 + + + + = + CH8 F 49 - = = -
= = CH9 F 40 + = = + = = Example of positive attribution table Domains SHA explicit SHA implicit Other factors C. Health NS6 NS16 NS18 NS 05 NS10 IY2 IY6 IY 21 IY 13 IY 22 IY 11 NS7 CH 7 CH10 CH 13 PA21 PA11 PA1 FGNS1 FGIY1 FGPA1 FGNS2
PA2 P12 FGNS1 FGIY1 FGNS2 FGIY2 FGCH2 NS16 NS12 IY2 IY 21 IY22 NS7 NS4 NS11 IY3 IY26 PA 3 PA5 CH9 PA18 PA2 IY3 CH4 CH9 FGNS1 NS6 NS16 NS10 IY2 IY6 IY 21 IY 13 IY 22 IY 11 NS7 IY9 IY26 IY19 IY15 CH 7 CH11 CH8 CH10 CH 13 PA 3 PA11 PA16 PA20 PA2 FGCH1 FGNS2 FGCH2 NS12 NS10 NS7 NS11 PA18 FGNS2 FGIY2 IY15 CH 13 CH4 PA18 F. Who eats what and when? Example of graphical illustrations Example of a positive drivers-outcomes table Providing information on better farming methods More land cultivated Increase in attendance at community meetings/training In c re a s e in a g ric u ltu r a l in p u ts
Joined cooperative/s In c er a s e in liv e s to c k n u m b e rs Training on post-harvest processing and handling Im p ro v e d h y g ie n e Joined savings group In c re a s e d cro p v a rie ty Provision of livestock/poultry (SHA) Im p ro v e d IY C F p ra c tic e s WASH information In c r e a s e d y ie ld Government (social cash transfer) Im p ro v e d c o m m u n ity c o h e s io n Access to fertiliser/seeds/chemicals (SHA provided or purchased from SHA-related income) Provision of information on nutrition In c r e a s e d h h
in c o m e / s a v in g s DRIVERS OUTCOME 5 (5) - 18 (18) 2 (2) 27 (27) - 1 (1) 4 (4) - - - 26 (23) - - -
- 3 (3) - 8 (6) 1 (1) 1 (1) - - - - 11 (10) - - - - - - Example of dashboard based causal chain analysis
with drill down Comparison with other appraoches Group 1. Approaches with specific features that the QuIP also incorporates Appreciative Enquiry; Case Studies; Causal Link Monitoring; Collaborative Outcome Reporting; Critical Systems Heuristics; Goal Free Evaluation; Outcome Mapping; Positive Deviance; Success Case Method; Utilisation Focused Evaluation. Group 2. The QuIP is a Beneficiary Assessment; Contribution Analysis; Developmental more narrowly Evaluation; Innovation History; Institutional Histories; Outcome defined version of a Harvesting; Process Tracing; Realist Evaluation. broader approach. Group 3. More quantitative approaches Cost Benefit Analysis; Difference-in-Difference Evaluation; Qualitative Comparative Analysis; Randomized Control Trials; Social Return on Investment. Group 4. Approaches with stronger participatory and formative goals. Democratic Evaluation; Empowerment Evaluation; Horizontal Evaluation; Most Significant Change; Participatory Assessment of Development; Participatory Impact Assessment for Learning and Accountability; Participatory Evaluation and Participatory Rural Appraisal.
See also www.betterevaluation.org Caveat: what QuIP does not do Does not provide Responses Estimates of the magnitude of average treatment effects Use as one input into microsimulation Run alongside a quantitative impact evaluation. Statistically representative frequency counts Reveals scope and range of responses Combine with Bayesian updating Use to design or follow-up on quantitative surveys. Objective facts Triangulate Perceptions matter! Recommendations for action Combine QuIP with process evaluation and follow-up stakeholder engagement. 18
Ongoing agenda To conduct more studies in different contexts. To promote further research into qualitative impact evaluation approaches and standards. To build stronger communities of good enough evaluative practice To research the political economy of evaluative practice. [CEDIL = DFID Centre for Development Impact and Learning] BSDR theory of change More and better More inclusive and sustainable development More and better evidence of social impact understand ing about producing good social impact evidence Wider support for producing good social impact evidence QuIP related research outputs by UoB and/or BSDR More and better QuIP
studies conducted by BSDR and by other organisations (demand) Increased capacity to produce good impact evidence (supply) Capacity building work (including training, networking and dissemination) to build a QuIP community of practice (dashed lines suggest feedback loops) UoB support for BSDR and QuIP related research Internal capacity of BSDR (staff, networks, identity, systems, Other external support for BSDR finance etc.) and/or promotion of QuIP Issues for discussion
Blindfolding Why do it? To enhance credibility of evidence by reducing the risk of pro-project and confirmation bias (of intended beneficiaries and field investigators). To give equal weight to all possible drivers of change. Can be combined with unblindfolded data collection: e.g. through joint follow-up interpretation of findings. Blindfolding is not always possible (e.g. in very low trust contexts). The QuIP does not require it. For confirmatory research the analyst cannot be blindfolded and this may also be a source of bias. However, their work can be audited. 22 The ethics of blindfolding The positives of blindfolding (more credible evidence) should outweigh the negatives (the greater good argument). Negatives include restricting field researchers ability to probe project impact in more detail. The rationale for blindfolding should be explained prior to participants in advance otherwise it is not informed consent. Blindfolding can be partial e.g. by revealing the identity of the commissioner, but not detailed project activities. Blindfolding can be temporary e.g. by holding unblindfolded joint follow-up meetings or sensemaking workshops. 23 Rethinking the qual/quant distinction 1. Two epistemological traditions (facts/ numbers vs words/meaning). 2. Different methods: that can be nested but remain distinct. 3. Quantification as the process of codification that can be utilised within all research traditions 24
Research as (de)codificatiion Evaluation process based on artful simplification of reality (time, space, ontology) Framing + Codifying Design, data collection, analysis, and reporting through time Meshing recall with intervention periods Decodifying + reframing Meshing feedback with follow-up actions Complex reality Positionality of the analyst ? Intended beneficiaries Commissioner of study Study findings Selected respondents
Project Theory of Change Photos and field report Field researchers Project staff QuIP Analyst Transcripts QuIP guidelines Lead evaluator Bath SDR Project context Other Others Inner ring: data and documents Outer ring: people Arrows indicate selected flows of information and influence Decision to fund Timeliness
Agree informally Select lead evaluator Identify field team Ethical approval Sign contract Supply sample frame data and approve data collection instruments Clarify theory of change TIMELY ACTION Sign contract Agree informally Sign contract Sign contract Finalise data collection design and line up trained analyst Data analysis and quality checks Organise field access, pilot data instruments and train staff Collect and transfer data Written and verbal feedback
Client Lead Field research organisation(s) evaluator team Definitions Attribution How do the actions of development agencies (X) contribute to improving peoples wellbeing (Y) in diverse, complex and risky contexts (Z)? Strategies Statistical inference based on variable exposure to X controlling for Z: Confirmatory - starting with X Rely on intended beneficiaries own self-reported account of causal mechanisms linking X to Y and to Z: Exploratory start with Y and work back to X and Z Credibility Good enough argument and evidence to enable A to convince B that X caused Y, subject to Z, on the basis of reasonable assumptions. Sufficiently transparent to be open to peer review or audit. Credibility threshold dependent on the prior knowledge and triangulation opportunities of B Credibility is closer in meaning to reasonable than rational A credible causal claim (that X caused Y) Evidence that X and Y happened;
Independent assertions by several stakeholders that X was an INUS cause of Y (with limited prompting); The absence of any more credible counter-explanation for why they might have said this; Their account of how X caused Y is consistent with a plausible theory. INUS - a possibly Insufficient but Necessary part of a causal package that is an Unnecessary but Sufficient cause of Y. A complex context A setting in which The influence of X on Y is confounded by factors Z that are: impossible to fully identify hard to measure accurately interactive and cumulative in their influence on Y impossible fully to control
DEAD METAPHOR - An overused metaphor that has gained a new meaning. IMPLIED METAPHOR - A more subtle use of a metaphor. MIXED METAPHOR - Two or more metaphors used together that appear illogical. EXTENDED METAPHOR - Metaphor that continues...
Science Review. 4th Grade. What do we call each stage of the water cycle? Water travels in a never-ending cycle of precipitation, accumulation, evaporation, and condensation. Answer. The steam coming from this cup is __. water vapor smoke. Precipitation.
ENWORKS Partnership The Partnership Board includes representatives from: Economic Solutions Drew Thomas Northwest Regional Development Agency Mark Atherton Environment Agency Toby Wilison Greater Manchester Chamber of Commerce Clive Memmott North West Chambers of Commerce Mike Damms North West Universities Association...
"Lady Lilith" Dante Gabriel ... (The witch he loved before the gift of Eve,) That, ere the snake's, her sweet tongue could deceive, And her enchanted hair was the first gold. And still she sits, young while the earth is...
January 15, 2002 18th AMS International Conference on IIPS Using a LDAP Directory Server for Environmental Data Discovery Donald Denbo NOAA-PMEL/UW-JISAO Presented by Eugene Burger NOAA-PMEL/UW-JISAO Objective Develop a data discovery service that enables users to find data sets available...
what to tackle . highly dependant on . local context. Priorities shaped by resources and opportunities. ... (pro tip! Google "taking stock and making hay") For starters you could read our paper which is only 28 pages. (Go ahead, we'll...
b. 25% of the children of two carriers will get cystic fibrosis. c. 50% of the children of two carriers will get cystic fibrosis. d. There is a 25% chance that a child will have cystic fibrosis if both parents...
Ready to download the document? Go ahead and hit continue!