With the transition to market-based systems, many countries are designing and implementing social policies targeted to specific populations, e.g. social protection to poor people, job training programs to the youth and unemployed, agricultural development programs to farmers. Decision-makers, donors and taxpayers are interested in knowing whether the programs have the expected benefits. A rigorous assessment may lead to improvements in the design and implementation of the programs, fostering accountability.
The evaluation of the effectiveness of programs began with welfare and job training programs in the United States and is now increasingly applied in developing countries. Many interesting lessons emerged from this literature with respect to how to conduct a rigorous evaluation. Scientifically sound evaluations are needed because they are crucial for generating political support for the continuation or expansion of the programs. Moreover, evaluation findings may also inform policy makers outside national boundaries. Their value as a public good justifies funding evaluations with resources beyond those available domestically. There are several international institutions devoting resources to impact evaluation.
However, consensus about the need for serious impact evaluation at the policy-making level is not yet accepted across the board. There are many programs in need of evaluation. Further research would shed light on similar/new activities, but these programs either go unevaluated or they are evaluated using techniques which are far from best practices in impact evaluation.
However, consensus about the need for serious impact evaluation at the policy-making level is not yet accepted across the board. There are many programs in need of evaluation. Further research would shed light on similar/new activities, but these programs either go unevaluated or they are evaluated using techniques which are far from best practices in impact evaluation.
Activities that promote impact evaluation methodology (in terms of capacity building and bridging research and policy) in more disadvantaged Latin American and Caribbean countries will be important to closing the gap between state of the art evaluation techniques in developed countries and their application in Latin America and the Caribbean.
CEDLAS is committed to generating/supporting scientifically sound studies and capacity building activities thataim to produce evidence-based results for policy making.
Experimental Evaluation
Youth unemployment is a pervasive phenomenon in Latin America and the Caribbean. Governments have widely used training programs in order to mitigate the problem. This paper documents the effects of a training program designed for low income youths, which comprises of vocational training, life skills and work experience. Results show large gains in employment, with effects that remain stable for more than two years after the intervention. The program also shows substantial effects on access to credit. Program participants exhibit a higher probability of having requested formal consumer credit, and higher probability of having bank debts in good standing. The evidence suggest that our results are driven mainly by men and younger participants who have higher gains in terms of outcomes, contrary to previous evidence from Latin America.
Alzúa, M. l., Cruces, G. and Lopez, C. (2016), Long-run Effects of Youth Training Programs: Experimental Evidence from Argentina. Economic Inquiry. doi: 10.1111/ecin.12348
The objective of the intervention is to reduce the incidence of diseases related to poor sanitation and manage public risks posed by the failure to safely confine the excreta of some community members. The way to achieve this objective is by empowering communities motivated to take collective action. Local governments and other agencies perform at best a facilitating role. There is a growing recognition that this approach, referred to as Community-Led Total Sanitation (CLTS), may help with the reduction of open defecation practices. However, no rigorous impact evaluation of CLTS has been conducted so far. This study presents the results of the baseline analysis of a randomized controlled trial for studying the effect of CLTS in rural Mali.
An approach adopted in South Asia has drawn attention. This approach shifts away from the provision of subsidies for toilets for individual households and towards a promotion of behavioral change at the individual level. It emphasizes collective decision-making in order to produce ‘open defecation-free‘ villages. The objective of the intervention is to reduce the incidence of diseases related to poor sanitation and manage public risks posed by the failure to safely confine the excreta of some individuals. The way to achieve this objective is by empowering communities that are motivated to take collective action. Local governments and other agencies perform at best a facilitating role. There is a growing recognition that this approach, referred to as Community-Led Total Sanitation (CLTS), may help with the reduction of open defecation practices. However, no rigorous impact evaluation of CLTS has been conducted so far. This study presents the results of the baseline analysis of a randomized controlled trial for studying the effect of CLTS in rural Mali. It produces sound evidence that can be used to evaluate the extent to wich CLTS improves health outcomes and help determine what drives collective action, in order to increase sanitation coverage.
The direct recipients of the intervention are members of rural communities in Mali who aspire to live in a cleaner environment. The donor community includes international organizations and governments in developing countries. They will benefit from having simple and clear evidence on the effectiveness of an innovative program for improving sanitation in rural areas. They will learn whether the program has worked or failed to achieve its objective of eradicating open defecation, and the key factors explaining success and failure. This evaluation aims to provide useful information to help guide decisions on how to help meet the MDG sanitation target.
This research was recognized in the 2015 Best of UNICEF Research competition as Outstanding research.
Pickering, A. J., Djebbari, H., Lopez, C., Coulibaly, M., & Alzua, M. L. (2015) Effect of a community-led sanitation intervention on child diarrhoea and child growth in rural Mali: a cluster-randomised controlled trial. The Lancet Global Health, 3(11), e701-e711.
This paper exploits the unique institutional setting of U.S. campaign finance to provide new evidence on social incentives for political participation. We conducted a field experiment in which letters with individualized information about campaign contributions were sent to 91,998 contributors in the 2012 U.S. presidential election. The effect of those letters on recipients’ subsequent contributions is examined using administrative data. We find that exogenously making an individual’s contributions more visible to her neighbors significantly increased her subsequent contributions if the majority of her neighbors support her same party, but decreased her contribution if the majority of her neighbors support the opposite party. This constitutes evidence that individuals give preferential treatment to neighbors of the same party. In another treatment arm, we randomized the information observed by recipients about neighbors’ contribution behavior. Consistent with existing evidence on social norms, individuals contribute more when neighbors of the same party contribute higher average amounts. Furthermore, we find that the individuals also care about the total amounts raised by the same and opposite parties. These findings result in implications for fundraising strategies, the design of optimal disclosure policies and the understanding of geographic polarization.
Ricardo Perez Truglia and Guillermo Cruces. Partisan Interactions: Evidence from a Field Experiment in the United States. Journal of Political Economy. Volume 125, Number 4. August 2017Ricardo Perez Truglia and Guillermo Cruces (2014) Social Science Research Network
Individual perceptions of income distribution play a vital role in political economy and public finance models, yet there is little evidence regarding their origins or accuracy. This study examines how individuals form these perceptions and explores their potential impact on preferences for redistribution. A tailored household survey provides original evidence on systematic biases in individuals’ evaluations of their own relative position in the income distribution. The study discusses one of the mechanisms that may generate such biases, based on the extrapolation of information from endogenous reference groups. It presents some suggestive evidence that this mechanism has significant explanatory power. The impact of these biased perceptions on attitudes toward redistributive policies is studied by means of an experimental design that was incorporated into the survey, which provided consistent information on own-ranking within the income distribution to a randomly selected group of respondents. The evidence suggests that those individuals who overestimated their relative position and thought that they were, relatively richer than they were tend to demand higher levels of redistribution when informed of their true ranking.
Cruces, G., Pérez-Truglia, R and Tetaz, M. (2013) Biased Perceptions of Income Distribution and Preferences for Redistribution: Evidence from a Survey Experiment. Journal of Public Economics 98, pages 100-112.
Non-experimental Evaluation
In the context of the project “Fostering capacities in Impact Evaluation in Latin America,” the Center for Distributive, Labor and Social Studies, CEDLAS with the support of the International Development Research Centre (IDRC-Canada), organized three calls for thesis proposals that used Impact Evaluation techniques for PhD and Master’s students based in Latin America. Nineteen papers were selected.
In 1993 the Argentine Congress passed a law (Ley Federal de Educación, LFE henceforth) aimed at changing some important characteristics of the educational system. Chief among them, were an extension in the years of compulsory education and a change in the structure of educational curricula. While in the previous system a child was obliged to attend seven years of primary school, under the new legislation compulsory educational level was extended to nine years.
By increasing the obligatory number of years of education, the government sought to force mostly poor children to increase their human capital accumulation and induce some of them to continue studying in the secondary level and hopefully into college. Youths with more education are expected to perform better in the labor market, and hence have a lower probability of falling into poverty. While there is well-established evidence on the relationship of time spent at school and improvements in the labor market for developed countries, evidence for developing countries is much more scarce (Duflo, 2001, Galiani et al, 2007, Madeira, 2005, Rodriguez, 2005).
In this paper we evaluate the impact of the LFE on several educational and labor outcomes by exploiting the regional heterogeneity in the timing of the reform. Argentina is a federal country where primary and secondary public education are administered and financed at the provincial level. Although the LFE was a federal law to be complied with in all provinces, there was flexibility for state governments to decide on the timing of the reform. In some provinces the reform was quickly implemented after the LFE was passed; in others, the pace of the changes was slower. In fact, in some districts many central aspects of the reform were never implemented.
Taking advantage of this source of variation in the exposition to the “treatment”, we study the impact on different educational and labor market outcomes. In particular, we are interested in evaluating whether impoverished youngsters who were required to attend two additional years of schooling were more likely to finish high school, and whether they performed better in the labor market.
Alzúa, M. L., Gasparini, L. and F. Haimovich (2015), Educational reform and labor market outcomes: the case of Argentina’s Ley Federal de Educación. Journal of Applied Economics. Volume 18, 2015 – Issue 1
Other Initiatives
While the region has seen an increase in the number of impact evaluations aiming at measuring the causal impact of programs or policies, the technical capacities to conduct such studies in the region are still limited, and specially so in poorer countries. We will develop a series of training activities aimed at filling part of this gap, by providing short intensive training courses, supporting dissertations in specific methodologies and conducting professional development activities.
With the support of the International Development Research Centre (IDRC-Canada), CEDLAS aims to develop a capacity building initiative based on three different but closely related activities. The aim is to provide training activities in impact evaluation in the region on a regular basis.
The first actions is related to conduct ande promote a series of training courses on impact evaluation aimed at graduate students, junior university faculty, research assistants and junior researchers at think tanks, and mid-level technical government staff in Latin America.
The second action is to offer a set of scholarships ffor different levels and types of relevant research: i) Thesis development: PhD or Master level students using impact evaluation techniques can apply to obtain financial support for their research. ii) Technical staff in government agencies: these scholarships go beyond basic training and provide and involves internships for professional and technical staff working in government impact evaluation who may benefit from help from our researchers.
Third, CEDLAS will engage in activities to foster professional education. The Impact Evaluation Network will provide specific workshops with advanced material for researchers and academics interested in using impact evaluation in the region. The Impact Evaluation Network (IEN) at LACEA is an initiative that aims to advance the state of knowledge and expertise regarding the impact evaluation of different policies among scholars and policy makers in Latin America. Finally, we will develop a series of informal brown bag lunches held at Universidad Nacional de La Plata that are open to interested researchers both at La Plata and in other universities. At these lunches, new technical material will be informally discussed.
Brown Bag Lunches
- August 8th, 2014. “Cluster-Robust Inference” Discussion seminar about the paper: Cameron and Miller. A Practitioner’s Guide to Cluster Robust Inference. Journal of Human Resources. Spring 2015 vol. 50 no. 2 317-372. Walter Sosa Escudero (UDESA, UNLP, CONICET) led the discussion.
- December 5th, 2014. “New Development in Non-experimental Techniques for Impact Evaluation” discussed by Guillermo Cruces
- March 19th, 2015: “Gestión de la evaluación experimental del Programa Juventud y Empleo. Lecciones aprendidas” Brígida García (Ministerio de Trabajo – República Dominicana)
- May 15th, 2015: “Big data, mining and learning: perspectives for social data analysis” by Walter Sosa Escudero. Reading Material: Hal, Varian. Big Data, new tricks for econometrics, and, Wired. The End of Theory: The Data Deluge Makes the Scientific Method Obsolete
- November 9th, 2015: “Medium Term Educational Consequences of Alternative Conditional Cash Transfer Designs: Experimental Evidence from Colombia” Felipe Barrera-Osorio (Graduate School of Education – Harvard University)
Training
- October 20-24, 2014. Training on Impact Evaluation of Public Policies – Intermediate Level. La Plata, Argentina
- October 19-23, 2015. Training on Impact Evaluation of Public Policies – Intermediate Level. La Plata, Argentina
- November 24-26, 2015. Topics on Labor Markets and Impact Evaluation. Marco Manacorda (Queen Mary University of London). La Plata, Argentina
- March 10, 2016. “Constructing a Research Agenda using RCTs: Information and Educational Decisions”. Francisco Gallego (Pontificia Universidad Católica de Chile). 9th Meeting IEN. La Plata, Argentina
Grants for graduate thesis
We organized three calls for thesis proposals that used Impact Evaluation techniques for PhD and Master’s students based in Latin America. Nineteen papers were selected.
A new evaluation paradigm originated in academia is making inroads in policy circles in the region – there is an increasing understanding of the need to evaluate the impact of different government policies in terms of causality. However, until a few years back, most work in this field was conducted by researchers in developed countries, with academic research as its principal outcome. While Latin America has witnessed an increase in the number of impact evaluations conducted in the region and led by local researchers, many research centers still lag behind. This project develops several activities oriented to enhancing capacities for impact evaluation in the region in the context of a joint collaboration between the Centro de Estudios Distributivos Laborales y Sociales (CEDLAS), at Universidad de la Plata, Argentina and Grupo de Analisis para el Desarrollo (GRADE), Peru.
Basic training courses
These training courses aim to strengthen the capacities of Think Tank Initiative researchers,and other technical staff from poorer countries (researchers, government staff, etc.), and to help more experienced researchers train themselves and discuss issues at the frontier of impact evaluation research. The course covered basic and intermediate facts about experimental and non-experimental techniques. The training covered theoretical material in the morning and during the afternoon, existing evaluations were replicated, providing hands-on experience.
The course covered basic and intermediate facts about experimental and non-experimental techniques. This basic course was offered twice.
Intermediate/Advanced training course
This course aims to deepen students’ understandings of the theoretical problems involved in impact evaluation, including qualitative evaluations and mixed methods, while emphasizing problems associated with practical implementation. The course covers different evaluation techniques and theoretical lectures, that were supplemented with in-depth examination of case studies that are representative of each technique. Topics covered include: evaluation planning, research design (experimental, quasi-experimental, qualitative, and mixed methods), data collection, and identification strategies and methods (natural experiments, encouragement designs, difference-in-difference and instrumental variable techniques). As with the basic course, there was a hands-on component in which participants will have the opportunity to replicate evaluations with existing data sets. The course also used a workshop format in which participants present their current projects and get feedback from selected discussants as well as from other participants and experts. This course was also offered twice.
Mentoring
This mentoring activity represents a call for proposals for current Think Tank Initiatives grantees. Our Mentoring program looks to support TTI grantee institutions in the development of their impact evaluation studies. GRADE and CEDLAS designate an experienced researcher to provide technical support and guidance to institutions in demand of this type of services. Mentoring and support can cover projects at several stages: (i) impact evaluation studies commissioned to TTI institutions; (ii) research papers using impact evaluation techniques carried out by TTIs; (iii) design of evaluations commissioned by third parties (Governments, NGOs, etc.) to the TTIs; (iv) other non-specified technical help (questionnaire construction, sampling, etc.)
Five proposals were selected:
- Evaluación del Impacto de la Alianza Público-Privada en el Empleo e Ingreso de los Hogares – FOSDEH, Honduras
- Análisis de evaluabilidad de programas sociales nacionales de El Salvador – FUNDAUNGO, El Salvador
- Los efectos de la introducción de la Computadora en el Aula – Evidencias del “Programa Una Laptop por Niño” – CADEP e Instituto Desarrollo, Paraguay
- Diseño de evaluación de impacto del Proyecto “Atención educativa oportuna para el desarrollo integral a niños y niñas de 3 a 4 años” – Instituto Desarrollo, Paraguay
- Evaluación del Impacto de los riesgos de infestación de Dengue para la articulación con los gobiernos locales – Instituto Desarrollo, Paraguay
Seminars
Impact evaluation and policy influence workshop. This workshop was co-organized by CEDLAS, GRADE and CIPPEC and has received support from IDRC-Canada and 3ie. It took place November 19 2014, in Sao Paulo, Brazil. It brought together a group of researchers with expertise in design and development of impact evaluations in Latin America and provide a forum for researchers to share experiences and extract lessons about what works, discuss impact assessments’ influence in public policy and collaboratively develop strategies to maximize the impact of the results and findings.
On April 2, 2013 the Ciudad de La Plata, capital of the Argentinean Province of Buenos Aires, suffered intense rainfall and flooding that caused one in every three households to be affected. In a joint effort among Universidad Nacional de La Plata (UNLP), UNLP’s Facultad de Ciencias Económicas (FCE) and the Center for Distributive, Labor and Social Studies (CEDLAS), a representative survey of the city was conducted to measure different variables affected by the heavy precipitations.
The survey was carried out April 27-28 and May 4-5. The interviewers and the coordinators were advanced students from the FCE and researchers from CEDLAS, respectively.
The survey included a socioeconomic section, that intended to characterize the interviewed households, and an economic losses section, which measured the direct impact of the floods on tangible assets in surveyed households.
The report also included a survey of the health of children under 6, (considered the most vulnerable population), in order, to determine how this group was affected. In this specific case, the intention of the survey was to measure the impact of the floods on rapid transmission diseases as a consequence of the sanitary deficiencies. The illnesses considered in the survey were diarrhea and respiratory diseases.
Additionally, a group of questions were included to measure the psychosocial impact of this catastrophe. The test measured the frequency and intensity of assessed symptoms and is known as Davidson Trauma Scale. This test was previously used in other disasters, including the earthquake/tsunami registered in Chile in 2010. The scale consists of 17 items and measures the frequency and severity in a scale of 0 to 4 (ascendant order), allowing a score between 0 and 136. Some authors propose a cut-off point of 40 to determine if post-traumatic stress is present among respondents.
It is worthwhile to mention that even though the final results will not provide information on causation, they will evidence the importance of conducting surveys after catastrophes as a method of measuring their scope in multiple dimensions.
The course covered basic and intermediate facts about experimental and non-experimental techniques. This basic course was offered twice.
Press
La Nación – 06/02/2013
El Día – 06/02/2013
Página/12 – 06/03/2013
El Economista – 04/15/2013
Ámbito Financiero – 06/02/2013
InfoRegión – 06/03/2013
Radio La Voz – 06/03/2013
Nuevo Diario – 06/03/2013
La Arena – 06/03/2013
DERF – 06/03/2013
infoNEWS – 06/03/2013
El Comercial – 06/02/2013
La Mañana de Neuquén – 06/03/2013
Diario Inédito – 06/02/2013
The Impact Evaluation Network (IEN) at LACEA is an initiative that aims to advance the state of knowledge and expertise regarding impact evaluation of different policies.
- Abdul Latif Jameel Poverty Action Lab (J-PAL)
- International Initiative for Impact Evaluation (3ie)
- Innovation for Poverty Action
- Poverty and Economic Policy Reasearch Network (PEP)
- Poverty Action Lab
- Office of Evaluation and Oversight (OVE)
- Network of Networks of Impact Evaluation
- CLEAR (Regional Centers for Learning on Evaluation and Results)
- African Evaluation Association
- Coalition for Evidence-Based Policy “Social Programs That Work”
- The Development Impact Evaluation (DIME) Initiative
- BRAC