Using Public Policy for Social Change - Part 8

Program Evaluation Examples Part 1

Let's start by analyzing the research design used to evaluate the Creating Moves to Opportunity Program in Seattle, Washington. We start by identifying the issue at hand, which in this instance is the difficulty of publicly subsidized housing that this program seeks to address. The United States invests over $45 billion each year in initiatives aimed at providing affordable subsidized housing, which includes the Housing Choice Voucher Program. This is sometimes called the Section 8 Program. Each year, over two million families, totaling around nine million Americans, benefit from vouchers that provide reduced rent for apartments and homes throughout the United States. Tenants pay a portion of the rent, while the government covers the rest by making direct payments to the landlords. Although housing vouchers allow families to rent anywhere in the city, the majority tend to remain in high-poverty, low-opportunity neighborhoods when they receive a new housing voucher. The Creating Moves to Opportunity Program in Seattle was designed to improve the relocation of families with housing vouchers to higher opportunity neighborhoods by addressing specific barriers to their mobility.

The Creating Moves to Opportunity Program consists of three essential elements (Intervention Components).

1. First, it provides search assistance to families with vouchers, which includes education about higher opportunity neighborhoods throughout the city, support in completing rental applications, and services to help locate housing.
2. The second component focuses on landlord engagement. Because not all landlords are willing to accept families with housing vouchers, this part of the intervention seeks to recruit more landlords throughout the city who are open to working with voucher clients, while also simplifying the approval process for their participation in the program and assisting with relocations to diverse neighborhoods. Furthermore, this intervention established a new insurance fund to cover property damage, addressing landlords' concerns that tenants might cause more harm than the security deposit could compensate for. This insurance fund aims to alleviate landlords' concerns, whether they are justified or not, regarding property damage.
3. Third, the program offers short-term cash assistance to families to support their relocation to different areas of the city. As is well known, moving within a city can be expensive, so the Creating Moves to Opportunity Program provides grants to help individuals accumulate the necessary funds to cover their moving costs. This includes costs like renting a moving truck, setting up new cable services, and assisting with application fees and security deposits. The average support provided to families through the program is around $1,100. The Creating Moves to Opportunity Program is currently being evaluated using a Randomized Controlled Trial (RTC) design. In this design, there is no pretest observation, but this is acceptable since the intervention and the families participating in the research study are all new to public housing vouchers. 

The counterfactual refers to the control or comparison group. Families recently approved for the Housing Voucher Program in Seattle were randomly assigned to participate in the Creating Moves to Opportunity Program, and its three components compared to receiving standard services. This includes providing information about the process and encouraging families to search for housing across the entire city, but without the additional support and resources offered by the new intervention. The researchers plan to monitor the families involved over an extended period. At present, they are conducting an impact evaluation that focuses on specific short- and near-term outcomes. This includes the types of neighborhoods where individuals live after being approved for the Housing Voucher Program, distinguishing between high - and low-opportunity neighborhoods, as well as whether they make use of their vouchers. Next, we will look at adults' feedback regarding their satisfaction with their neighborhood. 

Let’s explore some initial findings that compare outcomes between the treatment and control groups. In the intervention or treatment group, 54% lived in high-opportunity areas, 88% used the housing voucher for a new lease or rental agreement, and 68% expressed a high level of satisfaction with their neighborhood. In the control group, only 14% ended up living in a high-opportunity area after receiving a housing voucher. However, this was not because of a lack of voucher usage or participation in the program; 84% utilized their housing vouchers for new leases or rental agreements, yet only a small percentage took advantage of the opportunity to relocate to a different neighborhood. In the control group, just 33% reported a high level of satisfaction with their neighborhood. The preliminary results from the Creating Moves to Opportunity study in Seattle suggest that this program has substantially increased the number of families living in higher opportunity areas within the city and has also improved family satisfaction with their neighborhoods. It is widely recognized that individuals typically prefer to remain in familiar neighborhoods or communities. The findings presented here are quite fascinating, and many cities are eagerly anticipating further updates from the evaluation team. 

Program Evaluation Examples Part 2

Let’s take a look at an example of a time series design. Keep in mind that a time series research design analyzes data on trends and outcomes of interest over various time periods. Analyzing shifts in the slope or intercept of the trend line at the intervention or policy change indicated by x. The primary threat to the internal validity of this design is the influence of history. An alternative event may have taken place concurrently with the policy change, which could account for some or all of the changes observed after the intervention. 

This graph depicts one of several potential scenarios within the time-series design. In this particular case, we see a flat trend, with the slope of the line remaining consistent prior to the implementation of the intervention. However, after the intervention or policy change, the line's slope begins to decline. Following the intervention or policy change, the slope of the line starts to decrease. For example, an evaluation of the impact of a fare increase in a metropolitan subway system might reveal that an increase in prices, following a long period of stable fares, led to a subsequent decline in ridership. In this case, the intervention was the rise in subway fares, and after this change, ridership could decrease

Now, let's turn our attention to evaluating a significant change in alcohol policy and regulation that took place in Russia in 2006. The primary question driving this evaluation was: given the established connection between alcohol abuse and suicide, did the 2006 policy change impact the national suicide rate? The 2006 policy involved several components, including new regulations on both the volume of sales and the quality of products available in the market. The 2006 policy implemented new regulations requiring alcohol production and distribution facilities to register with the government and undergo enhanced monitoring. Both types of regulations operated similarly to a tax, resulting in a decrease in the number of alcohol producers and distributors, as well as higher prices for consumers.

Researchers utilized a time-series research approach to track monthly suicide rates in the country, comparing data from both before and after the major policy changes. Additionally, they conducted this analysis separately for males and females. Their primary findings indicated significant seasonal variations in suicide rates. It is essential to take seasonal variations and the outcomes under investigation into account in a time series analysis. The results also showed that monthly suicide counts for both males and females decreased consistently during the entire stable period. There was a steady decline in suicide rates over the years of the time series, which started before the intervention was implemented. Nevertheless, statistical analysis of the time-series data revealed an immediate and sustained 9% reduction in male suicides.

No similar outcome was observed for females. Here is a clear graph displaying the results. The time series demonstrates the seasonal or monthly fluctuations in suicide deaths each year. The graph also depicts a downward trend for males, which appears to accelerate following the 2006 policy change. We can visually inspect the figure to identify the main findings. However, the actual analysis was quite complex, involving advanced statistical techniques such as regression methods to determine whether the trend line altered after the intervention. Whether you choose to be a detailed policy evaluator or not, having a basic understanding of this critical aspect of the policy-making cycle will be vital for your work.

Implementation Evaluation: A Real Story

For instance, the federal program established in Michigan provides additional nutrition and food support for pregnant individuals and children under five. This initiative, known as the Women, Infants, and Children (WIC) Nutrition Benefit, has seen many eligible individuals fail to enroll. Evaluators recognize that accessing the program can be challenging; some people are unaware of their eligibility, while others do not know how or where to start the application process. To address this, the evaluators implemented a new strategy by sending text messages to potential clients who likely qualify but are not enrolled. The evaluators aimed to determine whether (a) informing them of their likely eligibility for benefits and (b) providing a simple, one-touch method to initiate the application would enhance their access to the program. 

To address the question, the evaluator conducted a Randomized Control Trial. They compared WIC enrollment rates between a test group that received text messages and a control group that did not. Ultimately, the test group experienced a 91% increase in enrollment compared to the control group. This project made all the challenges faced during statistics at the Ford School completely worthwhile.

It’s not uncommon to discover insights during an evaluation that suggest the need for a new agenda item. For instance, a recent evaluation of a food assistance program revealed that many low-income individuals and families facing hunger did not qualify for benefits simply because their assets exceeded the $5,000 limit, which included the value of their vehicle.

The evaluation revealed that the evaluator needed to shift the focus from preventing fraud and abuse to aligning with the economic realities of today. It highlighted that individuals with $6,000 to $10,000 in assets, who are struggling with hunger, should be eligible for food assistance. Consequently, the department aimed to increase the asset limit, exclude car values from the calculation, and allow most applicants to self-report their net worth. This update was in line with the governor's initiative to address food insecurity and poverty in Michigan. Additionally, it streamlined the application process, saved time, and reduced paperwork for state caseworkers.

Today, the evaluator continues to assess the implementation of all their support programs to identify who is being served and who is being overlooked, in order to develop the next set of agenda items.

Real-World examples of Experimental Design: Voter Information & Political Selection in India

An example of experimental design can be observed in India. In general, India is known as the most populous democracy in the world, with over 800 million registered voters. This environment is marked by a significant prevalence of criminal charges against both political candidates and elected officials. For instance, 34% of current members of India’s national parliament face serious criminal charges, including offenses such as assault, kidnapping, attempted murder, and even murder.

Furthermore, recent high-quality empirical research conducted by the study's authors demonstrates a causal relationship between the presence of legislators facing multiple or serious criminal charges and subsequent declines in economic growth and insufficient public service delivery. With this in mind, a paper authored by an assistant professor and his colleagues at the Ford School of Public Policy at the University of Michigan aims to address several important research questions.

The first research question focuses on the extent to which voters in India are aware of candidates' criminal charges and whether they feel indifferent to or may even view such criminality positively in their elected officials. There is a general notion that voters might prefer officials with criminal backgrounds, as it could imply they possess the capability to hold office, irrespective of the legality of their actions. Conversely, it is also possible that limited access to information about candidates' criminal records is preventing many voters from rejecting those with serious criminal histories or multiple charges.

The second research question investigates whether a gentle approach to providing information—specifically through mobile phone messaging and voice/text communication—can help close any existing information gap and subsequently influence voting behavior. The focus is on the Northern Indian state of Uttar Pradesh, which is highlighted in red on the map. Here are some additional details about Uttar Pradesh (UP).

First and foremost, Uttar Pradesh is the most populous state in India, with a current population exceeding 200 million residents. It is also one of the poorest states in the country, with a substantial rural population that makes up nearly 80% of its inhabitants. This situation suggests that many individuals may encounter difficulties in accessing publicly available information about candidates' criminal records on websites.

On the other hand, researchers noted a high prevalence of mobile phone usage in these areas. At the time of the study, more than 85% of households owned at least one mobile phone, suggesting that mobile messaging could be an effective means of communicating information about candidates' criminal charges. Uttar Pradesh is not an exception in having a substantial number of political candidates and elected officials with criminal charges. In fact, during the 2017 State Assembly elections, it was discovered that 25% of the winning candidates faced serious criminal charges. If the researchers concentrate more intensely on the particular region within Uttar Pradesh where their experiments were conducted, they can provide several relevant statistics. Notably, 80% of electoral contests involved at least one major party or incumbent candidate facing some form of criminal charge, while 26% of these contests included at least one candidate charged with murder or attempted murder. Additionally, when analyzing candidates with criminal charges, it was discovered that they were involved in an average of 2.2 criminal cases each. What was the focus of the intervention, and what methods did the researcher employ to implement it?

To begin, it is important to note that the 2007 State Assembly elections in Uttar Pradesh were conducted over seven distinct phases. The elections were organized by geographic regions, with election days scheduled in phases over several weeks, and the researchers operated within Phase 4. Two days before election day, the researcher sent voice and text messages to mobile phone subscribers, informing them about any criminal charges against major party and incumbent candidates, along with the number of cases associated with those charges. It is important to highlight that a significant challenge in initiating this project was securing a partnership with a willing telecom company. As expected, there were concerns regarding potential backlash from political parties or individual politicians due to their involvement in disseminating information about candidates' criminal charges. 

Now, as the researcher examines the details of the experimental design, they should take into account their experimental sample and the precise locations of their work. Within the phase 4 area, there were roughly 3,800 villages and about 5,000 polling stations—locations where voters would go to cast their ballots on election day. These villages were spread across 38 assembly constituencies, which are regions that elect one representative to serve in the state assembly. The researchers were operating within 38 out of a total of 403 constituencies. To ensure they were focusing on relatively average villages, they restricted their sample to those with either one or two polling stations. Furthermore, these villages neither had excessively large nor small populations, and they did not exhibit extreme variations in Vodafone Idea's mobile phone subscriber coverage. To provide context, the villages in the sample had an average of approximately 1,200 registered voters. To determine the average impact of the messaging treatment, the researcher needed to randomly assign villages to either the treatment group or the control group. These villages included those where mobile phone subscribers received the researcher’s voice and text messages containing information about candidate criminality, as well as the control group, where no such messages were sent. The researcher allocated approximately 1/3 of the villages, roughly 3,800, to the treatment group and 2/3, also roughly 3,800, to the control group. 

By means of  random assignment, the researcher can assess the average impact of the messaging treatment on voting outcomes by comparing the average results at polling stations in treatment villages with those in control villages. Significantly, the researcher ultimately sent messages to more than 370,000 individuals in the treatment villages. Essentially, the implementation of the messaging treatment allowed the researcher to evaluate the intervention's effects using multiple datasets. At the outset, a key consideration for the researcher was gaining access to information about candidates, their criminal cases, and the related charges. 

Fortunately, the Indian government requires all candidates seeking office to disclose this information, which is then made public. Consequently, the researcher was able to obtain this data. Secondly, the researcher needed various datasets related to the villages. The first dataset concerned the number of mobile phone subscribers in each village, which was sourced from Vodafone Idea. The second dataset included population data for each village, which the researcher obtained from public census data. Additionally, the researcher required shapefiles—essentially digital maps for each village—which were acquired from the company ML Info Map India. The researcher needed those map files to combine them with polling station location data. This enabled the researcher to ascertain whether each polling station was located in a treatment village that received messages or in a control village that did not. 

To evaluate the effects of the treatment on voting, the researcher needed polling station-level voting results, which were publicly accessible from the government of Madhya Pradesh. By merging all these datasets, the researcher could first carry out the experiment and then assess the impact of their intervention. The researchers are presenting a figure that summarizes the key voting outcomes from their mobile information provision experiment, categorizing the candidates into five groups based on the severity of their criminal charges. On the left are the candidates with the least severe charges, specifically those with no pending criminal charges. Next are the candidates whose most serious charge is non-violent. Following that are candidates with violent charges that do not involve murder. The fourth group includes those facing attempted murder charges, and finally, the rightmost group consists of candidates charged with murder-related offenses. 

This analysis explores the average effects on voting outcomes for candidates in villages that received the messaging treatment compared to those that did not. Being situated above the red line indicates that candidates of a certain type experience positive voting outcomes in treatment villages, while being below the red line suggests that these candidates encounter negative impacts on their voting results in treatment villages.

The researcher has identified a distinct negative correlation between the voting impacts of the messaging treatment and the severity of charges faced by candidates. Notably, upon closer examination, the leftmost portion of the graph shows a positive impact on votes for candidates with no criminal charges at all.

We can conclude that candidates without criminal charges experienced an average increase of 2.8% in votes in areas that received the messaging treatment compared to those that did not. In contrast, candidates facing murder charges experienced a substantial average decrease of 11.1% in votes in the same comparison. This analysis reveals a negative correlation between the voting impacts of our messaging treatment and the severity of candidates’ charges. Beyond the specifics of these research findings, it is crucial to consider their implications for policymaking and policy decisions.

The primary implications the researcher aims to highlight are as follows:

They observed that voters did respond to this information. This contradicts a scenario in which most voters are already aware of the candidates' criminal charges and remain indifferent or view them positively. This indicates a potential positive outcome from conducting these information provision campaigns.

Secondly, this relates to the nature of the information provision. They found that their approach was relatively low-profile, utilizing a few text messages and voice recordings sent out prior to the elections. The positive results we achieved are encouraging for two main reasons. First, this method reduces the risk of political parties intervening during the information dissemination process. For example, if we were to conduct door-to-door outreach or hold public meetings in a village center, there would be a significantly greater chance of political parties becoming aware of these activities and disrupting the sharing of information. Second, from a budgetary standpoint, this approach is more cost-effective for reaching a target audience compared to methods like in-person outreach or purchasing advertising space in newspapers or on radio. Therefore, if our aim is to maximize outreach within a limited budget, these findings are quite promising.

Finally, when considering not only the specific context of this experiment but also the broader implications for the political and logistical feasibility of future campaigns—whether on a large scale within a single electoral cycle or across multiple cycles—several positive insights emerge from this work. Firstly, it is noteworthy that we encountered no negative repercussions from political parties or politicians, either for ourselves as authors, our team, or Vodafone Idea. This is encouraging because, in future scenarios where one might aim to scale up a similar intervention in subsequent election cycles, it provides a clear example to telecom companies that may be hesitant. They can refer to this instance where there were no adverse effects for the telecom involved.

However, it is crucial to acknowledge that while operated in thousands of villages, this was still a relatively small portion of the overall election landscape. If such efforts were to be conducted across multiple elections over time, it becomes more probable that political parties would take notice and potentially attempt to interfere with the information dissemination or undermine its effectiveness. Despite this, the current indications are encouraging regarding the potential benefits of this intervention in enhancing the quality of elected officials in India, particularly with regard to criminal charges.

Source: coursera