Using Public Policy for Social Change - Part 6

Research Designs and Concepts for Causal Inferences - Part 1

Research Design

Research design is a critical component of the research process, serving as the blueprint for how a study will be conducted. It guides researchers in collecting, analyzing, and interpreting data to answer specific questions or objectives. The foundation of any research design begins with clearly defined research questions or objectives. These questions guide the entire research process and help researchers focus on what they aim to discover. Here are some key points about defining research questions: 

1. Clarity: Research questions should be clear and concise, allowing for a focused investigation. 
2. Relevance: They should address significant issues within the field of study, contributing to existing knowledge.
3. Feasibility: Questions must be answerable within the constraints of time, resources, and methodology.

Once the research questions are established, researchers can outline their objectives. The primary objectives often include:

• Exploratory Insights: To uncover new patterns or relationships that have not been previously studied.
• Explanatory Insights: To understand the reasons behind certain phenomena or behaviors.
• Predictive Insights: To forecast future trends based on current data.

With clear questions and objectives in place, researchers can develop a specific plan or research design. 

This plan typically includes:

1. Methodology: Choosing between qualitative, quantitative, or mixed methods based on the nature of the questions.
2. Data Collection: Determining how data will be gathered (e.g., surveys, interviews, experiments).
3. Sampling: Deciding on the population to be studied and how participants will be selected.
4. Analysis Techniques: Planning how data will be analyzed to draw meaningful conclusions.

The ultimate goal of research is to provide deeper, explanatory insights rather than merely documenting facts or providing superficial descriptions. This involves:

• Explaining Rather Than Describing: Researchers strive to understand underlying causes and relationships rather than just reporting what is observed.
• Contextual Understanding: Providing context to findings helps in interpreting results in a meaningful way.
• Contribution to Knowledge: Effective research contributes to the broader field by answering critical questions and addressing gaps in understanding.

The primary objective of such research is to provide deeper, explanatory insights, rather than merely documenting facts or providing superficial descriptions. In simple language, to explain rather than to describe.

Causal Inference

Causal inference is a critical process in research aimed at determining whether an observed relationship between variables is indeed causal. This involves not just identifying correlations but understanding the underlying mechanisms that lead to specific outcomes. The primary goal of causal inference is to ascertain whether a particular intervention or exposure has a direct effect on an outcome. This is especially relevant in fields such as epidemiology, where researchers seek to understand the causes of health outcomes and diseases. The process involves comparing actual outcomes resulting from an intervention with hypothetical outcomes that would have occurred without it, often referred to as counterfactuals.

Causal inference is closely linked to the study of etiology, which investigates the reasons behind occurrences of events or conditions. Understanding why things happen requires establishing causal relationships, which can inform interventions and policy decisions. For instance, identifying the causal impact of a public health program on disease prevention can guide future health initiatives

To effectively establish causal inference, researchers employ various methodologies:

1. Randomized Controlled Trials (RCTs): Considered the gold standard for establishing causality, RCTs involve randomly assigning participants to treatment or control groups to observe differences in outcomes
2. Longitudinal Studies: These studies track the same individuals over time, allowing researchers to observe changes and infer causality by establishing temporal relationships between exposure and outcome
3. Statistical Models: Techniques such as regression analysis can help control for confounding variables and strengthen causal claims by estimating the independent effect of an intervention

Despite its importance, establishing causality poses significant challenges:

• Complexity of Relationships: Many factors can influence outcomes, making it difficult to isolate the effect of a single variable.
• Ethical Considerations: In some cases, it may be unethical to manipulate variables (e.g., exposing individuals to harmful substances), limiting the ability to conduct RCTs
• Debate Over Evidence: There is often disagreement among researchers regarding interpretations of evidence, particularly when financial or political implications are involved

Causal inference plays a vital role in understanding relationships between variables and informing effective interventions. By rigorously comparing actual outcomes with counterfactuals and employing robust methodologies, researchers can provide valuable insights into the causes of various phenomena, ultimately aiding in decision-making processes across different fields.

The Counterfactual

A counterfactual refers to a hypothetical scenario that explores what would have happened if a specific intervention had not occurred. It serves as a critical baseline for evaluating the impact of an intervention. In essence, it answers the question: "What would be the outcome if the intervention had not taken place?" 

In experimental research, counterfactuals are typically operationalized through control conditions. Here’s how they function:

1. Control Group: In a randomized controlled trial (RCT), participants are divided into at least two groups: one that receives the intervention (treatment group) and one that does not (control group). The control group represents the counterfactual scenario, providing a baseline against which the effects of the intervention can be measured.
2. Comparison of Outcomes: By comparing outcomes between the treatment and control groups, researchers can infer the effect of the intervention. If the treatment group shows significantly different results than the control group, it suggests that the intervention had an impact.

Counterfactuals are crucial for several reasons:

• Establishing Causality: They help clarify whether changes in outcomes can be attributed to the intervention rather than other external factors. The ability to compare actual outcomes with counterfactual scenarios strengthens causal claims.
• Understanding Mechanisms: Counterfactual analysis allows researchers to explore not just whether an intervention works, but how and why it works. This understanding can inform future interventions and policy decisions.
• Policy Evaluation: In public policy and social sciences, counterfactuals are used to evaluate the effectiveness of programs. For example, policymakers can assess what health outcomes might have been without a particular health initiative, guiding future funding and resource allocation.

While counterfactuals are essential for causal inference, establishing them can be challenging:

• Ethical Constraints: In some cases, it is unethical to deny an intervention to a control group (e.g., withholding life-saving treatments), complicating the establishment of a true counterfactual.
• Complex Interactions: Real-world scenarios often involve multiple variables interacting in complex ways, making it difficult to isolate the effect of a single intervention.
• Statistical Limitations: Relying solely on observational data to create counterfactuals can introduce biases due to confounding variables that may influence outcomes.

Counterfactuals are a vital component of causal inference, allowing researchers to compare actual outcomes with hypothetical scenarios to determine the impact of interventions. By utilizing control groups and rigorous experimental designs, researchers can establish clearer causal relationships and provide valuable insights for effective policy-making and program evaluation. Understanding counterfactuals enhances our ability to make informed decisions based on evidence, ultimately leading to better outcomes across various fields.

Consider a new educational program is implemented in certain schools to improve student performance. To evaluate the program's effectiveness, researchers compare the test scores of students who participated in the program (the intervention group) to those who did not (the control group or counterfactual condition). The counterfactual condition represents what would have happened to the students in the intervention group if they had not received the educational program. By comparing the outcomes of the intervention and control groups, researchers can determine the actual impact of the program, isolating it from other factors that may have influenced student performance. The counterfactual condition serves as the baseline for comparison, allowing researchers to establish causality and determine whether the educational program was the primary factor responsible for any observed improvements in test scores. A strong research design enables an accurate evaluation of the counterfactual.

Let’s examine some policy evaluation research designs as an example: 

1. The simple pre-test, post-test design.

Evaluating a Public Transportation Policy

Objective: To evaluate the effectiveness of a new public transportation initiative that offers free bus rides for one month to encourage more residents to use public transit.

Participants: Residents of the city who regularly commute to work.

Pre-Test: Prior to the implementation of the free bus rides, researchers conduct a survey to collect data on participants' commuting habits, including the mode of transportation they currently use (e.g., car, bus, bike) and their average commute times.

Intervention: The public transportation policy is implemented, offering free bus rides for one month.

Post-Test: At the end of the month, researchers conduct the same survey to measure any changes in commuting habits and average commute times.

Analysis: Researchers compare the pre-test and post-test data to determine if there was a significant increase in the number of residents using public transportation and a corresponding decrease in average commute times.

If the post-test results show a notable shift towards increased bus usage and reduced traffic congestion, the researchers can infer that the policy was effective in achieving its goals. This pre-test, post-test design allows for a straightforward assessment of the policy's impact by measuring changes in behavior before and after the intervention.

2. The two-group comparison or difference-in-differences research design.

This design involves two groups, one of which receives the intervention while the other serves as a control group and does not receive the intervention. Both groups are observed at two time points: before and after the intervention is implemented.

The key aspect of this design is the comparison of changes over time between the two groups. By comparing the differences in outcomes between the intervention and control groups from the first observation point to the second, researchers can estimate the effect of the intervention. For instance, let's say childhood obesity rates in a city that implemented a soda tax declined from 25% to 20%. However, in a similar comparison city without the soda tax, obesity rates also decreased from 23% to 15% during the same period. In this case, the comparison city serves as the counterfactual, representing what would have happened in the intervention city without the soda tax. The fact that obesity rates declined in both cities suggests that the soda tax policy may not have been solely responsible for the decline in the intervention city.

The two-group comparison or difference-in-differences design allows researchers to account for underlying trends that may be influencing the outcome, providing a more strong estimate of the intervention's effect.

3. The time-series design, making observations using data across a long period of time.

Example of policy evaluation research using a time series design, the smoking ban was introduced to reduce secondhand smoke exposure and improve public health. Prior to the ban, there were growing concerns about the health risks associated with smoking and secondhand smoke, prompting the need for effective public health policies.

Methodology

Data Collection: Researchers collected monthly data on hospital admissions for respiratory and cardiovascular diseases from January 2005 to December 2015. The pre-intervention period included 36 months (January 2005 to December 2007), while the post-intervention period spanned 96 months (January 2008 to December 2015).

Time Series Analysis: Using an Interrupted Time Series design, researchers conducted segmented regression analysis to assess changes in hospital admission rates before and after the smoking ban. This approach allowed them to evaluate both the immediate and long-term effects of the policy.

Control for Confounding Factors: The analysis controlled for potential confounding variables such as seasonal trends in hospital admissions, changes in population demographics, and other public health initiatives implemented during the study period.

Findings

Pre-Intervention Trends: Prior to the smoking ban, hospital admissions for respiratory and cardiovascular diseases showed a consistent upward trend.

Immediate Impact: Following the implementation of the smoking ban, there was a significant immediate reduction in hospital admissions for both respiratory and cardiovascular diseases. The segmented regression analysis indicated a sharp decline in admission rates right after the policy was enacted.

Long-Term Trends: Over the post-intervention period, the data showed a sustained decrease in hospital admissions, suggesting that the smoking ban had a lasting positive impact on public health. By the end of the study period, hospital admissions for respiratory diseases had decreased by approximately 25%, while cardiovascular admissions declined by around 15%.

Implications: This study exemplifies how time series analysis, particularly Interrupted Time Series design, can effectively evaluate the impact of public health policies such as smoking bans. The findings support the effectiveness of the smoking ban in improving public health outcomes, providing evidence for policymakers regarding the benefits of such interventions.

Conclusion: Through the use of ITS analysis, researchers were able to rigorously assess the effects of the smoking ban on hospital admissions, contributing valuable insights to the ongoing discourse on public health policies aimed at reducing smoking-related health risks. This example highlights the importance of time series designs in evaluating the long-term impacts of health interventions in the absence of randomized controlled trials.

Source: coursera