Finding Evaluation Evidence for Policy Guidelines and Evaluations
Using Research in Policy Cycle
The policymaking cycle relies heavily on research and evidence at all its stages. In-depth analysis helps policymakers understanding the complexities of social problems, their underlying factors and outcomes, enabling accurate issue identification and informed agenda setting. Through research, we can generate policy ideas, analyzing various policy options, and guiding decisions on which policies to implement. Research is crucial for effective policy implementation, particularly in evaluating the processes involved. Certainly, program and policy evaluation represent a critical area of research, enabling us to determine the effectiveness of implemented policies and whether they achieve their intended impacts and outcomes. If data, research outcomes, and scientific evidence play a crucial role in the policymaking process, how is this information communicated to those involved? This occurs through various means. Many policymakers have received formal education and training that includes a strong focus on research. Additionally, individuals can certainly access scientific journals and reports that present research findings.
How does scientific information get to people involved with policymaking?
- Formal education and training
- Scientific journals
- Media
- Legislative aids and other staff
- Lobbyists
- Social action and advocacy groups
- Expert testimony
- Government research
- Many other mechanisms
Experts are regularly asked to lend their expertise, including testifying before legislators and regulatory agencies. In many countries, there are several governmental agencies and departments that prioritize science and research. Yet, we recognize that public policy is not always driven by scientific findings and research. Beyond scientific information, a wide range of influences contribute to the policy process. What is the reason for this? While it's a complicated issue, it largely has to do with three categories of barriers: those related to the research process itself, communication hurdles, and political obstacles.
First and foremost, what barriers exist within the research process? Because strong studies with good internal validity are costly and take a long time to conduct, there is a substantial amount of research available that includes limitations and caveats. There are also several important protections concerning research on and data collection from humans, which is certainly a positive thing. There are significant ethical concerns when it comes to researching human subjects, which can impede progress and limit the kinds of studies that can be conducted. Due to these reasons, implementing randomized control trials in the policy sector is difficult. Consequently, this makes it challenging to determine causal relationships, as we understand that association does not equate to causation. Yet, all too often, we must rely on association-based research to guide the policymaking process. Additionally, policymakers often find that the research process takes significantly longer than they would like. It can take a considerable amount of time, sometimes years, to witness the impact of policies and programs on outcomes. Moreover, some outcomes are difficult to assess, even within a short timeframe. As a result, it often takes considerable time for research findings to be published in journals or other types of publications.
Additionally, several communication-related barriers hinder the use of research and evidence in the policy process. This also highlights that scientific research tends to be technical and highly specialized, which can make it difficult for laypeople, including policymakers, to grasp. Moreover, scientists often contribute to the challenge because their fields are filled with jargon and specialized terminology. Truthfully, researchers and scientists are not always the best at communicating their findings to general audiences. There are timing challenges too. Scientific insights might lag behind policymakers' needs due to delays in availability or communication. Lastly, media coverage of studies can be misleading. These situations arise because of valid reasons, journalists seek to communicate sophisticated concepts in a straightforward manner and at an elevated level. However, researchers possess specialized knowledge and technical expertise, often accompanied by numerous caveats regarding their findings, as no study is without its limitations.
Ultimately, the key point is that the policy process, While the policy process is rational in some respects, it is also influenced by a multitude of factors, with scientific information being just one among many. All policy is influenced by values, encompassing those related to political ideology, ethics, moral reasoning, and cultural norms and biases. While we all understand the crucial and undeniable impact of politics and values on the policymaking process, thorough research and analysis are essential at every stage of the policymaking process. The objective should be to develop evidence-based policy whenever feasible, as this approach is crucial for maintaining the core pillars of the public sector strong. It is essential for a government to support rather than hinder the economy, to utilize public resources efficiently, and to create and implement effective policies and programs. Additionally, it plays a critical role in addressing and reducing societal inequalities and inequities. Let me clarify that being objective— which is why we rely on reliable data and evidence— is not the same as being neutral. The generation and application of objective data and research evidence in the policy process can certainly guide conclusions and recommendations at every stage, including problem definition, agenda setting, policy formulation, decision-making, evaluations, and more. The aim is to utilize research to maintain objectivity regarding problems, their causes, and potential policy solutions. However, this does not imply that the data cannot also inform opinions and recommendations.
As policymakers, it is crucial to assess the effectiveness of different policies and to comprehend why some policies succeed while others do not achieve their intended goals. Systematic literature reviews (SLRs) are an essential resource for comprehending the current landscape of evaluation research, supporting evidence-based decision-making, and enhancing scientific knowledge in public policy. Thus, it is crucial for policy professionals to develop the skills necessary to search for, identify, and summarize SLRs effectively.
A Systematic Literature Review (SLR) is is a complete and comprehensive review of the existing research evidence on a particular topic, typically requiring a more structured and rigorous methodology than other types of literature reviews, such as scoping or argumentative reviews. SLRs characteristically frame the research discoveries within evaluations of the study designs' validity. Thus, SLRs are crucial for collecting and evaluating evidence in a reliable and unbiased manner.
What is the importance of a Systematic Literature Review, and what are the methods for conducting one? Researchers conduct systematic literature reviews to draw comprehensive conclusions regarding the current state of evidence in both science and policy practice. In a systematic literature review, researchers gather all published studies relevant to a specific topic or clearly defined research question, following predetermined inclusion criteria. They then assess the quality of the research designs and findings critically, both overall and by the quality of each study design, in order to identify weaknesses, inconsistencies, and gaps in the evaluation research.
Figure 1 illustrates the typical stages involved in performing a systematic literature review.
The initial stage of a systematic literature review entails formulating a clear question and defining the focus of the review. The questions should be sufficiently focused to ensure that a systematic literature review is feasible. The next stage is protocol development, where researchers establish the inclusion and exclusion criteria that will direct the selection of relevant studies. Criteria often include:
- Intervention and Settings
- Peer-Reviewed vs. Grey Literature
- Language
- Study Designs
Researchers generally use a variety of strategies to locate potential studies and then apply the criteria to the studies they have identified. Furthermore, they develop and implement a coding form to review the intervention studies. Sample details that can be recorded for studies include:
- Dates
- Setting
- Population(s) under study
- Statistical analysis approach
The significance of evaluating systematic literature reviews, in the policymaking process, it is crucial to identify and analyze high-quality evaluation studies of policies and programs to gain insights into what the overall body of evaluation research indicates about a particular policy strategy. "Systematic Literature Reviews (SLRs) can be valuable in answering questions about the efficacy of an intervention or the effects of exposure to a specific program or initiative." When presented in policy briefs, Systematic Literature Reviews (SLRs) offer essential information to policymakers, decision-makers, and other stakeholders who often lack the time or resources to thoroughly review and evaluate the evidence pertaining to a specific concern or policy issue. Moreover, program evaluation findings contribute to every stage of the policymaking cycle.
Finding systematic literature reviews, identifying systematic literature reviews doesn’t need to be an overwhelming task. Google Scholar can serve as a useful resource in your search for systematic literature reviews that you can include in your upcoming memo or policy brief. To improve your search, you can begin your policy topic of interest with the phrase “systematic literature review.” Some examples include:
Finding Policy Resources for Issues You Care About
Public Policy Evaluation: A Real Story in Detroit, Michigan, United States
Post-Great Recession, Detroit struggled with a surge in tax foreclosures. As incomes declined, city officials were slow to adjust property taxes to reflect the decreasing property values. One of the many unfortunate repercussions of this crisis was that, alongside the foreclosure of numerous owner-occupied homes, many renter households were also evicted from properties that had been subject to tax foreclosure. As the owners of those homes, their landlords had neglected to pay the owed property taxes. These properties would subsequently be auctioned off at the Wayne County tax foreclosure sale. Often purchased in large quantities by investors who failed to maintain them, these properties were quickly abandoned, leading to greater instability and decline in Detroit's housing environment. In this context, the city, along with the United Community Housing Coalition and the Rocket Community Fund, initiated a program known as Make It Home. Under this program, tax foreclosed homes that housed renter families would be taken off the tax foreclosure list, allowing the tenants living there to purchase the properties at a significantly reduced price, often around $1,000. The Rocket Community Fund will take care of the remaining back taxes. The goal of the program was to keep homes in productive use while providing greater housing stability for these renting families. However, there were significant concerns about whether this program could succeed, as this low-income population might struggle to maintain stable homeownership due to the costs associated with owning a home. How would their situation compare to that of households with similar incomes that chose to remain renters? Poverty Solutions allocated funding for certain individuals, Faculty members at the University of Michigan will investigate these questions to contribute to the program's design.
They conducted interviews with program participants and two different control groups. One group comprised individuals who were eligible for the program but chose different rental housing options, while another group included those who obtained homeownership through other avenues. To gain insight into the experiences of individuals who participated in the program. The findings were mixed. It was revealed that participants in the program experienced greater housing stability compared to those in the other two control groups. Nonetheless, their stability remained at serious risk. They encountered substantial housing repair expenses, and many quickly fell behind on their property taxes. This is precisely where the importance of high-quality evaluation is highlighted. The following year, the program continued to function as before, but with an added modification informed by the evaluation results. Households received up to $8,000 to cover emergency home repairs, enhancing their chances of achieving stability. They assessed this program as well and discovered that, although homeowners reported an improvement in perceived housing stability, many were still struggling to keep up with their property taxes. The following year, counseling services were introduced to help participants apply for property tax relief. After each new phase, they continued to assess the program's outcomes, resulting in adjustments and further evaluations, creating an ongoing cycle of improvement.
As you work to discover your niche in the wide-ranging and complex field of public policy, take some time to reflect on and express your own personal guiding or core principles:
