2020 State Standard of Excellence
10. Evidence Definition and Program Inventory
Did the state or any of its agencies release a common evidence framework, guidelines, or standards to inform its research and funding decisions and make publicly available an inventory of state-funded programs categorized based on at least two tiers of evidence?
Defining evidence and conducting program inventories can help state governments ensure that their programs are using proven practices and that their interventions are effective in meeting the state’s desired goals as well as improving outcomes for residents.
A 1999 New Mexico law required all state agencies to submit annual performance-based budget requests that include outputs, outcomes, performance, and evaluation data. The 2019 Evidence and Research Based Funding Requests Act amended the 1999 law by defining four tiers of evidence and further requiring certain state agencies (selected annually by the state legislature), to “identify each sub-program as evidence-based, research-based, promising, or lacking evidence of effectiveness” and report on the amount allocated for each of these evidence tiers. The law builds on New Mexico’s long-standing series of inventory and cost-benefit reports in the areas of children’s behavioral health; adult behavioral health; early education; child welfare; criminal justice; healthcare, infant and maternal health; and education.
The California Department of Social Services created the California Evidence-Based Clearinghouse for Child Welfare, which allows child welfare providers and professionals to “identify, select, and implement evidence-based child welfare practices that will improve child safety, increase permanency, increase family and community stability, and promote child and family well-being.” This tool helps identify best practices and provides guidance and support for program implementation. The Clearinghouse’s numerical rating scale categorizes programs into six tiers of evidence and uses a relevance scale as a complement to the scientific rating scale and to demonstrate applicability for client populations.
The Colorado Governor’s Office of State Planning and Budgeting, along with the Legislature’s Joint Budget Committee, established a common set of evidence definitions and a five-step evidence continuum that includes actions to increase a program’s level of evidence. The Office also publishes periodic Results First reports that categorize all state-funded prevention programs in the areas of child welfare, criminal justice, and juvenile justice according to three tiers of evidence: evidence-based, promising, and needs additional research.
A 2015 Connecticut law defines three tiers of evidence for programs operated by the Connecticut Departments of Correction, Children and Families, and Mental Health and Addiction Services, and the Court Support Services Division of the Judicial Branch: evidence-based, research-based, and promising. The law requires these agencies to categorize their programs by the evidence tiers in even-numbered fiscal years. Additionally, the law charges the Institute for Municipal and Regional Policy at Central Connecticut State University with submitting an annual report containing a cost-benefit analysis of the programs, which, in 2019, was entitled Benefit-Cost Analyses of Evidence-Based Programs. The findings showed that the Judicial Branch’s Court Support Services Division spent 79% of their adult program funding and 86% of juvenile funding on evidence-based programs; the Department of Children and Families spent 20%; and the Department of Corrections spent 98% on such programs.
Florida’s Department of Juvenile Justice requires all residential commitment prevention contractors to implement at least one evidence-based model from the agency’s Sourcebook of Delinquency Interventions. The sourcebook lists all juvenile justice programs according to three levels of evidence: evidence-based practices, promising practices, and practices with demonstrated effectiveness, as defined in the book. The Department also introduced a Standardized Program Evaluation Protocol, a monitoring tool to ensure providers implement programs with fidelity.
As a result of a 2015 Minnesota law, Minnesota Management and Budget has developed numerous inventories of evidence-based programs, including criminal justice, mental health, child welfare, and higher education programs. Minnesota Management and Budget also maintains the Minnesota Inventory, a searchable clearinghouse of more than 500 state programs. As part of the inventory, the state developed a guide for using evidence in policymaking and evidence definitions to categorize interventions as proven effective, promising, theory-based, or having no effect. These resources informed funding decisions in the state, including $87 million in new or expanded evidence-based programming in the FY 2020-2021 budget.
A 2019 amendment to Mississippi’s 2014 performance-based budgeting law provided strong, more rigorous evidence definitions for evidence-based programs, intervention programs, research-based programs, promising programs. While the law continued to require the Mississippi Departments of Corrections, Health, Education, and Transportation to report during the annual budget process on their programs’ performance and cost-benefit ratio, the amendment authorized the Joint Legislative Committee on Performance Evaluation and Expenditure Review to designate additional agencies to comply with the law. Additional agency inventories include: Departments of Revenue and Medicaid. The Mississippi Joint Legislative Committee on Performance Evaluation and Expenditure Review’s 2018 Results First Mississippi analysis on juvenile justice programs at a residential facility should invest more resources in high-quality interventions and ensure implementation fidelity. The analysis identified the state’s need to define high-quality research.
The North Carolina Office of State Budget and Management released program inventories in the areas of child and family health and juvenile justice. The state also publishes periodic Results First progress reports and has created evidence definitions to categorize programs based on their levels of evidence.
The Ohio Department of Education’s evidence-based clearinghouse provides a common evidence framework by aligning evidence standards from various clearinghouses to assist educators in identifying evidence-based interventions according to content focus area, subject, grade bands, urban/rural settings, student demographics, and the four levels of evidence in federal Every Student Succeeds Act.
A 2003 Oregon law created evidence definitions for agencies that provide direct client engagement, including the Oregon Youth Authority, Department of Corrections, and the Oregon Health Authority. These evidence definitions require programs to incorporate “significant and relevant practices based on scientifically based research” by “[u]tilizing randomized controlled trials when possible and appropriate.” The law requires these agencies to perform cost-benefit analyses and compile a biennial program inventory with results from funded programs, such as the inventory of evidence-based practices produced by the Oregon Health Authority’s Health Evidence Review Commission.
In 2020, the Pennsylvania Workforce Development Board unanimously voted to adopt a four-tiered evidence framework for the state’s workforce system. The definitions for evidence of effectiveness include: strong evidence, moderate evidence, preliminary evidence, and pre-preliminary evidence.
In 2020, the Pennsylvania Department of Education launched the PDE Evidence Resource Center to help Pennsylvania schools thoughtfully apply high-quality, relevant research to their local settings. By listing evidence-based interventions in the areas of instruction and curriculum, student support and professional development, and student wellness, this tool helps school districts identify and select strategies to address every aspect of school improvement. This tool is also being used for the PDE’s research agenda that allows the department to identify and measure its progress in meeting its research and learning priorities.
In 2020, the Texas Workforce Commission, in partnership with the Office of the Governor, Texas Workforce Investment Council, and Rural Capital Area Workforce Board, collaborated to develop a new Evidence Framework to better measure the impact of publicly funded workforce development programs. In July 2020, the Commission approved $1.8 million in Workforce Innovation and Opportunity Act (WIOA) statewide funding for the Building Construction Trades project, which will utilize the evidence framework to better measure the success of the grant program.
In 2018, the Vermont Agency of Education adopted four tiers of evidence, based on the federal Every Student Succeeds Act (ESSA) for use in distributing education funds, including the bulk of federal education and state-funded grants (such as BEST/Act 230 and Flexible Pathways; see sample statement of evidence). The guidance states: “applying the same requirement to both federal and state grants will allow for greater consistency in the review and approval of grant applications.”
A 2012 Washington law stated that “prevention and intervention services delivered to children and juveniles in the areas of mental health, child welfare, and juvenile justice [must] be primarily evidence-based and research-based” (p. 2); directed the Washington State Institute for Public Policy (WSIPP) to develop definitions for three levels of evidence: evidence-based, research-based, and promising practices (p. 4); and tasked WSIPP with creating an inventory of evidence-based programs. The most recent inventories include evidence reviews in children’s mental health, child welfare, and juvenile justice (2019), cannabis prevention and treatment (2019), adult corrections (2018), and K-12 educational programs for struggling students (2020) (which is also featured on the state superintendent’s website as Menus of Best Practices and Strategies).