ECONOMIC TOOLS TO EVALUATE SOCIAL SCIENCE PROGRAMS BIDISHA MANDAL SCHOOL OF ECONOMIC SCIENCES WASHINGTON STATE UNIVERSITY November 16, 2011 WSAC, 2011 Extension Directors Conference Overview Economic tools Why evaluate? What are economic tools, economic analyses? When to evaluate? How to evaluate? Case studies from Health Extension Spokane Public schools: Nutrition services intervention for middle school students Strengthening Families Program: Substance abuse prevention for adolescents Providence health care services: Transitional care model to reduce preventable hospital readmissions Supplemental Nutrition Assistance Program Education: University of Idaho Extension Magnitude of the Problem Primary prevention Health promotion Secondary prevention Screening, diagnosis, therapies Tertiary prevention Treatment to prevent or postpone complications A Crucial Difference Program effectiveness Outcome oriented Directly links the intervention with health outcomes of interest Program evaluation Ways to maximize the intended impact with available resources, or Ways to obtain a particular impact with as little resources as possible Importance of Evaluation Resource constraints To eliminate or reduce waste Evidence of return on investment Ideally… Inform decisions Help make choices about future allocations Example: Health-care system Getting value for money is a widely accepted and legitimate goal Quality medical care in part translates into potentially expensive demands for new drugs and technology Pressure to improve efficiency, make trade-offs, and develop incentive systems for patients and physicians while holding down healthcare costs Address Two Questions What works? What works best? Multiple interventions could work Identify the one intervention that provides the greatest bang for the buck! Example: Increased prevalence of diabetes among adults Strategies and reasonable alternatives Physical activity – Fitness programs (worksite, community, less TV/computer time) Diet – cooking programs, education program to change food consumption behavior What is the objective Increased physical activity? Short-term Reduced risk of diabetes? Long-term Economic Analysis Evaluation is part of the program design and planning There are always competing use of resources Frame the study in order to consider opportunity costs for each of our choices Identify, measure, value and compare the costs and consequences of alternative prevention strategies Quantitative and analytic methods Cost-benefit analysis Cost-effectiveness analysis Cost-utility analysis Identify the Risks Factors Define the target population for the intervention Define the problem or question, and magnitude of impact Define the information needs of the target population in reference to the program or intervention These steps will Influence the types of benefits and costs to be included Help to determine which analytic method is most appropriate Identify the Intervention(s) Indicate clearly the preventive strategies under consideration, including baseline if any Specify perspective of the program and analysis Limit perspectives to those relevant to the study Define relevant time frame in which program will be delivered Determine how far into the future costs and effects that accrue from the intervention will be considered Background on the Intervention Can it work? Will it work? Efficacy Degree to which intervention strategies can work under ideal conditions, with carefully selected participants, and optimal resources. Example: Randomized controlled trials Effectiveness How well these strategies actually work in community settings Demonstrates real-world effectiveness under practical resource constraints Effectiveness is likely to be lower than efficacy What are the benefits and costs of the intervention? Units of measurement How do benefits compare with costs? What additional benefits could be obtained with additional resources? Identify the Methods Determine the analytic methods for decision-making The choice will depend on the policy question, the outcomes of interest, and the availability of data Determine whether analysis is to be marginal or incremental Marginal analysis: Examines the effect of expanding or contracting an intervention Incremental analysis: Compares the effects of alternative programs Identify the Outcomes Identify the relevant costs Program costs Productivity losses Identify the relevant outcomes Number and nature of health outcomes Specify the discount rate or time preference for monetary and non-monetary costs Identify sources of uncertainty and plan sensitivity analysis The Role of Discount Rate Individuals generally weight costs and benefits in the near future more heavily than in the distant future This applies to the valuation of capital and investments and to health outcomes Societal preference is for health benefits received today versus health benefits received in the future Using an appropriate discount rate in an economic analysis allows Adjusting the value of receiving benefits today versus in the future or of incurring costs today versus in the future Makes benefits and costs comparable over time Discount rate is selected based on the study perspective Social or private or individual The Role of Uncertainty Precise estimates of costs and benefits/effects are often not available Limited literature Different population settings Important to list all assumptions upon which estimation is dependent Perform sensitivity analyses How will result of evaluation change if the assumptions change? Evaluation Tools 3 most commonly used techniques Cost-benefit analysis (CBA) Cost-effectiveness analysis (CEA) Cost-utility analysis (CUA) Each method Allows comparison of different intervention strategies Calculates resources consumed and outputs generated Requires quite similar cost analysis Assessment of outcomes, both benefits and harms (negative benefits) Scope of analysis determines the appropriate analytic method Cost-Benefit Analysis All costs and benefits valued in dollars Costs include Cost of program Cost to participants - out-of-pocket expenses, productivity losses, travel time, child care, intangible costs (pain, suffering) Benefits include All types of beneficial and harmful health outcomes, whether intended or not Have to be valued in monetary terms CBA is well suited to comparisons with interventions that include cross-sector considerations Housing, education, transportation interventions Cost-Effectiveness Analysis Usually examines direct medical, non-medical, and productivity costs Compares costs with outcomes in standard health units Example: costs per case averted Most suitable when comparing interventions that have similar health outcomes Cost-Utility Analysis Modified version of CEA Compares direct medical and non-medical costs with health outcomes converted to a standard health unit, often a quality adjusted life year (QALY) combining both mortality and morbidity Often used to compare health intervention which have different type of health outcomes CASE STUDIES Spokane Public Schools: Objective Spokane public schools’ lunch program With Doug Wordell, Ruth Bindler, Kenn Daratha, Sue Butkus Intervention program included reducing vending machine beverages, limiting ala carte offerings, and adding seasonal fruits and vegetables to student lunch menus Compare pre-program and post-program behavior Objective My involvement If there were associations between an altered school food environment and food choices of middle school students both in and outside of school Retroactive, after program was delivered and surveys were conducted My role Analyze survey data Determine food behavior change Spokane Public Schools: Method & Results Results Healthful modifications in the school food environment associated with some positive food behaviors The cost of conducting the intervention was approximately $24,000/year Lost ala carte sales, loss in vending machine sales More expenditure on produce In this study, is it possible to show the benefits of improved food choices outweigh the costs? NO - Related data was not collected for CBA Spokane Public Schools: What did I learn? Difference between outcomes and impacts In this study, there is no way to link improvement in behavior to improvement in health (short-term or longterm) Even if they are positively related, we have no quantitative measure for the benefits Economic analysis is not always possible Unless evaluation is part of the program design Strengthening Families Program: Objective Strengthening Families Program (SFP) for Parents and Youth 10-14 years With Laura Hill, Robby Rosenman, Ron Mittelhammer Voluntary, family-based intervention Designed to discourage future substance abuse among adolescents and youth Compare pre-program and post-program behavior Randomized clinical trials (RCT) have shown SFP to be costeffective and that benefits outweigh costs How does SFP’s impact in community dissemination compare to results from RCTs? My involvement Retroactive, after programs were delivered and surveys were conducted Strengthening Families Program: Method & Results Community dissemination has many practical issues CBA, CEA and CUA not possible Do not have necessary data But, have data to determine which factors in community dissemination of the program are different Variation in program delivery across counties, states Data not recorded systematically or consistently Attrition – high incompletion rates, some people come to the sessions but do not respond to surveys Self-selection – more motivated parents are more likely to attend SFP Results People who come to the sessions but do not respond to surveys have lower self-assessment scores Strengthening Families Program: What did I learn? Validity of assumptions Does sample match population? Differential dropouts Are facilitators similar? Some sessions have additional orientation session Language of delivery Providence Hospital Transitional Care: Objective Transitional care model With Cindy Corbett Innovative model to improve and synchronize hospital discharge planning and deliver core transitional care intervention to patients at high risk for potentially preventable readmissions Secondary objective: document barriers and facilitators of successful delivery in different hospital environments My involvement Contacted during the planning process But, resource constrained Unable to collect/record some necessary data for a complete CEA Future studies will look at CUA Providence Hospital Transitional Care: Method & Results Cost-effective analysis of transitional care model Have necessary costs Cost of transitional care nurse, inpatient pharmacy, pharmacotherapy clinic, administrative costs, home health care costs, hospital care costs Have some necessary effects (in $) Compared to patients not receiving transitional care Hospital care avoided due to lower re-hospitalizations Revenues from inpatient pharmacy, pharmacotherapy clinic not recorded Result Decrease in re-hospitalizations and ED visits Total savings over 4 months = $55,752.34 (for ~ 100 patients) Savings likely to increase over time since some of the cost items were fixed costs, and some revenues were unknown Providence Hospital Transitional Care: What did I learn? Know your audience Who is using the results? What is the study perspective Results of this study to be used by hospital administrators CBA not appropriate Supplemental Nutrition Assistance Program: Objective University of Idaho Extension, Nutrition and Food Safety With Shelly Johnson, Joey Peutz and others Follow Virginia Tech report for CBA (1996) to calculate the costs and benefits of UI’s Supplemental Nutrition Assistance Program Education (delivered in Coeur d’ALene) My role A complete CBA with sensitivity analysis My involvement: Quite proactive Involved in program design – control and intervention Introduce new questions to pre- and post-program surveys to improve CBA Supplemental Nutrition Assistance Program: Method & Results Costs and benefits Collect up-to-date information on costs of health conditions/diseases averted due to improvement in nutritional intake and health behavior Cost of program delivery (compared to control group) Results Control and intervention will take place in February/March 2012 Analysis results expected middle of next year Supplemental Nutrition Assistance Program: What did I learn? Practical issues Participant enrollment Differences in incentives to participants in control and intervention groups Sample size for robust results Are cost data in literature suitable for Idaho population? Challenges Time consuming process Uncertain monetary values in CBA Uncertain QALY values in CUA Comparison of results under different situations Validity of assumptions Other questions? Contact me! firstname.lastname@example.org 509-335-7553 Reference Haddix, A.C., Teutsch, S.M., Corso, P.S. (2003). “Prevention Effectiveness: A Guide to Decision Analysis and Economic Evaluation”. Second Edition, Oxford University Press.