Evidence Check: Title page Commissioning better research syntheses Sally Redman, Danielle Campbell, Gai Moore Sax Institute Mission To improve health and increase the quality and performance of health services and programs by: building excellent policy and practice focused research and increasing the impact of this research on health policy, programs and services Is research used to inform policy in NSW? • Policy Maker Survey - 38 senior policy makers from the NSW Department of Health and Area Health Services • Researcher Survey - 41 senior researchers from 29 Universities and research centres across NSW Proportion perceiving a HIGH need to improve the use of research in policy 90 80 70 60 % 50 40 30 20 10 0 Policy makers Researchers Source: Policy Maker Survey and Researcher Survey Perceived facilitators of evidence in policy NSW policy makers Better access to researchers Better access to research/reviews Policy-relevant research Innvaer et al, 2002 (Review of 24 interview studies of health policy makers) Personal contact with researchers Timely relevance Inclusion of summaries & recommendations Increasing the impact of research: Some goals of the Sax Institute Increase research in areas where NSW Health has knowledge priorities Give policy makers more timely and effective access to research findings Bring policy makers and researchers closer together to more easily understand each others expertise and priorities Identify research findings that are of potential importance to policy and assist researchers to package and communicate their findings in ways that can influence the policy agenda. Ability to find research to inform policy in last 12 months 68% found brief research summaries difficult or impossible to find 84% found more systematic reviews difficult or impossible to find Difficulty in finding research summaries nominated as one of greatest barriers to using evidence in policy Policy Maker Survey What goes wrong? Policy makers… Reviews are: Too slow Too focused on researcher’s interests rather than policy question Too time consuming to commission Too long with key messages buried Focused on need for more research rather than best summary of current position Policy makers’ feedback about research reviews “I don’t think we use reviews much at all – people don’t even think of it” “A couple of times either I didn’t specify clearly enough what I wanted or the specs changed and what they produced...was awkward for the Department” “Also there’s the timing issue, you don’t want to have to put a lot of effort into a tendering process that won’t get you a product when you really need it” What goes wrong? Researchers.. Commissioned reviews: Are often buried and cannot be published Are frustrating because policy makers do not know what they want and change their mind Compete with research through urgent tender requests Prop’n of researchers perceiving that little value is placed on efforts to get research into policy 90 80 70 60 50 % 40 30 20 10 0 When funding bodies assess track record When universities consider employment/promotion Source: Researcher Survey Evidence Check To enable policy makers to more readily access high quality reviews of research suited to their needs Will also: • Encourage researchers to undertake reviews for government • Result in greater knowledge of research expertise in NSW among policy makers Evidence Check Identification of expertise Review Commissioning Tool RADAR Partnership and production guidelines Knowledge Brokers Preferred Provider Rules Diagnosis Evidence Check: Diagnosis Evidence Check Commissioning Tool: Series of structured questions to assist policy makers to analyse and make explicit their needs Knowledge brokers: Individuals with experience in policy AND research - trained and supported to work between researchers and policy makers Commissioning Tool 1. Background and purpose of the policy • How will the review be used, at what stage in the policy process, and by whom? • What is the political context? • What risks are associated with the policy issue? 2. Questions to guide the review • What are the specific questions that need to be answered (define: intervention, target population and outcome)? Commissioning Tool 3. Timeframe and funding • Is an answer required immediately, in the short term, or in the long term? • What funds are available? 4. Depth of analysis of evidence • Systematic review or a summary of key findings? • Are strength and recency of evidence important? 5. Format of final product • How long should the report be? • What level of language is appropriate? Case Study: Advanced Care Planning Evidence need initially defined by policy makers Review questions agreed after meeting with broker Evidence to support a policy approach to comprehensive advanced care planning (ACP) for end-of-life choices Is there evidence of interventions that support a communitywide approach to ACP for end-of-life care? What types of interventions that support a communitywide approach to ACP have been implemented and proven effective? Is there evidence of interventions for other sensitive issues that could provide lessons to support a community-wide approach to ACP? Findings from scoping exercise by broker A recent meta-analysis provides a basis for identifying key features of interventions, and other reviews of ACP and palliative care may support these findings. Reviews of approaches to other issues (eg domestic violence) may also describe relevant strategies Recommendation Commission researcher to: identify relevant high-quality metaanalyses and reviews; identify key features of effective interventions in community settings; and develop recommendations for a policy and program approach in NSW Case Study: Type 2 Diabetes Prevention Evidence need initially defined by policy makers Review questions agreed after meeting with broker Evidence to support the development of strategies to reduce the incidence of type 2 diabetes in CALD communities Are there CALD groups in NSW with increased incidence of type 2 diabetes or increased prevalence of risk factors? What evidence exists for interventions What are the factors that are likely to be that may contribute to effective in reducing higher risk levels in type 2 diabetes risk specific CALD groups? factors in specific CALD groups? Findings from scoping exercise by broker Risk factor prevalence and disease incidence data is available from government bodies and diabetes associations. Data on factors that influence risk and incidence in specific CALD groups can be derived from these sources and the peer reviewed literature Recommendation Commission researcher to undertake a review and critical analysis of epidemiological data on risk factor and disease prevalence, and evidence on the most effective prevention approaches for specific CALD groups Evidence Check Identification of expertise Review Commissioning Tool RADAR Partnership and production guidelines Knowledge Brokers Preferred Provider Rules Diagnosis Evidence Check: Identifying expertise RADAR: Database 38 research centres and Universities that are members of the Sax Institute used to generate a database of expertise Selecting reviewers: Preferred provider Competitive tender avoided by decision rules for selecting individuals with best expertise based on discipline base (eg epidemiology, health economics), content knowledge (i.e. diseases, settings, interventions, and populations), and review experience Prop’n of researchers interested in commissioned research reviews 90 80 70 60 50 % 40 30 20 10 0 Funded by government High interest in High interest in joining to conduct review in conducting register of reviewers last 2 years commissioned reviews Source: Researcher survey Evidence Check Identification of expertise Review Commissioning Tool RADAR Partnership and production guidelines Knowledge Brokers Preferred Provider Rules Diagnosis Evidence check: the review Range of review types depending on policy issue (eg systematic review, review of evaluations, ‘headline’ review) Guidelines for reviewers (eg CHSRF 1:3:25 format) Translation process by knowledge broker Feedback from the pilot study Provided a valuable framework for critical thinking and discussion among policy makers Helped policy makers move from a general issue to specific questions Helped policy makers anticipate the potential challenges and risks that could result in a product that does not meet policy needs Outcomes so far 7 reviews using Evidence Check commissioned so far by NSW Health in broad range of areas: e.g. health workforce, diabetes prevention, HIV services, obesity, population health priorities Rapid increase in demand over the past few months 2 groups that used Evidence Check during pilot/early stages have returned with new project requests Case Study: Health Workforce Evidence need initially defined by policy makers Evidence to support improved workforce planning and education for future health needs especially overseas trained doctors Review questions agreed after meeting with broker What are the characteristics of newly appointed nonspecialist doctors trained overseas or in Australia? Findings from scoping exercise by broker Little evidence exists to clarify: the extent to which NSW doctors are supported and trained; community perceptions of types of doctors; barriers and facilitators of effective workforce participation; and flow-on effects of new training/recruitment methods. Recommendation What skills and knowledge gaps can be identified in newly appointed doctors trained overseas and in Australia ? How are these doctors perceived by the community and the health system? Commission researcher to provide an objective, descriptive and analytical profile of doctors practicing in NSW who have been trained in Australian medical schools or overseas.