Use of mixed methods in the
evaluation of suicide
prevention strategies and
interventions
Stephen Platt
Qualitative research and suicide
Seminar, Cardiff University, 2 July 2007
Structure of presentation






Mixed methods research
Suicide in Scotland: trends and context
Choose Life strategy and action plan
Evaluation plan
Selected main findings
Selected recommendations
Mixed methods research




Adoption of a research strategy involving
more than one type of research method
May be mix of qualitative and quantitative
methods or mix of quantitative methods or
mix of qualitative methods
Increasing use of mixed methods strategies,
especially combining qualitative and
quantitative approaches
Many reasons why this is occurring
Why the increasing popularity of
mixed methods research? (1)




Opportunity for skills enhancement
Broadening methodological repertoire
mitigates vs. “trained incapacities” (Reiss)
Encourages thinking ‘outside the box’
Cross-national research (e.g. EU) provides
increased opportunities for mixed methods
research
Why the increasing popularity of
mixed methods research? (2)

Fits with political currency accorded to ‘practical
enquiry’ that speaks to policy/ makers and informs
practice


“Whole industry” (Brannen) of mixed methods
research created around evidence-based policy and
in policy evaluation
Increasing emphasis upon dissemination


Researchers need to communicate in “double speak”
(Brannen): technical/specialised language of
research and popular language that can easily
communicate findings/messages to ‘users’
Words as important as numbers in writing up
research
Misconceptions of ‘the other’

Quantitative attitudes  qualitative:





Too context specific
Selection of data to fit preconceptions
Unrepresentative samples/examples
Unwarranted claims
Qualitative attitudes  quantitative:




Overly simplistic
Decontextualised
Reductionist in terms of generalisations
Failing to capture subjective meaning
Rationales underlying choice of
method(s) (1)

Paradigms/philosophical assumptions



Qualitative and quantitative research seen as
intrinsically different (particularly with regard to
philosophical traditions)
But surveys not necessarily conducted on basis of
positivist assumptions and qualitative researchers
using participant observation often work in realist
tradition
Micro-level emphasises subjective interpretations.
Macro-level concerned with larger patterns/
trends and seeks structural explanations. But all
aim to understand individuals in society.
Methods need to be congruent with this quest.
Rationales underlying choice of
method(s) (2)

Pragmatics




Commonplace to argue that methods should be
appropriate (and subordinate) to research Q
But usually many research Qs. Some may be
underpinned by realist assumptions, others by
interpretevist assumptions.
And practicalities of research process may
change original intention anyway, with outcomes
of research taking precedence
Mixed methods believed/claimed to produce
“better” outcomes than reliance on single method
(pragmatic orientation linked to emphasis upon
policy/practice application)
Rationales underlying choice of
method(s) (3)

Politics



Political rationales for using mixed methods
Example: addressing condition of women in
society requires use of large-scale quantitative
data (structural) as well as in-depth qualitative
data (personal experience/perspective) in order
to understand/expose gendered inequalities
Crucial issue is the purpose to which methods are
put rather than the methods per se
Combining methods in research
process: context of justification



Stage of data analysis and interpretation
Data derived from different methods cannot be
added together to produce unitary reality
Possible outcomes when methods are combined




Corroboration (‘triangulation’) (‘same’ findings
from different methods)
Elaboration (one method exemplifies how findings
apply in particular cases)
Complementarity (findings differ but together –
synergistically – they generate insights)
Contradiction (findings from different methods
conflict)
Mixed methods designs:
key dimensions

Logic of enquiry: inductive (aimed at
discovery) or deductive (aimed at
hypothesis testing)



No one-to-one correspondence between one
particular logic of enquiry and one type of
method
(If mixed method approach is warranted)
the ordering of methods needs to be
considered: sequential or simultaneous?
How dominant is a particular method going
to be (consider scarce resources)?
Possible permutations of mixed
methods designs

Examples of
simultaneous
designs




QUAL + quan
QUAN + qual
QUAL + QUAL
QUAL + qual

Examples of
sequential designs







Qual  QUAN
QUAL  quan
Quan  QUAL
QUAN  qual
Qual  QUAL
QUAL  QUAL
QUAL  QUAL
Suicide rates across the world (2002)
Age-standardised suicide rates, by
country, 1991/93-2002/04, males
Standardised suicide rate
35
England
Scotland
N Ireland
25
20
15
10
5
0
4
/0
02
20
3
/0
01
20
2
/0
00
1
20
00
/2
99
0
19
00
/2
98
19
9
/9
97
19
6
/9
96
19
7
/9
95
19
6
/9
94
19
5
/9
93
19
4
/9
92
19
3
/9
91
19
Wales
30
Age-standardised suicide rates, by
country, 1991/93-2002/04, females
Standardised suicide rate
15
England
Scotland
N Ireland
9
6
3
0
4
/0
02
20
3
/0
01
20
2
/0
00
1
20
00
/2
99
0
19
00
/2
98
19
9
/9
97
19
6
/9
96
19
7
/9
95
19
6
/9
94
19
5
/9
93
19
4
/9
92
19
3
/9
91
19
Wales
12
Intentional self harm & undetermined
deaths, Scotland, 15+ years, 1970-2005
Suicide rate per 100,000
40
35
30
25
Male
Female
20
15
10
5
0
1970 1974 1978 1982 1986 1990 1994 1998 2002
Year
Scotland’s national suicide prevention
strategy & action plan: Choose Life






Launched in December 2002
Major element of Scottish Executive’s work
on health improvement and mental health
Plan being implemented in phases
Budget for phase 1 (April 2003-March
2006) was £12m
Additional £8.4m allocated for first two
years of phase 2 (2006-08)
Overall aim: to reduce suicide rate in
Scotland by 20% in 2013 (cf 2002)
Choose Life: national and local
infrastructure




Designated National Implementation Support Team
(NIST) co-ordinates and supports national
development and implementation
NIST’s core functions include: awareness
raising/campaigning; working with the media;
development/dissemination of information and
knowledge; and guiding and supporting local
implementation
In each local authority Choose Life action plans
have been developed by the Community Planning
Partnership (CPP)
National and local budgetary allocation
Evaluation: main objectives




Assess whether sustainable infrastructure is
being developed nationally and locally to
support achievement of Choose Life strategy
Measure and review progress towards
implementation of Choose Life milestones
Examine whether and how Choose Life is
stimulating effective forms of practice
Provide detailed recommendations to guide
the next phase of the action plan
Theory-based evaluation




Theory-driven approaches are intended to
address needs of programme implementers
Aims to articulate and test the explicit and
implicit theories that shape the design and
planned implementation of a programme
Attempts to determine
 whether a programme has been delivered as
intended
 what aspects of the programme work, for
whom and in what
circumstances/contexts/settings
Theory of change: “a systematic and cumulative
study of the links between activities, outcomes
and contexts…” (Weiss)
Theory-based evaluation:
the role of context



Context as a powerful ingredient in
programme evolution and success – not a
variable to be ‘controlled for’
Social programmes can’t be tested/rolled
out in laboratories, so getting to grips with
the interface between context and
intervention is of fundamental importance
Qualitative approaches can do this to a
degree but rarely in a way that address
concerns about programme impact
Theory-based evaluation:
the process


Through a collective/collaborative process, the
evaluator encourages programme stakeholders to
articulate at the earliest possible stage :
 The programme’s rationale
 Its intended outcomes
 The activities that will be implemented to achieve
these
 Contextual factors and their influence
The ensuing ‘theory of change’ can be used to
improve programme planning and of make
evaluation decisions
Main methods




Two electronic surveys of local coordinators
Two rounds of interviews with key
informants at national level (including
NIST)
Detailed and in-depth exploration of
theories of change in 8 selected local areas
(using interviews, workshops, observation,
documentary analysis)
Two workshops with local coordinators and
national informants
Co-ordinator surveys

Covering range of areas:










(Progress towards) Local vision for change
Progress in the development of the local infrastructure
Resource allocation and generation
Examples of innovative and effective practice
Monitoring and evaluation
Sustainability and mainstreaming
Collection of data on suicide and deliberate self-harm
Reflections on national support and on local progress
Open and closed questions
Rating scales to measure satisfaction with national
action towards achievement of milestones and to
review local implementation progress
Interviews with key informants

Key elements of semi-structured interviews:





Considering progress towards the achievement of the
national milestones set out in Choose Life
Understanding the criteria applied at the national level to
assess local implementation
Considering the quality of collaboration between the NIST
and major national agencies for the achievement of
Choose Life objectives.
Interviews with members of NIST and sample of
key national informants (e.g. ChildLine, Samaritans,
SAMH, NUJ)
Additional component: exploration of the NIST
‘story’ via individual interviews and joint workshop
Local area case studies (1)

Case studies were main vehicles for



Representative sample selected by:



exploring implementation process
identifying overall theoretical framework within
which local suicide prevention teams worked
Geographical type (rural/remote, urban and
mixed) [1º criterion]
Local suicide rate, focus on priority groups,
interaction between national and local levels, and
approaches to coordination [2º criteria]
Two fieldwork visits 6 months apart
Local area case studies (2)

Interviews with case study informants


Purposive sampling: 4 key informants involved in
decision-making process (e.g. members Choose
Life partnership responsible for priority setting)
and 2 representatives involved in Choose Life
funded activities
16 project activities selected for more
detailed examination (2 per case study site)



Intended to provide representative selection
across Choose Life priority groups and objectives
Selection negotiated with coordinators
Interviews conducted with project representative,
usually project lead/manager
Local area case studies (3)

Observational activities and collection of
documentation



Observation of Choose Life events (usually
partnership meetings but also training days,
evaluation days and practitioner fora)
Key documents included: minutes of Choose Life
partnership meetings, locality reports on progress,
reports of previous needs assessment
Evaluation team aimed to understand
theories of change at overall programme
level as well as at project level
Local area case studies (4)

Changes to approach




Workshops replaced individual interviews with
stakeholders at second fieldwork visit
Provided opportunity for joint testing of, and
reflection on, local area theories of change
Local participants encouraged to assess progress
towards Choose Life objectives and milestones,
and how this could be demonstrated
Problems created as result of mixed levels
and responsibilities of participants (e.g.
strategic versus operational)
National workshops (1)


Two workshops one year apart brought
together evaluation team, NIST, local
coordinators and several key stakeholders
1st workshop aimed to:



develop understanding of the different models of
national and local actions and activities being put
in place
explore the evolving relationship between the
approaches of the centre and of local areas
identify ways in which progress could be
measured.
National workshops (2)

2nd workshop aimed to:



Review progress and learning in relation to
objectives of Choose Life
Test out key themes emerging from evaluation
Identify future priorities for development, support
required, measures of progress and outcome, and
implications for information collection and
research
Data analysis (1)





Continuous (iterative) process throughout study
Evaluation database designed and used to store (and
retrieve) data on all 32 local areas
Findings from each element of data collection (case
studies, workshops, national interviews, surveys)
written up in detailed reports which were then used
for comparative analysis
Data analysed according to predefined themes (e.g.
sustainability, partnerships) …
… and themes also developed from inductive
analysis conducted at each phase of evaluation
Data analysis (2)

Analytic framework developed to guide team
through the evaluation


This was expanded and changed according to
themes emerging from data
Analysis primarily drew on ‘charting’ method of
systematically handling complex datasets by
drawing out dimensions relating to each theme
across all ‘cases’
Evaluation: main findings





Sustainable infrastructures for
implementation
Allocation and use of resources
Innovative practice and use of evidence
Sustainability
Decision making processes and learning
Sustainable infrastructures for
implementation



Demonstrable progress made by NIST
CPP successes … but not as evident in
less ‘mature’ partnerships and in
engaging with clinical services
Various models of local coordination
developed


Preference for a dedicated (full-time)
coordination post …
… but evaluation unable to demonstrate the
superiority of this model
Allocation and use of resources





CPPs have attracted considerable additional
investment at local level and in-kind contribution …
… but areas have not been equally successful in
raising additional funding …
… and there has been unnecessary duplication of
effort at local level
Choose Life has stimulated a considerable amount of
activity relating to self-harm …
… but local areas have different understandings of
‘high risk’ suicidal behaviour and have adopted
different responses to address the problem
Innovative practice and use of
evidence



Many examples of locally defined
innovative practice …
… and multiple sources of information and
evidence used to inform local planning and
activity …
… but research rarely used systematically
Sustainability


NIST identified several achievements in
building a sustainable infrastructure for
suicide prevention
At local level, most success achieved in
mainstreaming training activities
Decision making processes
and learning






Local stakeholder consultation: key approach to
set implementation priorities
Short timescale to develop first action plan was
a major challenge
National support for learning has been delivered
through diverse routes
NIST has highlighted a strong commitment to
evaluation …
… but a national framework for evaluation
remains to be completed
In local areas different levels of priority and
attention have been attached to evaluation.
Recommendations:
mainstreaming at national level




Incorporate Choose Life objectives and
priorities into other policy streams/initiatives
Involve clinical services in population-based
suicide prevention activities
Involve national voluntary sector
organisations in awareness raising and
campaigning
Engage in purposive innovation to test out,
evaluate, learn and implement
Recommendations:
mainstreaming at local level



Using intelligence from a range of sources,
as tools in planning for sustainability
Building in mechanisms to track and review
progress towards objectives across policy
areas
More focused targeting of action is required
Recommendations: self-harm

More consideration to be given in phase 2
to the integration of self-harm into Choose
Life


The strategy should continue to encompass
high risk self-harm …
… but the less ‘serious’ component of selfharm cannot be ignored
Recommendations: CPP


The CPP remains the most appropriate vehicle
for developing strategy and overseeing delivery
in relation to Choose Life at the local level
But its limitations should be recognised



Need to examine the necessary partnerships that
have yet to be put in place
Priority should be given to establishing/building on
effective links with clinical & drug/alcohol services
NIST should continue to work closely with CPPs to
ensure that Choose Life budgets are fully spent on
suicide prevention activities
Recommendations: central
coordination body



Some type of central coordination body will
continue to be required in the immediate future
Key tasks: provide national oversight, assess
and support performance and ensure
accountability at local level, promote learning
and effective knowledge transfer, and coordinate action.
There should be a review of how the central
coordinating function is delivered and where it
is situated
Choose Life evaluation (first phase):
research team
University of Edinburgh
Stephen Platt (RUHBC)
Emma Halliday (RUHBC)
Margaret Maxwell (General Practice)
Scottish Development Centre for Mental Health
Joanne McLean
Allyson McCollam
Amy Woodhouse
London School of Economics
Dave McDaid (Health & Social Care)
Glasgow University
Mhairi Mackenzie (Public Health & Health Policy)
Avril Blamey (Public Health & Health Policy)
Descargar

Slide 1