Patient-Reported Health
Outcome Measures:
Overview and Application
in Clinical Trials Research
Marie Martin, PhD
Vertex Pharmaceuticals, Cambridge
March 24, 2009
1
A Show of Hands:
Who Has Experience with PROs?
Not A Statistical Talk
Statistics
Overview of PROs in Clinical Trials
Research
Overview of PROs in Clinical Trials
Research
Overview of PROs in Clinical Trials
Research
Overview of PROs in Clinical Trials
Research
Fast Pace
Try to hold questions til the end
My goal for you

My aim:
 for you to have a sound general sense of
what a PRO is and its role in clinical trials
research
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
What is the goal of medical treatment?
What is a PRO?
Why use PRO measures?
How do PRO measures complement traditional
clinical outcome measures?
What characterizes PRO measures?
 How do PROs differ?
What characterizes a well-constructed PRO
instrument?
 How are PRO instruments developed?
What is the process necessary to achieve a PRO
label?
 Putting it all together
What is the role of PROs in clinical trials research?
Review and conclusions
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
What is the goal of medical treatment?
What is a PRO?
Why use PRO measures?
How do PRO measures complement traditional
clinical outcome measures?
What characterizes PRO measures?
 How do PROs differ?
What characterizes a well-constructed PRO
instrument?
 How are PRO instruments developed?
What is the process necessary to achieve a PRO
label?
 Putting it all together
What is the role of PROs in clinical trials research?
Review and conclusions
What is the goal
of medical
treatment?
“Begin at the beginning
and go on till you come to the end;
then stop.”
Lewis Carroll
What is the goal of medical treatment?

What is an outcome?
 That which represents the goal of all
treatments: “to make the patient better”
Fries, 1993

“The best measure of quality is not how well
or how frequently a medical service is given,
but how closely the result approaches the
fundamental objectives of prolonging life,
relieving distress, restoring function, and
preventing disability.” Lembcke, 1952
What is the goal of medical treatment?

What is an outcome?
 That which represents the goal of all
treatments: “to make the patient better”
Fries, 1993

“The best measure of quality is not how well
or how frequently a medical service is given,
but how closely the result approaches the
fundamental objectives of prolonging life,
relieving distress, restoring function, and
preventing disability.” Lembcke, 1952
What concepts (outcomes) measure
the goal of medical treatment?

According to the Agency for Healthcare Research
and Quality, outcomes research is:


“A multidisciplinary endeavor that assesses the end
results of health services that are important to the
people who experience them.” (AHRQ, Clancy & Eisenberg, 1997)
According to the World Health Organization,
health is:

“a state of complete physical, mental, and social wellbeing and not merely the absence of disease or
infirmity.” (http://www.who.int/mediacentre/factsheets/fs220/en/)
What concepts (outcomes) measure
the goal of medical treatment?

According to the Agency for Healthcare Research
and Quality, outcomes research is:


“A multidisciplinary endeavor that assesses the end
results of health services that are important to the
people who experience them.” (AHRQ, Clancy & Eisenberg, 1997)
According to the World Health Organization,
health is:

“a state of complete physical, mental, and social wellbeing and not merely the absence of disease or
infirmity.” (http://www.who.int/mediacentre/factsheets/fs220/en/)
What concepts (outcomes) measure
the goal of medical treatment?

According to the Agency for Healthcare Research
and Quality, outcomes research is:


“A multidisciplinary endeavor that assesses the end
results of health services that are important to the
people who experience them.” (AHRQ, Clancy & Eisenberg, 1997)
According to the World Health Organization,
health is:

“a state of complete physical, mental, and social wellbeing and not merely the absence of disease or
infirmity.” (http://www.who.int/mediacentre/factsheets/fs220/en/)
Here we see ads for rheumatoid arthritis treatment that lead to
improvement of physical function, activities of daily living, and
health-related quality of life.
Here we see ads for rheumatoid arthritis treatment that lead to
improvement of physical function, activities of daily living, and
health-related quality of life.
Orencia Label for RA:
“ORENCIA is indicated
for … improving physical
function in adult patients
with moderately to
severely active
rheumatoid arthritis …”
“Bird watching
was my passion.
RA took that
away. Now,
promising new
treatments have
changed that. I’m
doing what I love
again, and it’s
wonderful.”
“Bird watching
was my passion.
RA took that
away. Now,
promising new
treatments have
changed that. I’m
doing what I love
again, and it’s
wonderful.”
Humira label for
rheumatoid arthritis:
“Humira is indicated
for … improving
physical function in
adults with
moderately to
severely active
rheumatoid arthritis”
Through the use of
patient-reported
outcomes, both Humira
and Orencia were able
to arrive at label claims
that also address
improvements in
physical function and
health-related quality
of life.
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
What is the goal of medical treatment?
What is a PRO?
Why use PRO measures?
How do PRO measures complement traditional
clinical outcome measures?
What characterizes PRO measures?
 How do PROs differ?
What characterizes a well-constructed PRO
instrument?
 How are PRO instruments developed?
What is the process necessary to achieve a PRO
label?
 Putting it all together
What is the role of PROs in clinical trials research?
Review and conclusions
What is a PRO?
What is a Patient-Reported Outcome?
An “Umbrella” term
PRO
Outcomes Reported by Patients
Bodily pain
Symptoms & their impact on functioning
Perceptions of one’s health
Satisfaction w Tx
Disease impact on Functioning
(physical, social, emotional)
Quality of Life
Outcomes (QoL)
Depression
Energy
Anxiety
Enjoyment
Positive Well-being
Impact of tx
on functioning
Bradley, 2006
What is a Patient-Reported
Outcome (PRO) according to the FDA?

FDA: “A PRO is a measurement of any aspect of a
patient’s health status that comes directly from the
patient


(ie, without the interpretation of the patient’s responses by a
physician or anyone else).”
PROs are patient reports of a health condition,
EMPHASIS
is on patient’s
voice:
treatment, feeling,
or function,
including




symptoms
Responses
intensity
to come
and frequency
directly from the patient
disease impact on functioning
satisfaction with treatment
health-related quality of life
Source: FDA guidance, 2006
The Challenge

To measure something that you can’t
observe directly  so you ask the patient
Impact of illness on
ability to function
Pain intensity
Fatigue / Vitality
Social Function
Physical Function
Quality of Life
Ability to concentrate
at work due to health
Worry about
one’s health
Satisfaction w Tx
Anxiety
Positive Well-being
Impact of symptoms on quality of life
Source: FDA guidance, 2006
The Challenge

To measure something that you can’t
observe directly  so you ask the patient
Impact of illness on
ability to function
Pain intensity
Fatigue / Vitality
Social Function
Physical Function
Quality of Life
Ability to concentrate
at work due to health
Worry about
one’s health
Satisfaction w Tx
Anxiety
Positive Well-being
Impact of symptoms on quality of life
Source: FDA guidance, 2006
The Method

Must be
 Systematic
 Reliable
 Replicable
 Based on good science
Source: FDA guidance, 2006
Examples
(Preview)
Examples
(Preview)
An introductory
“taste” to give you
a feel for PROs.
What concepts do PRO
instruments measure? (Examples)
What concepts do PRO
instruments measure? (Examples)

Overall health status
 In general, would you say your health is:


Symptoms and their impact on functioning
 How much did your pain limit your usual activities or enjoyment of everyday life?


{none of the time, a little of the time, some of the time, most of the time, all of the time}
Satisfaction with treatment or preference for treatment
 How satisfied or dissatisfied are you with the way the medication relieves your
symptoms?


{none of the time, a little of the time, some of the time, most of the time, all of the time}
Health perceptions
 Over the past week, how much did you worry about your health?


{not at all, a little, moderately, quite a lot, extremely}
Disease impact on functioning (physical, psychological, or social)
 During the past four weeks, how much of the time has your physical health or
emotional problems interfered with your social activities (like visiting friends,
relatives, etc.)?


{excellent, very good, good, fair, poor}
{scaled on a seven point bipolar scale from 'Extremely Satisfied' to 'Extremely Dissatisfied' }
Adherence to medical treatment
 In the past four weeks, I have skipped my medication

{none of the time, a little of the time, some of the time, most of the time, all of the time}
What concepts do PRO
instruments measure? (Examples)

Overall health status
 In general, would you say your health is:


Symptoms and their impact on functioning
 How much did your pain limit your usual activities or enjoyment of everyday life?


{none of the time, a little of the time, some of the time, most of the time, all of the time}
Satisfaction with treatment or preference for treatment
 How satisfied or dissatisfied are you with the way the medication relieves your
symptoms?


{none of the time, a little of the time, some of the time, most of the time, all of the time}
Health perceptions
 Over the past week, how much did you worry about your health?


{not at all, a little, moderately, quite a lot, extremely}
Disease impact on functioning (physical, psychological, or social)
 During the past four weeks, how much of the time has your physical health or
emotional problems interfered with your social activities (like visiting friends,
relatives, etc.)?


{excellent, very good, good, fair, poor}
{scaled on a seven point bipolar scale from 'Extremely Satisfied' to 'Extremely Dissatisfied' }
Adherence to medical treatment
 In the past four weeks, I have skipped my medication

{none of the time, a little of the time, some of the time, most of the time, all of the time}
What concepts do PRO
instruments measure? (Examples)

Overall health status
 In general, would you say your health is:


Symptoms and their impact on functioning
 How much did your pain limit your usual activities or enjoyment of everyday life?


{none of the time, a little of the time, some of the time, most of the time, all of the time}
Satisfaction with treatment or preference for treatment
 How satisfied or dissatisfied are you with the way the medication relieves your
symptoms?


{none of the time, a little of the time, some of the time, most of the time, all of the time}
Health perceptions
 Over the past week, how much did you worry about your health?


{not at all, a little, moderately, quite a lot, extremely}
Disease impact on functioning (physical, psychological, or social)
 During the past four weeks, how much of the time has your physical health or
emotional problems interfered with your social activities (like visiting friends,
relatives, etc.)?


{excellent, very good, good, fair, poor}
{scaled on a seven point bipolar scale from 'Extremely Satisfied' to 'Extremely Dissatisfied' }
Adherence to medical treatment
 In the past four weeks, I have skipped my medication

{none of the time, a little of the time, some of the time, most of the time, all of the time}
What concepts do PRO
instruments measure? (Examples)

Overall health status
 In general, would you say your health is:


Symptoms and their impact on functioning
 How much did your pain limit your usual activities or enjoyment of everyday life?


{none of the time, a little of the time, some of the time, most of the time, all of the time}
Satisfaction with treatment or preference for treatment
 How satisfied or dissatisfied are you with the way the medication relieves your
symptoms?


{none of the time, a little of the time, some of the time, most of the time, all of the time}
Health perceptions
 Over the past week, how much did you worry about your health?


{not at all, a little, moderately, quite a lot, extremely}
Disease impact on functioning (physical, psychological, or social)
 During the past four weeks, how much of the time has your physical health or
emotional problems interfered with your social activities (like visiting friends,
relatives, etc.)?


{excellent, very good, good, fair, poor}
{scaled on a seven point bipolar scale from 'Extremely Satisfied' to 'Extremely Dissatisfied' }
Adherence to medical treatment
 In the past four weeks, I have skipped my medication

{none of the time, a little of the time, some of the time, most of the time, all of the time}
What concepts do PRO
instruments measure? (Examples)

Overall health status
 In general, would you say your health is:


Symptoms and their impact on functioning
 How much did your pain limit your usual activities or enjoyment of everyday life?


{none of the time, a little of the time, some of the time, most of the time, all of the time}
Satisfaction with treatment or preference for treatment
 How satisfied or dissatisfied are you with the way the medication relieves your
symptoms?


{none of the time, a little of the time, some of the time, most of the time, all of the time}
Health perceptions
 Over the past week, how much did you worry about your health?


{not at all, a little, moderately, quite a lot, extremely}
Disease impact on functioning (physical, psychological, or social)
 During the past four weeks, how much of the time has your physical health or
emotional problems interfered with your social activities (like visiting friends,
relatives, etc.)?


{excellent, very good, good, fair, poor}
{scaled on a seven point bipolar scale from 'Extremely Satisfied' to 'Extremely Dissatisfied' }
Adherence to medical treatment
 In the past four weeks, I have skipped my medication

{none of the time, a little of the time, some of the time, most of the time, all of the time}
What concepts do PRO
instruments measure? (Examples)

Overall health status
 In general, would you say your health is:


Symptoms and their impact on functioning
 How much did your pain limit your usual activities or enjoyment of everyday life?


{none of the time, a little of the time, some of the time, most of the time, all of the time}
Satisfaction with treatment or preference for treatment
 How satisfied or dissatisfied are you with the way the medication relieves your
symptoms?


{none of the time, a little of the time, some of the time, most of the time, all of the time}
Health perceptions
 Over the past week, how much did you worry about your health?


{not at all, a little, moderately, quite a lot, extremely}
Disease impact on functioning (physical, psychological, or social)
 During the past four weeks, how much of the time has your physical health or
emotional problems interfered with your social activities (like visiting friends,
relatives, etc.)?


{excellent, very good, good, fair, poor}
{scaled on a seven point bipolar scale from 'Extremely Satisfied' to 'Extremely Dissatisfied' }
Adherence to medical treatment
 In the past four weeks, I have skipped my medication

{none of the time, a little of the time, some of the time, most of the time, all of the time}
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
What is the goal of medical treatment?
What is a PRO?
Why use PRO measures?
How do PRO measures complement traditional
clinical outcome measures?
What characterizes PRO measures?
 How do PROs differ?
What characterizes a well-constructed PRO
instrument?
 How are PRO instruments developed?
What is the process necessary to achieve a PRO
label?
 Putting it all together
What is the role of PROs in clinical trials research?
Review and conclusions
Why use PRO
measures?
Why use PRO measures?

Because some treatment effects are known only
to the patient


Because we need formal, reliable assessments
for clinical trials (vs informal interview)



Eg, pain intensity and pain relief
Instead of informally asking questions like “Is your pain less
intense?” or “Are you able to concentrate at work?” PROs
formalize this assessment w a method that yields measures that
are reliable and valid
Well-constructed PROs measure outcomes with precision and
accuracy
Because patient perspective augments what is
known about the product

In a clinical trial, PROs can augment what is known about the
product, as clinical measures do not necessarily correspond to
meaningful improvements in how the patient functions or feels
Source: FDA guidance, 2006
Why use PRO measures?

Because some treatment effects are known only
to the patient


Because we need formal, reliable assessments
for clinical trials (vs informal interview)



Eg, pain intensity and pain relief
Instead of informally asking questions like “Is your pain less
intense?” or “Are you able to concentrate at work?” PROs
formalize this assessment w a method that yields measures that
are reliable and valid
Well-constructed PROs measure outcomes with precision and
accuracy
Because patient perspective augments what is
known about the product

In a clinical trial, PROs can augment what is known about the
product, as clinical measures do not necessarily correspond to
meaningful improvements in how the patient functions or feels
Source: FDA guidance, 2006
Why use PRO measures?

Because some treatment effects are known only
to the patient


Because we need formal, reliable assessments
for clinical trials (vs informal interview)



Eg, pain intensity and pain relief
Instead of informally asking questions like “Is your pain less
intense?” or “Are you able to concentrate at work?” PROs
formalize this assessment w a method that yields measures that
are reliable and valid
Well-constructed PROs measure outcomes with precision and
accuracy
Because patient perspective augments what is
known about the product

In a clinical trial, PROs can augment what is known about the
product, as clinical measures do not necessarily correspond to
meaningful improvements in how the patient functions or feels
Source: FDA guidance, 2006
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
What is the goal of medical treatment?
What is a PRO?
Why use PRO measures?
How do PRO measures complement traditional
clinical outcome measures?
What characterizes PRO measures?
 How do PROs differ?
What characterizes a well-constructed PRO
instrument?
 How are PRO instruments developed?
What is the process necessary to achieve a PRO
label?
 Putting it all together
What is the role of PROs in clinical trials research?
Review and conclusions
How do PRO
measures
complement
traditional clinical
outcome measures?
WILSON-CLEARY MODEL OF HEALTH OUTCOMES:
Where do PROs fit in a greater model of health outcomes?
Characteristics of Individual
Biological and
Physiological
Variables
Symptom
Status
Functional
Status
General Health
Perceptions
Characteristics of Environment
Wilson & Cleary JAMA (1995); Adapted from Revicki, 2006
Quality of Life
WILSON-CLEARY MODEL OF HEALTH OUTCOMES:
Where do PROs fit in a greater model of health outcomes?
These are
concepts
captured via
PROs.
Biological and
Physiological
Variables
Characteristics of Individual
Symptom
Status
Functional
Status
General Health
Perceptions
Characteristics of Environment
Wilson & Cleary JAMA (1995); Adapted from Revicki, 2006
Quality of Life
WILSON-CLEARY MODEL OF HEALTH OUTCOMES:
Where do PROs fit in a greater model of health outcomes?
Characteristics of Individual
Biological and
Physiological
Variables
Symptom
Status
Functional
Status
General Health
Perceptions
Quality of Life
Characteristics of Environment
From CELL to INDIVIDUAL to the interaction of person as a MEMBER OF SOCIETY
-- Impact of DISEASE and of TREATMENT can be captured at each of these boxes
Wilson & Cleary JAMA (1995); Adapted from Revicki, 2006
Because OUTCOMES:
impact of DISEASE and
WILSON-CLEARY MODEL OF HEALTH
of TREATMENT can be captured
Where do PROs fit in a greater model
ofathealth
outcomes?
each of these boxes,
measuring such impact
enhances assessment of benefit
Characteristics of Individual
of tx by only clinical measures.
Biological and
Physiological
Variables
Symptom
Status
Functional
Status
General Health
Perceptions
Quality of Life
Characteristics of Environment
From CELL to INDIVIDUAL to the interaction of person as a MEMBER OF SOCIETY
-- Impact of DISEASE and of TREATMENT can be captured at each of these boxes
Wilson & Cleary JAMA (1995); Adapted from Revicki, 2006
Why use PRO measures?
Revisited


Because concepts measured by PROs
complement traditional clinical outcome
measures for a more comprehensive assessment
of impact of disease and treatment
Because with strategic planning, including PROs
in a clinical trial can result into successful PRO
label claims
Why use PRO measures?
Revisited


Because concepts measured by PROs
complement traditional clinical outcome
measures for a more comprehensive assessment
of impact of disease and treatment
Because with strategic planning, including PROs
in a clinical trial can result into successful PRO
label claims
Why use PRO measures?
Revisited


Because concepts measured by PROs
complement traditional clinical outcome
measures for a more comprehensive assessment
of impact of disease and treatment
Because with strategic planning, including PROs
in a clinical trial can result into successful PRO
label claims
 Including a successful publication strategy
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
What is the goal of medical treatment?
What is a PRO?
Why use PRO measures?
How do PRO measures complement traditional
clinical outcome measures?
What characterizes PRO measures?
 How do PROs differ?
What characterizes a well-constructed PRO
instrument?
 How are PRO instruments developed?
What is the process necessary to achieve a PRO
label?
 Putting it all together
What is the role of PROs in clinical trials research?
Review and conclusions
What
characterizes
PRO instruments?
How do PROs differ?
What characterizes PRO instruments?

PROs differ on:

Item characteristics




Concepts measured






Type of item
Number of response choices of each item
Number of items per concept measured
Number of concepts measured in a single instrument
Disease specific vs Generic measures
Complexity of concepts measured
Association of concept w bio-physiologic mechanisms
Recall period
Method of administration
What characterizes PRO instruments?

PROs differ on:

Item characteristics




Concepts measured






Type of item
Number of response choices of each item
Number of items per concept measured
Number of concepts measured in a single instrument
Disease specific vs Generic measures
Complexity of concepts measured
Association of concept w bio-physiologic mechanisms
Recall period
Method of administration
What characterizes PRO instruments?

PROs differ on:

Item characteristics




Concepts measured






Type of item
Number of response choices of each item
Number of items per concept measured
Number of concepts measured in a single instrument
Disease specific vs Generic measures
Complexity of concepts measured
Association of concept w bio-physiologic mechanisms
Recall period
Method of administration
Type of Items & Response Options

Items in PRO instruments may be:
 VAS (Visual Analog Scale)
 Anchored or categorized VAS
 Likert scale
 Rating scale
 Pictorial scale
 Checklist
Source: FDA Guidance, 2006
Type of Items: VAS
(Visual Analogue Scale)
VAS: A line of fixed length (usually 100 mm) w words that anchor
the scale at the extremes w/o words describing intermediate
positions. Subjects are to place a mark on the line that corresponds
to their perceived state.
Example: worry about one’s health (below)
Please place a mark on the line below to describe, during the past 4
weeks, how often you have worried about your health.
None of
the
time
All of
the
time
Type of Items: Anchored VAS
Anchored or categorized VAS: A VAS w one or more
intermediate marks and words describing intermediate position (eg,
half-way btw ends of the scale)
Example: current pain intensity (below)
Please place a mark on the line below to describe how intense your
pain is now.
No pain
at all
Moderate
Pain
Severe
Pain
Unbearable
pain
Type of Items: Likert
Likert scale: Refers to an item with response options that are an
ordered set of discrete statements—usually, the response options
form a bipolar (from positive to negative) set of response categories.
Example: the SF-36’s “Role-Physical” (role limitations due to
physical health) scale (below)
D u rin g th e p a s t fo u r w e e k s , h o w m u c h o f th e tim e h a v e yo u h a d a n y o f th e
fo llo w in g p ro b le m s w ith yo u r w o rk o r o th e r re g u la r d a ily a c tiv itie s a s a
re s u lt o f yo u r p h ys ic a l h e a lth ?
M ost
A ll o f
o f th e
th e tim e tim e
Som e
o f th e
tim e
A little
o f th e
tim e
N one
o f th e
tim e
a . C u t d o w n o n th e a m o u n t o f tim e
yo u s p e n t o n w o rk o r o th e r a c tivitie s
b . A c c o m p lis h e d le s s th a n yo u w o u ld
1
2
3
4
5
lik e
c . W e re lim ite d in th e k in d o f w o rk o r
1
2
3
4
5
o th e r a c tivitie s
d . H a d d iffic u lty p e rfo rm in g th e w o rk
1
2
3
4
5
1
2
3
4
5
o r o th e r a c tivitie s (fo r e x a m p le , it to o k
e x tra e ffo rt)
Source: Version 2 of the SF-36 Health Survey
Type of Items: Rating
Rating scale: A set of numerical categories from which to
choose the response that best describes one’s state or experience.
The ends of the rating scale are anchored w words but the categories
do not have labels.
Example: the Fatigue Severity Scale (below)
Read each statement and circle a number from 1 to 7, based on how accurately it reflects your
condition during the past week and the extent to which you agree or disagree that the
statement applies to you.
During the past week, I have found that:
1. My motivation is lower when I am fatigued.
2. Exercise brings on my fatigue.
3. I am easily fatigued.
4. Fatigue interferes with my physical functioning.
5. Fatigue causes frequent problems for me.
6. My fatigue prevents sustained physical functioning.
Disagree
1
2
1
2
1
2
1
2
1
2
1
2
3
3
3
3
3
3
4
4
4
4
4
4
5
5
5
5
5
5
Agree
6
7
6
7
6
7
6
7
6
7
6
7
Type of Items: Pictorial
Pictorial scale: A set of pictures applied to any of the types
of response options (often used in pediatrics and cognitively
impaired pts)
EXAMPLE: the Wong-Baker FACES scale (below) applied to a
Likert scale
Type of Items: Pictorial
EXAMPLE:
These are 3
pictorial scales
of exhaustion,
chest tightness,
and throat
constriction of
the Dalhousie
dyspnea
(breathlessness)
scale for
children and
adolescents.
Source: McGrath et al.,
BMC Pediatrics, 2005
Source: http://www.biomedcentral.com/1471-2431/5/33
Type of Items: Checklist
Checklist: Provide a simple choice btw a limited set of options (eg,
Yes, No, Don’t know). Some may only ask subjects to mark if the item
statement is true. Checklists are reviewed for completeness and
nonredundancy.
Example: the 35-item Pediatric Symptom Checklist (below) used to
screen 4-5 yr olds for psychosocial dysfunction.
Please mark under the heading that best describes your child:
Never
1
2
3
4
5
6
7
8
Sometimes
Often
Complains of aches and pains
Spends more time alone
Tires easily, has little energy
Fidgety, unable to sit still
Has trouble with teacher
Less interested in school
Acts as if driven by a motor
Daydreams too much
Source: http://www.brightfutures.org/mentalhealth/pdf/professionals/ped_sympton_chklst.pdf
What characterizes PRO instruments?

PROs differ on:

Item characteristics




Concepts measured






Type of item
Number of response choices of each item
Number of items per concept measured
Number of concepts measured
Disease specific vs Generic measures
Complexity of concepts measured
Association of concept w bio-physiologic mechanisms
Recall period
Method of administration
Number of Items Per Concept:
How many items are utilized to capture an underlying concept?

PRO instruments vary in total number of items
per concept measured:

Single item for single concept


Multiple items for single concept


Eg, a VAS to measure pain intensity or pictorial pain
intensity scale
Eg, the Fatigue Severity Scale (9 items)
Multiple items for multiple domains within a concept

Eg, the HAQ (Health Assessment Questionnaire) assessing
8 domains (or concepts) of physical function (disability), with
2+ items each

{Dressing/grooming, rising, eating, walking, hygiene, reaching,
gripping, getting in/out of a car}
Number of Items Per Concept:
How many items are utilized to capture an underlying concept?

PRO instruments vary in total number of items
per concept measured:

Single item for single concept


Multiple items for single concept


Eg, a VAS to measure pain intensity or pictorial pain
intensity scale
Eg, the Fatigue Severity Scale (9 items)
Multiple items for multiple domains within a concept

Eg, the HAQ (Health Assessment Questionnaire) assessing
8 domains (or concepts) of physical function (disability), with
2+ items each

{Dressing/grooming, rising, eating, walking, hygiene, reaching,
gripping, getting in/out of a car}
Number of Items Per Concept:
How many items are utilized to capture an underlying concept?

PRO instruments vary in total number of items
per concept measured:

Single item for single concept


Multiple items for single concept


Eg, a VAS to measure pain intensity or pictorial pain
intensity scale
Eg, the Fatigue Severity Scale (9 items)
Multiple items for multiple domains within a concept

Eg, the HAQ (Health Assessment Questionnaire) assessing
8 domains (or concepts) of physical function (disability), with
2+ items each

{Dressing/grooming, rising, eating, walking, hygiene, reaching,
gripping, getting in/out of a car}
Number of Items Per Concept:
How many items are utilized to capture an underlying concept?

PRO instruments vary in total number of items
per concept measured:

Single item for single concept


Multiple items for single concept


Eg, a VAS to measure pain intensity or pictorial pain
intensity scale
Eg, the Fatigue Severity Scale (9 items)
Multiple items for multiple domains within a concept

Eg, the HAQ (Health Assessment Questionnaire) assessing
8 domains (or concepts) of physical function (disability), with
2+ items each

{Dressing/grooming, rising, eating, walking, hygiene, reaching,
gripping, getting in/out of a car}
Number of Items Per Concept:
How many items are utilized to capture an underlying concept?

PRO instruments vary in total number of items
per concept measured:

Single item for single concept


Multiple items for single concept


Eg, a VAS to measure pain intensity or pictorial pain
intensity scale
Eg, the Fatigue Severity Scale (9 items)
Multiple items for multiple domains within a concept

Eg, the HAQ (Health Assessment Questionnaire) assessing
8 domains (or concepts) of physical function (disability), with
2+ items each

{Dressing/grooming, rising, eating, walking, hygiene, reaching,
gripping, getting in/out of a car}
Trade-off Between Number of Items and
Properties of the PRO Instrument:
Problems Afflicting Short Forms
Ideal
Ruler
Floor
Problem
7
7
6
5
Loss of
Measurement
Precision
Ceiling
Problem
7
5
5
4
3
3
3
1
1
2
1
Short Forms
Trade-off Between Number of Items and
Properties of the PRO Instrument:
Problems Afflicting Short Forms
Ideal
Ruler
In general, the greater
then number of items to
measure a single
concept, the greater the
measurement precision.
7
7
6
5
Floor
Problem
Loss of
Measurement
Precision
Ceiling
Problem
7
5
5
4
3
3
3
1
1
2
1
Short Forms
Trade-off Between Number of Items and
Properties of the PRO Instrument:
Problems Afflicting Short Forms
Ideal
Ruler
In general, the greater
then number of items to
measure a single
concept, the greater the
measurement precision.
7
7
6
5
Floor
Problem
Loss of
Measurement
Precision
Ceiling
Problem
7
5
5
4
3
3
3
1
1
2
It is important to understand
the range of1 the concept that
one needs to measure for
your target population and
product.
Short Forms
What characterizes PRO instruments?

PROs differ on:

Item characteristics




Concepts measured






Type of item
Number of response choices of each item
Number of items per concept measured
Number of concepts measured in a single instrument
Disease specific vs Generic measures
Complexity of concepts measured
Association of concept w bio-physiologic mechanisms
Recall period
Method of administration
Number of Concepts Measured
How many concepts does a single PRO instrument measure?

PRO instruments vary in total number of
concepts measured:
 Instrument measures one concept

Eg, the Fatigue Severity Scale


1 scale with 9 items
Instrument measures many concepts

Eg, the SF-36


8 scales (Phys. Function, Role Physical, Bodily Pain,
General Health, Vitality, Social Function, Role
Emotional, Mental Health)
2 summary concepts (Physical and Mental Health)
Number of Concepts Measured
How many concepts does a single PRO instrument measure?

PRO instruments vary in total number of
concepts measured:
 Instrument measures one concept

Eg, the Fatigue Severity Scale


1 scale with 9 items
Instrument measures many concepts

Eg, the SF-36


8 scales (Phys. Function, Role Physical, Bodily Pain,
General Health, Vitality, Social Function, Role
Emotional, Mental Health)
2 summary concepts (Physical and Mental Health)
Number of Concepts Measured
How many concepts does a single PRO instrument measure?

PRO instruments vary in total number of
concepts measured:
 Instrument measures one concept

Eg, the Fatigue Severity Scale


1 scale with 9 items
Instrument measures many concepts

Eg, the SF-36


8 scales (Phys. Function, Role Physical, Bodily Pain,
General Health, Vitality, Social Function, Role
Emotional, Mental Health)
2 summary concepts (Physical and Mental Health)
One Instrument Measures One Concept:
Fatigue Severity Scale
Entire tool consists of 9 items, with the aim of having all items measure
the same concept, fatigue.
Read each statement and circle a number from 1 to 7, based on how accurately it reflects your
condition during the past week and the extent to which you agree or disagree that the
statement applies to you.
During the past week, I have found that:
Disagree
1. My motivation is lower when I am fatigued.
1
2
3
2. Exercise brings on my fatigue.
1
2
3
3. I am easily fatigued.
1
2
3
4. Fatigue interferes with my physical functioning.
1
2
3
5. Fatigue causes frequent problems for me.
1
2
3
6. My fatigue prevents sustained physical functioning.
1
2
3
7. Fatigue interferes with carrying out certain duties and responsibilities.
1
2
3
8. Fatigue is among my three most disabling symptoms.
1
2
3
9. Fatigue interferes with my work, family, or social life.
1
2
3
4
4
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
5
Agree
6
7
6
7
6
7
6
7
6
7
6
7
6
7
6
7
6
7
Number of Concepts Measured
How many concepts does a single PRO instrument measure?

PRO instruments vary in total number of
concepts measured:
 Instrument measures one concept

Eg, the Fatigue Severity Scale


1 scale with 9 items
Instrument measures many concepts

Eg, the SF-36


8 scales (Phys. Function, Role Physical, Bodily Pain,
General Health, Vitality, Social Function, Role
Emotional, Mental Health)
2 summary concepts (Physical and Mental Health)
One Instrument Measures Many Concepts:
SF-36
EXAMPLE:
This
conceptual
framework
shows
expected
relationships
between
items
measuring 8
concepts
summarized
into 2 general
concepts.
Source: http://www.sf-36.org/tools/SF36.shtml
What characterizes PRO instruments?

PROs differ on:

Item characteristics




Concepts measured






Type of item
Number of response choices of each item
Number of items per concept measured
Number of concepts measured in a single instrument
Disease specific vs Generic measures
Complexity of concepts measured
Association of concept w bio-physiologic mechanisms
Recall period
Method of administration
Disease Specific vs. Generic PROs

Disease-Specific PROs


Refer to PROs that assess concepts that are defined
specifically for a particular population
They focus on the impact of a particular condition on
the patient’s functioning and experience


Asthma Control Questionnaire
Generic PROs

Refer to PROs that assess concepts that apply to a
broad range of diseases or conditions

Eg, SF-36 developed to measure, among other things, a
wide range of physical functioning
Example: Disease-Specific PRO

The Asthma Control Questionnaire
Circle the number of the response that best describes how you have been
during the past week.
1. On average, during the past week, how often were you woken by your
asthma during the night?
{NEVER to UNABLE TO SLEEP B/C OF ASTHMA}
2. On average, during the past week, how bad were your asthma symptoms
when you woke up in the morning?
{NO SYMPTOMS to VERY SEVERE SYMPTOMS}
3. In general, during the past week, how limited were you in your activities b/c
of your asthma?
{NOT LIMITED AT ALL to TOTALLY LIMITED}
Disease Specific vs. Generic PROs

Disease-Specific PROs


Refer to PROs that assess concepts that are defined
specifically for a particular population
They focus on the impact of a particular condition on
the patient’s functioning and experience


Asthma Control Questionnaire
Generic PROs

Refer to PROs that assess concepts that apply to a
broad range of diseases or conditions

Eg, SF-36 developed to measure, among other things, a
wide range of physical functioning
Example: Generic PRO

The SF-36’s Physical Function scale
The following questions are about activities you might do during a typical day.
Does your health now limit you in these activities? If so, how much?
a. Vigorous activities, such as running, lifting
heavy objects, participating in strenuous sports
b. Moderate activities, such as moving a table,
pushing a vacuum cleaner, bowling, or playing golf
c. Lifting or carrying groceries
d. Climbing several flights of stairs
e. Climbing one flight of stairs
f. Bending, kneeling, or stooping
g. Walking more than a mile
h. Walking several hundred yards
i. Walking one hundred yards
j. Bathing or dressing yourself
Yes,
limited a
lot
Yes,
limited a
little
No, not
limited at
all
1
2
3
1
2
3
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
Source: Version 2 of the SF-36 Health Survey
Helpful Strategy

To include in a clinical trial

A disease-specific PRO


A generic PRO


Eg, Asthma Control Questionnaire
Eg, SF-36
They complement each other in describing
The burden of disease
 The benefit of treatment

Helpful Strategy

To include in a clinical trial

A disease-specific PRO


A generic PRO


Eg, Asthma Control Questionnaire
Eg, SF-36
They complement each other in describing
The burden of disease
 The benefit of treatment

Helpful Strategy

Complementarity:
Disease-specific PROs zoom-in on diseasespecific concepts with disease-specific
relevance and conceptual precision
 Generic PROs

1. Disease Burden: Allow for comparing disease
scores to population norm
 2. Relative Disease Burden: Allow for crossdisease comparison of disease impact
 3. Benefit of Treatment: Allow for comparing trial
endpoint scores to population norm

Helpful Strategy

Complementarity:
Disease-specific PROs zoom-in on diseasespecific concepts with disease-specific
relevance and conceptual precision
 Generic PROs

1. Disease Burden: Allow for comparing disease
scores to population norm
 2. Relative Disease Burden: Allow for crossdisease comparison of disease impact
 3. Benefit of Treatment: Allow for comparing trial
endpoint scores to population norm

1. BURDEN OF DISEASE: Generic PROs allow for
comparing disease to population norm
Example:
Generic
PRO
Measure,
the SF-36
Figure 1. Total Burden of Restless Leg Syndrome on HRQOL
SF-36 Scores for USA (N=158) Compared to
1998 U.S. General Population Norms (N=6742)
Best
Health
Average
Adult
U.S.
Norm
(Unadj.)
60
60
55
55
50
50
*
45
40
Poorest
Health
*
*
*
*
*
*
*
*
45
40
*
35
35
30
30
25
25
20
20
PF
RP
BP
SF-36 Scores
US Adult Norms (Adjusted)+
GH
VT
SF
SF-36 Scales
RE
MH
All scales:
Mean = 50
SD = 10
PCS
MCS
Summaries
* Significantly lower mean score than 1998 U.S. Norm. + US adult norms adjusted to the age and gender of the target RLS sample.
Source: Martin et al. 2005
1. BURDEN OF DISEASE: Generic PROs allow for
comparing disease to population norm
Example:
Generic
PRO
Measure,
the SF-36
Figure 1. Total Burden of Restless Leg Syndrome on HRQOL
SF-36 Scores for USA (N=158) Compared to
1998 U.S. General Population Norms (N=6742)
Best
Health
Average
Adult
U.S.
Norm
(Unadj.)
60
60
55
55
50
50
*
45
40
*
*
*
*
*
*
*
45
40
*
35
35
30
Poorest
Health
*
25
20
PF
RP
BP
SF-36 Scores
US Adult Norms (Adjusted)+
GH
VT
SF
SF-36 Scales
RE
Burden
is defined as a
30
negative deviation in SF-36
25
scores from norm associated
with20 the presence of a medical
MH
PCS
MCS
condition Summaries
(e.g., RLS).
All scales:
Mean = 50
SD = 10
* Significantly lower mean score than 1998 U.S. Norm. + US adult norms adjusted to the age and gender of the target RLS sample.
Source: Martin et al. 2005
Helpful Strategy

Complementarity:
Disease-specific PROs zoom-in on diseasespecific concepts with disease-specific
relevance and conceptual precision
 Generic PROs

1. Disease Burden: Allow for comparing disease
scores to population norm
 2. Relative Disease Burden: Allow for crossdisease comparison of disease impact
 3. Benefit of Treatment: Allow for comparing trial
endpoint scores to population norm

2. RELATIVE DISEASE BURDEN:
Generic PROs allow for cross-disease
comparison of disease impact
Placement of burden of disease among other diseases
Example: Generic PRO Measure, SF-36
Table 1. Total Burden of RLS: Study Sample Compared With General US Population Normative Data
Norms Adjusted for Comorbidities
RLS US Sample
(N=158)
General US
Population Norms
(N=6742)
US Population Norms
for
Type-2
Diabetes
(N=570)
(N=570)
US Population Norms
for
Depression
(N=1334)
US Population Norms
for
Osteoarthritis
(N=968)
Mean
SE
Mean
SE
Mean
SE
Mean
SE
Mean
SE
SF-36 scales
Physical functioning
Role physical
Bodily pain
General health
Vitality
Social functioning
Role emotional
Mental health
38.36
40.70
38.45
40.52
41.62
42.91
45.47
44.92
1.04
0.96
0.83
0.92
0.98
1.08
1.05
1.06
48.04d
48.91d
49.14d
49.48d
50.30d
49.71d
49.90d
50.43d
0.23
0.24
0.24
0.24
0.24
0.24
0.24
0.24
43.43d
42.99a
48.46d
42.42
47.83d
48.37d
46.37
49.50d
0.61
0.61
0.59
0.52
0.54
0.57
0.63
0.55
42.76d
40.26
45.55d
41.88
44.05b
43.14
39.48d
40.35d
0.37
0.39
0.38
0.36
0.35
0.40
0.47
0.42
41.34b
41.85
39.98a
42.10
44.86c
43.76
45.23
47.54
0.66
0.68
0.55
0.62
0.59
0.72
0.76
0.63
SF-36 summary measures
Physical summary
Mental summary
37.56
46.79
0.97
1.09
48.32d
50.82d
0.23
0.23
43.22d
49.95b
0.58
0.54
44.24d
41.00d
0.38
0.45
39.67b
47.94
0.63
0.67
a
p<.05; bp<.01; cp<.001; dp<.0001
Source: Martin et al. 2005
2. RELATIVE DISEASE BURDEN:
Generic PROs allow for cross-disease
comparison of disease impact
Placement of burden of disease among other diseases
Example: Generic PRO Measure, SF-36
Table 1. Total Burden of RLS: Study Sample Compared With General US Population Normative Data
Norms Adjusted for Comorbidities
RLS US Sample
(N=158)
General US
Population Norms
(N=6742)
US Population Norms
for
Type-2
Diabetes
(N=570)
(N=570)
US Population Norms
for
Depression
(N=1334)
US Population Norms
for
Osteoarthritis
(N=968)
Mean
SE
Mean
SE
Mean
SE
Mean
SE
Mean
SE
SF-36 scales
Physical functioning
Role physical
Bodily pain
General health
Vitality
Social functioning
Role emotional
Mental health
38.36
40.70
38.45
40.52
41.62
42.91
45.47
44.92
1.04
0.96
0.83
0.92
0.98
1.08
1.05
1.06
48.04d
48.91d
49.14d
49.48d
50.30d
49.71d
49.90d
50.43d
0.23
0.24
0.24
0.24
0.24
0.24
0.24
0.24
43.43d
42.99a
48.46d
42.42
47.83d
48.37d
46.37
49.50d
0.61
0.61
0.59
0.52
0.54
0.57
0.63
0.55
42.76d
40.26
45.55d
41.88
44.05b
43.14
39.48d
40.35d
0.37
0.39
0.38
0.36
0.35
0.40
0.47
0.42
41.34b
41.85
39.98a
42.10
44.86c
43.76
45.23
47.54
0.66
0.68
0.55
0.62
0.59
0.72
0.76
0.63
SF-36 summary measures
Physical summary
Mental summary
37.56
46.79
0.97
1.09
48.32d
50.82d
0.23
0.23
43.22d
49.95b
0.58
0.54
44.24d
41.00d
0.38
0.45
39.67b
47.94
0.63
0.67
a
p<.05; bp<.01; cp<.001; dp<.0001
Source: Martin et al. 2005
2. RELATIVE DISEASE BURDEN:
Generic PROs allow for cross-disease
comparison of disease impact
Placement of burden of disease among other diseases
30
34 36
40
Average
Adult
Average
Well
Adult
50
55
SF-36’s Physical Component Summary (PCS)
Source: Adapted from Ware and Kosinski, 2001
2. RELATIVE DISEASE BURDEN:
Generic PROs allow for cross-disease
comparison of disease impact
Placement of burden of disease among other diseases
Restless
Legs
Syndrome
Type-2
OA Diabetes
Depression
30
34 36
40
Average
Adult
Average
Well
Adult
50
55
SF-36’s Physical Component Summary (PCS)
Source: Adapted from Ware and Kosinski, 2001
2. RELATIVE DISEASE BURDEN:
Generic PROs allow for cross-disease
comparison of disease impact
Placement of burden of disease among other diseases
Restless
Legs
Syndrome
Type-2
OA Diabetes
Depression
Congestive
Heart
Failure
Chronic
Lung
Disease
30
34 36
Asthma
40
Average
Adult
Average
Well
Adult
50
55
SF-36’s Physical Component Summary (PCS)
Source: Adapted from Ware and Kosinski, 2001
Helpful Strategy

Complementarity:
Disease-specific PROs zoom-in on diseasespecific concepts with disease-specific
relevance and conceptual precision
 Generic PROs

1. Disease Burden: Allow for comparing disease
scores to population norm
 2. Relative Disease Burden: Allow for crossdisease comparison of disease impact
 3. Benefit of Treatment: Allow for comparing trial
endpoint scores to population norm

3. BENEFIT OF TX: Generic PROs allow for
comparing endpoint scores to population norm
Baseline
(BL) Burden
and Endpoint
(EP) Leg
scores
– MOCKonDATA
Figure
1. Total
of Restless
Syndrome
HRQOL
Example:
Generic
PRO
Measure,
the SF-36
SF-36 Scores for USA (N=158) Compared to
1998 U.S. General Population Norms (N=6742)
Best
Health
Average
Adult
U.S.
Norm
(Unadj.)
60
60
55
55
50
50
*
45
40
Poorest
Health
*
*
*
*
*
*
*
*
45
EP
BL
40
*
35
35
30
30
25
25
20
20
PF
RP
BP
SF-36 Scores
US Adult Norms (Adjusted)+
GH
VT
SF
SF-36 Scales
RE
MH
All scales:
Mean = 50
SD = 10
PCS
MCS
Summaries
* Significantly lower mean score than 1998 U.S. Norm. + US adult norms adjusted to the age and gender of the target RLS sample.
Adapted from Martin et al. 2005
Helpful Strategy -- Punchline

Complementarity of Disease-specific and
Generic PROs:

When both are included in a clinical trial, it is
possible to capture
Disease-specific concepts
 Generic concepts to compare to norm and
demonstrate




Burden of illness
Relative burden of illness
Benefit of illness in lifting burden of illness
Helpful Strategy -- Punchline

Complementarity of Disease-specific and
Generic PROs:

When both are included in a clinical trial, it is
possible to capture
Disease-specific concepts
 Generic concepts to compare to norm and
demonstrate




Burden of illness
Relative burden of illness
Benefit of illness in lifting burden of illness
What characterizes PRO instruments?

PROs differ on:

Item characteristics




Concepts measured






Type of item
Number of response choices of each item
Number of items per concept measured
Number of concepts measured in a single instrument
Disease specific vs Generic measures
Complexity of concepts measured
Association of concept w bio-physiologic mechanisms
Recall period
Method of administration
Complexity of concepts measured

Concepts measured by PROs differ in their
degree of complexity:
 From simple


To more complex concepts


eg, presence of a symptom
eg, ability to carry out activities of daily living
To extremely complex concepts

eg, quality of life
WILSON-CLEARY MODEL OF HEALTH OUTCOMES:
Range of complexity of concepts captured with PROs
Characteristics
of Individual
Simple:
More complex:
presence of a
ability to carry
symptom (eg,
out activities of
headache)
daily living
Biological and
Physiological
Variables
Symptom
Status
Functional
Status
Most complex:
Quality of Life
General Health
Perceptions
Quality of Life
Characteristics of Environment
From CELL to INDIVIDUAL to the interaction of person as a MEMBER OF SOCIETY
-- Concepts are increasingly difficult to define and measure – INCREASINGLY COMPLEX
Wilson & Cleary JAMA (1995); Adapted from Revicki, 2006
What characterizes PRO instruments?

PROs differ on:

Item characteristics




Concepts measured






Type of item
Number of response choices of each item
Number of items per concept measured
Number of concepts measured in a single instrument
Disease specific vs Generic measures
Complexity of concepts measured
Association of concept w bio-physiologic mechanisms
Recall period
Method of administration
WILSON-CLEARY MODEL OF HEALTH OUTCOMES:
Association of concept w bio-physiologic mechanisms
Characteristics
of Individual
Simple:
More complex:
presence of a
ability to carry
symptom (eg,
out activities of
headache)
daily living
Biological and
Physiological
Variables
Symptom
Status
Functional
Status
Most complex:
Quality of Life
General Health
Perceptions
Quality of Life
Characteristics of Environment
From CELL to INDIVIDUAL to the interaction of person as a MEMBER OF SOCIETY
-- Concepts are increasingly more distant from bio-physiologic mechanisms
Wilson & Cleary JAMA (1995); Adapted from Revicki, 2006
What characterizes PRO instruments?

PROs differ on:

Item characteristics




Concepts measured






Type of item
Number of response choices of each item
Number of items per concept measured
Number of concepts measured in a single instrument
Disease specific vs Generic measures
Complexity of concepts measured
Association of concept w bio-physiologic mechanisms
Recall period
Method of administration
Recall Period

Refers to the period over which the study
participant is asked to recall his/her
experience

Eg, current/now, today, last week, last 4
weeks, last month
How intense is your pain now?
 Mark a response that describes the way you feel
today.
 During the past week, how was your …?
 During the past four weeks, how …?

Recall Period

Refers to the period over which the study
participant is asked to recall his/her
experience

Eg, current/now, today, last week, last 4
weeks, last month
How intense is your pain now?
 Mark a response that describes the way you feel
today.
 During the past week, how was your …?
 During the past four weeks, how …?

Recall Period (Cont.)

In general, shorter reporting
periods are preferred over
longer ones.


Of concern is the difficulty in
remembering accurately, which
may threaten the accuracy of
the item responses.
“PRO instruments that require
patients to rely on memory …
may threaten the accuracy of
the PRO data.“ FDA Guidance
Source: A. Stone, 2006; FDA Guidance, 2006
Recall Period (Cont.)

In general, shorter reporting
periods are preferred over
longer ones.


Of concern is the difficulty in
remembering accurately, which
may threaten the accuracy of
the item responses.
“PRO instruments that require
patients to rely on memory …
may threaten the accuracy of
the PRO data.“ FDA Guidance
Source: A. Stone, 2006; FDA Guidance, 2006
Recall Period (Cont.)

When examining a PRO tool for use in a clinical
trial, it is important



to assess the appropriateness of the recall period.
to consider patients’ ability to accurately recall the
information requested over the recall period proposed.
The choice of recall period that is most suitable
depends on



the purpose and intended use of the instrument
the characteristics of the disease/condition
the treatment to be tested
Source: FDA Guidance, 2006
Recall Period (Cont.)

When examining a PRO tool for use in a clinical
trial, it is important



to assess the appropriateness of the recall period.
to consider patients’ ability to accurately recall the
information requested over the recall period proposed.
The choice of recall period that is most suitable
depends on



the purpose and intended use of the instrument
the characteristics of the disease/condition
the treatment to be tested
Source: FDA Guidance, 2006
Recall Period (Cont.)


When evaluating PRO-based claims, the FDA
 Will review the study protocol to determine
what steps were taken to ensure that patients
understand the appropriate recall period.
If a patient diary or some other form of
unsupervised data entry is used,
 the FDA will review the protocol to determine
what measures are taken to ensure that
patients make entries according to the study
design and not, for example, just before a clinic
visit when their reports will be collected.
Source: FDA Guidance, 2006
Recall Period (Cont.)


When evaluating PRO-based claims, the FDA
 Will review the study protocol to determine
what steps were taken to ensure that patients
understand the appropriate recall period.
If a patient diary or some other form of
unsupervised data entry is used,
 the FDA will review the protocol to determine
what measures are taken to ensure that
patients make entries according to the study
design and not, for example, just before a clinic
visit when their reports will be collected.
Source: FDA Guidance, 2006
What characterizes PRO instruments?

PROs differ on:

Item characteristics




Concepts measured






Type of item
Number of response choices of each item
Number of items per concept measured
Number of concepts measured in a single instrument
Disease specific vs Generic measures
Complexity of concepts measured
Association of concept w bio-physiologic mechanisms
Recall period
Method of administration
Method of Administration of PRO

Self-administered




Interviewer-administered



Paper and pencil
Internet
Handheld device
In person
Over the telephone
Interactively-administered


Interactive voice response system (IVRS)
Interactive web response system (IWRS)
What characterizes PRO instruments?

PROs differ on:

Item characteristics








Type of item
Number of response choices of each item
Number of items per concept measured
Concepts measured


REVIEW
Number of concepts measured in a single instrument
Disease specific vs Generic measures
Complexity of concepts measured
Association of concept w bio-physiologic mechanisms
Recall period
Method of administration
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
What is the goal of medical treatment?
What is a PRO?
Why use PRO measures?
How do PRO measures complement traditional
clinical outcome measures?
What characterizes PRO measures?
 How do PROs differ?
What characterizes a well-constructed PRO
instrument?
 How are PRO instruments developed?
What is the process necessary to achieve a PRO
label?
 Putting it all together
What is the role of PROs in clinical trials research?
Review and conclusions
Long Segment Ahead
“Put on your seat belt”
What characterizes
a well-constructed
PRO instrument?
How are PRO instruments
developed?
What defines a well-constructed
PRO instrument?

The challenge is to define and capture
(operationalize) the target concept well

To achieve this, PRO instruments

Must be based on sound qualitative research




Patient input
Literature review
Expert opinion
Must be based on sound quantitative methods

Appropriate psychometric properties and construct
validation must be demonstrated
FDA Draft PRO Guidance
www.fda.gov/cder/guidance/5460dft.pdf
February 2006
FDA’s 2006 PRO Draft Guidance

This guidance
 Emphasizes that, when defining clinical benefit, it is “extremely
important” to determine how a patient feels or functions as a
result of treatment.

Underscores that PRO instruments used in clinical trials must be




well-developed and
adequately validated to measure what they are intended to
measure.
Describes how the FDA evaluates PRO instruments when used
as efficacy endpoints in clinical trials
Provides recommendations for developing and validating PRO
instruments to support labeling claims.
Burke & Patrick, FDA/Industry Statistics Workshop, 2007
FDA’s 2006 PRO Draft Guidance

This guidance
 Emphasizes that, when defining clinical benefit, it is “extremely
important” to determine how a patient feels or functions as a
result of treatment.

Underscores that PRO instruments used in clinical trials must be




well-developed and
adequately validated to measure what they are intended to
measure.
Describes how the FDA evaluates PRO instruments when used
as efficacy endpoints in clinical trials
Provides recommendations for developing and validating PRO
instruments to support labeling claims.
Burke & Patrick, FDA/Industry Statistics Workshop, 2007
FDA’s 2006 PRO Draft Guidance

This guidance
 Emphasizes that, when defining clinical benefit, it is “extremely
important” to determine how a patient feels or functions as a
result of treatment.

Underscores that PRO instruments used in clinical trials must be




well-developed and
adequately validated to measure what they are intended to
measure.
Describes how the FDA evaluates PRO instruments when used
as efficacy endpoints in clinical trials
Provides recommendations for developing and validating PRO
instruments to support labeling claims.
Burke & Patrick, FDA/Industry Statistics Workshop, 2007
FDA’s 2006 PRO Draft Guidance

This guidance
 Emphasizes that, when defining clinical benefit, it is “extremely
important” to determine how a patient feels or functions as a
result of treatment.

Underscores that PRO instruments used in clinical trials must be




well-developed and
adequately validated to measure what they are intended to
measure.
Describes how the FDA evaluates PRO instruments when used
as efficacy endpoints in clinical trials
Provides recommendations for developing and validating PRO
instruments to support labeling claims.
Burke & Patrick, FDA/Industry Statistics Workshop, 2007
PRO Instrument Development
FDA Draft Guidance to Industry--Figure 1:
The PRO Instrument Development and Modification Process
This 2006
guidance is
for PRO
development.
Any preexisting or
new PRO
must be
based on this
methodology.
PRO Instrument Development
FDA Draft Guidance to Industry--Figure 1:
The PRO Instrument Development and Modification Process
This 2006
guidance is
for PRO
development.
Any preexisting or
new PRO
must be
based on this
methodology.
More recent
and detailed
steps from
PRO dev’t to
PRO claims
i.
Hypothesize Conceptual Framework
•
•
•
•
•
•
Outline hypothesized concepts & potential claims
Determine intended population
Perform literature/expert review
Develop hypothesized conceptual framework
Place PROs within preliminary endpoint model
Document preliminary instrument development
v. Modify Instrument
•
•
•
•
Change wording of items,
populations, response
options, recall period, or
method of administration
Translate & culturally
adapt to other languages
Evaluate as appropriate
Document all changes
ii. Adjust Conceptual
Framework & Draft
Instrument
PRO
Claim
iv. Collect, Analyze, &
Interpret Data
•
•
•
•
Prepare protocol & statistical
analysis plan
Collect & analyze data
Evaluate treatment response using
cumulative distribution & responder
definition
Document interpretation of
treatment benefit in relation to claim
Burke & Patrick,
FDA/Industry
Statistics
Workshop, 2007
•
•
•
•
•
•
•
Generate new items
Create instrument
Select administration mode,
recall period & response
options
Format instrument
Conduct cognitive debriefing
Pilot test draft instrument
Document content validity
iii. Confirm Conceptual Framework &
Assess Other Measurement Properties
•
•
•
•
Confirm conceptual framework with scoring rule
Assess score reliability, construct validity, & ability to
detect change
Finalize instrument content, formats, scoring, procedures
& training materials
Document measurement development
i.
Hypothesize Conceptual Framework
•
•
•
•
•
•
Outline hypothesized concepts & potential claims
Determine intended population
Perform literature/expert review
Develop hypothesized conceptual framework
Place PROs within preliminary endpoint model
Document preliminary instrument development
v. Modify Instrument
•
•
•
•
Change wording of items,
populations, response
options, recall period, or
method of administration
Translate & culturally
adapt to other languages
Evaluate as appropriate
Document all changes
ii. Adjust Conceptual
Framework & Draft
Instrument
PRO
Claim
iv. Collect, Analyze, &
Interpret Data
•
•
•
•
Prepare protocol & statistical
analysis plan
Collect & analyze data
Evaluate treatment response using
cumulative distribution & responder
definition
Document interpretation of
treatment benefit in relation to claim
Burke & Patrick,
FDA/Industry
Statistics
Workshop, 2007
•
•
•
•
•
•
•
Generate new items
Create instrument
Select administration mode,
recall period & response
options
Format instrument
Conduct cognitive debriefing
Pilot test draft instrument
Document content validity
iii. Confirm Conceptual Framework &
Assess Other Measurement Properties
•
•
•
•
Confirm conceptual framework with scoring rule
Assess score reliability, construct validity, & ability to
detect change
Finalize instrument content, formats, scoring, procedures
& training materials
Document measurement development
Stage 1: Preliminary activity
Hypothesize Conceptual Framework










Determine intended population
Perform literature review
Obtain patient input
Obtain expert opinion
Outline hypothesized PRO concept(s)
Outline potential PRO-based claim(s)
Place PROs within preliminary endpoint model
Develop hypothesized conceptual framework
Document preliminary instrument development
Emphasis on this stage is qualitative activity:


Thinking, hypothesizing, drafting, reviewing
Stage 1: Preliminary activity
Hypothesize Conceptual Framework










Determine intended population
THINKING
Perform literature review
OBTAINING
Obtain patient input
HYPOTHESIZING
Obtain expert opinion
Outline hypothesized PRO concept(s)
Outline potential PRO-based claim(s)
Place PROs within preliminary endpoint model
Develop hypothesized conceptual framework
Document preliminary instrument development
Emphasis on this stage is qualitative activity:


Thinking, hypothesizing, drafting, reviewing
Stage 1: Preliminary activity
Hypothesize Conceptual Framework










Determine intended population
Perform literature review
Obtain patient input
Obtain expert opinion
Outline hypothesized PRO concept(s)
Outline potential PRO-based claim(s)
Place PROs within preliminary endpoint model
Develop hypothesized conceptual framework
Document preliminary instrument development
Emphasis on this stage is qualitative activity:


Thinking, hypothesizing, drafting, reviewing
Endpoint Model Example:
Rheumatoid Arthritis
Pain
Intensity
Mobility
Swollen
Joints
Pathophysiologic
Mechanism
Social
Function/
Activities
Fatigue
HealthRelated
Quality of
Life
Activities of
Daily Living
Tender
Joints
Emotional
Well-Being
Sleep
Problems
Physical
Function
In Stage 1, this is an initial draft
Adapted from Revicki, 2006
Endpoint Model Example:
Rheumatoid Arthritis
These concepts are
typically captured via
PROs.
Pain
Intensity
Mobility
Swollen
Joints
Pathophysiologic
Mechanism
Social
Function/
Activities
Fatigue
HealthRelated
Quality of
Life
Activities of
Daily Living
Tender
Joints
Emotional
Well-Being
Sleep
Problems
Physical
Function
Adapted from Revicki, 2006
What is an Endpoint Model?

An endpoint model

Identifies appropriate endpoint concepts



Both clinical and PRO concepts
Knowledge of disease and likely tx impact drives the concepts to be
measured by PROs
Relates such concepts to one another

Articulates associations among different health outcomes





States the set of concept relationships in terms of specific measures
Demonstrates importance of health outcomes
Is not a theoretical framework/model
Provides rationale for PRO measurement strategy


Identifies important PRO domains (concepts)
Explains association with treatment benefit


Proximal versus distal outcomes
Is a key driver of protocol design


Provides links btw clinical and PRO outcomes
Points to appropriate trial designs to support claims
Informs identification of PRO label targets
Adapted from Rock, FDA, 2006; Revicki, ISOQOL, 2006
What is an Endpoint Model?

An endpoint model

Identifies appropriate endpoint concepts



Both clinical and PRO concepts
Knowledge of disease and likely tx impact drives the concepts to be
measured by PROs
Relates such concepts to one another

Articulates associations among different health outcomes





States the set of concept relationships in terms of specific measures
Demonstrates importance of health outcomes
Is not a theoretical framework/model
Provides rationale for PRO measurement strategy


Identifies important PRO domains (concepts)
Explains association with treatment benefit


Proximal versus distal outcomes
Is a key driver of protocol design


Provides links btw clinical and PRO outcomes
Points to appropriate trial designs to support claims
Informs identification of PRO label targets
Adapted from Rock, FDA, 2006; Revicki, ISOQOL, 2006
What is an Endpoint Model?

An endpoint model

Identifies appropriate endpoint concepts



Both clinical and PRO concepts
Knowledge of disease and likely tx impact drives the concepts to be
measured by PROs
Relates such concepts to one another

Articulates associations among different health outcomes





States the set of concept relationships in terms of specific measures
Demonstrates importance of health outcomes
Is not a theoretical framework/model
Provides rationale for PRO measurement strategy


Identifies important PRO domains (concepts)
Explains association with treatment benefit


Proximal versus distal outcomes
Is a key driver of protocol design


Provides links btw clinical and PRO outcomes
Points to appropriate trial designs to support claims
Informs identification of PRO label targets
Adapted from Rock, FDA, 2006; Revicki, ISOQOL, 2006
What is an Endpoint Model?

An endpoint model

Identifies appropriate endpoint concepts



Both clinical and PRO concepts
Knowledge of disease and likely tx impact drives the concepts to be
measured by PROs
Relates such concepts to one another

Articulates associations among different health outcomes





States the set of concept relationships in terms of specific measures
Demonstrates importance of health outcomes
Is not a theoretical framework/model
Provides rationale for PRO measurement strategy


Identifies important PRO domains (concepts)
Explains association with treatment benefit


Proximal versus distal outcomes
Is a key driver of protocol design


Provides links btw clinical and PRO outcomes
Points to appropriate trial designs to support claims
Informs identification of PRO label targets
Adapted from Rock, FDA, 2006; Revicki, ISOQOL, 2006
What is an Endpoint Model?

An endpoint model

Identifies appropriate endpoint concepts



Both clinical and PRO concepts
Knowledge of disease and likely tx impact drives the concepts to be
measured by PROs
Relates such concepts to one another

Articulates associations among different health outcomes





States the set of concept relationships in terms of specific measures
Demonstrates importance of health outcomes
Is not a theoretical framework/model
Provides rationale for PRO measurement strategy


Identifies important PRO domains (concepts)
Explains association with treatment benefit


Proximal versus distal outcomes
Is a key driver of protocol design


Provides links btw clinical and PRO outcomes
Points to appropriate trial designs to support claims
Informs identification of PRO label targets
Adapted from Rock, FDA, 2006; Revicki, ISOQOL, 2006
Endpoint Model Example:
Rheumatoid Arthritis
These concepts are
typically captured via
PROs.
Pain
Intensity
Mobility
Swollen
Joints
Pathophysiologic
Mechanism
Social
Function/
Activities
Fatigue
HealthRelated
Quality of
Life
Activities of
Daily Living
Tender
Joints
Emotional
Well-Being
Sleep
Problems
Physical
Function
Adapted from Revicki, 2006
Endpoint Model Example:
Rheumatoid Arthritis
For example, in this
stage, we might identify
these 3 PRO concepts
circled as most relevant
to the product studied,
and draft a PRO-based
label claim based on
them.
Pain
Intensity
Mobility
Swollen
Joints
Pathophysiologic
Mechanism
Social
Function/
Activities
Fatigue
HealthRelated
Quality of
Life
Activities of
Daily Living
Tender
Joints
Emotional
Well-Being
Sleep
Problems
Physical
Function
Adapted from Revicki, 2006
Linking PRO Concepts from
Endpoint Model & Desired Claim
In Stage 1, this is an initial draft
Example
Desired claim
Drug Y relieves
pain and
improves ability
to function
physically and
carry out daily
activities
Concepts from
endpoint model
in claim
PRO
instruments
used
Pain
VAS Pain
Physical
Function
HAQ-DI
Activities of
Daily Living
ADL
Questionnaire
Linking PRO Concepts from
Endpoint Model & Desired Claim
In Stage 1, this is an initial draft
Example
Concepts from
endpoint
model
Next, we need to take
a
in claim
look at the PRO
Desired claim
instruments we are
considering using in our
Pain
Drugclinical
Y relieves
trial:
* Ifand
new, draft it.
pain
* If pre-existing,
assess it.
improves
ability
Physical
to function
Function
physically and
carry out daily
activities
Activities of
Daily Living
PRO
instruments
used
VAS Pain
HAQ-DI
ADL
Questionnaire
Next Step is the PRO Instrument’s
Conceptual Framework
If Instrument is new, draft it. If pre-existing, assess it.
A conceptual framework refers to the expected relationships
between items, domains, and measurement concepts.
Item A
Item B
Item C
Item D
Item E
Item F
Item G
Domain
Score 1
The domains
and overall
score
represent
related but
separate
concepts
Overall
Score
Domain
Score 2
It is a description of how each item
relates to the intended measurement
concepts (measurement model).
Source: FDA Guidance, 2006; Rock, FDA, 206
Example 1: Conceptual Framework-SF-36
EXAMPLE:
This
conceptual
framework
shows
expected
relationships
between
items
measuring 8
concepts
summarized
into 2 general
concepts.
These are the PRO
concepts measured
by the SF-36.
Source: http://www.sf-36.org/tools/SF36.shtml
Example 2: Conceptual Framework of three important
aspects of head & neck cancer measured with PROs
Items
Difficulty in
Swallowing liquids
Swallowing saliva
Swallowing solid foods
PRO Concept
Swallowing
Difficulty in
Speaking loud enough
Being understood by others
Speaking
Difficulty in
Eating
Dressing
Bathing
Toileting (using the bathroom)
Transferring (moving back and
forth from bed to chair)
Remaining continent
Basic Activities of Daily
Living
See how this conceptual framework
specifies exactly how each concept
is measured by each set of items
Adapted from Patrick D et al. Value in Health
In This Stage, it is Important
to Align Target Concepts

Concepts measured in the following need to be aligned:
 In PRO Instrument:


In Endpoint Model:


PRO target concepts in endpoint model that applies to your
product
In Study Protocol:


Concepts operationalized by the conceptual framework of the
PRO instrument
Concepts that are part of clinical trial objectives as stated in
the protocol
In Target PRO Label:

Concepts included in desired PRO-based labeling statement
Stage 1: Preliminary activity
Hypothesize Conceptual Framework










Determine intended population
Perform literature review
Obtain patient input
Obtain expert opinion
Outline hypothesized PRO concept(s)
Outline potential PRO-based claim(s)
Place PROs within preliminary endpoint model
Develop hypothesized conceptual framework
Document preliminary instrument development
Emphasis on this stage is qualitative activity:


Thinking, hypothesizing, drafting, reviewing
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
What is the goal of medical treatment?
What is a PRO?
Reminder
Why use PRO measures?
of where
How do PRO measures complement traditional
we are
clinical outcome measures?
What characterizes PRO measures?
 How do PROs differ?
What characterizes a well-constructed PRO
instrument?
 How are PRO instruments developed?
What is the process necessary to achieve a PRO
label?
 Putting it all together
What is the role of PROs in clinical trials research?
Review and conclusions
i.
Hypothesize Conceptual Framework
•
•
•
•
•
•
Outline hypothesized concepts & potential claims
Determine intended population
Perform literature/expert review
Develop hypothesized conceptual framework
Place PROs within preliminary endpoint model
Document preliminary instrument development
v. Modify Instrument
•
•
•
•
Change wording of items,
populations, response
options, recall period, or
method of administration
Translate & culturally
adapt to other languages
Evaluate as appropriate
Document all changes
ii. Adjust Conceptual
Framework & Draft
Instrument
PRO
Claim
iv. Collect, Analyze, &
Interpret Data
•
•
•
•
Prepare protocol & statistical
analysis plan
Collect & analyze data
Evaluate treatment response using
cumulative distribution & responder
definition
Document interpretation of
treatment benefit in relation to claim
Burke & Patrick,
FDA/Industry
Statistics
Workshop, 2007
•
•
•
•
•
•
•
Generate new items
Create instrument
Select administration mode,
recall period & response
options
Format instrument
Conduct cognitive debriefing
Pilot test draft instrument
Document content validity
iii. Confirm Conceptual Framework &
Assess Other Measurement Properties
•
•
•
•
Confirm conceptual framework with scoring rule
Assess score reliability, construct validity, & ability to
detect change
Finalize instrument content, formats, scoring, procedures
& training materials
Document measurement development
Stage 2: Draft of PRO
Adjust Conceptual Framework & Draft Instrument


Create PRO instrument





Finalize items in instrument (focus groups, pt input)
Select administration mode, recall period & response options
Format instrument
Draft instructions
Conduct cognitive debriefing



a qualitative method to assess subjects’ understanding of the
items included in the PRO instrument

Improve instrument based on cognitive debriefing

extent to which a measure represents all facets of a given
concept
Pilot-test the draft PRO instrument
Document content validity
Emphasis on this stage is qualitative activity:


Drafting instrument, cognitive debriefing
Stage 2: Draft of PRO
Adjust Conceptual Framework & Draft Instrument


Create PRO instrument





Finalize items in instrument (focus groups, pt input)
Select administration mode, recall period & response options
Format instrument
Draft instructions
Conduct cognitive debriefing



a qualitative method to assess subjects’ understanding of the
items included in the PRO instrument

Improve instrument based on cognitive debriefing

extent to which a measure represents all facets of a given
concept
Pilot-test the draft PRO instrument
Document content validity
Emphasis on this stage is qualitative activity:


Drafting instrument, cognitive debriefing
Stage 2: Draft of PRO
Adjust Conceptual Framework & Draft Instrument


Create PRO instrument





Finalize items in instrument (focus groups, pt input)
Select administration mode, recall period & response options
Format instrument
Draft instructions
Conduct cognitive debriefing



a qualitative method to assess subjects’ understanding of the
items included in the PRO instrument

Improve instrument based on cognitive debriefing

extent to which a measure represents all facets of a given
concept
Pilot-test the draft PRO instrument
Document content validity
Emphasis on this stage is qualitative activity:


Drafting instrument, cognitive debriefing
Stage 2: Draft of PRO
Adjust Conceptual Framework & Draft Instrument


Create PRO instrument





Finalize items in instrument (focus groups, pt input)
Select administration mode, recall period & response options
Format instrument
Draft instructions
Conduct cognitive debriefing



a qualitative method to assess subjects’ understanding of the items
included in the PRO instrument

Improve instrument based on cognitive debriefing

extent to which a measure represents all facets of a given
concept
Pilot-test the draft PRO instrument
Document content validity
Emphasis on this stage is qualitative activity:


Drafting instrument, cognitive debriefing
i.
Hypothesize Conceptual Framework
•
•
•
•
•
•
Outline hypothesized concepts & potential claims
Determine intended population
Perform literature/expert review
Develop hypothesized conceptual framework
Place PROs within preliminary endpoint model
Document preliminary instrument development
v. Modify Instrument
•
•
•
•
Change wording of items,
populations, response
options, recall period, or
method of administration
Translate & culturally
adapt to other languages
Evaluate as appropriate
Document all changes
ii. Adjust Conceptual
Framework & Draft
Instrument
PRO
Claim
iv. Collect, Analyze, &
Interpret Data
•
•
•
•
Prepare protocol & statistical
analysis plan
Collect & analyze data
Evaluate treatment response using
cumulative distribution & responder
definition
Document interpretation of
treatment benefit in relation to claim
Burke & Patrick,
FDA/Industry
Statistics
Workshop, 2007
•
•
•
•
•
•
•
Generate new items
Create instrument
Select administration mode,
recall period & response
options
Format instrument
Conduct cognitive debriefing
Pilot test draft instrument
Document content validity
iii. Confirm Conceptual Framework &
Assess Other Measurement Properties
•
•
•
•
Confirm conceptual framework with scoring rule
Assess score reliability, construct validity, & ability to
detect change
Finalize instrument content, formats, scoring, procedures
& training materials
Document measurement development
Stage 3: Finalize PRO
Confirm Conceptual Framework & Assess Other
Measurement Properties






Confirm conceptual framework with scoring rule
Assess score reliability, construct validity, & ability to
detect change
Finalize instrument content, formats, scoring,
procedures & training materials
Document measurement development
Refine and finalize PRO-based label claim
Emphasis on this stage is quantitative activity:


Confirming scoring rule, assessing psychometric
properties
Stage 3: Finalize PRO
Confirm Conceptual Framework & Assess Other
Measurement Properties






Confirm conceptual framework with scoring rule
Assess score reliability, construct validity, & ability to
detect change
Finalize instrument content, formats, scoring,
procedures & training materials
Document measurement development
Refine and finalize PRO-based label claim
Emphasis on this stage is quantitative activity:


Confirming scoring rule, assessing psychometric
properties
Stage 3: Finalize PRO
Confirm Conceptual Framework & Assess Other
Measurement Properties






Confirm conceptual framework with scoring rule
Assess score reliability, construct validity, & ability to
detect change
Finalize instrument content, formats, scoring,
procedures & training materials
Document measurement development
Refine and finalize PRO-based label claim
Emphasis on this stage is quantitative activity:


Confirming scoring rule, assessing psychometric
properties
Stage 3: Finalize PRO
Confirm Conceptual Framework & Assess Other
Measurement Properties






Confirm conceptual framework with scoring rule
Assess score reliability, construct validity, & ability to
detect change
Finalize instrument content, formats, scoring,
procedures & training materials
Document measurement development
Refine and finalize PRO-based label claim
Emphasis on this stage is quantitative activity:


Confirming scoring rule, assessing psychometric
properties
Stage 3: Finalize PRO
Confirm Conceptual Framework & Assess Other
Measurement Properties






Confirm conceptual framework with scoring rule
Assess score reliability, construct validity, & ability to
detect change
Finalize instrument content, formats, scoring,
procedures & training materials
Document measurement development
Refine and finalize PRO-based label claim
Emphasis on this stage is quantitative activity:


Confirming scoring rule, assessing psychometric
properties
Stage 3: Finalize PRO
Confirm Conceptual Framework & Assess Other
Measurement Properties






Confirm conceptual framework with scoring rule
Assess score reliability, construct validity, & ability to
detect change
Finalize instrument content, formats, scoring,
procedures & training materials
Document measurement development
Refine and finalize PRO-based label claim
Emphasis on this stage is quantitative activity:


Confirming scoring rule, assessing psychometric
properties
Stage 3: Finalize PRO
Confirm Conceptual Framework & Assess Other
Measurement Properties






Confirm conceptual framework with scoring rule
Assess score reliability, construct validity, & ability to
detect change
Finalize instrument content, formats, scoring,
procedures & training materials
Document measurement development
Refine and finalize PRO-based label claim
Emphasis on this stage is quantitative activity:


Confirming scoring rule, assessing psychometric
properties of the PRO instrument
Stage 3: Finalize PRO
Confirm Conceptual Framework & Assess Other
Measurement Properties






Confirm conceptual framework with scoring rule
Assess score reliability, construct validity, & ability to
detect change
Finalize instrument content, formats, scoring,
procedures & training materials
Document measurement development
Refine and finalize PRO-based label claim
Emphasis on this stage is quantitative activity:


Confirming scoring rule, assessing psychometric
properties
Measurement Properties of
a PRO Instrument

To ensure that the instrument measures the
intended concept, we need to demonstrate
the instrument’s:


Reliability
 Internal consistency reliability
 Test-retest reliability
Validity
 Content validity
 Construct validity
Measurement Properties of
a PRO Instrument

To ensure that the instrument measures the
intended concept, we need to demonstrate
the instrument’s:


Reliability
 Internal consistency reliability
 Test-retest reliability
Validity
 Content validity
 Construct validity
Measurement Properties of
a PRO Instrument

To ensure that the instrument measures the
intended concept, we need to demonstrate
the instrument’s:


Reliability
 Internal consistency reliability
 Test-retest reliability
Validity
 Content validity
 Construct validity
Measurement Properties of
a PRO Instrument

To ensure that the instrument measures the
intended concept, we need to demonstrate
the instrument’s:


Reliability
 Internal consistency reliability
 Test-retest reliability
Validity
 Content validity
 Construct validity
Begin here
1. Demonstrate PRO’s
Content Validity


Refers to the
extent to which an
instrument includes
or represents all of
the content of a
particular construct
for the target
population
It relies on patient
input from the
target population,
literature review,
and expert opinion
Basic Activities of Daily Living
Hygiene
Bathe
Dress
Feed self
Toilet
Walk
Transfer
from bed
to chair
Continence
1. Demonstrate PRO’s
Content Validity


Refers to the
extent to which an
instrument includes
or represents all of
the content of a
particular construct
for the target
population
It relies on patient
input from the
target population,
literature review,
and expert opinion
Basic Activities of Daily Living
Hygiene
Bathe
Dress
Feed self
Toilet
Walk
Transfer
from bed
to chair
Continence
1. Demonstrate PRO’s
Content Validity

Instrument content must be appropriate
for its target population

Example: An instrument to assess physical
functioning


For pts w rheumatoid arthritis, need to include
items that assess ability to open doors or open
jars
For pts w congestive heart failure, need to
include items that assess ability to walk
distances or climb stairs.
1. Demonstrate PRO’s
Content Validity

Instrument content must be appropriate
for its target population

Example: An instrument to assess physical
functioning


For pts w rheumatoid arthritis, need to include
items that assess ability to open doors or open
jars
For pts w congestive heart failure, need to
include items that assess ability to walk
distances or climb stairs.
1. Demonstrate PRO’s
Content Validity

Instrument content must be appropriate
for its target population

Example: An instrument to assess physical
functioning


For pts w rheumatoid arthritis, need to include
items that assess ability to open doors or open
jars
For pts w congestive heart failure, need to
include items that assess ability to walk
distances or climb stairs.
1. Demonstrate PRO’s
Content Validity

Established qualitative methods

There are established methods that assess patient
input utilized to ensure that all relevant concepts are
identified



focus groups
patient input
From this,


Items capturing identified concepts can then be developed
and incorporated into a new PRO instrument.

Staying close to the actual language used by pts themselves
1. Demonstrate PRO’s
Content Validity

Established qualitative methods

There are established methods that assess patient
input utilized to ensure that all relevant concepts are
identified



focus groups
patient input
From this,


Items capturing identified concepts can then be developed
and incorporated into a new PRO instrument

Staying close to the actual language used by pts themselves
2. Demonstrate PRO’s
Internal Consistency Reliability

Internal-consistency reliability
 Refers to the extent to which the items in a
domain are intercorrelated


As all items in a domain are defined as
contributing to the concept, items are
expected to intercorrelate
This is captured with a Cronbach’s alpha
coefficient
It ranges from 0 to 1.0
 0.7 is considered minimally acceptable

2. Demonstrate PRO’s
Internal Consistency Reliability

Internal-consistency reliability
 Refers to the extent to which the items in a
domain are intercorrelated


As all items in a domain are defined as
contributing to the concept, items are
expected to intercorrelate
This is captured with a Cronbach’s alpha
coefficient
It ranges from 0 to 1.0
 0.7 is considered minimally acceptable

2. Demonstrate PRO’s
Internal Consistency Reliability

Internal-consistency reliability
 Refers to the extent to which the items in a
domain are intercorrelated


As all items in a domain are defined as
contributing to the concept, items are
expected to intercorrelate
This is captured with a Cronbach’s alpha
coefficient
It ranges from 0 to 1.0
 0.7 is considered minimally acceptable

3. Demonstrate PRO’s
Test-Retest Reliability

Test-retest reliability

Refers to stability of scores over time when no
change has occurred in the concept of interest




Of subjects who have not changed in status of target
disease, how stable are their PRO scores over time?
Eg, of Ss who remain the same in their level of severity of
rheumatoid arthritis, have their scores in physical functioning
stayed the same through time?
Very important in clinical trials that involve change
over time
Most informative when the time interval between the
test and the retest is appropriate for determining
stability
3. Demonstrate PRO’s
Test-Retest Reliability

Test-retest reliability

Refers to stability of scores over time when no
change has occurred in the concept of interest




Of subjects who have not changed in status of target
disease, how stable are their PRO scores over time?
Eg, of Ss who remain the same in their level of severity of
rheumatoid arthritis, have their scores in physical functioning
stayed the same through time?
Very important in clinical trials that involve change
over time
Most informative when the time interval between the
test and the retest is appropriate for determining
stability
3. Demonstrate PRO’s
Test-Retest Reliability

Test-retest reliability

Refers to stability of scores over time when no
change has occurred in the concept of interest




Of subjects who have not changed in status of target
disease, how stable are their PRO scores over time?
Eg, of Ss who remain the same in their level of severity of
rheumatoid arthritis, have their scores in physical functioning
stayed the same through time?
Very important in clinical trials that involve change
over time
Most informative when the time interval between the
test and the retest is appropriate for determining
stability
3. Demonstrate PRO’s
Test-Retest Reliability

Test-retest reliability

Refers to stability of scores over time when no
change has occurred in the concept of interest




Of subjects who have not changed in status of target
disease, how stable are their PRO scores over time?
Eg, of Ss who remain the same in their level of severity of
rheumatoid arthritis, have their scores in physical functioning
stayed the same through time?
Very important in clinical trials that involve change
over time
Most informative when the time interval between the
test and the retest is appropriate for determining
stability
4. Demonstrate PRO’s
Construct Validity

Construct Validity
Is central to the measurement of abstract
theoretical concepts
 Is concerned with the extent to which a
particular measure relates to other measures
in a manner consistent with theoretically
derived hypotheses concerning the concepts
(constructs) that are being measured.


It is the total configuration of empirical evidence
that lends credence to a PRO Instrument’s
construct validity
4. Demonstrate PRO’s
Construct Validity

Construct Validity
Is central to the measurement of abstract
theoretical concepts
 Is concerned with the extent to which a
particular measure relates to other measures
in a manner consistent with theoretically
derived hypotheses concerning the concepts
(constructs) that are being measured.


It is the total configuration of empirical evidence
that lends credence to a PRO Instrument’s
construct validity
4. Demonstrate PRO’s
Construct Validity

Construct Validity
Is central to the measurement of abstract
theoretical concepts
 Is concerned with the extent to which a
particular measure relates to other measures
in a manner consistent with theoretically
derived hypotheses concerning the concepts
(constructs) that are being measured.


It is the total configuration of empirical evidence
that lends credence to a PRO Instrument’s
construct validity
4. Demonstrate PRO’s
Construct Validity

Construct Validity
Is central to the measurement of abstract
theoretical concepts
 Is concerned with the extent to which a
particular measure relates to other measures
in a manner consistent with theoretically
derived hypotheses concerning the concepts
(constructs) that are being measured.


It is the total configuration of empirical evidence
that lends credence to a PRO Instrument’s
construct validity
Endpoint Model Example:
Rheumatoid Arthritis
Pain
Intensity
Mobility
Swollen
Joints
Social
Function/
Activities
Pathophysiologic
Mechanism
Fatigue
HealthRelated
Quality of
Life
Activities of
Daily Living
Tender
Joints
Emotional
Well-Being
Sleep
Problems
For example, if the physical
function PRO is in fact measuring
physical function, there are
certain relationships that should
be observed among these
endpoints.
Physical
Function
Adapted from Revicki, 2006
4. Demonstrate PRO’s
Construct Validity

Generates hypotheses for validation Re. internal
structure of the PRO instrument
 Item to concept(s) relationships:
This is based on the PRO’s conceptual framework
 Multi-trait scaling




Item convergence: whether each item correlates
substantially with the scale it is a part of
Item discrimination: whether each item correlates
significantly higher with the scale it is part of than with
other conceptually similar scales
Concept 1 to Concept 2 relationship:
Relationships among PRO domains
(concepts)  Based on endpoint model

Convergent Validity, Discriminant Validity
4. Demonstrate PRO’s
Construct Validity

Generates hypotheses for validation Re. internal
structure of the PRO instrument

Concept 1 to Concept 2 relationship:

Relationships among PRO domains (concepts)  Based on
endpoint model


Convergent Validity, Discriminant Validity
Item to concept(s) relationships:
This is based on the PRO’s conceptual framework
 Multi-trait scaling



Item convergence: whether each item correlates
substantially with the scale it is a part of
Item discrimination: whether each item correlates
significantly higher with the scale it is part of than with
other conceptually similar scales
Endpoint Model Example:
Rheumatoid Arthritis
Pain
Intensity
Mobility
Swollen
Joints
Social
Function/
Activities
Pathophysiologic
Mechanism
Fatigue
HealthRelated
Quality of
Life
Activities of
Daily Living
Tender
Joints
Emotional
Well-Being
Sleep
Problems
For example, if the physical
function PRO is in fact measuring
physical function, there are
certain relationships that should
be observed among these
endpoints.
Physical
Function
Adapted from Revicki, 2006
4. Demonstrate PRO’s
Construct Validity

Generates hypotheses for validation Re. internal
structure of the PRO instrument

Concept 1 to Concept 2 relationship:

Relationships among PRO domains (concepts)  Based on
endpoint model


Convergent Validity, Discriminant Validity
Item to concept(s) relationships:
This is based on the PRO’s conceptual framework
 Multi-trait scaling



Item convergence: whether each item correlates
substantially with the scale it is a part of
Item discrimination: whether each item correlates
significantly higher with the scale it is part of than with
other conceptually similar scales
4. Demonstrate PRO’s
Construct Validity

Generates hypotheses for validation Re. internal
structure of the PRO instrument

Concept 1 to Concept 2 relationship:

Relationships among PRO domains (concepts)  Based on
endpoint model


Convergent Validity, Discriminant Validity
Item to concept(s) relationships:
This is based on the PRO’s conceptual framework
 Multi-trait scaling



Item convergence: whether each item correlates
substantially with the scale it is a part of
Item discrimination: whether each item correlates
significantly higher with the scale it is part of than with
other conceptually similar scales
4. Demonstrate PRO’s
Construct Validity

Generates hypotheses for validation Re. internal
structure of the PRO instrument

Concept 1 to Concept 2 relationship:

Relationships among PRO domains (concepts)  Based on
endpoint model


Convergent Validity, Discriminant Validity
Item to concept(s) relationships:
This is based on the PRO’s conceptual framework
 Multi-trait scaling



Item convergence: whether each item correlates
substantially with the scale it is a part of
Item discrimination: whether each item correlates
significantly higher with the scale it is part of than with
other conceptually similar scales
Example: Conceptual Framework-SF-36
EXAMPLE:
This
conceptual
framework
shows
expected
relationships
between
items
measuring 8
concepts
summarized
into 2 general
concepts.
Source: http://www.sf-36.org/tools/SF36.shtml
4. Demonstrate PRO’s
Construct Validity

Generates hypotheses for validation Re.
relationships of PRO concepts with clinical
measures
Generate hypotheses of how the PRO concepts
captured by the PRO instrument must relate to
clinical measures if indeed the concept measured
is the concept intended
 Known-groups Validity



The usefulness of a measure in distinguishing among
groups of people with “known” characteristics
Eg, in rheumatoid arthritis, ACR-criteria 20, 50, 70%
responders must have physical function scores that are
monotonically increasing reflecting greater improvement
4. Demonstrate PRO’s
Construct Validity

Generates hypotheses for validation Re.
relationships of PRO concepts with clinical
measures
hypotheses of how the PRO concepts captured by
the PRO instrument must relate to clinical
measures if indeed the concept measured is the
concept intended
 Known-groups Validity



The usefulness of a measure in distinguishing among
groups of people with “known” characteristics
Eg, in rheumatoid arthritis, ACR-criteria 20, 50, 70%
responders must have physical function scores that are
monotonically increasing reflecting greater improvement
4. Demonstrate PRO’s
Construct Validity

Generates hypotheses for validation Re.
relationships of PRO concepts with clinical
measures
hypotheses of how the PRO concepts captured by
the PRO instrument must relate to clinical
measures if indeed the concept measured is the
concept intended
 Known-groups Validity



The usefulness of a measure in distinguishing among
groups of people with “known” characteristics
Eg, in rheumatoid arthritis, ACR-criteria 20, 50, 70%
responders must have physical function scores that are
monotonically increasing reflecting greater improvement
Endpoint Model Example:
Rheumatoid Arthritis
Pain
Intensity
Mobility
Swollen
Joints
Pathophysiologic
Mechanism
Social
Function/
Activities
Fatigue
Tender
Joints
Emotional
Well-Being
Sleep
Problems
For example, in this end point
model for RA, we see
relationships among endpoints
that indicate that changes in the
severity of RA (left) should have
an impact on PRO concepts
(right), including Physical
Function.
HealthRelated
Quality of
Life
Activities of
Daily Living
Physical
Function
Adapted from Revicki, 2006
Example of Known-Groups Validity:
Physical Function PROs Demonstrate
Responsiveness to Changes in Disease Severity
Baseline to 6 and 12 Months Change Scores in Physical Functioning across Levels of ACR
Improvement Criteria.
Three measures of
physical function
ACR at 6 Months (N=262)
Greater Improvement
ACR<20
20-49
MHAQ
1.87
9.19
PF-10
2.34
5.58
IRT-based
2.14
7.93
MHAQ
2.24
6.71
PF-10
2.54
4.68
IRT-based
2.42
6.18
50-69
ACR 70+
p-value
Data from clinical trial
of Orencia on RA
F-ratio
RV
As part of the construct
10.06
11.75
<.0001
23.86of measures
0.71
validation
of physical function, we
9.81
11.58
<.0001
17.77
0.53
identify “known groups”
of patients
11.35
13.76
<.0001
33.78 w various
1.00
degrees of improvement
ACR at 12 Months (N=227)
in a RA and see if
9.01
11.84
<.0001
0.70
degree15.87
of improvement
in PRO score increases
7.54
11.85
<.0001
12.30
0.54
with degree of RA
improvement.
8.82
14.76
<.0001
22.62
1.00
4. Demonstrate PRO’s
Construct Validity

Construct Validity



Is evaluated after Content Validity is established
Must be evaluated within the specific measurement
application considered (e.g., population and condition
in the clinical trial)
Validation study results are deemed adequate if:


Consistent with what was hypothesized
Testing is focused on the ability of the instrument to
measure the concept represented in the desired claim
4. Demonstrate PRO’s
Construct Validity

Construct Validity



Is evaluated after Content Validity is established
Must be evaluated within the specific measurement
application considered (e.g., population and condition
in the clinical trial)
Validation study results are deemed adequate if:


Consistent with what was hypothesized
Testing is focused on the ability of the instrument to
measure the concept represented in the desired claim
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
What is the goal of medical treatment?
What is a PRO?
Reminder
Why use PRO measures?
of where
How do PRO measures complement traditional
we are
clinical outcome measures?
What characterizes PRO measures?
 How do PROs differ?
What characterizes a well-constructed PRO
instrument?
 How are PRO instruments developed?
What is the process necessary to achieve a PRO
label?
 Putting it all together
What is the role of PROs in clinical trials research?
Review and conclusions
i.
Hypothesize Conceptual Framework
•
•
•
•
•
•
Outline hypothesized concepts & potential claims
Determine intended population
Perform literature/expert review
Develop hypothesized conceptual framework
Place PROs within preliminary endpoint model
Document preliminary instrument development
v. Modify Instrument
•
•
•
•
Change wording of items,
populations, response
options, recall period, or
method of administration
Translate & culturally
adapt to other languages
Evaluate as appropriate
Document all changes
ii. Adjust Conceptual
Framework & Draft
Instrument
PRO
Claim
iv. Collect, Analyze, &
Interpret Data
•
•
•
•
Prepare protocol & statistical
analysis plan
Collect & analyze data
Evaluate treatment response using
cumulative distribution & responder
definition
Document interpretation of
treatment benefit in relation to claim
Burke & Patrick,
FDA/Industry
Statistics
Workshop, 2007
•
•
•
•
•
•
•
Generate new items
Create instrument
Select administration mode,
recall period & response
options
Format instrument
Conduct cognitive debriefing
Pilot test draft instrument
Document content validity
iii. Confirm Conceptual Framework &
Assess Other Measurement Properties
•
•
•
•
Confirm conceptual framework with scoring rule
Assess score reliability, construct validity, & ability to
detect change
Finalize instrument content, formats, scoring, procedures
& training materials
Document measurement development
Stage 4: Utilize PRO in Clinical Trial
Collect, Analyze, and Interpret Data





Prepare protocol and SAP (statistical analysis plan)
Collect and analyze the trial data
Evaluate treatment response
Document interpretation of treatment benefit in relation
to claim
Emphasis on this stage is quantitative activity:


Write analysis plan, analyze the data
Stage 4: Utilize PRO in Clinical Trial
Collect, Analyze, and Interpret Data





Prepare protocol and SAP (statistical analysis plan)
Collect and analyze the trial data
Evaluate treatment response
Document interpretation of treatment benefit in relation
to claim
Emphasis on this stage is quantitative activity:


Write analysis plan, analyze the data
Stage 4: Utilize PRO in Clinical Trial
Collect, Analyze, and Interpret Data





Prepare protocol and SAP (statistical analysis plan)
Collect and analyze the trial data
Evaluate treatment response
Document interpretation of treatment benefit in relation
to claim
Emphasis on this stage is quantitative activity:


Write analysis plan, analyze the data
Some General Analyses



Assess and demonstrate adequate
measurement properties

Internal consistency reliability

Test-retest reliability among stable patients
Demonstrate that scores change through time
and not equally for all groups

Repeated measures analyses

Analyses of baseline-endpoint change scores
Demonstrate treatment benefit on PRO scores
Some General Analyses



Assess and demonstrate adequate
measurement properties

Internal consistency reliability

Test-retest reliability among stable patients
Demonstrate that scores change through time
and not equally for all groups

Repeated measures analyses

Analyses of baseline-endpoint change scores
Demonstrate treatment benefit on PRO scores
Some General Analyses



Assess and demonstrate adequate
measurement properties

Internal consistency reliability

Test-retest reliability among stable patients
Demonstrate that scores change through time
and not equally for all groups

Repeated measures analyses

Analyses of baseline-endpoint change scores
Demonstrate treatment benefit on PRO scores
i.
Hypothesize Conceptual Framework
•
•
•
•
•
•
Outline hypothesized concepts & potential claims
Determine intended population
Perform literature/expert review
Develop hypothesized conceptual framework
Place PROs within preliminary endpoint model
Document preliminary instrument development
v. Modify Instrument
•
•
•
•
Change wording of items,
populations, response
options, recall period, or
method of administration
Translate & culturally
adapt to other languages
Evaluate as appropriate
Document all changes
ii. Adjust Conceptual
Framework & Draft
Instrument
PRO
Claim
iv. Collect, Analyze, &
Interpret Data
•
•
•
•
Prepare protocol & statistical
analysis plan
Collect & analyze data
Evaluate treatment response using
cumulative distribution & responder
definition
Document interpretation of
treatment benefit in relation to claim
Burke & Patrick,
FDA/Industry
Statistics
Workshop, 2007
•
•
•
•
•
•
•
Generate new items
Create instrument
Select administration mode,
recall period & response
options
Format instrument
Conduct cognitive debriefing
Pilot test draft instrument
Document content validity
iii. Confirm Conceptual Framework &
Assess Other Measurement Properties
•
•
•
•
Confirm conceptual framework with scoring rule
Assess score reliability, construct validity, & ability to
detect change
Finalize instrument content, formats, scoring, procedures
& training materials
Document measurement development
Stage 5: Modify PRO Instrument
if Needed
Modify Instrument




Change wording of items, populations, response
options, recall period, or method of administration
Translate & culturally adapt to other languages
Document all changes
Emphasis on this stage can be both qualitative
and quantitative:



Qualitative (changing wording of items, translating
items)
Quantitative (psychometric work to demonstrate crosslinguistic equivalence of translated versions of PRO)
Stage 5: Modify PRO Instrument
if Needed
Modify Instrument




Change wording of items, populations, response
options, recall period, or method of administration
Translate & culturally adapt to other languages
Document all changes
Emphasis on this stage can be both qualitative
and quantitative:



Qualitative (changing wording of items, translating
items)
Quantitative (psychometric work to demonstrate crosslinguistic equivalence of translated versions of PRO)
Stage 5: Important to Validate PRO
Modifications

If PRO is modified for trial, modification must
be validated:






Deleted or added items or domains
Changed mode of administration
Translated or adapted for use in another population
Altered recall periods or response options
Changed instructions, procedures, or significantly altered
formats
Revised order of questions or assessed as part of a
battery
Stage 5: Important to Validate PRO
Modifications

Evidence of validity of new version must
be demonstrated:
New version reflects the conceptual framework
 New version supports full patient response (no
missing data)
 Measurement properties are retained to support
pooling of data across study sites using different
versions of formats

More recent
and detailed
steps from
PRO dev’t to
PRO claims
i.
Hypothesize Conceptual Framework
•
•
•
•
•
•
Outline hypothesized concepts & potential claims
Determine intended population
Perform literature/expert review
Develop hypothesized conceptual framework
Place PROs within preliminary endpoint model
Document preliminary instrument development
v. Modify Instrument
•
•
•
•
Change wording of items,
populations, response
options, recall period, or
method of administration
Translate & culturally
adapt to other languages
Evaluate as appropriate
Document all changes
ii. Adjust Conceptual
Framework & Draft
Instrument
PRO
Claim
iv. Collect, Analyze, &
Interpret Data
•
•
•
•
Prepare protocol & statistical
analysis plan
Collect & analyze data
Evaluate treatment response using
cumulative distribution & responder
definition
Document interpretation of
treatment benefit in relation to claim
Burke & Patrick,
FDA/Industry
Statistics
Workshop, 2007
•
•
•
•
•
•
•
Generate new items
Create instrument
Select administration mode,
recall period & response
options
Format instrument
Conduct cognitive debriefing
Pilot test draft instrument
Document content validity
iii. Confirm Conceptual Framework &
Assess Other Measurement Properties
•
•
•
•
Confirm conceptual framework with scoring rule
Assess score reliability, construct validity, & ability to
detect change
Finalize instrument content, formats, scoring, procedures
& training materials
Document measurement development
What defines a well-constructed
PRO instrument? (Revisited)

The challenge is to define and capture (operationalize)
the target concept well via

sound qualitative research


sound quantitative methods


Patient input, Literature review, Expert opinion to establish
CONTENT VALIDITY
Appropriate psychometric properties and construct validation to
establish CONSTRUCT VALIDITY
FDA seeks that the instrument be developed according
to this process for


Newly-developed instruments
Pre-existing instruments
What defines a well-constructed
PRO instrument? (Revisited)

The challenge is to define and capture (operationalize)
the target concept well via

sound qualitative research


sound quantitative methods


Patient input, Literature review, Expert opinion to establish
CONTENT VALIDITY
Appropriate psychometric properties and construct validation to
establish CONSTRUCT VALIDITY
FDA seeks that the instrument be developed according
to this process for


Newly-developed instruments
Pre-existing instruments
What defines a well-constructed
PRO instrument? (Revisited)

The challenge is to define and capture (operationalize)
the target concept well via

sound qualitative research


sound quantitative methods


Patient input, Literature review, Expert opinion toVery
establish
CONTENT VALIDITY
Important
Appropriate psychometric properties and construct validation to
establish CONSTRUCT VALIDITY
FDA seeks that the instrument be developed according
to this process for


Newly-developed instruments
Pre-existing instruments
Tired?
Hang in there
Almost there
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
What is the goal of medical treatment?
What is a PRO?
Why use PRO measures?
How do PRO measures complement traditional
clinical outcome measures?
What characterizes PRO measures?
 How do PROs differ?
What characterizes a well-constructed PRO
instrument?
 How are PRO instruments developed?
What is the process necessary to achieve a PRO
label?
 Putting it all together
What is the role of PROs in clinical trials research?
Review and conclusions
What is the process
necessary to achieve
a PRO label?
Putting it all together
Warning
 “If
you don’t know
where you are going,
any road will get you
there.”

Lewis Carroll
Warning
 “If
you don’t know
where you are going,
any road will get you
there.”

Lewis Carroll
Begin with the End in Mind
We want to
start with the
end in mind
and know the
path that
leads us
there.
FDA
Label
Strategic Planning for Successful
PRO Labels


Think about what you are going to do before you
do it
 Consistent with good science
 Ask: What statement or implication of tx
benefit can be captured w PRO?
FDA: Label claim drives measurement strategy
 Desired label statements  PRO instrument
Adapted from Revicki, 2006; Rock, FDA 2006
Strategic Planning for Successful
PRO Labels


Think about what you are going to do before you
do it
 Consistent with good science
 Ask: What statement or implication of tx
benefit can be captured w PRO?
FDA: Label claim drives measurement strategy
 Desired label statements  PRO instrument
Adapted from Revicki, 2006; Rock, FDA 2006
Successful Endpoint Model 
Successful PRO Measurement Strategy 
Successful PRO label claim

Key steps:



Begin with the
end in mind

Understand disease area and expected tx effects
Identify outcome concepts relevant to patient and
disease area
Articulate the endpoint model
Articulate desired PRO-based label statement(s)
Select/develop PRO instruments to measure target
concepts (domains) of interest



Articulate conceptual framework
Implement PROs in well-designed clinical trials informed
by endpoint model
Communicate early and often with FDA about plans
Adapted from Revicki, ISOQOL, 2006
Successful Endpoint Model 
Successful PRO Measurement Strategy 
Successful PRO label claim

Key steps:



Begin with the
end in mind

Understand disease area and expected tx effects
Identify outcome concepts relevant to patient and
disease area
Articulate the endpoint model
Articulate desired PRO-based label statement(s)
Select/develop PRO instruments to measure target
concepts (domains) of interest



Articulate conceptual framework
Implement PROs in well-designed clinical trials informed
by endpoint model
Communicate early and often with FDA about plans
Adapted from Revicki, ISOQOL, 2006
Successful Endpoint Model 
Successful PRO Measurement Strategy 
Successful PRO label claim

Key steps:



Begin with the
end in mind

Understand disease area and expected tx effects
Identify outcome concepts relevant to patient and
disease area
Articulate the endpoint model
Articulate desired PRO-based label statement(s)
Select/develop PRO instruments to measure target
concepts (domains) of interest



Articulate conceptual framework
Implement PROs in well-designed clinical trials informed
by endpoint model
Communicate early and often with FDA about plans
Adapted from Revicki, ISOQOL, 2006
Successful Endpoint Model 
Successful PRO Measurement Strategy 
Successful PRO label claim

Key steps:



Begin with the
end in mind



Understand disease area and expected tx effects
Identify outcome concepts relevant to patient and
disease area
Articulate the endpoint model
Articulate desired PRO-based label statement(s)
Select/develop PRO instruments to measure target
concepts (domains) of interest
 Articulate conceptual framework
Implement PROs in well-designed clinical trials informed
by endpoint model
Communicate early and often with FDA about plans
Adapted from Revicki, ISOQOL, 2006
Successful Endpoint Model 
Successful PRO Measurement Strategy 
Successful PRO label claim

Key steps:



Begin with the
end in mind



Understand disease area and expected tx effects
Identify outcome concepts relevant to patient and
disease area
Articulate the endpoint model
Articulate desired PRO-based label statement(s)
Select/develop PRO instruments to measure target
concepts (domains) of interest
 Articulate conceptual framework
Implement PROs in well-designed clinical trials informed
by endpoint model
Communicate early and often with FDA about plans
Adapted from Revicki, ISOQOL, 2006
Successful Endpoint Model 
Successful PRO Measurement Strategy 
Successful PRO label claim

Key steps:



Begin with the
end in mind



Understand disease area and expected tx effects
Identify outcome concepts relevant to patient and
disease area
Articulate the endpoint model
Articulate desired PRO-based label statement(s)
Select/develop PRO instruments to measure target
concepts (domains) of interest
 Articulate conceptual framework
Implement PROs in well-designed clinical trials informed
by endpoint model
Communicate early and often with FDA about plans
Adapted from Revicki, ISOQOL, 2006
Endpoint Model Example:
Rheumatoid Arthritis
Pain
Intensity
Mobility
Swollen
Joints
Pathophysiologic
Mechanism
Social
Function/
Activities
Fatigue
HealthRelated
Quality of
Life
Activities of
Daily Living
Tender
Joints
Emotional
Well-Being
Sleep
Problems
Physical
Function
Adapted from Revicki, 2006
Successful Endpoint Model 
Successful PRO Measurement Strategy 
Successful PRO label claim

Key steps:



Begin with the
end in mind



Understand disease area and expected tx effects
Identify outcome concepts relevant to patient and
disease area
Articulate the endpoint model
Articulate desired PRO-based label statement(s)
Select/develop PRO instruments to measure target
concepts (domains) of interest
 Articulate conceptual framework
Implement PROs in well-designed clinical trials informed
by endpoint model
Communicate early and often with FDA about plans
Adapted from Revicki, ISOQOL, 2006
Successful Endpoint Model 
Successful PRO Measurement Strategy 
Successful PRO label claim

Key steps:



Begin with the
end in mind



Understand disease area and expected tx effects
Identify outcome concepts relevant to patient and
disease area
Articulate the endpoint model
Articulate desired PRO-based label statement(s)
Select/develop PRO instruments to measure target
concepts (domains) of interest
 Articulate conceptual framework
Implement PROs in well-designed clinical trials informed
by endpoint model
Communicate early and often with FDA about plans
Adapted from Revicki, ISOQOL, 2006
Linking Claims, Concepts in Claims,
& PRO Endpoints
Example 1
Desired claim
Drug Y relieves
pain and
improves ability
to function
physically and
carry out daily
activities
Concepts in
claim
PRO
instruments
used
Pain
VAS Pain
Physical
Function
HAQ
Activities of
Daily Living
ADL
Questionnaire
Linking Claims, Concepts in Claims,
& PRO Endpoints
Adapted from J. Scott, Study Endpoints & Label Dev’t, FDA 2004
Example 2
Desired claim
Velpaz relieves
pain quicker
without
upsetting your
stomach
Concepts in
claim
Pain
Relief
Time to
Onset
Stomach
Upset
PRO
instruments
used
Pain Diary
Time to
Pain Relief
GI-symptom
diary
Linking Claims, Concepts in Claims,
& PRO Endpoints
Adapted from Patrick et al., Value in Health, 2008
Example 3
Desired
claim
Product X
reduces
problems with
swallowing &
speaking to
others and
improves daily
activities
PRO
concept
PRO instrument
Swallowing
Swallowing diary &
retrospective report
Speaking to
others
Conversation diary
Daily activities
Activities of Daily
Living
REMEMBER:
PRO’s Conceptual Framework
A conceptual framework refers to the expected relationships
between items, domains, and measurement concepts.
The domains
and overall
score
represent
Item A
Domain
related but
Item B
Score 1
separate
Item C
concepts
Overall
Score
Item D
Domain
Item E
Score 2
Item F
Item G
It is a description of how each item
relates to the intended measurement
concepts (measurement model).
Source: FDA Guidance, 2006; Rock, FDA, 206
Why develop an Endpoint Model
and Conceptual Framework?

Because doing so ensures that researchers
identify and define

What they intend to measure—the PRO concept



How they will measure it—the PRO instrument



Identifies concepts to be measured—both clinical and PRO
These are the goals of treatment
Guides instrument selection/development to support claims
Clarifies how each concept is measured by each set of items
Why they are measuring it—the PRO-based label claim

Clarifies goals for labeling claims
Adapted from Rock, Medical Officer, Office of Oncology Drug Products, FDA, 2006
Why develop an Endpoint Model
and Conceptual Framework?

Because doing so ensures that researchers
identify and define

What they intend to measure—the PRO concept



How they will measure it—the PRO instrument



Identifies concepts to be measured—both clinical and PRO
These are the goals of treatment
Guides instrument selection/development to support claims
Clarifies how each concept is measured by each set of items
Why they are measuring it—the PRO-based label claim

Clarifies goals for labeling claims
Adapted from Rock, Medical Officer, Office of Oncology Drug Products, FDA, 2006
Why develop an Endpoint Model
and Conceptual Framework?

Because doing so ensures that researchers
identify and define

What they intend to measure—the PRO concept



How they will measure it—the PRO instrument



Identifies concepts to be measured—both clinical and PRO
These are the goals of treatment
Guides instrument selection/development to support claims
Clarifies how each concept is measured by each set of items
Why they are measuring it—the PRO-based label claim

Clarifies goals for labeling claims
Adapted from Rock, Medical Officer, Office of Oncology Drug Products, FDA, 2006
Successful Endpoint Model &
Measurement of Endpoint Concepts
Important for Successful Label Claims

Must ensure that the PRO we select or develop
is measuring the right target concept in claim.
 First, ensure you are measuring the right
concept (content validity).
 Then, ensure PRO instrument has strong
measurement properties (construct validity)
Powers, FDA, 2006
Successful Endpoint Model &
Measurement of Endpoint Concepts
Important for Successful Label Claims

Must ensure that the PRO we select or develop
is measuring the right target concept in claim.
 First, ensure you are measuring the right
concept (content validity).
 Then, ensure PRO instrument has strong
measurement properties (construct validity)
Powers, FDA, 2006
Successful Endpoint Model &
Measurement of Endpoint Concepts
Important for Successful Label Claims

Must ensure that the PRO we select or develop
is measuring the right target concept in claim.
 First, ensure you are measuring the right
concept (content validity).
 Then, ensure PRO instrument has strong
measurement properties (construct validity)
Powers, FDA, 2006
Successful Endpoint Model &
Measurement of Endpoint Concepts
Important for Successful Label Claims

Must ensure that the PRO we select or develop
is measuring the right target concept in claim.
 First, ensure you are measuring the right
concept (content validity).
 Then, ensure PRO instrument has strong
measurement properties (construct validity)
Powers, FDA, 2006
Successful Endpoint Model &
Measurement of Endpoint Concepts
Important for Successful Label Claims

“It is often much worse to have a good
measurement of the wrong thing—especially
when, as is so often the case, the wrong thing
will IN FACT be used as an indicator of the right
thing—than to have poor measurement of the
right thing.”
J. Tukey
Quoted by Powers, Office of Med.Policy, FDA, 2006
Successful Endpoint Model 
Successful PRO Measurement Strategy 
Successful PRO label claim

Key steps:



Begin with the
end in mind



Understand disease area and expected tx effects
Identify outcome concepts relevant to patient and
disease area
Articulate the endpoint model
Articulate desired PRO-based label statement(s)
Select/develop PRO instruments to measure target
concepts (domains) of interest
 Articulate conceptual framework
Implement PROs in well-designed clinical trials informed
by endpoint model
Communicate early and often with FDA about plans
Adapted from Revicki, ISOQOL, 2006
Successful Endpoint Model 
Successful PRO Measurement Strategy 
Successful PRO label claim

Key steps:



Begin with the
end in mind



Understand disease area and expected tx effects
Identify outcome concepts relevant to patient and
disease area
Articulate the endpoint model
Articulate desired PRO-based label statement(s)
Select/develop PRO instruments to measure target
concepts (domains) of interest
 Articulate conceptual framework
Implement PROs in well-designed clinical trials informed
by endpoint model
Communicate early and often with FDA about plans
Adapted from Revicki, ISOQOL, 2006
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
What is the goal of medical treatment?
What is a PRO?
Why use PRO measures?
How do PRO measures complement traditional
clinical outcome measures?
What characterizes PRO measures?
 How do PROs differ?
What characterizes a well-constructed PRO
instrument?
 How are PRO instruments developed?
What is the process necessary to achieve a PRO
label?
 Putting it all together
What is the role of PROs in clinical trials research?
Review and conclusions
What is the Role
of PROs in
Clinical Trials
Research?
What is the Role of PROs?


To capture the patient’s voice about their
 illness and
 treatment intervention
To complement biological & physiological
outcomes to form a comprehensive
assessment of the patient.
WILSON-CLEARY MODEL OF HEALTH OUTCOMES:
Where do PROs fit in a greater model of health outcomes?
Characteristics of Individual
Biological and
Physiological
Variables
Symptom
Status
Functional
Status
General Health
Perceptions
Characteristics of Environment
Wilson & Cleary JAMA (1995); Adapted from Revicki, 2006
Quality of Life
WILSON-CLEARY MODEL OF HEALTH OUTCOMES:
Where do PROs
fit in a greater
model of health outcomes?
IMPORTANT
TO REMEMBER:
Impact of DISEASE and of TREATMENT can be
Characteristics
captured
at each ofofIndividual
these boxes
And so are part of comprehensive assessment
Biological and
Physiological
Variables
Symptom
Status
Functional
Status
General Health
Perceptions
Characteristics of Environment
Wilson & Cleary JAMA (1995); Adapted from Revicki, 2006
Quality of Life
Summary Review
Why are PROs included in clinical trials?

B/c some treatment effects are only known to patients


B/c patients provide a unique perspective on
treatment effectiveness


eg, how patients function in daily activities
B/c it is part of a comprehensive assessment of
impact of disease and treatment


eg, pain intensity and pain relief
Eg, capture disease burden & patient’s view of benefit of
treatment
B/c capturing improvement in patients-reported
measures can be a powerful addition to a label claim
of treatment efficacy
Summary Review
Why are PROs included in clinical trials?

B/c some treatment effects are only known to patients


B/c patients provide a unique perspective on
treatment effectiveness


eg, how patients function in daily activities
B/c it is part of a comprehensive assessment of
impact of disease and treatment


eg, pain intensity and pain relief
Eg, capture disease burden & patient’s view of benefit of
treatment
B/c capturing improvement in patients-reported
measures can be a powerful addition to a label claim
of treatment efficacy
Summary Review
Why are PROs included in clinical trials?

B/c some treatment effects are only known to patients


B/c patients provide a unique perspective on
treatment effectiveness


eg, how patients function in daily activities
B/c it is part of a comprehensive assessment of
impact of disease and treatment


eg, pain intensity and pain relief
Eg, capture disease burden & patient’s view of benefit of
treatment
B/c capturing improvement in patients-reported
measures can be a powerful addition to a label claim
of treatment efficacy
Summary Review
Why are PROs included in clinical trials?

B/c some treatment effects are only known to patients


B/c patients provide a unique perspective on
treatment effectiveness


eg, how patients function in daily activities
B/c it is part of a comprehensive assessment of
impact of disease and treatment


eg, pain intensity and pain relief
Eg, capture disease burden & patient’s view of benefit of
treatment
B/c capturing improvement in patients-reported
measures can be a powerful addition to a label claim
of treatment efficacy
Humira label for
RA:
“Humira is
indicated for …
improving physical
function in adults
with moderately to
severely active
rheumatoid
arthritis”
Orencia Label for RA:
“ORENCIA is indicated
for … improving physical
function in adult patients
with moderately to
severely active
rheumatoid arthritis …”
Humira label for
RA:
“Humira is
indicated for …
improving physical
function in adults
with moderately to
severely active
rheumatoid
arthritis”
Orencia Label for RA:
“ORENCIA is indicated
for … improving physical
function in adult patients
with moderately to
severely active
rheumatoid arthritis …”
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
What is the goal of medical treatment?
What is a PRO?
Why use PRO measures?
How do PRO measures complement traditional
clinical outcome measures?
What characterizes PRO measures?
 How do PROs differ?
What characterizes a well-constructed PRO
instrument?
 How are PRO instruments developed?
What is the process necessary to achieve a PRO
label?
 Putting it all together
What is the role of PROs in clinical trials research?
Review and conclusions
Review &
Conclusions
Outline
1.
2.
3.
4.
5.
6.
7.
8.
9.
What is the goal of medical treatment?
What is a PRO?
Why use PRO measures?
How do PRO measures complement traditional
clinical outcome measures?
What characterizes PRO measures?
 How do PROs differ?
What characterizes a well-constructed PRO
instrument?
 How are PRO instruments developed?
What is the process necessary to achieve a PRO
label?
 Putting it all together
What is the role of PROs in clinical trials research?
Review and conclusions
My goal for you

My aim:
 for you to have a sound general sense of
what a PRO is and its role in clinical trials
research
WILSON-CLEARY MODEL OF HEALTH OUTCOMES:
WhereIMPORTANT
do PROs fit in aTAKE-HOME
greater model of
health outcomes?
MESSAGE:
Impact of DISEASE and of TREATMENT can be
Characteristics of Individual
captured at each of these boxes
And so are part of comprehensive assessment
Biological and
Physiological
Variables
Symptom
Status
Functional
Status
General Health
Perceptions
Characteristics of Environment
Wilson & Cleary JAMA (1995); Adapted from Revicki, 2006
Quality of Life
Review & Conclusions


A lot needs to happen at protocol concept
 Develop an endpoint model
 Define relevant PRO concepts to product &
target population
 Define target PRO-based label claim
 Identify pre-existing, well-constructed & wellvalidated PRO for target PRO concepts, OR
 Develop new PRO (for this case, plan ahead!)
Incorporate in trial
By following
the steps
discussed
today, you will
be well on your
way to a label
claim.
The End
Thank You
[email protected]
Critical Path opportunities to
discuss measurement issues
Adapted from Rock, Medical Officer, Office of Oncology Drug Products, FDA, 2006
Descargar

Slide 1