WMO
Social and Economic Benefit Studies
Non-Economic Social Science Methods
Assessing the Uptake of Services
Gerald Fleming
Chair, OPAG/PWS
Mahé, Seychelles, 5th May 2015
WMO - Public Weather Services, WDS
Social Sciences


Social science is a major branch of science, and
a major category of academic disciplines,
concerned with society and the relationships
among individuals within a society.
Social science dates from the Age of
Enlightenment, and attempts to use scientific
methods to examine social behaviour and
characteristics.
WMO Public Weather Services
2
Social Sciences

Some of the Social Sciences:
 Sociology
 Human Geography
 Economics
 Political Science
 Psychology
 Demography
 Anthropology
 etc etc...
WMO Public Weather Services
3
Social Sciences

Some of the Social Sciences:
 Sociology
 Human Geography
 Economics
 Political Science
 Psychology
 Demography
 Anthropology
 etc etc...
WMO Public Weather Services
4
“Assessing the Uptake of Services”

What does this mean?
It’s much easier to assess the uptake of Products




Trial new products
Refine them
Put them on sale
Measure the uptake (sales)
WMO Public Weather Services
5
“Assessing the Uptake of Services”

How do we measure Services?
NMHSs have historically been very good at
producing forecasts and warnings.
Once they are issued – what happens to them?
Do we know HOW people use them?
Often we have no clue about this
WMO Public Weather Services
6
Weather Warnings
What ways can we assess these?
 Measure accuracy of the warnings
 Measure number of warnings issued
 Measure whether warnings reached
intended users
 Measure whether or not the warning was
understood
WMO Public Weather Services
7
Weather Warnings
What ways can we assess these?
 Measure whether the consequences of
the likely weather were understood
 Measure the intended behaviour of the
user before and after receiving the
warning
 Measure the warning outcomes
 Measure the satisfaction of the users
WMO Public Weather Services
8
Weather Warnings
What ways can we assess these?
 Measure accuracy of the warnings
√
 Measure number of warnings issued
√
 Measure whether warnings reached
intended users
X
 Measure whether or not the warning was
understood
X
WMO Public Weather Services
9
Weather Warnings
What ways can we assess these?
 Measure whether the consequences of
the likely weather were understood
X
 Measure the intended behaviour of the
user before and after receiving the
warning
X
 Measure the warning outcomes
X
 Measure the satisfaction of the users
√
WMO Public Weather Services
10
Weather Forecasts and Warnings
We need to know more about the behaviour of those
who receive the warnings
 Especially – decision-making
o
Was it rational?
o
Was it logical?
o
Was it intuitive?
WMO Public Weather Services
11
Weather Forecasts and Warnings
Decision-making by Users
 What are the constraining factors?
 Are there options?
 Were these options weighed-up in an appropriate
manner?
 How did other peoples behaviour influence the
decision-maker?
 What were the uncertainties?
All-in-all, a complex area for study.
WMO Public Weather Services
12
Weather Forecasts and Warnings
How can we get to understand users – individually
and collectively?
 Surveys
 Focus Groups
 Seminars
 Field Research
WMO Public Weather Services
13
Weather Forecasts and Warnings
Surveys
 Can be random (e.g. telephone surveys) or selfselecting (e.g. users of websites)
 Can be Closed (questions with a limited range of
answers) or Open (invite free text / comment)
 Were the questions properly designed (e.g. not asked
in a leading manner)
 Were the number of respondents statistically
significant?
WMO Public Weather Services
14
Weather Forecasts and Warnings
Field Surveys
 Can entail close collaboration with decision-makers, to
observe their activities and processes at first hand
(intrusive)
 Can use other data sets
•
Health statistics
•
Traffic information
•
Consumer behaviour
•
Agricultural inputs and outputs
•
Communication / Social Media data
WMO Public Weather Services
15
Weather Forecasts and Warnings
All surveys

Must be careful to recognise the differences
between “Stated Preference” (what people say
they will do) and “Revealed Preference” (what
they actually do, as indicated by the evidence.
Understanding user needs is much more complex
than just asking them!
A research area in itself.
WMO Public Weather Services
16
Weather Forecasts and Warnings
In this context...

Economics can be seen as the study of social
behaviour as revealed by financial evidence

Social Sciences encompass a broader, more
holistic view of social behaviour
WMO Public Weather Services
17
WMO Public Weather Services
18
Surveys of Users
Need to know the profile of the survey respondents
 Male / Female
 Age group
 Occupation
 Location
 etc etc..
WMO Public Weather Services
19
Surveys of Users
A recent example from Met Éireann
 Survey conducted of users of the website met.ie
 Therefore not a random survey, but self-selecting
 Website users could choose whether or not to
take the survey
 Each user asked only once (minimise the
nuisance value)
 Mix of Closed and Open questions
WMO Public Weather Services
20
General information
Current survey
Previous Survey
Survey period
13.01.2015 – 13.02.2015
Survey period
21.01.2013 – 22.02.2013
Questionnaire views
497,699
Questionnaire views
364,414
Number of answers
50,221
Number of answers
37,891
Response rate
10.1%
Response rate
10.4%
First time visitors
2,864
First time visitors
2,161
Open answers
18,125
Open answers
11,246
Benchmark
Benchmark Group
Meteorological benchmark
Survey Period
25.11.2013 – 13.02.2015
Number of Websites
4
Total Number of Answers
215,108
Content
Demographics
User feedback
Benchmark
Summary
About the survey
General information
Swedish Meteorological and Hydrological Institute
Finnish Metrological Institute
Survey period
03.03.2014 – 31.03.2014
Survey period
25.11.2013 – 23.12.2013
Questionnaire views
1,180,762
Questionnaire views
2,074,919
Number of answers
53,199
Number of answers
100,814
Response rate
4.5%
Response rate
4.9%
First time visitors
2,024
First time visitors
2 ,076
Open answers
11,082
Open answers
19,599
Danish Meteorological Institute
Survey period
14.12.2014 – 22.12.2014
Questionnaire views
794,330
Number of answers
23,204
Response rate
5.5%
First time visitors
433
Open answers
6,378
Who Are Your Users?
Gender – Device
Met Éireann 2015 tablet
Women
41%
Men
59%
51%
49%
Number of answers: 5,172
Met Éireann 2015 desktop
40%
60%
Number of answers: 38,857
Met Éireann 2015 overall
41%
59%
Number of answers: 44,029
0%
20%
40%
60%
Female
Gender - Website
Meteorological benchmark
35%
Danish Meteorological Institute
100%
Male
65%
31%
Finnish Metrological Institute
80%
Number of answers: 37,863
69%
36%
64%
Number of answers: 92,193
Swedish Meteorological and Hydrological Institute
33%
67%
Number of answers: 48, 204
Met Éireann 2013
33%
67%
Number of answers: 33,138
Met Éireann 2015
41%
0%
10%
20%
59%
30%
40%
50%
Female
Note: Percentages are rounded to the closest whole number.
60%
Male
70%
Number of answers: 44,029
80%
90%
100%
Occupation
Device
Met Éireann 2015 tablet
20%
2%
Met Éireann 2015 desktop
17%
33%
Met Éireann 2015 overall
2%
32%
0%
Office worker
12%
3% 5%
16%
2%
10%
10%
16%
11%
3%
3%
Number of answers: 6,509
32%
7%
7%
10%
20%
30%
40%
50%
60%
Labourer
CEO/senior manager/manager
Self-employed
6%
6%
23%
Number of answers: 41,515
24%
Number of answers: 48,024
70%
80%
90%
100%
Unemployed
Trainee/student
Retired
Other
Website
Meteorological benchmark
32%
Danish Meteorological Institute
27%
Finnish Metrological Institute
28%
12%
15%
11%
8%
42%
Met Éireann 2013
27%
Met Éireann 2015
Office worker
10%
18%
2%
20%
Labourer
Note: Percentages are rounded to the closest whole number.
30%
16%
40%
CEO/senior manager/manager
11%
50%
16%
4%
3%
60%
Self-employed
11%
27%
7%
5%
12%
3% 5%
2% 4%
15%
15%
2%
31%
0%
10%
18%
Swedish Meteorological and
Hydrological Institute
9%
9%
4%
7%
7%
13%
2% 4%
18%
7%
5%
Number of answers : 41,977
Number of answers: 97,815
8%
6%
24%
Number of answers: 52,111
7%
6%
24%
Number of answers: 34,963
70%
Unemployed
80%
90%
Trainee/student
100%Number of answers: 48,024
Retired
Other
Age - Device
35%
Met Éireann 2015 overall
Number of answers: 48,152
30%
Met Éireann 2015 desktop
Number of answers: 41,613
Met Éireann 2015 tablet
25%
Number of answers: 6,539
20%
15%
10%
5%
0%
0-9 years
10-19 years
20-29 years
Note: Percentages are rounded to the closest whole number.
30-39 years
40-49 years
50-59 years
60-69 years
70-79 years
80-89 years
over 90 years
Frequency
Device Met Éireann 2015 tablet
8%
46%
34%
6%
5%
Met Éireann 2015 desktop
6%
45%
36%
8%
6%
Met Éireann 2015 overall
6%
45%
36%
8%
6%
Number of answers: 6,728
Number of answers: 42,584
0%
10%
20%
30%
40%
This is my first visit
Every day
50%
60%
Every week
70%
80%
Every month
100%Number of answers: 48,005
90%
Less often
Website
Meteorological benchmark
3%
55%
Danish Meteorological Institute 1%
33%
65%
6% 3%
29%
4% 1%
27%
4% 2%
Number of answers: 42,147
Finnish Metrological Institute 2%
65%
Number of answers: 98,599
Swedish Meteorological and
Hydrological Institute
4%
Met Éireann 2013
5%
Met Éireann 2015
6%
0%
44%
39%
46%
38%
46%
10%
20%
30%
This is my first visit
Note: Percentages are rounded to the closest whole number.
35%
40%
Every day
50%
60%
Every week
70%
Every month
9%
4%
7%
5%
8%
80%
Less often
90%
5%
Number of answers: 51,787
Number of answers: 36,852
100% Number of answers: 48,005
Device Type
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
86%
14%
Desktop
Tablet
Met Éireann 2015
Note: Percentages are rounded to the closest whole number.
Number of answers: 49,312
Exciting/Boring
«
Desktop
Very exciting
Exciting
5%
Boring
Very boring
more interactive features
More detail for the areas of interest. Maybe live stats for particular areas around the
country.
9%
Add 5 to 15 day location forecast for specific locations. And product more images
with animations
more interactive and county specific illustrations
36%
More interactive, the radar option is very good.
50%
Some Video ?
Take a look at how other Weather Sites do it. No need to re-invent the wheel. But
you are so so so boring :)
Pictures are always interesting to look at. More pictures would break up the text
better. Pictures of today's weather?
Number of answers: 43,620
«
Tablet
Look at other providers, I like rainfall radar and thing that are more integrated with
users
Desktop
Tablet
Perhaps more interactive regarding maps at regional level.
4%
5%
Strategic webcams would be interesting and more info about why a aprticular
forecast has been made.
9%
9%
36%
38%
50%
49%
Your website could have better graphics and more interesting content. I have
always found met.ie quite boring however it does supply what's important and that's
the weather.
More local information. I especially like yr.no and the detailed local info that it gives.
More graphical info as opposed to text might also be good
Would like more responsive elements in it
Number of answers: 38,631
Number of answers: 4,989
Note: Percentages are rounded to the closest whole number.
Some video forecast
Attractive/Ugly
«
Desktop
Very attractive
Attractive
Ugly
Very ugly
Make it slightly more visually appealing.
I really like the short range and 5 day forecasts. However, I think thye could be
jazzed up a bit and be more prominent on the site. Also often the time on the scale
does not reflect the time on the map - thye can be out of synch.
5%
21%
rainfall radar is brilliant but it could always be better more information or improved
animation would be nice
25%
the image of the website could be brightened up.A little bit more user friendly,
I humbly suggest that it should be more graphical. Include Animations for Forecasts
as well as last few hours. (Similar to rain radar, but also projected into the future.)
Include Wind Speed, Temperature and Rain fall in animations. Have a look at
www.theyr.com to get ideas.
49%
would be nice to have the animated forecast screen bigger, it's all a bit small
Number of answers: 43,818
maybe a little more visual
«
Tablet
I would Maybe change the Back ground Colours but other the That always very
Professional
Desktop
Tablet
4%
Make the map on the right hand side a littlebigger. I always zoom in.
4%
More colour, bigger print
22%
25%
20%
Please make the website a little more attractive and educative for the kids as well!!
27%
surely you can design a better, more attractive website?
49%
49%
It could be significantly updated to look more attractive and make gathering
information more intuitive
Larger text; more interaction; more colour
Number of answers: 38,791
Number of answers: 5,027
Note: Percentages are rounded to the closest whole number.
Better graphics of current and incoming weather.24hr,48hr72hr.
Language
Text is written in a language that is easy to understand
Strongly agree
Agree
Disagree
Strongly disagree
2% 1%
«
Desktop
Please use more paragraphs when describing the weather. Bick chunks of text are
too difficult to read.
Leagan as GAEILGE le do thoil! Irish Language versions please.
Content in Irish language is incomplete, this should be addressed.
Its old and not user friendly, also its difficult to get just a simple weather forecast for
your town or county. You should take a look at how BBC Weather do it. Much
simpler and in layman terms. Use less of the isotopes language, keep it simple.
46%
51%
at times some charts and information do not correspond and scattered. Language is
too vague.Tidy it up.
Easier language please
A link to explaining key meterological terms to people of modest understanding
Number of answers: 43,943
Tablet
«
Tablet
Desktop
2%
Please set the regional forecasts with a more laymans language with a more friendly
site‘
3%
1%
42%
46%
51%
Content and language over technical for common usage.
55%
Note:
Strongly disagree
had a 0.34% response rate
Number of answers: 38,873
Number of answers: 5,070
Note: Percentages are rounded to the closest whole number.
Avoid lengthy sentences. Short punchy key points are more effective. Language
used is often more conversational than professional.
Speak in a language that the average citizen can understand, not just those with 3rd
level education or a professional qualification. Also, with regard to compatibility with
tablets, the site needs serious attention.
Short range forecast into charts or words easier yo understand. Precipitation in
charts defined as rain, sleet or hail if possible.
Ambiguous language used to describe forecast
Make able to use the site in other languages ​, such as Russian , Polish , German.
Trust
The website gives an impression that the organization
behind it is professional and trustworthy
Strongly agree
Agree
Disagree
Strongly disagree
1%
3%
«
Desktop
Some of the comments never get updated and hence lose their message (Water
Safety bulletin as example)
Gives me what I want and I trust it.
How about slicker images! I'm thinking along the lines of 3D. Having said that, I'd
rather the content to remain as accurate and professional as it is.
44%
Sometimes the graphics suggest weather different to the text. Maybe on the home
page there should be three maps depicting morning, afternoon and evening and
including temps and wind on the one map.
52%
There should be proper editing of the written forecast on the home page. It is
unusual not to find several typos/errors on any given day. This looks unprofessional
and could easily be eliminated.
If it was possible to have a rainfall radar with a predictive sequence for 3/6 hrs ahead
it would be super as I use this every day for work and fully depend on this
Number of answers: 44,051
«
Tablet
As a teacher I use the website occasionally as a teaching tool perhaps some
simplified links for children to help them read weather update sect more easily
Desktop
Tablet
3%
3% 1%
Keep it professional/technical, don't simplify it in an effort to modernise/attract larger
audiences. For the outlook forecast, breaking it out more clearly by day would make
it easier to follow - eg Monday - showers... Monday night -...
1%
Avoid lengthy sentences. Short punchy key points are more effective. Language
used is often more conversational than professional.
44%
52%
Number of answers: 38,974
45%
51%
Number of answers: 5,077
Note: Percentages are rounded to the closest whole number.
It could look a bit more modern and professional but otherwise I feel its a very good
website for updated weather warnings and of course regional weather warnings.
More information please and a more professional appearance. But I like its simplicity
so don't lose that. Define the terms and phrases used in the forecasts. For example I
don't know what winter showers are as opposed to showers. Anyone I ask guesses
but don't know either. thanks and good luck.
Design index - benchmark
Modern/outdated
Overall
assessment
Modern/outdated
80.0
80.0
75.0
75.0
70.0
70.0
65.0
Attractive/ugl
y
60.0
65.0
Overall
assessment
60.0
55.0
55.0
50.0
50.0
Well-organised/messy
Attractive/ugl
y
45.0
Exciting/boring
Well-organised/messy
Exciting/boring
Met Éireann 2015
Met Éireann 2013
Met Éireann 2015 overall
Met Éireann 2015 desktop
Met Éireann 2015 tablet
Note: Percentages are rounded to the closest whole number.
Swedish Meteorological and Hydrological Institute
Finnish Metrological Institute
Danish Meteorological Institute
Meteorological benchmark
Usability index - benchmark
Navigation
Trust
90.0
85.0
80.0
75.0
70.0
65.0
60.0
55.0
50.0
Speed
Navigation
Relevance
Communication
Trust
90.0
85.0
80.0
75.0
70.0
65.0
60.0
55.0
50.0
Speed
Relevance
Communication
Met Éireann 2015
Met Éireann 2013
Met Éireann 2015 overall
Met Éireann 2015 desktop
Met Éireann 2015 tablet
Note: Percentages are rounded to the closest whole number.
Swedish Meteorological and Hydrological Institute
Finnish Metrological Institute
Danish Meteorological Institute
Meteorological benchmark
Open Survey Answers
WMO Public Weather Services
34
Open Survey Answers
WMO Public Weather Services
35
WMO Service Delivery Strategy
 One of the key strategic thrusts of WMO
 Helping NMHSs to better deliver their products
and services to users
 Incorporates an Implementation Plan
 Designed to be a practical guide to the steps
we must all take to improve our services
 Incorporates the “Service Delivery Progress
Model”
 A technique to help us find out how good (or
bad) we are at Service Delivery, and how to
improve.
WMO Public Weather Services
36
Service Delivery Progress Model
 Defines five different stages of development
for NMHSs (each over a number of
elements)





Undeveloped
Development Initiated
Development in Progress
Developed
Advanced
WMO Public Weather Services
37
Service Delivery Progress Model
 Defines six elements of development
 Evaluate user needs and decisions
 Link service development and delivery to user
needs
 Evaluate and monitor service performance and
outcomes
 Sustain improved service delivery
 Develop skills needed to sustain service
delivery
 Share best practices and knowledge
WMO Public Weather Services
38
Service Delivery Progress Model
WMO Public Weather Services
39
Service Delivery Progress Model
WMO Public Weather Services
40
Service Delivery Progress Model
WMO Public Weather Services
41
Undeveloped
Evaluate user
needs, decisions
Development
Initiated
√
√
Sustain improved
SD
Develop skills
needed to sustain
SD
Share best
practices and
knowledge
Developed
√
Link SD and
delivery to user
needs
Evaluate and
monitor service
performance and
outcomes
Development
in Progress
√
√
√
WMO Public Weather Services
42
Advanced
Social Sciences and Met Services





Its all about the users!
Improving our knowledge and understanding of
those who use forecast and warnings services
Helping users to get more from our forecast and
warnings services
Trying to understand decision-making, in all its
complexity
Try to get to some measure of the ultimate
outcomes for society of the services provided by
NMHSs
WMO Public Weather Services
43
WMO Public Weather Services
44
Descargar

Presentation title here - World Meteorological Organization