Methods for Learning
Metrics
A workshop in conjunction with
Metrics 2005
September 19, 2005
Como, Italy
Organized by:
Carolyn Seaman
Letizia Jaccheri
Rough Schedule for Today
9am
Introductions, team-building, and
preliminaries
10am
Break
10:30m
Presentations of position papers and
discussion
12:30pm Lunch
2pm
3:30pm
Teaching design exercises
Break
4:00pm Group photo
4:05pm Discussion
5:30pm Close
Introductions
• Step 0: Name, institution, job title
• Step 1:
– Pair up
– Share work details – research, teaching,
management, development, consulting,
etc.
– Introduce your partner
Introductions
• Step 2
– Pair up differently
– Share who you are outside of work:
• What’s important to you?
• What do you enjoy?
• What defines you?
– Introduce your partner
Introductions
• Step 3
– Pair up differently
– Share what you have taught
•
•
•
•
Other than software engineering courses
Subject
Context
Audience
– Introduce your partner
Presentations
• Luigi Buglione, Alain Abran:
– Improving Measurement Plans from multiple dimensions:
Exercising with Balancing Multiple Dimensions - BMP
• Linda M. Ott
– Developing Healthy Skepticism not Disbelief - Problems in
Teaching Software Metrics
• Giuseppe Lami
– Learning Metrics for Natural Language Requirements in an
Under-Graduate Course
• Letizia Jaccheri, Sandro Morasca
– Understanding the role of software metrics in an Empirical
Software Engineering Course for PhD students
– overview of course
• Maurizio Morisio, Marco Torchiano
– TDE: a method and tool to teach OOP
Teaching Design Exercise
• Split into 3 groups
• Everyone does task 1 in their group
• Each group presents their course
design
• Divide tasks 2, 3, and 4 among the
three groups
• Each group presents their work
• Discussion
Teaching Design Exercise
Task 1
• Subject to be taught:
How to change money in a
foreign country
• Design a course, including:
– Objectives and overall structure
– Outline of material to be presented
– Exercises
– Assessments
Task 1 (cont.)
• Course should be based on collaborative
exercises, e.g.
–
–
–
–
Producing a film or video as a group
Creating a piece of artwork
Collaboratively designing a poster on some topic
Deciding how something should be built (i.e.
making design decisions)
• … and student reflection and feedback, e.g.
–
–
–
–
–
Student presentations on reading material
Personal logs or journals written by students
Essays about lessons learnt
Oral examinations
Etc.
Task 1 Results
• Group A
– Objectives:
• To provide the skills needed to end up with the needed currency in
hand without breaking rules
– Materials
• Talks – money, value of money, different exchange rates, why it is a
problem
• Motivate importance of topic – why it will be helpful to them
• Example scenario – going abroad to a conference
• Converting cash and converting with credit card
– Group exercises
• Teams – one person on each team doesn’t know how to convert
money, others teach him
– Assessments
• Short term – give same value of money in different currencies to
different students and they have to convert it to dollars, then they have
to write a log, which is evaluated against the objectives
• Public exposition – so they can learn from mistakes and from other
students
Task 1 Results
•
Group B
– Main objective – to teach someone how to make available local currency
– Half day course
– Present an IDEF0 Process model for explaining the conversion process –
inputs to outputs and mechanisms to do that
– Topics:
•
•
•
•
•
•
•
•
•
What is money – different forms, cash, credit cards, etc.
Cost of living in different countries
Fees, commissions
Exchange rates
Where can we change money?
Outputs – traveler’s checks, cash, receipt
Changing money back into original currency
Teller machines, bureaus de change
Flowchart to describe different options
– Exercise – Excel spreadsheets with change rates and world events related
to changes
– Role playing – purchase of currency
– Assessment – role play, ordinal scale (0-3), strengths, areas for
improvement
– SMART evaluation
– Feedback – oral to group; written through evaluation forms; improvements
and teaching style
Task 1 Results
•
Group C
– 8 sections: objectives, different ways to change, considerations, economy,
safety and security, convenience, exercises, assessment
– Objectives – assumed transactions are legal
• Complete the transaction – end up with different currency than started with
• Most economically and good exchange rate and good commission
• Convenient and safe transaction
– Different ways
• Before the trip – bank or airport
• During the trip – bank, airport, or hotel, ATM or change office
– Considerations
• Economy, safety, convenience
– Ranked methods in decreasing order of:
• Economy: Home bank, foreign bank, ATM, airport/hotel/change office
• Safety and security: Bank, hotel, ATM, airport, change office
• Convenience (depends on circumstances): Hotel, home bank
– Exercises
• Describe scenarios – circumstances of trip, situation
• Students write what, how, why
– Assessment
• Groups of some size – critique each others’ responses and share ideas
– Poster – Foreign Exchange 101
Teaching Design Exercise
Task 2
• Add to or modify the course you just
designed to include a multi-faceted
approach, including, e.g.
– Psychological and emotional dimensions
– Ethical concerns
– Legal issues
Task 2 Results
• Increased the number of constraints in the process
model
• Psychological and emotional – language barrier,
style of writing (characters), may not know exchange
rate, i.e. what to expect, security issues (older or notcomputer-literate people), location of ATM, events
you’re not aware of at an ATM
• Ethical concerns – uneasy with government, not
inclined to change a lot of cash, reduce expenditures
if not comfortable with government
• Legal constraints – constraints on amount of
currency allowed in or out, types of money that can
be exchanged (coins or bills), black market,
merchants changing under the table
Teaching Design Exercise
Task 3
• Now consider the situation where you are
teaching a course on changing money, but in
the context of a more general course on
travel
• What are the learning objectives concerning
changing money?
• What learning objectives about travel could
be illustrated by teaching how to change
money?
• How would your course be different?
Task 3 Results
• Travel considerations – planning strategies (long
term/prepaying vs. short term/more expensive/very flexible)
• How to assess the need for money – local customs (credit
cards accepted or not)
• How to manage difficulties and travel emergencies – have
multiple potential solutions
• Travel planning illustrated by changing money
• Differences in culture – orderly vs. chaotic, also affects
changing money strategy
• Different points of view
• Urban vs. rural
• Environment and government
• Whether or not you need to change money – some places
accept multiple currencies, esp. borders
Teaching Design Exercise
Task 4
• Now suppose that you are teaching your
course, and some students say…
“Why do we need to know this? With the
Euro, we’ll never need to change
money again!”
• How do you teach them that the concepts are
still important?
Task 4 Results
• Teach them that the concepts are still
important
• Use an experience that they might have, e.g.
someone has to plan where they work in the
next 5 years, have to compare wages in
different countries
• Traveling and buying goods – examples to
show them that this will help them
understand what’s going on
• Experiment – go to EBay and buy Italian wine
in Euros – how much will it cost?
Discussion
• Can we apply what we’ve designed for
teaching money-changing skills to
teaching software metrics?
• Task 1 – collaborative exercises and
student feedback
• Task 2 – balancing multiple
perspectives
• Task 3 – “sneaking in” metrics
• Task 4 – dealing with “disbelievers”
Applying Ideas to Metrics
Teaching Currency Exchange:
• Learning Objectives
– To provide the skills needed
to end up with the needed
currency in hand without
breaking rules
– To teach someone how to
make available local
currency
– Complete the transaction –
end up with different
currency than started with
– Getting a good exchange
rate and good commission
– Convenient and safe
transaction
Teaching Metrics:
•
Learning Objectives
–
–
–
–
–
–
–
–
–
To provide skills needed to develop metrics
without violating personnel rules and rules
of measurement theory
Goal-driven measurement (GQM) – how
derive measures from local goals (multiple
levels)
End up with a quantity that is meaningful
and that is understood – measure what you
intend to measure – and that is useful,
provides some benefit
Designing efficient measures
Measuring efficiently
Cost of collecting metrics
Return on investment of measurement
Not accepting counterfeit information –
knowing when the data is bogus
Audience of data – how to use, who should
use it, security of data
Applying Ideas to Metrics
Teaching Currency Exchange:
• Material covered:
–
–
–
–
Why it is important
Why it’s a problem
Basic concepts - money, value of
money, different exchange rates,
Different ways of converting
•
•
–
–
Different forms of money – cash,
credit cards, travelers checks
Process
•
•
•
–
–
–
–
–
Before and after trip
Bank, ATM, bureau de change,
airport/hotel
IDEF0 process model
Flowchart
Inputs and outputs
Cost of living in different countries
Fees, commissions
Exchange rates
Changing money back into original
currency
Economy, safety, convenience
Teaching Metrics:
•
Material covered:
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
Motivation
Measurement foundations and theory
Operational definitions of metrics
Different analysis methods
Process vs. product vs. personnel metrics
Dangers in converting measurement units; different
scales for effort, e.g. – needed for benchmarking
Different modes of data collection
Different measurement purposes; application of
specific metrics
Different measurement scales – ordinal, interval, etc.
Measurement process
Comparing different processes
Variety of programmer salaries/costs
How measures work in different types of
organizations
Use of normalized metrics
Costs of data collection/analysis
Auditing
Using measurement results for improvement
Showing progress through measurement results
Tie the results to process, not to people
Cost/benefit analysis of measurement
Evaluating/assessing intangibles
Applying Ideas to Metrics
Teaching Currency Exchange:
•
Material covered (cont.):
–
–
–
–
–
–
–
–
–
Psychological and emotional – language
barrier, style of writing (characters), may
not know exchange rate, i.e. what to expect,
security issues (older or not-computerliterate people), location of ATM, events
you’re not aware of at an ATM
Ethical concerns – uneasy with
government, not inclined to change a lot of
cash, reduce expenditures if not
comfortable with government
Legal constraints – constraints on amount
of currency allowed in or out, types of
money that can be exchanged (coins or
bills), black market, merchants changing
under the table
Strategy – for both travel and money
changing
Travel planning illustrated by changing
money
Differences in culture – orderly vs. chaotic,
also affects changing money strategy
Urban vs. rural
Environment and government
Whether or not you need to change money
– some places accept multiple currencies,
esp. borders
Teaching Metrics:
•
Material covered:
–
–
–
–
–
–
–
–
–
–
–
–
–
Culture of quality
Measurement textbooks translated into
different languages
Legal obligations of measurement
Acceptance of measurement programs by
developers
Paranoia
Personal and professional vulnerability
Inappropriate uses of measurement
Distinguishing randomness from trends
Using measurement for building SLA’s and
other types of contracts
Third party assessment – ethics and
legalities
Subversion of metrics by changing
behavior
Using different measures at different stages
of project and converting between them
Influence of different programming
languages
Applying Ideas to Metrics
Teaching Currency Exchange:
• Exercises:
–
–
–
–
–
–
–
–
–
one person on each team doesn’t
know how to convert money, others
teach him/her
Excel spreadsheets with change
rates and world events related to
changes
Role playing – purchase of currency
Oral feedback to group; written
feedback through evaluation forms;
improvements and teaching style
Describe scenarios – circumstances
of trip, situation; students write what,
how, why
Poster – Foreign Exchange 101
Comparing wages in different
countries
Examples to show them how
changing money will help them
understand larger issues of
economics and world events
Experiment – go to EBay and buy
Italian wine in Euros – how much will
it cost?
Teaching Metrics:
•
Exercises:
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
Group members teach one member about metrics;
students create pedagogical materials
Project manager version of Monopoly
Process control charts – show real examples; relate
to real events, e.g. vacations, reorgs
Refactoring exercise – take measurements, refactor
according to metrics, re-measure
Evaluating the validity of metrics in a published
study
Role playing manager and developer
Students look at own company’s data and apply
estimation models and relative errors with actuals
Scenarios – discuss how expected values would be
adjusted in that situation
Give a real scenario and set of measurement data,
students make inferences and predictions about
project
Differences between metrics on different types of
software (e.g. OS, embedded, etc.)
Compare functionally equivalent software using a
set of metrics to see differences
Measurement profiles of different teams of
developers – what can you get with the same
amount of XXX from different teams?
Design a metrics that demonstrates your value as a
software engineer
Look at history of software industry through
measurement snapshots
Outsourcing exercise
Applying Ideas to Metrics
Teaching Currency Exchange:
• Assessments:
– Short term – give same value of
money in different currencies to
different students and they have
to convert it to dollars, then they
have to write a log, which is
evaluated against the objectives
• Public exposition – so they can
learn from mistakes and from
other students
– role play, ordinal scale (0-3),
strengths, areas for
improvement
• SMART evaluation
– Groups of some size – critique
each others’ written responses
to set of questions and share
ideas
Teaching Metrics:
• Assessments:
–
–
–
–
–
–
Compare different students’ designs
for the same measurement problem
Compare different students’ process
assessments given same data and
assessment process
Look at different metrics and how
they relate to the underlying value of
the software and how the metrics
could be combined properly
Given a scenario, which metrics are
most suitable and why?
Manager/developer role play – “why
do I have to collect this?”
Role playing with appraisals –
making sense of information given by
a project team
Descargar

Methods for Learning Metrics