Topics in Software Engineering
(Software Testing and Analysis)
Darko Marinov
September 20, 2011
• First few lectures to help you select projects
– Shared memory: CHESS, IMUnit, CAPP
– Message passing: Setac
– Comment analysis: iComment
– Regression testing: survey paper
– Model-based testing
– Today: test assertions
– Thursday: mock-based testing
• Project proposals due on Sep 27 (in a week!)
– Your project can be something we don’t cover
Next Week
• Project proposals (due 9/27)
– If we didn’t talk (email/Skype/in person),
schedule a meeting with me
– Create a Wiki page (low overhead but helps in
keeping track)
• Paper reports (due 9/29)
– Choose one paper, ideally related to your
– Write a four-bullet report on that paper
Paper Today
• Augmenting Automatically Generated UnitTest Suites with Regression Oracle Checking
Tao Xie (ECOOP 2006)
• Follows: one-slide overview, project ideas,
questions for discussion
Paper Overview
• Problem
– Automatically generated tests lack assertions
• Solution
– Orstra
– Insert assertions based on observers
• Evaluation
– 11 small programs
– Mutation testing for fault seeding
– Increased fault-exposure ratio
Potential Projects (1)
• This idea could be combined with the
regression testing techniques presented in
the survey paper on 9/13 [KB]
• how to identify which of the assertions
generated are actually useful [FS]
• there should be selection among assertions
to prevent false alarms in the modified
version of the program [SO]
• generate new test inputs instead to consider
only those existing [FS]
Potential Projects (2)
• Find a better way to deal with nondeterminism that could be impacting the
quality of the generated assertions [AY]
• annotate the methods of your class with
information that assists Orstra in determining
whether a particular method's value should
be asserted, and in what way [DwG]
• Perform a more extensive evaluation on
some large code base [DeG]
• apply some of the techniques in this paper to
manually written tests [JT]
Potential Projects (3)
• Integrate this tool into an IDE (like Eclipse) to
give indications about when code changes
are modifying the state of outside objects[SB]
• We can also extend this to implement
contracts in a program… add pre-conditions,
post-conditions and invariants [AD]
• evaluate the effectiveness of the various
[oracle generation] tools on software systems
of varying size [CM]
• Creating an Orstra based test augmenter for
other test frameworks in other languages[KN]
Questions for Discussion (1)
• Find test inputs that differentiate the behavior
of a modified version respect to the original
version [FS]
• consider information other than the simple
return value [FS]
• Can Method-Sequence State Representation
represent singleton? [JT]
• there are no general rules to get expected
output when providing an arbitrary input for a
program? [YL]
• how Orsta can work with JUnitEE [AD]
Questions for Discussion (2)
• generating assertions which test only the
specification you want to test, and not the
implementation details? [DwG]
• Should they have used observer methods
when generating for results that are not
primitive and not the class under test? [DeG]
• Integration with code contracts / annotations /
aspect-oriented programming? [KB]
• Couldn't they have used the existing test
suite and have the developer choose states
to generate assertions for? [SB]
Questions for Discussion (3)
• Was the need for random test generation
really required? [SB]
• How does the technique presented in the
paper scale to much larger systems? [CM]
• How to be sure that the execution during
“capture phase” is correct? [SO]
• Are there alternate ways to deal with nondeterminism than just running the application
multiple times? [AY]
• What other types of cost, besides execution
time, does adding Orstra to tests cost? [KN]

cs527-09 - Darko Marinov