Software Testing &
Quality Assurance
Lecture 11
Created by: Paulo Alencar
Modified by: Frank Xu
1
Overview
Structural Testing
Introduction – General Concepts
Flow Graph Testing
DD-Paths
Test Coverage Metrics
Basis Path Testing
Guidelines and Observations
Data Flow Testing
Hybrid Methods
Retrospective on Structural Testing
2
An Overview of Path Testing
Coverage Criteria
•
Basic Measures
–
–
–
–
–
–
–
•
Statement Coverage
Decision Coverage
Condition Coverage
Multiple Condition Coverage
Condition/Decision Coverage
Modified Condition/Decision Coverage
Path Coverage
Other Measures
– Function Coverage
– Call Coverage
– Loop Coverage
•
Reference for this lecture: http://www.bullseye.com/coverage.html [1]
3
Coverage Analysis
• The objectives of coverage analysis are:
– To determine “gaps” in test cases (i.e. parts of the program that are not
tested)
– To establish a quantitative measure for the testing efficiency
– To identify additional tests that may be needed to eliminate “gaps”
• Coverage analysis assists on evaluating the efficiency of the test cases,
and not the quality product itself. Of course the objective is through
testing you can eliminate faults and increase the quality of the product
at the end.
• As we have seen there are several coverage criteria to identify paths in
the CFG and consequently generate test cases that cover these paths.
4
Coverage Analysis and Test Plans
• A good test plan utilizing coverage analysis must:
– Choose the appropriate coverage methodology or criterion (i.e.
statement coverage, decision coverage, condition coverage etc.)
– Establish minimum acceptable percentage level of coverage
– Utilize more than one coverage criterion
• Eventually, it is not a good idea to rely only on code
coverage. Other techniques are useful as well (i.e. BVA,
ECT, DTT).
5
Statement Coverage
• The measure reports on the percentage of executable
statements of the code exercised by a set of test cases. Also
known as basic bloc coverage or segment coverage.
• The major advantage of this measure is that it can be
applied directly to object code and does not require
processing source code. Performance profilers commonly
implement this measure.
• The major disadvantage of statement coverage is that it is
insensitive to some control structures
6
Statement Coverage Examples
if(A) then
F1();
F2();
Test Case: A=True
Statement Coverage
achieved
int* ptr = NULL;
if (B)
ptr = &variable;
*ptr = 10;
Test Case:
B=True
Statement Coverage
Achieved
Problem : if B is false the code
will fail with a null pointer
exception.
7
Statement Coverage Comments
• Statement coverage does not report whether loops reach their
termination condition - only whether the loop body was executed.
• Since do-while loops always execute at least once, statement coverage
considers them the same rank as non-branching statements.
• Statement coverage is completely insensitive to the logical decisions.
• One argument in favor of statement coverage over other measures is
that faults are evenly distributed through code; therefore the percentage
of executable statements covered reflects the percentage of faults
discovered.
• Statement coverage requires in most cases very few test cases to
achieve. Not acceptable to release code based on statement coverage
alone.
8
Decision Coverage
• Decision coverage (also known as branch coverage, all-edges
coverage, basis path coverage, decision-decision-path testing) reports
whether boolean expressions in control structures are evaluated to both
true and false values by the test cases.
• Note in decision coverage we are not interested in all combinations of
True/False values of all program predicates, just that we have
exercised all the different values in our test cases.
• The entire boolean expression is considered one true-or-false predicate.
• Additionally, this measure includes coverage of switch-statement
cases, exception handlers, and interrupt handlers.
9
Decision Coverage Examples
if(A)
F1();
else
F2();
if(B)
F3()
else
F4();
Test Cases for Decision Coverage:
A=T, B=T
A=F, B=F
if (A && (B || F1()))
F2();
else
F3();
Test Cases for Decision Coverage:
A=T, B=T
A=F
Problem: F1() never gets called. This
problem occurs in languages with
short circuiting boolean operators
10
Decision Coverage Comments
• This measure has the advantage of
simplicity without the problems of
statement coverage.
• A disadvantage may produce gaps in tested
code in programs written in languages that
have short-circuit logic operators (C, C++,
Java, etc.)
11
Condition Coverage
• Condition coverage reports the true or false outcome of each boolean
sub-expression.
• Condition coverage measures the sub-expressions independently of
each other.
• This measure is similar to decision coverage but has better sensitivity
to the control flow.
• An extension of Condition Coverage is the Condition/Decision
Coverage which is a hybrid measure composed by the union of
condition coverage and decision coverage. It has the advantage of
simplicity but without the shortcomings of its component measures.
12
Condition Coverage Examples
if(A && B)
F1();
else
F2();
if(C)
F3()
else
F4();
if(A && B)
F1();
else
F2();
if(C)
F3()
else
F4();
Test Cases for Condition Coverage:
A=T, B=T, C=F
A=F, B=F, C=T
Test Cases for Condition Coverage:
A=T, B=F, C=F
A=F, B=T, C=T
Problem: decision coverage not achieved
13
Multiple Condition Coverage
• Multiple condition coverage reports whether every
possible combination of boolean sub-expressions
occurs.
• The test cases required for full multiple condition
coverage of a condition are essentially given by
the logical operator truth table for the condition.
14
Multiple Condition Coverage Examples
if(A && B) // condition 1
F1();
else
F2();
if(C || D)// condition 2
F3()
else
F4();
Test Cases for MCC:
For condition 1
A=T, B=T
A=T, B=F
A=F, B=T
A=F, B=F
For condition 2
C=T, D=T
C=T, D=F
C=F, D=T
C=F, D=F
15
Multiple Condition Coverage Comments
•
For languages with short circuit operators multiple condition coverage is very
similar to condition coverage (but with more test cases).
•
For languages without short circuit operators multiple condition coverage is
effectively path coverage.
•
A disadvantage of this measure is that it can be tedious to determine the
minimum set of test cases required, especially for very complex boolean
expressions.
•
An additional disadvantage of this measure is that the number of test cases
required could vary substantially among conditions that have similar
complexity. Consider a && b && (c || (d && e)) and the
((a || b) && (c ||
d)) && e [1]
•
As with condition coverage, multiple condition coverage does not include
decision coverage.
16
Modified MCC
• Also known as MC/DC and MCDC.
• This measure also known as MC/DC and MCDC requires enough test
cases to verify every condition can affect the result of its encompassing
decision.
• For C, C++ and Java, this measure requires exactly the same test cases
as condition/decision coverage.
• Modified condition/decision coverage was designed for languages
containing logical operators that do not short-circuit.
• The short circuit logical operators in C, C++ and Java only evaluate
conditions when their result can affect the encompassing decision.
17
Modified MCC Examples
• To test if (A or B)
A:
T
F
B:
F
T
F
F
• To test if (A and B)
A:
F
T
T
B:
T
F
T
• To test if (A xor B)
A:
T
T
F
B:
T
F
T
18
Path Coverage
• This measure reports whether each of the possible paths in each
function have been followed. A path is a unique sequence of branches
from the function entry to the exit. This measure is also known as
predicate coverage.
• Since loops introduce an unbounded number of paths, this measure
considers only a limited number of looping possibilities.
• A large number of variations of this measure exist to cope with loops.
Boundary-interior path testing considers two possibilities for loops:
zero repetitions and more than zero repetitions. For do-while loops, the
two possibilities are one iteration and more than one iteration.
19
Path Coverage Examples
if (A)
F1();
F2();
if (A)
F3();
F4();
A
F1
F2
A
F3
F4
Paths:
A-F1-F2-A-F3-F4
A-F2-A-F3-F4
A-F1-F2-A-F4
A-F2-A-F4
Problem: only two
are feasible
A=T
A=F
20
Path Coverage Comments
• Path coverage has the advantage of requiring very
thorough testing.
• Path coverage has two severe disadvantages.
– The first is that the number of paths is exponential to the number
of branches.
– The second disadvantage is that many paths are impossible to
exercise due to relationships of data.
• There are many variations of path coverage to deal with
the large number of paths. Examples include, n-length subpath coverage reports whether we have exercised each path
of length n branches. Other examples include linear code
sequence and jump (LCSAJ) coverage and data flow
21
coverage.
Loop Coverage
• This measure reports whether each loop body in the
program has been executed zero times, exactly once, and
more than once (consecutively).
• For do-while loops, loop coverage reports whether the
body has been executed exactly once, and more than once.
• The valuable aspect of this measure is determining whether
while-loops and for-loops execute more than once,
information not reported by others measure [1].
22
Function Coverage
• This measure reports whether each function
or procedure in the program has been
invoked.
• It is useful during preliminary testing to
assure at least some coverage in all areas of
the software.
23
Call Coverage
• This measure reports whether each function
call in the program has been executed. The
hypothesis is that faults commonly occur in
interfaces between modules.
• Also known as call pair coverage.
24
Coverage Measures Comparison
• Relative strengths when a stronger measure
includes a weaker measure.
– Decision coverage includes statement coverage since
exercising every branch must lead to exercising every
statement.
– Condition/decision coverage includes decision
coverage and condition coverage (by definition).
– Path coverage includes decision coverage.
– Predicate coverage (paths as possible combinations of
logical conditions) includes path coverage and multiple
condition coverage, as well as most other measures.
25
Testing Strategy (1)
• Your highest level of testing productivity occurs when we
identify the most failures with the least effort.
• Effort is measured by the time
– Required to create test cases,
– Add the test cases to your test suite and
– Run the test suites.
• It follows that we should use a coverage analysis strategy
that increases coverage as fast as possible. This provides
the greatest probability of finding failures sooner rather
than later.
26
Testing Strategy (2)
• One strategy that usually increases coverage quickly is to first attain
some coverage throughout the entire test program before striving for
high coverage in any particular area [1]
• The sequence of coverage goals listed below illustrates a possible
implementation of this strategy [1]
–
–
–
–
Invoke at least one function in 90% of the source files (or classes).
Invoke 90% of the functions.
Attain 90% condition/decision coverage in each function.
Attain 100% condition/decision coverage.
• Notice we do not require 100% coverage in any of the initial goals.
This allows us to defer testing the most difficult areas and maintain
high testing productivity; by achieving maximum results with
minimum effort [1]
Source: http://www.bullseye.com/coverage.html [1]
27
Testing Productivity
Source: http://www.bullseye.com/coverage.html [1]
28
Descargar

Document