Testing safety-critical
software systems
Marcos Mainar Lalmolda
Quality Assurance and Testing
20th November 2009
What a safety-critical software
system is
Programming features and
Approaches on design
What a safety-critical software
system is
 A safety-critical software system is a computer system
whose failure or malfunction may severely harm people's
lives, environment or equipment.
 Some fields and examples:
 Medicine (patient monitors)
 Nuclear engineering (nuclear power station control)
 Transport (railway systems, cars anti-lock brakes)
 Aviation (control systems: fly-by-wire)
 Aerospace (NASA space shuttle)
 Civil engineering (calculate structures)
 Military devices
 Etc.
Safety-critical Standards
 Industries specific
 Medical device software: IEC 62304
 Nuclear power stations: IEC 60880
 Aerospace: AS9100A
 Airbone: DO178B
 Scale of 5 safety integrity levels: 4 is very
high, 0 not safety related.
 Safety engineering
Programming features and
languages (I)
 General principle: Try to keep the system as simple
as possible.
 Programming features not recommended:
 Pointers and dynamic memory
 Unstructured programming (gotos)
 Variant data
 Implicit declaration and initialisation
 Recursion
 Concurrency and interrupts
Programming features and
languages (II)
Features which increase reliability:
Strong typing
Run time constraint checking
Parameter checking
Language to be avoided: C
Language recommended: Ada
Ada subset for safety-critical software:
Other languages: increased overhead
Approaches on design
 Formal methods
 Assume that errors exist and design prevention and
recovery mechanisms.
 “Program verification does not mean error-proof
programs […]. Mathematical proofs can also be
faulty. So whereas verification might reduce the
program-testing load, it cannot eliminate it” (F.P.
Brooks, No Silver Bullet, 1987).
Testing safety-critical software
systems (I)
 Basic idea: Identify hazards as early as possible in the
development life-cycle and try to reduce them as much as
possible to an acceptable level.
 Remember: Always test software against specifications!
 Independent verification required
 If formal methods have been used then formal mathematical
proof is a verification activity.
 Already known techniques used for typical systems
 White box testing
 Black box testing
 Reviews
 Static analysis
 Dynamic analysis and coverage
Testing safety-critical software
systems (II)
 Specific procedures and techniques from safety
 Probabilistic risk assessment (PRA)
 Failure modes and effects analysis (FMEA)
 Fault trees analysis (FTA)
 Failure mode, effects and criticality analysis
 Hazard and operatibility analysis (HAZOP)
 Hazard and risk analysis
 Cause and effect diagrams (aka fishbone diagrams
or Ishikawa diagrams)
Probability Risk Assessment
Risk Criteria
*From Safety-Critical Computer Systems –
Open Questions and Approaches
presentation, Andreas Gerstinger,
February 16, 2007, Institute of Computer
Technology, Wien
Fault tree analysis (FTA)
 A graphical technique that provides a systematic description of
the combinations of possible occurrences in a system which can
result in an undesirable outcome (failure).
 An undesired effect is taken as the root of a tree of logic
 Each situation that could cause that effect is added to the tree
as a series of logic expressions.
 Events are labelled with actual numbers about failure
 The probability of the top-level event can be determined using
mathematical techniques.
An example of a Fault tree
*From http://syque.com/quality_tools/toolbook/FTA/how.htm
 Complex subject
 Suitably trained and experienced people are key to
the success of any software development.
 Main objective of testing techniques: minimise risk of
implementation errors.
 Above all, the best way to minimise risk both to
safety, reliablity and to the timescale of a software
project is to keep is simple.
 Wikipedia. http://en.wikipedia.org
 IPL Information Processing Ltd, An Introduction to Safety Critical Systems, Testing
Papers. http://www.ipl.com/include/download/CookieRequestPage.php?FileID=p0820
 IPL Information Processing Ltd, An Introduction to Software Testing, Testing Papers.
 Evangelos Nikolaropoulos, Testing safety-critical software, Hewlett-Packard Journal,
June 1997.
 Frederick P. Brooks, Jr. , No Silver Bullet: Essence and Accidents of Software
Engineering, 1986.
 Andreas Gerstinger, Safety-Critical Computer Systems – Open Questions and
Approaches presentation, February 16, 2007, Institute of Computer Technology, Wien.
 Fault Tree Analysis: How to understand it.

Testing safety-critical software systems