The Future of Test Automation
Worksoft, Inc.
© 2003 All Rights Reserved
Business Case for Test Automation
– Why automate?
Test Automation Challenges
– Common mistakes
Software Development Trends
– Risk is increasing
Test Automation Trends
– Technology is evolving
Future of Test Automation
– Quality maturity
Business Case for
Test Automation
Why Automate?
Save time
Leverage resources
Reduce costs
Increase coverage
 Repeatable tests
– Multiple builds
– Multiple versions
 Unattended execution
– Time to design more tests
– Time to do manual testing
 Concurrent testing
– Write tests during requirements
– Develop tests during coding
 Reproducible tests
– Audit trail of test results
– Consistency from one tester to another
 Higher volumes
– Acquire test data from production
– Generate data programmatically
 Reduced rework
– Catch errors early
– Fewer emergencies, patches
Reduce failures in production
– Loss of revenues, market share
– Loss of customers, confidence
– Loss of user productivity
Reduce support, rework costs
– Costs to recover from errors
– Costs to remedy defects
– Costs to retest, redeploy
 Functionality increases over time
Inventory of existing features is cumulative
New features are added
Fixes must be verified
Ad hoc tests are needed
 Do the math
Cycle times are shorter
Developers outnumber testers
A 10% software change requires 100% testing
A single release will have multiple builds
Test Automation Challenges
Common Mistakes
You don’t have time
You don’t have resources
You don’t understand costs
You can’t measure coverage
Time is Limited
Shorter cycle times
– Schedules are market-driven, not effort-based
– Original schedule is always compressed
– Poor quality means longer cycles
Test automation takes time
– It’s a development project
– 5X to 10X manual effort
– Ongoing maintenance effort
Resources are Limited
 Recording tests takes manual effort
– Tests have to be performed manually
– You can’t start until software is ready
 Scripting takes skill
– Tools are programming languages
– Automation is a development project
 Implementation takes resources
– Design, document tests
– Develop framework, scripts
– Execute, evaluate results
Costs Aren’t Fully Apparent
Cost of acquisition
– Educate, evaluate, negotiate, license
 Cost to implement
– Planning, training, design, development,
– Requires development resources
 Cost of ownership
– Continuous maintenance, support
– Change, version control
– Training for new personnel
Coverage is Difficult to Measure
 Requirements are missing, obsolete
– Test cases are ad hoc
– Test knowledge is organic
 Regression tests are undefined
– Existing functionality not captured
– Past defects are forgotten
 New features are informal
– Poor or missing specifications
– Casual or inconsistent hand-off
Software Development Trends
Risks are Increasing
Increased exposure
Decreased cycle times
Greater functionality
More complexity
Increased Exposure
 Business rules exposed to users
– Rules engines enable analyst involvement
– More changes made faster
 Applications exposed to customers
– Web access becomes customer interface
– Direct impact on credibility
 Applications exposed to suppliers
– Web services open applications
– Direct impact on production
Decreased Cycle Times
 Market-driven schedules
– Software is competitive weapon
– Customer, competition drive demands
– Supply chain integration drives dates
 Accelerated development techniques
– Agile, extreme programming practices
– Short interval, incremental deliveries
– Cycles times of weeks or months
Greater Functionality
 Component-based development
– Incorporate new functionality quickly
– Less development, more testing
 Integrated applications
– Massive ERP, CRM systems
– Easy to configure, difficult to test
 Integrated enterprises
– Supply chain integration
– Standards for development, none for test
Test Automation Trends
Technology is Evolving
Data-Driven/Keyword Driven
Class/Action Framework
 Test cases executed manually and recorded
– No leverage
– No concurrent test development
 Scripts are unreliable
– Sensitive to timing, changes
– No logic for error handling
 Scripts are not maintainable
– Unstructured, undocumented
– Hard coded data means huge volumes
Example Script
# Add Account
set_window ("Account Manager", 15);
edit_set ("txtName", “Worksoft");
obj_type ("txtName","<kTab>");
edit_set ("txtAddress", “123 Main Street");
obj_type ("txtAddress","<kTab>");
edit_set ("txtCity", “Dallas
list_select_item ("cbState", "Texas"); # Item Number 42;
edit_set ("Edit", "Texas");
edit_set ("txtZip", “99999");
obj_type ("txtZip","<kTab>");
button_press ("Save_2");
 Test cases executed manually, recorded then modified
– No concurrent test development
– Requires programming skills
 Scripts are complex
– Writing programs to test programs
– Code added for synchronization, logic, error handling
– Code added for external data
 Scripts are difficult to maintain
– Script code can exceed application code
– Turnover means obsolescence
Example Script
# Add Account
set_window ("Account Manager", 15);
if ( obj_exists “txtName != E_OK )
gsStepStatus = "edit_Input: Edit object does not exists";
status = edit_set ( sAccountName );
if ( status != E_OK )
gsStepStatus = "edit_Input: Value could not be input";
Data-Driven/Keyword Driven Framework
 Keywords recorded or generated, then modified
– Script language specific
– Requires programming skills
 Scripts are complex, high volume
– Complete framework must be developed
– May be hundreds of action/keyword variations
 Scripts are difficult to maintain
– Added or modified for new functionality
– Multiple changes for application UI changes
Keyword Framework Structure
Proprietary Tool
Configuration Managed Assets
Execution Scripts
Calls to BPs
Example Test Case
Begin testcase Cust-AUD
Add Account
123 Main
Update Account
Acme AC
456 Elm
Delete Account
Bay Brick
789 Shore
San Jose California
Class/Action Framework
 Document then execute
– Pre-written libraries, customer-extensible
– Proven, supported application
 Point and click development
– Enforced structure, conventions
– Add, manage data easily
 Powerful framework
– Synchronization, error handling, recovery
– Standard and shared processes
 Relational database
– Automated impact analysis, maintenance
– Multi-user, centralized maintenance, administration
– All test assets can be housed in a database—no flat files
 Open architecture
– Single interface across multiple platforms, tools
Class/Action Framework Structure
Configuration Managed Assets
(in DB)
Proprietary Tool
Business Processes and data are decoupled from tool
Application Map
Define Process
Add Application
Add Window
Select Object
Select Action
Input Value
Literal data
Variable data
Generic Functions
public function edit_Input ( sObjectId, sObjectName, sWindowId, sWindowName )
if ( obj_exists ( sObjectId ) != E_OK )
gsStepStatus = "edit_Input: Edit object does not exists";
if ( GetStringArgument ( "Value", gsExpectedValue, sDefaultValue ) == FAILED )
gsStepStatus = "edit_Input: Action parameter (Value) could not be retrieved";
status = edit_set ( sObjectId, gsExpectedValue );
if ( status != E_OK )
gsStepStatus = "edit_Input: Value could not be input";
gbActionStatus = PASSED;
Specify Result on Pass, Fail
Specify Log Result
Add Data
Automated Documentation
Process / Window
Step Description
*Account Master
Press New CommandButton
Input Worksoft into Account Name TextBox
Input 123 Main into Account Address TextBox
Input Dallas into Account City TextBox
Select Item Texas from Account State ListBox
Press Save CommandButton
Verify Text is Equal To “Account Added”
Press OK CommandButton
Automated Impact Analysis
Automated Maintenance
Extensive Reporting
Quality Maturity
 Proprietary languages are being retired
– Shift to native languages
 Testability is becoming accepted
– Shift to cooperative development
 Automation is becoming automated
– Tools are maturing
 Increased management awareness
– Risks are higher
Shift to Native Languages
 Proprietary languages too costly
Skilled resources are scarce
10X development costs
Deployment license costs
Not all tools support all platforms
 Major vendors in transition
– Mercury: WinRunner to QuickTest Pro/VBScript
– Compuware: QA Run to Test Partner/VBA
– Rational: Robot VB to Java
Shift to Cooperative Development
 Development practices improving
– Coding standards, conventions
– Object properties, methods exposed
– Unique, persistent, meaningful names
 Development support enhanced
– Code instrumentation more likely
– Standards compliance more rigorous
– Test APIs more common
Tools are Maturing
 Next generation automation products
– Certify
– Unified TestPro, ABT, TestFrame
 Automated unit testing
– JUnit, NUnit
 Testing designed into architectures
– Test standards for Web services
Risks are Higher
 Sarbanes Oxley raises stakes
– CXO-level responsibility for financial accuracy
– Duty imposed for internal controls
– Potential civil, criminal liability
 Customer, exterprise integration raises
– Defects are visible to outside world
– Errors can impact entire supply chain
Build/Buy the Right Framework
 Less development, more testing
– Introduce development to testability
– Focus test engineers on changes and exceptions
– Empower business analysts to automate
 Reduce cost of implementation, ownership
– Less training, faster deployment
– Less maintenance, support
– Lower tool license costs
 Introduce enterprise standards
– Enable seamless end to end execution
– Provide portability across tools, platforms
– Leverage resources across applications

Automated Web Testing Strategies - SCQAA-IE