Test & Evaluation/Science & Technology Program
Unmanned and Autonomous Systems Test
FY2009 Program Execution Review
Project Status
UAST Accomplishments
UAST Lessons Learned
T&E/S&T Program FY2009 End-of-Year Program Execution Review
UAST Technology Overview
T&E capability shortfalls
UAS Layered Architecture
Predicting UAS Intelligence and Behavior
Emulating Mission & Environmental Complexity
Assessing UAS Effects and Capabilities
UAST Protocols and Design
UAS Test Bed and Environment
Cross-layer Mgmt
and Control
Comms (GIG)
UAST Reference Data (Ground Truth)
UAST Tools and Techniques
Major technology challenges (in the S&T solution development)
Develop algorithms for predictive behavior by integrating genetic algorithm, probability theory, and
autonomous behavior theory
Construct models to analyze UAS decision operations in complex and uncertain environment
Develop realistic test bed components, and environments to support UAS evaluation approaches to
unfolding missions via range assessments
Compile ‘truth data’ to support range safety, and protocols development
Develop methodologies and metrics to evaluate distributive control and situational awareness
Develop test protocols and algorithms to support on-board management system evaluation and UAS
T&E/S&T Program FY2009 End-of-Year Program Execution Review
UAST Technology Overview (cont.)
How does the focus area prioritize the
technology challenges (state your rationale)?
Comms (GIG)
UAS Layered Architecture
Cross-layer Mgmt
and Control
Mapping of T&E needs to POR or NPOR as defined by:
– DoD Roadmaps
– SPO/ Project Managers
By technology having broad applications across UAS
Cross-layer Mgmt
and Control
Metrics and Analysis
Comms (GIG)
Mission and
and Actuation
Terrain, Weather, Comms, Threats, Friendlies, Neutrals
Air, Space, Surface, Subsurface
UAS Architecture
Layer Emulations
And Live
(Physical, Tactical,
Social, etc.)
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Roadmap leverage of accepted reference
models for testing UAS
• NIST Performance
Decomposition Model
• As opposed to
componentized UAS
view presented in
• Another alternative
would be a test
framework over
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Operational UAST Challenge
UAS Test Challenge
UAS Environment
Test Control
Data to
Scoring and Feedback
System of Systems
Under Test
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Autonomy-Base UAST Challenge
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Cognitive Autonomous System
Tester (CAST)
Project Description
The overall CAST methodology learns each mission’s • CAST integration strategy report
goals, learns vehicle-specific characteristics and
• Stand-alone toolkit for UAS autonomous behavior
behaviors, and identifies vehicle behavioral anomalies.
In addition, a self-organizing knowledge representation
• Final Project report
system will organize and categorize the information in a
way that permits the tester to test complex emergent
behaviors and anomalous behaviors to be differentiated
and evaluated during highly difficult missions and
mission failure modes.
Key Future Events & Dates
Start/Finish : October 2008/September 2010
PDR: July 2009
Phase 2 Component Demo: September 2009
Phase 2 Scientific Report: September 2009
CDR: January 2010
Final Integrated CAST Demo: August 2010
Final CAST Scientific Report: September 2010
Final CAST Transition Report: September 2010
Project 1 of 13
Transition Partners & Dates
• Army Research Lab-BCT (FCS) Mission Based
Applications. 4Q FY10
• Army Advanced Technology Directorate (AATD), TBD
T&E/S&T Program FY2009 End-of-Year Program Execution Review
High-Fidelity Communications,
Modeling & Analysis (HFCMA)
Project Description
HFCMA is a Live/Virtual/Constructive T&E framework
that merges Live and Constructive assets, including
heterogeneous collaborative UxVs with the ability to
mirror One to Many Live UxV’s performance in the V/C
simulation. T&E Use Case is a) multiple vehicles
interacting, b) each with collateral systems and c) a test
system that must coherently stimulate the vehicles
under test and capture limits of UxVs whole
system performance and integrity. HFCMA is
concerned with testing the dynamic limits of autonomy
Key Future Events & Dates
Transition Partners & Dates
Start/Finish : January 2008/May 2010
Phase 2 Documentation: September 2009
Collaborate with transition partner: December 2009
Define User Cases: January 2010
Incorporate Selected assets Demo: February 2010
Effectiveness of interaction and multiple assets report
April 2010
• Final Integrated report: May 2010
Project 2 of 13
LVC framework for test data recovery and analysis
Delivery of prototype test framework
Framework training manual
Final project report
Incorporate use cases for air, ground & sea
Verify adequacy with transition partner
ATEC-BCT(FCS) Robotic convoy 1Q/FY10
NAVAIR/452 FTS-Scan Eagle/Predator 2Q/FY10
NUWC-Undersea environments TBD
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Remote Embedded System Test
Project Description
REST is the integrated set of technologies that
replaces a wired instrumentation system with a
wireless, self powered, self organizing, self healing
network of sensor nodes designed to be
manufactured (embedded and assembled) into the
platform structure and provide continuous test and
evaluation capability as well as integrate into the
platform Integrated System Health Manager.
REST is designed to provide UAST capability for
multiple platform types and DoD services.
• Prototype TRL 6 system components
• Flight and Ground Demonstration
• Final Report and White Papers
Key Future Events & Dates
Transition Partners & Dates
Start/Finish : October 2007/December 2009
Field Demonstration: August 2009
Final Report: October 2009
Conduct Field Test November, 2009
Final Report: December, 2009
Project 3 of 13
• Advanced Composite Cargo Aircraft (ACCA)
Program, Phase III FY09 (REST TRL 6 Baseline
Technology) – Installed and evaluated in ACCA
Fuselage Fatigue Test Article (Mar CY2010)
• Vulture (DARPA), In-Work
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Software Configurable MultiChannel Transceiver (SCMT)
Project Description
Flexible multi-band command and control
• Project will provide a 4-channel SCMT demonstration
communications system supports test data capture and prototype
command/control applications during complex multi• Set of capability demonstrations for core functionality
agent (UAS) test scenarios. Effectively implements a
• Aeronautical TM modulation formats
software-customizable transceiver system that is
frequency agile with modulation and protocol
independence. SCMT encompasses four transmitters
and four receivers in a single unit the size of a
commodity disk drive programmable for any frequency
band between 1 MHz and 6 GHz.
Key Future Events & Dates
• Start/Finish : August 2008/August 2009
• Transceiver Completion Report: August 2009
• Transceiver link Demonstration: October 2009
Project 4 of 13
Transition Partners & Dates
• Phase 1 completion Oct 2009. Must coordinate with
SET to determine if project will continue
• Test community interest: 452nd Test Wing, NAVAIR,
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Micro-beacon Tracking of
Autonomous Systems (MTAS)
Project Description
Effective evaluation of a UAS performance depends on • Three prototype beacons
ground truth about the world, the vehicle and what the • Full scale system demonstration at Ft. Devens airfield
• Final project report
vehicle senses. MTAS is designed to provide vehicle
localization data utilizing small self-contained, and
inexpensive RF micro-beacons affixed to test vehicles.
Portable ground stations receive the signal and
calculate range to each test vehicle.This supports
multiple autonomous systems, low latency, sub-meter
positioning performance, and a full performance
operation in GPS-denied environments.
Key Future Events & Dates
Transition Partners & Dates
• Brigade Combat Team Modernization (BCTM)
Experiment (RCX), 4Q/FY09
Start/Finish : October 2007/September 2009
System Design Review: June 2008
Field Demonstration: August 2009
Final Report: October 2009
Project 5 of 13
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Virtual Autonomous Teaming Tool
Project Description. VATT is a tool used for
autonomous vehicle simulation, teaming, and decision • Requirements Document
making analysis. It aims to develop a VATT
• VATT-T Software Toolkit
configuration for T&E applications, that can predict
• Final Report
how modifications to external stimuli will produce
changes within the UAS autonomy logic processes.
VATT environment strives to seamlessly integrate
simulation subsystems with real UAS hardware
subsystems. Simulated systems could replicate
multiple platforms for the development of mutli-platform
behaviors thus supporting repeatable events and
detailed system/ event logging.
Key Future Events & Dates
Transition Partners & Dates
• RTTC; John Hardison- 4 Q FY10
Start/Finish : April 2008/November, 2010
End of Phase 2 Demo: February, 2010
Start of Phase 3: February, 2010
End of Phase 3: November, 2010
Project 6 of 13
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Standards, Software and Hardware for
Multi-Resolution Maps/Models (MRMM)
Project Description
High precision site data is an essential tool in deriving
ground truth. Comparison not only of the vehicle’s
position with actual location but with what the vehicle
senses in its surroundings. This S&T project is based
upon analytical studies and laboratory studies to
physically validate theoretical foundations using
software and standards for the implementation of
multiresolution models for unmanned autonomous
systems testing. The analysis, models, software and
standards will be evaluated through both hardware
emulation and experimental evaluation.
• Software module libraries modified for hardware
• Ground / Air testing of laptop payload
• Detail design specifications of Prototype FPGA-based
I/O Board.
Key Future Events & Dates
Transition Partners & Dates
• Army Research Laboratory, 4 Q/FY10
• Air Force Research Laboratory, 4 Q/FY10
Other dates of interest:
• ARL Test Collaboration TBD LIVEFIRE Program
• AFRL/NASA Phase II SBIR – TBD Helicopter flight
April 2008/December 2010: Start/Finish
Sept 27-28 : Final Large Prototype Static Test
Oct 15-16 : Final Small Prototype Static Test
Nov 5-6 : Final Large Prototype Dynamic Test
Nov 26-27: Final Small Prototype Dynamic Test
Dec 10-11: Final Demo Phase II
Project 7 of 13
T&E/S&T Program FY2009 End-of-Year Program Execution Review
An Integrated Agent-based Framework for
Simulated, Virtual and Live Testing of Teams of
Unmanned Vehicles (IABF)
Project Description
IABF will allow for systematic and structured T&E of
technologies in the cognitive/social domain using a
combination of simulation, hardware in the loop testing
and live testing. Testers will quickly and efficiently be
able to develop test plans and validate UAS Team
performance with respect to:
• C2 navigation approaches utilizing NLOS techniques
• Collision avoidance
• UAS Team coordination and performance
• Fault tolerance under various failure modes and
bandwidth constraints
Key Future Events & Dates
Transition Partners & Dates
• Picatinny Arsenal, ARDEC,1Q FY10
• Naval Research Lab (NRL),4QFY09
Start/Finish : September 2008/September 2010
CDR: April 2008
Phase 1 Demo: September 2008
Phase 2 Demo: August 2009
Final Report and User Documentation
Project 8 of 13
DCTF Software and libraries
Phase I : Indoor demonstration of Test-framework
Phase II: Outdoor advanced Technology Demo
Final report and User documentation
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Reconfigurable Wireless
Measurement System (RWMS)
Project Description
Ground truth measurement infrastructure to support
UAS testing and vehicle safety analysis. The measure
system consists of components that can provide an
alternative source of data to compare with those
organic to the system under test. Three levels of
modularity, namely, i) physical interface (face plate), ii)
hardware stack, and iii) computational fabric (FPGA
based) are employed to provide maximum flexibility
• RWMS prototype (HW and SW)
• Documentation of the miniaturized RWMS design
RWMS Demo in 2 or more UAS test case Scenarios
• Final report and analysis of demonstrated capability
Key Future Events & Dates
Transition Partners & Dates
• Start/Finish: January 2009/Sep 2009
• LVC & RWMS Demonstration: August 2009
• Phase 2: Prototype & Documentation: Sept. 2009
• ATEC White Sands Missile Range, 1QFY10
Project 9 of 13
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Flexible Command and Control
Systems (FCCS)
Project Description
FCCS is a user facing application framework that
• Prototype Demonstration
provides C2 capabilities, its primary functions are to
• FCCS Documentation
reduce the amount of time to display meaningful data to • FCCS Final System
the user and improve the C2/HRI functionality of the
targeted robotic device. When a new UAS is presented
for testing, The C2 system must be able to be rapidly
adapted to the unique qualities of the new UAS. The
C2 system must also have a standardized, welldocumented framework so that UAS developers can
prepare their UAS for integration with the C2 system
before arriving at the test facility.
Key Future Events & Dates
Transition Partners & Dates
• Global Observer 412 FTS EDWARDS, 1QFY10
Start/Finish: October 2007/December 2009
Acquisition of telemetry source: June 2008
Delivery of Second Telemetry Source: Sept. 2008
Initial Demonstration: December 2008
FCCS workbench demonstration: August 2009
Delivery of Final Documentation: December 2009
Project 10 of 13
T&E/S&T Program FY2009 End-of-Year Program Execution Review
A Prescriptive and Adaptive Testing
Framework (PATFrame)
Project Description
PATFrame is a decision support tool encompassing a
• Tool delivery (Software)
prescriptive and adaptive framework for UAS SoS
• Final demonstration
Testing. PATFrame will be implemented using a
• Final Documentation
software dashboard that will enable improved decision
making for the UAS T&E community. The technology
addresses the Modeling & Architecture and Testbeds &
Test Environments areas in the UAST capability
Key Future Events & Dates
Transition Partners & Dates
• U.S. Army Operational Test Command (USAOTC), Ft.
Hood, TBD
• Naval Air System Command (NAVAIR), Patuxent
River, MD, TBD
• Air Force: 303rd Aeronautical Systems Wing, WrightPatterson Air Force Base, OH, TBD
Start/Finish : October 2009/September 2012
PDR: September 2010
Event Workshop 1&2: September 2010
CDR: March 2011
Event Workshop 3: March 2011
Phase2 :Progress Report Tool: September 2011
Event Workshop 4: September 2011
Final Demonstration: September 2012
Final Demonstration: September 2012
Final out brief & Deliverables: September 2012
Project 11 of 13
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Synthetic Collective Unmanned
Underwater Laboratory (SCUUL)
Project Description
SCULL applies dynamic systems theory to design
reduced order models of cooperative UUV navigation,
sampling, performance, and control. UUV technology
includes operations in remote/inaccessible, and
dynamic environments; autonomous trajectory
generation, tracking, and coordination. The value
added to UUV testers is (1) tests to guide UUV
development and deployment; (2) help to define T&E
for cooperating underwater vehicles; and (3) provide
T&E with a capability for fielding autonomous systems
that are suitable, effective, and survivable.
• Interim report on reduced-order modeling of
cooperative behavior
• Interim & final reports on model-based simulation
• Prototype micro-UUV platform and software
Key Future Events & Dates
Transition Partners & Dates
• Start/Finish : October 2009/October 2012
• Demo underwater motion capture system: Oct 2010
• Demo Model-based simulation and hardware
prototype: October 2011
• Phase 2 Scientific Report: October 2011
• Prototype deployment and demonstration: 2012
• Naval Undersea Warfare Center (NUWC), 1Q FY12
Project 12 of 13
T&E/S&T Program FY2009 End-of-Year Program Execution Review
M&S for UAST
Project Description
Proposed technology is a developmental support tool
that focuses on a collection of entities for UAS testing.
This support tool is known as an Ontology/Taxonomy
application containing knowledge about existing UASs.
Real time positioning of equipment to monitor UAS
experiments via the internet for T&E observations
occurring within the UAST Enclave. The
ontology/taxonomy will be implemented as a decision
support system to improve UAS T&E outcomes.
Appropriate UAS choice for the mission & environment.
Key Future Events & Dates
Transition Partners & Dates
• UAST WG/ UAST Prototype Testers, 4Q FY10
Start/Finish : October 2008/December 2009
Progress report of current tasks: December 2008
Ontology/Taxonomy task begun: January 2009
Protocols Document: December 2009
Software solution for Ontology researched and
identified: December 2009
Development of Test Protocols
Evaluation tool
Ontology/Taxonomy working application
Final Report
UAST Technical Topic Areas = Framework
& Protocols
Modeling and
Test Beds & Test Environments
LU Lab
( ”, t ”,”, x”, y”, z”)
Ft. Bliss
Project 13 of 13
T&E/S&T Program FY2009 End-of-Year Program Execution Review
’, t’, x’, y’, z’)
,t, x, y, z)
UAST FY09 Achievements
Remote Embedded Development of alternative energy
harvesting and storage technology.
System Test
Self healing power harvesting
Implemented and demonstrated
switching between vibration and
thermal energy harvesting modes as
well as dynamic power summing from
vibration and thermal.
Benefit T&E
Self powered wireless structure
sensor network replacement for
wired structure network provides for
continuous T&E at low installation
weight and O&M.
Micro Beacon
Track of
System (MTAS)
First available T&E/S&T technology
which addresses high accuracy full I
performance TSPI in GPS denied
environments. Demonstrated on
manned and small UGV (static and
moving vehicles with up to 3 vehicles
tracked simultaneously.
•Sub-meter 3d positional accuracy.68m
•Position fix latency-1.0 s
•1km2 tracking range
•Applicable to manned or unmanned
systems and dismounted soldiers
Integrated Agentbased Framework
Unified platform independent
distributed control framework for
simulated and mixed mode testing
(live-simulated) of unmanned systems
Extensive use of M&S with
integrated comm/networking
capability, allowing support for a
broader and more robust set of
testing in a more realistic simulation
T&E/S&T Program FY2009 End-of-Year Program Execution Review
UAST FY09 Achievements (cont.)
Benefit T&E
System (RWMS)
Miniaturized RWMS to provide
unobtrusive real-time situational
measurements and perform
override/shutdown actions if safety is
compromised. Report the kinematics,
monitor the operating environment,
UAS health monitoring.
Non-intrusive, user-defined data
interface device for real time data
collection of UxV system and subsystem levels for both truth and
perceived truth data. Addresses the
need for multi-platform
modular,scalable data collector tool.
Candidate solution for ATEC’s Field
Data Collector.Utility for TENA in a
Resource Constrained Environment
Configurable MultiChannel
Emerging and next-generation test
instrumentation increasingly require
network communication to manage
complex and dynamic test scenarios,
particularly for UAST. Adapting
wireless communication infrastructure
to accommodate challenging testing
needs can benefit from reconfigurable
radio technology.
SCMT enables four transmitters and
four receivers in a single unit,
programmable for any frequency
band between 500 MHz and 6 GHz.
Multiple communications links
needed for interface to different
systems. Real-time reprogramming
of transceiver will allow versatility of
T&E/S&T Program FY2009 End-of-Year Program Execution Review
FY 09 Lessons Learned
UAST Working Group MUST shift focus from 'marveling about UxS's' to 'T&E'ing UxS's.'
We must devise a UAST framework so that the S&T that will enable various 'paths' through
the framework can be highlighted and justified then the ROI of S&T projects
can be measured.
We can move from a focus on a) measuring the autonomy algorithm in the payload to b)
measuring the autonomy of the Whole System and doing so sufficiently to Predict, with
stated Confidence, the anticipated Suitability, Safety, Effectiveness and Survivability of the
whole system. "How can you 'prove' that your systems design will solve the customer's
problem before you build and prove that design?"
The UAST process must be synchronized with PMO's view of the world and the TRMC
process. Drop 'ConOps' and talk in terms of Use Cases and Use Case Portfolios. Then
S&T projects must state Objective in terms of Use Case hot spots and a) S&T intent
regarding ameliorating the hot spots and b) requirements allocated to other of TRMC's S&T
Use Cases must state the T&E implications of the whole UxS. T&E implications are
objectives and concepts as stated at the Effects --> Capabilities -->Algorithm/Timeline level
(not System Shall statements) across development, prototype, IOC, production
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Accelerating UAS Deploy through Test
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Safety in the Battlespace
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Three Phases of Startups
Glimpse of “Market”
Learning enough to match
market with technologies
Create value proposition
for garnering ROI
Ambiguity of Operational
Necessity; Fast Tracking;
Need for Testing for
Safe, Suitable, Effective,
and Survivable UAS
We are currently finishing Phase II
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Three Questions
• Are we progressing fast enough to help the Warfighters?
– Prototypes, Demos, Beta-Testing
• How can we accelerate S&T activity for UAST?
– Use cases
– Connections to UAS programs: workshops, outreach, conferences,
web site, magazine, flyer
– Other? AUVSI, Robotic Rodeo, PERMIS, NIST, SAE, INCOSE, IEEE,
NDIA, UARCs, Academia
• These rules for triage?
– Must address a T&E need
– Must require S&T
– Should have a big payoff
– Should have wide applicability
– Should have a transition partner
T&E/S&T Program FY2009 End-of-Year Program Execution Review
Working Group
MUST devise a UAST ROI framework
Use-case mapping of test needs for safe, suitable, survivable and effective UAS
Technology forecasting
•CTEIP Mitigation
– Solicit needs from Warfighters, Testers, Developers, PMs
– Transition partner development
– How do we accelerate S&T investments into T&E capability?
•UAST overarching role
– Filling in the gaps for testing UAS that are being deployed using rapid acquisition
– Focus on autonomy T&E
– Focus on safety of UAS
•Stakeholder collaboration (COI-UAST)
– Use Cases must state the T&E implications of the whole UxS
– T&E implications are objectives and concepts as stated at the
– Effects --> Capabilities -->Algorithm/Timeline level (not System Shall statements) across
development, prototype, IOC, production
T&E/S&T Program FY2009 End-of-Year Program Execution Review