Evolution of a Measurement-Based Data Processing System for
Erich Franz Stocker NASA/GSFC Code 610.2
Mary Cleave NASA Headquarters
Stocker - Page 1
Evolution Measurement Based System-IGARSS 2006
Aug 1, 2006
• Determining measurements
• Architectural characteristics of a measurementbased system
• Evolving TRMM mission data processing system to
measurement based Precipitation Processing
System (PPS)
• Architecture-based process in use
• Summary
Stocker - Page 2
Evolution Measurement Based System-IGARSS 2006
Aug 1, 2006
Establish of a Measurement
NASA Earth Science is evolving
– Measurement-based programs for mature measurements
– Away from single point missions.
NASA intends to evolve to measurements
– The measurement has been made via remote sensing on previous missions
– When a program has matured to the point that follow-on flights with a number of appropriate
instruments have been identified.
– Flights and instruments need not all be provided by NASA
Measurement programs are warranted when NASA needs
– A parameter or set of related parameters needed for the measurement have emerged as
key to a climate research area with an NASA Earth Sciences research program
– To support climate research with a seamless data set from mission to mission.
– Consensus of the research community as to the importance of the parameter to that
The selection of parameters that constitute a climate data record
(measurement) is being developed within NASA Earth Sciences by
Stocker - Page 3
No fixed “one-size fits all” process but focused on an identified research need
Science Divisions lead and
Managed by the science program managers
in conjunction with the science community
Evolution Measurement Based System-IGARSS 2006
Aug 1, 2006
Characteristics of Measurement Approach
• Evolution from a science team per mission to a science team per
– TRMM science team has become the Precipitation Measurement Missions
(PMM) team
– Funding and research announcements are focused on the single mission
science team
• Programmatic direction from a single Program Scientist within NASA
Hq Earth Sciences.
• Measurement based data processing systems may have evolved from
a single mission system but support multiple missions (instruments)
for processing the measurement
– Build on existing relationship with the measurement science team
– Maintain processing capabilities from one mission to next
– Handle new missions, parameter data, or related instrument data as needed
with minimum incremental cost for the processing
Stocker - Page 4
Evolution Measurement Based System-IGARSS 2006
Aug 1, 2006
Advantages of the Measurement Approach
Ensures continuity of measurements and data products key to NASA Earth
Science research focus
Helps ensures that missions are focused on the science measurement needs
rather than on the hardware being developed
Places missions within the overall direction of the measurement science
Ensures that measurement based science data processing systems
– Are under science umbrella rather than the computer science one
– Have the science expertise to produce measurement products correctly, efficiently and with
– Provides infrastructure support from mission to mission to
• Keep costs low – through incremental expansion costs
• Focus on the needs of the measurement data collection and processing
• Provide measurement expertise and continuity
• Ensure that processing not so general that all focus on the measurement is lost
Facilitates science research on the measurement even when no mission is
immediately available.
Stocker - Page 5
Evolution Measurement Based System-IGARSS 2006
Aug 1, 2006
Precipitation Processing System
• Precipitation identified as ready for evolution to measurement
– Many missions
– Long history of producing the measurement
– Consensus of the science community
• TRMM Data and Information System (TSDIS) designated as a
prototype measurement based data processing system
• Basic requirements
– Be able to handle data from a number of different sources
– Be able to able to support TRMM, existing precipitation data products and the
Global Precipitation Measurement mission
– Be able to add or remove data streams (with a known incremental cost for
– Be able to transition the data processing system to another provider when
desired (i.e. move processing )
– Provide seamless consistent data set of precipitation through many missions
Stocker - Page 6
Evolution Measurement Based System-IGARSS 2006
Aug 1, 2006
Single Point to Measurement Based
Stocker - Page 7
Evolution Measurement Based System-IGARSS 2006
Aug 1, 2006
PPS Evolution Approach
Totally architectural approach
– Analysis
– Requirements
– Design
– Implementation
– Testing
– Configuration Management
Start with the TSDIS architecture and reuse as many architectural components
– Identify all architectural components
– Identify those that are fixed in single source approach
– Determine all dependencies (language, design, and code)
From this starting part evolve PPS by
– Identifying the characteristics needed because of the given charge
– Identify all the components required
– Identify all the properties and constraints on processes, data flows, and
– Multiple architectural walkthroughs before first design/implementation and
walkthroughs throughout the process
Stocker - Page 8
Evolution Measurement Based System-IGARSS 2006
Aug 1, 2006
Architecture Characteristics for
“Measurement Based”
Minimum human interaction
Easily moved to different hardware platforms
Recover automatically for
predetermined anomalies
Easily moved to different operating system platforms
Ability to use with different supporting COTS
Able to call for help
Recover automatically from
Run on small Equipment
Run on large equipment
No hard software
Operations settable via data
No breakage on scaling
No lock-up on scaling
GPM Req for Algorithms
Simplified deletions
Simplified modifications
Simplified additions
Run in different modes
Ease of maintenance
Expand functionality beyond initial evolution
Adaptable to new and different situations/uses
Usable in unanticipated manner
Data streams addable or
changeable via data
Process interactions settable
via data
DB and other tools settable
via data
Support running of “special or injected code” in plug-in mode
Stocker - Page 9
Evolution Measurement Based System-IGARSS 2006
Aug 1, 2006
Evolving for “Measurement Based” Processing
Processing Infrastructure
Stocker - Page 10
Evolution Measurement Based System-IGARSS 2006
Aug 1, 2006
PPS Architecture Overview
Project Specific
Existing Core
Stocker - Page 11
Evolution Measurement Based System-IGARSS 2006
Aug 1, 2006
Some PPS Aspects
Tied to the ongoing NASA REASON CAN community standards efforts
– Support ESML (GHCC)
– Support OpenDaPS (URI)
– Data storage in HDF4 and HDF5
– Evaluating other standards as identified
• Working to ensure availability in generic GIS data formats
Allow dynamic subsetting online (in mulitple formats)
– Geographic
– dataset parameter subsetting
Flexible toolkit
– Easily add formats (currently binary, HDF4, HDF5, NetCDF)
– Add languages (currently C, C++, and F77)
– Dynamically produce read/write routines for subsetted data
Use of SOAP to provide access to special services (e.g. user code within PPS)
Close working with other precipitation centers
Stocker - Page 12
Evolution Measurement Based System-IGARSS 2006
Aug 1, 2006
Examples Working with Others
• Working with CSU to prototype
– Intercalibrated 1C radiometer data from multiple sensors
– A appropriate logical format for the representation of such data (this also with a
wider community using radiometer data
• Working with Univ of Utah and MSFC on precipitation features
• Working with UAH/GHCC on prototyping aspects important for
– XML based data ingest formats
– ESML for HDF4 and F77
• Working with several different groups on GIS versions of
precipitation data
• Testing dynamic subsetting capabilities
– To satisfy user requests on TSDIS
– To satisfy study needs for GPM
• Driven by the needs of the precipitation community and the work
done through REASON CAN
Stocker - Page 13
Evolution Measurement Based System-IGARSS 2006
Aug 1, 2006
Prototype Science Discipline Center
Stocker - Page 14
Evolution Measurement Based System-IGARSS 2006
Aug 1, 2006
PPS/GPM Operations Concept
Stocker - Page 15
Evolution Measurement Based System-IGARSS 2006
Aug 1, 2006

IGARSS Measurement Based Systme