Results of the Assessment of the
Defense Experimental Program to Stimulate
Competitive Research (DEPSCoR)
Brian Zuckerman
Presented to
COSEPUP Evaluation of the National Science
Foundation's Experimental Program to Stimulate
Competitive Research (EPSCoR) and Similar Programs
in Other Federal Agencies
December 17th, 2012
Slide 1
Study Origin and Timeline
• FY2008 Defense Authorization Act (Public Law 110-181),
Section 241, instructed the Secretary of Defense to utilize a
defense Federally Funded Research and Development Center
(FFRDC) to carry out an assessment of the DEPSCoR
program.
• Institute for Defense Analyses was tasked to conduct study
February 2008
• Results briefed to Senate and House Armed Services
Committee staff (SASC/HASC) November 2008
• Study results cleared for public distribution January 2009
Slide 2
Legislative Mandate for Study
1. Tangible results and progress toward the objectives of the
program
a. Applications used by, or supportive of, operational users
b. Expanded national research infrastructure
2. Activities consistent with statute
3. Assessment of program elements
4. Assessment of activities of state committees
5. Advantages and disadvantages of institution & state based
formulas
6. Mechanisms for improving the management and
implementation of the program
Slide 3
Descriptive Statistics
• Between 1993 and 2008, 729 total DEPSCoR awards
– 546 individual Principal Investigators (PIs)
» 121 PIs with multiple awards
» 42 with three or more
» 1 PI with eight awards
– 22% of PIs have won 42% of awards
• 1993-2008 funding of $243 million
– Decline after 2000 peak partially reversed in 2008 competition
• 27 states and territories (states) have been eligible for at least one
year since program authorized in current form in 1995, plus Missouri
(eligible in 1993)
– All eligible states except for the Virgin Islands have won awards
• 19% (5) of eligible states have won 35% of awards
– Montana, Alabama, Oklahoma, Nebraska, South Carolina
• 7% (5) of institutions have won 28% of awards
– Montana State, U. Nebraska-Lincoln, U. Wyoming, West Virginia U.,
U. Arkansas
Slide 4
DEPSCoR Program Objectives
1. To enhance the capabilities of institutions of
higher education in eligible states to develop,
plan, and execute science and engineering [S&E]
research that is competitive under the peer-review
systems used for awarding federal research
assistance
2. To increase the probability of long-term growth in
the competitively awarded financial assistance
that institutions of higher education in eligible
states receive from the federal government for
science and engineering research
Slide 5
The DEPSCoR State Share of
DOD S&E Increased
Share of DoD University Funding to DEPSCoR States,
Adjusting for DEPSCoR Programmatic Funding
18%
16%
14%
12%
10%
8%
6%
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
Year
R&D (Actual)
R&D (Linear Model)
Research (Actual)
Research (Linear Model)
Source: National Science Foundation (NSF) Survey of Federal Science and Engineering Support to Universities,
Colleges, and Nonprofit Institutions and the DEPSCoR program database of funding totals by state and year
Note: Graph includes all states ever involved in DEPSCoR program 1995-2008
Note: The dotted lines represent linear regression models applied to the data
Slide 6
2005
DEPSCoR Funding As a Percentage of
DOD Funding in DEPSCoR States
Percentage of DoD Funding in DEPSCoR-Eligible
States Accounted for by DEPSCoR Program Funds
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
Year
Percentage of DoD Total University S&E R&D
Percentage of DoD Total University S&E Research
Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions
and the DEPSCoR program database of funding totals by state and year
• DEPSCoR has declined in importance as a source of funding for
eligible states since 2000
Slide 7
2005
“Success” Varied Among States
20.0%
18.0%
Sources: NSF Survey of Federal Science and
Engineering Support to Universities, Colleges, and
Nonprofit Institutions, the IDA DEPSCoR
database, and the DEPSCoR program database of
funding totals by state and year
Share of DoD University S&E R&D
16.0%
14.0%
12.0%
10.0%
8.0%
6.0%
4.0%
2.0%
0.0%
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
Year
Near or Above Threshold
Rising Fast
Middle
Lagging
• DEPSCoR-eligible states fell into four groups:
– 6 states Near or above threshold (AL, HI, LA, MS, NM, SC)
– 9 states Rising fast (AK, ID, KY, ME, MT, NE, NV, ND, SD)
– 6 states Middle (AR, DE, KS, OK, RI, TN)
– 4 states + 2 territories Lagging (NH, PR, VT, VI, WV, WY)
Slide 8
Average DEPSCoR Funding as a
Fraction of DOD Funding by State
6
NH
0
WA
54
MT
28
ND
33
ID
SD
0
0
CA
0
UT
2
RI
MI
0
0
IA
49
NE
36
NV
0
MA
0
NY
0
WI
12
41
WY
26
ME
31
VT
0
MN
0
OR
PA
0
IL
0
CO
38
0
OH
0
IN
0
0
DC
38
WV
6
KS
5
0
VA
37
KY
MO
NJ
DE
0
0
0
AZ
52
OK
4
NM
5
TN
0
MD
NC
38
AR
47
SC
35
MS
TX
0
CT
53
AL
0
GA
28
LA
0
6
GU
AK
0
FL
1
HI
0 DEPSCoRs
1-10 DEPSCoRs
11-20 DEPSCoRs
21-30 DEPSCoRs
31+ DEPSCoRs
18
PR
0
VI
Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions, the IDA DEPSCoR
database, and the DEPSCoR program database of funding totals by state and year
Note: Average calculated as total DEPSCoR funds during eligible years divided by total DOD funds during eligible years
•
DEPSCoR share of university S&E R&D funding varies substantially by state
•
More than 60% for VT, WY
•
“Rising fast” states’ (AK, ID, KY, ME, MT, NE, NV, ND, SD) DEPSCoR shares decline
2001-2005 compared with 1993-2000
Slide 9
Legislative Mandate for Study
1. Tangible results and progress toward the objectives of the
program
a. Applications used by, or supportive of, operational users
b. Expanded national research infrastructure
2. Activities consistent with statute
3. Assessment of program elements
4. Assessment of activities of state committees
5. Advantages and disadvantages of institution & state based
formulas
6. Mechanisms for improving the management and
implementation of the program
Slide 10
Expanded National Research
Infrastructure (1)
• Involving new investigators:
– Recent (2006-2008) cohorts had about 60% new PIs
– Most Army-funded DEPSCoR awardees (82%) had not
previously received funding from Army Research Office (ARO)
– 56% of PIs had been funded by the NSF either previous to or
within the same year of their first DEPSCoR award
• Training graduate students and postdoctoral fellows:
– ARO and Office of Naval Research (ONR) data suggest that
awards fund about 1 PhD, 1 Master’s degree, 2 postdocs
• Building physical infrastructure:
– DEPSCoR awards have supported purchase and maintenance
of equipment but data not collected systematically by services
Slide 11
Expanded National Research
Infrastructure (2)
• Leveraging new funding for defense-related research is
limited:
– 8% of non-DEPSCoR ARO awardees in DEPSCoR states
received a DEPSCoR award before (or in the same year as)
their first non-DEPSCoR ARO award
– 4 DEPSCoR awardees (less than 1%) won a DOD
Multidisciplinary University Research Initiative (MURI) award
after or in the same year as their first DEPSCoR award
» 2 of these investigators received non-DEPSCoR DOD funding
before their first DEPSCoR award
• Leveraging other funding:
– 63 DEPSCoR awardees (12%) received their first NSF funding
subsequent to their first DEPSCoR funding
Slide 12
Legislative Mandate for Study
1. Tangible results and progress toward the objectives of the program
a. Applications used by, or supportive of, operational users
b. Expand national research infrastructure
2. Activities consistent with statute
• Activities were found to be consistent with statute
3. Assessment of program elements
• Available data on DEPSCoR program activities and outcomes are
insufficient for monitoring and evaluation purposes
4. Assessment of activities of state committees
• State committees prioritized proposals that met state infrastructure
development goals and reflected the mission/research needs of DOD
• Committee processes varied widely from state to state and limited and
variable data prevented detailed assessment
5. Advantages and disadvantages of institution & state based formulas
6. Mechanisms for improving the management and implementation of
the program
Slide 13
Legislative Mandate for Study
1. Tangible results and progress toward the objectives of the
program
a. Applications used by, or supportive of, operational users
b. Expanded national research infrastructure
2. Activities consistent with statute
3. Assessment of program elements
4. Assessment of activities of state committees
5. Advantages and disadvantages of institution & state based
formulas
6. Mechanisms for improving the management and
implementation of the program
Slide 14
Approach Taken
• Assessment focused on comparing the current state-based
formula with an institution-based criterion based on a
maximum threshold for DOD S&E research funding
• As directed in the study legislative mandate, particular
emphasis was given to supporting defense missions and
expanding the nation's defense research infrastructure
– Since the legislative charge for the assessment does not
specify how “expanding the nation’s defense research
infrastructure” should be interpreted, the assessment
considered advantages and disadvantages using a variety of
possible interpretations
• Assessment also considered alternative criteria:
– Indicators of state-level S&T capacity (SEI, Milken Institute)
– Normalization by state population (Census)
• State-based and institution-based criteria can be combined
– PI-level criteria are also possible
Slide 15
Effect of Institution Based
Funding Threshold
Maximum DOD
Funding Threshold
All Universities With
Non-Zero DOD Research
Funding in 2005
Eligible for the
2008 Competition
Potentially Eligible
for the 2009
Competition
No limit
360
77
114
$10 million
316
75
109
$5 million
269
65
90
$3 million
231
55
76
$1 million
157
36
52
•
77 institutions in 2008 DEPSCoR-eligible states (360 total) received nonzero
research funding from DOD in 2005
– 38 Carnegie “Very High” or “High” research institutions in DEPSCoR states
•
$5 million threshold would make 269 institutions eligible
– Twelve of the 77 institutions in currently eligible jurisdictions (e.g., University of
Delaware, University of Nevada, Brown, Clemson, Vanderbilt, University of
Nebraska) would become ineligible
•
Considering only Carnegie “Very High” or “High” research universities, shift
would increase number of eligible universities from 38 to 121
Slide 16
Definition of “Expanding National
Research Infrastructure”
•
If interpreted as increasing equity in funding among states or achieving
state-level infrastructure goals, a state-based formula would be
advantageous
– Current state-based formula for eligibility harnesses the state EPSCoR
committees to coordinate infrastructure and capacity-building at the state level
– If intent is to increase the competitiveness of historically-underrepresented
states, eligibility can easily be determined at a state level
•
If interpreted as involving new investigators or institutions in defense-related
research, an institution-based formula would be advantageous
– Allows targeting of programmatic resources toward investigators at institutions
that have not historically built relationships with DOD
– Approach taken by late 1980s/early 1990s DOD Research Initiation Program
– While a state-based approach includes the flexibility to channel DEPSCoR
proposals toward historically underrepresented universities or new
investigators within an eligible state, the institution-based approach allows
greater flexibility to target underrepresented universities and investigators
throughout the entire country
•
Could not be determined whether state-based or institution-based approach
would elicit more qualified applications to support defense missions
– Larger number of eligible institutions implies more proposals, but quality
indeterminate
Slide 17
Effect of State Population
Normalization
’05 DOD S&E R&D Funding, by State
• Green states were
eligible in 2008, red
states are “graduates”,
blue states were never
eligible
$350,000
FY 2005 DoD University S&E R&D Funding (K$)
$300,000
$250,000
$200,000
$150,000
• Comparing top chart with
bottom shows dramatic
difference in order
$100,000
$50,000
M
D
C
A
P
A
T
X
M
A
N
Y
D
C
F
L
G
A
N
M
V
A
N
C
I
WL
A
M
S
A
L
O
H
H
I
M
I
N
J
U
T
IN
C
O
A
Z
T
N
M
N
LA
S
C
M
O
D
E
C
T
W
NI
V
O
R
R
I
IA
A
K
O
K
A
R
N
D
N
E
K
Y
M
T
ID
M
E
S
D
K
S
N
H
P
R
W
V
V
WT
Y
$0
’05 Per capita DOD S&E R&D Funding, by State
State
$70.00
• Several DEPSCoReligible states (e.g., AK,
DE, ND, RI, MT, SD)
above average in funding
per capita
$50.00
$40.00
$30.00
$20.00
$10.00
$0.00
M
D
H
NI
M
D
E
M
S
N
D
A
K
M
A
P
A
U
T
R
I
A
L
M
T
S
D
W
A
U
S C
A GA
ve
ra A
ge
V
A
T
X
M
E
N
C
N
V
ID
N
E
C
O
N
J
N
Y
F
L
M
OI
H
C
T
IL
IN
S
C
LA
IA
A
R
O
R
M
N
A
Z
T
N
O
K
M
O
W
I
K
Y
N
H
K
S
V
T
W
Y
P
R
W
V
FY 2005 DoD University S&E R&D Funding Per Capita
$60.00
• DEPSCoR graduates
among highest per capita
recipients
State
Slide 18
Other Capacity Indicators
90
Milken Index, 2008 Score
80
70
60
50
40
30
20
10
0
• Some DEPSCoR-eligible states (e.g., NH, RI, DE, KS) above average
in Milken Institute state S&T index
– Some “graduated” states below average
• Some DEPSCoR-eligible states (e.g., DE, KS, NE, NH, RI, VT) in the
top or second quartiles for more than half of Science and
Engineering Indicators’ seven R&D output measures
Slide 19
Legislative Mandate for Study
1. Tangible results and progress toward the objectives of the
program
a. Applications used by, or supportive of, operational users
b. Expanded national research infrastructure
2. Activities consistent with statute
3. Assessment of program elements
4. Assessment of activities of state committees
5. Advantages and disadvantages of institution & state based
formulas
6. Mechanisms for improving the management and
implementation of the program
Slide 20
IDA Recommendations
• DOD should change the current process for review of proposals to
focus more heavily on investigators’ future potential to conduct
research rather than on their current research capabilities
• DOD program managers should be formally encouraged to serve as
mentors and facilitators for DEPSCoR investigators seeking to
engage in further defense-related research
• DOD should create data systems that will allow systematic tracking
of DEPSCoR activities and outcomes
• Congress should re-examine and consider clarifying ambiguities in
the DEPSCoR legislative mandate
• Once the DEPSCoR objectives have been clarified, redesign the
program with a strategy for enhancing competitiveness at relevant
level (e.g., individual, institution, state)
Slide 21
Some DEPSCOR Legislative
Language Is Inconsistent
• Objectives specify that the research institution is the level at
which competitiveness is to be enhanced but authorizing
legislation also specifies that eligibility for DEPSCoR be
determined at the state level
– 2008 change to eligibility criteria leaves ambiguous whether
Congress intends DEPSCoR to target competitiveness at the
institution or state level
• Objectives specify that goal is to increase probability of longterm growth in the competitively awarded financial assistance
that institutions of higher education in eligible states receive
from the federal government but eligibility determined based
on DOD funding
– Statute does not discuss whether Congress intends DEPSCoR
should focus narrowly on [6.1] research that is relevant to DOD
missions and priorities although this is how program is run
Slide 22
Recommendations for
Legislative Clarification
• To ensure that the program is implemented in a manner that
meets current legislative priorities, Congress should clarify
whether
– The program is intended to increase competitiveness for
federal research funding in general or for particular types of
research funding
– The program is intended to fund primarily basic research,
primarily applied research, or a combination
– The primary unit at which competitiveness should be
enhanced is the institution, state, or other (e.g., individual
investigator)
• Eligibility criteria, funded activities, and other program
elements should be structured in accordance with the
program’s objectives
Slide 23
Competitiveness can be
Enhanced at Different Levels
• DEPSCoR supports individual or small-group research
projects and can therefore be understood to primarily target
capacity-building at the level of the individual
– Though supporting training and purchase of equipment target
capacity-building at level of the institution as well
• DEPSCoR also operates at state level (e.g., involvement of
EPSCoR committees, state-based eligibility criteria)
• While it might be argued that institutional competitiveness
depends on individual competitiveness and state
competitiveness depends on institutional competitiveness,
these dependencies are neither straightforward nor selfevident
Slide 24
Structure Program to Enhance at
the Desired Organizational Level
• Assessment identified other programs that might serve as
potential models for DEPSCoR
– State-level: NSF EPSCoR/Research Infrastructure
Improvement (RII) Awards
» One per state; integrated into state S&T plan; EPSCoR
committee involvement; capacity building rather than research
– Centers: NIH IDeA/Centers of Biomedical Research
Excellence (COBRE)
» Large-team research, equipment, mentoring; milestones for
transition to support by standard NIH mechanisms
– Investigator-level: DOD Research Initiation Program (RIP) or
NIH/IDeA Academic Research Enhancement Award (AREA)
Program
» Institution-based eligibility criteria; small research awards to
individual investigators; build relationships/expertise to allow for
transition to support by standard research mechanisms
Slide 25
Backup Slides
Slide 26
State Eligibility over Time
Fiscal Year
Jurisdiction
95
96
97
98
99
00
01
02
03
04
AL
AK
AR
DE
HI
ID
KS
KY
LA
ME
MS
MT
NE
NV
NH
NM
ND
OK
RI
SC
SD
TN
VT
WV
WY
PR
VI
05
06
07
08
Source: IDA analysis of
DEPSCoR BAAs and DOD
DEPSCoR press releases
Note: Red cells denote
years in which jurisdictions
were not eligible, and blue
cells denote years in which
states were eligible.
Note: Missouri, which was
eligible only in the 1993-4
competition, was not
included in the table.
Slide 27
No Obvious Correlation
$25
90
$20
70
Funding (M$)
$15
50
40
$10
Number of awards
60
30
20
$5
Share of DoD University Funding to DEPSCoR States, Adjusting for
DEPSCoR Programmatic Funding
20.0%
80
18.0%
16.0%
14.0%
12.0%
10.0%
8.0%
6.0%
4.0%
2.0%
0.0%
10
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
Year
$0
0
1993-4 1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
R&D (Actual)
R&D (Linear Model)
Research (Actual)
Research (Linear Model)
2008
Competition
Funding (M$)
Number of Awards
Number of DEPSCoR awards and program
funding
Linear regression model of DEPSCoR state
share of DOD university funding
Slide 28
DEPSCoR Emphasis is on
Basic Research
• Research oriented towards developing operationally useful
devices or components is not a legislatively-mandated
program goal
• DEPSCoR projects mostly fund basic research
– Not a statutory requirement
– Administered by basic research organizations within Air Force
and Army, plus Office of Naval Research (ONR)
» Interviews with DEPSCoR program officers suggest that more
applied research does not perform well in review
– DEPSCoR program officers tend to track transitions to 6.2 or
6.3 research, but not further
• Two transitions to operational use were found
Slide 29
Two Transitions to
Operational Use Were Found
• Ronald DeVore, University of South Carolina: Wavelet mathematics
for image compression for tactical applications
– DeVore and colleagues collaborate with program managers at Naval
Air Warfare Center NAWC at China Lake to deliver wavelet-based
image processing platform
– Charles Creussere of NAWC implements wavelet-based image
processing system for navigation in the Tomahawk Block II program
• Michael Pursley, Clemson University: wireless, mobile, distributed,
multimedia communication networks
– Pursley and colleagues working since 1970s with ITT on tactical radio
development
– Group used DEPScoR funding to support research that provided
better anti-jam communications and greater multiple-access capability
– Research led to the Soldier Level Integrated Communications
Environment (SLICE) wideband networking waveforms that have
been integrated into the SINCGARS radio
Slide 30
Descargar

Introduction