Getting Up to Speed on Value-Added An Accountability Perspective
Presentation by the Ohio Department of Education
1
Agenda
1) Value-Added and the Accountability System
2) 2007 – Overall Value-Added Results
3) Analysis of Individual Districts/Schools
4) AYP Growth Model
5) Data Availability
2
Value-Added and the Accountability System
3
Former Accountability System Architecture
Indicators Met
(District #s in
parentheses)
Change in
PI Score
AYP Status
Excellent
94% - 100%
(29 or 30)
o
r
100 to 120
a
n
d
Met/Not Met
Effective
75% - 93.9%
(23 to 28)
o
r
90 to 99.9
a
n
d
Met/Not Met
0% - 74.9%
(0 to 22)
o
r
0 to 89.9
a
n
d
Met
50% - 74.9%
(15 to 22)
o
r
80 to 89.9
a
n
d
Not Met
Academic
Watch
31% - 49.9%
(10 to 14)
o
r
70 to 79.9
a
n
d
Not Met
Academic
Emergency
0% - 30.9%
(9 or fewer)
a
n
d
0 to 69.9
a
n
d
Not Met
Continuous
Improvement
4
Performance
Index Score
o
r
o
r
PI Score
Improvement
Criteria Met
Change in Improvement Measure:
Implementing Value-Added
• Up to 2007, Performance Index Growth has
been used
• Starting in 2008, Value-Added data will be
used for districts and schools with tested
grades 4-8
5
Ohio’s Value-Added Measure
• Measures the contribution of a school or
district to the progress of its students on
test scores
• Requires a Scaled Score metric
– Current Alternate Assessments are not
measured on a Scaled Score
6
Ohio’s Value-Added Measure
• Scores are measured in “Normal Curve
Equivalent” gains
• “0” gain represents the typical or “expected” gain
• “Value-Added” is based on the 2006 – 07
distribution of scores
• Scores also use a measure of precision
(1 “Standard Error”) to help describe the
Value-Added classification
7
Ohio’s Value-Added Measure
There are three classification “bands”
using “gain score” and 1 Standard Error
• + Above expected growth (Green)
•  Met expected growth (“one year of growth in
one year of time”) (Yellow)
• - Below expected growth (Red)
8
Ohio’s Value-Added Measure
• Scores calculated for:
 Grades (4-8)
 Subjects (Reading and Math only)
 Grade and subject composites
 School composite
 District composite
• Only District and School composite
scores will be used for ratings
9
Gain Score, Standard Error and Classifications
Relative to Value-Added Standard
10
Exceed (+)
Exceed (+)
Meet ( )
0
Meet ( )
Not
Meet (-)
Not
Meet (-)
-10
Gain = 4
SE = 3
Gain = 4
SE = 5
Gain = 3
SE = 1
Gain = -4
SE = 5
Gain = -3
SE = 2
Gain = -5
SE = 1
Value-Added Will Affect Ratings
• 2007-08 first year VA can change
LRC designation
• Reward / sanction
– Reward enough growth
– Penalize insufficient growth
11
Impact on Designation
Above expected gain is rewarded
• Each rating category is rewarded based on
above expected gain, including an “Excellent
with Distinction” rating
• At least two years of above expected gain
12
Impact on Designation
Below expected gain results in lower ratings
• Each rating category is impacted by below
expected gain
• Rating is lowered if you have three years of
below expected gain
• Academic Emergency will not be lowered
13
New 2007-08 Accountability System Architecture
Including VA Impact
Performanc
e Index
Score
Indicators
Met
94% 100%
75% 93.9%
0% 74.9%
50% 74.9%
31% 49.9%
0% 30.9%
o
r
o
r
o
r
o
r
o
r
a
n
d
100 to 120
90 to 99.9
0 to 89.9
80 to 89.9
70 to 79.9
0 to 69.9
AYP
Status
a
n
d
a
n
d
a
n
d
a
n
d
a
n
d
a
n
d
Met or
Not
Met
Met or
Not
Met
Preliminary
Designatio
n
Excellent
IF YES
STOP
HERE
Effective
Met
Continuous
Improvemen
t
Not
Met
Not
Met
Not
Met
Did the
Preliminary
Designatio
n increase
or
decrease
based on
the AYP
Status?
No
additional
change to
the
designatio
n can
occur
based on
the value
added
calculation
IF NO
CONTINU
E
Academic
Watch
Academic
Emergency
Valueadded
MAY
affect a
designatio
n when it
has not
been
changed
by the
AYP
Status
Preliminary
Designatio
n
Excellent
Effective
Continuous
Improvement
Academic
Watch
Academic
Emergency
a
n
d
a
n
d
a
n
d
a
n
d
a
n
d
Amount of growth using
value-added calculation
Final
Designation
Above expected growth for at
least 2 consecutive years
Excellent with
Distinction
Below expected growth for at
least three consecutive years
Effective
Otherwise no effect on rating
Excellent
Above expected growth for at
least 2 consecutive years
Excellent
Below expected growth for at
least three consecutive years
Continuous
Improvement
Otherwise no effect on rating
Effective
Above expected growth for at
least 2 consecutive years
Effective
Below expected growth for at
least three consecutive years
Academic
Watch
Otherwise no effect on rating
Continuous
Improvement
Above expected growth for at
least 2 consecutive years
Continuous
Improvement
Below expected growth for at
least three consecutive years
Academic
Emergency
Otherwise no effect on rating
Academic
Watch
Above expected growth for at
least 2 consecutive years
Academic
Watch
Otherwise no effect on rating
Academic
Emergency
Impact on Designation - Examples
District A:
PI = 85 (Effective Range)
Met AYP
VA – Above Expected in 2007 and 2008
Final Rating ?
15
Impact on Designation - Examples
District A:
PI = 85 (Effective Range)
Met AYP
VA – Above Expected in 2007 and 2008
Final Rating - Excellent
16
Impact on Designation - Examples
District B:
PI = 77 (Academic Watch Range)
Met AYP
VA – Above Expected in 2007 and 2008
Final Rating - ?
17
Impact on Designation - Examples
District B:
PI = 77 (Academic Watch Range)
Met AYP
VA – Above Expected in 2007 and 2008
Final Rating - Continuous Improvement
(Why?)
18
Proposed Report Card Graphic
Grade
4
5
6
7
8
Subject
Composite
Reading
+
+
-
-


Mathematics
+
+

+

+
Grade
Composite
19
Total
Composite*
+
*Used in
LRC rating
+
+

+

2007 – Overall Value-Added Results
20
Value-Added 2007 Data
Scores based on four years of data:
• 2006-07 Grades 4-8 results (Reading, Math, Writing,
Science and Social Studies)
•
• 2005-06 Grades 3-7 results (Reading and Math in
addition to Grade 4 Writing)
• 2004-05 Grade 3 results (Reading and Math);
Grades 4-5 results (Reading only)
• 2003-04 Grade 3 results (Reading only)
21
Achievement Test Results:
NCE Means based on 2007 results
Math
Grade
22
3
4
5
6
7
8
45.8
48.6
2005
42.8
2006
45.4
51.0
49.0
46.2
46.9
49.7
2007
50.0
50.0
50.0
50.0
50.0
50.0
Relationship of Gains to Grade Level
Percent
All Districts - Math
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
4th Grade
23
5th Grade
6th Grade
7th Grade
8th Grade
Achievement Test Results:
NCE Means based on 2007 results
Reading
Grade
24
3
4
5
6
7
8
2004
50.0
2005
48.9
47.1
47.1
2006
49.0
46.5
50.8
56.0
52.1
50.6
2007
50.0
50.0
50.0
50.0
50.0
50.0
49.8
Relationship of Gains to Grade Level
Percent
All Districts - Reading
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
4th Grade
25
5th Grade
6th Grade
7th Grade
8th Grade
Percent (number) per category
Value-Added 2007 Composite Results
26
60%
(314)
(77)
(1288)
50%
40%
30%
(159)
(137)
(666)
(50)
(660)
(37)
20%
10%
0%
School Districts
Traditional School
Buildings
Community Schools
Relationship of LRC Designation to District
Typology
Percent of Districts
100%
80%
60%
40%
20%
0%
Rural Poor Other Rural
AW
27
*Note: Urban 21 are a subset of All Urban
category
Suburb
CI
Eff
All Urban
Exc
Urban 21
Relationship of Composite VA Gains to
District Typology
Percent of Districts
100%
80%
60%
40%
20%
0%
Rural Poor Other Rural
28
*Note: Urban 21 are a subset of All Urban
category
Suburb
All Urban
Urban 21
Relationship of VA Gains to Achievement
'07 VA Composite Level by '06 Designation - All Schools
100%
80%
60%
40%
20%
0%
AE
AW
CI
2006 LRC Designation
29
Eff
Exc
Relationship of VA Gains to Math Achievement
'07 VA Math Level by '06 Designation - All Schools
100%
80%
60%
40%
20%
0%
AE
AW
CI
2006 LRC Designation
30
Eff
Exc
Relationship of VA Gains to Reading Achievement
'07 VA Reading Level by '06 Designation - All Schools
100%
80%
60%
40%
20%
0%
AE
AW
CI
2006 LRC Designation
31
Eff
Exc
Performance (Status) by % Poverty
120
2007 PI Score
100
80
60
40
20
0
0
20
40
60
% Poverty
32
80
100
Performance (VA Gain) by % Poverty
2007 VA Gain Score
25
20
15
10
5
0
-5
-10
-15
-20
0
20
40
60
% Poverty
33
80
100
Performance (Status) by % Minority
120
2007 PI Score
100
80
60
40
20
0
0
20
40
60
% Minority
34
80
100
Performance (VA Gain) by % Minority
2007 VA Gain Score
25
20
15
10
5
0
-5
-10
-15
-20
0
20
40
60
% Minority
35
80
100
Analysis of Individual Districts/Schools
36
Value-Added Gains and Performance Index
High Performance – Low Value-Added
2007 Composite Gain Score
8
6
4
2
0
-2
District A
-4
-6
70
80
90
2007 Performance Index
37
100
110
Value-Added Gains and Performance Index
Average Performance – Low Value-Added
2007 Composite Gain Score
8
6
4
2
0
-2
-4
District B
-6
70
80
90
2007 Performance Index
38
100
110
2007 Composite Gain Score
Value-Added Gains and Performance Index
Very Low Performance and Value-Added
8
6
4
2
0
-2
-4
District C
-6
70
80
90
2007 Performance Index
39
100
110
Value-Added Gains and Performance Index Very Low Performance – “Green” VA
But is it enough?
2007 Composite Gain Score
8
6
4
District D
2
0
-2
-4
-6
70
80
90
2007 Performance Index
40
100
110
Value-Added Gains and Performance Index
“Green”- But will it be enough to improve?
2007 Composite Gain Score
8
6
4
2
0
District E
-2
-4
-6
70
80
90
2007 Performance Index
41
100
110
Value-Added Gains and Performance Index
Low Performance – High Value-Added
2007 Composite Gain Score
8
6
4
District F
2
0
-2
-4
-6
70
80
90
2007 Performance Index
42
100
110
Value-Added Gains and Performance Index
High Performance – High Value-Added
2007 Composite Gain Score
8
6
District G
4
2
0
-2
-4
-6
70
80
90
2007 Performance Index
43
100
110
Value-Added Gains and Performance Index
High Performance – High Value-Added
2007 Composite Gain Score
8
6
4
District H
2
0
-2
-4
-6
70
80
90
2007 Performance Index
44
100
110
Within District Variability
School level VA Gains vs. Performance
Value-Added Gain Score
10
E6
M2
5
M4
E5
E10
0
E9
E1
M1
E8
E4
E3
E7
E2
M3
Alt
ES
-5
MS
-10
A1
-15
50
60
70
80
Perform ance Index
45
90
100
AYP Growth Model
46
Changes in 2008
• New Goals
• Making minimum N uniform
• Include a Growth Model criterion
47
New Goals
Elementary
Middle
High
48
2006-07
Reading
2007-08
Reading
2006-07
Math
2007-08
Math
Grade 3
71.2%
77.0%
60.6%
68.5%
Grade 4
68.3%
74.6%
67.1%
73.7%
Grade 5
68.3%
74.6%
49.6%
59.7%
Grade 6
75.8%
80.6%
55.1%
64.1%
Grade 7
68.6%
74.9%
47.3%
57.8%
Grade 8
73.8%
79.0%
47.5%
58.0%
OGT (Grade 10)
71.8%
77.4%
60.0%
68.0%
Uniform Minimum N Size
• Minimum size for evaluation was:
• 30 for all groups except students with disabilities
• 45 for students with disabilities
• Minimum N size starting with 2007-08:
• 30 for all groups
• Minimum N size change for 2007-08 report card data
to meet federal requirement
49
Meeting Adequate Yearly Progress
Up to 2006-07, districts and schools could meet
AYP achievement in one of three ways:
1) By meeting or exceeding all AYP targets;
2) By meeting or exceeding AYP targets with a twoyear average of previous and current year’s
reported data;
3) Via the AYP safe harbor provision – district/school
achieves a 10% reduction in the percentage of
non-proficient students from the previous year
and also meets graduation or attendance rate
goal.
50
Growth Model and AYP
• 4th way of meeting AYP
• Individual student projections
• Projected out to first year of “next
school” (3 years maximum)
• Includes “next school” effects
• Only counts “Full Academic Year”
students in computation
51
Growth Model and AYP
AYP% for subgroup =
Numerator (count subgroup students only):
• Sum of 3rd graders proficient or better +
• Sum of “on track” students (those
projected to reach proficiency) +
• Sum of other proficient students with no
growth computation.
52
Growth Model and AYP
AYP% for subgroup =
Denominator:
• Total number of full academic year students
in combined tested grades (including 3rd
where applicable)
53
Meeting AYP – Status Only
Students Not Moving To Proficiency
Percent
of
Students
Proficient
Students On-Track to Proficiency
A
B
MEET AYP
MEET AYP
MISS AYP
MISS AYP
D
C
Magnitude of Individual Student Gains
54
Meeting AYP – Growth Only
Students Not Moving To Proficiency
Percent
of
Students
Proficient
Students On-Track to Proficiency
A
B
MISS AYP
MEET AYP
MISS AYP
MEET AYP
D
C
Magnitude of Individual Student Gains
55
Meeting AYP – Status and Growth
Students Not Moving To Proficiency
Percent
of
Students
Proficient
Students On-Track to Proficiency
A
B
MEET AYP
MEET AYP
MISS AYP
MEET AYP
D
C
Magnitude of Individual Student Gains
56
AYP Growth vs. Value-Added Gains
AYP Growth
_______________
Value-Added Gains
_______________
Level
Student
School / District
Use
Help meet AYP
LRC rating
Data Source
OAT
OAT
Measure
Projected gain
Composite mean gain
Purpose
Students “on-track” to
proficiency
Value schools add to
students’ starting points
57
Value-Added Data Availability
58
School and District Value-Added Data
• Publicly available
• ODE home page – Accountability Menu –
Value Added and click on “Value-Added Data
and Reports”
• Data Available through Power User Reports
(iLRC) in “Ratings” folder
• Data limited to Gain Scores, Standard Errors,
and “VA – N” - by subject and grades level
and composites
59
Value-Added Data Availability –
EVAAS Diagnostic Reports
• Hosted on SAS web site at:
https://ohiova.sas.com/evaas/login.jsp
• Login and password required –
District superintendent has
administrative authority
60
Value-Added Data Availability –
EVAAS Diagnostic Reports
• Reports
–
–
–
–
Are interactive
Contain one year of data
Provide student information by SSID
List gain scores only for students who have
contiguous grade scores in a subject
- Includes only full academic year students
(WKC)
61
In the Works: Value-Added Tools for
Teachers and Parents
• Explaining the Value-Added measure in
the Accountability System to educators
• Practical knowledge for teachers to talk to
parents about value-added
• Disk format – available before 2008-09
school year
62
Getting Up to Speed on Value-Added An Accountability Perspective
• Questions?
63
Descargar

No Slide Title