Test-Driven Learning in
Early Programming Courses
David Janzen
California Polytechnic State University
San Luis Obispo, CA
Hossein Saiedian
University of Kansas
Lawrence, KS
SIGCSE’08
I Have a Dream
• That all students will some day…
– learn to design and test as they learn to program
• “... we have as many testers as we have
developers. And testers spend all their time
testing, and developers spend half their time
testing.”
Bill Gates in Information Week, May 2002
SIGCSE’08
How to Accomplish the Dream?
• Test-Driven Development
• TDD is a design (and testing) approach
involving short, rapid iterations of
Unit Test
Code
Refactor
Forces programmer to consider use of a method
before implementation of the method
SIGCSE’08
TDD in Action
class RectangleTest {
@Test
public void testArea() {
Rectangle r =
new Rectangle(5,8);
assertEquals(40, r.getArea());
}
} @Test
public void testPerim(){
Rectangle r =
new Rectangle(5,8);
assertEquals(26,r.getPerim());
}
}
class Rectangle {
private length, width;
public Rectangle(int l,int w) {
length = l; width = w;
}
public int getArea() {
return length * width;
}
} public int getPerim() {
return (length * 2) +
(width * 2);
}
}
SIGCSE’08
Predicting
Adoption Evolution
the Future
Objects
TDD
Research Labs
Dahl & Nygaard
NASA Project
Mercury
Industry
Smalltalk, C++
Industry Industry
XPAny SW Process
“beyond the visionary
Gradphase
& and into the
Advanced
earlyUG
mainstream”
IEEE Software 2007
Grad &
Advanced UG
First Year First
CS Year CS
SIGCSE’08
Questions We Want To Answer
1. Can we teach automated unit testing in
first year programming courses without
sacrificing other first year topics?
2. Is it better to teach test-first (TDD) or testlast programming to first year students?
3. What do students think about TDD?
SIGCSE’08
How Do We Teach TDD?
• Test-Driven Learning involves teaching
testing by example
XXXXXXXXXXXXX
System.out.println(“Result: ” + myObject.calculateResult());
assertEquals(25, myObject.calculateResult());
SIGCSE’08
Experiment Design
• Proposed TDD/TDL integration into CS1
and CS2 courses at University of Kansas in
2006 as part of PhD research
• We don’t always get what we ask for
– Given permission to teach two labs and assign
two projects
• Weeks 6 and 7 in a 16-week CS1 course
• Weeks 1 and 3 in a 16-week CS2 course
SIGCSE’08
Experiment Design
Project 1 solution
with tests provided
CS1
CS2
TF/TL
Instruction
& Survey
Project 1
CS1 Students
Assigned by ID
Survey
Project 2
Test-First
40
Test-Last
40
Test-Last
66
Test-First
66
Test-First
6
Test-Last
30
Project 1 solution
with tests provided
CS2 Students
Self-selected
SIGCSE’08
Experiment Context
• CS1 and CS2 both used C++, vi/emacs, and g++
Project Topics 1st Project
2nd Project
CS1
Arrays, functions, Classes, text I/O
text I/O
CS2
Classes, arrayClasses, pointerbased list, text I/O based list, text I/O
SIGCSE’08
Goal: Keep Unit Testing Simple
#include <cassert>
class Exams { . . . };
int main()
{
run_tests();
}
void run_tests()
{
{ //test 1 Minimum of empty list is 0
Exams exam1;
assert(exam1.getMin() == 0);
} //test 1
{ //test 2
Exams exam1;
exam1.addExam(90);
assert(exam1.getMin() == 90);
} //test 2
}
SIGCSE’08
CS1 Results: Grades & Effort
• Project grades were virtually the same
• Time spent on project was ~10% higher for
test-first programmers
Time (min) 1st Project 2nd Project
TF mean
285.29
343.48
TL mean
260.20
308.40
% diff
10%
11%
p-value
.60
.47
SIGCSE’08
CS1 Results: Unit Tests
• Test-first programmers wrote more tests on
the first project, but fewer on second
• Remember that students switched approaches
# asserts
1st Project 2nd Project
TF mean
5.85
1.89
TL mean
3.85
3.10
% diff
52%
-39%
p-value
.11
.11
SIGCSE’08
CS1 Results: Confidence
• No difference in confidence on first project
• On second project, test-first programmers were
more confident of their solutions (P1 tests)
2nd Project
Quality
Changes
Reuse
TF mean
3.98
3.90
3.69
TL mean
3.25
3.06
2.88
% diff
22%
27%
28%
p-value
.05
.02
.02
SIGCSE’08
CS1 Results: Student Opinions
CS1 Programmer Opinions
Characteristic
Choice
BestApproach
ThoroughTesting
Correctness
Simpler
FewerDefects
0%
20%
40%
60%
80%
100%
% Choosing
Test-First
Test-Last
SIGCSE’08
CS2 Results: Grades
• Project grades were higher for test-first
• No significant difference in previous
experience or gpa
Score
1st Project 2nd Project
TF mean
88.83
90.17
TL mean
79.47
72.83
% diff
12%
24%
p-value
.09
.04
SIGCSE’08
CS2 Results: Effort
• Time spent on project was lower for testfirst programmers
Score
1st Project 2nd Project
TF mean
12.42
16.00
TL mean
14.70
18.06
% diff
-16%
-11%
p-value
.30
.42
SIGCSE’08
CS2 Results: Unit Tests
• Test-first programmers wrote more tests
• Not significant, high standard deviation
• Students not graded on tests
# asserts
1st Project 2nd Project
TF mean
53.67
47.67
TL mean
13.43
25.03
% diff
300%
90%
p-value
.08
.39
SIGCSE’08
CS2 Results: Confidence
• No significant difference in confidence on
either project
Both Projects Quality
Changes
Reuse
TF mean
3.83
4.00
3.67
TL mean
3.21
3.89
3.71
% diff
16%
3%
-1%
p-value
.14
.87
.74
SIGCSE’08
CS2 Results: Student Opinions
CS2 Programmer Opinions
Characteristic
Choice
BestApproach
ThoroughTesting
Correctness
Simpler
FewerDefects
0%
20%
40%
60%
80%
100%
% Choosing
Test-First
Test-Last
SIGCSE’08
Reluctance to adopt test-first
despite perceived benefits
11% vs 63% would
choose test-first
SIGCSE’08
Validity Concerns
• Study too small to draw any conclusions
• Lack of randomization in CS2
• Confounding factors:
–
–
–
–
New concepts introduced with each project
Solutions with tests provided after 1st project
Simple assert-based unit tests, no framework
C++
• However, many of the results line up with results
from industry and more advanced courses
SIGCSE’08
Conclusions
• Unit testing can be integrated into CS1/2 courses
without taking time away from other topics
• Test-first programmers tend to write more tests
• Test-first programmers may do as well (CS1) or
better (CS2) on projects
• Test-first programmers may spend more time (CS1)
or perhaps less time (CS2) on projects
• Early programmers are reluctant to use test-first,
even after positive experiences with it
SIGCSE’08
Future Work
• Currently conducting a study in CS1.5
– Java and JUnit
– TDD from beginning
– TDD throughout 10-week quarter
• Early Results
– TDD takes no extra instruction time
– Existing course materials can be rewritten w/TDD
– TDD could influence course refactoring
• e.g. delay coverage of I/O
SIGCSE’08
Questions
• Resources at http://www.simexusa.com/tdl/
• If you are an early adopter of TDD in CS1/2,
I’d like to know, email me at
djanzen (at) calpoly.edu
• Acknowledgements
– SIGCSE Special Projects Grant – seed funding
– Lockheed Martin – current (future work) study
SIGCSE’08
Descargar

A Leveled Examination of Test