Three Colleges Collaborate
to Assess “Study Abroad”
– Influential Factors,
Experiences and Outcomes
Christine Brooks Cote, Bowdoin College
James C. Fergerson, Bates College
Mark Freeman, Colby College
North East Association for Institutional Research
Portsmouth, NH November 2004
Off-Campus Study Survey Participants
Bates
Bowdoin
Amherst
Colby
Smith
Project Development



Genuine interest in understanding the impact
upon students of a semester or a year spent
abroad
A need to evaluate the Colby-Bates-Bowdoin OffCampus Study Programs (CBB) in light of the
goals presented to the Mellon Foundation
Seized the opportunity to use the instrument in
a collaborative assessment project supported by
NEASC and funded by the Mellon Foundation
With Support From:
“Motivating Inquiry Regarding Teaching and Learning
at Critical Transition Points in Liberal Education”
“Motivating Inquiry Regarding Teaching and Learning
at Critical Transition Points in Liberal Education”
20 Liberal Arts Colleges Participated

First Year Seminar study
Senior/Capstone Experience study

Off-Campus Study survey


“Faculty Fellows” interviews of students about
focusing on an area of study
Institutions could opt in or out of each study
Comparing Study Away Programs
% of Class of 2004 that
studied away
Institutional Programs
CBB Consortium (expires in
FY2006)
Bates
Bowdoin
Colby
64%
51%
65%
"Fall Semesters
Abroad" (location
varies); May 4-4-1
"Short Term"
Ireland/UK and FY
CBB programs only
programs in
France/Spain
Quito
Cape Town
London
Outside Programs?
JYA/JSA, etc (2/3 of
study away students)
No
No
Who studies away?
ALL disciplines are
encouraged
About 1/3 are
language or area
studies majors
Heaviest user of
CBB
All areas; promotes
non-traditional areas
All areas
All areas
Not required
Not req'd for
study abroad (is a
graduation
req'mt)
Where?
Language Study?
Required in nonEnglish area
Survey Development




Based upon a survey designed at Bowdoin
Survey was reviewed and revised by the participating
institutions (IR offices, off-campus study offices, faculty,
deans, NEASC)
Flexible administration: Common “core” of questions
with some local questions; language was localized
Colby/Bates/Bowdoin also used the survey for a final
evaluation its CBB Consortium programs in London, Cape
Town, and Quito
Survey Administration





Bowdoin: Done in conjunction with senior exit interviews
Bates: Stand-alone Web survey
Colby: Web survey in conjunction with Senior Survey
HTML code shared, but
some questions and
wording were
“localized”; data
collected locally
Institutions sent data
to Colby, for
processing/clean-up;
SPSS file returned
School
Amherst
Response
Count
Rate
70%
102
% of
Sample
13%
Bates
66%
179
23%
Bowdoin
66%
131
16%
Colby
96%
281
35%
Smith
40%
102
13%
Bates: “Calendar Wars” & “Working the Web”
Bates Web Survey Responses
(Response rate: 67%)
30
May "Short Term"
# responses
25
Break
20
Winter Exam Week
15
10
5
0
May 17 - Remind 5
May 14
May 13
May 11
May 10 - Remind 4
May 6
May 5
May 4
May 3 - Remind 3
Apr 28
Apr 27
Apr 26 - Remind 2
Apr 25
Apr 22
Apr 20
Apr 17
Apr 16
Apr 15
Apr 14
Apr 13
Apr 12
Apr 11
Apr 10
Apr 9 - Remind 1
Apr 8
Apr 7
Apr 6
Apr 5 - Announce
Paper announcement with 5 e-mail reminders . Survey appears as “to do” link in
student’s personal academic affairs account until submitted. L.L. Bean gift
certificates raffled as “incentives”.
Survey Components





Students’ motivations for off-campus study
choices
Preparation and advising for study away
experience
Student experiences while away
Perceptions of the impact on academic
and personal growth
Senior academic activities that built upon
the study abroad experience
Major Survey Components
Students’ motivations for
off-campus study choices
Very strong influence
on choices



No influence
on choices
My desire to learn or become more fluent in the
language of this country or region of the world
My desire to study in a country or region of the
world that is very different from the one I have
lived in most of my life
My desire to have a lighter academic load than I
would normally have at [Bates or Bowdoin or
Colby]
Motivations for Study Away (All Institutions)
Strong/Very Strong Influence
Personal interest or attraction
Desire to be away from home institution for a period *
Desire to be more fluent in language of country
Program supported my major
Study in country that's very different
Take classes that applied toward my degree *
Program supported interests outside my major
Opinions of faculty or administrators **
Opinions or recommendations of your parents or peers *
Previous visit to this country
Desire to study without language barriers
Experiences of family or friends
Desire for a lighter academic load
Program cost
Study in country that's similar
* Not asked by Bowdoin
%
85.3%
63.0%
60.5%
55.5%
45.0%
42.0%
36.4%
31.8%
31.6%
27.8%
21.0%
19.0%
9.8%
8.5%
4.8%
** “Academic Advisors or Professors” at Bowdoin
Major Survey Components
Students’ experiences while
studying away
Strongly agree



Strongly disagree
While away, I was part of an intellectual community
that was invigorating and thought-provoking.
Most of my time outside of class was spent with
people who called the country or region of the
world their home.
While away, I nearly always spoke a language other
than English.
Major Survey Components
Students’ perception of impact of
off-campus study experience
Strongly agree




Strongly disagree
The experiences I had while away caused me to change
my understanding of how people around the world view
the United States.
My understanding of the culture, country or region of the
world I was in changed significantly by my living there for
a time.
The experiences I had while away taught me a great deal
about gender issues facing people in other parts of the
world.
Returning to [Bates or Bowdoin or Colby], I engaged in
research or independent study that directly built upon the
work I did while away.
Key Experiences/Outcomes Questions





Did students gain the skills that are desirable in a
liberal arts context?
How did outcomes vary by type of student, field of
study, or by institution?
Did study away change personal, academic, and
career goals?
Did students link study away experiences to other
academic work?
How do they compare with those who didn’t study
away?
Major Study Away Outcomes
Overview
Major Study Away Outcomes:
More than 50% agreed moderately/strongly (All Institutions)
Learned from the traveling I did
82.8%
Study away made me more mature and self-reliant
81.3%
My understanding of how people view the US changed
78.5%
Became well-immersed in the culture
73.6%
Understanding of culture changed by studying away
70.9%
Learned about political and economic issues
70.3%
I have more appreciation for cultures unlike my own
69.8%
Overall, learned more outside of class while away
69.6%
Learned a lot from my courses
63.0%
61.2%
Study abroad was single most rewarding part of education
I stay more informed about current world events
59.4%
51.6%
Learned about race and ethnicity
0%
10%
20%
30%
40%
50%
60%
70%
80%
90% 100%
Other Study Away Outcomes
Overview
Other Study Away Outcomes:
Less than 50% agreed moderately/strongly (All Institutions)
Proficiency in a language improved
48.0%
43.7%
Part of intellectual community
Took courses that built on what I did away
35.8%
Personal values and beliefs changed
34.0%
Learned about gender issues
33.9%
28.1%
Most of my time was spent with natives
Academic goals changed
24.7%
24.1%
Career goals changed
I always spoke a language other than Engilsh
23.0%
20.0%
Engaged in independent research that built on what I did away
I had a difficult time adjusting to the academic structure
9.3%
7.2%
Kept in touch with professors back home
0%
10%
20%
30%
40%
50%
Study Away Reasons - Cluster Analysis



Reasons for studying abroad items are the
best candidates for a priori or “causal”
variables.
If Reasons presuppose certain outcomes
then a model focusing on them has
diagnostic value in practice.
Cluster analysis into “types” to simply
interpretation.

A four cluster solution fit the five-school data
best
Selected Cluster Solution
Traditional Academic Focus?
SA in " similar" country?
Language learning important?
Listen to advice in SA choice?
SA is escape from [Colby]?
Previous visit to SA country?
Likely to be language major?
More than avg non-white?
More than avg. female?
Ex
pe
rie
nti
ali
sts
Na
Ac rrow
ad
em
ics
We
Int ll-Ro
ern und
ati
e
on d
ali
sts
Clu
bM
ed
s
Cluster
characteristics
Cluster names are subjective
interpretations of item response
patterns
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
No
No
Maybe
No
No
No
< Average
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
No
No
Yes
No
No
No
No
Does Cluster Membership Predict
OCS
Experiences?
Ay-uh…

Well-Roundeds and Experientialists strong on cultural
immersion, language learning experiences; weakest of
the four clusters on “adjusting to academic structure”

Narrow Academics stand out on “keeping in touch with
professors back home” and “course learning”, “part of
intellectual community”; weakest on cultural immersion
experiences

Club Meds weak pretty much everywhere, especially
traditional academics – but more cultural immersion than
Narrow Academics
Does Cluster Membership Predict
OCS Outcomes?
Ay-uh…

Well-Roundeds strongest on most outcomes, virtually
across the board – “academic integration” as well as
“personal growth” and “cross-cultural understanding”
outcomes


Narrow Academics weakest on “personal growth” and
“cross-cultural understanding” – almost as strong as WellRoundeds in “integration with post-OCS academics”
Club Meds bring up the rear virtually across the board,
especially on post-OCS academic integration. They do,
however, report that that the experience was “rewarding”
Comparing Students By College By Cluster
Clustering at Colby, Bates, and Bowdoin
35%
30%
25%
Bates
20%
Bowdoin
15%
Colby
10%
5%
0%
"Experientialists"
"Narrow
Academics"
"Well-Rounded
Internationalists"
"Club Meds"
Academic Division and Clusters - Bates
Academic Majors and Clusters - Bates
40%
35%
30%
25%
20%
15%
10%
5%
0%
"Experientialists"
Humanities
"Narrow
Academics"
Interdisciplinary
"Well-Rounded
Internationalists"
Natural Sciences
"Club Meds"
Social Sciences
Location of Study – By Cluster (Bates)
Study Abroad Location - Bates Students
Percentage Distribution by Cluster
60%
50%
40%
30%
20%
10%
0%
"Experientialists"
"Narrow
Academics"
Asia/Lat. America/Africa/Mult.
"Well-Rounded
Internationalists"
Europe
Australia/NZ/Oceania
"Club Meds"
UK/Ireland
Cumulative Average GPAs – By Cluster (Bates)
Cumulative GPA by Study Away Cluster vs. Class of 2004 - Bates Students
3.50
3.45
3.44
3.39
3.40
3.34
3.35
3.31
3.30
3.24
3.25
3.20
3.15
3.10
Experientialists
(N=53)
Club Meds (N=50) Narrow Academics Well-Rounded All in Class of 2004
(N=41)
Internationalists
(N=450)
(N=32)
Changes in Goals – Bates
Study Away Led to Significant Changes in My:
(% agreeing moderately/strongly -- Bates students)
60%
50%
40%
30%
20%
10%
0%
"Experientialists"
"Narrow
Academics"
Personal values and beliefs
"Well-Rounded
Internationalists"
Career goals
"Club Meds"
Academic Goals
Core Changes – By Division
Study Away Led to Significant Changes in:
(% agreeing moderately/strongly -- Bates students, by Area of
Major or Minor)
60%
50%
40%
30%
20%
10%
0%
Natural Science Social Science
Personal values and beliefs changed
Humanities
Interdisciplinary
Academic goals changed
Languages
Career goals changed
Linking OCS Survey Data With
Senior Survey Outcomes


Responses were
linked with HEDS
senior survey data
(Colby only) – can
leverage the OCS
survey results against
other institution-wide
outcomes
And can compare
outcomes with nonstudy abroad
graduates
Table 9. Gains in a global perspective on
diversity
% who “moderately” or “greatly” enhanced ability
while at Colby (Colby data only)
Enhanced ability to place current problems
in historical/ cultural/ philosophical
perspective during time at Colby?
Experientialists
81%
Narrow
Academics
80%
Well-Rounded
Internationalists
88%
70%
Club Meds
Did not study
abroad
50%
Regression Analysis: Are Other
Factors Linked To Outcomes?


So, “reasons for studying abroad” account for a
healthy portion of the variance in outcomes.
What of other factors?




Sex
Major
Country of study
Does school “uniquely” account for any variation
in outcomes?

If so, may help
Compared to
"Experientialists"
Compared
to W Europe
Compared
to Colby
Learned about political and economic issues
-.08 .11 -.06 .11 -.10 .13
.10 -.07 .11
.36
.09 -.03
My understanding of how people view the US changed
.10 -.11 -.15 -.05 -.14 .15 -.06 -.02 -.02 .06
.10
.11
Study abroad was single most rewarding part of education
.03
.08
.11
.00 -.10
Study away made me more mature and self-reliant
.18 -.06 -.11 -.01 -.09 .17 -.07 .02 -.07 .13
.03
.07 -.12
Personal values and beliefs changed
.04 -.02 -.03 .02
.02
.21
.06 -.04
Academic goals changed
.03
.00 -.06 .16 -.10 .05
.18
.09 -.03
Career goals changed
-.04 .04 -.01 .07
.09
.01
.06
.23 -.03 .05
.25
.10 -.07
Engaged in independent research that built on what I did away
-.04 .21 -.01 .20
.19 -.08 .12
.21 -.10 .02
.13
.14
Took courses that built on what I did away
-.03 .20
.15
Understanding of culture changed by studying away
.10 -.04 -.01 -.01 -.03 .09 -.10 -.04 -.11 .04
.20
.15
I have more appreciation for cultures unlike my own
.19 -.07 -.10 -.01 -.07 .11 -.07 .02 -.11 .03
.20
.12 -.13
I stay more informed about current world events
.11
.10
.03 -.09
Soc. Science Major
Nat. Science Major
.02
.10 -.05 .14
.03
.09 -.05 -.01 .07
.00
Language Major
Humanities Major
Female
Interdisc. Major
.03
.13
.00
.04
.08
.04 -.05 .10
.00 -.02 .35
.00
.01
.02 -.03 .09 -.27 .05
.06 -.04 .13 -.02 .13 -.09 .05 -.11 .13
.04
Bowdoin
.05 -.02 -.01 -.06 .04 -.03 .03
Bates
.06
Other Ctry
Learned about race and ethnicity
Eng-Speaking Ctry
.02 -.12 -.05 -.09 .01 -.07 .00 -.03 -.18 .29 -.02 -.01
Regression analysis
Club Meds
.17
Well-Rounded Inter.
Learned about gender issues
Field of Study
Narrow Academics
Table shows the standardized b coefficient from 13
linear regressions run with the outcome measure
(rows) as a dependent variable, and the dummycoded variables (columns) sex, field of study,
cluster, country of study, and school as independent
variables. Values that were statistically significant (p
< .06) are shaded.
Sex
.05
.03
.03
.08 -.01 -.03
.06
Survey Data Useful for CBB
Program Evaluation


CBB Off-Campus Study Programs were
funded by the Mellon Foundation
Two critical goals of the programs were to
generally enhance the academic quality of
study abroad experiences and improve the
integration of study abroad experiences
with the educational program at the home
college.
Survey Data Useful for CBB
Program Evaluation
Motivations
The academic opportunities within the
program or country support my major
CBB 61%
All Others 47%
My desire to have a lighter academic load
than I would normally have at [Bates or
Bowdoin or Colby]
CBB 5%
All Others 13%
Survey Data Useful for CBB
Program Evaluation
Experiences and Perception of Impact
While away, I was part of an intellectual
community that was invigorating and
thought-provoking.
CBB 55%
All Others 43%
I learned a great deal from the courses I
took while away.
CBB 81%
All Others 61%
Personal Growth / Reflection
“Study abroad gave me
the opportunity to have
my own thoughts; it
provided space from
academics that was filled
by reflection. That's very
important in college
because students are
confronted by so much
information that is difficult
to synthesize. With study
abroad, there is time and
space to reflect on Bates,
on the USA, on yourself.”
It is difficult to put in words. I
learned so much - and it is
still sinking in after a year. I
learned to interview powerful
political figures in a different
language. I learned to live and enjoy living in a home
without indoor plumbing. I
even learned a lot from being
mugged, getting parasites,
and getting sick - all of it
opened me up to a different
reality from life at Bates.
-- Anthropology major,
Ghana
-- Political Science major,
Madagascar
Connections to Research
“Broadly speaking, it was a lesson on life, what it means to
be human, and how to interact and communicate crossculturally. I worked so hard with subsistence campesino
potato farmers to produce our own food or kill our own
animals, and that has made me much more conscious of
the flow of food globally.
At Bates I took a Social Justice Spanish class, I was
rewarded an Otis Fellowship to return to Bolivia, and I
understood my thesis with greater depth. The mix of Bates
and Off Campus help me confront my own assumptions
and romanticisms about the life in Bolivia, Vermont, and so
on.
--Environmental Studies Major, Bolivia
Senior Thesis
Returning to Bates, I engaged in senior thesis or project
research that directly built on my study abroad experience
(39.2% agree "slightly" to "strongly")
60%
53.4%
50%
40%
31.3%
30%
20%
15.3%
10%
0%
Disagree
moderately/strongly
Slightly agree/disagree
Agree moderately/strongly
Senior Thesis / Research Links
“My senior thesis related
directly to my experience in
Tasmania. The first week of
my stay there was the most
influential in my thesis. I
returned to Tasmania the
following summer to do
research for my thesis, and
having already lived there,
made connections and gotten
to know the culture and area
was very helpful. I could hit
the ground running.”
-- Environmental Studies major
“My thesis was not about
Ecuador, but I extensively
worked with Spanish texts,
so that was important.
Although I did not like
Ecuador very much, the
experience provided me
with confidence to apply
for a Phillips Fellowship,
which allowed me to do
pre-thesis research in
Cuba.”
-- History major
Uses of Collaborative
Assessment – Advantages

Labor-saving: Divides up workload and tasks

Survey design involved faculty and other administrators

Improved analysis by sharing technical expertise and
discussing results

Promoted other data-sharing – Your First College Year
results shared

Importance of comparative data – why are we different,
and what does that mean?
Uses of Collaborative
Assessment -- Concerns


Different campus cultures and analytical approaches
Timing
 Campus calendars / office work schedules
 Assembling the right team(s)

Getting “Buy-in” to exchange data

FERPA concerns (exchanging non-survey outcomes data)

Integrating data from different collection methods
Policy Discussions and Faculty Reactions




“This is really interesting!”
At Bowdoin, the faculty Off-Campus Study
Committee has reviewed the entire report,
including qualitative data from one-on-one and
group interviews.
Data from the report will be used in our final
CBB report to the Mellon foundation.
Important assessment discussions would focus
on reasons behind college differences...yet to
happen.
Questions?




Contacts:
Christine Brooks Cote, [email protected]
Jim Fergerson, [email protected]
Mark Freeman, [email protected]
Descargar

NCAA Division III Financial Aid Report