STEM-Year-Three-Evaluation-Final-Report

Given the suspension of funding after the first program year, STEM 2 was not able to track more specific student indicators for those students participating in service-learning courses s

Service Learning Transforming Educational Models in Science, Technology, Engineering,

and Mathematics (STEM) 2 Year Three Evaluation Final Report

California State University

Submitted by Cobblestone Applied Research & Evaluation, Inc

Rebecca M Eddy, Ph.D., Erika Randall, M.Ed., & Mariana Schmalstig, M.A

September 29, 2014

Table of Contents

Executive Summary 4

Introduction to Service Learning Transforming Educational Models (STEM) 2 8

Background and Context of Service Learning 8

(STEM) 2 Program Purpose 9

Program Evaluation Methods 11

Key Evaluation Questions 11

Section 1: Academic Program Development 12

STEM Service-Learning Self-Assessment Rubric 12

STEM Service-Learning Self-Assessment Results 14

Summary of Service-Learning Capacity in STEM Departments at the CSU 19

Innovative Practices in STEM Departments 19

Campus 1: Chico 20

Campus 2: Fresno 20

Campus 3: Monterey Bay 21

Campus 4: San Marcos 21

Campus 5: Sonoma 21

Summary of Innovative Practices Updates 22

Laying the Foundations Subgrants Summary 22

Engaged Department Initiative 23

Engaged Department Institute 23

Additional Funding Opportunities that Support Service Learning in STEM at the CSU 32

Energizing Career Opportunities through Local Education and Development (ECO LED) 32

CSU’s Service Learning and Energy Efficiency and Education Grant Program 35

Section 2: Student Development 40

Replicating Successful Service-Learning Initiatives: STEM C3 40

Creation of the STEM C3 Manual 41

STEM C3 Presentations 41

Follow-up to STEM C3 Presentations: One Year Later 42

Follow-up to STEM C3 Presentations: Two Years Later 43

Summary of STEM C3 Follow Up 45

STEM C3 Awards and Impact 45

Section 3: Statewide Partnership Development 46

Strengthen Current Partnerships 46

California Campus Compact 46

Society for Science and the Public 47

Project Kaleidoscope (PKAL) 47

Southern California Edison (SCE) 47

Develop New Partnerships 48

Southern California Gas 48

SESYNC 48

Helmsley Charitable Trust 49

AmeriCorps VISTA 49

Summary of Partnership Development 50

Program Dissemination 50

Summary of Program Dissemination 51

Section 4: (STEM) Program Results and Sustainability 52

Results of the (STEM) 2 Program 52

Lessons Learned and Sustainability of the (STEM) 2 Program at the CSU 54

Conclusions 56

References 57

Tables Table 1 Components of Service-Learning Capacity Dimensions 13

Table 2 Laying the Foundation Subgrants Participants and Activities 23

Table 3 Engaged Department Institute Follow-Up Survey Respondents 27

Table 4 Mean Participant Ratings for Engaged Department Institute: Comparison of Pretest, Posttest, and Follow-up Surveys 28

Table 5 Engaged Department Institute Participant Mean Ratings, Service Learning Benefits 29

Table 6 Engaged Department Institute Efficacy Mean Ratings 30

Table 7 Summary of ECO LED Program Activities 33

Table 8 Challenges and Possible Solutions for the ECO LED Program 35

Table 9 Summary of Additional Grant Activities 37

Table 10 Offices Involved in Planning STEM Student Success Programs 43

Table 11 Most Viewed Pages on Chancellor’s Office Center for Community Engagement Website 51

Table 12 Program Indicators, Goals, and Status After Years 1 and 3 52

Table 13 Overall and STEM Service Learning Courses 2008-09 through 2012-13 54

Figures Figure 1 Pretest Self-Assessment Ratings for All CSU Campuses 14

Figure 2 Posttest Self-Assessment Ratings for All CSU Campuses 15

Figure 3 Average Pretest and Posttest Scores by Component 15

Figure 4 Pretest and Posttest Self-Assessment Ratings for All CSU Campuses 18

Figure 5 Continuums of Service Conference Presentation Ratings 42

Executive Summary

In 2010, the CSU consortium was awarded $1.5 million in funding (over 3 years) from Learn and Serve America: Corporation for National and Community Service to support efforts to promote service learning and community engagement in Science, Technology, Engineering, and

Mathematics (STEM) fields throughout the California State University (CSU) The Learning Transforming Educational Models (STEM) 2 program was delivered to the CSU

Service-campuses and supported by the CSU Chancellor’s Office from September 2010 through August

2011 Although funding did not continue beyond the initial $500,000 grant in the first year, the CSU campuses continued STEM service-learning efforts with support from the Chancellor’s Office This report serves as a follow-up study from the first year of funding intended to

summarize the impact of (STEM) 2 in which we attempt to identify how the elements of the

original (STEM) 2 program produced sustainable change on CSU campuses

Engaged Department Initiative

Engaged Department Institute

(EDI)

EDI Grants

The EDI was offered in June 2011 and was specific to science departments It included five teams from four campuses The three-day long conference brought together teams of faculty, staff, and community partners

to develop strategies to support service-learning and community engagement activities in their departments Participating teams received $5,000 subgrants to carry out the action plans they developed at the Institute during the 2011-2012 academic year

Laying the Foundation Subgrants

Fifteen of the twenty-three CSU campuses received funding for service-learning offices to assess, showcase and support current service-learning activities in their STEM departments

Student

Development

STEM C3 (Careers,

Community and Connections)

Program Directors from CoyoteCareers in San

Bernardino created a manual and presented at multiple events that provided information on how to develop and sustain a successful campus program for STEM students which includes tutoring, career preparation, service- learning internships, and alumni networking

Partnership

Development

Statewide Partnership Development

Partnerships were established or strengthened with key organizations including: California Campus Compact, Cal STEM, California Volunteers, Society for Science and the Public, Project Kaleidoscope, the investor- owned utilities, foundations and the AmeriCorps VISTA program

The overarching goals of (STEM) 2 were to institutionalize service learning in STEM departments and promote college student retention and achievement in STEM disciplines Three areas were targeted to support the grant goals: academic program development, student

development, and partnership development throughout the CSU campuses and communities The grant supported these three areas with several activities throughout the CSU system

Cobblestone Applied Research & Evaluation, Inc was again hired to evaluate the

effectiveness of the (STEM) 2 initiatives and to conduct a follow-up study of program activities that have occurred since the first year of the program We wanted to determine if any changes made from the original LSA funding were sustained over time and investigate the current state of service learning in STEM at the 23 CSU campuses The evaluation questions and corresponding results are summarized below

How has service-learning

capacity in STEM

departments on CSU

campuses changed over

time? What is the current

state of this capacity?

According to the pretest self-assessment rubric, STEM departments on

4 of the 23 campuses were at the “sustained institutionalization” stage

of service learning During the posttest, this doubled to eight campuses

at the “sustained institutionalization” stage Most campuses improved from pretest to posttest with thirteen campuses moving onto the next stage On average, improvement was made for each component of the rubric, although differences between the dimensions were found among the campuses

To what extent was

initiatives related to the

capacity for service

learning in STEM on CSU

campuses?

In general, campuses that were engaged in more program components tended to have larger increases from their pretest to posttests scores On average, the six campuses who participated in three to five activities

had an increase in FURCO score by 10.6 points Of the campuses (n =

9) who were involved in one or fewer activities, only 7 points of improvement were made

What was the impact of the

(STEM) 2 activities such as

Innovations subgrants,

Laying the Foundation

subgrants, and Engaged

Department Initiative in

promoting Academic

Program Development?

Innovations subgrants: The subgrants clearly provided CSU campuses

with the ability to include service learning in a way that was tailored to the specific needs of the institution and surrounding community Some

of the work that was started from these projects has continued over time, while other work has been expanded or suspended Sustainability

of such projects requires ongoing effort and resources and many projects have been supported by faculty members at each campus

Laying the Foundation subgrants: These subgrants allowed all fifteen

campuses to gain a better understanding of the state of service learning

in their STEM departments, deliver planned activities and create plans for future growth Despite lower than expected faculty support, the subgrants provided the necessary focus on STEM service-learning courses to facilitate the long-term growth of STEM service-learning courses system wide

Engaged Department Institute: Results indicated that respondents

benefitted from their experiences at the EDI Comparisons between the original EDI survey and the follow-up survey revealed that attitudes towards service learning and the EDI have remained favorable, despite the challenges that some groups have faced in implementing the vision that they created at the EDI

What additional funding

opportunities have

facilitated service learning

in STEM in the CSU?

The Energizing Career Opportunities through Local Education and

Development and Service-Learning and Energy Efficiency and Education grant programs allowed for some campuses to continue

improving service-learning efforts Six individual campuses implemented a wide array of activities These accomplishments include the assessment and development of service-learning courses, the implementation of new curriculum, and the facilitation of learning communities

Additionally, (STEM) 2 created the foundation upon which a newly launching CSU STEM VISTA was built In AY 2014-2015, fifteen AmeriCorps VISTA members will be placed in STEM departments and colleges to build capacity for STEM student success through the

development of high-impact practices like service learning

What was the impact of the

(STEM) 2 activities in

promoting Student

Development? Have CSU

campuses implemented

long-term plans after

attending the STEM C3

symposia?

All of the seven participating campuses felt that the printed manual they received at the symposium was helpful A year after attending the symposium, five of the campuses confirmed that they plan to establish

a new STEM student success program using the tools learned from the

symposium While the momentum generated from STEM C3 inspired

several campuses to create plans for STEM student success programs, two years later this enthusiasm has largely waned While many individual faculty members have devoted time in planning and grant- writing for these initiatives, it is clear that grant funding is required to

support intensive efforts on campus

How has Statewide

How have results of

(STEM) 2 initiatives been

disseminated in the CSU?

To increase visibility of program activities and results, the “Center for Community Engagement” (CCE) and “STEM Engaged Learning” websites have been updated with recent and relevant information This has promoted dissemination of activities and service-learning resources Additionally, the 2011-2012 annual publication distributed by CCE (in print and electronically) highlighted STEM service-learning courses and programs at CSU campuses

Did the activities

program produce intended

program results?

A total of 529 STEM SL courses are offered at the CSU—an increase

of 245 new courses since 2008-09 The percent of service-learning students in the STEM fields has increased from 9.6% to 14.9% from 2008-09 to 2012-13 and 17.7% of all service-learning courses are offered in the STEM fields (from 10.1% in 2008-09)

What were the lessons

program?

While initial enthusiasm to integrate service learning into STEM curriculum may be strong, external funding is required to sustain such efforts Similarly, dedicated faculty and a commitment to inquiry-based learning are also required to maintain momentum for such initiatives

Innovations subgrants were a useful process for creating new

service-learning courses at multiple campuses The longer-term

impacts have had mixed results with some efforts expanding while others have been suspended

The STEM C3 presentations garnered a lot of interest from attendees;

however, longer-term implementation of similar programs has not been fully integrated at other CSU campuses

What were the most

sustainable aspects of the

(STEM) 2 program?

Service-learning course experiences have positive impacts on students, faculty and the community but adequate time and preparation must be allowed to maximize the student learning experience and community benefit

STEM service-learning courses that were developed as part of

(STEM) 2 continue to be offered at CSU campuses and others have also been developed recently The continued increase in the number

of STEM service-learning courses is evidence of this sustainability New initiatives such as VISTA program and the partnerships with Business and Finance around Campus as a Living Lab effort will continue to build capacity for long-term sustainability

Service learning in STEM has expanded throughout the CSU over the past three years, in part from the LSA funding and subsequently other funding partnerships The increase in service-learning courses and numbers of students served, particularly in STEM, can be attributed to this effort Despite the lack of funding beyond the first year, the CSU Chancellor’s Office, Center for Community Engagement (COCCE) has successfully leveraged new and existing partnerships to continue integration of service learning in STEM throughout the state Other evidence that service learning in STEM has broadened throughout the CSU was captured by the self-

assessment instruments conducted as pretests and posttests Most CSU campuses increased assessment ratings in the integration of service learning in STEM, and a good number have reached the sustained institutionalization stage These data are encouraging both in terms of evidence that the COCCE efforts have been successful over the past few years, as well as the idea that more students, faculty and communities are expected to benefit from this increased integration

self-Introduction to Service Learning Transforming Educational Models (STEM) 2

The California State University (CSU) system is the largest system of higher education in the United States and recognizes the responsibility to provide its students with knowledge and opportunity to fully participate in civil society For more than a decade, the CSU system has been committed to increasing opportunities for student community engagement, and has been

expanding partnerships to do so Learn and Serve America (LSA) has been a partner to the CSU

at various points since 2000 In 2010, the CSU consortium was awarded $1.5 million in funding (over 3 years) from Learn and Serve America: Corporation for National and Community Service

to support efforts to promote service learning and community engagement in the Science

Technology, Engineering, and Mathematics (STEM) fields throughout the 23 campuses of the

CSU The Service Learning Transforming Educational Models (STEM) 2 program was delivered

to the CSU campuses and supported by the CSU Chancellor’s Office from September 2010 to August 2011 The focus of this initiative was threefold: academic program development, student development, and statewide partnership development

Unfortunately, funding for the Learn and Serve program overall was eliminated by

Congress after the first year of the (STEM) 2 program and therefore the ability to implement the three-year program was thwarted Despite this challenge, the CSU Chancellor’s Office Center for Community Engagement (COCCE) has been able to continue with its STEM-focused support from a combination of other funding sources and the Chancellor’s Office directly This report serves as a follow-up study from the first year of funding intended to summarize the impact of

(STEM) 2 This study attempts to capture how the elements of the original (STEM) 2 produced sustainable change on CSU campuses and provides some evidence of what might be possible if similar initiatives are supported in the CSU in the future

Background and Context of Service Learning

Service learning is an educational methodology that combines community service with classroom learning to engage students in the educational process, teach civic responsibility, and

strengthen communities (Bringle & Hatcher, 1995) Service learning helps students gain socially responsive knowledge, that is, education obtained through direct academic-based problem

solving of social issues (Altman, 1996) Students participating in service learning apply the skills and knowledge they learn in their academic coursework to identify and solve real-world

community problems and also access the expertise of community partners in addressing these

problems Additionally, students become contributing citizens and active community members through the service they perform

Research indicates that service learning has an overall positive impact on students’ social, personal, and cognitive outcomes (Giles & Eyler, 1994) Students who participated in service-learning education had greater gains in perspective-taking, complex problem solving and critical thinking than students who did not participate in service-learning education (Batchelder & Root, 1994; Markus, Howard, & King, 1993) Additionally, service learning increases students’

awareness of contemporary social issues (Driscoll, Holland, Gelmon & Kerrigan, 1996),

enhances students’ self-efficacy relating to community service (Reeb, Katsuyama, Sammon, & Yoder, 1998), and increases positive attitudes about civic engagement and social responsibility (Markus et al., 1993) Although this form of community-based education has been linked to positive student outcomes, it also creates new challenges for the institution Specifically, this type of undertaking changes the function of the institution, creates new demands for faculty roles, and affects not only what students learn but also what is taught

Through the (STEM) 2 program, students and faculty at a number of CSU campuses participated in one of several service-learning courses focused specifically on STEM These courses covered a wide variety of disciplines and strategies Given the suspension of funding

after the first program year, (STEM) 2 was not able to track more specific student indicators for those students participating in service-learning courses such as retention, graduation, self-

efficacy relating to community service or other key indicators proposed However, overall rates

of STEM service-learning courses and other institution-level data have been tracked In addition,

the COCEE has captured some of the key activities related to (STEM) 2 initiatives and related programs that have occurred at CSU campuses during the past three years From this

information, we can make some inferences about the relationship between these activities

facilitated by (STEM) 2 and subsequent impacts on students and faculty Although these

conclusions are far from definitive, they suggest how LSA funding has provided the CSU with both direct and indirect impacts even after suspension of initial funding The following report details these activities and findings

(STEM) 2 Program Purpose

(STEM) 2 was designed to integrate service learning as part of the undergraduate

educational experience for students in the STEM fields Specifically, program goals were to 1)

institutionalize service learning in STEM disciplines, and 2) promote college student retention and achievement in STEM disciplines These goals were addressed through academic program development, student development, and statewide partnership development A website was established to provide information about the initiative, and can be found at:

http://www.calstate.edu/cce/stem/

(STEM) 2 included a variety of program activities throughout the CSU campuses in the first program year These activities were intended to simultaneously expand service-learning experiences for CSU students while advancing their retention, graduation and career placement

in the STEM disciplines Activities included:

Laying the Foundation subgrants were awarded to fifteen service-learning offices on

CSU campuses to assess current STEM service-learning activities on their campuses and promote service learning in STEM departments

Innovation subgrants were awarded to five CSU campuses (Chico, Fresno, Monterey

Bay, San Marcos, and Sonoma) to provide funding to carry out innovative projects and serve as demonstration sites from which to grow STEM service-learning initiatives throughout the CSU

An Engaged Department Institute (EDI) was held with five teams from multiple CSU

campuses (Bakersfield, Fresno, Los Angeles, and Monterey Bay) to provide training, support, and resources to strategically embed service learning and community

engagement into academic programs These five teams then received follow-up

subgrants to implement the plans they developed during the Institute

A how-to manual for STEM student success was written by a team at CSU San

Bernardino that provided specific information on developing a program modeled after

their award-winning CoyoteCareers program, which includes facilitation of career soft

skills, service-learning internships, tutoring support, and integration of alumni

Efforts to promote partnerships and participate in service-learning activities with key organizations such as Project Kaleidoscope and the Society for Science and the Public were also developed

Program Evaluation Methods

Cobblestone Applied Research & Evaluation, Inc was again hired to evaluate the

effectiveness of the (STEM) 2 initiatives and to conduct a follow-up study of program activities that have occurred since the first year of the program We wanted to determine if any changes made from the original LSA funding were sustained over time and investigate the current state of service learning in STEM at the 23 CSU campuses The evaluation relied on a variety of data collection activities including: administering a follow-up assessment of STEM departments’ capacity for service learning on all 23 CSU campuses; administering a follow-up survey to those campus teams that attended the Engaged Department Institute to determine the sustainability of participant growth on key dimensions addressed during the institute; administering other follow-

up inquiries for those campus representatives that attended the STEM C3 symposium or received Innovations subgrant funding to determine any sustainable activities that were in place since the initial intervention; and archival analysis of service-learning courses and student counts on each

of the CSU campuses

Key Evaluation Questions

The purpose of the evaluation was to answer key evaluation questions to understand the extent to which any sustainable results have occurred at the CSU in STEM service learning during the past two years Key evaluation questions included:

1) How has service-learning capacity in STEM departments on CSU campuses changed over time? What is the current state of this capacity?

2) To what extent was participation in (STEM) 2 initiatives related to the capacity for service learning in STEM on CSU campuses?

3) What was the impact of the (STEM) 2 activities such as Innovations subgrants and

Engaged Department Initiatives in promoting Academic Program Development?

4) What additional funding opportunities have facilitated service learning in STEM in the CSU?

5) What was the impact of the (STEM) 2 activities in promoting Student Development? 6) Have CSU campuses implemented long-term plans after attending STEM C3 symposia? 7) How has Statewide Partnership Development from (STEM) 2 continued or expanded?

8) How have results of (STEM) 2 initiatives been disseminated in the CSU?

9) Did the activities conducted in the (STEM) 2 program produce intended program results?

10) What were the lessons learned from the (STEM) 2 program?

11) What were the most sustainable aspects of the (STEM) 2 program?

How has service-learning capacity in STEM departments on CSU campuses changed

over time? What is the current state of this capacity?

Section 1: Academic Program Development

Program activities were focused on three major areas: academic program development, student development, and statewide partnership development The purpose of the first area, academic program development, was to facilitate service-learning initiatives in STEM

departments on all CSU campuses The rationale for supporting academic program development was to allow high-impact practices like service learning to become a core part of STEM

curriculum, as opposed to an ancillary activity Only by integrating service learning into core programs would this allow for long-term sustainability of service learning in the CSU and have the intended outcomes on students, faculty and communities During the first year of the

program, an assessment was conducted to understand the current state of service learning in STEM departments Recently, this assessment was completed again for each of the CSU

campuses as a posttest measurement, which usually included participation by many members of the original teams (e.g., service-learning director and STEM faculty) The following section includes a description of these assessments and a comparison of the pretest and posttest results for each of the CSU campuses

STEM Service-Learning Self-Assessment Rubric

The rubric administered was adapted from the work of Andrew Furco, University of California, Berkeley, 1999 (revised in 2002, 2003, 2006) and based on the Kecskes/Muyllaert

Continuums of Service Benchmark Worksheet Each campus self-rated twenty-two aspects of

service-learning capacity divided into five dimensions of their service-learning program

specifically related to STEM departments as a pretest (Fall 2010) and as a posttest (Fall 2013),

see Table 1 Dimension 1 measures the development of a definition, philosophy, and mission of

service learning on the campus Dimension 2 measures STEM faculty support of, and

involvement in, service learning Dimension 3 measures STEM student support for, and

involvement in, service learning Dimension 4 measures community participation and

partnerships, and finally, Dimension 5 measures departmental support for service learning

Table 1 Components of Service-Learning Capacity Dimensions

Dimension 1: Philosophy and

Mission of Service Learning

1.1 Definition of Service Learning in STEM Departments 1.2 Strategic Planning

1.3 Alignment with Departmental Mission 1.4 Alignment with Educational Reform Efforts Dimension 2: STEM Faculty

Support for and Involvement in

Service Learning

2.1 STEM Faculty Knowledge and Awareness 2.2 STEM Faculty Involvement and Support 2.3 STEM Faculty Leadership

2.4 STEM Faculty Incentives and Rewards Dimension 3: STEM Student

Support for and Involvement in

Service Learning

3.1 STEM Student Awareness 3.2 STEM Student Opportunities 3.3 STEM Student Leadership 3.4 STEM Student Incentives and Rewards Dimension 4: Community

Participation and Partnerships

4.1 Community Partner Awareness 4.2 Mutual Understanding

4.3 Community Partner Voice and Leadership

Dimension 5: Departmental

Support for Service Learning

5.1 Coordinating Entity 5.2 Policy-Making Entity 5.3 Staffing

5.4 Funding 5.5 Administrative Support 5.6 Engaged STEM Department Status 5.7 Evaluation and Assessment

For each dimension component, respondents rated their current status on a scale of 1 to 9, with 1 representing the lowest possible rating and 9 representing the highest possible rating The numerical ratings were accompanied by a statement to be used by the respondent as a guide for their rating

A rating of 1 to 3 represented that a campus was in the stage of critical mass building,

meaning that the STEM departments were in the beginning stages of service-learning

institutionalization; a rating of 4 to 6 represented the area of quality building, which suggested

that the STEM departments were aware of their service-learning needs and were working toward

service-learning institutionalization; and a rating of 7 to 9 represented the area of sustained institutionalization, meaning that service learning was fully integrated into the STEM

departments Following each of the 22 ratings, a space was provided for explanation should the respondent feel an explanation of the rating was necessary

STEM Service-Learning Self-Assessment Results

Each campus also received a total score for their self-assessment from the pretest and posttest The average for each dimension was added together to obtain the total score We used the average of each dimension thereby making each dimension of equal weight in the total score Out of a total 45 points possible, campuses averaged a total score of 17.6, with a range of 6.3 to

37.2 points for the pretest Figure 1 shows the distribution of scores across all campuses for the

pretest The total score for the self-assessment rubric shows that eleven campuses were in the

critical mass building stage, eight were in the quality building stage, and four were in the

sustained institutionalization stage overall

Figure 1 Pretest Self-Assessment Ratings for All CSU Campuses

By comparison, posttests scores demonstrated improvement for most of the campuses

On average, the campuses scored a 22.5 rating, which is at the quality building stage Scores

ranged from the lowest stage, with a score of 14.3, to the highest stage at 44.2 Figure 2

illustrates the distribution of scores for each campus at posttest Out of the 23 campuses, only

two were in the critical mass building stage; meanwhile, thirteen were in the quality building stage and eight in the sustained institutionalization stage

Figure 2 Posttest Self-Assessment Ratings for All CSU Campuses

The majority of the items in a particular stage also differed between the tests Seventeen

out of the twenty-two components were in the critical mass building stage for the pretest On the posttest, 16 items were scored at the quality building stage Interestingly, components 1.4

(highest on the posttest) and 5.3 (lowest on the pretest) had the largest increase from pre to post,

1.49 and 1.35 points respectively Figure 3 demonstrates the average score for each component

on the pretest (lighter shade) and the posttest (darker shade)

Figure 3 Average Pretest and Posttest Scores by Component

Reviewing the individual components (see Figure 3) also shows a similar pattern of

improvement from pretest to posttest On average, across all of the campuses, scores improved

Quality Building

Sustained Institutionalization

even within components During the pretest, the lowest rated component was 5.3 Staffing (M = 2.7) whereas the highest rated component was 3.4 STEM Student Incentives and Rewards (M =

5.0) The lowest component on average for the posttest was 2.4 STEM Faculty Incentives and

Rewards (M = 3.72) and 1.4 Definition of Service Learning in STEM Departments (M = 5.74) had the highest score This indicates that the lowest score in the pretest was at a critical mass building stage as opposed to being in the quality building stage during the posttest

The comparison of pretest and posttest scores for each dimension reiterates the

development theme For the most part, campuses improved at least slightly from pretest to

posttest Additionally, dimensions 2 and 5 only (faculty and departmental support) had the most

campuses (n = 6) stay at the critical mass building stage from pretest to posttest Most

dimensions also had at least five campuses increase their score by at least two points, with the exception of Dimension 1 (Philosophy of Service Learning) which had only three campuses move up Subsequently, two particular campuses reduced their rating scores across all of the dimensions For both pretests and posttests, Dimensions 1 (Philosophy) and Dimension 3

(Student Support) had the lowest number of campuses (n = 2) at the critical mass building stage

as well as the greatest number of campuses (n = 6) at the sustained institutionalization stage

For each component, campuses also had an opportunity to provide explanations in

addition to their rating scores The relationship between the pretest and posttest revealed

interesting differences between the ratings Also, a focus on each stage of the assessment

indicated what may have been lacking for those in the critical mass building stage and what allowed some campuses to reach the sustained institutionalization stage A summary of

differences for each dimension is as follows:

Dimension 1: Philosophy and Mission of Service Learning The majority of campuses

rated this dimension at the quality building stage for both the pretest and posttest,

although more campuses rated individual components at the sustained institutionalization

stage during the posttest By the posttest, campuses generally had a more robust idea of how to incorporate service learning, such as creating a handbook for faculty

Dimension 2: STEM Faculty Support for and Involvement in Service Learning Campuses

had similar responses from pretest to posttest with ratings straddling the critical mass building and quality building stages Since the pretest, it appears that faculty members are

becoming more involved in service learning, within some campuses more than others

To what extent was participation in (STEM) 2 initiatives related to the capacity for

service learning in STEM on CSU campuses?

Dimension 3: STEM Student Support for and Involvement in Service Learning Some

improvements were made on this dimension, specifically; student incentives increased from pretest to posttest with more campuses indicating which courses are service learning

on student transcripts

Dimension 4: Community Participation and Partnerships Ratings on this dimension stayed fairly consistent from pretest to posttest Explanations revealed that there has consistently been some form of partnership developed between the institution and the community, but that it has not reached its full potential

Dimension 5: Departmental Support for Service Learning Ratings on this component

increased from the critical mass building stage to the quality building stage The

descriptions also varied from pretest to posttest Initially, campuses reported the need for more emphasis on departmental changes to include service learning The posttests

indicate that many of the campuses are now in the beginning stages of planning and implementing these changes

Ratings on the self-assessment rubric for serving learning in STEM on CSU campuses increased from pretest to posttest, although for a small number of campuses overall ratings decreased The original evaluation design allowed for tracking these changes over time as

campuses were engaged in various (STEM) 2 program activities We wanted to investigate how these varying levels of participation impacted the institutionalization of service learning on CSU campuses since not all campuses or STEM departments took advantage of available program offerings While there is a general trend toward increasing capacity on campuses, we wanted to

understand how participation in (STEM) 2 initiatives might be related to this increased capacity

We gathered information related to which campuses participated in (STEM) 2 initiatives,

including attending the EDI as a campus team, receiving funding from (STEM) 2 such as Laying the Foundations or Innovations subgrant, or attending a STEM C3 symposium Campuses were

then arranged to reflect the level of participation from minimal/ low participation to high

participation Figure 4 reflects the level of participation by campus from low (left) to high

(right)

Figure 4 Pretest and Posttest Self-Assessment Ratings for All CSU Campuses

In general, campuses that were engaged in more program components tended to have a larger difference between their pretest and posttests total scores On average, the six campuses who participated in three to five activities had an increase in FURCO score by 10.6 points Of

the campuses (n = 9) who were involved in one or fewer activities, only 7 points of

improvement were made All other campuses (n = 8) were in the middle range, participating in

two activities and gaining 4.8 points from pretest to posttest

Figure 4 also highlights the difference between the pretest and posttest scores for each

campus All but three of the campuses improved in their self-assessment rating during this time

Of these campuses, two decreased their score and one stayed the same Thirteen of the campuses

moved up a stage, eight from the critical mass building stage to the quality building stage, four from the quality building stage to the sustained institutionalization stage, and one from the first

stage to the third stage

Sustained Institutionalization

Participation

What was the impact of the (STEM) 2 activities such as Innovations subgrants, Laying the Foundation subgrants and Engaged Department Initiative in promoting Academic

Program Development?

Summary of Service-Learning Capacity in STEM Departments at the CSU

The self-assessment rubric revealed that most CSU campuses were either in the critical mass building stage or quality building stage during the pretest A majority of campuses, in contrast, were in the quality building and sustained institutionalization stages when completing

the posttest A comparison of pretest and posttest ratings shows that almost all of the campuses improved; approximately half of the institutions continued onto the subsequent stage An

analysis of the individual components also demonstrated the typical pattern of moving from the

critical mass building stage onto the quality building stage The movement from pretest to

posttest also varied across dimensions Some dimensions were initially rated by the campus at a lower stage or made more improvements than other dimensions Explanations highlight the improvements made by each campus but also emphasized their varying needs

Grant activities that targeted academic program development at CSU campuses included:

Innovation subgrants (awarded to 5 campuses), Laying the Foundation subgrants (awarded to 15 campuses) and the Engaged Department Initiative, which included the Engaged Department Institute and the Engaged Department follow-up subgrants (awarded to 5 teams from 4 CSU

campuses) Given that all of these activities occurred during Year 1, we wanted to follow up with participating campuses that received funding for Innovation subgrants or who attended the Engaged Department Institute to assess the sustainability or changes from these initiatives that have occurred over time

Innovative Practices in STEM Departments

Within the original Learn and Serve funding, academic program development was

supported by providing five CSU campuses with small grants for innovative service-learning

practices, referred to as Innovation subgrants These campuses were located in the three regions

of the state (north, central, south) and were required to contribute matching funds from their campus to support the initiative In 2013, we inquired about the status of these projects to

determine if any longer-term activities resulted from the initial funding The following is a brief description of each project and accompanying status update

Campus 1: Chico

The Innovation subgrant at Chico supported the development of materials and activities

for the Geological and Environmental Sciences department Service-learning students developed

15 presentations for the Chico Gateway Science Museum on topics such as monitoring

earthquakes, demonstrations of volcanic eruptions and other local geologic phenomenon

Visitors of all ages interacted with service-learning students and their projects While the specific service-learning courses were a success, Chico reported that garnering faculty involvement was a

“much more difficult prospect than originally envisioned.” This was attributed to the additional workload on faculty that comes with preparing for and teaching service-learning courses

Update: Since the initial Innovation subgrant funding, the relationship between Chico

State and the Chico Gateway Science Museum has continued and been largely dependent on the work of one faculty member In 2012, students in a Mineralogy class designed a hands-on

activity for the Gold exhibit at the museum and in spring 2013 a Volcanology class also did presentations at the museum It is unknown whether additional service-learning activities have occurred at the museum in the past two years One major barrier to integrating more service learning on campus has been to attract more faculty members to design similar experiences for their courses

Campus 2: Fresno

Fresno offered three new service-learning courses through the Innovation subgrant: the

first course was offered for chemistry students using a peer tutoring model to improve oral communication skills; a second, similar course was offered for physics students; a third course was offered for physics students using a planetarium docent model The students who enrolled in the tutoring courses provided tutoring to both local high school students and general physics and chemistry students at the university The planetarium model of service-learning course was a great success Through the course, service-learning students developed a pre-show video and throughout the year, the Planetarium noticed a marked increase in visitor engagement

Update: One Associate Professor in Chemistry has led efforts to develop two sustainable

STEM service-learning courses The peer tutoring model course has continued In addition, another course provides students with “authentic industry experience while conducting EPA compliant drinking water quality testing for the Scout Island Outdoor Education Center.”

Campus 3: Monterey Bay

The Monterey Bay campus is an obvious hub of activity focused on service learning for its students The campus reported in 2011 that “Service learning at CSUMB has proven itself to

be sustainable and STEM-based service learning is a major part of that sustainability.” Specific

activities that occurred during the Innovation subgrant process included: formation of the STEM

SL-Learning Community (LC) including six faculty and twelve community partners; creation of

a Service-learning Course Development Workbook to outline upper division learning outcomes; integration of social justice issues and service-learning outcomes into six service-learning courses and piloting of two STEM service-learning courses; and development of projects for seniors to use as capstone projects

Update: There were no specific updates obtained for service learning in STEM at the

Monterey Bay campus, however, there remains a campus-wide emphasis on integration of service learning as a core part of the curriculum

Campus 4: San Marcos

CSU San Marcos’ planned activities included launching service-learning opportunities for students, developing a brochure targeted at STEM faculty regarding the benefits of service learning, and supporting faculty innovation in STEM service learning through mini-grants for which the faculty could apply Over 200 brochures, developed by physics faculty and the Office

of Community Service Learning (OCSL), were distributed in the original project However, OCSL did not receive any faculty proposals for mini-grants, so staff redirected their efforts and ultimately, funded a faculty member in biology to start a science center with CSU San Marcos service-learning students at a local elementary school As a result, more than 1,000 elementary students have been engaged in hands-on STEM activities with CSU San Marcos students

Update: This project will require additional funding to be sustainable and therefore they

are currently seeking funding opportunities for this project

Campus 5: Sonoma

Sonoma State University used Innovation subgrant funds to support the Sustainable

Waterways Educational Engagement Program (SWEEP) The primary purpose of this project was to embed service learning emphasizing sustainability of local waterways, in particular, Copeland Creek Seven new service-learning courses were developed and agreements were

established for data collection and restoration of Copeland Creek Positive relationships were built between the Sonoma State facilities department, Academic Affairs, Sonoma Preserves and the Sonoma County Water Agency In addition, funds were used to convert the Campus

Community Garden (adjacent to the creek) into an on-campus “laboratory” for biological pest control and watershed contamination analysis Through the service-learning courses and efforts, the Campus Community Garden became more accessible and can now be used as an outdoor classroom Additionally, there is an informational hub providing materials about STEM, service learning, and the garden and creek partnerships, which is available for students and the general public

Update: In 2011, Sonoma State received funding from the National Science Foundation

(NSF) for the STEPing Up STEM program that provides students with inquiry-based,

multidisciplinary experiences in STEM In addition, the Sonoma County Waters Agency Board

of Directors recently approved a three-year contract for the Watershed Academics for

Sustainability Collaborative (WATERS) project that expands previous community-based work to provide students with additional hands-on learning experiences that also benefits the community

Summary of Innovative Practices Updates

The subgrants clearly provided CSU campuses with the ability to include service learning

in a way that was tailored to the specific needs of the institution and surrounding community

Some of the work that was started from the Innovation subgrant projects has continued over

time, while other work has been expanded or suspended Sonoma State was particularly

successful in securing additional funding for their community-based projects The most

sustainable efforts were based in Monterey Bay where the integration of service learning is a core component of the curriculum Overall, most campuses reported successful integration of

Innovation subgrant funds into their STEM service-learning programs Sustainability of such

projects generally requires ongoing effort and resources and many projects have been supported

by one or more enthusiastic faculty members at each campus

Laying the Foundations Subgrants Summary

Another component of academic program development was the implementation of the

Laying the Foundation subgrants These subgrants were offered to service-learning offices at

each campus as an opportunity to assess, showcase and support current service-learning activities

in their STEM departments Recipients of the subgrant received $4,000 in funds that required 100% match by the campus A total of 15 campuses received funds as part of this component Program goals were accomplished in 2011 and all fifteen campuses were able to gain a better understanding of the state of service learning in their STEM departments While the goal of the subgrants was to assess, showcase and support current service-learning activities, important

information was also gathered from this effort Table 2 summarizes key activities and outputs

from these subgrants

Table 2 Laying the Foundation Subgrants Participants and Activities

of Activity

Participating Faculty members

Participating Community partners

Participating College students

Participating K-12 students

Engaged Department Initiative

The final component of academic program development was the implementation of the

Engaged Department Initiative This initiative consisted of two parts: 1) the Engaged

Department Institute and 2) the Engaged Department follow-up subgrants

Engaged Department Institute

The Engaged Department Institute (EDI) was hosted by the CSU Center for Community Engagement and lead by expert presenters in community engagement The EDI was held in June

2011 in Long Beach, California at the CSU Chancellor’s Office The institute was offered to academic science departments from CSU campuses interested in the department as a unit of engagement and change Specifically, the three-day long conference brought teams of faculty, staff, and community partners together to develop strategies to 1) include community-based work in both their teaching and their scholarship, 2) include community-based experiences as a standard for majors, and 3) develop a level of unit coherence that allows them to successfully model community engagement and progressive change in the department The EDI was attended

by five teams from four CSU campuses Team members completed posttest surveys on the last day of the institute

Engaged Department Institute Participants

The EDI was attended by departmental teams in the STEM fields from Bakersfield, Fresno, Los Angeles and Monterey Bay Of the 23 departmental team members who attended the EDI, many had multiple roles which include the following: eighteen were faculty members, five were community partners, four were department chairs, and three were service-learning

directors/coordinators EDI participants reported an average of 12.94 years of teaching

experience at the postsecondary level Most team members had experience teaching in the

natural sciences and nursing fields

Engaged Department Institute Activities

Prior to the beginning of the institute, departmental teams were asked to complete a

“team preparation guide,” to both customize the Institute to the participants’ needs and to

provide a baseline assessment of where the department was before the Institute Participants were given time everyday throughout the three-day long institute to work with their team members and develop their engaged department plan During the first day of the institute, team members learned about the history of community engagement across the STEM disciplines and the many ways that service learning contributes to student success, fulfilling the mission of the University, and promoting community change Participants also discussed what community engagement currently looked like in their department, what it means to them, their students, university, and the community in which they belong The participants also discussed what they hoped to gain from participating in the institute During the second day of the institute, participants learned about successful existing campus-community partnerships, heard student perspectives on ways to

engage students in service-learning opportunities, and discussed successful ways to navigate the politics of community engagement On the final day of the EDI, participants’ brainstormed ways

to include, expand, and deepen community-based learning in their departments, and provide specific short-term and long-term goals for their departments Departmental teams presented their community-engagement plans, discussed the faculty resources they currently had in place

and the resources that are needed to create and maintain an engaged department Although all the

participating teams left the EDI with a plan for community engagement, their individualized plans were at different stages of development, had various short-term and long-term goals, and included unique approaches to achieving these goals For example, one team planned to establish multiple service-learning lower division, upper division, and capstone courses; another team planned to integrate community engagement into retention, tenure and promotion guidelines Teams then received $5,000 subgrants to implement their plans during the 2011-2012 academic year

Engaged Department Follow-Up Subgrants

After finishing the 2011 EDI, teams received $5,000 subgrants to implement the plans they outlined during the Institute Overall, the projects were successful as explained below:

CSU Bakersfield: the chemistry department, in collaboration with the student chemistry club and

several K-12 after-school programs, developed and hosted Chemical Circus, a series of

engaging, entertaining and educational chemistry activities designed to promote interest in science among youth and engage CSUB chemistry students Additionally, the department RTP guidelines were revised to more formally incorporate community engagement into faculty

responsibilities and service learning was incorporated into both general education and major courses Finally, the Chemistry Department was one of only nine winners nationwide to receive funding from “Partnering for Excellence: Innovations in STEM Education.”

CSU Fresno (Chemistry): the chemistry department enhanced their curriculum by incorporating

service-learning activities into their general education Chemistry and Society course, the

first-year undergraduate general chemistry laboratories, and into an upper-division major course in analytical chemistry As a result, in 2011-2012, 500 student volunteers contributed 1,800 hours

of service to create 16 finalized science kits, facilitate “Saturday Science visits” at the Discovery

Science Center, and created 25 other demonstrations to be used with primary and level students

secondary-CSU Monterey Bay: the division of science and environmental policy conducted a second year

of their previously established STEM SL Learning Community comprised initially of 6 CSUMB service-learning faculty members and 12 community partners (2 per course) to meet to

continuing developing their course-community partnerships During this second year the learning community developed a list of characteristics that define a strong course-community partnership and transformed that list into a Partnership Assessment Rubric The learning community

members applied the rubric to their own partnerships, developed a set of service-learning student reflection guidelines designed to assess the “Integration of Students” components of the rubric, and collaboratively examined those student reflections to gain greater insights into the quality of their partnerships Students were invited to the final learning community meeting to share their own service-learning experiences and ideas about what constitutes a strong course-community partnership The Partnership Assessment Rubric has been published and a complementary

webinar and discussion guide were created All are available on the CSU STEM Engaged

Learning website (www.calstate.edu/cce/stem)

Cal State Los Angeles: the department of chemistry and biochemistry worked with a community

partner, the Weingart East LA YMCA, and the chemistry department at East Los Angeles

Community College to offer an environmental chemistry-focused community engagement

experience The experience was embedded in a 1 unit seminar that was part of a group of linked general education courses (chemistry, statistics, and composition) for non-science majors in Spring term, 2012 Issues of environmental concern for the community partner were identified through two focus groups, and were incorporated into the course as five research topics for five groups of students working together The course culminated in group presentations about the topics, which contained recommended action plans for the community partner

CSU Fresno (Nursing): the nursing department hosted guest speakers and trainings for all

nursing faculty about service learning to introduce the pedagogy and begin developing a culture

of engagement One required nursing course was approved by the nursing faculty council as an

“S-designated” service learning course and several other courses are in the process of becoming service-learning designated courses

Follow-up with Engaged Department Institute Participants

After finishing the 2011 EDI, participants were asked to indicate how much they agreed

with statements about their experiences both before and after attending the institute through a

posttest and retrospective pretest survey To investigate the extent to which the EDI had

sustained effects on participants, we administered a follow-up survey during Fall 2013

Table 3 Engaged Department Institute Follow-Up Survey Respondents

Chemistry and Biochemistry Team

Chemistry Team A (Fresno)

Chemistry Team B (Bakersfield)

Division of Science and Environmental Policy Team

Nursing Team

A total of 23 team members attended the EDI in 2011 For the follow-up survey, there were 15

respondents Table 3 provides the details of who responded to the follow-up survey Four

participants did not respond to requests to participate, and four participants could not be

contacted most likely because they had changed positions

EDI Follow-up Survey Results

After finishing the 2011 EDI, participants were asked to indicate how much they agreed with statements about their experiences both before and after attending the institute through a posttest and retrospective pretest survey The follow-up survey contained many of the same items and results from the 2011 survey are indicated below Agreement to all survey items was

measured on a five point Likert scale items from 1 = Strongly Disagree to 5 = Strongly Agree;

therefore, general agreement to a survey item is stronger as the mean approaches “5” and disagreement is stronger as the mean approaches “1” A mean score close to “3” indicates the group’s indifference towards the particular survey item

Table 4 Mean Participant Ratings for Engaged Department Institute: Comparison of Pretest, Posttest, and

Follow-up Surveys

Survey Item

Before

Attending the EDI

Mean

After

Attending the EDI

Mean

Follow-up Survey

Mean

Our EDI team has a unified vision of community

I can articulate what engagement in an academic

Students participating in service-learning classes benefit

Faculty members have a responsibility to partner with the

community as members of an academic institution 3.63 4.16 4.36 Our communities benefit from working with students from

I can articulate why engagement in an academic program/

The students benefit from a service-learning experience in

ways that are not possible in a classroom 4.19 5.00 4.93 Working with community partners provides faculty with

Community partners benefit from service more than the

Generally, students benefit from service learning more than

Scale: 1 = Strongly Disagree to 5 = Strongly Agree

Statistical comparisons were not appropriate given the sample size of participants; however, overall trends suggest participation in the EDI has continued to have an impact on those who participated, and follow-up ratings were similar to those obtained at posttest

immediately following the institute (see Table 4) In general, the EDI experience has sustained

positive attitudes over two years for those who attended the institute One noteworthy exception

to this is the question regarding students benefitting from service learning more than the

community partner—this rating increased by more than a full point over time

Benefits of Service Learning

Departmental team members were also asked to answer questions about the benefits of service learning to students and the community Again, results of the follow-up survey show attitudes towards service learning are fairly consistent to the results seen immediately following the

posttest EDI survey (see Table 5) These results showing favorable attitudes towards service

learning are expected considering the successes that respondents shared via the follow-up survey Participants mentioned several courses that have been developed or that have had elements of service learning incorporated into the course curriculum For example, one participant

mentioned, “We have expanded our [service-learning] offerings to include a lower division environmental science class We have also added a new major (Environmental Studies), which expands on the service-learning experience up into the senior capstone.” Another participant indicated they had “inclusion of a [service-learning] component into the general chemistry

course.” and “creation of a new service-learning course in instrumental analysis.” Relationships with community partners were also mentioned as a success that has resulted from the EDI

Table 5 Engaged Department Institute Participant Mean Ratings, Service Learning Benefits

Survey Items

EDI Posttest Survey

Mean

Follow-up Survey

Mean

Accessing the community partner's expertise is an essential

Service learning can aid in fulfilling the mission of the

Service learning can improve student academic achievement 4.62 4.50 Service learning can stimulate community change 4.43 4.33 Service learning can support student retention 4.47 4.43

Scale: 1 = Strongly Disagree to 5 = Strongly Agree

Efficacy of the EDI

Lastly, the faculty members responded to survey items regarding the efficacy of the EDI

and service learning (see Table 6) Results indicated most respondents agreed or strongly agreed

with most survey items In fact, all respondents provided favorable feedback as to the efficacy of the EDI

“The EDI had a huge impact on our chemistry department In a way, it was like a faculty retreat for a few of the faculty to sit, focus, and discuss ideas related to student success and learning, and especially how to engage with our community Our chemistry department efforts have been featured by the College news, University news, and have benefited our student undergraduate chemistry club They have been nationally

mini-recognized for their combined community efforts with the [service-learning] activities and school age (K-12) children in the central valley.”