NGC Final Evaluation Report Feb 8_2010

Table of Contents Section 1: Introduction 2-4 1.1 Introduction 2 1.2 Executive Summary 2-3 1.3 Methodology 4 1.4 Evaluation Team Biographies 4-5 Section 2: Program Background 6-15 2.1

Acknowledgments

This report has been made possible with the assistance of Nate Waas Shull representing

Portland Schools Foundation, and Reese Lord representing the Office of the Mayor of the City

of Portland (referred to throughout the report as project directors) They have worked

diligently to ensure that all available data, materials, and records were expeditiously provided

to the Center staff, and they were always available to provide additional explanation and insight into this vast collection of materials In instances when surveys, interviews, and/or focus group data required additional clarification, the communication and organization for these was always thorough and timely Without this level of support and follow through, this report could not have successfully been completed The extensiveness of this report is an indication of their willingness to commit their time in support of providing precision and

thoroughness to this evaluation process

Also, it should be noted that this Ninth Grade Counts project was successfully planned and implemented in an abbreviated timeline As decisions were made at the highest levels,

individuals charged with the responsibility for setting these plans in motion immediately

proceeded with the design and implementation required for the successful launch of the Ninth Grade Counts initiative Ultimately their work resulted in the successful implementation of this broad array of summer transition programs Again, careful follow-through and clear

communication was pervasive throughout implementation Recognition of this careful,

thoughtful, and thorough effort is requisite and fundamental to its continued successful

503-725-9519 www.centerforsuccess@pdx.edu

Report completed: February 2010

Table of Contents

Section 1: Introduction 2-4

1.1 Introduction 2

1.2 Executive Summary 2-3 1.3 Methodology 4 1.4 Evaluation Team Biographies 4-5

Section 2: Program Background 6-15

2.1 Overview 6-9

2.2 Key Materials 10

2.3 Data Sources 11-13

2.4 “Connected by 25” Research Synopsis 13-15

Section 3: Impact of Program on Students 16-30

3.1 2009 Youth Corps Student Survey Results 16-26

3.2 Student Focus Group Results 27-30

Section 4: Impact of Program on Services and Resources 31-50

4.1 Debrief Session – Program Providers 31-34

4.2 2009 Ninth Grade Counts Partner Program Survey Results 35-46

4.3 Project Directors Focus Group Results 47-50

Section 5: Conclusions/Recommendations 51-53

Appendix A Table: Cohort Graduation Rate by District (Class of 2008)

Appendix B Total number of completing students by program and district

Appendix C Student Focus Group Protocols

Appendix D Ninth Grade Counts and Summer Youth Corps Debriefing

Session Agenda

Appendix E Project Directors’ Focus Group Protocols

Section 1: Introduction

1.1 Introduction

In October 2009, representatives from the Portland Schools Foundation and the City of Portland Mayor’s Office began meeting with representatives from Portland State University’s Center for Student Success (Graduate School of Education) to begin discussions leading to an agreement to provide third-party evaluation services related to the Ninth Grade Counts summer transition

project The scope of the work became more clearly defined to include a review and analysis of existing data, surveys, archived documents, and available qualitative and quantitative data

Further discussion of the requisite components of the Center’s evaluation process included data to

be collected through student and program staff focus groups and interviews From the compilation and analysis of this extensive information and substantial set of materials, a composite of

conclusions, insights, and recommendations for programmatic modifications and thoughtful next steps was to be prepared It should also be noted that the project directors retained the services of the Northwest Evaluation Association for quantitative analysis of participant demographics and performance data That analysis is on-going and results will be reported separately from this report

What follows is the summary of findings related to program evaluation work completed primarily in

2009 The report is divided into the following sections for ease of accessing specific findings: overview of the project, impact of program on students, impact of program on services and

resources, conclusions and recommendations, and appendices

1.2 Executive Summary

This report examines Ninth Grade Counts, a network of 22 distinct summer transition programs targeting at-risk incoming 9th graders from Multnomah County’s six major school districts during the summer of 2009 The initiative, coordinated by the Portland Schools Foundation (PSF),

represents a broad-based effort to increase high school graduation rates by helping more students start high school prepared to succeed The report also includes examination of Youth Corps – Exploration, an initiative of the Education Cabinet led by the Portland Mayor and the Multnomah County Chair, which provided college and career exposure opportunities for a subset of Ninth Grade Counts participants Detailed information on findings, recommendations, and methodology

is included in this report Several key items from the report are highlighted below:

• Nearly 800 students completed summer transition programs operated by 22

community-based organizations and school districts as part of the Ninth Grade Counts initiative; 420 of those students also took part in weekly career and college exploration trips through the Youth Corps – Exploration

Section 1: Introduction

• Approximately 500 participating students received high school credit through their

participation in the program Both students and partner program staff indicated that the opportunity to earn credit is a key motivator for students to participate in the

program

• Leaders of the initiative worked with 6 local school districts to identify the students

most at-risk of dropping out by applying a common set of academic indicators to the incoming 9th grade class; programs worked to focus recruitment on these students

• The initiative provided partner programs with significant in-kind resources to help

increase their capacity These included 36 full-time AmeriCorps staff, 28 Masters of Arts

in Teaching students from Marylhurst University, transit passes, student gift card

incentives, and coordination and transportation for weekly career site visits – all totaling more than $550,000 of in-kind support

• Students who participated in the career and college exploration experience reported

overwhelmingly positive perceptions and impacts of the program Students agreed or

strongly agreed that the program made them think more about their futures (83%);

made them feel better prepared for 9th grade (70%); motivated them to graduate high school (78%); helped them think about new career and postsecondary options (66%); and made them think college was possible for them (82%)

• Survey and focus group findings show that students were consistently motivated to

participate in the program due to personal factors including the desire to improve themselves, have fun, and gain new experiences; encouragement from a family member

or school advisor was also crucial; other motivating factors included to possibility of earning credit, and the promise of a transit pass or gift card incentive

• Representatives of the 22 partner programs that form the Ninth Grade Counts network

reported very positive perceptions of their participation in the initiative Staff from

these partner programs agreed or strongly agreed that participating in the Ninth Grade

Counts network had a positive impact on their programming (95%); increased their focus on incoming 9th graders (75%) and students at high risk of dropping out (60%); enhanced their services to youth (90%); and allowed them to serve more students (65%) 21 of the 22 programs indicated an intention to participate in the initiative again, and 100% of programs agreed that the effort was worthwhile and should be continued

These represent a selection of high-level findings from this evaluation More detailed findings and recommendations for further development of the initiative are included in the following report

Section 1: Introduction

1.3 Methodology

For purposes of this evaluation, our methodology included a study of all evidence and associated materials that were provided; focus groups of students and project staff; in-depth analysis of

student and program provider surveys; review of meeting notes and minutes; and the development

of an understanding of the project’s historical background to include the work of key initiators and providers

With an emphasis on qualitative research, this study focuses on the accomplishments of the Ninth Grade Counts 2009 project, while endeavoring to answer questions related to program meaning and relevance, program effects and impact, and accomplishment of stated goals and objectives

Following the necessary protocols and design work associated with qualitative research has been important in this evaluation process, and the result is a valid and credible set of conclusions and recommendations intended for program improvement Again, this level of work was made possible with extensive assistance from project directors from the Mayor’s Office and the Portland Schools Foundation

1.4 Evaluation Team Biographies

Linda Jessell, MA

Center for Student Success, Director

Portland State University

Linda Jessell’s extensive educational background is both varied and comprehensive and includes

33 years in K-12 education combined with 5 years of work in higher education In addition to her work in the Center for Student Success, Linda has most recently taught courses at both Portland State University and Lewis and Clark College Previous to joining the Center, Linda held several administrative positions for the Gresham-Barlow School District; Director of Curriculum K-12, Director of Secondary Schools, and Principal of Gresham High School Linda’s expertise lies in the areas of staff development, and curriculum and instruction

Deborah Sommer, EdD

Center for Student Success, Assistant Director

Portland State University

Deborah Sommer has a rich background in education of over 38-years in public education; as a staff development specialist, vice-principal, principal, curriculum director, school superintendent and university faculty Having worked in large and small school districts, suburban and rural schools,

Section 1: Introduction

and at an ESD, she has extensive experience in curriculum development, including power

standards, curriculum mapping and program audits A specialist in team building and conflict resolution, contract negotiations and building parent and community support for schools, she was the recipient of the Oregon Superintendent of the Year award in 2006-Canby School District

Richard Dills, EdD

Educational Consultant

Rick Dills has 32 years of experience as an educational practitioner Dr Dills was the director of student achievement for the Gresham-Barlow School District in Gresham, Oregon, a district serving 12,000 students in 20 schools that has received national and international recognition for its

leadership in using Rasch-based assessment information to study and promote student growth His more recent assignments focused on curriculum, assessment, instructional technology and professional development Dr Dills has taught classes on performance assessment and educational measurement for Portland State University, and previously worked for the Oregon University System to develop its Proficiency-Based Admission System (PASS) As coordinator for standards and assessment for PASS, Dr Dills collaborated with university faculty and teachers from Oregon high schools to develop a set of educational standards that have been lauded as among the most comprehensive and meaningful in the country

Section 2: Program Background

2.1 Overview

As a foundation for the Ninth Grade Counts project in Multnomah County, the “Connected by 25” research base inspired community leaders to reach consensus around a commitment to focus on high risk teenagers in jeopardy of not completing their high school experience A deeper

understanding of the issues and challenges associated with disaffected youth provided the

motivation for this group of city, county, and school district leaders to collectively commit their political will around a program targeted toward supporting high risk teenagers as they transition from 8th to 9th grade

The “Connected by 25” research (available at www.connectedby25.org ) drove the initiatives that emerged from the Mayor’s Office and the Portland Schools Foundation (PSF) during 2008 and 2009 PSF funds were committed to support high school interventions and a pilot summer program was

in place during the summer of 2008 Lessons learned from the pilot included a need to provide clearer communication and to have agreements about expectations between PSF and program partners Most importantly, these lessons served to inform the design and processes that shaped the work on the development of the 2009 programs

Simultaneously, the Mayor’s Education Cabinet committed to a multi-year initiative around three strategies These included: Ninth Grade Counts (students transitioning from 8th to 9th grade), Youth Corps - Exploration (incoming 9th graders participating in Ninth Grade Counts), Youth Corps- Internships (incoming 10th graders), and Youth Corps - Work Experience (incoming 11th and 12th

graders) The implementation of the Ninth Grade Counts and Youth Corps – Exploration summer programs for incoming 9th graders launched alongside a pilot version of the 10th grade internship program, as well as a large-scale work experience program for older students funded through federal stimulus dollars and managed by Worksystems, Inc This evaluation focuses exclusively on the interventions for incoming 9th graders

More specifically, the Ninth Grade Counts initiative emerged from PSF staff with the intent of

connecting identified youth with the necessary support to begin high school on the right track The target number of students for participation in this program was 1,000 The Youth Corps-

Exploration, an initiative out of the offices of Mayor Adams and County Chair Wheeler, focused on

9th graders and was intended to provide college and career exploration to approximately 500 of those students participating in the Ninth Grade Counts program PSF, the Mayor and Chair’s

offices, and Worksystems are currently working to further align this series of targeted summer programs

Section 2: Program Background

As these efforts progressed, involved parties developed a deepened knowledge and understanding

of the consequences related to young people who do not complete their high school education And, as a next critical step these leaders agreed upon a process for early identification of these high risk students The ensuing development of a common set of criteria set the stage for a coordinated effort on the part of six Multnomah County school districts to identify students as “academic

priority” and then to begin the process of securing a commitment from them to participate in a summer transition program

Noteworthy at this point in the project’s development was the work done by project directors (from the Portland Schools Foundation and the Mayor’s Office) to secure agreement across the county’s six major school districts to develop these common indicators of high risk eighth grade students, and then to commit to a concerted effort to secure their participation in one of the supportive summer transition programs These indicators, originally conceived and used in the Portland Public School District, correlated with the “Connected by 25” research data and provided the metric for early identification of eighth graders potentially at risk of dropping out of high

school

As each school district’s education strategy team moved forward with the process of student identification, the interpretation of these criteria varied to some degree, and some modification of indicators occurred Early in this process, the eighth grade students considered to be highly at-risk

of dropping accounted for roughly 40% of the entire class This figure is consistent with some national studies of the high school dropout rate, particularly as it relates to some minority

populations

Given pervasive inconsistency and inaccuracy of reporting of dropout data nationally and locally,

we recommend use of Oregon Department of Education (ODE) data for the “Cohort Graduation Rate” (see Appendix A) for the purpose of monitoring the progress of this first group of students participating in summer transition programs The data provides the necessary information related

to each of the six districts’ graduation rates over time as well as a comparison to the statewide average More specific information related to progress over time for students participating in each

of these programs should be targeted for monitoring within an on-going student data tracking system (e.g credits by grade level, attendance, GPA, etc.)

Recruitment of students into summer transition programs was conducted either by school staff or

by staff working in nonprofit community programs It became apparent during this recruitment effort that those adults who had the closest relationships with a particular student had the greatest impact on securing their commitment to participate At the school sites, counselors were critical to successful recruitment, and in the nonprofit partner programs adults who had developed close relationships with students and their families were most able to successfully recruit them into these summer programs

Section 2: Program Background

Although these community nonprofit programs were not legally able to access the school’s student information base (FERPA regulations) regarding academic priority students, a great deal of

successful recruitment resulted from their knowledge of the students in the program and the relationships that they had developed with students’ families These trusting relationships served them extremely well as they proceeded with the recruitment process As consideration is given to future recruiting processes, other factors that to a somewhat lesser degree contributed to securing student participation included: incentives such as gift cards and TriMet passes, the Ninth Grade Counts guidebook, and opportunities for college and career visitations

Of particular significance to this project’s success was the extensive level of community support and associated resources committed to the summer 2009 programs (Ninth Grade Counts and Youth Corps – Exploration) School districts and community-based organizations used their existing resources and programs to join the broader effort to support students through this key transition PSF and the Mayor/Chair’s offices also worked together to leverage additional resources to support these programs With an estimate of in-kind resources available to support students and families

at a half-million dollar level, there can be no question as to the resolve of Multnomah County constituents around successful program implementation And given this level of commitment, it is also critical to the continued evolution of these programs that accountability and reporting systems are firmly in place to provide explicit and comprehensive data to program supporters related to student achievement and growth over time

Review of intended outcomes provides for another means of analyzing the summer programs’ accomplishments Reaching as many of the county’s academic priority students (year 1 targeted incoming 9th graders) as possible and involving them in one or more of the summer support

programs was one targeted outcome The project came close to meeting the original goal of 1,000 students participating in a summer program Early indicators from program self-reports indicate that 830 students completed one program Further analysis showed that several students

participated in more than one program A total of 773 unduplicated students completed the

program (see Appendix B) The program component that includes career and college visitations reached 420 students Given the commitment to develop a student data tracking system,

determining student attrition rates must be an important feature of the system in order to provide validation and accuracy to student participation numbers

Another intended outcome was to increase the capacity and enrich the quality of summer

transition programs Specific examples indicative of successful accomplishment of this outcome included: the staffing commitment from AmeriCorps, and the participation of Marylhurst MAT students, the provision of TriMet passes and student incentives, and the coordination and expense

of the career and college site visits These enhancements to the original program design added viable enrichment and expansion opportunities for students Program providers in their survey completed at the conclusion of the summer work (2009) indicated that they had the capacity to involve more students and they also reported that they could become more creative with

interventions Overall success related to this outcome is apparent given this increase in the

Section 2: Program Background

number of possible summer program slots available as well as having a broader array of

intervention possibilities

With the knowledge that student credit deficiencies are a key contributor to students dropping out

of high school, the opportunity for students to earn credit provided an incentive for many students

to enroll in the program Current information indicates that approximately 500 of the roughly 800 students who completed the summer program were successful in securing credit toward

graduation Generally these credits were elective with one district awarding elective credit within the content areas of Language Arts and Science (1/2 credit of each) to about 80 students PSF worked with the Portland Public School District to extend the opportunity for community-based organizations to help students to receive credits through an off-campus learning experience These also were elective credits that were validated by an adult the students worked with during their

“field experience” Student focus group participants repeatedly noted that this opportunity to earn credit(s) prior to high school was a significant contributor to their decision to participate in a

summer transition program

Two surveys (student and program provider) were administered at the conclusion of the summer transition programs with the intent of collecting useful qualitative data for future program

modifications Of the two, the student survey was considered to be less useful as students

commented on specific aspects of the program but did not necessarily answer the specific

questions Future planning includes the use of a single comprehensive student survey that is

administered in a pre/post format and provides insight into students’ attitudinal shifts as well as information specifically assessing the value of the college and career visitations On the other hand, partner surveys provided valuable actionable data such as the strengths and challenges related to the AmeriCorps component and the Marylhurst MAT effort

In reflecting (via surveys, focus groups, and interviews) on the usefulness of the strategies used to solicit and secure resources in support of these programs, notes and follow-up calls to businesses from the Mayor and County Chair proved to be extremely successful Other supporting resources identified as useful included TriMet passes for students in order to address transportation issues up front, and AmeriCorps who provided 36 full time staff to work with students at no cost to the program Useful resources specific to supporting the Youth Corps college and career exploration effort included bus passes and gift cards (although these were the most challenging to secure) Another challenge was securing the “coaches” assigned to assist students with their college and career visitations This should be one aspect of the Youth Corps structure that needs to be refined

so that coaches ensure that students have a say in the focus for the visitations

In reflecting on public relations strategies incorporated into this project, it will be important in the future to clearly establish a brand/identity in order to communicate with potential contributors as well as to a broader constituent group Recognition of the importance of working specifically with the student population identified as most at risk of not completing high school must be a priority for future communication efforts The development of a set of succinct information documents and strategies that communicate to the community the “cost” of not educating these young people will contribute to the galvanization of this effort Equally as important will be providing accurate

Section 2: Program Background

progress reports about student participants and their progress over time Accountability related to student success will be mandatory in order for this focus to become a long-term community

priority

2.2 Key Materials – Ninth Grade Counts & Youth Corps, 2009

Provided on 10/12/09 to PSU Center for Student Success for purposes of evaluation of Ninth Grade Counts/Youth Corps 2009

1 Ninth Grade Counts Overview (2-page)

2 Ninth Grade Counts Guidebook

3 Document for Press Conference: Youth Corps and Ninth Grade Counts

4 Resource Description and Request Form

5 Partnership Pledge

6 In-kind resource accounting

7 An example of in-kind resource tallies sent to programs

8 Responses from programs to the in-kind resource tallies

9 Academic Priority criteria proposal

10 Evaluation Plan proposal presented to Cabinet (2-sided with diagrams) – presented prior to conversations with PSU regarding more in-depth qualitative analysis

11 Debrief Meeting Agenda/Facilitator’s notes

12 Combined notes from 3 sub-groups at debrief meeting

13 Message to programs re: securing academic credit for students

14 Oregonian article on Ninth Grade Counts from 8/17/09

15 Mid-summer “e-update” message sent to stakeholders

16 Recruitment and Outreach Working Document

17 Program Providers Survey

18 Youth Corps Survey

19 Descriptive Graphic for the “big picture” of Ninth Grade Counts and Youth Corps

20 Youth Corps coordinator job description

21 Employer site host approach letter

22 Employer letter to request they encourage staff to be volunteer coaches

23 Coach flyer

24 Coach application

25 Coaches training agenda & materials

26 Youth Corps Provider meeting agenda

27 Coaches Handbook

28 Facilitator schedule and agenda for first day with Youth Corps students

29 Site host thank you

30 Education Cabinet update 4/21/09

31 Youth Corps Coaches Nuts and Bolts information meeting

32 Youth Corps background

33 Ninth Grade Counts Community of Practice Introduction

34 Youth Corps brochure

Section 2: Program Background

2.3 Data Sources

This evaluation of the 2009 Ninth Grade Counts summer transition programs was conducted by

the evaluation team from the Center for Student Success in the Graduate School of Education at Portland State University The Center was hired to conduct this evaluation in November 2009, to

be completed and submitted to the Portland Schools Foundation and the Mayor’s Office by the conclusion of January 2010 This evaluation includes and builds upon data, summaries, and surveys completed at the conclusion of the summer 2009, up to mid-January 2010

Given that this evaluation model was designed and implemented following the conclusion of the summer program, some limitations regarding accessibility to participants tempered the findings of this study For example, the opportunity to interview parents of participating students was not possible at this time, but our recommendation is to have this in place as the next phase of the summer program is implemented for summer 2010 The team that the Center worked with from the Portland Schools Foundation and the Mayor’s Office are to be commended for providing us with the extensive materials for review, focus group opportunities, and other helpful information The following sources provide the basis for this evaluation:

Policy documents:

• Academic priority criteria evaluation plan proposal

• Coordinator job description

• Education cabinet update – 4/21/09

• Email message regarding the awarding of academic credit

• Introduction to Community of Practice

• Resource description and request form

• In-kind resource accounting

• Example of in-kind resource tally

• Email responses from providers to resource tally

• Employer site host approach letter

• Employer thank you letter

Section 2: Program Background

Public Relations/Recruitment Documents:

• 9th Grade Counts overview

• Press conference: Youth Corp/9th Grade Counts

• Oregonian article

• Descriptive graphic

• Youth Corps background flyer

• Summer Youth Corp brochure

• Midsummer e-update to stakeholders

• Coaches’ training – June 12, 2009

• Youth Corps Coaches Nuts and Bolts

• Program providers meeting – June 18, 2009

• Youth Corps student agenda

• August 20, Debrief Meeting agenda and facilitator notes

Background/History:

• “Connected by 25” research

 “Connected by 25: Real Results for Young Portlanders”

 “New Growth in Stumptown: Young Portlanders Face Twenty-First Century

Challenges”

 “The Fourth R: New Research Shows Which Academic Indicators Are Best Predictors

of High School Graduation”

Focus Groups:

• Student

 Madison High School – El Programo Hispano

 David Douglas High School – Ninth Grade Counts

• Representatives from Ninth Grade Counts and the Mayor’s Office

Section 2: Program Background

Using the information accumulated from all of these data sources, the Center for Student Success has captured progress towards stated goals and outcomes We have analyzed the data to

determine if it is suggestive of longer-term trends or prospects We have also identified program components for commendation and consideration as developmental work on the Ninth Grade Counts project continues to unfold

2.4 “Connected by 25” Effort and Research Synopsis

Research studies included:

1 “Connected By 25 – Real Results for Young Portlanders”, Spring 2007

2 “New Growth in Stumptown: Young Portlanders Face Twenty-First Century Challenges”, John Tapogna, M.P.P., Managing Director, ECONorthwest, Sharon Bernstein, M.A., Lois Leveen, Ph.D., Spring 2007

3 “The Fourth R”, Mary Beth Celio, M.A., Senior Researcher Northwest Decision Resources and Lois Leveen, Ph.D., Spring 2007

The “Connected by 25” effort is comprised of more than 35 Portland area corporations, nonprofit organizations, foundations, and educational advocacy groups Its intent is to connect young

Portlanders to school, work, and the community by age 25 This research-based initiative has created a citywide network that includes programs and services targeted toward ensuring success for the city’s young people, with an initial focus on the 9th grade transition

The “Connected by 25” project noted the presence of many unrelated programs directed toward increasing the number of students graduating from high school, including alternative schools, school-based interventions, and after-school and summer programs By strategically aligning around common objectives, “Connected by 25” envisions creating even more powerful and

effective initiatives on behalf of struggling students

Key points and research findings published in the 2007 “Connected by 25” research:

(research available at www.connectedby25.org)

 Changes in the city include population growth with significant increases in racial diversity and households speaking languages other than English

 Over one-third of students in 4 of Portland’s 5 school districts are children of color

 Poor students are the majority in 4 of the 5 districts

Section 2: Program Background

 High wage jobs are increasing for highly educated and/or highly skilled workers, and low wage jobs are increasing for low skilled workers Educational level is the key in this job market

 Young people who are successfully connected to school or work by age 25 will earn higher wages and have better long-term employment prospects Contrastingly, youth who are not connected by age 25 will not enjoy these benefits

 In Portland, over 86% of young people were connected to either school or work while approximately 8,000 others were not

 Young people successfully connected to education and/or the workforce by age 25 will be more engaged in their community which in turn will produce social benefits

 Family, community, and school-based networks result in more school and work success for students When this support is continuing and complementary the best results occur

 Specific academic indicators provide an accurate predictor of students who will leave high school without graduating Use of these indicators increases the likelihood that students most in need will receive the necessary support directed toward high school graduation

 Students’ performance on academic indicators provides the most accurate assessment of whether or not a student will graduate When two of these academic indicators are

considered (number of 8th grade standards met and the number of core classes passed or failed during 9th grade), demographic variables such as race and poverty are no longer statistically significant

 Students tend to disconnect from their school programs primarily during the summer and following the conclusion of grade 12

 Prior academic achievement is an important indicator for predicting graduation Targeted support to these students testing in the lowest scoring groups will result in more students graduating on time

 The ninth grade year is pivotal in terms of success or failure resulting in either graduation or not Students failing one or more 9th grade core classes were 4 times more likely to leave

high school before graduation

 Failure to make necessary progress on course credits results in a significant increase in the risk of the student dropping out of high school

 Students experiencing one or more educational disruptions during high school (e.g

transferring schools, temporary withdrawal, etc.) will affect the student’s likelihood of graduation

Section 2: Program Background

 By using the longitudinal cohort group method for determining graduation rates, individual student’s progress over the course of four years is best tracked This method reveals

student progress as evaluated using various academic indicators and demographic factors and results in increased accuracy related to eventual graduation

Key challenges:

 The area’s population is increasingly impacted by changing demographics These

population shifts directly affect schools, community groups, and government agencies

 The economy is shifting to include an increasing number of high-wage jobs for highly

educated and skilled individuals while low-skilled workers only have access to increasing numbers of low-wage jobs

 Young people need support to be successful

Portland realities:

 Portland is comprised primarily of white individuals while young people are increasingly diverse

 Portland’s child poverty rate is increasing more rapidly than other major cities

 Portland is increasingly divided by race, class, and language with young people in the most diverse and poorest neighborhoods having experiences much different than others

 Portland is nationally recognized for its reduction of young people in the criminal detention system

 Enrollment, while declining in Portland Public Schools, is increasing in the city’s other four districts

 School complexities include the fact that many students are performing well while many others are not The system’s youngest students perform relatively well, while in higher grades the number of students meeting standards drops and the achievement gap between white students and students of color widens

Given these challenges and demographics, the Ninth Grade Counts initiative is particularly timely in that it focuses on a difficult time in the lives of many students – the transition of students moving from middle school to high school Those supporting this effort have clearly read the research and want to make a difference in the lives of Portland area youth entering the 9th grade

Section 3: Impact of Program on Students

3.1 2009 Youth Corps Student Survey Results

Highlights: When Youth Corps student participants were surveyed at the end of the summer in

2009, they overwhelmingly indicated that the program had been a positive experience for them Analysis of their survey responses suggests that:

1 Youth Corps Career and College Exploration experiences have made students think more about their futures, be more motivated to graduate from high school, and think that

attending college is possible for them

2 Students feel better prepared for 9th grade as a result of their summer experiences

3 Students report that they learned from their career and college site visits and are

considering different career and post high school educational options because of them

4 Students’ Summer Youth Corps Coaches helped them benefit from the experiences

5 Students would overwhelmingly recommend Youth Corps to a friend and are interested in participating in the future

6 While incentives like the TriMet pass and gift card influenced many students to participate

in Youth Corps, they were also strongly influenced by the encouragement of family

members and school advisors; moreover, their “biggest reasons” for participating were

most often personal

Background: In the summer of 2009, approximately 800 students from six Portland metropolitan

school districts participated in one of 22 Ninth Grade Counts partner programs Thirteen of the partner programs chose to take advantage of the Youth Corps, which provided career and college exploration experiences for nearly 500 of the Ninth Grade Counts participants To study students’ perceptions of their Youth Corps experiences, attitudes resulting from those experiences, and reasons for participating, Youth Corps administered a student survey in August 2009, near the end

of students’ participation in the Ninth Grade Counts and Youth Corps programs The survey was completed by students within the partner programs who then returned survey results to Youth Corps staff A total of 221 student surveys were then tallied by Youth Corps, and the results

analyzed by Portland State University’s Center for Student Success

Survey Instrument: The 2009 Youth Corps Student Survey was a twenty-item instrument

developed by the Youth Corps program staff The first ten items used a five-point Likert scale to measure the extent to which students agreed with a series of positively worded statements about Youth Corps, their experiences, and their attitudes about their academic futures Two key items related to the stated goals for both Youth Corps and Ninth Grade Counts were identified in the analysis process: “The Youth Corps helped me feel more motivated to graduate high school” and

“Youth Corps helped me think that college is actually possible for me.” The second section of the survey consisted of two “Yes/No” items addressing whether the students “would recommend Youth Corps to a friend” and were “interested in participating in Youth Corps next year.” The third section of the survey asked students to indicate all the reasons they participated in Youth Corps (from a list of eight possible reasons), then indicate the single biggest reason for their participation

Section 3: Impact of Program on Students

and (in an open-ended format) explain why The final six items were open-ended questions

designed to further probe students’ experiences in Youth Corps career and college exploration activities

While the survey was developed by Youth Corps, it should be noted that students’ responses may have been influenced by their feelings about and experiences within their Ninth Grade Counts partner programs, since the survey was administered within those programs Ninth Grade Counts did not itself administer a single comprehensive survey to students in 2009, but the results from the Youth Corps Survey may also be indicative of students’ perceptions and attitudes about Ninth Grade Counts, and about their particular partner programs

Analysis and Results: The evaluator studied the frequencies of the 221 surveyed students’

responses to the ten Likert-scale items and also compared response patterns across the ten partner programs that returned surveys to Youth Corps Because students’ perceptions and attitudes are a potential first indicator of changes in their future school behavior and success, it was important to ascertain the degree to which students left the program with positive perceptions of themselves, their futures, and the program’s impact on both In the data displays that follow this narrative, the percentage of students indicating that they “strongly agree,” “agree,” “sort of agree,” “disagree,”

or “strongly disagree” for each item are represented in a pie-graph format Because the survey employed a five-point scale, “sort of agree” was interpreted by the evaluator as a neutral response, and therefore the analysis and interpretation focused on the frequency of student responses at the

“agree” or “strongly agree” levels

In the case of all ten items, the frequency of positive responses was very high, ranging from 61% to 83% of students who agreed/strongly agreed that the statement was true for them Students indicated the highest frequency of positive responses for items related to how the program: “made

me think more about my future” (83%); “helped me think that college is actually possible for me” (82%); and “feel more motivated to graduate from high school (78%), which were also the broadest indicators of students’ perceptions of positive academic futures 79% of students indicated that

“the career site visits were interesting and I learned something,” while 74% indicated positive responses to a similar item related to “the college and other post high school visits.”

Students indicated more ambivalent (but still very positive) response patterns to more specific items related to the impact of those visits: 34% said that they “sort of agree” or “disagree” to statements about how the site visits “helped me to think about new and different career options” and caused me to “consider new or different college and post high school options.” When asked about their summer Youth Corps coaches, about three-quarters of students agreed that their coach

“helped me to get more out of this experience,” while six in ten agreed that “I wish I could continue

to met with my Summer Youth Corps Coach.” In emphasizing again the overwhelmingly positive picture represented by this survey data, it should be noted that the greatest level of disagreement with any of the ten positive statements about Youth Corps was a mere 9% (for the item about students’ wish to continue meeting with their summer coach)

Section 3: Impact of Program on Students

Students’ responses to the “Yes/No” items were even more positive 92% agreed that they would recommend Youth Corps to a friend, and 83% indicated that they were interested in participating in Youth Corps next year

The responses to two of the attitudinal items were also compared across the ten programs that submitted surveys (data summarized in bar charts at the end of the display section) For the item

“The Youth Corps helped me feel more motivated to graduate high school,” the frequency of

students who agreed/strongly agreed ranged from a high of almost 90% to a low of just over 50% For the item “Youth Corps helped me think that college is actually possible for me,” the frequency

of students who agreed/strongly agreed ranged from a high of 95% to a low of just under 60%

While these results indicate some variance in perceptions of Youth Corps across the Ninth Grade Counts partnership programs, the differences were not determined to be significant, reinforcing that the patterns for all students are indicative of their experiences no matter which partner

program they attended However, it should be noted that the 221 respondents (out of

approximately 800 partnership program students and nearly 500 Youth Corps participants) do not necessarily represent a statistical sampling across all programs, and therefore the very positive conclusions about students’ experiences implied by the survey data should be considered in this context

Students not only reported their attitudes about the Youth Corps program, but also communicated their reasons for participating On the survey, students were given seven possible reasons for participating and an eighth, open-ended option (“A different reason?”) They were asked to check all reasons that applied to them, and then to indicate the “biggest reason” they had participated Their response patterns (summarized in a bar graph in the data section that follows this narrative) suggest that students most often indicate the incentive card, TriMet bus pass, and family

encouragement as reasons they participated in Youth Corps Their “biggest reason,” however, was most often one different than the listed incentives When their open-ended responses for

“different reasons” are studied, the most common are related to students’ desires to improve themselves, their desire to meet new friends and have new experiences, and their wish to have fun and avoid the boredom of “nothing to do at home.” Students’ primary reasons for participating in Youth Corps seem therefore to be largely personal, although several students also mention their desire to learn about and “go to college.”

Commentary and Recommendations: The 2009 Youth Corps Student Survey results indicate very

positive student perceptions of the program, and they likely suggest positive feelings about the entire Ninth Grade Counts experience as well Notably, all items received very high levels of

agreement (the lowest frequency of agree/strongly agree responses was still over 60%), and the responses across all ten programs were also consistently very positive Students’ overwhelmingly (and consistently) positive responses to the survey would seem to indicate that the 2009 Youth Corps program served them well, engendering positive attitudes both about the program’s career and college exploration experiences and about the impact on students’ perceptions of their

futures Of particular note are students’ high levels of agreement with the items related to

preparation for 9th grade, high school graduation, and college, especially when one considers that

Section 3: Impact of Program on Students

“academic priority” students would be expected to have less positive responses to questions about their scholastic futures Unfortunately, it is not possible to interpret the survey as a measure of change in students’ attitudes, since there is no baseline – either from previous student cohorts or from a pre-experience survey of this year’s students

The survey data related to reasons for participating in the program is also revealing It appears that the extrinsic incentives offered to students (TriMet pass, $100 gift card) were a reason why

students chose to participate in the program; however, they were not the primary reason The data would indicate that continuing these sorts of incentives is warranted, but would also suggest that appealing to the more personal reasons mentioned by students, and continuing to work through families and school advisors, is even more important

In planning for future student surveys, the evaluator recommends:

 Administration of the student survey in a pre-/post model so that it is possible to study changes

in students’ perceptions from the start to the end of their summer experiences

 Administration of the student survey across all programs as an expectation

More detailed suggestions for such a survey will be developed by the Center for Student Success to inform future planning and data gathering

Graphic Data Summaries:

A Likert Scale Perception of Program Items – ALL Students

B Yes/No Items – ALL Students

C Reasons for Participating – ALL Students

D Comparison of Graduation and College Items Across Partner Programs

Section 3: Impact of Program on Students

The Youth Corps Career and College Explorations made

me think more about my future

83% of students agree or strongly agree

34%

49%

15%

I feel better prepared for 9th grade as a result of my

Youth Corps experiences

70% of students agree or strongly agree

The Youth Corps helped me feel more motivated to

graduate high school

78% of students agree or strongly agree

The career site visits helped me to think about new and

different career options that I hadn’t thought of

Section 3: Impact of Program on Students

I am considering new or different college and post high

school options because of our visits

66% of students agree or strongly agree

Section 3: Impact of Program on Students

Youth Corps helped me think that college is actually

possible for me

82% of students agree or strongly agree

Overall, the college and other post high school visits

were interesting and I learned something

74% of students agree or strongly agree

My Summer Youth Corps coach helped me get more

out of this experience

72% of students agree or strongly agree

Section 3: Impact of Program on Students

I wish I could continue to meet with my Summer Youth

Section 3: Impact of Program on Students

C: Indicate ALL the reasons you participated in Youth Corps this summer Which of these is the

BIGGEST REASON?

I would recommend Youth Corps to a friend

92% of students say yes

YES 92%

NO 8%

I am interested in participating in Youth Corps next

year

83% of students say yes

YES 83%

N O 17%

144 165 108

49

129 86

40

71.5

5.5 29.5 14 4 26.5 22.5 4

I have participated with the sum m er program in the past.

M y parents or fam ily m em ber wanted m e to participate.

A friend encouraged m e to participate.

S om eone else talked to m e about the program

Section 3: Impact of Program on Students

D: Graduation and College: Comparison of Survey Responses by Program

The majority of students in all ten programs agreed/strongly agreed that Youth Corps made them feel more motivated to graduate The level of agreement ranged from a high of 87% (EPH-PHS, PSD-ELL) to a low of 52% (IRCO-Sabin)

6 0

11

16

3 0

8

3

1 1 0

8

5

2 1

6 2 4

2 1

1 5

1 2 1

4

7

2 0

10

8

2 0

10 16

13 1

15

14

5 0

IR CO_

MHS

IR CO_

Sa bin

NAY

A NH _R GMS

PSD _ELL Reynolds

Section 3: Impact of Program on Students

The majority of students in all ten programs agreed/strongly agreed that Youth Corps helped them think college is possible for them The level of agreement ranged from a high of 95% (EPH-PHS) to a low of 58% (NAYA)

15

13 1

6

3 2

2 0

9

6 1

7

5

2 0

2 4

1 2 1

3 6

3

0 1

11

5

2 0

13 19

7 1

15

14

4 3 1

Youth Corps helped me think that college is actually possible for me.

Section 3: Impact of Program on Students

3.2 Student Focus Group Results

Background:

On December 9th and 11th ,2009, representatives from the Center for Student Success met with students from Madison and David Douglas High Schools in a focus group format to discuss with them their experiences during the Ninth Grade Counts summer transition program Student

participants were targeted for participation in the summer program as a result of having been identified during their eighth grade year as an “academic priority” student Each focus group lasted for approximately one hour and the facilitator placed an emphasis on making certain that every student had an opportunity to respond to some or all questions (see Appendix C) The group size at Madison High School was eight participants and the group size at David Douglas High School was twelve participants Students were provided with pizza and soda pop as an incentive to attend and

to participate

The facilitator began with introductions, described the focus group process (length, participation and expectations, and how the information will be used), and reviewed confidentiality agreements Students in both groups were receptive to the process and seemed anxious, in most instances, to participate in the conversation

 “I was forced to come but I already knew I had to go.” (probe: Was it helpful?) “Yes,

I am actually turning in my work now.”

Perceptions about the summer program were generally positive and students felt that the program had been beneficial to them All seemed comfortable openly discussing the issues that were

challenges in their own lives that often prevented them from being more successful in school A number of these students obviously have issues other than academics impeding their progress in school

 “It helped me keep my grades up, to keep my stuff organized (probe: What did they do?) “Once I got my homework done, you would bring a notebook and it would

be organized by each class”

Section 3: Impact of Program on Students

 “My cousin said he was a slacker like me so I am trying to get my grades up (probe: Did the summer program help?) “Yes You know where you want to go before it is too late.”

 (Student now living with grandparents) “My mom and dad really aren’t part of my life right now.”

Student responses also indicated that they felt that the field trips/ college visitations were of

interest and these experiences had an impact on them Their comments provided insight into their current thinking about what they intend to do upon completion of high school In all instances, their comments implied an intent to finish their high school program Students in both groups knew the “buzz words” and had obviously heard the lectures about time management, getting organized, and why attendance is important They were able to repeat back what they needed to

do to progress toward college, for example, but they did not seem equally able to articulate a plan

to make second semester more successful than the first

 “On one of the field trips they talked about the income you get depending on how long you go to college.”

 “I am going to try not to be sick, to get my homework in on time and complete, and

to pay attention I need to work on my study habits – people distract me I will try

to focus better.”

 “It (the summer program) showed me colleges I could go to and scholarships I could get They told us that being Latina would help me because I am bilingual.”

Students in both groups had specific advice for incoming 8th graders (2010 freshman) and

understood clearly that the expectations at high school were tougher than at their middle school One very interesting set of comments made by a participant (and positively acknowledged by most other group members) reflected a desire to have teachers who set high standards and demand that students reach these This was of particular interest in that the statements made indicated that this student preferred information straight up rather than “sugar-coated.” Others in the focus group indicated agreement with this

 ‘Teachers here sugar-coat everything – but not her She is like… “I’ll unlock the door but you have to walk through it.” She says, “Do this (Romeo and Juliet) or else you will get an F”, and other teachers say, “It’s OK, you can make it up.” I like that about her.’

 “We need more depth because in the program we did a lot of light stuff but now my teachers are really strict.”

Another noteworthy point made in both focus groups by a number of students was that in-school supports for these students appear to be in place in both high school settings These supports

Section 3: Impact of Program on Students

seem to take various forms; from attendance and tardy “round-ups”, to counselors who provide both information and discussions regarding next steps (in one group 8 students indicated that their counselor is the one who will help them with next steps)

 “I want to be a detective.” (probe: Who would you talk to here?) “I would go to my counselor and she would help me.”

 “I want to go to Cal Poly and do photography stuff.” (probe: Who would you talk to here?) “My counselor.”

Students in both groups had specific advice for incoming 8th graders, and this was reflected in comments that revealed a clear understanding that the expectations in high school were higher than at middle school Consistently, their comments revealed their appreciation for the summer program’s emphasis on this factor and the work done to help them develop an understanding of credits, the importance of grades, and relationship between these and attendance

 “They showed us what was required and how to pass your classes.”

 “They talked to us about what credits you had to have and grades.”

 “They said freshman year was hard because people don’t take it seriously enough.”

 “They helped us be more confident.”

Conclusions/Recommendations

A number of very important conclusions may be gleaned from these conversations with students (9th Grade Counts participants) who are now close to completing their first semester in high school

We found that in most instances their insights that produced these recommendations were

grounded in serious thought, and were intended for those who would be designing the program for future “academic priority” students They reflected a desire to be helpful to both program

providers as well as future student participants

 The supervision/oversight by concerned parents was consistently cited as a key factor in program enrollment in both groups Increased parent contact should be part of the

marketing and monitoring effort for next year’s students, and should be an emphasis while the program is underway

 The communication effort to get information out to students and parents about the

summer program appeared to be more extensive at one school than the other Enhanced communication should be a focus for every school/program in preparation for the next summer Both personal contacts with parents and distribution of materials should be a concerted effort throughout the spring of 2010

Section 3: Impact of Program on Students

 Students in both groups provided specific recommendations for program changes in the program for next summer Some of these included: more sports, more variety of classes, art, math that more closely replicates their high school math classes, essay writing,

computers, test preparation, study skills and habits, organization, and time management These insights should be included as a factor as decisions are made regarding the 2010 summer program’s content

It is our recommendation (consistent with current planning) that a solicitation of feedback related

to program changes be a component of the written student survey to be completed at conclusion

of the Ninth Grade Counts program in summer 2010 Project directors will then have a written set

of carefully considered recommendations from every participating student for evaluation purposes

as well as use in decisions about future program modifications