April-2019-CAEP-8-Annual-Reporting-Measures

4 Measure 1: Impact on P-12 Learning & Development At this time, Kentucky no longer has a mandatory, state-funded process for obtaining data related to completers’ performance and P-12

ANNUAL REPORTING MEASURES

COLLECTING INFORMATION ON PROGRAM IMPACT AND OUTCOMES

Berea College Education Studies Department Teacher Education Program

APRIL 30, 2019

1

Table of Contents

2

Purpose of Report

The Council for the Accreditation of Teacher Preparation (CAEP) requires that Education

Preparation Providers (EPPs) publicly share program information and data that is accessible to multiple stakeholders At Berea College, the EPP is known as the Teacher Education Program (TEP) within the Education Studies Department (EDS) There are eight annual reporting measures used to examine both program impact and program outcomes In the 2018 CAEP Handbook1, the measures are defined as “basic indicators” of EPP performance that are associated with “candidates

as they complete preparation, and to completers once they are on the job” (p 26) Data for these measures were gathered from multiple sources including departmental surveys and focus groups; reports generated by the Berea College Office of Institutional Research and Assessment; and reports published by state agencies such as the Kentucky Center for Statistics (KYStats) and the Kentucky Council for Postsecondary Education

1 At this time, the referenced (and most recent) handbook is the CAEP Handbook: Initial Level Programs 2018 The current CAEP Accreditation Handbook can be accessed at:

http://caepnet.org/accreditation/caep-accreditation/caep-accreditation-handbook

3

The Eight CAEP Annual Reporting Measures

IMPACT: Performance of program completers once they are employed as P-12 teachers

1 Impact on P-12 Learning & Development (Component 4.1)

2 Indicators of Teaching Effectiveness (Component 4.2)

3 Employer Satisfaction and Completer Persistence (Component 4.3)

4 Completer Satisfaction (Component 4.4)

OUTCOME: Outcomes of program preparation and consumer reporting information

5 Completion/Graduation Rates

6 Licensure/Certification Rates

7 Employment Rate

8 Consumer Information

4

Measure 1: Impact on P-12 Learning & Development

At this time, Kentucky no longer has a mandatory, state-funded process for obtaining data related to completers’ performance and P-12 student growth This is due to change in regulation that once required newly inducted teachers (program completers) to participate in the Kentucky Teacher Internship Program (KTIP) Participation in KTIP generated performance data related to

completers’ performance in the classroom Additionally, new leadership at the Kentucky

Department of Education has indicated that the state will continue its practice of not releasing P-12 student data Therefore, this impacts plans for collecting data on P-12 student learning and

development

With these changes in state leadership and mandated processes, EDS will focus on working with local school districts to obtain P-12 student data Relationship building with community

partners has been a priority for the EDS thus far during the 2018-2019 academic year EDS has started a discussion with the Community of Teachers (a stakeholder group consisting of P-12

educators) on how to obtain P-12 student data in a way that respects school districts, completers, and P-12 students EDS will continue this collaboration and formulate plans for obtaining P-12 student data during summer retreats planned for the Community of Teachers and for EDS faculty and staff EDS anticipates that data collection will begin during the 2019-2020 academic year after receiving guidance and permission from school partners

EDS will continue conducting completer focus groups so that the department can maintain strong relationships with completers and learn how they utilize program preparation to positively influence their students From analysis of focus group discussions, EDS has learned that completers apply program knowledge to measure student growth through the use of formative assessments Completers discussed how they analyze and interpret assessment results in order to better

understand students’ abilities, to measure students’ progress, and to inform classroom interventions P-12 student growth data is available through the use of completers’ classroom assessments EDS must ensure that the processes for sharing assessment results and drawing conclusions based upon the data have mutual benefits for completers and EDS

5

Measure 2: Indicators of Teaching Effectiveness

To successfully meet this component, CAEP requires that teaching effectiveness be

demonstrated “through structured and validated observation instruments and/or student surveys” Currently, the EDS has collected employer survey and completer survey data that asked about teaching effectiveness (data are presented below) However, since observation instruments and students surveys are the desired sources of evidence, EDS will collaborate with our partners to develop processes as well as observation tools and P-12 student perception surveys The measures need to be developmentally appropriate for users and must be approved by the school districts who are our collaborators Additionally, data from these measures should inform the professional growth

of program completers and provide them with feedback that can be used to refine their practice

Employer Survey Results The employer survey is a departmental survey that was administered for the first time in January 2019 to the principals of program completers Survey indicators are elements from the Charlotte Danielson Framework for Teaching2 that were selected by stakeholders; content validity for the indicators was established using the Lawshe (1975 method) Employers were selected based upon the responses from a survey of 2015-2016 and 2016-2017 completers who specified that they were currently employed as P-12 teachers Twelve out of 29 completers indicated that they are current P-12 teachers and provided their employers’ contact information Six of the 13 employers who were contacted replied to the survey resulting in a 46.2% response rate

Employers were asked to rate their employees’ performance for each element using the following categories:

 Exemplary (4): Meets the objective or standard with a high level of consistency

 Accomplished (3): Cleary developing the ability to meet the objective or standard with some degree of consistency

 Developing (2): Shows some development in addressing the objective or standard

 Ineffective (1): Shows little or no evidence of development in addressing the objective or standard

The survey data reveal that employers perceive completers as effective teachers and are satisfied with their preparation from the TEP Overall, employers rated completers’ teaching

effectiveness higher on elements related to planning and preparation and lower on elements related

to instruction and management of instructional groups

2 The Framework for Teaching that has been adapted for the Kentucky Department of Education can be accessed at:

https://education.ky.gov/teachers/PGES/TPGES/Documents/Kentucky%20Framework%20for%20Teaching.pdf

6

Employer Ratings of Completers’ Teaching Effectiveness

1B Demonstrating Knowledge of Students – Knowledge of the Learning Process 3.67 1C Selecting Instructional Outcomes – Value, Sequence, and Alignment 3.67 1E Designing Coherent Instruction – Learning Activities 3.67 2B Establishing a Culture for Learning – Expectations for Learning and Achievement 3.67 1A Demonstrating Knowledge of Content and Pedagogy – Knowledge of Content-Related Pedagogy 3.50 1E Designing Coherent Instruction – Lesson and Unit Structure 3.50 1F Designing Student Assessment – Congruence with Instructional Outcomes 3.50 1F Designing Student Assessment – Design of Formative Assessments 3.50 2A Creating an Environment of Respect and Rapport – Teacher Interaction with Students 3.50 3A Communication with Students Expectations for Learning 3.50 3D Using Assessment in Instruction – Monitoring of Student Learning 3.50

2D Managing Student Behavior –Monitoring of Student Behavior 3.33 2D Managing Student Behavior – Response to Student Misbehavior 3.33 2C Managing Classroom Procedures – Management of Instructional Groups 3.17 3C Engaging Students in Learning – Activities and Assignments 3.17 3D Using Assessment in Instruction – Feedback to Students 3.17

Completer Survey Results The completer survey is a departmental survey that was administered for the first time in

January 2019 Survey indicators are elements from the Charlotte Danielson Framework for

Teaching3 that were selected by stakeholders; content validity for the indicators was established

using the Lawshe (1975 method) The survey was sent to completers who graduated in either

2015-2016 or 2015-2016-2017 Of the 29 completers who were contacted, 21 responded to the survey resulting

in a 72.4% response rate Thirteen completers indicated that they are currently employed full-time as

P-12 teachers Completers represented the following specialty licensure areas: Elementary

Education, Art, Physical Education and Health, Middle School Science, Middle School Math,

English, Vocal Music, Instrumental Music, and Engineering and Technology

3 The Framework for Teaching that has been adapted for the Kentucky Department of Education can be accessed at:

https://education.ky.gov/teachers/PGES/TPGES/Documents/Kentucky%20Framework%20for%20Teaching.pdf

7

Completers who are currently teaching were asked about their perceptions of their teaching effectiveness Overall, completers rated their teaching effectiveness at the acceptable level (scores of

2 or above) Scores were highest for elements regarding lesson planning and using assessments

Completers were asked to rate their own performance for each element using the following

categories:

 Exemplary (4): Meets the objective or standard with a high level of consistency

 Accomplished (3): Cleary developing the ability to meet the objective or standard with some

degree of consistency

 Developing (2): Shows some development in addressing the objective or standard

 Ineffective (1): Shows little or no evidence of development in addressing the objective or

standard

Completers’ Ratings of Perceived Teaching Effectiveness

2A Creating an Environment of Respect and Rapport – Teacher Interaction with Students 3.30 3E Demonstrating Flexibility and Responsiveness – Lesson Adjustment 3.30 1D Demonstrating Knowledge of Resource – Uses available technology to design, plan and implement

instruction that facilitates student learning 3.17 3A Communication with Students – Expectations for Learning 3.10 3C Engaging Students in Learning – Activities and Assignments 3.10 1E Designing Coherent Instruction – Learning Activities 3.08 2B Establishing a Culture for Learning – Expectations for Learning and Achievement 3.00 2C Managing Classroom Procedures – Management of Transitions 3.00 2D Managing Student Behavior – Monitoring of Student Behavior 3.00 3D Using Assessment in Instruction – Monitoring of Student Learning 3.00 3D Using Assessment in Instruction – Feedback to Students 3.00 3E Demonstrating Flexibility and Responsiveness – Response to Students 3.00 1A Demonstrating Knowledge of Content and Pedagogy – Knowledge of Content-Related Pedagogy 2.92 1F Designing Student Assessment – Congruence with Instructional Outcomes 2.92 1F Designing Student Assessment – Design of Formative Assessments 2.92 2B Establishing a Culture for Learning – Importance of the Content 2.90 2C Managing Classroom Procedures – Management of Instructional Groups 2.90 2D Managing Student Behavior – Expectations 2.90 3D Using Assessment in Instruction – Assessment Criteria 2.90

8

1F Designing Student Assessment – Criteria and Standards 2.83 2D Managing Student Behavior – Response to Student Misbehavior 2.80 1E Designing Coherent Instruction – Lesson and Unit Structure 2.75 3B Using Questioning and Discussion Techniques – Quality of Questions 2.70 3B Using Questioning and Discussion Techniques – Discussion Techniques 2.70 1C Selecting Instructional Outcomes – Suitability for Diverse Learners 2.67

9

Measure 3: Employer Satisfaction and Completer Persistence

Data from the Employer Survey demonstrate that employers perceive completers as

effective teachers and are satisfied with their preparation from the TEP Although the data show areas for improvement regarding completers’ teaching effectiveness (available in the Employer Survey data table on page 6), employers indicated that they were highly satisfied with completers’ preparation for their assigned teaching responsibilities Additionally, employers rated completers’ preparation as “much better” when compared to their peers trained at different EPPs

Employer Satisfaction Survey Items

Overall, how satisfied are you with your employee's preparation for his/her assigned responsibilities during his/her first year of teaching?

Response Values: 5 = Highly Satisfied, 4 = Satisfied, 3 = Neutral, 2 = Dissatisfied, 1 = Highly Dissatisfied

Average Rating = 4.83

How would you rate this employee's preparation compared to the preparation of other first- year teachers trained at different institutions?

Response Values: 5 = Much better, 4 = Somewhat better, 3 = About the Same, 2 = Somewhat Worse, 1 = Much Worse

Average Rating = 5.00 Note: The data reflect the input from 6 employers

Employment Milestones Based on the responses from the Completer Survey (see Measure 2 for more survey details), two out of the 13 completers who are currently employed as full-time P-12 teachers indicated a rank change from Rank III to Rank II According to the Education Professional Standards Board (EPSB) Division of Teacher Licensure and Quality4, a rank change to Rank II refers to those who have earned a “master's degree in a subject field approved by EPSB or equivalent continuing education.” Additionally, five completers indicated that they have been teaching for two years or more

4 Information on the EPSB Rank System and Change can be accessed at:

http://www.epsb.ky.gov/mod/page/view.php?id=101

10

Measure 4: Completer Satisfaction

Feedback from the completer survey demonstrates that completers are satisfied with the

quality of education they received from the TEP Completers also indicated that TEP experiences,

particularly the Student Teaching Experience, were effective in preparing them for teaching

positions Additional information regarding the survey items and sample can be found on page 6

Completer Satisfaction Survey Items

Overall, how satisfied are you with the quality of education that you received in the Teacher Education Program?

Response Values: 5 = Highly Satisfied, 4 = Satisfied, 3 = Neutral, 2 = Dissatisfied, 1 = Highly Dissatisfied

Average Rating = 4.22 Note: The data reflect the input of 18 completers who responded to this item

Please rate the effectiveness of the following experiences in preparing you for your teaching position?

*This indicator was only presented to completers who indicated that they were currently employed as full-time teachers (N = 13)

Response Values: 5 = Highly Effective, 4 = Effective, 3 = Neutral, 2 = Ineffective, 1 = Highly Ineffective

Clinical Field Experiences in Pre-Professional Terms 4.38