USING ORGANIZATIONAL CHANGE SURVEY (OCS) TO EVALUATE CYFAR EFFECTIVENESS IN ADDRESSING COMMUNITY NEEDS LESSONS FROM A FOUR YEAR STUDY IN ILLINOIS

Introduction The objective of the CYFAR Program is to improve state extension system’s capacity to support community-based programs for children, youth and families at risk.. In the area

USING ORGANIZATIONAL CHANGE SURVEY (OCS) TO EVALUATE CYFAR EFFECTIVENESS IN ADDRESSING COMMUNITY NEEDS: LESSONS FROM A FOUR

YEAR STUDY IN ILLINOIS

Al Zwilling1 and Andre N Mbassa2 University of Illinois Extension1 and University of Illinois2

Documenting the current state of and changes in states’ abilities to develop and sustain effective programs for targeted groups is paramount to the successful implementation of many community projects This presentation provides guidelines to help CYFAR projects understand how to assess organizational change overtime and how to utilize findings in making program modifications throughout the course of a CYFAR project The University of Illinois Extension utilized the Organizational Change Survey (OCS) to establish general system baseline for measuring changes of the organization and changes in the communities as part of its state strengthening process Cross sectional primary data was collected from U of I Extension staff working with youth, families and communities, two local community advisory groups and task forces of one of the advisory groups over a four year period Initial findings indicated that this instrument can be sensitive to organizational and community change The instrument was used to establish a baseline for measuring Extension’s commitment to developing and implementing the project’s common vision and strategic plan and assess the progress over time There appears to be a link between offering staff development on specific topics (mini grant program) and organizational change Increased participation in training did not always reflect a change in the organization Rather, the quality of training and its relevance to the staff needs contributed to an increase in programming Project support and motivation in using communication technologies and internet resources enhanced staff effectiveness in conducting programming as well as extending their expertise to other community needs

Introduction

The objective of the CYFAR Program is to improve state extension system’s capacity to support community-based programs for children, youth and families at risk Over a four year period the University of Illinois Extension utilized the Organizational Change Survey (OCS) developed at the University of Arizona as a measure to evaluate and improve its community programs The goal of CYFAR at the state level is to “build capacity among vulnerable youth, families, communities and extension staff to pragmatically and effectively address problems related to youth in risky environments” (Peeples, Zwilling, Wiley & Spelke, 2000) At the community level the project has specific emphasis toward at risk issues while building staff development and capacity to address these issues The Knox County Healthy Youth Alliance (HYA) and Task Forces (TF) work towards reducing teen pregnancy, youth crime, school dropout and teen use of alcohol, tobacco and other drugs The HYA goals were for Knox County schools, communities and families to implement a comprehensive community based prevention education strategy that included: utilization

of comprehensive health curriculums for grades K-12 in the schools; community training in understanding youth development, risk and protective

factors, and effective community planning and evaluation The Prep Plus Advisory Committee (PPAC) in St Clair County was designed to provide a series of intensive personal and professional development opportunities for pregnant and parenting teens who have demonstrated their desire to achieve The second goal of this group was to develop the skills of group members in identifying and analyzing community issues, strategic planning and proposal writing

Research on organizational change identifies several organizational indicators of readiness This section identifies these indicators as another way to examine if and how they have contributed to organizational change Change can vary in complexity from the introduction of relatively simple processes into small workgroups to transforming an entire organization Cummings and Worley (2005) organize the activities of managing change into five categories These categories are motivating change, creating a vision, developing political support, managing the transition, and sustaining the momentum Napier et al (1998) described eleven work principles for creating the capacity for change within institutions of higher education These work principles include leadership imperative, compelling futures, process by design, data gathering as a creative act, team learning, trust and relationships, changing frames of reference, large group intervention, action learning, governance for action and linking change and resource allocation As one examines these indicators of readiness, one can find their presence within the groups being discussed

Research findings show that OCS instrument can be sensitive to organizational and community change Results indicate that over time, Extension staff at the state and local communities had a clear vision of the task assigned, and the training and support provided to each of the groups

is contributing towards achieving the goals of the organization It is worth noting that community coalitions’ vision and strategic plan for CYFAR initiative was clear to its members at the start of the project while CYFAR vision evolved over a longer period of time for Extension staff at the state level In the area of training, community coalition members received more training on programs for at risk audiences compared to Extension staff at the state level The Technology Mini Grant process engaged over 122 Extension staff However, the level of utilization of CYFAR focused resources such as CYFERnet remained low for all groups indicating a need for providing incentives for participants in using this source of information

Methods The Survey

• The Organizational Change Survey comprises six components that cover topics ranging from implementation of common vision to training opportunities, collaboration within and outside of Extension, diversity of Extension staff, program sustainability and respondent characteristics

• The survey was administered to Extension staff from across the state once in 2001, 2002, 2003, and 2005

• The survey was adapted and administered to both Healthy Youth Alliance and Task Force members in Knox County at four points in time and

to Prep Plus Advisory Committee group in St Clair County over a period of two years

• The OCS instrument consisted of two types of questions (multiple choice and text open end questions) and two scales (rating scales and agreement scales)

• The 5-point Likert scale ranging from 5: “Strongly Disagree” and 5- “Strongly Agree”, a 5-point agreement scale ranging from 5:

1-“Too Little” and 5- 1-“Too Much” and a rating scale ranging from 1-4: 1- “Poor” and 4- “Excellent” were used

Data Collection and Analysis

• Cross sectional primary data was collected from U of I Extension staff, HYA, TF and PPAC members

• On-line and paper versions of the instrument were used to collect data

• On-line data were recorded in a Microsoft Excel spread sheet and imported into SPSS statistical software

• Paper versions of the instrument were mailed to participants from the two communities and responses were returned to the research team on campus

• Paper survey was manually coded and data entered directly into the Data Entry program of SPSS

• SPSS statistical software was used for data analysis

• Data analysis included frequency distributions, descriptive statistics, and crosstabulations

Results and Discussions

• Extension Commitment And Staff Time Assignment To CYFAR Programming

Figure 1 presents the data regarding proportion of assignment devoted to children at risk Statistics show that at the state level over the four years of study, more respondents reported working zero to quarter time with at risk audiences With the exception of the first year participants, less than 40% of respondents devoted 1/2 to full time working with children at risk The trend was somewhat reversed at the community level In Knox County, over 50% of the HYA and TF participants devoted 1/2 to full time to programming for CYFAR Similar trend was observed with the Prep

Plus Advisory Committee in St Clair County where all committee members in year 4 devoted 1/2 to full time working with at risk audiences compare to less than 50% in year 1

In assessing the state vision for CYFAR program, less than 30% or respondents in year 1 and 2 respectively agreed or strongly agreed that the state extension had articulated a clear vision for CYFAR program (Figure 2) Apparently, the vision was still evolving in the first two years and with the introduction of the University of Illinois Extension Strategic Agenda in 2003, the percentage of respondent agreeing or strongly agreeing more than double in the third and fourth year The Strategic Agenda identified and communicated to all extension staff a limited set of priorities including catering to at risk audiences in the state that Extension can pursue successfully to improve its programming, strengthen its organization, and make it more agile to pursue engagement in a rapidly changing environment At the community level, an even greater percentage of respondents (40% for HYA and TF) agreed or strongly agreed in the first year that their coalition had a clear vision and strategic plan for carrying out programming for at risk audiences This percentage increased rapidly to 100% the third and fourth year respectively Similar trend was observed in St Clair County

• Assessment of Participants Knowledge and Expertise in Delivering CYFAR Programming

Respondents assessed their knowledge and practice relative to principles of positive youth development, risk and resiliency factors, developing, implementing and evaluating programs, and policy and legislation as it affects youth at risk Table 1, 2 and 3 summarize the findings for all three organizations Less than 50% of all respondents at the state level rated themselves good or excellent on all six topics under evaluation, with the rating for knowledge in risk and resiliency posting the lowest score (10%) in the first year These scores greatly improved the second and third year for all six topics and remain substantially high the fourth year for principles of youth development, risk and resiliency factor and program evaluation This improvement in knowledge and expertise is the results of a series of trainings delivered at CYFAR conferences and other seminars in these topics The goals of the first statewide conference were to increase the participants understanding of and appreciation for the importance of working with diverse audiences and establishing U of I Extension’s strategic plan for working with children, youth and families in risky environments The goals of the second year conference were to increase the knowledge of the research that supports youth, family and community programs, recognize the importance of designing comprehensive programs for youth, examine previous and current CYFAR programs in practice and how to evaluate community programs No conference the third year, only the implementation of the Technology Mini Grant process The fourth year conference goals were to improve internal cooperation between teams and become aware of positive youth development collaboration and resiliency research The final year conference goals were to review the goals of the state strengthening process and see how the goals were achieved, increase participant understanding of longitudinal evaluation and select models of effective programming and evaluation for use in participant’s

communities In addition to the annual Illinois Conferences, CYFAR offered a number of statewide professional development opportunities for Extension staff These topics included risk and resilience, helping youth increase their understanding of diversity and sexual health issues, teen pregnancy prevention and utilizing electronic tools for program development, implementation and evaluation In Knox county HYA project, 57% of participant rated themselves good or excellent in principles of youth development and knowledge of risk and resiliency factor in the first year (Table 2) The survey showed that participants rating improved over time for three areas only namely principles of youth development, risk and resiliency and programming for youth at risk Less than 50% of all participants rated themselves good or better in the remaining three areas evaluated Similar trend was observed in St Clair County where more than 50% of participants involved with Prep Plus project rated themselves good or excellent in principles of youth development, risk and resiliency and programming for youth at risk (Table 3)

• Capacity Building Through Staff Development and Training

Tables 4, 5 and 6 present data on extension staff participation in ten training topics At the state level, between 28% and 76% of respondents reported training participation, with half or more taking advantage of six of the ten topics in the first year (Table 4) Participants at the training were asked to rate the sufficiency of these trainings from 1=too little, 3=adequate, to 5=too much The mean ratings in the first year ranged from 2.11 to 2.79 Subsequent year’s results showed a remarkable improvement in training participation and rating, indicating that participants were interested in training and needed more as they progress in their work with at risk audiences Also, Extension perceived the need and provided necessary assistance

to enhance the organization’s performance More than half of respondents were trained in all ten topics in the second year, in eight topics the third year and nine topics the fourth year respectively The mean rating also improved ranging between 2.12 to 3.09 the second year, 2.39 to 3.05 the third year and 2.24 to 3.00 the fourth year showing Extension’s commitment to providing adequate staff development A similar trend was observed in Knox County as shown by the results compiled and presented in Table 5 While more than half of participants attended training in seven or more topics throughout the life of the project, their rating of training was even better with a mean sufficiency rating of 3 or higher for five topics in year 3 and four topics in year 4 respectively In St Clair County, the majority of PPAC participants who received some form of training where trained in four or less topics (Table 6) Over a two year period, project participants at various trainings posted a mean sufficiency rating of 3 for training on research-based information on youth growth, development and comprehensive health only

• Access to Internet, Financial and Other Technical Resources

Technology affects changes and the internet has virtually transformed society The OCS survey assessed Extension professionals’ use of internet for work relevant to children, youth, and families at-risk by measuring the frequency with which respondents accessed the CYFERnet web

page and other websites, and the reasons for using internet technology Respondents reported using both CYFERnet and other websites with varying frequencies A greater proportion of participants used non-Extension websites and other state and national Extension websites compare to using CYFERnet, despite the various trainings and awareness campaigns in the value of using CYFERnet for programming The low usage of CYFERnet suggests that Illinois staff may need more training and encouragement to use this powerful tool

The most common ways the internet is used throughout the four years of study are to locate background information for a news article or presentation, locate information or fact sheets to share with program staff and/or participants, and to locate information to develop new programs or modify existing ones (Table 7) Although 71% of respondents in year 1 acknowledged using internet services to communicate with a colleague within the state, this percentage dropped substantially the following years Less than half of Knox County participants in year 1 and 2 used internet service

to accomplish the seven tasks listed in Table 8 However there was a substantial improvement in year 3 and year 4 where more than 50% of them reported using internet services for all tasks except participating in an online tutorial on interactive learning experience In St Clair County, internet resources were used at time period one mostly to locate background information for a news article or presentation (67% of respondents said yes), and

to locate information or facts sheets to share with program staff and/or participants (67%) Whereas at time 2, Table 9 shows that internet services were utilized to communicate with colleagues within the state (83%), communicate with a colleague in another state (67%), locate information to develop new programs or modify existing programs (67), and to locate information for news article or presentation

Figure 3 presents responses to the question: Our state extension service has allocated financial resources specifically for work in CYFAR The charts show that in year 1 an year 2, less than 20% of participants agreed to receiving financial resources to carry out programs for at risk children However this percentage rose dramatically to 50% in year 3 and 59% in year 4 This may stem from the fact that during year 3, CYFAR offered mini-grants for interdisciplinary groups to utilize technology in the development and delivery of programs to reach youth and families in risky environments This process seemed to have engaged the largest number of Illinois Extension staff of any of the CYFAR interventions over the course

of the project Over 122 different individuals were named in the proposals This process proved to be successful, in that all groups completed their technology projects, shared their accomplishments at the 2004 statewide conference, and have sustained their projects

Conclusions

• Sharing a common vision and strategic plan is crucial in designing activities to accomplish a goal While knowledge of the vision grew, after four years some participants were still unsure or disagreed that Illinois has a clear vision for CYFAR Extension should provide more information to its personnel on the state vision as it fits into the national agenda

• Training in areas relevant to CYFAR programming will ensure the success of the program Extension services should emphasize the need and encourage staff involved with at risk audiences to participate at seminars and workshops For instance, training on project evaluation should

be promoted since knowledge in evaluation guidelines could assist participants in focusing their programs on appropriate activities

• CYFERnet is a valuable internet resource with useful information on children, youth and families at risk that Extension staff at state and community levels should utilize Extension should devise a means to promote the use of this source of information One could integrate the CYFERnet resource into all professional development for youth, family and community staff offered through U of I of Extension

References

Cummings, T G., and Worley, C G (1993) Organization Development and Change (6th ed.) Cincinnati, OH: South-Western College Publishing Napier, R., Sidle, C C., Sanaghan, P., and Reed, W S (1998) Metamorphosis: Creating the Capacity for Change, National Association of College

and University Business Officers, January, pp 18-28

Peterson, D.J., Roebuck, J.C, and Betts, S.C (2003) Iowa Cooperative Extension’s Capacity to Support Programs for Children, Youth and Families

at Risk: Results of the Third Round of the Organizational Change Survey Arizona Youth, Family and Communities The University of Arizona Tucson, Arizona

Peeples, G.G., Zwilling, A., Wiley, A., and Spelke, K (2000) Building Extension Capacity and Adding Strengths in Local Communities in Illinois

State Strengthening proposal funded by USDA Urbana, IL: University of Illinois

Table 1: Sate Staff Rating of Knowledge and Expertise Relative to Working with At-Risk Youth (%)

Good/Excel Fair Poor Good/Excel Fair Poor Good/Excel Fair Poor Good/Excel Fair Poor

Principles of positive youth

development and

comprehensive health

Risk and resiliency factor 10 64 26 56 18 26 100 0 0 65 33 2

Programming for youth at risk 26 52 22 79 21 0 99 1 0 76 22 2

Obtaining resources and future

funds to support programs for

youth at risk

Evaluating social programs for

youth at risk

Policy and legislation as it

affects youth at risk

Table 2: Rating of Knowledge and Expertise Relative to Working with At-Risk Youth (Knox HYA/TF)

Topic Year 1 Year 2 Year 3 Year 4

Good/Excel Fair Poor Good/Excel Fair Poor Good/Excel Fair Poor Good/Exce

l

Fair Poor

Principles of positive youth

development and

comprehensive health

Risk and resiliency factor 57 43 0 57 39 4 83 17 0 92 8 0

Programming for youth at

risk

Obtaining resources and

future funds to support

programs for youth at risk

Evaluating social programs

for youth at risk

Policy and legislation as it

affects youth at risk

Table 3: PPAC Rating of Knowledge and Expertise Relative to Working with At-Risk Youth (%)

Good/Excel Fair Poor Good/Excel Fair Poor