11.10.16 Final Chickasaw County IPFS Evaluation Plan

The project coordinator and evaluator will work to ensure the identified youth are included in implementation activities and in quantitative and qualitative data collection.. For qualita

Iowa Partnerships for Success

Evaluation Plan Template

Chickasaw County

Revised: October 26, 2016 Approved: November 10, 2016

If you have questions about how to complete the document or the evaluation planning process, please email Cheryl Davidson, IPFS Evaluator at cheryl.davidson@iowa.gov or call 515-281-6929.

Once you have completed all the questions, please submit this document

tocheryl.davidson@iowa.gov The deadline is July 1, 2016.

Evaluation Plan Template Part 1: Evaluation Summary

A Audience

i What groups or stakeholders will be sought for input in the evaluation process?

Input will be sought from key stakeholders and groups in Chickasaw County including: Chickasaw Connections coalition members, schools, teachers, youth, parents, law enforcement, juvenile court services, city and county elected officials, park oversight committees, grocery/convenience store retailers, county attorney, community event organizers, behavioral health, healthcare, and faith-based

organizations.

ii What, if any, efforts will be in place to ensure that minorities or special

populations are represented?

U.S Census demographic data from the County Assessment Workbook will help to assess the county’s youth population and identify youth being served in the project When possible, the coalition will aim to reach as broad and diverse a group

as possible

In the strategic plan for the project, youth who are not involved in

extracurricular activities and females between the ages of 16 and 18 from New Hampton Schools were identified as being disproportionately impacted by underage drinking The project coordinator and evaluator will work to ensure the identified youth are included in implementation activities and in quantitative and qualitative data collection Whenever possible, a representative approach will be used when administering quantitative evaluation assessments in order to reach the entirety of the targeted youth in Chickasaw County For instance, if a school survey is

administered to evaluate the effectiveness of the media campaign, all schools will be asked to invite all students in the targeted age range to participate For qualitative data collection, such as focus groups, the project coordinator and evaluator will recruit a range of participants to mirror the demographics of Chickasaw County If needed, minority or special population focus groups will be recruited to augment findings from focus groups reflecting a broad swath of the Chickasaw population

iii What kinds of information do you anticipate stakeholders will be interested in knowing about your project?

 Underage drinking and binge drinking rates

 Accident and Arrest trends

 Substance abuse treatment trends

 Number of youth involved in extracurricular activities and

extracurricular activity suspensions

 Iowa Youth Survey trends

 Changes in youth's attitudes and beliefs around alcohol

 Community readiness

 Fidelity of strategy implementation

 Adaptations made to strategies and the impact of adaptations

 Strengths and weakness of the implementation and delivery of the

strategies

 Persons/groups who received the services/interventions

 Costs of implementation and project delivery

 Effectiveness of the strategies

 Potential sustainability of strategies in terms of effectiveness, cost,

community support, etc.

 Strategies that resulted in the most challenges

iv How will you share data and reports with your coalition and stakeholders?

Reports on aggregated data will be prepared by the IPFS project coordinator and evaluator and shared at coalition meetings Members will split into groups and answer questions to help them analyze the data This information will be used to prepare handouts/case statements to share results with stakeholders at meetings and during speaking engagements Handouts and case statements will also be

available on the coalition website The project coordinator will provide additional summarized findings to the coalition and stakeholders upon request and

availability

B Data Methods

i List all the methods that you plan to use to collect data on your strategies (e.g secondary data, survey, focus group, interviews, pre-post- tests, and any other methods).

Secondary data, meeting minutes/notes, surveys, focus groups, interviews, pre-post tests, administrative data, and observational data will be used to collect data on strategies.

a If using secondary data, how will you obtain the data and from what sources?

Data will be obtained from the following secondary data sources:

 Iowa Youth Survey: IPFS staff will work with the Chickasaw County schools to ensure youth participate in the survey County-level data

from the survey will be accessed online, and school-specific data will

be requested from school leaders

 County Assessment Workbook tables: This information will be

maintained by the coalition assessment committee with assistance from IDPH.

 Information on substance abuse treatment and data from the Justice Data Warehouse will be accessed

 Administrative data from law enforcement: The two municipal police departments in the County and one sheriff's department will be asked

to provide information through their charge analysis reports and the party patrol and bar walkthrough forms

 Records from media outlets: Media outlets will assist in identifying the number of youth reached by the media campaign.

b If conducting surveys, how will you solicit participants and approximately how many will be surveyed? What information will you seek from them? How will you store the completed surveys? How do you plan to tabulate the results?

Schools will assist in conducting surveys on how many youth are reached by the IDPH "What Do You Throw Away" Media Campaign The project coordinator will request permission from the school

administrators to send a web link survey invitation to students via email

It is anticipated that all 1300 students in 6th-12th grade in Chickasaw County public and parochial schools will be invited to participate

Students will be asked if they've seen or heard of the campaign and, if so, where Students will take the survey during the school day to try to boost the response rate The surveys will be tabulated and stored electronically, which will allow for better security and easier tabulation of the results.

c If conducting focus groups, what groups will be represented and how many participants will be recruited? What information will you seek from them? Who will conduct the focus group and where will it take place? How do you plan to tabulate the results?

Youth between 12-20 years of age will be recruited for focus groups The purpose of these focus groups is to gain insight on youth’s perceptions of the consequences of and reasons for underage drinking The

assessment/planning committee will arrange the focus group(s) to take place

at an accessible location(s) Notes will be taken during the focus group(s) and recordings will potentially be transcribed and coded to identify emerging

concepts Transcriptions will not contain information that could personally identify the respondents The results will be stored electronically

d If using pre-post-tests, what information will you collect and how? Will data be available to you through the individual strategy curriculum? If not, what process will you use to collect the data? How will you store the completed tests? How do you plan to tabulate the results?

We will use pre-post-tests for the All Stars strategy that will measure student's perceptions of the risks/harms of underage alcohol use The

curriculum includes a pre- and post-test that the IPFS project coordinator and All Stars facilitator will deliver to the students during the course Data collection, analysis, and compiling of results will be provided by Brilliance Analytics as part of the student materials to be purchased Storage of

completed tests, datasets, and results will be decided following discussions with Brilliance Analytics.

e If using another method, please explain how the data will be collected.

Interviews with various stakeholders such as retailers and youth aged 18 to 20 will be arranged on an individual basis by

Assessment/Planning Committee members This will be done in order to get qualitative data on the progress in implementing the strategies Interviewer notes will be used to identify themes from the interviews about awareness, knowledge, and perceptions of the strategies Results will be stored

electronically and personally identifying information will not be included in the original notes or electronic copies Other program data collected on an ongoing basis will be stored in paper form and kept in a locked office or electronically in the Chickasaw County Assessment Workbook on a password protected laptop

Assessments and environmental scans will be conducted by coalition members, project staff, and volunteers throughout the project to collect data

on alcohol advertisements in convenience/grocery stores and alcohol use in public places within the county

Lastly, every All Stars facilitator will be required to complete a fidelity monitoring log form with information on the number of sessions, parent involvement, and activity completion

ii How do you plan to protect the confidentiality of the people you collect

information from?

The Center for Social and Behavioral Research (CSBR) at the University of Northern Iowa is assisting with the evaluation of the IPFS grant and will provide guidance on best practices for protecting the confidentiality of participants in the evaluation CSBR will review all forms, surveys, pre-post tests, and interview questions to ensure confidentiality is protected CSBR will also provide advice on how to de-identify the data for analysis and reporting

Participants will be given informed consent in which they can choose to

participate in surveys, interviews, and focus groups or opt out Signed consent will be obtained prior to recording any sessions Survey participants’ names will not be stored with their responses The original data files will be stored in a secure, locked office, and raw survey responses will be kept on a secure

password protected laptop

The All Stars program has received above average Quality of Research

Ratings for reliability of the measures in its assessments (for more information, see: http://vfhy.org/sites/default/files/pdf/All_Stars_NREPP.pdf).

For other data collection activities, pre-existing questionnaires and based instruments will be used, when possible, that have demonstrated

evidence-reliability In the event that new instruments are needed, the materials will be reviewed by the assessment committee to ensure the items are appropriate Pre- testing of new questions will also be conducted with youth in the target

population, when needed CSBR will advise on questionnaire design and testing strategies before quantitative and qualitative data collection begins When possible, questions will be derived from items known to have internal consistency for measuring a specific concept or set of concepts Finally, to reduce inter-observer reliability concerns, the same observer/interviewer will be used for qualitative data collection when possible

pre-ii Are the data sources valid (they measure what you want them to measure)? Why or why not?

The secondary data sources are strong in external validity because the

research designs have been constructed to be generalized to populations across the state of Iowa The data collected as part of the other evaluation activities,

such as the focus group notes, will have limited external validity because of the lack of random selection of participants and the qualitative nature on the data collection However, in instances where a census approach is used, such as the school survey, the effectiveness of the strategies may be generalized, with

specified limitations, to the targeted subpopulations in Chickasaw County

The assessments that are part of the All Stars Curriculum have received average scores for internal validity as rated by Quality of Research (for more information, see: http://vfhy.org/sites/default/files/pdf/All_Stars_NREPP.pdf) Questionnaires and assessment tools used will be derived from previously used tools that have been tested locally and in communities that have

successfully implemented these strategies in the past.

New instruments developed for the evaluation activities will be reviewed by the assessment committee to ensure the instruments have face validity The project coordinator will review the instruments for content validity, and both the project coordinator and CSBR will seek out established measures whenever possible to ensure criterion validity As resources allow, pre-testing will also be conducted in part to ensure construct validity

iii How will you ensure that the data you collect are representative of your county’s IPFS priority population (12-20 year olds)?

The data sources used (survey, interviews, focus group, secondary sources,

or administrative data) will focus on sampling the 12-20 year old age group when possible In-depth interviews and secondary data will provide data on underage drinking from the harder-to-reach 18 to 20-year-old population

working in the county or Chickasaw residents attending college outside of the county Some data on 18 to 20 year olds will be available through law

enforcement, including minor in possession of alcohol citations and fines

Demographic information collected from participants during the evaluation will

be compared to secondary data, primarily the American Community Survey, to ensure adequate representation of the priority population

D Analysis

a How will you use the data to help you find answers about your project? Both quantitative and qualitative data will be collected for the outcome evaluation All strategies will include some indicators to measure changes over time Baseline data collected prior to the start of the strategies will be compared

to post-activity assessments to determine the extent to which the strategies

were effective For example, the All Stars school curriculum incorporates pre- and post-assessments to measure changes after the intervention Also, there are plans to conduct a school survey targeting both students and parents before the start of several strategies and on a yearly basis thereafter to help assess

awareness, knowledge, attitude change, and impact

Table 1D shows the strategies and indicators the coalition will collect data on to show change over time Interviews will be conducted near the end of the project

to get an idea about what changes community members and key stakeholders feel have been most successful and have the greatest potential for sustainability Other measures of effectiveness will include any changes observed in

community policy, changes in the environment in terms of the promotions or

counter-advertisements youth are exposed to, and changes in the attitudes and beliefs youth have about drinking

Table 1D

Alcohol Use in Public

Places Number of policies before implementation and at the end of the project Underage Drinking Media

Campaign Number of youth who report seeing or hearing the media campaign in the first year of implementation and

at end of implementation Apply Appropriate

Penalties to Minors in

Possession (MIP)

The percentage of youth reporting it would be easy or very easy to get alcohol in their community in the 2014 Iowa Youth Survey compared to the percentage of youth who report easy/very easy alcohol access in the

2018 Iowa Youth Survey All Stars Changes in student's attitudes, beliefs, and behaviors

before beginning All Stars compared to after they have taken the program.

Alcohol Advertisements in

Public Places The number of alcohol advertising policies before implementation and at the end of the project

b How will you use the data to identify what changes/modifications are

needed?

The indicators listed under the “Monitoring Your PFS Process and Activities” section of Part 2: Data Reporting Plan will comprise the main elements of the

process evaluation The project coordinator, with assistance from CSBR, will

review the data collected and make recommendations to the

Assessment/Planning committee for changes or modifications to the project

activities and strategies Before making changes/modifications, the findings will

be shared with the parties and committees implementing the given strategy and action steps to ensure they are following fidelity guidelines and core

components will be addressed

All changes or modifications will be informed by the indicators planned for the outcome evaluation The outcome indicators include: 1) measurement of the perception of risk/harm of alcohol use according to pre- and post-test surveys administered by the All Stars program; 2) the number of alcohol retailers in Chickasaw county with new or revised policies for alcohol advertisements; 3) the average number of impressions among the target population reached by the media campaign; 4) number of new or revised policies for alcohol use or sales in parks; and 5) proportion of students in the Iowa Youth Survey who report the ease or difficulty in obtaining alcohol.

Part 2: Data Reporting Plan

Indicator A Method

From what source do youplan to collectthe data?

B

Measurement What question

or measure will be used to capture this indicator?

C Target PopulationWhat group will you be collectinginformation from?

D ReportingHow often will the data

be available for

collection?

E

Timeframe What period

of time will the data you report cover?

F Data LimitationsAre there any problems with the data source? Will the data be a limited measure?

G NamePerson or group responsible for

collecting the data

Monitoring Your PFS Process and Activities

What was the content of the discussions regarding sustainability?

What plans were made to sustain _

strategy?

Coalition, subcommittees, and strategy stakeholders Ongoing Annually No anticipated limitations/problems IPFS CoordinatorModifications

or adaptations

for cultural

groups

Minutes and notes from coalition and subcommittee meetings

What modifications oradaptations were made to accommodate for cultural groups and disparate populations?

What was the impact of the adaptations?

Where is drinking

Coalition, subcommittees, and strategy stakeholders

Ongoing Quarterly No anticipated

limitations/problems IPFS Coordinator

happening and who is drinking?

(Is there a difference among cultural groups?)Costs to

implement and

deliver

Price quotes and invoices

What did it cost

to implement and deliver strategy

Sub-contractors, media venues, curriculum companies, and IPFS coordinator Ongoing Annually No anticipated limitations/problems

IPFS Coordinator and

SupervisorResources

generated Attendance

forms, newly recruited coalition members, donation/in-kind log

Donated materials, time, mileage, meeting space, food,

print/copying, advertising, and funds

Coalition, subcommittees, and strategy stakeholders Ongoing Annually No anticipated limitations/problems

IPFS Coordinator, Coalition SecretaryMedia coverage

of IPFS

initiative

Media venues' data on reach New Hampton Tribune, Nashua Reporter, Fredericksbur

g Review, Facebook, Twitter, Chickasaw Connections Blog,95.1 the Bull Radio

What was the reach, venues used, when were printed, recorded, and posted (online) products released?

Counts on the number of articles (electronic and print), recorded ads, photos of media pieces General population Ongoing Monthly No anticipated limitations/problems IPFS Coordinator Individuals and

organizations

who received

IPFS services

Focus groups, surveys, interviews, preand post tests

Who was reached by the activity/

intervention?

Was knowledge

12 to 20-year-oldsand anyone who received IPFS Services

Ongoing Monthly No anticipated

limitations/problems IPFS Coordinator

and Evaluator

increased? Wereskills

developed? Did behavior change?

Was the experience positive?

Barriers and

successes

Capacity checklist, minutes and notes from coalition and subcommittee meetings

What were barriers to success? What successes happened? Whatwent well? How can things be improved? Wereoutcomes achieved?

Were strategies implemented with fidelity?

How?/Why not?

Coalition and subcommittee members and strategy stakeholders Ongoing Quarterly

No anticipated limitations/

problems

IPFS Coordinator and

Evaluator

Indicator A Method

From what source do youplan to collectthe data?

B Measurement What question or measurewill be used to capture thisindicator?

C Target PopulationWhat groupwill you be collecting informationfrom?

D

ReportingHow oftenwill the data be available for collection?

E

TimeframeWhat period of time will the data you reportcover?

F Data LimitationsAre there any problems withthe data source? Will the data be a limited measure?

G NamePerson or group responsible for collecting the data

Strategy Outcomes - Best Practices for Alcohol Use in Public Places

#

alcohol-free signs in n/a: The group does not n/a n/a n/a n/a n/a n/a