SUPERVISOR CORE MANAGING FOR RESULTS - Trainer’s Guide -

The trainee will be able to develop a plan to improve the delivery of child welfare services in his or her agency through monitoring outcomes with the CFSR process.. Supervisors will beg

county-between the federal and state-enhanced measures).

K3. The trainee will be able to explain how to navigate the Child Welfare Dynamic Report System website of the UC Berkeley Center for Social Services Research, and understand that all data on the C-CFSR quarterly, county-specific reports (aswell as much more detailed information) can be located there

K4. The trainee will be able to explain the interrelated nature of the outcome

measures for safety, permanency, and well-being that are tracked on the

quarterly county-specific outcome reports

S1. The trainee will be able to demonstrate the ability to ask policy and practice

questions related to trends identified in the C-CFSR outcome measures

S2. The trainee will be able to develop a plan to improve the delivery of child

welfare services in his or her agency through monitoring outcomes with the CFSR process

C-Values

V1. The trainee will value the need to supervise and monitor in ways to achieve

ASFA outcomes

V2. The trainee will value ongoing recognition of trends identified in the outcome

measures that enhance culturally sensitive and responsive practice by helping target services appropriately to ethnic, age, and gender groups

V3. The trainee will value incorporating various practice issues into supervision:

A Outcomes vs process

B Fairness and equity

C Incorporating data

3 Background and Concepts for Outcomes in Child Welfare

(Trainer's Slides PowerPoint [PPT] Slide Show)

The child welfare system has traditionally not done a very good job of monitoring and evaluating its performance

Slide 2

Child welfare policy and practices need to be guided by clear and specific goals, and progress toward those goals requires good performance data Yet—in spite of the large quantity of information often collected about children in out-of-home care—child welfare agencies are often unable to provide quick and reliable responses to questions posed by policymakers, administrators, and the public

ACTIVITY: Ask trainees to call out reactions

they have to the idea of using data in their work, reasons that they may not like data, etc Use flip chart to record answers.

Sentiment among child welfare agency staff is that, while they may be inundated by data, this information is not useful for planning, evaluation,

or other decisions Their experience has often been that data systems used by the agency were designed for other purposes and cannot provide answers to questions that are pertinent to their practice Further, they likely feel that entering data into CWS/CMS is a waste of time and takes away valuable time that they could be doing actual casework This attitude is largely due to a legacy of state data systems like CWS/CMS as being “roach motels” of data—workers enter information in, but virtually nothing ever comes out Or, when data does come out, it is not

Slide 4 (continued) presented in a way that is easily accessible or

useful

A first step, then, in overcoming this aversion to data is to make the connection for workers between the information they input into CWS/CMS about their cases, and how successful their agency is helping its clients To do so requires analyzing data that has been put into CWS/CMS and sharing it—on an ongoing basis

—in a format that workers can understand Whenworkers hear the word “data,” they immediately think of “MATH” or “STATISTICS” and can be intimidated or turned off A common reaction is

“I hate math! I became a social worker so that I could help people—not work with numbers!” Another likely reaction is, “I've been a social worker for 25 years, and I know very well what area most needs work in our agency—it’s placement disruptions (or reunifying kids more quickly, or cutting down on the number of kids reentering care, etc.).” These typical responses are not unreasonable—

given that workers have probably never seen examples of how data can be useful

If supervisors can be shown that information drawn regularly from CWS/CMS is a form of feedback on the impact of their efforts (as evidenced by performance on important goals such as preventing recurrence of maltreatment, orachieving reunification for children, etc.), their distrust of the usefulness of data will decrease It

is important to recognize that each worker’s practice experience provides a very real and validperspective of their agency’s areas of success and failure This validation can be used to point out

to supervisors that each of them is actually a data person whether they realize it or not That is, their experience is their data source from which they draw conclusions and make decisions about what areas have greater or lesser importance

Slide 4 (continued)

Extrapolating from each worker's experience with

their specific caseload to the information on all

workers’ caseloads that is captured in CWS/CMS,provides a more comprehensive picture of where

a child welfare agency needs to focus its efforts Clearly, improvements could be made in all parts

of the child welfare system—but limited time andresources demand that efforts be made in the areas of greatest need As a supervisor, it is critical to know where your attention would best

be focused—and to keep on top of trends becausethe area of focus changes over time In the ER unit—are referrals and/or substantiations going

up or down? For which age or ethnic groups? Are these trends reflected in entries to care? In the permanency unit—are reunifications increasing or not? If they are, is our re-entry rate the same or are many of the kids we send home later coming back into care? Drawing on information in CWS/CMS will help supervisors

to identify these trends and keep track of progresstoward addressing them Regularly updating staff

in unit meetings—and celebrating any progress orsuccess—can positively reinforce and motivate staff

Fancy statistics are not necessary Simple line or pie charts can easily point out those areas where the agency is succeeding best and where it most needs to focus its efforts to help families

Supervisors will begin to understand that access

to timely data and the ability to analyze this information can help agency staff to identify and respond to changing population needs—and enable workers to monitor the results of their work Overcoming skepticism regarding the usefulness of data is therefore a crucial first step toward better outcomes—and thus for improved lives of children and families

Ask for ways that might make using data and

outcomes measures more accessible for supervisors and workers Record suggestions on flip chart If training staff from only one or two counties, ask them to tell you general trends in child welfare in their county (e.g., how many children enter care each year, how long do children generally stay in care, what proportion

of children reunify after 12 months in care, etc.) Record these answers and check them against actual trends for the counties during the live demo of the UC Berkeley website Then underscore the necessity to make these efforts to

be aware of key trends, keep staff informed about areas of need, and to track progress, because… monitoring child welfare agency performance is

no longer simply a good idea—it's the law

Accountability System

The Children and Family Services Review (CFSR) is an ongoing nationwide process conducted by the U.S Department of Health and Human Services The first stage of the CFSR examined service delivery by child protection, foster care, adoption, family preservation, family support, and independent living programs in all states

The CFSR examines seven outcomes for children and families as well as seven systemic factors States were measured against national standards During the first round of the CFSR, no state achieved substantial conformity on all measures Thus, all states are implementing a Program Improvement Plan (PIP) The CFSR reviews all states periodically to follow progress and may impose fiscal penalties on those states that fail to improve

In 2003, California did not achieve conformity onany of the seven outcomes, and passed only one

Slide 5 (continued)

of seven systemic factors The state faced $18 million in penalties if it had failed to meet federal requirements by 2005 In January 2008, the federal government levied a nearly $9 million penalty against the state for failing to meet the outcome goals regarding re-entry and placement stability Loss of this funding during times of fiscal hardship will likely result in loss of services

to children and families

Assembly Bill 636, the Child Welfare Outcomes and Accountability Act of 2001, was developed as

a means to carry out California's PIP The bill provides a legislative framework for monitoring and assessing county performance to ensure the safety, health and well-being of the state’s children Since the passing of this legislation, California was instrumental in influencing the federal government to review and subsequently revise their federal outcome measures in 2007 Although many of the revised federal measures mirror the previously enhanced California measures, there are still a few vital outcome areasthat California chooses to track in the absence of federal mandates Those measures include the

“Timely Response” of social workers regarding Immediate and 10-day referrals and “Timely Social Worker Visits.” Once again, California is leading the way in the creation of well-being measures for children who are in foster care As

of June 2009, the measures for health, dental, andeducational outcomes for children in care were under development

Each county has created a self-assessment that identifies strengths and areas for improvement, aswell as a system improvement plan (SIP) to outline actions needed to improve county performance with respect to the outcomes

Key Concepts to Understand Child Welfare

Before examining the specific outcomes tracked under The California Child Welfare Outcomes and Accountability System, it is important to know the background about the reasons (i.e., sound methodological basis) why many of the measures were chosen.

Point in time estimates are biased They represent children who have the worst experiences in care—that is, those who tend to stay in care for long periods Slide #6 illustrates this phenomenon: the x-axis represents a timeline from January 1, 2007, up toJanuary 1, 2009, and the 10 numbered horizontal lines represent the duration in care of 10 children in foster care A point in time snapshot taken of the caseload on January 1, 2008, will only observe about half the children (line numbers 1, 2, 3, 5, 8 and 10 from the top) The other children will not be detected

over-by this snapshot Therefore, any mean or median measure of length of time spent in care will be biased

by the children observed in the snapshot who have longer lengths of stay Placement stability estimates would be similarly biased Using snapshot data, staffmay not recognize the opportunity to effectively focus resources by assuming that most children in thesystem are poorly served

1 This discussion adapted from: Webster, D., Needell, B., & Wildfire, J (2002) Data are your friends:

Child welfare agency self-evaluation in Los Angeles County with the Family to Family initiative Children

and Youth Services Review, 24(6/7), 471-484.

Slide 6 (continued) ACTIVITY: Referring to PPT slide #6, ask trainees

to call out how many children would comprise the calendar year 2007 entry cohort (Answer: 7) Then ask trainees to call out how many children would comprise the calendar year 2008 entry cohort (Answer: 3).

Fun Exercise with Medians and Means

“How long do children stay in care in your county?”

is an important question that people often ask The following brief exercise illustrates the need to reflect

on the approach taken to answer the question about length of stay.

ACTIVITY: Prior to class, trainer cuts out rows

(make more as needed) from the following table and distributes one row each to student at the beginning

of this exercise (The trainee given the “Bill Gates” row is asked to keep his/her identity secret).

UCB Ph.D., Consultant Roughly minimum wage

(let’s say, $10 K)Administrative Assistant $20 K

Administrative Assistant $20 K

Social Worker II $50 KSocial Worker II $50 K

$3 Billion

Ask students (except for Bill Gates) to state their

Slide 6 (continued) work title and income, and then to line themselves up

from highest to lowest income The instructor points out that the line up is another way to talk about a

“DISTRIBUTION” and asks the class to identify the MEDIAN income from the lineup The class will see that the median income is $50 thousand.

The instructor then asks Bill Gates to identify himself and to join the lineup, and asks the class to identify the median income now that a new person has been added The class will see that the median income will still be $50 thousand Thus the students will see that that the median income is a reasonable

description of the middle point of the incomes—even with an extreme observation.

Finally, the instructor will ask Bill Gates to step away, and for the class to identify the MEAN income from the lineup minus Bill Gates The students will find that that the average or mean income is about

$46 thousand When Bill Gates rejoins the group and the class is asked to re-compute the mean income, the average income will be over $250 million—a poor description of the individuals’ incomes (i.e., much too high for everyone except Bill Gates, for whom the description is much too low) The instructor can use the exercise to point out how medians are a much more reasonable concept to use when talking about measures (particularly length of stay) that may contain extreme, “outlier” data points.

The Cycle of Experiences in the Child Welfare System

As a last word before the break, it is important to underscore the interrelated nature of interpreting outcome data This is a main reason why the federal

“standards” are inherently misleading—i.e., if one

“achieves conformity” on a given measure, that is considered success However this “success” may adversely affect other outcomes and therefore cannot

be viewed in isolation.

Trainee Handout # 2

Slide 7 (continued)

(Handout #2)

There is no “gold standard” for determining success

on any outcome Any particular measure, viewed in isolation, tells you nothing useful about county performance

Data tells a story, and the relationship between outcomes must be taken into account when evaluating performance For example, while we mayseek to reduce the number of children entering care

—if/when we succeed in doing so, we must be aware

of how the smaller population entering care will likely be more difficult to work with (i.e., we have excluded more families from oversight by the agency

—but those families that do require intervention havemore challenging circumstances) Thus a reduction

in entries may affect other measures such as time to permanency or placement stability

The supervisor needs to understand, therefore, that monitoring and drawing conclusions about any givenoutcome must be done with respect to the

relationships between outcomes (Other potential relationships—reduction in entries could lead to higher recurrence of maltreatment; increase in reunification could lead to increase in reentry; increase in guardianship or adoption could lead to decrease in reunification.) Also, other factors outside

of child welfare, such as an economic downturn, etc.,may lead to an increase in entries even in spite of best practice

By being aware of the interrelated nature of outcomes, the supervisor can make a more realistic assessment of the effect their work is having on their clients They will not be overly pessimistic, for example, when the number of children reunified during a period of time drops—if they are also monitoring entries to care and note that a smaller number of children with greater needs is coming into care

By the same token, they will not prematurely think

Slide 7 (continued)

that they don’t need to continue efforts to improve reunification services just because an increasing number of children are going home from the agency They will also know to also track the proportion of those reunifying who later return to care—to ensure that the apparent, successful reunification events are not being followed by an unintended negative outcome (re-entry)

Finally, the supervisor will be able to use this understanding of relationships between the outcomes

to more clearly, forcefully, and accurately “tell the story” of how well children and families are being helped—whether the supervisor is motivating staff in

a unit meeting, communicating needs for practice change to a child welfare director, or being

“accountable” to relevant members of the community(e.g., judges, child advocates, legislators, board of supervisors) A couple of initiatives in California thatfocus on using outcome data:

Family to Family, an initiative developed by the

Annie E Casey Foundation and also supported by the Stuart Foundation, is being implemented in 22 counties that represent 84 % of the children in foster care The four core components are: community partnerships; recruitment, training and support of foster parents; team decision-making; and self-evaluation Self-evaluation, the use of data, allows community partners to focus on neighborhoods with the highest concentration of CPS referrals and entries

into foster care Using Family to Family strategies,

children who must be removed from the home can beplaced with foster families in their neighborhoods

http://www.aecf.org/initiatives/familytofamily

California Permanency for Youth Project, a

three-year program supported by the Stuart Foundation andthe Walter S Johnson Foundation, is dedicated to ensuring that all youth leave the child welfare systemwith a permanent lifelong connection with a caring adult The project collaborates with the courts, grouphomes, foster families, adoption agencies and the child welfare system to help them achieve

permanency for foster youth in Alameda, Monterey, San Mateo and Stanislaus counties The project also

works to improve public policy, increase training and

raise public awareness on the issue It tracks its

success by utilizing data on older youth in foster careand on exits from care http://www.cpyp.org

Summarize points supporting why data is important for supervisors:

 It is critical to know where your limited time

would best be focused—information on all

workers’ caseloads captured in CWS/CMS provides a more comprehensive picture of where a child welfare agency is performing well and where it needs improvement

 Regularly updating staff in unit meetings—and celebrating any progress or success—can positively reinforce and motivate workers

 Regular use of data and understanding relationships between the outcomes “tells the story” of how well children and families are being helped—whether supervisors are motivating staff in a unit meeting, communicating needs for policy and practice change to a child welfare director, or being

“accountable” to relevant members of the community (e.g., judges, child advocates, legislators, board of supervisors)

Break