Tài liệu Organizational Improvement and Accountability ppt

xiii Acknowledgments ...xix CHAPTER ONE Introduction ...1 Accountability in Education...1 Other Approaches to Educational Accountability ...4 Accountability in Other Sectors ...7 Organiz

Prepared for The William and Flora Hewlett Foundation

IMPROVEMENT

AND

ACCOUNTABILITYLESSONS FOR EDUCATION FROM OTHER SECTORS

Brian Stecher and Sheila Nataraj Kirby

EDITORS

The RAND Corporation is a nonprofit research organization providing objective analysis and effective solutions that address the challenges facing the public and private sectors around the world RAND’s publications do not necessarily reflect the opinions of its research clients and sponsors.

R® is a registered trademark.

© Copyright 2004 RAND Corporation

All rights reserved No part of this book may be reproduced in any form

by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from RAND.

Published 2004 by the RAND Corporation

1700 Main Street, P.O Box 2138, Santa Monica, CA 90407-2138

1200 South Hayes Street, Arlington, VA 22202-5050

201 North Craig Street, Suite 202, Pittsburgh, PA 15213-1516

RAND URL: http://www.rand.org/

To order RAND documents or to obtain additional information, contact

Distribution Services: Telephone: (310) 451-7002;

Fax: (310) 451-6915; Email: order@rand.org

Library of Congress Cataloging-in-Publication Data

Organizational improvement and accountability : lessons for education from other sectors / Brian Stecher [et al.].

p cm.

“MG-136.”

Includes bibliographical references.

ISBN 0-8330-3500-2 (paperback)

1 Educational accountability—United States 2 School management and

organization—United States 3 Organizational effectiveness—Evaluation I Stecher, Brian M II Rand Corporation.

Educational accountability became the law of the land with the sage of the No Child Left Behind Act of 2001 As states rushed todevelop accountability systems that met the requirements of the newlaw, the Hewlett Foundation asked RAND Education to undertaketwo tasks The first was to write a paper that delineated carefully theprovisions of the new law and the underlying assumptions on whichthe new test-based accountability systems are based, and to dissemi-nate it broadly to experts and practitioners With help from a group

pas-of experts who participated in an Accountability Forum held atRAND in 2002, we produced a RAND Corporation White Paper,

Working Smarter to Leave No Child Behind: Practical Insights for School Leaders (Stecher, Hamilton, and Gonzalez, 2003).

The second task we undertook was to examine models of countability in sectors other than education The goal of the investi-gation was to understand how such models work and their applica-bility to education This monograph documents the results of thatexamination It should be of interest to educational policymakers,educational administrators, and others who are interested in improv-ing the effectiveness of schools The work was sponsored by the Wil-liam and Flora Hewlett Foundation

Preface iii

Figure and Tables xi

Summary xiii

Acknowledgments xix

CHAPTER ONE Introduction 1

Accountability in Education 1

Other Approaches to Educational Accountability 4

Accountability in Other Sectors 7

Organization of the Monograph 9

CHAPTER TWO Malcolm Baldrige National Quality Award Program 11

Background 12

The Baldrige Criteria for Performance Excellence 14

Education Criteria for Performance Excellence 16

Organizational Performance Results 17

Strategic Planning 18

Information and Analysis 18

Process Management 19

Evidence on the Baldrige Framework: Effects on Performance 19

Validating the Framework 20

Effect on Operating Performance 21

The K–12 Educational Sector Experience with the MBNQA 25

Winners of the Baldrige Award in Education 25

Chugach School District (CSD), Alaska 26

Pearl River School District (PRSD), New York 27

Other Users of the Baldrige Criteria 28

Implications for Education 29

Institutional Self-Assessment May Be Inherently Beneficial 29

The Baldrige Process Supports Accountability 30

Educators May Find It Difficult to Understand and Translate the Criteria 31

Some Caveats 32

CHAPTER THREE Toyota Production System/Lean Manufacturing 35

Overview of TPS/Lean Manufacturing 35

Focus on the Value Stream 37

Standardization 38

Worker Empowerment 39

Evidence for Effects of TPS/Lean Manufacturing on Production and Workers 40

Worker Empowerment or Worker Exploitation? 42

Applying TPS/Lean Manufacturing to Education 43

Implications for Education 45

Greater Focus on Value Streams Could Improve Educational Outcomes 45

Teaching Practices May Benefit from Standardization 47

Worker Empowerment Could Improve Educational Practices 48

TPS/Lean Production Must Be Used as a Complete System 49

Closely Translating TPS Practices to Education Is Challenging 49

CHAPTER FOUR The Job Training Partnership Act and the Workforce Investment Act 51

Overview of the Job Training Partnership Act 51

Performance Standards 52

Performance Incentives 53

How JTPA Worked in Practice 54

Measures of Performance 54

Effects on Provider Behavior 55

Overview of the Workforce Investment Act 57

Early Implementation of the WIA 58

Implications for Education 59

Explicit Performance Objectives May Produce Mixed Outcomes 59

Data Collection Alone Does Not Drive Improvement 61

Educational Systems Should Use Multiple Measures of Performance 61

The JTPA Framework Highlights the Need to Adjust for Differences in Ability 62

CHAPTER FIVE Accountability in the Legal Profession 65

The Nature of Professions 65

Overview of Accountability in the Legal Profession 68

Professional Authority 68

Controls on Entry 69

Alternative Legal Training 71

Continuous Professional Development 72

Professional Responsibility 72

Addressing and Protecting Client Concerns 73

Collective Self-Regulation 74

Lawyer Discipline System 75

Market Forces 76

Applying the Legal Model to Educational Accountability 76

Implications for Education 79

The Knowledge Base in Education is Not Yet Sufficiently Well-Regarded for Professional Accountability 79

Professional Accountability and Alternative Certification Can Coexist 81

Educators Should Foster a Sense of Public Responsibility 81

Educators Would Need Methods of Self-Policing and Self-Regulation 82

Professional Accountability Is Complemented by Market Accountability 84

CHAPTER SIX

Clinical Practice Guidelines in the Health Sector 85

Background 85

Health Care Guideline Development 86

Topic Selection 86

Identification of Decisionmaking Group 87

Gathering the Evidence 87

Decisionmaking Process 88

Dissemination of Guidelines 88

The Effect of Guidelines on Health Care Practices 89

Applying the Concept of Practice Guidelines to Education 89

Implications for Education 91

Practice Guidelines Could Address Variations in Teaching Practices 91

Guideline Development Would Be Limited by the Lack of Scientific Evidence 92

Educators Need a Common System of Classification to Develop Guidelines 93

CHAPTER SEVEN Risk Adjustment Methods in Health Care Accountability 95

Making Fair Comparisons 95

How Risk Adjustment Is Done 96

Limitations 98

Applying Risk Adjustment to Education 99

Current Uses of Risk Adjustment in Education 100

Implications for Education 103

Risk Adjustment in Education May Be Controversial 103

Risk Adjustment Requires Agreement About Outcomes and Measures 103

Identifying Risk Factors Accurately Requires Extensive Data 104

Educators Should Understand and Accept the Risk-Adjustment Model 104

CHAPTER EIGHT Health Care Report Cards and the Public Release of Data 107

Facilitating Improvement by Means of Information 107

History of Health Care Report Cards and Public Release of Data 108

How Report Cards Have Affected the Health Care Industry 111

Health Care Organizations 111

Physicians and Clinicians 112

Consumers 112

Mixed Reactions to Health Care Data Reporting 113

Public Release of Data in Education 114

Implications for Education 115

Performance Report Cards Work Best with Competition 115

Publicizing Performance Data May Have Undesired Consequences 115

CHAPTER NINE Conclusions 117

Enhancing Effective Accountability in Education 119

Broaden Performance Measures 121

Make Sure Performance Goals Are Fair 121

Develop Standards of Practice in Promising Areas and Encourage Professional Accountability 121

Develop an Integrated, Comprehensive Strategy to Help Schools and Districts Improve Their Performance 122

Final Conclusions 124

References 125

Performance-Based Accountability in Education

In December 2001, the U.S Congress approved a reauthorization ofthe Elementary and Secondary Education Act (ESEA) and renamed itthe “No Child Left Behind Act” (P.L 107-110, H.R 1) The corner-stone of the No Child Left Behind Act (NCLB) is a performance-based accountability system built around student test results Thisincreased emphasis on accountability represents an important changefrom past federal educational initiatives, which focused primarily onthe provision of services Supporters of NCLB argued that previouseducational reforms were unsuccessful in large measure because theyignored student outcomes Borrowing from successful private-sectormanagement practices, they made the case that student achievementwould only improve when educators were judged in terms of studentperformance and consequences were attached to the results

Three basic elements make up the performance-based ability system required by NCLB: goals; assessments for measuringthe attainment of goals and judging success; and consequences (re-wards or sanctions) The goals are embodied in a set of content orperformance standards that schools and teachers use to guide cur-riculum and instruction Tests are developed to measure studentlearning and determine whether students have mastered the stan-dards Improved performance on the tests leads to rewards that areintended to reinforce effective behavior; poor performance on thetests leads to sanctions and improvement efforts that are intended to

account-modify ineffective behavior Some of the incentives operate throughparents If a child’s school is deemed to be in need of improvement,parents can request a transfer to another school and/or supplementaleducational services from private providers.

As clear as these procedures may seem, the key principles derlying NCLB accountability are largely untested in education Themechanisms through which the system is intended to work to im-prove student achievement and eliminate failing schools are not wellunderstood In this environment, decisionmakers at the state, district,and school levels are looking for guidance to help them make theirsystems as effective as possible One place to look for possible insightsinto effective accountability mechanisms is outside the educationalsector The purpose of this project is to examine accountability inother fields to find lessons that might be relevant for educators

un-Accountability in Other Sectors

We cast our net widely before selecting specific instances of ability to study We solicited recommendations from educational re-searchers as well as research colleagues who study organizations inother fields We also reviewed the debate within education sur-rounding the passage of NCLB for references to accountability inother domains The final set of cases reflects our desire to present ex-amples that are relevant, interesting, and diverse Our sample includescases from both the manufacturing and service sectors In each case,

account-we tried to understand the processes through which providers areheld accountable, how well these processes have worked, and whetherthey might be applicable to education

We examined five accountability models:

• Two accountability models drawn from the manufacturing tor (although now spreading to service industries): the MalcolmBaldrige National Quality Award Program and the Toyota Pro-duction System (TPS) Strictly speaking, these are models oforganizational improvement set within the larger context of

sec-market accountability rather than full-fledged accountability tems Both, however, offer ways to improve organizational effi-ciency.

sys-• A performance incentive model used in the evaluation of jobtraining programs for the poor established by the Job TrainingPartnership Act (JTPA) of 1982 (now replaced by theWorkforce Investment Act (WIA) of 1998)

• Accountability in the legal sector The legal accountabilitymodel is largely based on notions of “professional accountabil-ity,” which entail controlling entry into the profession, manda-tory capacity-building, self-policing, and protecting client con-cerns

• Accountability in health care We explored three aspects ofhealth care accountability that seemed particularly relevant foreducation: clinical practice guidelines, use of statistical risk ad-justment methods, and the public reporting of health perform-ance measures

These models differ in terms of their comprehensiveness, tiveness, and applicability to education In this monograph, we de-scribe each model, summarize the relevant research on effectiveness,and draw specific lessons for educators

effec-Implications for Education

We recognize that the education sector has unique characteristics thatset it apart from the other sectors we examined Yet we believe theanalyses of these different accountability models offer useful insights

on ways to enhance system-wide accountability in education, cluding how to improve the operation of schools and districts toachieve higher performance Specific lessons learned for educationinclude the following:

in-Broaden performance measures Educators should be careful when

setting performance objectives because the objectives will drive havior—for better or for worse Broadening “what counts” in the sys-

be-tem is one way to diffuse the pressure to focus too narrowly and todeemphasize other important priorities.

Make sure performance goals are fair to all students and schools.

The accountability system should establish reasonable improvementtargets for all schools and should not reward or penalize schools ordistricts for factors beyond their control The goal of fair comparisonsalso needs to be balanced against the goal of closing the gap betweensuccessful and unsuccessful students Nevertheless, the experiences ofJTPA/WIA and health point out the advantages of performance tar-gets that are sensitive to initial inputs

Develop standards of practice in promising areas and encourage professional accountability Movements to create more-explicit stan-

dards of practice would foster professional accountability and provideguidance to help schools and districts improve their performance Weencourage educators to select promising areas in which more-detailedpractice guidelines might be developed Such guidelines can form thebasis for more-detailed standards for the teaching profession so teach-ers can be more aggressive about monitoring their own professionalcompetence These steps would help broaden and deepen account-ability in education

Develop an integrated, comprehensive strategy to help schools and districts improve their performance This research points to four key

elements of an improvement strategy:

• Undertaking a focused institutional self-assessment (includingasking the right questions and assembling the right kinds of in-formation)

• Understanding the school system as a linked process

• Developing and applying an expanded knowledge base about fective practice in varying situations

ef-• Empowering participants in the process (notably teachers) tocontribute to improvement efforts

Developing and adopting such a strategy in education will quire time, effort, and a willingness to adapt principles from outsidethe educational sector Pilot efforts to adapt and test these compo-

re-nents in diverse schools settings and focused efforts to create tional applications would be a good starting place to try to take ad-vantage of the successful experiences of other sectors.

educa-In Conclusion

This investigation of accountability in other sectors sharpens ourthinking about accountability in education It suggests ways in whicheducators can develop better strategies for improving the performance

of schools and districts and policymakers can redefine educationalaccountability to make it more effective It is worth pointing out that,although education has much in common with business, law, andhealth care, it faces unique challenges that other sectors do not face.Nevertheless, educators have much to learn from these other fields Inthe end, they will have to develop an accountability model that ad-dresses their unique situation However, there is much they can draw

on from accountability efforts outside of education

We are particularly grateful for the help we received from Jennifer Li,who worked with us to make the presentations consistent acrosschapters We want to thank Cynthia Cook and Paul Shekelle for theirhelpful reviews of the manuscript and their excellent suggestions ofadditional sources of information We also want to thank Jorge Ruiz-de-Velasco, Marshall Smith, and the William and Flora HewlettFoundation for their patience and support as the task grew largerthan originally anticipated

educa-a veduca-ariety of reguleduca-atory mecheduca-anisms—school buildings heduca-ad to meetstrict safety codes, teachers had to obtain formal certification, stu-dents had to study from approved textbooks, and a cornucopia ofspecific programs was mandated by state and federal governments.Schools were required to comply with these rules to ensure that stu-dents received an adequate education.

As the 21st century begins, educational accountability has taken

a different turn In December 2001, the U.S Congress approved areauthorization of the Elementary and Secondary Education Act(ESEA) and renamed it the “No Child Left Behind Act” (P.L 107-

110, H.R 1) The cornerstone of the No Child Left Behind Act(NCLB) is an emphasis on accountability based on student test re-sults Supporters of test-based accountability argue that previous re-forms failed because they focused on inputs (e.g., facilities, teachers,

textbooks) or on specific practices (e.g., remedial instruction, healthservices, school lunch programs), but never on outcomes Advocates

of NCLB further contend that educators have never been held sponsible for student learning; instead teachers and administrators arepaid (i.e., rewarded) on the basis of their educational backgroundsand their longevity in the profession As a result, they feel no personal

re-or collective responsibility fre-or how much students learn Bre-orrowingfrom successful private-sector management practices, accountabilityadvocates believe that student achievement will improve only wheneducators are judged in terms of student performance and experienceconsequences as a result

Although the No Child Left Behind act is hundreds of pageslong and the details of its implementation are quite complex, thelogic of its accountability system is quite simple The accountabilitysystem has three major components:

• Goals—explicit statements of desired student performance—toconvey clear and shared expectations for all parties

• Assessments for measuring attainment of goals and judgingsuccess

• Consequences (rewards or sanctions) to motivate administrators,teachers, and students to maximize effort and effectiveness

Figure 1.1 shows how these elements work together The goals

of the system are embodied in a set of content or performance dards that schools and teachers use to guide curriculum and instruc-tion Tests are developed to measure student learning and determinewhether students have mastered the standards Improved perform-ance on the tests leads to rewards that reinforce effective behavior;poor performance on the tests leads to sanctions and improvementefforts that modify ineffective behavior The actual NCLB rules are amore complex version of this basic model

stan-A few of the specific details of NCLB are worth mentioning cause they pervade the discussion of the reform Accountability comes

be-in the forms of be-increasbe-ing annual goals for student achievement and

escalating incentives for schools and districts based on student

achievement NCLB requires that, by 2014, all students must be

pro-ficient in reading and mathematics based on state-adopted tests

Schools and districts must make adequate yearly progress (AYP) toward

meeting these targets Furthermore, the same annual targets must bemet by students in every significant population subgroup, includingracial and ethnic groups, socioeconomic groups, and special educa-tion students Although some of the Act’s strictest sanctions applyonly to Title I schools (low-income schools that are eligible for extraresources under Title I of ESEA/NCLB), the major accountabilityprovisions of NCLB affect all the nation’s public K–12 schools, in-cluding charter schools

NCLB couples greater accountability for student performancewith increased local control and flexibility It emphasizes high-qualityteachers using scientifically based practices and expanded options forparents While the NCLB accountability system is multilevel, in-volving state policymakers, district leaders, school staff, and local par-ents, the state has the least-active role in the improvement process.Instead, the primary responsibility for improvement is assigned to thelocal level, i.e., the individual school or the district, rather than to thestate government, as was the case in the past NCLB also establishesminimum standards for teacher quality (and for the qualifications ofinstructional aides) and mandates that schools use scientifically basedpractices to promote student achievement Another important feature

of NCLB is its emphasis on the right of parents to make decisionsabout how and where their children are educated If schools are notdoing well, parents can request that their child be transferred to an-other school or be given supplemental educational services from aprivate provider To exercise their options, parents must be informedannually about the professional qualifications of their children’steachers, about the success of their school, and about the performance

of their child

Other Approaches to Educational Accountability 1

The standards-based approach embodied in NCLB is not the onlyway to hold schools accountable Several other approaches exist,sometimes simultaneously, in public education (Finn (2002); O’Day(2002); Darling-Hammond (1991); Adams and Kirst (1998)) Forexample, Adams and Kirst (1998) describe six types of educational

accountability: bureaucratic accountability, legal accountability, sional accountability, political accountability, moral accountability, and market- or choice-based accountability The accountability models are

profes-defined by the way they answer four key questions:

• Who is held accountable?

• For what are they held accountable?

• To whom are they accountable?

• What are the consequences of failing to meet the goals that areset for them?

Table 1.1 compares and contrasts four models of accountability

In the NCLB version of performance-based accountability, schoolsand districts are the units that are held accountable The districtmonitors the performance of schools and is responsible for taking _

1 Because we are interested mainly in accountability in other sectors, we provide here only a brief overview of types of educational accountability Interested readers should consult sev- eral very good articles that provide a more detailed examination of the advantages and disad- vantages of these various types of accountability in education (see Adams and Kirst, 1998; Darling-Hammond, 1991; and O’Day, 2002); and digests provided by ERIC at http://eric- web.tc.columbia.edu

To whom are they held accountable?

For what are they accountable?

What are the consequences

of failing to meet goals? Performance- or

Raising dent profi- ciency (in NCLB, it is measured by standardized tests)

stu-Increasingly severe sanc- tions (e.g., student trans- fer, supple- mental services, re- constitution) Bureaucratic

Sanctions such as loss of accreditation, firing of prin- cipals/ teachers Professional

accountability

Teachers Professional

peers/

professional organizations

Following recognized professional practices

Professional sanctions; loss

of tion Market

certifica-accountability

standards, philosophical/

religious norms, stu- dent disci- pline, other features

Loss of dents, leading

stu-to loss of revenue, eco- nomic failure

action if they are not performing adequately The state monitors theperformance of districts, and the federal government monitors theperformance of the states Schools must meet increasing targets forstudent proficiency based on standards-based test scores If schoolsfail to meet their targets, they face increasingly severe sanctions Aftertwo years, the district must provide transportation to another schoolfor students who want to leave After three years, it must provide

supplemental educational services from outside providers Ultimately,the school staff can be replaced and the school reconstituted.

Under bureaucratic accountability—the norm in the recentpast—rules and regulations specify how districts, schools, and teach-ers are to behave Various public agencies review school performanceand monitor compliance Bureaucratic accountability makes implicitassumptions that both policy and practice can be standardized, i.e.,policymakers can devise general rules and create broad program ini-tiatives that make sense for all schools, and teachers can apply generalinstructional principles that make sense for all students Under theseassumptions, it makes sense to use regulatory and inspection systems

to minimize noncompliance If districts or schools do not followregulations, they may be sanctioned, e.g., suffer loss of accreditation

or removal of administrators

Professional accountability is built on the assumption thatteachers are professionals who possess sufficient expertise to deter-mine the best ways of meeting the individual needs of their students.Thus, professional competence and standards for professional practicebecome important Professional teacher organizations have a majorrole in establishing such standards Quality is ensured through ac-creditation of teacher preparation schools, certification and licensure

of teachers, and requirements for continuous professional ment Failing to meet professional standards could result in loss ofcertification and/or disciplinary proceedings similar to those in thelegal or medical professions There is considerable debate, however,about whether the knowledge base for teacher education is developedenough to embody in explicit standards for practice and/or whetherthe current certification and licensure processes ensure the “quality”

develop-of teachers

As the name implies, market accountability uses the interactionbetween consumers (parents) and providers (schools) to regulate prac-tice and ensure quality In a market system, parents are allowed toselect the schools their children attend rather than their children be-ing assigned to schools based on where they live A variety of schemesexists for bringing market mechanisms to education, includingvouchers, charter schools, and magnet programs In voucher systems,

parents receive vouchers that they can use to purchase educationalservices (much as the federal government subsidizes food consump-tion through food stamps) In magnet or charter school programs,individual schools are freed from some regulations and are given someability to compete directly with each other for students The under-lying theory is that competition among schools will lead to higherquality Good schools will be successful; underperforming schoolswill be driven out as they lose students and revenue However, evenunder ideal conditions, choice alone is not enough to guarantee fullaccountability in terms of ensuring a quality education for all stu-dents because of issues of supply, access, and information (Gill, Tim-pane, Ross, and Brewer, 2001).

These four accountability models need not exist in isolationfrom one another, and in practice they seldom do For example, theNCLB standards-based accountability model is layered on top of ex-isting bureaucratic provisions Obviously, some of these models cancreate tensions when implemented in combination For example, pro-fessional accountability, which emphasizes the authority and knowl-edge of individual teachers, is diametrically opposed to the traditionalform of bureaucratic accountability, which attempts to minimize therole of the teacher in decisionmaking However, many authors advo-cate combining elements of several models to mitigate the negativeeffects associated with individual models For example, Finn (2002)advocates combining standards-based or test-based accountabilitywith market accountability whereas O’Day (2002) suggests that theweakness of professional accountability could be alleviated to someextent by combining it with test-based accountability In fact, NCLBitself includes some aspects of these other models For example, theschool choice provisions embody a market-based approach, and themandates for scientifically based programs and highly qualified teach-ers entail a form of professional accountability

Accountability in Other Sectors

Many of the key principles underlying NCLB—e.g., universal testing,parental choice—are largely untested in education, and the mecha-nisms by which these principles should work to improve studentachievement and eliminate failing schools are not well understood Inthis environment, decisionmakers at the state, local, and school levelsare looking for guidance to help them make their systems as effective

as possible One place to look for possible insights about effective countability mechanisms is outside the educational sector

ac-The purpose of this project was to examine accountability inother fields to see what lessons educators might learn We cast our netwidely, examining cases from both the manufacturing and service sec-tors, first to understand how providers are held accountable in thosesectors and next to examine evidence on how well these accountabil-ity processes worked Our selection process was thoughtful but notcomprehensive We solicited recommendations from educational re-searchers as well as research colleagues who study organizations inother fields We reviewed the debate within education surroundingthe passage of NCLB for references to accountability in other do-mains The final set of cases reflects our desire to present examplesthat are relevant, interesting, and diverse The reader will have tojudge whether these objectives were met

The five “cases” we investigated were the Malcolm Baldrige tional Quality Award Program, the Toyota Production System(TPS)/Lean Manufacturing, the Job Training Partnership Act (JTPA)

Na-of 1982 and its successor the Workforce Investment Act (WIA), thelegal profession, and selected aspects of the U.S health care system.The Baldrige Awards and the Toyota Production System are easilyclassified as market-based accountability systems; the connection be-tween consumers and providers is clear and direct Firms and provid-ers producing goods and services in a competitive environment arelikely to be successful only to the extent that they satisfy the needs ofconsumers In the JTPA, performance-based incentives are used toincrease efficiency by tying payments to direct measures of output.Both the legal and medical cases involve markets to an extent, but

they are markets in which consumers cannot judge quality as easily.

As a result, both domains have seen the rise of intermediaries or fessional organizations that attempt to provide information on qual-ity This brings aspects of professional accountability to bothdomains The health care case is also relevant to performance-basedaccountability, because the health profession is far ahead of education

pro-in defpro-inpro-ing and measurpro-ing standards of practice Spro-ince education haselements of market-based accountability, performance-based account-ability, and professional accountability, we believe these cases mayoffer useful insights for educators

Organization of the Monograph

Chapters Two and Three discuss two accountability models drawnfrom the manufacturing sector (although they are now spreading toservice industries): the Malcolm Baldrige National Quality AwardProgram and the Toyota Production System/Lean Manufacturing.Strictly speaking, these are models of organizational improvement setwithin the larger context of market accountability, not full-fledgedaccountability systems; both offer a way to improve organizationalefficiency Chapter Four describes the experience of JTPA and WIA,which set performance-based goals and incentives for centers provid-ing employment-related training It offers important lessons for theperformance incentive provisions of the NCLB Chapter Five pro-vides an overview of how professional accountability operates in thelegal profession Chapters Six through Eight look at aspects of ac-countability in the health care system We explore three aspects ofhealth care accountability that seem particularly relevant for educa-tion: clinical practice guidelines, use of risk adjustment models, andthe public reporting of health performance measures and its impact

on providers and consumers

These five models differ widely both in terms of hensiveness and in their applicability to education As a result, thechapters vary in depth and detail, but all attempt to draw implica-tions for the education sector We hope they help educators think

compre-about a number of specific issues, including how to make their goalsand expectations clearer, how to use data as a basis for improvement,how to make use of multiple measures, how to adjust for the hetero-geneity of inputs, and how to establish standards for practice We willreturn to these themes in the final summary chapter

Program

Sheila Nataraj Kirby

The goal of the Malcolm Baldrige National Quality Act of 1987(Public Law 100-107) is to establish criteria for performance excel-lence and to provide organizations a framework for designing, im-plementing, and assessing a process for managing all business opera-tions to be able to meet those criteria Given that many schools anddistricts are struggling to improve themselves and the performance oftheir students, it seemed a useful exercise to examine the MalcolmBaldrige National Quality Award (MBNQA) to see what lessons itmight hold for education This chapter describes the MBNQA Pro-gram, its criteria for performance excellence, and the evidence thatexists regarding the link between implementation of the MBNQAframework and operating performance We then discuss how thesecriteria have been applied to the educational sector and the experi-ences of two school districts that were recent winners of the award.The last section provides some implications for the use of MBNQAwithin the NCLB educational accountability context

We should note that there are several quality awards—for ple, the Deming Prize in Japan, the European Quality Award (EQA),and the Australian Quality Award (AQA)—each of which is based on

exam-a perceived model of totexam-al quexam-ality mexam-anexam-agement Although there exam-aresome differences among the quality awards, they provide a universalaudit framework for evaluating management practice, quality ofmethods, techniques, and tools, deployment of quality plans, and re-

sults (Ghobadian and Woo, 1996).1 We have neither the expertisenor the resources to evaluate these different awards or frameworks tosee which one is the “best.” Our decision to focus on the Baldrigeaward was therefore based not on evaluative criteria but on our judg-ment that, because it is widely accepted in the United States (not only

in manufacturing or business), it seemed an appropriate avenue toexplore for lessons that might be applicable to the educational sector

Background

Since the 1980s, Total Quality Management (TQM) has emerged asone of the most significant and pervasive developments in U.S busi-ness practice Powell (1995), citing Ross (1993), describes TQM as:

An integrated management philosophy and set of practices that emphasizes, among other things, continuous improvement, meeting customers’ requirements, reducing rework, long-range thinking, increased employee involvement and teamwork, proc- ess redesign, competitive benchmarking, team-based problem- solving, constant measurement of results, and closer relation- ships with suppliers (p 16)

Easton and Jarrell (1998) point out that the focus on TQM gan in 1980 primarily in manufacturing companies that were facingstrong global competition from Japan TQM was widely creditedwith leading the revolution in industry that led to Japan’s rise toglobal prominence in the postwar years (see Powell, 1995) The U.S.TQM movement gained momentum in 1987 with the establishment _

be-1 Ghobadian and Woo (1996) provide an excellent comparison of the characteristics of these four major quality awards, pointing out, for example, that—unlike the other three—the Deming Prize is not based on an underlying framework linking concepts and practices to results Thus, it does not assume causality but is more prescriptive in that it recommends a list of desirable quality-oriented “best practices,” such as quality circles and standardization The MBNQA, EQA, and AQA are based on an underlying causal framework linking differ- ent constituents of quality management and are prescriptive in the sense that they expound a particular philosophy of good management However, they do not recommend particular methods or tools.

of the Baldrige Award program by Congress “to recognize U.S nizations for their achievements in quality and performance and toraise awareness about the importance of quality and perfor-mance excellence as a competitive edge” (see http://www.nist.gov/public_affairs/factsheet/baldfaqs.html) The award recognizes per-formance excellence in each of five following categories: manufactur-ing, service, small business, and, starting in 1999, education andhealth care Up to three awards may be given in each category eachyear, although in some areas and some years, no awards are given ifapplicants are judged as not meeting standards The award is notgiven for a specific product or service but for meeting the Baldrigecriteria for performance excellence.

orga-The U.S Commerce Department’s National Institute of dards and Technology (NIST) manages the MBNQA program withassistance from the American Society for Quality, a professional non-profit association The award program is a joint government/private-sector effort Private-sector and state and local organizations havecontributed over $100 million, including $10 million raised by pri-vate industry to help launch the program, and the time and efforts ofhundreds of largely private-sector volunteers The volunteers serve asmembers of the Board of Examiners to review applications, make sitevisits, and make recommendations regarding awards The boardcomprises more than 300 experts from industry, educational institu-tions, governments at all levels, and nonprofit organizations, who gothrough a training process to become Baldrige examiners The federalgovernment provides about $5 million annually to NIST to managethe program Application fees are charged to cover expenses associ-ated with distribution and review of applications and development offeedback reports The application fees range from $5,000 for largeorganizations to $500 for nonprofit education institutions

Stan-Organizations that wish to apply submit an Eligibility nation Package to establish eligibility in one of the five award catego-ries Once they are determined to be eligible, they submit a com-pleted application form along with an application report consisting of

Determi-an orgDetermi-anizational overview Determi-and responses to the Criteria for ance Excellence All applications go through an independent review

Perform-by examiners; some are selected for consensus review Perform-by a panel ofjudges (stage 1) Based on the consensus review, some are selected for

a site visit review (stage 2) Stage 3 consists of the site visits, review,and recommendations for winners All applicants receive a detailedfeedback report regardless of what stage they reach in the process.The feedback report is a written assessment of the organization’sstrengths and vulnerabilities and contains detailed, actionable com-ments on opportunities for improvement NIST estimates that appli-cants receive approximately 300 hours of feedback each from expertexaminers while organizations that are chosen for a site visit receiveover 1,000 hours of review.2 Several winners of the award havepraised the quality and usefulness of the feedback.3

Another important emphasis of the program is dissemination.Recipients of the award are asked to participate in the annual confer-ence at which the awards are announced, and several of them havecosponsored regional conferences They are also expected to share ba-sic materials on their organizations’ performance strategies and meth-ods and to answer news media inquiries

The Baldrige criteria focus on results and continuous improvementand provide a framework for designing, implementing, and assessing

a process for managing all business operations The Baldrige criteriaare used by thousands of organizations of all kinds for self-assessmentand training and as a tool to develop performance and business proc-esses The MBNQA is generally regarded as the most prestigiousquality award in the United States, and many states have used the _

2 See http://www.nist.gov/public_affairs/factsheet/baldfaqs.html.

3 See http://www.baldrige.nist.gov/Why_Apply.htm.

4 This section draws heavily from http://www.quality.nist.gov/PDF_files/2003_Business_ Criteria pdf Unless otherwise noted, all quotations are taken from this reference.

Baldrige criteria to establish their own quality awards (Przasnyski andTai, 2002).

The Baldrige criteria are built upon a set of interrelated corevalues and concepts Consistent with its emphasis on “continuousimprovement,” the core concepts and framework are continuouslyevolving over time As of 2002, the core values and concepts includedvisionary leadership; customer-driven excellence; organizational andpersonal learning; valuing employees and partners; agility; focus onthe future; managing for innovation; management by fact; social re-sponsibility; focus on results and creating value; and systems perspec-tive

These values and concepts provide a foundation for “integratingkey requirements within a results-oriented framework that creates abasis for action and feedback,” and are embodied in seven criteria thatform the basis for organizational self-assessments, for making awards,and for giving feedback to applicants:

Leadership. How senior executives guide the organization andhow the organization addresses its responsibilities to the public andpractices good citizenship

Strategic planning. How the organization sets strategic tions and determines key action plans

direc-Customer and market focus. How the organization determinesrequirements and expectations of customers and markets

Information and analysis. How the organization manages, uses,and analyzes data and information to support key organizationalprocesses and the organization’s performance management system

Human resource focus. How the organization enables itsworkforce to develop its full potential and how the workforce isaligned with the organization’s objectives

Process management. How key production and delivery andsupport processes are designed, managed, and improved

Business results. How the organization performs and improves

in its key business areas: customer satisfaction, financial and place performance, human resources, supplier and partner perform-ance, and operational performance The category also examines howthe organization performs relative to competitors

market-The framework clearly defines the relationship among these teria as a causal one and emphasizes the central relationship betweenleadership and business results It also emphasizes that this relation-ship is two-sided to indicate the importance of feedback in an effec-tive management system.

cri-Applicants provide information about their efforts in each of theseven areas listed above The areas are weighted to get a final scoreand, because of the emphasis on demonstrated performance, thegreatest weight is given to organizational performance results (450out of 1000 points)

In 1999, the Baldrige award program was extended to the educationand health sectors This expansion assumes that the same seven-partframework that underlies the business criteria is adaptable to all orga-nizations, but it recognizes that the guidelines need some adaptation

to fit these new sectors Thus, for education, “customer and marketfocus” translate into “student, stakeholder, and market focus,” “hu-man resource focus” into “faculty and staff focus,” and “business re-sults” into “organizational performance results.” The underlying be-lief is that using the same framework for all sectors of the economyfosters cross-sector learning and sharing of information on best prac-tices

Since 1999, 47 applications have been submitted in the tion category.6 Any for-profit or not-for-profit public or private orga-nization that provides educational services in the United States or itsterritories is eligible to apply for the award That includes elementaryand secondary schools and school districts; colleges, universities, and _

educa-5 This section draws heavily from www.quality.nist.gov/PDF_files/2003_Education_ Criteria pdf Unless otherwise noted, all quotations are taken from this reference.

6 It is not clear whether these are 47 distinct educational organizations applying for the award or 47 applications from some smaller number of organizations applying multiple times Typically, firms apply multiple times before they are successful.

university systems; schools or colleges within a university; professionalschools; community colleges; technical schools; and charter schools.The education criteria are designed to help organizations use anintegrated approach to organizational performance management totry to improve education quality in terms of delivery of ever-improving value to students and stakeholders, improvement of overallorganizational effectiveness and capabilities, and increased organiza-tional and personal learning.

Four of the education criteria are particularly interesting becausethey have special relevance to the current accountability systems ineducation In particular, the emphases on organizational performanceresults, strategic planning, information and analysis, and processmanagement fit in well with the current emphases in education onmeasurable progress against goals; school improvement planning;data-driven decisionmaking; and aligning instruction with standards,goals, and diverse learning styles as characteristics of high-performingschools

Organizational Performance Results

For education, the Baldrige criterion of excellence in organizationalperformance translates into “value-added” demonstrated perform-ance, as measured by (1) annual improvement in key measures of per-formance, especially student learning and (2) demonstrated leadership

in performance and performance improvement relative to comparableorganizations and/or benchmarks These measures are similar to whatNCLB has required states to establish in terms of adequate yearlyprogress measures that schools must meet each year—measures thatinclude goals for student achievement and progress, both overall andfor groups of students disaggregated by various characteristics (gen-der, race/ethnicity, English language learners, migrant status, povertystatus, and disability status)

However, the Baldrige criteria are broader than those established

by NCLB For example, the organizational performance areas arestudent learning (150 points); student- and stakeholder-focused re-sults (60 points); budgetary, financial, and market results (60 points);faculty and staff results (60 points); and organizational effectiveness

results (60 points) This composite of indicators is intended to ensurethat strategies are balanced—that they do not inappropriately maketrade-offs among important stakeholders, objectives, or short- andlonger-term goals—an important lesson education accountability sys-tems must take into account.

The rationale behind using a “value-added” concept of lence is that it (1) places the major focus on teaching and learningstrategies; (2) poses similar types of challenges for all organizationsregardless of resources and incoming students’ preparation and abili-ties; (3) is most likely to stimulate learning-related research and tooffer a means to disseminate the results of such research; and (4)offers the potential to create an expanding body of knowledge of suc-cessful teaching and learning practices in the widest range of organiza-tions

excel-Strategic Planning

This criterion examines how the organization develops strategic jectives and action plans, how they are deployed, and how progress ismeasured For example, strategy development looks at how the orga-nization prepares for the future, what kinds of projections and op-tions it uses to envision the future, what kinds of data it uses, how itbalances short-term and long-term objectives, and how it developsstrategies to address key challenges identified in the organizationalprofile In addition, organizations are asked to describe how theyconvert their objectives into action plans, including allocation of re-sources and measures to track progress, and to project their progress

ob-on key performance measures This is similar to calls for strategicplanning as the basis for improving the school Key to strategic plan-ning is the organization’s approach to measurement, analysis, andknowledge management, discussed below

Information and Analysis

This criterion examines how the organization selects, gathers, andanalyzes data and manages and improves its knowledge assets It ech-oes calls for principals and teachers to build continuous feedback intotheir school improvement efforts A key component of this criterion

is gathering comparative data on similar organizations and marks (defined as identifying processes and results that represent bestpractices and performance for similar organizations) Other key com-ponents are making timely, reliable, and accurate information avail-able to staff, students, and stakeholders and sharing best practicesamong faculty, staff, and stakeholders.

bench-Process Management

Process management refers to the ways in which the organizationidentifies and manages its key processes for creating student andstakeholder value and maximizing student learning and success Thus,for example, the organization is asked what the key learning-centeredprocesses that deliver the educational programs are; how the require-ments for these processes are determined; how faculty and staff areproperly prepared to deliver these processes; how individual differ-ences in student learning rates and styles are accommodated; hownew technology is incorporated; how sequencing and linkage amongthe educational offerings are addressed; how key performance meas-ures are used to control and improve these processes to ensure studentsuccess; and how support processes (such as finance, facilities, infor-mation services, and human resources) are used to support the learn-ing-centered processes Obviously, schools’ answers to these questionsshow how they would translate their goals and planning into results

Evidence on the Baldrige Framework: Effects on

implement-ologies: The first examines the linkages between the various criteriathat make up the framework and assesses the direct or indirect causalrelationships among them; the second merely uses subsets of firmsthat either implemented the Baldrige framework or won the Baldrigeaward and examines whether these firms had better operating per-formance than comparison groups of firms that did not implementthe framework These two approaches to validity are clearly related;however, because they are so different, we discuss them separatelybelow.

Validating the Framework

Winn and Cameron (1998) administered a 190-item survey data toall permanent noninstructional staff members of a large Midwesternuniversity to investigate the validity of the casual relationships un-derlying the MBNQA framework, in particular the direct relationshipassumed to exist between leadership, system dimensions, and out-comes Factor analysis revealed that the seven categories were reliableand appropriate However, confirmatory path analysis did not vali-date all the relationships in the framework For example, Winn andCameron found that, with few exceptions, leaders did not appear tohave a direct impact on organizational outcomes Rather, their influ-ence was felt through the systems and processes they had established.They hypothesize that this may be more typical of higher educationthan for-profit firms, because of a professionalized workforce and lowlevels of hierarchical control They offer the following quality im-provement plan for educational institutions: (1) put in place effectiveleadership committed to change; (2) gather information; (3) use thisinformation to guide strategic planning; (4) based on the strategicplan, design a human resource management system and organiza-tional processes focused on quality In their model, having effectiveand efficient organizational processes leads to quality outcomes.Pannirselvam and Ferguson (2001) used data from the ArizonaGovernor’s Quality Award (which is based on the MBNQA) to studythe relationships between the Baldrige categories Unlike Winn andCameron (1998), their results confirmed the validity of the Baldrigeframework and suggest this might be due to differences in the sample