Measuring Customer Satisfaction

1. Introduction This Guide is designed to help vocational rehabilitation (VR) programs serving American Indians/Alaska Natives collect customer satisfaction

Measuring Customer Satisfaction

A Resource Guide for Vocational Rehabilitation Programs Serving

American Indians and Alaska Natives

Walter Hillabrant, Ph.D

Judy Earp

Funded by:

American Indian Rehabilitation Research and Training Center

Northern Arizona UniversityInstitute for Human DevelopmentArizona University Center on Disabilities

Flagstaff, AZ

Measuring Customer Satisfaction

A Resource Guide for Vocational Rehabilitation Programs Serving

American Indians and Alaska Natives

April 2003

Walter Hillabrant, Ph.D

Judy Earp Support Services International, Inc

Suite 506

8609 Second AvenueSilver Spring, MD 20910

Funded by:

American Indian Rehabilitation Research and Training Center

Northern Arizona UniversityInstitute for Human DevelopmentArizona University Center on Disabilities

Flagstaff, AZ

We wish to express our gratitude and appreciation to Dr Catherine Marshall andPriscilla Sanderson for the guidance and feedback provided during the development ofthis Guide Appreciation is also extended to Dr Rick Roessler for his editorial feedbackand guidance

This manual is dedicated to the staff of Vocational Rehabilitation Programs and to thecustomers they serve

Measuring Customer Satisfaction

1 Introduction

This Guide is designed to help vocational rehabilitation (VR) programs serving AmericanIndians/Alaska Natives collect customer satisfaction data in culturally sensitive ways While thisGuide is written for VR program staff, it is intended to be useful to anyone interested in VR andrelated topics So, when we say “you” in the Guide, we are referring to VR program staff but, byextension, we mean any reader

Throughout this Guide, we refer to “customer” satisfaction Sometimes we, and other sources, refer to “consumer” or “client” satisfaction Generally, customer, consumer or client satisfaction

can be used interchangeably; however, we think it may be important for program staff to think

about your clients as “customers”—in many situations, dissatisfied customers can and do seek

products and services from a different provider Thus, providers are (or should be) motivated tomake and keep their customers satisfied with the services and products provided The goal of thisGuide is to help you keep your customers satisfied with your program’s services, activities, andproducts.1

Currently, there is little information about VR customer satisfaction for American Indians/AlaskaNatives at the national or local levels.2 In general, we lack information about the VR services,activities, and facilities American Indians/Alaska Natives find to be most and least useful andsatisfying, and we lack information on how customer satisfaction can be improved You can usethis Guide to help you and others collect and report customer satisfaction data, and to use thedata to improve the level of satisfaction of your customers

While VR program staff may have little available time, resources, or systems in place formeasuring customer satisfaction, such measurements must be made to demonstrate thatAmerican Indians/Alaska Natives are receiving needed VR services, and to determine howsatisfied they are with these services The 1998 Amendments (PL 102-569) to the RehabilitationAct mandate that federally funded programs providing VR services determine and reportcustomer satisfaction with those services, and address progress made in the inequitable treatment

1 Discussion of customer satisfaction can beg the question: who is the customer? The most frequent answer may refer to the persons being served by the (VR) program—the clients However, another, provocative answer has been suggested—our ultimate customers are employers—persons and organizations that employ our clients Viewing employers as customers of VR programs implies that measuring customer satisfaction might include employers of

VR program clients as well as the clients.

2 White, M., Williams-Joseph, D., & Gahungu, A (2002) An Examination of the Relationship Between Consumer

Satisfaction and Employment Outcomes for Rehabilitated and Non-Rehabilitated American Indians and Alaska Natives, Final Report Flagstaff: Northern Arizona University, Institute for Human Development, Arizona

University Center on Disabilities, American Indian Rehabilitation Research and Training Center

of minorities by VR programs In addition, the Government Performance and Reporting Act(GPRA) requires three categories of information from VR projects: 1) the number of customersserved under an Individual Plan of Employment (IPE), 2) the number of customers served whohave a successful employment outcome, and 3) the number of cases closed without a successfulemployment outcome

What is customer satisfaction? In this Guide, customer satisfaction refers to the way a

customer feels about the VR program on scales that range from very satisfied to very dissatisfied.

In addition to overall satisfaction with the program, customer satisfaction can be

measured with respect to specific program services, activities or other

aspects of the program such as job placement, skills training, program

staff, facilities and equipment A customer may have varying levels of

satisfaction with different aspects of the program For example, a

customer may be very satisfied with the services provided by the

program, very dissatisfied with the program’s physical facilities and

environment, and moderately satisfied with his/her employment

placement

Customer satisfaction can change over time At one point in time, for example early in anintervention or early stage of a program, the customer might be mildly dissatisfied with certainaspects of the program At a later point in time, the customer might be very satisfied, or very

dissatisfied with the same aspects of the program These changes in thecustomer’s level of satisfaction might be the result of greater experiencewith these program components; or the changes may be associated with areevaluation of the original experiences resulting in a new understanding

or appreciation of the original experiences and the context of thoseexperiences Changes in the level of customer satisfaction, and thed

ifferent reasons or explanation of such changes suggest that 1) the timing of measurement isimportant, and 2) measuring and interpreting customer satisfaction can be challenging

Customer satisfaction and program evaluation Program evaluation may involve discovering

the extent to which a program meets its goals and objectives, finding out if a program works aswas planned, and/or learning how satisfied its customers are Thus, customer satisfaction is onlyone aspect or component of program evaluation It is possible for many customers to be verysatisfied with most program components but the program might fail to meet some of its mostimportant objectives and visa versa—the program might achieve many of its key objectives, butmany of its customers might be very dissatisfied with some program components By focusing

on customer satisfaction in this Guide, we are not saying that it is the only important part of

program evaluation or the most important part; however, other aspects of program evaluation arediscussed elsewhere.3

This Guide was developed under a grant awarded to the American Indian RehabilitationResearch and Training Center (AIRRTC) at Northern Arizona University in Flagstaff, Arizona.The Guide is divided into six sections, separated by Tabs, to give the user easy access to aspecific section or topic The sections include:

Tab 1: Background/History

Tab 2: WHAT is evaluation?

Tab 3: WHY evaluate?

Tab 4: HOW to evaluate?

Tab 5: How to use the customer satisfaction data

Tab 6: Example of a customer satisfaction survey instrument

3 See, for example, Davis, J.D., Erickson, J.S., Johnson, S.R., Marshall, C.A., Running Wolf, P., & Santiago, R.L

(Eds) (2002) Work Group on American Indian Research and Program Evaluation Methodology (AIRPEM),

Symposium on Research and Evaluation Methodology: Lifespan Issues Related to American Indians/Alaska Natives with Disabilities Flagstaff: Northern Arizona University, Institute for Human Development, Arizona University

Center on Disabilities, American Indian Rehabilitation Research and Training Center

Section 1: Background/History

American Indians/Alaska Natives have the highest rates of disability among the U.S population,and these rates are increasing faster than that of the overall population (McNeil, 1997) Also,American Indians/Alaska Natives have lower rates of successful rehabilitation closures than therest of the U.S population The low rehabilitation rates contribute to the high unemploymentrates of American Indians/Alaska Natives and represent lost opportunities to enhance quality oflife

The federal government is seeking ways to address disparate disability and rehabilitation ratesfor American Indians/Alaska Natives and other minorities For example, the Rehabilitation ActAmendments of 1998 (PL 102-569) require that federally funded VR programs addressinequitable treatment of minorities, and mandate that the programs submit annual reportsdescribing progress toward remediation of disparate treatment of minority clients Additionally,the Amendments mandate that federally funded VR programs conduct customer satisfactionsurveys to identify areas of vocational rehabilitation that need to be expanded, improved, ormodified State VR programs do serve American Indians/Alaska Natives; in fact, sometimesstate VR programs and tribal VR programs share the responsibility for client interventions.However, there have been conflicts in the past about the manner in which some state VRprograms met the needs of American Indian/Alaska Native customers and provided services inculturally appropriate ways

The American Indian Vocational Rehabilitation Services (AIVRS) program officially began withthe 1978 reauthorization of the Rehabilitation Act of 1973 In 1981, the first tribal VR grant wasmade by the Rehabilitation Services Administration (RSA) in the Department of Education, tothe Navajo Vocational Rehabilitation Program By 1986, other tribes received VR grants TheAIVRS Program was originally funded by a 0.25 percent set aside of federal vocationalrehabilitation funding under PL 93-112 The 1992 Amendments increased the percentage setaside for tribal programs from 0.25 percent to 0.33 percent

In 1998, the American Indian Rehabilitation Research and Training Center (AIRRTC) wasawarded a grant from the National Institute on Disability and Rehabilitation Research (NIDRR)

to conduct a study on the relationship between consumer satisfaction and employment outcomesfor American Indians/Alaska Natives receiving VR services.4 The study examined surveyinstruments used by VR agencies in 18 states and reviewed available agency reports The studyfound little consumer satisfaction data for American Indians/Alaska Natives at either the national

or the local levels There are several reasons that VR programs fail to report customersatisfaction and outcome data for American Indians/Alaska Natives who receive services in VRprograms:

4 White, M., et al

 Failure to collect client race/ethnicity Most consumer satisfaction surveys examined

in the study failed to collect client race/ethnicity data Consequently, the servicesprovided to, outcomes experienced by, and satisfaction of American Indian/AlaskaNative customers could not be determined

 Some VR programs did not collect customer satisfaction data

 Different VR programs measured customer satisfaction in very different ways As aresult, it is difficult to compare the customer satisfaction at one program with that ofanother

 Customer satisfaction may be measured at different times during the program cycle(e.g., after completing some program requirement such as the IPE, or after jobplacement) Because of differences in the timing of the measurement, it is difficult tocompare VR programs or to compare customer satisfaction in the same program indifferent years

Findings from the AIRRTC study suggest that some VR programs need help in measuring andreporting consumer satisfaction and that such measurements should include determination ofcustomer race/ethnicity VR programs should be able to report the services provided to, theemployment outcomes experienced by, as well as the level of satisfaction of all clients includingAmerican Indian/Alaska Native clients This Guide was developed to assist VR project staff inmeasuring the satisfaction of American Indian/Alaska Native customers receiving VR services

Section 2: WHAT is evaluation?

For this Guide, evaluation is a process used to 1) determine how well your VR program ismeeting its goals (e.g., numbers of customers served, case closures, customer satisfaction, jobplacements)—often called outcome evaluation, 2) identify program areas that need improvement,and/or 3) determine if the program operates the way it was designed to operate—often calledprocess evaluation Finding out the degree to which your customers are satisfied with importantparts of your program can be an important part of program evaluation Customer satisfaction can

be measured You can learn what parts of your program customers are satisfied with, what partsthey are dissatisfied with, how strongly they feel, and their views on the causes of theirdissatisfaction Collecting customer satisfaction data is not difficult, and this Guide shows youhow to do it

For example, you can measure how satisfied your customers are with program services,activities, and conditions including:

 The ways they found out about the program (program outreach/advertising)

 Program enrollment procedures and activities

 Assessment, testing, and diagnostic procedures and activities

 Development of an Individual Plan for Employment (IPE)

 Vocational skills training

 Facilities and equipment

 Involvement of the client’s family in program activities

 Life skills training

 Health maintenance and improvement training

 Program staff

• Customers treated with respect

• Concern with customer’s well being

• Helping customers to achieve goals and abilities—client empowerment

• Knowledge and skill level

 Job placement efforts and results

 Support services such as transportation and financial assistance

 Post-placement support

 Post-placement job satisfaction

Section 3: WHY evaluate?

There are three basic reasons to measure the satisfaction of your customers:

These reasons are interrelated Governments (Federal, State, Tribal) are responsible for ensuringthat money they spend (mostly generated by taxes) is well spent The federal governmentprovides funds to State and tribal VR programs and seeks to ensure that these programs providevaluable services to clients, that minority clients are treated fairly, and that VR programcustomers are satisfied with the services provided Similarly, program staff want their programs

to be effective, and are proud of their contributions to the success of the program

Program evaluation is not a goal—it is an activity that is done to help meet goals such asimproving the program, guiding allocation of resources, and determining the degree to which aprogram is meeting its goals and objectives By collecting and studying the level of satisfaction

of your customers, you can find ways to improve the program, to maintain existing high levels ofcustomer satisfaction, and to improve customer satisfaction where needed

This Guide operates from a set of assumptions and research findings:

 There are few programs that cannot be improved in some way or ways

 Programs that are not working well can generally be improved

 It is easier to improve a program using objective clear information rather than usingsubjective or vague impressions, and

 Customer satisfaction data can help guide program improvement

In general, when customers are satisfied with a program component, the VR program shouldcontinue its practices relevant to the component or, perhaps, adjust its practices in accordancewith the reasons given by the customers for their satisfaction For example, customers may report

a high degree of satisfaction with the program’s recruitment/outreach efforts, with manycustomers citing radio announcements and health care providers as the primary sources of

 It is required by the federal government (Rehabilitation Act

Amendments of 1998 and GPRA)

 You can use the customer satisfaction data to make your program

better and to improve the level of satisfaction of its customers

 Knowing what makes satisfied customers (and what contributes to

successful employment of your customers), you can share successful practice with other VR programs to promote better VRoutcomes for American Indian/Alaska Native customers

information about the program; conversely, few customers might cite television as a source ofinformation about the program Thus, while most customers report a high level of satisfaction,the program might decide to change the station(s) used in television advertising, to dropexpensive television advertising, and to direct funds to other aspects of the program Thispositive customer satisfaction finding might be used in several other ways For example, staffresponsible for the radio advertising should be made aware of the customer satisfaction findings.Even if these staff and others anticipated the findings, confirmation of the success of their ideasand efforts could be rewarding In addition, program staff might share the positive findings withthe relevant health care providers acknowledging their contribution to customer satisfaction.Also, the customer satisfaction results might indicate that some providers who were expected to

be referral sources actually provided fewer referrals to the programs than were anticipated The

VR program might seek to determine if there are ways to increase referrals from such sources

When customer satisfaction measurements show that customers are dissatisfied with aspects of a

VR program, staff can examine the data to discover the reasons given for the dissatisfaction and

to determine if there are differences between satisfied (if there are any) anddissatisfied customers Using a hypothetical example, examination of thedata might reveal that many customers (e.g., 55%) say that they aredissatisfied with their level of involvement in designing their IPE Thesedata might guide program improvement through consultation with the staff

or consultant providing the specific component of training

Section 4: HOW to Measure Customer Satisfaction

There are many ways to evaluate a VR program—to determine how well different parts of a VRprogram are working This Guide focuses on one part of the evaluation—how to measurecustomer satisfaction

The first questions raised by VR program staff about data collection,

including the measurement of customer satisfaction, include:

 WHO is going to do it?

 HOW will they do it?

 WHEN will they do it?

 WHERE is it done?

Who is going to measure customer satisfaction?

Some programs, usually larger programs, have a staff position with responsibilities (andresources) that includes program evaluation Other programs may outsource the task of programevaluation to a consultant or other person or entity that helps conduct periodic evaluations Inany event, a first or, at least an early step in the process is the identification of a person orpersons who will be responsible for measuring customer satisfaction

Identifying a person to collect satisfaction data is important and can be difficult for two reasons:

It is critical that each customer be assured that 1) he/she will suffer no negative effects forexpressing dissatisfaction with aspects of the program, and 2) his/her privacy will be protected—customer’s opinions, judgments will not be shared in ways that identify him/her For example, acustomer might be reluctant to express dissatisfaction with aspects of the VR program to his/herprimary counselor, the program director, or anyone affiliated with the program, fearing thatexpressions of dissatisfaction could lead to denial of services, some form of retaliation, orneglect by program staff Thus, it is critical that:

Whoever is involved with the measurement of customer satisfaction must

convince the customer that the measurement is being conducted to ensure

that his/her needs are being met, and to guide improvement of the

program The persons involved must convince the customer that

expressions of satisfaction/dissatisfaction will not be kept with identifiable

information such as the customer’s name, and that the satisfaction data

are confidential, kept apart from customer records.

If VR project staff lack the skills, experience, and/or cultural sensitivity to collect the customersatisfaction data, there are sources for obtaining such training and/or training materials Theseinclude but are not limited to the AIRRTC, the Consortia of Administrators for Native American

Rehabilitation (CANAR), state VR programs, the National Clearinghouse of RehabilitationTraining Materials (NCRTM), and the Council on Rehabilitation Education (CORE)

How will customer satisfaction be measured?

Questions and scales Measuring customer satisfaction can be done by simply asking a series of

questions It is generally best if some of the questions measure the degree of satisfaction ordissatisfaction using a scale For example, the customer can be asked to indicate the item thatbest represents his or her view from a set of five alternatives:

Very Moderately Neutral Moderately Very

unsatisfied unsatisfied satisfied satisfied

Strongly Moderately Neutral Moderately Strongly

disagree disagree agree agree

In addition, it is necessary to record demographic information about the customer such as his/hergender, race/ethnicity, tribal affiliation, age, and nature of disability Such information willenable the program to determine if certain groups of customers are more or less satisfied with theprogram

Open-ended questions In addition to asking questions about the level of satisfaction, good

measurement involves asking open-ended questions about why the customer feels the way he/shedoes, about ways the program can be changed to improve his/her level of satisfaction, whathe/she liked best about the program, and what he/she disliked most about the program (see Tab6)

Improving customer satisfaction through measurement The process of measuring customer

satisfaction itself can contribute to the level of customer

satisfaction with the program If the questions are presented in

ways that communicate that the process is a burden to the staff, that

the staff member or consultant is “going through the motions” in

asking questions because he or she is required to, customers are likely to believe the program isnot really interested in their level of satisfaction Consequently, the customer is likely to formnegative feelings and opinions about the staff involved and, perhaps, the program in general Onthe other hand, if the staff or consultant involved in measuring customer satisfaction really caresabout the customer’s satisfaction, and communicates this interest, then the customer is likely tofeel valued and to form positive opinions about the staff or consultant and, perhaps, the program.

Incorporating measurement in program activities One approach to obtaining customer

satisfaction data is to build the data collection process into the standard program procedures Forexample, a customer satisfaction survey could be built into most or many program activities orprocedures such as

 enrollment in the program (record demographic information such as gender, andeducational experience needed for data analysis)

By using a small number of questions and rating scales to measure customer satisfaction, theburden on the customer and program staff can be minimized By making the measurement ofcustomer satisfaction part of regular program activities and procedures, we overcome theproblem of collecting all customer satisfaction data at just one or two points in time

Type of data collection Sometimes it is not possible or practical to build the measurement of

customer satisfaction into program procedures and activities In such cases, customer satisfactioncan be measured by asking customers to answer questions sent by mail, asked in a telephoneinterview, asked in a face-to-face, personal interview, or accessed through the internet Each of

these methods of collecting customer satisfaction (and other program evaluation) data isdescribed below:

 Mail surveys —the VR program (or a person or organization working for the program)mails a questionnaire to a group of customers or former customers The customerscomplete the questionnaire and mail it back to the sender Mail surveys tend to consumeless staff time as interviewers are not used; however, mail surveys tend to have lowresponse rates—many customers fail to respond to them, even when provided with anaddressed, stamped envelope for returning the completed questionnaire

 Telephone surveys —the VR program (or a person or organization working for theprogram) calls current and/or former customers asking questions by telephone.Telephone surveys are impractical in much of “Indian country” because of the highpercentage of homes that lack telephone service If telephone surveys are used as amethod for collecting customer satisfaction data, it is helpful to send a postcard to thecustomer in advance of the telephone call informing him/her of the survey

 Face-to-face (in person) interviews—A member of the VR program staff or anotherperson interviews current and/or former customers (often at a program office or at apost-placement workplace), reading and/or signing the

questionnaire items to the customer Face-to-face interviews, by

their nature, tend to be more intimate than the other methods

of data collection This intimacy can be used to

communicate care and concern to customers, but can have

the disadvantage that customers may be more reluctant to

criticize or speak negatively about the program in a

face-to-face interview than in an Internet, mail, or telephone

survey On balance, this Guide recommends collecting customer satisfaction data usingface-to-face interviews

 Internet surveys —The VR program (or a person or organization working for theprogram) develops an interactive questionnaire on a website Customers and/or formercustomers log on to the website and complete the questionnaire Alternatively, the

customers can print the questionnaire, complete it by hand, and mail

or fax it to the program or other designated place Internet surveys

tend to have the low response rates of mail surveys, in part,because internet access is unavailable or difficult to access

in many parts of Indian country

Cultural sensitivity and appropriateness Just as successful VR programs serving American

Indians/Alaska Natives design and deliver services in culturally sensitive ways, successfulmeasurement of customer satisfaction is done in a similar fashion Because of the diversity ofculture across the more than 550 federally recognized tribes and Alaska Native Villages, each

VR program must determine and meet the needs of its customers in measuring their satisfactionwith the program Some general guidelines include:

 Language—questions/items should be translated into the customer’s native language athis/her preference Sometimes, questions are best asked or presented with a

combination of English and the customer’s native language

 Relation of the interviewer to the customer—in some cultures, it is

inappropriate for a person to express opinions or judgments to certain

groups as a function of age, gender, clan, relative (e.g., uncle,

mother-in-law)

 Confidentiality—especially in small communities, customers may fear that theconfidentiality of their opinions and judgments cannot be maintained—that othermembers of the community will discover what has been said

Meeting Customer Needs As with all aspects of VR services and activities, measuring

customer satisfaction (and other evaluation activities) should be designed and conducted inaccordance with the needs of the customer being interviewed In general, we recommend that, inface-to-face interviews, the customer be given a printed (in Braille if appropriate) copy of thequestionnaire so that he/she has the option of reading the items while the interviewer reads orsigns each item In general, the interviewer will sign and/or read aloud each item on thequestionnaire and record the customer’s response In some cases, the customer will prefer to readand respond to each item without assistance of the interviewer; this option is acceptable

Examples of actions required to meet the special needs of some customers include:

 Using large type or Braille type for visually impaired customers

 Signing for hearing impaired customers

 Presenting the questions in two or more separate sessions for customers who may needbreaks

 Ensuring that the customer does not miss any program activity because of participating inthe interview; for example, the customer should not miss lunch or a training activity thatcannot easily be rescheduled while he/she is participating in the satisfaction survey

When to measure?

Currently, programs that measure customer satisfaction do so at varying points in the VRprocess For example, customer satisfaction may be measured after the development of the IPE,