Practicing Organization Development (A guide for Consultants) - Part 37 docx

Evaluation can occur at three points in the OD process: during the process formative, at the conclusion of the process summative, or at some time after the process has ended longitudinal

Figure 12.4 Organization Development Evaluation Model

10 Administer Longitudinal Evalutation

1 Do Assessment

7 Analyze Results

8 Present Results

9 Make Revisions

2 Determine OD Needs

3a Determine Level

of Evaluation

3b Design Intervention

4b Develop Evaluation Instrument 4a Develop Intervention

5b Administer Formative Evaluation

5a Implement Intervention

6 Administer Summative Evaluation

IMPORTANCE OF EVALUATION

If you don’t know where you are going, any path will get you there And if you don’t know where you are, you’ll never know where you are going The process

of evaluating is truly about determining whether the path you are on has been the most efficiently executed and effective path for the system you are in In turn, this will influence how you move forward into the future The funda-mental premise is self-awareness A system of any size—from one person to a whole organization—will be most amenable to change when it has all relevant data The more that the people in a client system are able to understand them-selves, the greater the likelihood that they will take ownership in determining how to best move forward

Within this context, the key change agents are more able to influence the sys-tem with data that demonstrate success and failure Action learning focuses on important outcomes—What was planned? What happened? What was learned? What will be done next time? One critical step to creating learning is addressed

in the question: What happened? Success and failure are concepts critical to

cre-ating a learning organization, and this is where the consultant as intervener comes into the picture By providing the system with avenues for having a bet-ter understanding of itself through rigorous evaluation, the system will begin to change its thinking paradigm When the system begins to change its thinking paradigm, the learning has begun There is no other more powerful way to cre-ate change than through learning rooted in the shifting of how we think This learning is what will lead to new and sustainable organizational habits that result in long-term health and prosperity

WHERE EVALUATION FITS IN THE INTERACTION WITH CLIENTS

As indicated earlier, it is important to establish the evaluation processes during the pre-launch phase In this way, what is important for the client, the consul-tant, the participants, and the process can be established before the process is begun With a plan in place, continuous improvement in the OD process, based

on identified evaluative feedback, can occur

Evaluation can occur at three points in the OD process: during the process (formative), at the conclusion of the process (summative), or at some time after the process has ended (longitudinal) The most important of these times is

during the OD process—formative evaluation To discover that something could

have been done differently or that the process did not create the intended

outcomes after the process is over does not allow for adjustments or modifica-tions along the way

WHO SHOULD DO THE EVALUATION?

This decision, too, should occur during the pre-launch phase There are three common answers to the question of who should perform the evaluation: the client organization, the consultant, or a third party Many factors will influence where the responsibility for evaluation will rest Given the preference for for-mative evaluation expressed earlier, the responsibility for evaluation will fall equally with the consultant and the client Regular, ongoing evaluation will pro-vide feedback continuously, if so designed, and will allow the consultant and the client to partner to maximize the potential in the OD process For summa-tive and longitudinal evaluation, there are pros and cons for each potential eval-uator These are summarized in Exhibit 12.1

KEY MEASURES IN OD

When taking a systems approach to measuring in OD, we advocate the inclu-sion and integration of six key dimeninclu-sions: performance, morale, emotion, crit-ical customer service behaviors, organization climate, and personal characteristics (Cady & Lewis, 2002) Each component falls into one of two categories of

mea-surement, soft or hard Soft measures are perceptual in nature That is, the

scor-ing of soft measures is based on an individual’s judgment or opinion For example, an employee is asked to respond in a survey to the statement, “I am satisfied with my job.” The employee chooses a number on a seven-point Likert scale ranging from strongly disagree with the statement (equal to 1) to strongly

agree with the statement (equal to 7) (Rea & Parker, 1997) Hard measures are

observable and can be objectively counted (that is, not based on opinion), often based on behavior or units of some kind For example, the number of units sold

or made is a hard measure

Taking a systems approach to evaluation is important in encouraging mem-bers of an organization to understand how their actions and behaviors (soft measures) influence important bottom-line outcomes (hard measures) A sys-tem model can be used to provide meaningful ongoing feedback at all levels, to diagnose departmental or organization-wide performance issues, and to iden-tify appropriate interventions that will positively impact performance We will now review the six key measurement dimensions mentioned earlier that should

be considered when taking a systems approach to evaluation

Exhibit 12.1 Advantages and Disadvantages to Possible Evaluators

Client Organization

• Not an additional cost to the organization

• Will be present to the organization long after the specific OD process is over

• Knows the culture of the organization better than an outside party can

• Knows what is important to the organization and, thus, what needs to

be evaluated

• Because of responsibility for participating

in planning the OD process, can build evaluation into pre-launch phase

• May be too closely associated with the organization to see what is necessary in the evaluation

• May fear that the outcome of the evaluation will cause them to lose credibility if they were responsible for the contract

• May not have time to do evaluation effectively because of other projects

• May not have the necessary competencies

to carry out the evaluation

Consultant

• Knows the OD process and its intended outcomes as well

• Should have the necessary competencies

to carry out the evaluation

• Because of the responsibility for pating in planning the OD process, can build evaluation into the pre-launch phase

• Gives the consultant the opportunity to modify processes used or provide adequate compensation for mistakes made

• Adds to the project‘s costs

• Unless a long-term involvement is envisioned, the consultant may not be around to complete the evaluation

• May not be viewed as objective as there

is an incentive to make the evaluation look good (for affirmation of client‘s choice) or for additional needs to surface (for additional work)

• May create a sense of client‘s dependence

on the consultant

Third Party

• Viewed as more objective because not involved in any other way in the OD process

• Usually more expert because of the ability

to specialize in evaluation

• Has nothing to lose or gain in the outcome

of the evaluation

• Clearly, an additional cost

• A third party may not want to take on such

a limited component of a project

• Is not integral to the project, so may not know either the process or the outcomes

to be assessed

• Is probably not in a position to influence improvements in the process or in im- proving the ODs consultants processes

• Guarantees that the evaluation will not be formative

Measurement Dimension 1—Profitability and the Bottom Line When discussing the bottom line, many people think of profitability first In this chapter, profitability is part of a broader definition of performance that is based

on a balanced approach to assessing the bottom line There are three perfor-mance measures, the first of which is profitability Profit is one measure of per-formance that has dominated throughout the history of economic ventures, and

it remains a central focus today However, there is growing recognition that

profit is not the only indicator of the real bottom line (for example, Kaplan & Norton, 1996) For non-profit organizations, consider cash flow as a measure that replaces profitability for a for-profit organization

There are other measures, along with profit, that provide a more balanced

approach to assessing the bottom line That is why the word performance is

used in place of bottom line In addition to profit, there are internal and exter-nal performance measures, such as customer retention, organizatioexter-nal learning, new products developed, and many more

Measurement Dimension 2—Morale Measures of morale are soft by nature That is, asking the employees or cus-tomers their opinions regarding a variety of issues assesses each measure Mea-sures of morale address attitudes and intentions from both the employee and customer perspective An attitude is a bi-polar affect regarding some target (for example, I like it or I don’t like it) Attitudes focus on such things as job satis-faction for employees and product/service satissatis-faction for customers Intentions are cognitive choices made with regard to pending action or lack of action (for example, I will do this or I will do that or I will do nothing) Intentions focus

on such things as intention-to-leave a job or intention-to-return to buy a product/ service again As mentioned earlier, these measures tend to focus on a degree

of agreement or disagreement on a Likert scale

Measurement Dimension 3—Emotion Emotions are similar yet distinct from morale Emotions encompass glad, sad, mad, afraid, and ashamed There are also the emotions of excited, peace, and numb These emotions tend to be related to discrete events and are reactionary

in nature The reason that they are considered similar to morale is that they are known to influence morale directly For example, a failed performance by an employee (for example, a lost sale) can lead to anger and guilt These emotions can subsequently lead to an exaggerated low level of job satisfaction and a heightened level of intention-to-leave Over time, these levels of satisfaction and intention-to-leave can return to the original levels prior to the failed perfor-mance That is, emotions generally have a temporary influence on morale, not

a permanent one

Measurement Dimension 4—

Critical Customer Service Behaviors Customers have certain expectations when receiving products and services There are all sorts of support required and questions to be asked In the process, the customer can define a set of employees’ behaviors that add value to the product or service These behaviors are applicable across industries, yet must

be customized to fit a particular industry (service or product) There are five critical employee behaviors that can lead to higher levels of customer satisfac-tion and retensatisfac-tion: response time, personalized attensatisfac-tion, professionalism, expertise, and comforting

Measurement Dimension 5—Organization Climate There is a considerable amount of research and numbers of survey measures that are directed toward a better understanding of those factors that directly influence task performance These task-dependent measures fall into three

variables: job climate factors, teaming behaviors, and leadership behaviors Job

climate factors are directly related to the work being done and include

envi-ronmental support, personal capabilities, and reward systems Teaming

behav-iors are the critical behavbehav-iors team members should exhibit that lead to a

high-performing team Teaming behavior measures include the following: antic-ipates and fulfills team needs; manages own personal contribution; displays a

positive attitude; and effectively communicates Leadership behaviors are those

critical behaviors that leaders should exhibit to create a high-performing team Leadership behaviors include the following: supporting, informing, monitoring, recognizing and rewarding, empowering, and clarifying

Measurement Dimension 6—Personal Characteristics There are two types of personal characteristics, each of which contains differ-ent measures First, there are biographical characteristics that are considered

hard measures and are usually found in most personnel files Biographical

char-acteristics include such things as age, race, gender, tenure, and function The

second type of personal characteristic is personality Personality refers to more

stable behavioral traits that people exhibit For example, there are a variety of personality models, such as the Myers-Briggs Type Indicator (MBTI), the Big Five (extraversion, emotional stability, agreeableness, conscientiousness, and openness to experience), and other non-dimensional measures, such as locus-of-control and pessimism While these tend to be very popular in use, their validity is suspect

TYPES OF EVALUATION

There are many ways to perform an appropriate evaluation In planning an OD evaluation, it is important that an agreement be reached in the beginning about what is important and how it will be determined, whether the outcome has

been achieved or not As described earlier in this chapter, it is important to decide who will conduct the evaluation Outcome measures must be identified

A time frame—formative, summative, and longitudinal—must be determined And then the appropriate tool or approach must be chosen

Formal/Informal

A formal approach to evaluation will be decided among the parties involved in the OD process Informally, however, all of the parties involved will be con-stantly looking at ways to improve their roles in the OD process Sometimes this may be apparent, as when a consultant might simply ask a client contact how things are going Often, however, the informal evaluation process will not be apparent It may consist of continually observing how people are responding and participating, checking on available statistics that might reflect on the desired outcome measures, and even watching stock prices The concern with informal evaluation is that the results are not shared and are not systematic As

a result, partnering between consultant and client loses synergy, and results that surface are biased by happenstance observations rather than systematic ones

It also makes great sense in summative and longitudinal evaluation to repeat whatever was done in the assessment stage Since that process was successful in identifying the focus for the OD process undertaken, it makes sense to use the same approach in evaluating the outcomes of that process

Thus, the recommendation of the authors is that a shared, systematic per-spective be selected from among the many formal modes of evaluation that are available There are many ways in which such approaches could be classified

We have chosen to classify them as quantitative (also called hard) and qualita-tive (also called soft) These will be detailed further in this section

Quantitative

A preference still exists in business to have quantifiable data, and many of the traditional approaches to evaluation in OD rely on quantifiable data These data come from a variety of sources—surveys (both simple and complex), measur-able outcomes (such as discussed earlier in this chapter—both in terms of meet-ing pre-specified goals and pre-process and post-process comparisons), and benchmarking (especially with competitors) are the primary approaches used

Surveys

These can be as simple as a basic satisfaction survey (often referred to as a happy sheet) to a somewhat more complicated feedback form (for example, an assessment of how meetings are going) to multi-rater feedback to a very com-plex culture survey of the organization

The basic satisfaction survey is probably the easiest to construct and use and, thus, is frequently used Items to be measured might include participants’ satis-faction with the OD process (activities, materials, consultant, facilities, and so on) Sample items are presented in Exhibit 12.2

A somewhat more complicated survey, although based on the same con-cepts as the satisfaction survey, is one that focuses on a specific function within the organization—for example, how well executive team meetings are going, level of customer satisfaction, product innovation, and so on See Exhibit 12.3 for an excerpt from a survey frequently used by one of the authors

Multi-rater assessment is another survey form that is often used when the

OD focus is on an individual or a small group of individuals This is often referred to as 360-degree feedback When completed in a survey form, it pro-vides quantitative data There are many issues with this type of approach, espe-cially when used inappropriately to determine salary and promotion potential (see McLean, 1997) A prime concern with the use of such an instrument is that research shows that different groups of reviewers provide different feed-back, naturally, as people are observed by different groups of people in differ-ent contexts

Finally, the most complex form of survey is the climate (or culture) survey This is usually a comprehensive review of the factors in the organization that affect the climate (or culture) A controversial issue that frequently surfaces with surveys is whether to use a customized survey (developed by the consultant or

a team in the client organization) or a commercial, standardized survey See Exhibit 12.4 for a comparison of the two approaches

Exhibit 12.2 Sample Items from an OD Process Satisfaction Survey

Evaluation of Team Building OD Process

In responding to the following questions, circle the number that corresponds with the following scale:

5 = Very Satisfied; 4 = Satisfied; 3 = Neutral; 2 = Dissatisfied; 1 = Very Dissatisfied

1 How well the need for team building 5 4 3 2 1 was originally identified

facilities were

activities at the retreat were

Exhibit 12.3 Survey to Determine Satisfaction with a Team Meeting Process

Team Meeting Assessment

Circle the number to the right of each item that matches your assessment of how well the team is currently functioning Use the following scale:

5 = Strongly Agree; 4 = Agree; 3 = Neutral; 2 = Disagree; 1 = Strongly Disagree

Planning

meetings.

before meetings.

outcomes.

6 Visual aids are available, including flip 5 4 3 2 1 charts, markers, tape, sticky notes, etc.

interaction.

materials to meetings.

scheduled.

10 Agendas adequately reflect meetings 5 4 3 2 1

Doing

rather than providing information.

meetings.

16 Discussion is focused on outcomes; 5 4 3 2 1

it does not ramble.

17 Each team member fully participates 5 4 3 2 1 when competent to do so.

discussion.

clearly developed action step.

team members.

Practicing Organization Development, 2nd Ed Copyright © 2005 by John Wiley & Sons, Inc Reproduced

Exhibit 12.3 Continued

Doing

each other.

22 I feel free to express my opinions 5 4 3 2 1 without fear of reprisals.

consensus basis.

24 All team members accept responsibility 5 4 3 2 1 for the quality of team meetings,

providing feedback during the meeting

on ways to improve meetings.

encouraged.

is completed.

facilitation skills.

30 Conflict is handled in a healthy way, 5 4 3 2 1 focusing on facts and “I” statements.

without responding with “Yes, but.”

another’s new ideas.

34 The agenda consists of items at an 5 4 3 2 1 appropriate level, delegating detail

to appropriate employees.

are identified before the end of each meeting.

are used during meetings.

for the team’s action steps to be carried out.

40 The next meeting date is set 5 4 3 2 1 prior to adjournment.

Practicing Organization Development, 2nd Ed Copyright © 2005 by John Wiley & Sons, Inc Reproduced