an investigation of accuracy, learning and biases in judgmental adjustments of statistical forecasts

AN INVESTIGATION OF ACCURACY, LEARNING AND BIASES IN JUDGMENTAL ADJUSTMENTS OF STATISTICAL FORECASTS DISSERTATION Presented in Partial Fulfillment of the Requirements for the Degree of

AN INVESTIGATION OF ACCURACY, LEARNING AND BIASES

IN JUDGMENTAL ADJUSTMENTS OF STATISTICAL FORECASTS

DISSERTATION

Presented in Partial Fulfillment of the Requirements for

the Degree of Doctor of Philosophy

In the Graduate School of The Ohio State University

By Cuneyt Eroglu

* * * * * The Ohio State University

2006

Dissertation Committee:

Professor Douglas M Lambert, Adviser

Professor Keely L Croxton

_

Business Administration

Copyright by Cuneyt Eroglu

2006

ABSTRACT

Judgmentally adjusting a statistical forecast before using it is a widespread

practice in business The goal of this study is to provide a deeper understanding of

judgmental adjustments of statistical forecasts to improve forecasting performance The

forecasting performance is measured with three dependent variables The first dependent

variable is accuracy improvement which represents the positive change in forecast

accuracy after a judgmental adjustment The second dependent variable is learning which

refers to the continuous improvement of a forecaster’s performance over time The third

variable is actually a set of variables that includes various biases A bias is a systematic

deviation in forecasts that is introduced by a particular forecaster The independent

variables included in this study are personality variables, motivational variables and

situational variables Personality is measured by the Big-Five model that analyzes an

individual’s personality in five dimensions including extraversion, conscientiousness,

neuroticism, agreeableness and openness to experience Motivational variables are

measured in terms of motivational orientation and various motivational stimuli

Situational variables include feedback, supervision, timing of adjustment, and

demographics

Data were collected from a company where store managers used judgmental

adjustments of statistical forecasts to improve the forecast accuracy for their stores The

data collection covered a period of 12 months and 390 stores over several Midwestern

and southeastern states The data for dependent variables were obtained from forecasting

records and the data for independent variables were collected using a survey instrument

The results indicate that, on average, judgmental adjustments of statistical

forecasts result in accuracy improvement The extent of the accuracy improvement is

affected by personality, motivational and situational variables Furthermore, there was

evidence for biases that were introduced through judgmental adjustments Biases were

also moderated by personality, motivational and situational variables This study detected

no evidence of learning

ACKNOWLEDGMENTS

A doctoral dissertation is rarely an exclusive work of a single individual This

dissertation is no exception I am deeply indebted to several people whose support, help

and encouragement made this dissertation a reality Their contributions to this study were

invaluable

First and foremost, I would like to thank Professor Douglas M Lambert, Director

of The Global Supply Chain Forum at The Ohio State University and Chairman of my

dissertation committee He has been very supportive throughout the entire course of my

doctoral program in countless ways His expert judgment and constructive critiquing of

my work enabled me to navigate safely and productively through every phase of my

dissertation from selecting a promising research topic to applying rigorous academic

standards to my data collection and analysis to communicating the research results in a

well-written dissertation He not only guided me from an academic perspective, but also

helped me obtain business data for my dissertation and make my research relevant to

practitioners His insights, feedback, encouragement and guidance have made this

dissertation remarkably better Above all, he has provided me with an excellent role

model as a researcher, a teacher and an academician, which I will always look up to and

which will surely shape my academic career for the years to come

Together with Professor Lambert, Professor Keely L Croxton and Professor A

Michael Knemeyer formed my dissertation committee that guided me through the

dissertation phase of my studies Their feedback and insights contributed immensely to

the quality of the end product They were always ready and willing to assist me with any

challenges and dilemmas that I was facing Furthermore, they dedicated many hours of

their time to review my work and provide me with further guidance I will forever owe

Professor Croxton and Professor Knemeyer a debt of gratitude

I also would like to thank Professor Martha C Cooper, Professor Walter Zinn and

Professor Thomas J Goldsby who have made significant contributions to my personal

and professional growth during my Ph.D program During my interactions with them,

both inside and outside the class room, our discussions have always been intellectually

stimulating and broadened my horizons Their support and guidance will forever be

remembered with much gratefulness

As a person who has always gone beyond the call of duty, Shirley J Gaddis

deserves much credit Without her crucial assistance and extraordinary organizational

skills, it would have been much more difficult for me to communicate with my

dissertation committee members, schedule meetings and presentations, and circulate

drafts of my work She has been an exceptional facilitator between my dissertation

committee members and myself, which, in turn, substantially shortened the time required

for writing this dissertation

I would also like to express my gratitude for my fellow doctoral students who

have helped create a friendly working atmosphere where we shared good times, debated

research ideas, and vented during more challenging times Furthermore, I have received

much help from the staff at the Marketing and Logistics Department, the Graduate

Programs Office and the Office of International Education Their assistance is greatly

appreciated

This dissertation is based on the data obtained from a company that is a member

of The Global Supply Chain Forum at The Ohio State University I would like to express

my deepest thanks to the executives at this company and other member companies of The

Global Supply Chain Forum who have provided me with data, stimulated my thinking

with their questions, and enriched my work with their suggestions and feedback Their

contributions added immensely to the value of this dissertation I shall forever be grateful

to them

Last but not least, I would like to thank Steven Robeano, a dear friend who has

always been there for me throughout the Ph.D program I could always count on him to

listen to me, to share his insights and ideas, to offer advice and provide encouragement

His friendship is greatly appreciated

VITA

EDUCATION

1992 Bachelor of Science, Industrial Engineering, Middle East

Technical University, Ankara, Turkey

1994 Master of Science, Management Science, University of

Miami, Coral Gables, Florida

2004 Master of Arts, Business Administration, Logistics, Fisher

College of Business, The Ohio State University 2002-present Graduate Teaching and Research Associate, The Ohio State

University

PROFESSIONAL EXPERIENCE

1994-1996 Logistics Analyst, Ryder Dedicated Logistics, Miami,

Florida 1997-1998 Analyst, Istanbul Gold Exchange, Istanbul, Turkey

1999-2001 Business Development Manager, Federal Express, Istanbul,

Turkey 2001-2001 Project Manager, Ericsson Telecommunications, Istanbul,

Turkey

FIELDS OF STUDY Major Field: Business Administration

Areas of Specialization: Logistics and Marketing

TABLE OF CONTENTS

ABSTRACT ii

ACKNOWLEDGMENTS iv

VITA vii

LIST OF TABLES xiii

LIST OF FIGURES xviii

CHAPTER 1 INTRODUCTION 1

1.1 Background 1

1.1.1 Theoretical Background 4

1.1.2 Business Background 7

1.2 Research Design 9

1.2.1 Research Purpose 10

1.2.2 Model Building and Hypotheses 11

1.2.3 Data Collection and Analysis 15

1.3 Limitations 16

1.4 Potential Contributions 19

1.4.1 Managerial Contributions 19

1.4.2 Academic Contributions 20

1.5 Organization 21

LIST OF REFERENCES 23

CHAPTER 2 LITERATURE REVIEW 25

2.1 Prevalence of Judgmental Adjustments 26

2.2 Efficacy of Judgmental Adjustments 27

2.2.1 The Case Against Judgmental Adjustments 28

2.2.2 The Case for Judgmental Adjustments 33

2.2.3 Domain Knowledge and Judgmental Adjustments 36

2.3 Factors Affecting the Accuracy of Judgmental Adjustments 38

2.4 A Critique of Current Literature 51

2.5 Summary 55

LIST OF REFERENCES 57

CHAPTER 3 MODEL BUILDING AND HYPOTHESES 61

3.1 The Working Model 61

3.1.1 Brunswik’s Lens Model 62

3.1.1.1 Functionalism 65

3.1.1.2 Probabilism 66

3.1.2 Hogarth’s Model 68

3.1.3 Information-Processing Model 72

3.1.4 Work Performance Model 76

3.1.5 Synthesis of Relevant Characteristics of the Above Models 79

3.1.6 The Working Model 81

3.2 The Dependent Variables 85

3.2.1 Accuracy Improvement 85

3.2.2 Learning 87

3.2.3 Biases 88

3.3 Independent Variables 92

3.4 Hypotheses 94

3.4.1 Intelligence 95

3.4.2 Personality 95

3.4.2.1 The Big-Five Personality Model 97

3.4.2.2 Conscientiousness 98

3.4.2.3 Emotional Stability 99

3.4.2.4 Extraversion 101

3.4.2.5 Agreeableness 102

3.4.2.6 Openness to Experience 102

3.4.3 Motivation 103

3.4.3.1 Motivational Orientation 107

3.4.3.2 Motivation-Behavior-Performance Connection 110

3.4.3.3 Motivational Behaviors 114

3.4.3.4 Motivational Stimuli 117

3.4.4 Situational Variables 124

3.4.4.1 Directional Bias 125

3.4.4.2 Information Sharing 126

3.4.4.3 Feedback 127

3.4.4.4 Presentation of information 127

3.4.4.5 Presentation of adjustment 129

3.4.4.6 Timing of adjustment 129

3.4.4.7 Experience 129

3.4.4.8 Demographics 130

3.4.4.9 Data variability 130

3.4.4.10 Sales Volume 131

3.5 Summary 131

LIST OF REFERENCES 132

CHAPTER 4 METHODOLOGY 140

4.1 Research Design 141

4.1.1 A Critical Overview of Laboratory Experiments 142

4.1.2 A Critical Overview of Field Studies 148

4.1.3 Design of Current Research Study 151

4.2 Research Setting 154

4.3 Data Collection 159

4.3.1 Adjustment Data 160

4.3.2 Survey Instrument 162

4.3.2.1 General Considerations and Cover Page 162

4.3.2.2 Personality 164

4.3.2.3 Work Locus of Control 168

4.3.2.4 Scale for Motivation 169

4.3.2.5 Forecasting Practices and Demographics 173

4.3.2.6 Determination of Sample Size 177

4.4 Summary 179

LIST OF REFERENCES 180

CHAPTER 5 DATA ANALYSIS AND RESULTS 183

5.1 Description of Input Data 183

5.1.1 Demand Data 184

5.1.2 Adjustments Data 189

5.1.3 Survey Responses 193

5.2 Description of Outcome Variables 197

5.2.1 Accuracy Improvement 198

5.2.2 Learning 202

5.2.3 Biases 203

5.2.4 Testing for Non-Response Bias 210

5.3 Hypotheses Testing 212

5.3.1 Personality 213

5.3.2 Motivation 220

5.3.2.1 Motivational Orientation 220

5.3.2.2 Motivational Behaviors and Stimuli 227

5.3.3 Situational Variables 244

5.3.3.1 Directional Bias 244

5.3.3.2 Information Sharing 245

5.3.3.3 Feedback 246

5.3.3.4 Presentation of information and presentation of adjustments 250

5.3.3.5 Timing of adjustment 252

5.3.3.6 Experience 255

5.3.3.7 Demographics 257

5.3.3.8 Data variability 260

5.3.3.9 Sales Volume 261

5.4 Summary 263

LIST OF REFERENCES 265

CHAPTER 6 SUMMARY AND CONCLUSIONS 266

6.1 Motivation for the Present Study 266

6.2 Research Setting 267

6.3 Summary of Results 268

6.3.1 Accuracy Improvement, Learning and Biases 268

6.3.2 Personality Variables 270

6.3.3 Motivational Variables 272

6.3.4 Situational Variables 275

6.4 Managerial Implications 277

6.4.1 Personnel Selection 277

6.4.2 Training 278

6.4.3 Task Design 279

6.5 Future Research Opportunities 280

6.6 Summary 282

BIBLIOGRAPHY 284

APPENDIX A THE SURVEY INSTRUMENT 297

APPENDIX B THE SELECTION OF CUTOFF POINT 302

APPENDIX C SCALE ASSESSMENT 307

LIST OF TABLES

Table 1: Characteristics of the judgmental adjustment at the research company 16

Table 2: Information provided to experimental groups 42

Table 3: Components of skill addressed by selected methods for improving forecasts 50

Table 4: Summary of studies reviewed in this chapter 52

Table 5: A forecast with n half-hour time intervals 86

Table 6: Accuracy improvement formulas 87

Table 7: Meta-analytic results (true validities) 96

Table 8: Strengths and weaknesses of lab experiments and field studies 141

Table 9: List of potentially unrealistic assumptions of lab experiments about judgmental adjustments 143

Table 10: List of less-than-ideal forecasting practices 149

Table 11: List of methodological limitations of field studies 149

Table 12: Guidelines for adjusting forecasts by store managers 156

Table 13: Data Sources 159

Table 14: Database table containing forecast adjustment data 160

Table 15: SONSO and Big-Five personality scales 164

Table 16: SONSO and Big-Five personality scales 165

Table 17: Corresponding scales of SONSO, NEO and Berkeley personality inventories 165

Table 18: Correlations between SONSO, NEO and Berkeley personality inventories 166

Table 19: Scales of the SONSO personality inventory 167

Table 20: WLOC scale questions 168

Table 21: Work Preference Inventory Items and Scale Placement 171

Table 22: Breakdown of adjustments by the number of intervals adjusted 191

Table 23: Number of managers by adjustment frequency 192

Table 24: The correlations between adjustment frequency and independent variables 193

Table 25: Survey response statistics 195

Table 26: Response rate by store managers and adjustments 196

Table 27: Number of managers by adjustment frequency 196

Table 28: A forecast with n half-hour time intervals 198

Table 29: Accuracy improvement formulas 199

Table 30: Changes in mean accuracy measures as a result of judgmental adjustments 200 Table 31: Learning model coefficient estimates (Type A) 203

Table 32: Learning model coefficient estimates (Type C) 203

Table 33: Bias formulas 204

Table 34: Summary statistics of optimism bias 205

Table 35: Summary statistics of conservatism bias 206

Table 36: Summary statistics of overreaction bias 207

Table 37: Non-response bias testing 211

Table 38: Correlation among personality dimensions 213

Table 39: Regression results for Type A accuracy improvement 214

Table 40: Regression results for Type C accuracy improvement 215

Table 41: Regression results for optimism bias 216

Table 42: Regression results for conservatism bias 216

Table 43: Regression results for overreaction bias 217

Table 44: Correlations between personality dimension and motivational orientation 219

Table 45: Effects of personality on accuracy improvement and biases 220

Table 46: Correlations between intrinsic and extrinsic motivational orientations 221

Table 47: Regression results for Type A accuracy improvement 222

Table 48: Regression results for Type C accuracy improvement 222

Table 49: Regression results for optimism bias 223

Table 50: Regression results for conservatism bias 223

Table 51: Regression results for overreaction bias 224

Table 52: Correlations between motivational orientation dimensions, subscales and perceived motivational stimuli 225

Table 53: Correlations between intensity, direction and persistence 229

Table 54: Regression model for Type A accuracy improvement 230

Table 55: Regression model for Type C accuracy improvement 230

Table 56: Regression model for optimism bias 231

Table 57: Regression model for conservatism bias 231

Table 58: Regression model for overreaction bias 232

Table 59: Summary statistics and correlations of instrumentality 232

Table 60: Correlations between motivational stimuli and accuracy improvement measures (Type A and Type C) 235

Table 61: Motivational stimuli and accuracy improvement measures (Type A and Type C) 236

Table 62: Correlations between motivational stimuli and optimism, conservatism and overreaction biases 241

Table 63: Motivational stimuli and optimism, conservatism and overreaction biases 242

Table 64: Effects of motivational variables on accuracy improvement and biases 243

Table 65: Directional bias and its effects on accuracy and other biases 244

Table 66: Information sharing and accuracy 246

Table 67: Information sharing and biases 246

Table 68: Feedback and accuracy improvement 247

Table 69: Feedback and Type A accuracy improvement 248

Table 70: Feedback and Type C accuracy improvement 248

Table 71: Feedback and biases 249

Table 72: Feedback and optimism bias 249

Table 73: Feedback and conservatism bias 249

Table 74: Feedback and overreaction bias 249

Table 75: Correlations for presentation of information and presentation of adjustments 251

Table 76: Presentation of information, accuracy improvement and biases 251

Table 77: Presentation of adjustments, accuracy improvement and biases 252

Table 78: Correlation of lead-time with accuracy improvement and biases 253

Table 79: Average accuracy improvement and biases 253

Table 80: Correlations of experience with accuracy improvement and biases 255

Table 81: Industry experience, accuracy improvement and biases 256

Table 82: Company experience, accuracy improvement and biases 256

Table 83: Experience in current store, accuracy improvement and biases 256

Table 84: Experience in current position, accuracy improvement and biases 257

Table 85: Gender, accuracy improvement and biases 258

Table 86: t-test results for gender differences 258

Table 87: Education, accuracy improvement and biases (means) 258

Table 88: Age, accuracy improvement and biases (means) 259

Table 89: Daily sales and accuracy improvement 260

Table 90: Daily sales and biases 260

Table 91: Mean daily sales and accuracy improvement 261

Table 92: Mean daily sales and biases 261

Table 93: Effects of situational variables on accuracy improvement and biases 262

Table 94: Summary statistics for Type A and Type C accuracy improvement 269

Table 95: Summary statistics for optimism, conservatism and overreaction biases 270

Table 96: Summary results obtained in Chapter 5 272

Table 97: Comparison of occasional and frequent users 306

Table 98: Cronbach’s alpha measures 308

Table 99: Fit measures for SONSO inventory model 310

Table 100: Fit measures for Work Preference Inventory model 312

LIST OF FIGURES

Figure 1: A classification of forecasting methods 3

Figure 2: Decision guide for combining forecast methods 6

Figure 3: Stepwise adjustment of statistical forecasts 8

Figure 4: Sample screenshot from the software used by the research company 9

Figure 5: A simplified version of the working model 14

Figure 6: An expanded lens model 48

Figure 7: Brunswik’s Lens Model 63

Figure 8: Interrelationships between person, actions and environment 68

Figure 9: Hogarth’s Conceptual Model of Judgment 69

Figure 10: Example of a salesman’s pricing decision 70

Figure 11: A general information processing model 73

Figure 12: A Model for Personality and Work Performance 77

Figure 13: A working model of judgmental adjustment of statistical forecasts 82

Figure 14: Various types of forecast adjustments 89

Figure 15: Motivation and performance 105

Figure 16: The relationship between motivation and independent variables 112

Figure 17: Stepwise adjustment of statistical forecasts 156

Figure 18: Sample screenshot from the software used by the research company 158

Figure 19: Cover page of the survey instrument 163

Figure 20: SONSO personality inventory as used in the survey instrument 167

Figure 21: WLOC scale as used in the survey instrument 169

Figure 22: WPI scale as used in the survey instrument 173

Figure 23: Questions about forecasting methods 174

Figure 24: Questions about reward structures 174

Figure 25: Questions about perceived incentive 175

Figure 26: Questions about information sharing 175

Figure 27: Questions about feedback 175

Figure 28: Questions about accountability 176

Figure 29: Questions about data presentation 176

Figure 30: Questions about invested effort and demographics 176

Figure 31: Total number of forecast adjustments during the 12 month study period 178

Figure 32: Average daily sales by month 185

Figure 33: The distribution of average daily sales among stores 186

Figure 34: Standard deviation of daily sales by month 187

Figure 35: The distribution of standard deviation of daily sales among stores 188

Figure 36: Distribution of adjustment frequency among store managers 197

Figure 37: Distribution of Type A average accuracy improvement among all store managers and respondents 201

Figure 38: Distribution of Type C average accuracy improvement among all store managers and respondents 201

Figure 39: Distribution of optimism bias among all store managers and respondents 209

Figure 40: Distribution of conservatism bias among all store managers and respondents 209

Figure 41: Distribution of overreaction bias among all store managers and

respondents 210

Figure 42: Direction operationalized in the survey instrument 228

Figure 43: Intensity operationalized in the survey instrument 228

Figure 44: Survey questions related to motivational stimuli 234

Figure 45: Perceived financial incentive and accuracy improvement 237

Figure 46: Perceived social incentive and accuracy improvement 237

Figure 47: Enjoyment as challenge and accuracy improvement 238

Figure 48: Enjoyment as other than challenge and accuracy improvement 238

Figure 49: Positive attitude towards accountability and accuracy improvement 239

Figure 50: Negative attitude towards accountability and accuracy improvement 239

Figure 51: Perceived supervision and accuracy improvement 240

Figure 52: Survey questions related to information sharing 246

Figure 53: Survey questions related to feedback 247

Figure 54: Questions related to presentation of information and presentation of adjustments 250

Figure 55: Changes in accuracy and biases with lead-time 254

Figure 56: Age, accuracy improvement and biases 259

Figure 57: Confidence interval statistics for Type A accuracy improvement 304

Figure 58: Confidence interval statistics for Type C accuracy improvement 304

Figure 59: SEM of SONSO inventory 309

Figure 60: SEM of Work Preference inventory 311

CHAPTER 1 INTRODUCTION

The practice of judgmentally adjusting forecasts that are generated by quantitative methods is a common one in business As such, the success of many managerial decisions depends on the accuracy of forecasts that are judgmentally adjusted Furthermore, the significance of this practice has led to proliferation of academic studies on this subject Hence, a better understanding of judgmental adjustments is of interest to researchers and practitioners This research is an investigation of judgmental adjustments of statistical forecasts in terms of accuracy improvement, learning and biases

1.1 Background

Forecasting accuracy is important for a single firm as well as for members of a supply chain First, forecasting provides essential information for many managerial decisions Managers take risks based on the information generated by the forecasting system, which may have significant effects on the financial well-being and competitive position of a firm As such, improving the accuracy of forecasts is an important goal for a company Second, the effects of poor forecasting transcend the boundaries of a single

firm and reach other members of the supply chain Forecast errors of a firm cause larger fluctuations in demand for the suppliers in the upstream tiers of the supply chain This phenomenon is referred to as the “amplification of demand” (Forrester 1961) or the

“bullwhip” effect (Lee, Padmanabhan and Wang 1997)

The need for increased forecast accuracy has fostered interest in better forecasting methods both in academia and in industry There is a significant body of literature that proposes many alternative forecasting methods However, all of these methods can be grouped in one of two categories: Statistical or judgmental (Armstrong 2001; see Figure 1) Statistical methods are referred to as objective or quantitative while judgmental

methods are referred to as subjective or qualitative Both categories of forecasting

methods have their advantages and disadvantages While statistical forecasting methods can handle large data sets with objectivity and consistency, they may ignore important contextual information and assume that future events will mimic past trends Judgmental methods, on the other hand, make use of both quantitative and qualitative data even though they are inefficient in dealing with large data sets, and can be subjective and inconsistent Therefore, forecasters often face a choice between statistical and judgmental methods when implementing a forecasting system

Figure 1: A classification of forecasting methods

When faced with a choice between statistical and judgmental forecasting

methods, some forecasters take a hybrid approach where they combine statistical and judgmental methods in order to improve the accuracy of the final forecast One

combination method is to use judgmental adjustments to statistical forecasts (Armstrong 2001) In this approach, statistical forecasts are generated using historical data and are adjusted based on the judgment of the forecaster(s) This way, the efficiency of statistical methods in processing large data sets is combined with the contextual information that can be added to the final forecast results in the form of managerial and/or expert

Statistical Judgmental

Combine forecasts Adjust judgmental forecast Adjust statistical forecast

accuracy of the judgmental adjustments However, current literature lacks in some

fundamental ways First, there is a need for a theoretical model of the process that

produces judgmental adjustments so that various aspects of judgmental adjustments can

be studied systematically Second, in addition to accuracy improvement, the effects of learning and biases should be studied Third, studies should preferably be conducted in a business setting so that they are more generalizable

The purpose of this research is to address the previously mentioned shortcomings

in the literature In other words, it is to deepen the understanding of judgmental

adjustments by (1) proposing a working model for the process which generates

judgmental adjustments to statistical forecasts, (2) exploring certain variables that affect the accuracy, learning and biases in judgmental adjustments Moreover, data from an actual business setting will be used in order to make the results more generalizable

1.1.1 Theoretical Background

Researchers have recognized the significance of judgmental adjustments of

statistical forecasts As a result, studies have been conducted both in experimental

settings and in business settings While these studies provide some basic insights about judgmental adjustments, many researchers maintain that further research is needed to explore numerous aspects of judgmental adjustments of statistical forecasts

A fundamental finding in previous studies is that human intervention in the form

of judgmental adjustments can help improve accuracy of statistical forecasts in two ways First, it can detect pattern changes Sanders (1992) shows instances when judgmental adjustments can improve accuracy and reduce biases by recognizing the patterns in data

and incorporating that information into the statistical forecast Second, it can incorporate expert knowledge about the data series when a statistical forecasting method ignores such information For example, an industry expert can add customer, product and market knowledge to a statistical sales forecast Sanders and Ritzman (1991) studied the effects

of pattern changes in data on the accuracy of statistical forecasts and judgmental

adjustments by practitioners Their results show that forecasts revised by practitioners are superior to statistical forecasts in estimating the magnitude, onset and duration of a

pattern change

Most statistical forecasting methods that use historical data assume that the data are based on an underlying pattern such as level, trend, seasonality or a combination of these Furthermore, these models assume that the past patterns will repeat themselves in the future However, in practice, there are abrupt changes in the environment that cause the patterns to change substantially so that the statistical forecasts lose their initial

accuracy Forecasting practitioners are well aware of this problem In a survey of

forecasting experts, Collopy and Armstrong (1992) identified the ability of a statistical forecasting method to deal with abrupt changes in the underlying patterns of data to be one of the most useful criteria in selecting extrapolation methods They further concluded that most statistical forecasting methods fail to provide such a capability Therefore, if a statistical forecasting method fails to recognize a shift in the underlying data patterns, it would be a sensible policy for the forecaster who detects this shift to incorporate the necessary information into the statistical forecast by a judgmental adjustment

Goodwin (2002) provided a set of guidelines for combining management

judgment with statistical methods to capture the synergy of complementary strengths of

each approach (see Figure 2) While he based his recommendations on the literature, he acknowledged that much research needed to be done and qualified his recommendations

as tentative

Adapted from Goodwin (2002)

Figure 2: Decision guide for combining forecast methods

Statistical

Forecasting

Series Type

Corrected Judgmental Forecasts regular irregular

Type of Period

Regular with special events

Statistical Forecasting normal

Past hard data

on special circumstances

special

Statistical Forecasting high

low Availability

Data on past judgmental adjustments and outcomes

for special periods

Correct and combine

high Availability

Likely impact of special events Impact low Statistical Forecasting

Judgmental Adjustment

low

high

1.1.2 Business Background

The empirical aspects of this research are based on the practices of the research company1 The research company is a member of The Global Supply Chain Forum based

at The Ohio State University Management makes extensive use of judgmental

adjustments of statistical forecasts in its daily operations The research company owns a large number of retail stores Due to the competitive nature of its business, the research company places great importance on improving forecast accuracy Management of the research company and researchers at The Ohio State University have agreed to support a research study to better understand and improve the process of judgmental adjustments

In the research company, judgmental adjustments were generated as follows: Every week, the supply chain management department at the headquarters generated sales forecasts using a commercially available software package which utilized advanced statistical methods These forecasts covered the next two weeks and were based on the historical sales data stored in the data warehouse of the research company The data warehouse contained up to two years of point-of-sale data for most of its products

Afterwards, the marketing department made necessary forecast adjustments based on new information such as new product introductions, price promotions, and store openings or closings Once these changes were made, the revised forecast was relayed to the store managers so that they could incorporate any additional information at the local level, such as weather conditions, local events and road closures Management wanted to better understand the process of judgmental adjustments so that the accuracy of the adjustments

1 For confidentiality reasons, the real name and any identifying information of the company that supported this research are withheld in this document This firm will only be referred to as “the research company”

made by store managers could be further improved This process is depicted in Figure 3 For illustration purposes, a sample screenshot of the software used by the research

company is given in Figure 4

Figure 3: Stepwise adjustment of statistical forecasts

Generated using historical sales data

Forecasts adjusted based on market and product information

Forecasts adjusted based on local information

Note: Certain parts of the screen shot were masked for confidentiality reasons

Figure 4: Sample screenshot from the software used by the research company

1.2 Research Design

This section describes the approach taken in the research More specifically, it outlines the goals and methods of the research as well as the working model and the relevant dependent and independent variables

1.2.1 Research Purpose

The purpose of this research is to better understand and improve the efficacy of judgmental adjustments of statistical forecasts The efficacy of judgmental adjustments is conceptualized in terms of three related measures The first measure is accuracy

improvement Accuracy improvement is the difference between the accuracy of the statistical forecast and the accuracy of the adjusted forecast Research has shown that this difference is positive when judgmental adjustments of statistical forecasts are applied in the right way under appropriate conditions The second measure is learning which

measures accuracy improvement over time As a forecaster repeatedly adjusts statistical forecasts by using his or her judgment, the accuracy improvement should improve in time due to a cognitive process called “complex skill acquisition” The third measure is bias A bias is a systematic error in a forecast For example, a forecaster may have an optimism bias where he or she tends to adjust forecasts in an upward direction In addition to

accuracy improvement and learning, the effects of biases on judgmental adjustments of statistical forecasts is a research topic that needs to be investigated

It should be noted that the data set provided by the research company was rich in detail so that a multi-faceted investigation of judgmental adjustments could be conducted First, this data set came from a business setting which improved the validity and

generalizability of the research findings Second, the data set contained a large number of store managers (forecasters) who made judgmental adjustment over a long period of time This study period extended over 12 months This allowed the measurement of learning and biases in addition to accuracy improvement Third, the data set included a

geographically diverse set of store locations covering several states in the Midwest and southeastern regions of the United States

The goal of this research was accomplished in three steps: (1) build a working model that illustrates the relationships between dependent and independent variables, (2) propose hypotheses based on the working model, and (3) test these hypotheses using empirical data obtained from the research company

The second step of this research entailed exploring certain environmental

variables that affect the accuracy of the judgmental adjustments Based on the proposed working model of the judgmental adjustment process, relevant variables were tested for their effect on judgmental adjustments The results of this step deepened the

understanding of the internal functioning of the judgmental adjustment process and provide important input in devising operating policies geared towards improving the accuracy of these adjustments

1.2.2 Model Building and Hypotheses

The first step involved proposing a working model for the process which

generates judgmental adjustments This model provided the groundwork for identifying the major dependent and independent variables, the interactions among the variables and basic cognitive mechanisms utilized Furthermore, this model helped visualize the

internal functioning of the judgmental adjustment process The significance of the

working model derived from the fact that it formed a theoretical foundation for proposing hypotheses First, a theoretical foundation enabled a systematic exploration of factors that affected the effectiveness of judgmental adjustments In other words, important factors

affecting judgmental adjustments were not overlooked In fact, Sanders (2001)

emphasized the need for systematic research on judgmental adjustments Second, a

theoretical foundation provided a means to combine empirical research findings in a coherent way Without a theoretical foundation, one could argue, and potentially find empirical support for, any number of variables that may affect judgmental adjustments However, if these variables are not studied with a prior theoretical framework the results will be very difficult to integrate into a solid body of knowledge For example,

researchers in the area of motivation have complained about the lack of a unified

theoretical framework in studies related to motivation (Judge 2002) Such studies focus

on a narrow set of traits and their effects on motivation However, without a theoretical framework, results of different studies cannot be compared and integrated Furthermore, since behavior is usually determined by an interaction of different traits, one would want

to study the interaction of traits which, in the absence of a theoretical framework, is virtually impossible For all these reasons, it was important that this research be based on

a working model of judgmental adjustments that identified relevant factors and illustrated the interaction among them Since such a framework was not present in the literature, we proposed a working model based on various, more generic models in the cognitive

psychology literature

The literature on cognitive psychology deals with cognitive faculties of humans such as perception, memory and information processing As such, researchers have proposed many models of human cognitive functioning These models describe how humans perform cognitive functions in a generic way that is independent of context Moreover, each model looks at human cognitive functions from a specific point of view

which depends on the research question that a particular researcher wants to answer Consequently, these models were too generic for use in this research

The solution to this situation was to build a working model for judgmental

adjustments that was a synthesis of relevant characteristics of various models in cognitive psychology The working model was based on four different models The first model was the Brunswik Lens Model (Brunswik 1956) which established the relationships between the organism (the forecaster) and the ecology (business setting) Given this background, the second model, Hogarth’s model (Hogarth 1987) illustrated the sequential nature of human judgment The third model by Arthur, Doverspike and Bell (2004) provided further detail in terms of information processing Finally, the work performance model (Robertson and Callinan 1998) tied various environmental and personal variables to task performance The working model was developed as a synthesis of these models and was customized to reflect the specifics of the judgmental adjustments process As a result, the working model was, on the one hand, generic enough to be applicable beyond the

operations of the research company, and, on the other hand, specific enough to accurately capture the relationships between different variables affecting judgmental adjustments

A simplified version of the working model is provided in Figure 4 The inputs to the working model are statistical forecast and contextual information When the store manager generates a judgmental adjustment, he or she also takes into account relevant previous experiences After the judgmental adjustment, actual sales are observed and forecasting performance can be determined in terms of accuracy improvement, learning and biases The information generated by this outcome is added to the memory of the store manager to be used again in future decisions The judgmental adjustments and the

memory processes are both moderated by personal and environmental variables which include intelligence, personality, motivation and situational variables

Figure 5: A simplified version of the working model

The dependent variables studied in this research were (1) accuracy, (2) learning and (3) biases Accuracy of judgmental adjustments refers to the “closeness” of a forecast

to the actual value at a given point in time Hence, the accuracy was analyzed from a cross-sectional perspective The learning effects, however, are longitudinal in nature,

Perception Processing Execution

Contextual Information

Statistical Forecast

Outcome

Memory Feedback Past Performance

Personal and Environmental Variables

Intelligence Personality Motivation Situation

Accuracy Improvement Learning

Biases

since learning occurs over time when the forecasters gain experience and their

performance improves In other words, learning can be described as the positive change

in accuracy improvement over time Biases are systematic deviations in judgmental adjustments Normally, errors should display a random behavior However, if there are errors that show some systematic recurrence then there may be a bias in judgmental adjustments The nature of the bias is expected to reveal the source of bias which can be addressed to eliminate this particular bias and improve accuracy

The working model’s conceptualization of judgmental adjustments led directly to the identification of independent variables to be investigated in this research The

working model viewed judgmental adjustments as a cognitive task, as a judgment and decision-making task and as a repetitive and complex task As such, three streams of research were reviewed to identify independent variables The first stream was related to judgmental adjustments and this review can be found in Chapter 2 The other two streams

of research were human judgment and task performance These findings are integrated into hypotheses proposed in Chapter 3 along with the findings reviewed in Chapter 2

1.2.3 Data Collection and Analysis

Data collection for this research was composed of two parts The first part

consisted of retrieving forecasting data from the research company’s data warehouse The data warehouse stored records of statistical forecasts, judgmental adjustments and actual sales These records allowed the calculation of dependent variables, that is, accuracy improvement, learning and biases The second part of the data collection was the

administration of a survey instrument among certain regions of the research company’s

stores The survey instrument focused on the measurement of independent variables After both parts of data were collected, survey responses were matched with computer records and the ensuing analysis tested hypotheses proposed

1.3 Limitations

As with any research, this study has certain limitations First, the data for

judgmental adjustments was taken from a single company operating under specific

conditions In other words, it is possible to make judgmental adjustments in a variety of slightly different ways However, the practice at the research company followed a

specific pattern As such, the results may not be generalizable beyond the characteristics

of the specific task performed by the store managers of the research company (see Table 1)

Characteristic The research company Other Possibilities

Nature of Variable Uncontrollable Controllable

Type of adjustment Unstructured Structured

Adjustment Type By percentage, by dollar amount

Table 1: Characteristics of the judgmental adjustment at the research company

For example, in the current research setting the forecasting was done for term sales However, the forecast horizon may be different in other business settings, which may potentially impact the external validity of results This concern was addressed

short-in two different ways: First, the hypotheses were based on well-established theory of cognitive psychology Therefore, the findings in this research are not mere statistical coincidences Second, the results can be compared with other pervious studies on

judgmental adjustments Ultimately, however, there is always a chance, albeit slim, that under different conditions the current results would suffer in terms of external validity

A second limitation of the research is that some variables remained outside the scope of project for practical data reasons A large number of variables affect judgmental adjustments, human judgment and task performance Yet, a comprehensive survey

instrument for data collection would be prohibitively large, which would decrease the accuracy of responses because of respondent fatigue Also, a long questionnaire would deter potential respondents from participation as they are busy with their daily tasks As a result, a decision was made to keep the survey instrument short so that it could be

completed in 15 to 20 minutes Although the results of this research may seem

incomplete with the exclusion of some variables, the presence of a working model makes

it very easy to extend the current research by further data collection Since the working model provides a theoretical foundation for the data collected in this research, further data collection and analysis would be easily integrated with the current results Therefore, this limitation can be remedied in the future

A third limitation is that the Big-Five personality model may represent personality traits too broadly to be useful in drawing specific conclusions The so-called “bandwidth-fidelity dilemma” refers to the debate whether broadly-defined or narrowly-defined personality traits are better at predicting work performance and explaining behaviors

Some researchers argue that specific personality traits may not predict overall

performance as well as the broadly-defined traits but specific work outcomes

(performance) are better explained by well-chosen and narrowly-defined personality traits (Hough 1998) According to this argument, considering the specifics of a work environment and choosing explanatory personality traits accordingly would improve the predictive power of the proposed model However, other researchers maintain that

broadly-defined personality traits are comparable across work environments and, thereby, provide a better understanding of the determinants of work performance (Ones and

Viswesvaran 1996) Both arguments have merit in their own respect Ashton (1998) points out that Big-Five may provide adequate predictive and explanatory power for some types of jobs, while other types of jobs may require more specific personality measures to explain and predict performance In fact, as our understanding of human judgments, in general, and judgmental adjustments, in particular, increases, the particular functioning of cognitive units is likely to call for more specific personality measures to explain performance At this stage, it would be wise to start with the broadly-defined Big-Five personality traits and refine the criteria as our knowledge grows

A fourth limitation is that the variables in this research are not manipulated in the sense of a lab experiment Instead, observations are recorded and analyzed using a linear regression model The advantage of a field study is that the study conditions are realistic and representative The lack of complete control on variables can be problematic if the variables do not show a natural dispersion and undermine the power of statistical tests A related limitation is that linear models were assumed unless otherwise noted

1.4 Potential Contributions

The goal was to make both managerial and academic contributions The

managerial contributions derived from better forecasting practices to be designed with indications from this research The academic contributions were a better understanding of judgmental adjustments based on the results of this research

First, these findings can help managers in finding and assigning the “better

suited” employees within their organizations to forecasting tasks The forecasting task may be part of an employees’ overall job responsibilities or only a small number of employees in an organization need make forecast adjustments Thus, there can be

potentially many employees who could be assigned to the forecasting task These tests can help differentiate among different employees in their potential performance based on their personality traits and other characteristics Given the findings from this research, managers can design tests or screening procedures that identify the best qualified

candidate for judgmental adjustments

Second, managers can design training programs for their forecasters based on the findings from this research on how to improve accuracy, how to accelerate learning and how to identify and reduce, and possibly, eliminate biases in forecasting In fact, a