Research "MANAGING SIMULATION BASED TRAINING A FRAMEWORK FOR OPTIMIZING LEARNING, COST, AND TIME " pot

Managing Simulation Based Training A Framework for Optimizing Learning, Cost, and Time Noah Joseph Richmond Chapter I: Risk Exposure and Reduction via Simulation 1 Chapter IA: M

INFORMATION TO USERS

This manuscript has been reproduced from the microfilm master UMI films the text directly from the original or copy submitted Thus, some thesis and dissertation copies are in typewriter face, while others may be from any type of computer printer

The quality of this reproduction is dependent upon the quality of the copy submitted Broken or indistinct print, colored or poor quality illustrations and photographs, print bleedthrough, substandard margins, and improper alignment can adversely affect reproduction

in the unlikely event that the author did not send UMI a complete manuscript and tnere are missing pages, these will be noted Also, if unauthorized copyright material had to be removed, a note will indicate the deletion

Oversize materials (e.g., maps, drawings, charts) are reproduced by sectioning the original, beginning at the upper left-hand comer and continuing from left to right in equal sections with small overlaps

ProQuest Information and Learning

300 North Zeeb Road, Ann Arbor, MI 48106-1346 USA

800-521-0600

® UMI

MANAGING SIMULATION BASED TRAINING

A FRAMEWORK FOR OPTIMIZING LEARNING, COST, AND TIME

A DISSERTATION SUBMITTED TO THE DEPARTMENT OF MANAGEMENT SCIENCE AND ENGINEERING

AND THE COMMITTEE ON GRADUATE STUDIES

OF STANFORD UNIVERSITY

IN PARTIAL FULLFILLMENT OF THE REQUIREMENTS

FOR THE DEGREE OF DOCTOR OF PHILSOPHY

Noah Joseph Richmond February 2003

UMI Number: 3085223

Copyright 2003 by Richmond, Noah Joseph All rights reserved

® UMI

UMI Microform 3085223 Copyright 2003 by ProQuest Information and Learning Company All rights reserved This microform edition is protected against unauthorized copying under Title 17, United States Code

ProQuest Information and Learning Company

300 North Zeeb Road P.O Box 1346 Ann Arbor, Mi 48106-1346 —

© Copyright Noah J Richmond 2003

All Rights Reserved

ii

I certify that I have read this dissertation and that, in my opinion, it is fully adequate in

scope and quality as a dissertation for the degree of Doctor of Philosophy

William Perry

I certify that I have read this dissertation and that, in my opinion, it is fully adequate in

scope and quality as a dissertation for the degree of Doctor of Philosophy

I certify that I have read this dissertation and that, in my opinion, it is fully adequate in

scope and quality as a dissertation for the degree of Doctor of Philosophy

MANAGING SIMULATION BASED TRAINING

A FRAMEWORK FOR OPTIMIZING LEARNING, COST, AND TIME

-Abstract-

This study provides a management framework for optimizing training programs for learning, cost, and time when using simulation based training (SBT) and reality based training (RBT) as resources Simulation is shown to be an effective means for

implementing activity substitution as a way to reduce risk The risk profile of 22 US Air

Force vehicles are calculated, and the potential risk reduction is calculated under the

assumption of perfect substitutability of RBT and SBT Methods are subsequently developed to relax the assumption of perfect substitutability The transfer effectiveness tatio (TER) concept is defined and modeled as a function of the quality of the simulator used, and the requirements of the activity trained The Navy F/A-18 is then analyzed in a case study illustrating how learning can be maximized subject to constraints in cost and time, and also subject to the decision maker’s preferences for the proportional and absolute use of simulation Solution methods for optimizing multiple activities across shared resources are next provided Finally, a simulation strategy including an operations planning program (OPP), an implementation program (IP), an acquisition program (AP), and a pedagogical research program (PRP) is detailed The study provides the theoretical tools to understand how to leverage SBT, a case study demonstrating these tools’

efficacy, and a set of policy recommendations to enable the US military to better utilize SBT in the future

IV

In Memory of Gian-Carlo Rota

` 1932-1999

Managing Simulation Based Training

A Framework for Optimizing Learning, Cost, and Time

Noah Joseph Richmond

Chapter I: Risk Exposure and Reduction via Simulation 1 Chapter IA: Military Personnel Risk Exposure and Risk Reduction Strategies 2 Chapter IB: Risk Assessment and Impact of 1:1 SBT:RBT exchange 3 Chapter II: Transfer Effectiveness Ratio and Total Training Resources 4 Chapter III: Optimization Framework and Computational Form 5

Chapter IVA: F/A-18 Training Program and Formulation 6 Chapter [VB and Chapter [VC: Formulation and Solution Methods of the LATT

Chapter IVD: Formulation of the Multi-Activity Strategic F/A-18 Optimization 8

Risk Reduction of Military Aviation through Simulation 10

If SBT is a perfect substitute for RBT, how should SBT be used? 14

The Link between simulation and reality based training: TER 33

vi

Boundary Solutions

Simulator Types and Measures of Quality

Measures of Quality: Evaluating Vision and Acceleration Systems

Solution Methods and Solution Properties

Parametric Solutions: Simultaneous Cost and Time Constraints

Chapter IV D

Combining PMA Readiness Functions to form Total Readiness

Interpreting Results: The Implied Allocation Model (AM)

Components of a Simulation Strategy

Executive Summary: Simulation Strategy

This study provides a management framework for understanding the impact of simulation based training Chapters [ through IV provide the analytic machinery necessary to

analyze, formulate, and optimize training programs This chapter previews and

summarizes the principle findings of the study Chapter V closes with recommendations for present policy planning, and future research

Organization and Principle Findings

The chart below illustrates the organization of the study along single activity (“micro”) and complete training program (“macro”) lines, and between theory and application

primary findings of each chapter is next provided

Chapter |: Risk Exposure and Reduction via Simulation

Chapter I introduces risk exposure and reduction methods First, historical trends in

accident rates, and strategies for risk reduction are described Then a numerical analysis

of risk, forecasting total losses in life and vehicles is conducted for a five year horizon It

is shown how simulation can be used to reduce this risk exposure, answering the

question, “If SBT and RBT are perfect substitutes, how should SBT be used?” The

chapter thus provides a descriptive analysis of current risk exposure, strategies for its reduction, and a first-cut analysis for the cost savings that can be achieved using

simulation

Chapter IA: Military Personnel Risk Exposure and Risk Reduction Strategies

The military has seen a dramatic reduction is risk exposure in the last twenty years In

1980, 2400 servicemen died The Air Force alone lost 350 due to unintentional injury By

1993, 1300 lives were lost The Air Force cut losses due to accidents to 100 persons While total accidental deaths fell by 71%, the accidental death rate fell a less dramatic 58% because the military downsized

The loss reductions were driven by improved training doctrine and equipment reliability

Though accident rates have been halved throughout the force, risk exposure remains high, and more recent gains have leveled off Today, five military personnel per month die in aviation accidents Over ten years, these deaths exceed combined losses from the Gulf War and military operations in Kosovo Training is now riskier than war; accident rates are sufficiently high to threaten operational tempo While the rate averages five per month, there are periods when three or four times the average may occur nearly

simultaneously When multiple accidents occur, operations must be addressed politically

in order to maintain confidence in the military

It is therefore imperative to find new ways to reduce risk exposure Chapter JA reviews

strategies for risk reduction, including activity substitution This strategy replaces high risk activities with lower risk substitutes, while maintaining peacetime readiness

Simulation based training (SBT) is one means of implementing activity substitution Simulation provides a synthetic training environment that can closely replicate high risk activities It is nearly risk free, and is inexpensive compared to live flight Understanding the value of simulation is therefore of key importance to policy analysts—and the stated goal of this study

Chapter IB: Risk Assessment and Impact of 1:1 SBT:RBT exchange

If SBT and RBT are perfect substitutes, then one training hour of SBT can be used to

replace one training hour of RBT without impacting the quality of learning that takes place The risk associated with the live flight hour is eliminated, replaced by the (nearly) risk free simulation training hour—when a simulator is “crashed” there is no loss of life

or equipment! While the assumption of perfect substitutability is naive, it is a good point

of departure Chapter IB provides an analysis of risk reduction through the use of

simulation under this assumption The remainder of this study provides methods for analyzing the more complex problem where training methods have different pedagogical

values, costs, and exercise times

Chapter IB begins by showing that time, i.e a training hour, is the proper measure of the independent variable Loss rates, the dependent variables, are then defined as the average number of deaths per training hour The loss rates are calculated for each of 22 vehicles

in the US Air force by analyzing their complete historical risk profiles Because loss rates

change over time, it is necessary to model the trend in these rates The cumulative

average loss rates are calculated, first for loss of life, and then for cost of capital These

rates are combined to forecast total losses for each vehicle type for a five year horizon

This analysis is referred to as the Single Vehicle Loss Model (SVLM), and measures how risky each vehicle platform is using a single summary statistic

These SVLM statistics are then combined in the Multi-Vehicle Loss Model (MVLM) to

show how losses can be reduced through the use of simulation The MVLM shows that total forecast losses for the five year period 2002-2006 is $3.3 Billion Savings can be made by substituting simulation based training for live flight Because simulation is assumed to be a perfect substitute, the optimal solution is degenerate—all live flight should be replaced with simulation because of its cost advantage As a result, two

constraints are added: (1) Proportional and (2) Total use of simulation Respectively

denoted by a and , the first limits the percentage of hours that may be substituted away for any single vehicle, while the latter limits the total number of hours substituted across

all vehicles Prioritizing the riskiest aircraft first, it is possible to calculate attainable

savings as a function of a and B As expected, the larger the total number of hours

permitted, the more savings that accrue However, marginal savings decline with

increased volume, because the optimal approach is to substitute away the riskiest (most costly) hours first

Chapter Il: Transfer Effectiveness Ratio and Total Training

Resources

Much of the debate surrounding the use of simulation today is based upon a lack of agreement on the impact of substituting one training activity for another Chapter I is performed under the restrictive analysis of perfect substitutability, despite the fact that different activities have different pedagogical value, and a method to account for these

contributions is referred to as Total Training Resources (TTR), and learning is modeled

as a function of the single variable TTR

Chapter [I provides methods for modeling and assessing TER The analysis proceeds in four steps The first is the definition of TER, which is expressed as a simple

multiplicative coefficient The second part discusses the factors that determine TER The discussion is split into physiology and technology The former provides an organizing framework for understanding how trainees experience the stimuli needed to create a training experience The latter describes how technology can provide these stimuli Taken together, these two pieces give a complete picture of how technology choices dictate the quality of training, and hence pedagogical value of different training methods The choice

of training method thereby leads to predictable outcomes—the amount of learning

accomplished

The third and final step discusses current methods for assessing TER Advantages and disadvantages of three current methods are discussed: (1) Pilot and expert opinion, (2) In-

simulator learning, and (3) Direct transfer In addition, a new method, termed ‘logistic

analysis,’ is developed It models TER as a function of two variables, Frequency and Quality Frequency is the rate at which practice cycles can be completed, and Quality is a

composite measure of the capabilities of the technology to deliver realistic synthetic stimuli This method for TER estimation thus builds on the previous analysis of

physiology and technology, and is the only method that separately accounts for the

quality and quantity of training delivered

Chapter Ill: Optimization Framework and Computational Form Chapter III describes the management framework in which complete training programs are modeled and optimized The chapter has two parts Part A formulates the objective

function and constraints in their most intuitive form Part B provides an equivalent

program more suitable for computation

Chapter IIIA: Optimization Framework

The optimization framework provides a method for the efficient trading off of learning, cost, and time for the training of a single activity Mathematical definitions of learning,

cost, and time are provided Each is described as a function of the types and quantities of

training resources consumed In addition, the constraints a and B on proportional and total

use of simulation are formalized The equations provide a framework in which one of

learning, cost, and time can be optimized subject to constraints in the other two Solving the program results in the designation of a training program where improvement in the objective function cannot be attained without increasing the resources available, or easing restrictions on @ and/or B

Chapter I1IB: Computational Form

The optimization of principal interest to the military is maximizing learning subject to constraints in cost and time Learning, L(x), is designated as the objective function By construction, L(x) is a composite function consisting of a concave function of a linear function of x As a result, the embedded linear function can be maximized instead of L(x) Because the constraints in cost, C(x), and time, T(x), are linear, minor algebra results in a linear program of the form:

Maximize f (x)

sử

Ax<b x20

Referred to as the “computational form,” this program can be solved rapidly using

simplex or any other basic solution methodology

Chapter IV: Case Study and Algorithmic Techniques

Chapter IV applies the theory of Chapters II and II in a case study of the Navy’s F/A-18 The principle strike fighter deployed on aircraft carriers, the F/A-18 forms the backbone

of sea-based, power projection capabilities Because of it’s sheer numbers, and

comparatively high risk profile, the F/A-18 is an ideal subject for a case study The analysis proceeds in four steps First, a thorough review of the Navy’s current training program for the F/A-18 is provided The Low Altitude Tactical Training (LATT) activity

is then formulated in the optimization framework A numerical analysis for LATT

follows Chapter IV closes by showing how multiple activities can be simultaneously optimized over a set of common resources so as to maximize total readiness

Chapter IVA: F/A-18 Training Program and Formulation

Chapter IVA provides a detailed overview of the Navy’s training program for the F/A-18 The Inter-deployment Training Cycle (IDTC) is a structured set of exercises that the Navy uses to train its aviators across each of seven major units of skills These blocks are referred to as primary mission areas (PMAs) Each consists of critical functions, or

“activities,” of the F/A-18 By modeling activity level readiness as a learning function L(x) in the optimization framework, the tools of Chapters II and III can be applied

Chapter IVA also describes the Navy’s simulator assets, and shows how they are

modeled as training resources in the optimization framework The TER for each

simulator when applied to Low Altitude Tactical Training (LATT) is determined using the logistic analysis method Provision of the cost of training for each method completes the necessary data sets Chapter IVA concludes by providing the general form for the LATT tactical optimization program

Chapter !VB and Chapter IVC: Formulation and Solution Methods of the LATT Activity in the (L,C,T) Framework

Chapter IVB takes up where Chapter [VA ends, translating the general form of LATT

into its computational form Learning is designated as the objective function, subject to constraints in cost and time Chapter [VC then demonstrates solution methods The numerical analysis begins by parametrically maximizing learning subject to simultaneous constraints in cost and time The program is first optimized with a and B unrestricted, and

subsequently repeated with a gradually restricted and B free, and then B restricted while a

is free The step by step analysis shows how the constraints interact to select the optimal training program Two principle concepts arise: (1) The efficient frontier, and (2) a

solution heuristic

The efficient frontier is found by plotting the training methods on two dimensions: TER and Cost The efficient frontier is the boundary formed by taking convex combinations of those points with the best combinations of TER and Cost When a and B are unrestricted, the training methods selected must lie on the efficient frontier The efficient frontier is a useful concept for understanding which technologies offer advantageous TER-Cost combinations in the framework of a system of training platforms

The solution heuristic provides a way for understanding what solutions on the efficient

frontier will be selected, and how constraints in a and B will cause optimal solutions to

deviate from the efficient frontier When a and B are free, problems in which time is scarce will favor methods on the efficient frontier with a high TER ratio When budgets are tight, methods with a high TER/$ ratio will be preferred The heuristic states that as a

is constrained, the most efficient (i.e., closes to the efficient frontier) RBT methods will

enter the solution mix When B is constrained, RBT may or may not be used Together, these principles provide a way of looking at the training methods on a simple graph, and then understanding how the program’s constraints shape the optimal solution

Chapter IVD: Formulation of the Multi-Activity Strategic F/A-18

Optimization

The final piece of the methodology puzzle is to show how several different activities can

be optimized over a shared set of resources The total readiness (TR) function shows how

the objective functions L(x) of the individual activities can be joined into a single

strategic objective function It is simply a linear combination where the weights assigned

reflect the relative importance of the activities trained: TR(Z) =Ze L(x)

The constraints of the multi-activity (strategic) level problem are analogous to the single activity (tactical) problem Total cost and time constraints are enforced as limits on the sum of the individual costs and time limits Likewise, a and B limit the proportional and total use of simulation across all activities Capacity constraints for total available hours

on each training method are added

Two properties follow from this structure The first is the Strategic-Tactical Optimality

Condition (STOC) This condition states that the solution to the strategic program

consists of solutions to the tactical programs that are optimal when viewed alone The property holds when the constraints of the tactical problems are set to the values implied

by the strategic solution For example, if the strategic solution requires that $4,000 and five hours be spent for Low Altitude Tactical Training, then the tactical solution is

optimal when viewed alone, under respective cost and time constraints of $4,000 and five hours If there was a better solution, i.e one that produced more learning under these constraints, then it would have been found during the strategic optimization

The STOC property is useful because it leads to the Implied Allocation Model (LAM) This model describes strategic optimization as occurring in two steps First, the strategic constraints are “allocated” across the individual sub-problems For example, the total cost budget is divided across the individual tactical models, each of which may use up to its

allocated portion of the total available budget Second, after allocation, each model is

optimized given its now fixed resource constraints Optimal solutions for the strategic model can be found informally by trying different allocations, or through a standard search algorithm

The STOC property parallels how actual budgeting is accomplished It provides a method for conducting the what if analyses and re-planning current training regimes For

example, the strategic program can be optimized given current budget and time

allocations It can then be re-optimized with these tactical allocations left free The two model runs indicate where budgets should be tightened or increased, where time should

be reduced or increased, and in what training activities simulation should be emphasized

or reduced Guidance on how to change the training program is thereby provided

Closing Remarks

The first four chapters of this study provide a set of analytic tools for planning training programs given resources and stated preferences Tools for understanding what solutions may be had are provided, as are algorithms for finding them However, this study does not make any recommendation as to what the decision maker’s (DM) preferences ‘ought’

to be This is a study that clarifies what ‘is,’ providing a management framework for driving conclusions from the DM’s assumptions It does not attempt to suggest that one set of working assumptions is better or more valid than another Chapter V may appear to depart from this vein because it makes recommendations on what organizational changes are needed in order to successfully use SBT In actuality, these recommendations do not imply any prejudgment of the value of simulation Instead, they provide guidance on how

to use the method effectively Whether or not the method is desired follows first from the preferences of the DM

Chapter |

Risk Reduction of Military Aviation through Simulation

Aviation accidents are a major source of risk for the US military Flight training in

particular is a dangerous business In World War II, 120,000 American aviators were

killed Of these, 15,000—one in eight—died in training.' Today, Accidents are the

number one cause of death amongst US servicemen Accidents have superceded enemy action as the military’s number one source of fatalities, claiming half of all fatalities from the period spanning 1980 to 1993 The charts below show that for both men and women,

accidents amount to a clear majority of deaths, claiming respectively 58.4% and 50.7% of

persons This rate more than doubles that of death due to disease and illness.”

Fuzure Air Force ta Oismi Sut on co Maes = artes by

Cause ofCeam 1930 1993 No5 292

* hchudes deaths O.¢ 20 hustite action (n= ZH} and hose deaths where

Couse ss ther unclessfed_ cr uns mown [ns 18) * Includes ore death where Cause 1s csher

Accidents during flight compose one of the greatest risks undertaken today: It is one of

the major drivers of how our soldiers die, and how skills and proficiency are lost It is

also a critical means by which capital is destroyed Because the other sources of death

(disease, homicide, and suicide) are not associated with major capital losses, accidents are the primary peace time driver for the loss of capital equipment Peacetime losses from the last 10 years exceed the combined losses of the 1991 Persian Gulf and Kosovo air campaigns

The degree to which the problem of accidental death can interfere with military

operations cannot be overstated Beyond even the loss of life and capital equipment— accepted risks in military service—is the strategic issue of maintaining operational

tempo Twice in the past five years the military has temporarily ceased non-essential operations precisely because these costs became too high The strategic military mission

of maintaining readiness was compromised This limitation has the potential to

significantly undermine the capability of the US military; the most recent stand down

came during the war in Afghanistan, illustrating the seriousness of this problem

Flight Training: Key Element of Aviation Accidents

Training accidents are a key driver of aviation accidents In the period spanning 1984 to

1994, 1523 aircraft crashed and 1600 lives were lost in training exercises Pro-rated, lives lost during flight training amount to over half of those lost through unintentional injury, hence over 25% of all fatalities There are approximately 30,000 pilots in the armed

forces today From an operations perspective, there are typically 1.5 major accidents for

every 100,000 flight hours As a result of the force size, operational tempo, and

associated risk of flight training, accidents resulting in death during flight accrue at a rate

of about five aircraft per month.* The costs of these accidents are enormous Consider the losses due to flight training alone: At an estimated loss rate of 60 planes and 100 lives per year, with an aircraft cost of $30 million, life value of $2 million, and loss of skills of $5 million per person, the total cost of these accidents amounts to $2.5 Billion per annum The strategic value of op tempo is perhaps an order of magnitude greater

3 http://www.cnn.com/2002/US/02/08/air.force.safety/index.html

See also http://www.cnn.com/US/9709/17/military.crash/index.html

* This statistic does not take into account type of aircraft or missions flown, but conveys the general level

of risk faced Note that a Class A mishap, in which the aircraft is essentially lost or destroyed, typically results in more than one death Hence the loss rate of five aircraft per month results in 60 aircraft per year, and 100 lives per year

Il

Strategies for Risk Reduction

The US military faces a clear problem that can be summarized in three key points:

(1) Unintentional accidents are responsible for the majority of deaths of US

There are three basic strategies that the military can pursue to offset these losses:

(1) Reduce the frequency of activities that lead to risk

(2) Make the activities that lead to risk safer

(3) Find alternatives to the activities that lead to risk

Option (1) is essentially the concession of op tempo: The reduction of lost life and

equipment at the expense of reduced readiness.” Options (2) and (3) are preferable

because they attempt to maintain readiness while simultaneously reducing risk The rigorous pursuit of option (2) explains the past reduction in the rate of unintentional accidents By all accounts, this strategy has been successful, accounting for a 25%

reduction in the total rate of death amongst servicemen from the early eighties Previous gains have come through the improvement in the reliability of military equipment, and improved safety precautions in training doctrine However, the low-hanging fruit has been picked—there are fundamental risks that cannot be engineered away, either through improved reliability or operations management Unless the activities undertaken during mission rehearsal are significantly changed, it will be difficult to further improve safety Hence we turn to option (3): Reduced Risk Training

Reduced Risk Training

According to the military credo, “We train as we fight,” pilots learn by doing The

consequence is that high quality training equates to risky, high cost training—a cost appraised in billions of dollars per year Pursing option (3) requires that the military find new ways to train its personnel that have the same pedagogical value as the current high- risk training methods but with a lower risk profile Simulation is one approach, but has met with resistance

Juxtaposed to the military credo, ““We train as we fight,” the concept of substituting low

risk activities for high risk activities—including the use of simulators—is anathema

Two assumptions typically bolster this viewpoint: (1) Reality based training is “more real” than Simulation based training, and (2) More realism is better However, both assumptions are patently false Empirical evidence shows that (1) Reality based training need not be “more real” than Simulation based training, and (2) More realism need not have any pedagogical value whatsoever—it can, in fact, be counter-producti ve." The Gordian knot that links high quality training to high risk activities can be cut Activity substitution, specifically simulation, offers a means to this end However, it must be used appropriately, or the premonition of inadequate training may be fulfilled This chapter

sets forth to answer the following critical question: How can simulation be used to reduce

the risk of training, and what is its potential value?

Estimating the Value of Simulation Based Training

Simulation Based Training reduces risk exposure by substituting an essentially risk free activity for a high risk activity By reducing the amount of real flight training, the

military can reduce the number of accidents that occur, and the subsequent loss of life, expertise, and capital This study next calculates the exact savings gained by using SBT

as a substitute for RBT Real data is used The number of lives and dollars saved are calculated as functions of both the proportional and absolute quantity of SBT used

Perfect substitutability of RBT and SBT is assumed for this first cut analysis

® See Taylor, Lintern, Kunde, Tschopp, and Talleur, “Effects of Scene Content, Field of View and Amount

of Simulator Training in First Officer Training.” Appearing in Flight Simulation Technology, Capabilities, and Benefits, [7-18 May 1995, pp 7.1-7.11

13

If SBT is a perfect substitute for RBT, how should SBT be used?

If SBT is a perfect substitute for RBT, then the calculation of how much SBT to use is direct If we replace one unit of RBT with one unit of SBT, then no change in readiness occurs However, the risk associated with the unit of RBT is removed, replaced by the risk associated with the unit of SBT added Because SBT leads to neither loss of life nor capital equipment loss, the risk associated with SBT is zero Thus every unit of SBT substituted in effectively removes the risk of one unit of RBT From this fundamental relationship we can build a function that shows the dollars and lives saved as a function

of the amount of SBT used (in place of RBT) This analysis is carried out in 6 basic steps:

(1) Define the independent unit of measure

(2) Define a loss rate associated with the independent unit of measure

(3) Calculate the single vehicle loss rate for human life

(4) Expand the single vehicle loss rate to include the cost of lost capital

(5) Combine data from the single vehicle loss models to generate summary statistics

of losses across aircraft types

(6) Combine data from the single vehicle loss models and multi-vehicle loss models

to calculate risk reduction as a function of proportional and absolute use of SBT

This calculations tells the DM exactly what the impact of different SBT usage policies will be on cost through loss of life and equipment The policy maker can then make explicit tradeoffs between the use of SBT and RBT, and anticipate the results

The Independent Unit of Measure: Flight Hours

In order to determine how to reduce risk through activity substitution, it is necessary to have a, ‘common currency,’ for the exchange of units of one activity for those of another The essential criterion is that each activity unit should have associated with it a standard quantity of risk Then substituting units of one activity for units of the other will have a predictable effect on the amount of risk incurred Without having a standard unit of activity, and an associated standard quanta of risk, there can be no rational trade off of risks through changes in activity levels The selection of the unit of measure determines

14

how this exchange will be made The independent unit of measure used in this study is flight hours

Estimable Loss Rates

Risk is a Statistical phenomena Showing that a flight hour has associated with it a

‘standard’ quanta of risk is complex because it requires the estimation of a probability density: We must calculate the probability of an accident per flight hour We must

calculate a mean and confidence interval for the losses per flight hour In mathematical terms, the loss rate is considered ‘estimable’ if the mean can be calculated with low or ne

bias (accuracy), and the confidence interval is small (precision) In the real world,

‘enough’ accidents have to occur for us to be able to develop the first order mean and

standard deviation statistics

Unfortunately (from an analytic perspective), the loss rate changes over time As a result,

the mean—a function of time—cannot be computed by taking the simple average of

accidents over flight hours Likewise, the standard formulae for the standard deviation for stationary means cannot be applied Fortunately, the rate of loss for most aircraft

approaches an asymptotic limit As a result, the mean at any time T can be calculated as

the cumulative mean—the total number of accidents over the cumulative flight hours

The benefit of this method over a stationary mean is that it illustrates the change in

accident rates over time, and that the emerging trends can be forecast The ability to forecast future loss rates is critical, because the policy decisions that this study informs are designed around the amount of simulation that should be used now and in the future While this method will slightly overestimate the accident rate at time T (because the function is approaching an asymptote from above), the use of an unbiased moving

average is precluded by the relative rarity of accidents Such a measure would be too

volatile to be useful, or to forecast

15

The Single Vehicle Loss Model

We next describe how to estimate and

forecast the loss rates for a single vehicle

We first show through graphical inspection

that risk is an estimable function of flight

hours We then validate this explanation

with computational results

Our objective is to forecast the total number of deaths that will be incurred by a particular aircraft type for the next five years The calculation is computed in two parts First, we forecast the number of flight hours for each year, and the rate of death for each year Second, these forecasts are used to compute the total number of forecast deaths as the vector product of the flight hours and death rates:

flight hours ¢ death rates

Forecast Deaths = -

discount factor The discount factor is included in this model to reflect the change in real value of

nominal losses over time

Flight Hours: Confirming the Risk Measure

We begin with a graphical demonstration of how flight hours and loss rates are related The chart at right depicts the cumulative flight hours and deaths accrued by the A-10 Thunderbolt Both cumulative hours and cumulative fatalities are plotted by year It is

There is a small non-linearity in the relationship between these variables, which is

significant Note that the line depicting cumulative fatalities ‘bows’ out and away from cumulative flight hours This curvature occurs because, with each passing year, fewer accidents accrue per flight hour: The rate of accidents is undergoing a very progressive decline

Estimating the Death Rates

This trend is confirmed with

death rate for cach year, and

the cumulative death rate In

the chart at right, the bars

to exist at approximately I-1.5 deaths per 100,000 flight hours The equations for these statistics are given below:

deaths, Fatality Rate d, = ————_ Oey BBE =F 700,000

> deaths, Cum Ave Fatality Rate d* = —*

The standard deviation of the cumulative average fatality rate is estimated as a

generalized auto-regressive conditional heteroscedastic (GARCH) process The basic equation is 0° =a@eo,,+fe", where £ =d, —d,™” The variance at time t is

calculated as a weighted average of the previous periods’ estimated variance, and the

current one-period observed variance This one period variance is ý”, the square of the

difference of the single period death rate from the cumulative average death rate This difference is interpreted as the one period deviation of the observed value from the estimated mean

The chart at right illustrates the time series d™*, ¢,, and a All three statistics are well

behaved As noted, d,”* follows a gradual exponential decline, then asymptotically

approaches a limit at 1-1.5

deaths per 100,000 flight

hours The series ¢, is

conditionally normal, and

the resulting series o7 is

stable, approaching a mean

value of one (hence

G, approaches one also)

The next chart shows the

time series for d,““ando,,

In conclusion, the estimate of d®* is numerically r yp precise (i.e., confidence is high) The & chart below illustrates confidence intervals centered around a mean death rate of | person per 100,000 flight hours The results indicate that the confidence intervals are valid (i.e., contain only non-negative values) up to a confidence level of 90% These values are sufficient for a first cut analysis Though the estimated loss of life will vary by a factor of

2, the risk levels associated with different aircraft vary by as much as a factor of 100 As

a result, these values allow a definitive prioritization between aircraft The policy maker can identify the riskier aircraft with considerable confidence, a useful tool when

determining how best to leverage SBT

Forecasting Flight Hours and Death Rates

The final step is to model both the future flight hours and death rates, and then to

combine them as a vector product:

flight hours ¢ death rates Forecast Deaths = -

discount factor

For these estimates, the ten year period spanning 1992 to 2001 is used for historical data, and the five year forecast spans the years 2002 to 2006.® Data is fit both using the linear model y =ax+b, and also the exponential model y=e**?

” Setting o = 0.6 when p = | is equivalent to setting o equal to 60% of pt, reflecting the observed

relationship between o, and d,""*

19

The maximum likelihood estimator can be solved for explicitly In the linear case, we have a= no 0Q) and b=Y —aX , where X and ¥ are respectively the

nà x” - (> xƑ

mean of the independent variable x (year), and dependent variable y (death rate, or flight

hours) For the exponential model the calculation ¡s the same, except that we replace the

observed dependent values y with the natural logarithm of these values, effectively estimating In y=ax+b The total error of both models is computed as the sum of

squared errors over the estimated and observed values, and the model with the smaller total error is selected for forecasting

Future values are estimated with the selected model for the forecast years The chart at right depicts the ten year death rates and flight hours for the A-10 The shaded box

indicates forecast values

Forecast Cumulative Average Fatality Rate and Hours

FATAL AVE RATE HOURS |

In the case of the A-10, the trend is clearly linear, and the results are intuitive (the

standard deviation is similarly forecast, but not shown) The final calculation is:

flight hours ¢ death rates discount factor

For the ALO, the estimated flight hours, death rates, and ‘real deaths’ for periods zero through four are shown in the following chart

The estimated 5.86 deaths is the mean forecast deaths By forecasting the standard

deviation of the loss rates, it is possible to estimate a low and high forecast The method

used for this forecast is the same as for the mean loss rate and the number of flight hours: Linear and exponential models are fit, and the series is forecast using the method with the smaller sum of square errors The high (low) forecasts for the loss rates is then calculated

as the mean loss rates plus (minus) the forecast standard deviation.'® The results for the A-10 are illustrated below

The Single Vehicle Loss Model: Summary of Results

The principal value of the single vehicle loss model is that it allows us to quantify,

forecast, and compare the risk attributed to aviation accidents by aircraft type For the A-

10 Thunderbolt, the expected number of deaths over five years is 5.9 persons, with a low

of 1.8 persons, and a high of 10.0 persons This range reflects the mean estimate plus or

minus one standard deviation, hence a confidence interval of 70% The values fall within

one order of magnitude

Because of the enormity of value attributed to a human life, this information is sufficient

to enable effective decision making The questions of what percentage of reality based training should be replaced with simulation based training, and how much simulation should be used in total can be answered The multi-vehicle loss model, subsequently

described, is a method for leveraging this information to inform these decisions

However, we first summarize the findings of the single vehicle loss model across all

airplanes, and then extend the model to allows us to account for the loss of capital

equipment and skills in addition to the loss of life

22

Losses by Aircraft Type

The Single Vehicle Loss Model (SVLM) is used to determine the amount of risk incurred

by each aircraft, and

Mean, Low, and High Death Estimates by Aircraft

across air craft 100

types The chart at % 80 a 1

right illustrates the J 60 — — | : High |

death estimates for 20

0 et etetetetete+ee+ hf tt ttt i

each of twenty two PRSSERSTSLYRERaREe Seas

There are three critical points to observe First, the aircraft fall loosely into three

categories: Low, medium, and high risk The low risk group consists of the T41, B2, C10, C17, TI, T39, C12, F4, C9, F1 L7, and B52 All of these aircraft are predicted to incur losses of fewer than two persons in the next five years The medium risk group consists

of the T43, C141, T37, B1, T38, Al0 and F15 aircraft Each of these aircraft are

predicted to have more than two, but fewer than 10 lives lost The high risk group

contains the C5, F16, C135, and C130 aircraft, all of which are predicted to lose 10 or more lives

The second key observation is that the low, medium, and high risk rankings are consistent

with respect to the vehicles’ low, mean, and high loss rate estimates With the exception

of the high estimate of the T43, and aircraft with highly similar loss rates, ranking the aircraft by their low risk estimates produces nearly the same results as ranking the aircraft

by their mean risk rankings.'' Ranking by the high risk rankings produces similar results

As a result, the analysis results are robust from the standpoint of relative risk

'' The T43 has a very high o for its loss rate, which results in a large variance in the forecast death rats The

“high” estimate is extreme because of this result When a single crash occurs during a small number of flight hours, it is unclear whether the aircraft if intrinsically unsafe, or whether a safe aircraft “happened” to have a malfunction during the small flight sample More data would help to reduce this uncertainty and provide a more stable estimate.

The third point is that the amount of risk exposure between aircraft can vary by an order

of magnitude At the low end, some aircraft have negligible death rates because either

there are few flight hours flown, or because the risk per flight hour is very low (or both)

These vehicles are expected to have no or few losses—one a year would be a surprise In contrast, there would be no surprise if the three riskiest airplanes resulted in losses of 50

or more persons each As a result, the return to investment on simulation will vary

dramatically based on which airplanes the military chooses to target for simulation

Targeting Simulation Hours

When applying simulation to reduce risk, the economical solution is to replace reality based training hours with simulation based training hours for those aircraft that pose the

greatest risk per hour In order to have the maximum possible effect, the military ought to

pursue the ‘low-hanging fruit’ first—that is, the deaths that are cheapest to prevent The strategy of replacing flight hours with the highest loss rates is the most economical

because simulator hours are approximately fixed in price, irrespective of airplane type rẻ The value of substituting a simulator hour for a reality based training hour is thus

determined by the amount of risk removed We thus prioritize those aircraft with the

highest loss rates The chart illustrates the low, medium, and high estimates together with

the total flight hours for the five year forecast period

24

The loss rates govern the lives saved per training hour substituted, while the total forecast deaths for a given airplane places a ceiling on the maximum possible savings The chart

below illustrates the relationship between total flight hours and deaths accrued Aircraft with similar total deaths appear in the same ‘row.’ Those with similar flight hours

appears in the same column For two aircraft with the same number of total deaths, the aircraft to the left on the chart has a higher loss rate because it has fewer associated flight

Recall that the loss models’ confidence intervals precision is typically one order of

magnitude, hence the logarithmic risk scale is an intuitive way to compare the risk of

aircraft in this context.

The Value of Human Losses

As described, the SVLM describes how human losses accrue with aircraft accidents The cost associated contains two major components: The value associated with the loss of life, and the cost of replacing the skill set of the aviator The first figure is typically found

as the implied expected value of a human life by the manifest risk preferences of decision makers In other words, if a decision maker is willing to spend $20,000 to reduce the risk

of loss of life by 1%, then the associated value of a human life is

100 e $20,000 = $2 million Estimating the intrinsic value of a life at $2.0M, and the value

of skills lost at $5.0M, the cost attributed to loss of a skilled personnel is approximately

$7.0M Totaling the loss of life across all aircraft produces low, mean, and high estimates

of 21, 158, and 360 lives for respective dollar equivalents of $147M, $1.1B, and $2.5B

respectively

The Value of Capital Losses

The second major cost is the loss of the aircraft The value of lost aircraft is taken to be

the dollar cost of the vehicle Note that this definition also has complications Older aircraft, when lost, are not replaced by equivalent aircraft Some analysts would argue that the value of such losses should be lower, because some useful proportion of the vehicle’s life has already been consumed These analysts value aircraft in a way similar

to how insurance companies treat automobiles—they depreciate with time Others point out that older vehicles will be replaced with more expensive, newer vehicles Because the unit cost of vehicles has steadily increased, the dollar cost of replacement is actually higher than the purchase price of the lost airplane A more thorough approach might

consider the value derived from the aircraft through actual use Aircraft central to the

military’s planning might be assessed at a higher value than those considered ancillary The options are endless, for which reason this study begins at the beginning—the dollar cost of the vehicle Methods for more accurately assessing a vehicles ‘worth’ presents one area of future research for subsequent studies

'? Note that these values are for the Air Force only, spanning a five year horizon

The chart below illustrates the loss of aircraft (in units) Low, medium, and high

estimates are provided with the total flight hours for the five year forecast period Losses

are expressed in numbers of aircraft, not the value of the aircraft lost

Mean, Low, and High Aircraft Losses, |

The Value of Combined Losses

Ultimately, the loss of lives and aircraft must be combined into a single measure

The chart below illustrates the combined losses of life, skills, and capital equipment by

aircraft type Three critical features emerge on inspection First, the scope of the problem

can now be deduced Over the five year forecast period, the total of mean losses is $3.3

Billion, while the high and low estimates are respectively $6.0 Billion and $1.1 Billion

Mean, Low, and High Value Losses, Flight Hours,

by Airplane, Projected Five Years

Second, the cost by aircraft type varies Aircraft Value Lost: % of Total

considerably The F16 contributes fully 37.9% of - Other, 2.8%

lost value The top five sources of loss, the F16, | - C141, 1.1%

| | 7 C135, C130, BI, and F15 together contribute approximately 80% of the cost In contrast, the | | |} B52 1.5% || (3521 F117 2.8%

bottom ten contribute less than 3% of the losses | | [ A10, 4.0%

F15, 8.6% simulation needed to replace flight hours—varies

widely by aircraft type Substituting 10% of the

simulator hours, while the B1 yields 1,000

C135, 12.9% C130, 12.4%

28

The Multi-Vehicle Loss Model

The Single Vehicle Loss Model (SVLM) calculates the risk associated with each aircraft

It is a normative model, describing the mean and standard deviation of loss rates—our

principle measure of risk In contrast, the Multi Vehicle Loss Model (MVLM) is a prescriptive model, and describes how to apportion simulator hours across multiple

vehicles based upon the data produced by the SVLM Given a fixed number of simulator

hours to substitute for reality based training, the MVLM specifies which RBT hours to replace, and computes the savings resulting from the removal of the risk associated with the eliminated flight hours The method combines a single criterion for optimality, and

two constraints:

(1) Optimality: The hours with the highest associated risk, measured through dollar

equivalent losses, are eliminated first

(2) Constraint 1: A maximum percentage of training hours may be replaced for each

aircraft type This percentage, a, is held fixed across all aircraft types

(3) Constraint 2: A maximum total number of simulation hours, f, is specified This parameter reflects the maximum capacity to deliver simulation hours

For example, suppose that there are only two types of aircraft, A and B Suppose further that A has 100 flight hours, and B has 200 flight hours The losses associated with A are

$1.0 per hour, and with B are $0.5 per hour Suppose that we wish to allocate B = 10

flight hours We should first substitute flight hours for A, because each hour eliminated results in a cost avoidance of $1.0 If a = 5%, then we can substitute away up to 5 type A flight hours, and 10 type B flight hours In this case, we use up the maximum 5 type A flight hours, and an additional 5 type B flight hours for a total cost avoidance of $7.5 If

a is increased to 10%, then we substitute away up to 10 type A flight hours, and 20 type

B flight hours In this case we will exclusively use type A flight hours for a cost

avoidance of $10 because B = 10 If we increase the maximum simulation hours by

raising B to 20, we will first use 10 type A flight hours, and then 10 type B hours for a

savings of $15

29