Safer Surgery part 13 pot

the work on development of behavioural markers systems for assessing cockpit crews, as well as early work related to healthcare and surgery with a focus on teamwork or communication, thr

Exemplar behaviours and demonstrative scenarios for each sub-team/stage of a procedure are fully described in the OTAS user manual (Undre and Healey 2006, freely available for research use at: <http://www.csru.org.uk>)

Further Empirical Testing: Urological Cases (Undre et al 2007a)

This study aimed to further assess:

feasibility of the revised OTAS© tool;

usefulness of the revisions;

reliability in the behavioural scoring

The study also aimed to compare general surgery with urology elective procedures As in the previous study, care was taken to inform staff about the study and to reassure them that data would be used for research purposes only

Methods

Data were collected in 50 urological surgery operations in two operating theatres, one in our own institution (central London teaching hospital) and the other at a treatment centre Twenty operations were the first operation of the list; the remaining 30 operations were the second or subsequent operation The typical mix of operations contained cystoscopy, ureteroscopy, ureterorenoscopy, transurethral resection of the prostate (TURP) and short procedures such

as orchidectomy, vasectomy and circumcisions Data were collected from procedures that lasted 30–240 minutes Tasks and behaviours were assessed from Pre-op Stage 1 to Post-op Stage 2 The last OTAS© stage was not feasible

to assess

In six additional procedures, behavioural ratings only were collected by two psychologist observers to assess inter-observer reliability

Results and Comments

Task completion Table 6.2 (urology columns) presents the task completion

rates Overall, task completion was higher in urology than in general surgery The pattern of task completion rates between different types of tasks was strikingly similar, with patient tasks showing highest completion rates, followed

by equipment/provisions and communication tasks In addition, some variability was observed in urology theatres too, with significantly lower levels of equipment tasks in the Pre-op Phase than in the other two phases and significantly higher levels of communication tasks in the Pre- and Post-op Phases than in the Intra-operative Phase

•

•

•

Behaviour ratings As in the previous study, these were relatively high (scores

above four on a seven point scale) Of significance:

Anaesthetists’ and nurses’ ratings were highest on cooperation and lowest on communication, with no significant different across operative phases

Surgeons’ ratings exhibited a similar pattern, but, in addition, their scores were significantly lower in the post-operative phase

The Pearson r correlation coefficients between the two psychologists’

ratings were as follows:

– communication: 0.35, p < 0.05;

– coordination: 0.72, p < 0.001;

– cooperation/back up behaviour: 0.64, p < 0.001;

– leadership: 0.62, p <0.001;

– monitoring/awareness: 0.53, p < 0.001

In conclusion, team assessment appeared feasible in urology theatres Importantly, the revised OTAS© application was successful The revised tool replicated some of the findings of the initial version in both task completion and behaviour ratings The acceptable reliability of the behavioural scoring suggests that the addition of exemplar behaviours and demonstrative scenarios did assist the behavioural assessment, as intended

Current Work and Future Directions

Comprehensive and robust assessment of teamwork in the context of surgery is becoming increasingly important It has been shown that poor teamworking in surgical teams is associated with the occurrence of adverse events to patients (e.g., Davenport et al 2007, Gawande et al 2003, Greenberg et al 2007, JCAHO 2000) Recent increases in shift-working in the delivery of surgical services mean that operating theatre staff are now much less likely than in the past to be working in stable teams, in which individuals know each other as well as their strengths and weaknesses (Royal College of Surgeons of England 2007) Such changes have been followed by an increased emphasis on teamwork skills in the modern surgical training curriculum (ISCP 2005) Taken together, such developments are likely to increase the importance of robust teamwork assessment

The aim of this chapter was to present in detail the development and initial application of the Observational Teamwork Assessment for Surgery© (OTAS©) The origins of OTAS© can be traced in the empirical work on teamwork in complex work environments that started more than 40 years ago with a focus on military teams Conceptually, OTAS© is grounded on (Dickinson and McIntyre 1997) a model of teamwork Empirically, it follows attempts to assess teamwork in expert teams that work in complex environments via observation These attempts include

•

•

•

the work on development of behavioural markers systems for assessing cockpit crews, as well as early work related to healthcare and surgery with a focus on teamwork or communication, threats and errors, or error recovery strategies in operating theatres

Building on this work, our research team constructed the initial version of the OTAS©, which consisted of a task checklist (to be completed by a surgeon observer) and five behavioural scales (to be scored by a psychologist observer) Empirical testing of this first version suggested that team observations and assessment is indeed feasible and also led to modifications, aiming to assist the behavioural scoring – thus enhancing reliability and also allowing new observers

to be trained Subsequent testing supported empirically the feasibility, applicability and reliability of OTAS©

At present, we are conducting more developmental as well as validation work using OTAS© Regarding the validation process, three aspects are currently under investigation First of all, we are in the process of assessing effects of observer’s expertise in the behavioural scoring of OTAS© The hypothesis is that ratings obtained from expert OTAS© users should exhibit higher correlations than those obtained by expert and novice users Initial empirical evidence is consistent with this hypothesis Secondly, we are in the process of using OTAS© to analyse teamwork in simulated crises-ridden procedures (Undre et al 2007a) Procedures have been carried out, they have been video/audio-recorded and analysed using OTAS© Initial evidence from the ratings of these procedures suggests that OTAS©

can be used to rate teams retrospectively, in addition to its original real-time usage Thirdly, we are using OTAS© alongside other observational tools that we have developed and that assess aspects of surgical process other than teamwork – including interruptions and disruptions to surgical workflow Meaningful correlations between aspects of teamwork as captured by OTAS© and other surgical processes will contribute to the cross-validation of all tools involved

Regarding the developmental work that is being carried out, we are in the process

of producing a version of OTAS© to be used in simulation-based team training Current work carried out by members of our group suggests that simulation-based training with formative feedback/debriefing has potential application to training surgeons how to cope most effectively with stressors that occur during procedures (Arora et al 2009) Such stressors include technical difficulties (e.g., unexpected bleeding), but also lack/failure of equipment, unnecessary distractions and uncooperative team members This work builds on previous team training modules addressed to the entire operating theatre team that we have piloted successfully (Moorthy et al 2005, 2006, Undre et al 2007a) Successful application of OTAS©

in a training context will expand the domain of application of the tool and will contribute to its further validation

OTAS© is one of the first assessment tools to be designed exclusively for surgical teams In addition to OTAS©, we have also developed and tested a version of the NOTECHS (Avermate van 1998, Flin et al 2003) to be used in surgical teams (Sevdalis et al 2008a) In the past five years, other research groups working in

parallel have developed tools that also aim to capture teamwork skills – including the Anaesthetists’ Non Technical Skill ((ANTS) Fletcher et al 2003), the Non Technical Skills for Surgeons ((NOTSS) Yule et al 2006b, 2008) and the Mayo High Performance Teamwork Scale (MHPTS) (Malec et al 2007 – see Chapters

2, 11 and 12 in this volume) The proliferation of tools with a focus on operating theatre teams signals an increasing recognition of the importance for teamwork in surgery The uniqueness of OTAS© lies in that it captures the entire team, instead

of individual members of it (i.e., the NOTECHS-based approach) Thus OTAS©

offers a holistic, non-threatening assessment that can be used to assess teamwork via observation (as opposed to self-report) in any operating theatre Importantly,

it can also be used to provide formative feedback as part of a crisis management training module; such modules are becoming increasingly embedded in surgical training Taken together with other tools and complementary measures, OTAS©

can contribute to our understanding of surgical teamwork and is a potentially useful tool in attempts to improve teamwork, train surgical teams and, ultimately enhance surgical patient care

Acknowledgements

This chapter is based on a large and long-lasting research programme on teamwork

in surgical teams that is being carried out by our research group Dr Andrew N Healey, former member of the group, played an instrumental role in the shaping and development of the OTAS© since its inception and over a number of years

Dr Mary Koutantji and Mr Peter McCulloch also contributed significantly to this work – especially to the refinement of the tool

The authors would like to thank the Department of Health: Patient Safety Research Programme (CAV), the BUPA Foundation (CAV), the Smith and Nephew Foundation (CAV), the British Academy (NS) and the Economic and Social Research Council (ESRC) Centre for Economic Learning and Social Evolution (NS) for funding for the work reported in this chapter

References

Arora, S., Sevdalis, N., Nestel, D., Tierney, T., Woloshynowych, M and Kneebone,

R (2009) Managing intra-operative stress: What do surgeons want from a crisis

training programme American Journal of Surgery 197, 537–543.

Avermate van, J.A.G (1998) NOTECHS: The Evaluation of Non-technical Skills

of Multi-pilot Air Crew in Relation to the JAR-FCL Requirements Research Report for the European Commission Amsterdam: National Aerospace Laboratory, DGVII NLR-CR-98443

Beard, J.D (2007) Assessment of surgical competence British Journal of Surgery

94, 1315–16

Calland, J.F., Guerlain, S., Adams, R.B., Tribble, C.G., Foley, E and Chekan,

E.G (2002) A systems approach to surgical safety Surgical Endoscopy 16,

1005–14

Carthey, J (2003) The role of structured observational research in health care

Quality of Safety in Health Care 12, Suppl 2, ii13–ii16.

Catchpole, K R., Giddings, A.E.B., de Leval, M.R., Peek, G.J., Godden, P.J., Utley, M., Gallivan, S., Hirst, G and Dale, T (2006) Identification of systems failure

in successful paediatric cardiac surgery Ergonomics 49, 567–88.

Davenport, D.L., Henderson, W.G., Mosca, C.L., Khuri, S.F and Mentzer, R.M (2007) Risk-adjusted morbidity in teaching hospitals correlates with reported levels of communication and collaboration in surgical teams but not with scale

measures of teamwork climate, safety culture or working conditions, Journal

of the American College of Surgeons 205, 778–84.

de Leval, M.R., Carthey, J., Wright, D.J., Farewell, V.T and Reason, J.T (2000)

Human factors and cardiac surgery: A multicenter study, Journal of Thoracic and Cardiovascular Surgery 119, Pt 1, 661–72.

Dickinson, T.L and McIntyre, R.M (1997) A conceptual framework for teamwork

measurement In M.T Brannick, E Salas and C Prince (eds) Team Performance Assessment and Measurement, 1st edition (pp 19–44) New Jersey: Lawrence

Erlbaum Associates, Inc

Dwyer, D.J., Fowlkes, J.E., Oser, R.L., Salas, E and Lane, N.E (1997) Team performance measurement in distributed environments: The TARGETs

methodology In M.T Brannick, E Salas and C Prince (eds) Team Performance Assessment and Measurement, 1st edition (pp 137–54) New Jersey: Lawrence

Erlbaum Associates, Inc

Fletcher, G., Flin, R., McGeorge, P., Glavin, R., Maran, N and Patey, R (2003) Anaesthetists’ Non-Technical Skills (ANTS): Evaluation of a behavioural

marker system British Journal of Anaesthesia 90(5), 580–8.

Flin, R., Martin, L., Goeters, K., Hoermann, J., Amalberti, R., Valot, C and Nijhuis, H (2003) Development of the NOTECHS (non-technical skills)

system for assessing pilots’ CRM skills Human Factors and Aerospace Safety

3, 95–117

Fowlkes, J.E., Lane, N.E., Salas, E., Franz, T and Oser, R.L (1994) Improving

the measurement of team performance: The TARGETs methodology Military Psychology 6, 47–61.

Fried, G.M and Feldman, L.S (2008) Objective assessment of technical

performance World Journal of Surgery 32, 156–60.

Gawande, A.A., Zinner, M.J., Studdert, D.M and Brennan, T.A (2003) Analysis

of errors reported by surgeons at three teaching hospitals Surgery 133(6),

614–21

Greenberg, C.C., Regenbogen, S.E and Studdert, D.M (2007) Patterns of

communication breakdowns resulting in injury to patients Journal of the American College of Surgeons 204, 533–40.

Guerlain, S., Shin, T., Guo, H., Adams, R and Calland, J.F (2002) A Team Performance Data Collection and Analysis System Proceedings of the Human

Factors and Ergonomics Society, 46th Annual Meeting

Guerlain, S., Adams, R.B., Turrentine, F.B., Shin, T., Guo, H., Collins, S.R and Calland, J.F (2005) Assessing team performance in the operating room: Development and use of a ‘black-box’ recorder and other tools for the

intraoperative environment Journal of the American College of Surgeons

200(1), 29–37

Guerlain, S., Shin, T., Guo, H., Adams, R and Calland, J.F (2007) A Team Performance Data Collection and Analysis System Proceedings of the Human

Factors and Ergonomics Society 51st Annual Meeting, Baltimore, USA Healey, A.N., Undre, S and Vincent, C.A (2004) Developing observational

measures of performance in surgical teams Quality of Safety in Health Care

13, Suppl 1, i33–i40

Healey, A.N., Sevdalis, N and Vincent, C (2006a) Measuring intra-operative interference from distraction and interruption observed in the operating theatre

Ergonomics 49, 589–604.

Healey, A.N., Undre, S and Vincent, C.A (2006b) Defining the technical skills of

teamwork in surgery Quality of Safety in Health Care 15(4), 231–4.

Healey, A.N., Undre, S., Sevdalis, N., Koutantji, M and Vincent, C.A (2006c) The complexity of measuring interprofessional teamwork in the operating

theatre Journal of Interprofessional Care 20(5), 485–95.

Healey, A.N and Vincent, C (2007) The systems of surgery, Theoretical Issues in Ergonomic Science 8, 429–43.

Helmreich, R.L and Davies, J (2007) Human factors in the operating: Interpersonal

determinants of safety, efficiency and morale In A.R Aitkenhead (ed.) Clinical Anaesthesiology: Safety and Risk Management in Anaesthesia London:

Balliere Tindall

Helmreich, R.L and Foushee, H.C (1993) Why crew resource management? Empirical and theoretical basis of human factors training in aviation In E.L

Weiner, B.G Kanki and R.L Helmreich (eds) Cockpit Resource Management

(pp 3–45) New York: Academic

Helmreich, R.L., Schaefer, H.G and Sexton, B (1995) The Operating Room Checklist Technical Report 95-10 Austin, TX: NASA, University of Texas,

FAA

Helmreich, R.L, Klinect, J.R., Wilhelm, J.A., Tesmer, B., Gunther, D., Thomas,

R., Romeo, C., Sumwalt, R and Maurino, D (2002) Line Operations Safety Audit (LOSA) DOC 9803-AN/761 Montreal: ICAO.

ISCP (2008) Intercollegiate Surgical Curriculum Programme Available at:

<http://www.iscp.ac.uk> [last accessed November 2008]

JCAHO (2000) The Joint Commission: Operative and post-operative complications:

Lessons for the future Sentinel Events Alert 12 Available at: <http://www.

jointcommission.org/sentinelevents/sentineleventalert/> [last accessed January 2008]

Kaufman, L (2003) The wrong nephrectomy Bulletin of the Royal College of Anaesthetists 7, 848–9.

Klampfer, B., Flin, R Helmreich, R, Häusler, R., Sexton, B., Fletcher, G., Field,

P., Staender, S., Lauche, K., Dieckmann, P and Amacher, A (2001) Enhancing Performance in High Risk Environments: Recommendations for the Use of Behavioural Markers Report from the behavioural markers workshop, Zürich,

June Berlin: Damler Benz Foundation

Klinect, J.R., Murray, P., Merritt, A.C and Helmreich, R.L (2003) Line Operations

Safety Audit (LOSA): Definition and operating characteristics Proceedings

of the 12th International Symposium on Aviation Psychology (pp 663–8)

Dayton, OH: The Ohio State University

Lingard, L., Reznick, R., Espin, S., Regehr, G and DeVito, I (2002) Team communications in the operating room: Talk patterns, sites of tension, and

implications for novices Academic Medicine 77(3), 232–7.

Lingard, L., Espin, S., Whyte, S., Regehr, G., Baker, G.R., Reznick, R., Bohnen, J., Orser, B., Doran, D and Grober, E (2004a) Communication failures in the operating room: An observational classification of recurrent types and effects

Quality of Safety in Health Care 13(5), 330–4.

Lingard, L., Garwood, S and Poenaru, D (2004b) Tensions influencing operating

room team function: Does institutional context make a difference Medical Education 38, 691–9.

Mackenzie, C.F., Jefferies, N.J., Hunter, W.A., Bernhard, W.N and Xiao, Y (1996) Comparison of self-reporting of deficiencies in airway management with video analyses of actual performance LOTAS Group Level One Trauma Anesthesia

Simulation Human Factors 38(4), 623–35.

Mackenzie, C.F., Xiao, Y and The IPO Group (2003) Observational Analysis of Video Records of Team Performance Proceedings of the Human Factors and

Ergonomic Society 47th Annual Meeting, Denver, Colorado, 1493–7

Malec, J.F., Torsher, L.C., Dunn, W.F., Weingmann, D.A., Arnold, J.J., Brown, D.A and Phatak, V (2007) The Mayo high performance teamwork scale: Reliability and validity for evaluating key crew resource management skills

Simulation in Healthcare 2, 4–10.

Moorthy, K., Munz, Y., Adams, S., Pandey, V and Darzi, A (2005) A human factors analysis of technical and team skills among surgical trainees during

procedural simulations in a simulated operating theatre Annals of Surgery

242(5), 631–9

Moorthy, K., Munz, Y., Forrest, D., Pandey, V., Undre, S and Vincent, C (2006) Surgical crisis management skills training and assessment: A simulation-based

approach to enhancing operating room performance Annals of Surgery 242,

631–9

O’Connor, P., Hormann, H.J., Flin, R., Lodge, M and Goeters, K.M (2002) Developing a method for evaluating crew resource management skills: A

European perspective International Journal of Aviation Psychology 12(3),

263–85

Paris, C.R., Salas, E and Cannon-Bowers, J.A (2000) Teamwork in multi-person

systems: A review and analysis Ergonomics 43(8), 1052–75.

Roth, E.M., Christian, C.K., Gustafson, M., Sheridan, T.B., Dwyer, K., Gandhi, T.K., Zinner, M.J and Dierks, M.M (2004) Using field observations as a tool for discovery: Analysing cognitive and collaborative demands in the operating

room Cognition, Technology & Work 6(3), 148–57.

Royal College of Surgeons of England (2007) Safe Shift Working for Surgeons in Training: Revised Policy Statement from the Working Time Directive Working Party London: Royal College of Surgeons of England

Sevdalis, N., Davis, R., Koutantji, M., Undre, S., Darzi, A and Vincent, C (2008a)

Reliability of a revised NOTECHS scale for use in surgical teams American Journal of Surgery 196(2), 184–90.

Sevdalis, N., Forrest, D., Undre, S., Darzi, A and Vincent, C (2008b) Annoyances, disruptions and interruptions in surgery: The Disruption in Surgery Index

(Disi) World Journal of Surgery 32(8), 1643–50.

Undre, S and Healey, A.N (2006) Observational Teamwork Assessment for Surgery© OTAS© User Manual Available at: <http://www.csru.org.uk> [accessed June 2008]

Undre, S., Healey, A.N., Darzi, A and Vincent, C.A (2006a) Observational

assessment of surgical teamwork: A feasibility study World Journal of Surgery

30(10), 1774–83

Undre, S., Sevdalis, N., Healey, A.N., Darzi, S.A and Vincent, C.A (2006b)

Teamwork in the operating theatre: Cohesion or confusion? Journal or Evaluation in Clinical Practice 12(2), 182–9.

Undre, S., Koutantji, M., Sevdalis, N., Selvapatt, N., Williams, S., Gautama, S., McCulloch, P., Darzi, A and Vincent, C (2007a) Multi-disciplinary crisis

simulations: The way forward for training surgical teams World Journal of Surgery 31, 1843–53.

Undre, S., Sevdalis, N., Healey, A.N., Darzi, A and Vincent, C.A (2007b) Observational teamwork assessment for surgery (OTAS): Refinement and

application in urological surgery World Journal of Surgery 31(7), 1373–81.

Vincent, C., Moorthy, K., Sarker, S K., Chang, A and Darzi, A.W (2004) Systems approaches to surgical quality and safety: From concept to measurement

Annals of Surgery 239(4), 475–82.

Yule, S., Flin, R., Paterson-Brown, S and Maran, N (2006a) Non-technical skills

for surgeons in the operating room: A review of the literature Surgery 139(2),

140–9

Yule, S., Flin, R., Paterson-Brown, S., Maran, N and Rowley, D.R (2006b)

Development of a rating system for surgeons’ non-technical skills Medical Education 40, 1098–104.

Yule, S., Flin, R., Maran, N., Rowley, D R., Youngson, G.G and Paterson-Brown,

S (2008) Surgeons’ non-technical skills in the operating room: Reliability

testing of the NOTSS behaviour rating system World Journal of Surgery 32,

548–56

Rating Operating Theatre Teams –

Surgical NOTECHS

Ami Mishra, Ken Catchpole, Guy Hirst, Trevor Dale and

Peter McCulloch

introduction

In this chapter we discuss the development of a tool for rating operating teams, and sub-teams of surgeons, anaesthetists and nurses, on their non-technical performance in the operating theatre Validated in laparoscopic cholecystectomy (LC) and carotid endarterectomy (CEA), this work built upon earlier studies of behaviour and process in orthopaedic and paediatric cardiac surgery, where the importance of non-technical skills was identified through direct observation of behaviour and process in the operating theatre (Catchpole et al 2005, 2006, 2007)

No operation observed was performed perfectly, and, in all, deviations from the optimal course of the operation were found In some, these small deviations escalated into more serious situations that compromised the safety of the patient

or the success of the operation Often these problems derived from threats in the system of surgery that originated from outside the operating theatre – or at least could not be attributed solely to errors by the teams As our understanding of how small events can escalate to more serious problems, and the role that non-technical skills may play in reducing or increasing the chances of harm, our methods of measuring those skills have been refined, and in turn our understanding has become more sophisticated In this chapter, we attempt first to describe the process of intellectual and methodological development, and to provide a substantive analysis

of our current tool for assessing non-technical skills in the operating room

A Model of error Causation in Surgery

Early work by the research team in orthopaedic and paediatric cardiac surgery at Great Ormond Street Hospital, London (Catchpole et al 2005, 2006) set out to examine why errors in operating theatres occurred using a model similar to that proposed by Helmreich and colleagues for aviation (Helmreich and Musson 2000, Helmreich et al 1999) The model adapted for surgery (see Figure 7.1) suggests that system threats can predispose and lead to human errors, revealing further