Safer Surgery part 6 doc

2008b Debriefing surgical trainees on non-technical skills NOTSS.. Chapter 3 Competence Evaluation in Orthopaedics – A ‘Bottom-up’ Approach David Pitts and David Rowley introduction The

Safer Surgery

24

Flin, R., Yule, S., Paterson-Brown, S., Maran, N., Rowley, D.R and Youngson,

G.G (2007) Teaching surgeons about non-technical skills The Surgeon 5(2),

86–9

Frank, J.R (ed.) (2005) The CanMEDS 2005 Physician Competency Framework Better Standards Better Physicians Better Care Ottawa: The Royal College

of Physicians and Surgeons of Canada

Gawande, A.A., Zinner, M.J., Studdert, D.M and Brennan, T.A (2003) Analysis

of errors reported by surgeons at three teaching hospitals Surgery 133,

614–21

Giddings, A.E.B and Williamson, C (2007) The Leadership and Management of

Surgical Teams London: The Royal College of Surgeons of England.

Glaser, B.G and Strauss, A.L (1967) The Discovery of Grounded Theory Chicago:

Aldine

GMC (2001) Good Medical Practice London: General Medical Council.

Gordon, S.E (1993) Systematic Training Programme Design: Maximizing Effectiveness and Minimizing Liability Englewood Cliffs, NJ: Prentice Hall Hall, J.C., Ellis, C and Hamdorf, J (2002) Surgeons and cognitive processes

British Journal of Surgery 90, 10–16.

Helmreich, R and Davies, J (1996) Human factors in the operating room: Interpersonal determinants of safety, efficiency and morale In A Aikenhead

(ed.) Balliere’s Clinical Anaesthesiology 10(2), 277–95.

Helmreich, R and Schaefer, H (1994) Team performance in the operating room

In M Bogner (ed.) Human Error in Medicine Hillsdale, NJ: LEA, 225–53.

Helmreich, R., Sexton, B and Merritt, A (1997) The Operating Room Management

Attitudes Questionnaire (ORMAQ) University of Texas Aerospace Crew

Research Project Technical Report 97-6 Austin, TX: The University of

Texas

Hoffmann, R., Crandall, B., and Shadbolt, N (1998) A case study in cognitive task analysis methodology: The Critical Decision Method for the elicitation of

expert knowledge Human Factors 40, 254–76.

Jacklin, R., Sevdalis, N., Darzi, A and Vincent, C (2008) Mapping surgical practice decision making: An interview study to evaluate decisions in surgical

care The American Journal of Surgery, 195, 689–96.

Kneebone, R and Darzi, A (2005) New professional roles in surgery British

Medical Journal 330, 803–804.

Klampfer, B., Flin, R., Helmreich, R.L., Hausler, R., Sexton, B., Fletcher, G., et al (2001) Group Interaction in High Risk Environments: Enhancing Performance

in High Risk Environments, Recommendations for the use of Behavioural Markers Berlin: GIHRE Available on <www.abdn.ac.uk/iprc> [last accessed November 2008]

Mishra, K., Catchpole, K., Dale, T and McCulloch, P (2008) The influence of non-technical performance on non-technical outcome in laparoscopic cholecystectomy

Surgical Endoscopy 22, 68–73.

Development and Evaluation of the NOTSS Behaviour Rating System 25

Perrow, C (1999) Normal Accidents: Living with High-risk Technologies

Princeton, NJ: Princeton University Press

Reason, J.T (1997) Managing the Risks of Organizational Accidents Aldershot: Ashgate

SASM (2003) Scottish Audit of Surgical Mortality Annual Report – 2001 data Glasgow: SASM

Seamster, T., Redding, R and Kaempf, G (1997) Applied Cognitive Task Analysis

in Aviation Aldershot: Avebury.

Sevdalis, N., Healey, A.N and Vincent, C.A (2007) Distracting communications

in the operating theatre Journal of Evaluation in Clinical Practice 13, 390–4

Sevdalis, N., Davis, R., Koutantji, M., Undre, S., Darzi, A and Vincent, C.A

(2008) Reliability of a revised NOTECHS scale for use in surgical teams The

American Journal of Surgery 196, 184–90.

Sexton, B., Thomas, E and Helmreich, R (2000) Error, stress, and teamwork in

medicine and aviation: Cross sectional surveys British Medical Journal 320,

745–9

Studdert, D.M., Mello, M.M., Gawande, A.A., et al (2006) Claims, errors, and

compensation payments in medical malpractice litigation The New England

Journal of Medicine 354, 2024–33.

Taylor-Adams, S and Vincent, C (2004) Systems analysis of clinical incidents:

The London protocol Clinical Risk 10, 211–20.

Undre, S., Sevdalis, N., Healey, A.N., Darzi, A and Vincent, C (2007) Observational Teamwork Assessment for Surgery (OTAS): Refinement and

application in urological surgery World Journal of Surgery 31, 1373–81.

Vincent, C.A., Neale, G and Woloshynowych, M (2001) Adverse events in

British hospitals: Preliminary restrospective record review British Journal of

Medicine 322, 517–19.

Youngson, G.G (2000) Surgical Competence: Acquisition, Measurement, and

Retention Edinburgh: The Royal College of Surgeons of Edinburgh.

Yule, S., Flin, R., Paterson-Brown, S and Maran, N (2006a) Non-technical skills

for surgeons: A review of the literature Surgery 139, 140–9.

Yule, S., Flin, R., Paterson-Brown, S., Maran, N and Rowley, D (2006b)

Development of a rating system for surgeons’ non-technical skills Medical

Education 40, 1098–104.

Yule, S., Flin, R., Maran, N., Rowley, D R., Youngson, G.G and Paterson-Brown, S (2008a) Surgeons’ non-technical skills in the operating room: Reliability testing

of the NOTSS behaviour rating system World Journal of Surgery 32, 548–56.

Yule, S., Flin, R., Rowley, D., Mitchell, A., Youngson, G.G., Maran, N and Paterson-Brown, S (2008b) Debriefing surgical trainees on non-technical

skills (NOTSS) Cognition, Technology & Work 10, 265–74.

Yule, S., Rowley, D., Flin, R., Maran, N., Youngson, G.G., Duncan, J and Paterson-Brown, S (2009) Experience matters: Comparing novice and expert ratings of

non-technical skills using the NOTSS system Australian Journal of Surgery

79, 154–160

This page has been left blank intentionally

Chapter 3 Competence Evaluation in Orthopaedics – A

‘Bottom-up’ Approach David Pitts and David Rowley

introduction

The design and implementation of what we now know as Procedure Based Assessments (PBAs) began in the UK in the early 1990s In 2008, PBAs are in use

in all UK surgical specialties, embedded in all surgical curricula as the primary tool for evaluating perioperative competence in the middle and later years of surgical training The motivation driving their development has been practical problem solving In this respect their development has much in common with other ‘need pull’ innovations (Langrish et al 1972) in that their wider foundations can only be seen retrospectively and although they have much in common with other surgical assessments, their early development occurred completely independently

PBAs have been developed and introduced against a backdrop of transition

in surgical training Their development has involved not only the design of an assessment tool but also the battle to gain acceptance of the concept and practice

of overt competence evaluation in the surgical workplace This chapter describes the evolution of PBAs from instigation to practical usage and describes ongoing evaluation of the outcome in terms of the instrument and its use

Surgical Training in Transition

Since the early 1990s UK surgical training has been in a state of constant transition Not only have the regulations governing training changed radically but the political, social and healthcare environments in which training occurs have swung between extremes A review of some of these changes will show why gaining acceptance

by the surgical community for the use of a competence assessment tool such as the PBA has been so vital

Changes in Structure and Regulation

Until the publication of the Calman Report (Department of Health 1993), surgical training in the UK involved a lengthy apprenticeship punctuated by knowledge tests but without any assessment of practical skills and no formal requirement to

Safer Surgery

28

address non-technical areas such as communication or teamwork Although the Calman reforms introduced some degree of structure, it was not until the Richards Report of 1997 (Richards 1997) and the subsequent report of the Competence Working Party of the Joint Committee for Higher Surgical Training (JCHST) in

2001 (Rowley et al 2002) that assessments of practical ability or competence were openly recommended

Royal Colleges should give serious consideration to establishing innovative procedures, other than written exit examinations, to assess clinical competence

of candidates for the award of a certificate of Completion of Specialist Training (Richards 1997)

It is essential that trainers and trainees extend their assessment of operative and clinical performance Speciality Advisory Committees (SACs) in surgery should determine which operations should occur and to what extent, and what level

of operative ability is required for a given stage of training Simply recording

a minimum number of operations is insufficient – the quality of the training experience is more important than the number of experiences (Rowley et al 2002: 21)

Following the publication of Unfinished Business (Donaldson 2002), a report on

the current state of training, further reforms were introduced and the ‘Modernizing Medical Careers’ project coincided with the inception of the Postgraduate Medical Education and Training Board (PMETB) in 2003 which insisted on the introduction

of comprehensive curricula for each specialty and principles established whereby regular assessment of practical skills was encouraged PMETB’s key task has been

to establish standards defining medical education, training and assessment and to assure these standards (including competence based curricula) through external management of quality

The Trauma and Orthopaedics surgical curriculum (the first time such a document has been produced in the specialty in the UK), in which competence-based training and assessment were enshrined, was approved by PMETB in September 2006 (Pitts et al 2007) PBAs have been introduced against this changing structural backdrop

Changes in Public Attitude

There is no doubt that the public attitude towards medicine in general, and to surgery in particular, has changed This change was most notably precipitated by the Bristol (Kennedy 2001) and Shipman (Smith 2005) inquiries into high death rates in paediatric surgery and general practice respectively High mortality rates

in the Bristol Paediatric Cardiac Unit resulted in action from the Department of Health in 1994 and the suspension of operating in that unit in 1995 The subsequent inquiry’s report into that unit (Kennedy 2001) coincided with the very public

Competence Evaluation in Orthopaedics 29

trial and eventual incarceration of Harold Shipman, a general practitioner, for actions resulting in the deaths of a number of his patients The Shipman Inquiry, reporting from 2002–2005 (Smith 2005), revealed serious shortfalls in processes and procedures surrounding the use of controlled drugs, certification of death and the monitoring of clinical performance stretching back, in Shipman’s case,

to his time as a medical trainee The Donaldson White Paper in 2007, for new revalidation processes in the UK for clinicians and other medical professionals (Department of Health 2007), has been one of the longer-term outcomes of the Shipman Inquiry which will undoubtedly culminate in the use of PBAs or similar tools in the revalidation process

Changes in Time Available for Training

The European Working Time Directive introduced in 1998 reduced the number of hours a trainee might stay in the workplace to 58 in 2004 and are likely to reduce those hours further, to 48 in 2009 There have undoubtedly been benefits from this directive but it has significantly reduced the access to surgical experience for trainees, particularly with respect to unusual trauma cases arriving out of normal working hours

Changes in Service Delivery

Recent years have seen the growth of Independent Sector Treatment Centres (ISTC) Such centres, normally operating outwith the control of NHS local management, have been used to reduce waiting lists, particularly for common surgical procedures conducted on anaesthetically less challenging patients This has further reduced the access to routine surgical experience, particularly for more junior trainees

PBAs have been developed against this background of sudden and discontinuous change with reduced access to surgical experience necessitating the introduction

of training tools that help to derive maximum benefit from the time available Facilitating positive change in such circumstances is (and always has been) difficult

The innovator has for enemies those who did well under the old system and only faint friends in those who might do well under the new (Machiavelli 1515, Chapter VI)

What is a PBA?

A PBA is a collection of behavioural markers (elements) for observing activities around a surgical operation set in seven domains covering the whole of a surgical procedure from consent to post operative management

Safer Surgery

30

A PBA is a formal, structured assessment of a trainee’s competence in performing surgery An individual PBA provides a formative assessment to the trainee and evidence for the trainer on which to base their future input and level

of supervision A collection of PBAs (assembled over several years, conducted by

a variety of trainers) provides summative evidence of the trainee’s progress and competence in learning surgical procedures and techniques, performing them to the required protocol and quality

A PBA happens in real time, in a real operating theatre with a live patient

It is normally undertaken, without pressure, between a trainee and their trainer (with whom a relationship is already established) surrounded by an operating team who will not take any unusual measures to support the trainee A PBA will not normally be conducted on the first occasion a trainer and trainee operate together

It is normally conducted on a procedure with which the trainee is already familiar There is no limit to the number of times a trainee may attempt a particular PBA

so there is no pressure to succeed on a particular occasion All of these conditions help the trainee to give a ‘normal’ performance and, more importantly, protect the patient

PBAs in Practice – Applying the Seven Domains

Within each domain there are a number of related yet unique elements which identify activities which must be performed successfully in order to achieve a

‘satisfactory’ score Most elements are identical across all procedures but in some domains there is opportunity for procedure-specific items which identify the trainee’s grasp of the unique aspects of particular surgical procedures Table 3.2 illustrates both generic and specific items

Although superficially the structure of a PBA resembles a two-page checklist (see Figure 3.1) a PBA is not a schedule of how to perform the procedure, rather

it identifies places in the procedure where competence is observable In the same

Table 3.1 PBA domains

Competence Evaluation in Orthopaedics 31

Competencies and definitions Score n/u/S Comments intra-operative technique

IT1 Follows an agreed, logical sequence or protocol for the procedure

IT2 Consistently handles tissue well with minimal damage

IT3 Controls bleeding promptly by an appropriate method

IT4 Demonstrates a sound technique of knots and sutures/staples

IT5 Uses instruments appropriately and safely

IT6 Proceeds at appropriate pace with economy of movement

IT7 Anticipates and responds appropriately to variation e.g anatomy

IT8 Deals calmly and effectively with untoward events/complications

IT9 Uses assistant(s) to the best advantage at all times

IT10 Communicates clearly and consistently with the scrub team

IT11 Communicates clearly and consistently with the anaesthetist

IT12 Dislocates hip safely

IT13 Cuts femoral neck appropriately to match design of implant

IT14 Demonstrates familiarity and understanding of acetabular preparation including osteophyte trimming

medially and at rim

IT15 Broaches the femur properly and prepares the bony surface

IT16 Uses trials and checks component orientation properly

IT17 Fix acetabular component appropriately

IT18 Implants femoral component appropriately

IT19 Performs final reduction and checks for stability

Table 3.2 example elements for total hip replacement PBA, taken from T&O

curriculum (Pitts et al 2007)

Safer Sur

Figure 3.1 Total hip replacement PBA T&O curriculum (Pitts et al 2007)

Competence Evaluation in Orthopaedics 33

way that a driving examiner looks for key behaviours (mirrors, signal, manoeuvre) the assessor is guided by the PBA to key performance points in the procedure Both the trainer and trainee may trigger a PBA It is normally conducted with the trainer scrubbed (able to observe trainee’s actions closely) The trainee conducts the agreed sections of the procedure taking care to verbalize their intentions (in order to not only enable more effective assessment but also to avoid any compromise in the quality of patient care) At any point, the trainer may step

in and perform all or some remaining sections of the procedure, if there is the slightest risk that the trainee will provide less than optimal care

After the surgery is complete the trainee and trainer review the PBA form and complete it Each element of relevant domains assessed is scored as satisfactory or unsatisfactory according to whether there is sufficient evidence from the trainer’s observation that the required standard was met The final domain of the PBA is the global assessment (see Table 3.3)

The global assessment gives the trainer the opportunity to comment on the trainee’s overall performance Even though the individual elements may have been performed to a satisfactory finished quality, the trainer is still able to apply an overall expert judgement For example, the trainee may have been slow or hesitant

or struggled to deal with an unexpected complication

The results of the PBA are transferred to a PBA summary sheet where they are seen alongside results from other PBA assessments This document’s key function

is to demonstrate clearly, to the annual review panel, whether the trainee is making progress, to indicate if certain areas of competence require further attention or highlight whether there are serious causes for concern

Level at which completed elements of the PBA were

performed Tick as appropriate Comments

Level 0 Insufficient evidence observed to support a judgement

Level 1 Unable to perform the procedure under supervision

Level 2 Able to perform the procedure under supervision

Level 3 Able to perform the procedure with minimum supervision (would need

occasional help)

Level 4 Competent to perform the procedure unsupervised (could deal with

complications)

Table 3.3 global assessment taken from T&O curriculum (Pitts et al

2007)