Safer Surgery part 7 pps

Which Procedure?In order to guarantee a sufficiently wide range of assessment, each surgical specialty has selected a number of index procedures.. These procedures are selected on the b

Which Procedure?

In order to guarantee a sufficiently wide range of assessment, each surgical specialty

has selected a number of index procedures These procedures are selected on the

basis of their broad accessibility to trainees, observability and in most cases an aspect of the procedure which contributes something unique to the assessment range In orthopaedics there is presently a collection of 14 index procedures (e.g., carpal tunnel decompression, total knee replacement, compression hip screw for intertrochanteric fracture neck of femur) A trainee may submit PBA assessments

on any number of procedures but a successful example of all 14 must be included before the completion of training By the end of training all the index procedures must be scored at the defined competence level of four Naturally in the early years

an intermediate score is inevitable for all or some domains It is very important for both trainer and trainee to appreciate that it is progression towards competence which is being assessed primarily Less than a score of four is to be expected early

on in training, culminating in ‘straight fours’ towards the completion of training

PBA and the Curriculum

It should be noted that PBAs are one element of a wider specialty curriculum They are linked to the learning agreement and work in synergy with other tools that vary to some degree between specialties

Designing and Developing PBA

Historically the roots of PBA go back in the authors’ experience to the early 1990s1 when a desire to evaluate the change in performance before and after a fracture fixation course lead to the development of a 20-item multisource feedback tool assessing performance in inserting a dynamic hip screw (DHS) into a fractured neck

of femur The potential of this approach went unrecognized until 2002 when the recommendations of the JCHST Competence Working Party (Rowley et al 2002) made it possible to proceed with PBA development in orthopaedics, at which time

it was referred to as a performance-based assessment Parallel developments in other specialties led to the development of the Operative Competence (OPCOMP) tool by Jonathan Beard in vascular surgery (Thornton et al 2003) In 2004, elements of both systems were integrated into what became the Procedure Based Assessment (PBA) In 2005 the tool was introduced to all surgical specialties through workshops conducted for the Specialist Advisory Committee Chairs (Pitts

1 The first PBA was designed in 1994 as a follow up to a project investigating the change in competence following a training course (Oliver et al 1997) but was not published until the report of the JCHST Competence working party (Rowley et al 2002), when it was included as an appendix.

and Rowley 2005) and minor amendments made to the wording of elements and domains to make them accessible to the widest possible user group Since this time they have been embedded in both the trauma and orthopaedic (T&O) curriculum (Pitts et al 2007) and the Intercollegiate Surgical Curriculum Project (ISCP 2008)

In the latter case some minor changes have been made but the instruments remain broadly identical

Design Considerations

Features and characteristics of the surgical workplace, alongside the personality of the surgical team and requirements of assessment have influenced the development

of the PBA

Surgical environment The surgical environment is special and although many

aspects of it may be simulated, there is at present no adequate simulation of the high stakes of a real operative procedure In order to make a valid assessment of operative competence, the real world has to be used This imposes considerable constraints on assessment, not the least of which is the central purpose of providing

an overwhelmingly safe service to the patient Each operation is unique Not only

do the physical circumstances of the operating environment vary but also the composition of the team, type of instruments in use (even for similar procedures) and most fundamentally the patients in whom there is a wide variation of largely similar anatomy and variation in the severity of disease

Nature of the surgical task The basic separation of surgical procedures into

emergency and elective shows that some procedures are conducted on suitable patients who may be screened and selected for surgery by a variety of measures beforehand whereas others will arrive unscheduled with possibly life threatening conditions in a variety of states of ill health The operating room is inevitably a stressful environment in which the formal assessment of trainees’ competence is

of secondary concern

Characteristics of assessors (trainers) and trainees An ‘early years’ specialist

surgical trainee is by no means a novice She/he will have undergone at least five years of medical school education followed by between two and five years

of postgraduate training before she/he enter specialty training A senior surgical trainee, towards the end of training, will be a widely experienced practitioner who regularly operates on her/his own with accessible supervisors who are never the less outside the room in which the surgery is being conducted The trainer (a consultant surgeon) is primarily responsible for the care of the patient and often for the leadership of a large team of professionals during the operative procedure The introduction of a novel activity such as conducting a PBA is (rightly) questioned in

order to ensure it does not compromise the primacy of provision of patient care.

Scale of the community The orthopaedic community is one of the largest in

surgery comprising over 40 per cent of practising surgeons In the UK this involves approximately 3000 surgeons (including trainees) in over 450 hospital locations This imposes considerable demands on the innovation process, not the least of which being to provide effective assessor training for the entire community

Connecting to patients All patients who undergo surgery in a UK teaching hospital

consent to part of their care being undertaken by trainees under supervision PBAs form a part of that patient care process and as such should ideally be understandable

by patients and their representatives

Curriculum requirements The same principles that have guided the development

of the orthopaedic curriculum as a whole also guided the design of PBA These principles were derived from a series of centre case studies The list below is adapted from the Trauma & Orthopaedic curriculum (Pitts et al 2007):

A radical alternative – PBAs have been introduced into an environment

where there were no established assessment tools and no foundations on which to build They have been designed with the intention of gaining

as much support from the orthopaedic community as possible in order to facilitate their implementation

Competence focused – There are debates about the nature or meaning of the

word ‘competence’ One conceptual standpoint states that a competence is simply a demonstrable ability to do something, using directly observable performance as evidence Another understands competence as being a holistic integration of understandings, abilities and professional judgments, where ‘competence’ is not necessarily directly observable, rather it is inferred from performance (Eraut 1994) The integration of these two aspects acknowledges a much greater level of complexity within surgical competencies and avoids the problem that individuals may well be able

to demonstrate that they can ‘do’ something, but that does not necessarily mean that they understand what they are doing or why until they give evidence for it Within our particular competence model we look not only for the three key domains i.e., knowledge, skills and attitudes, but also for the unique combination of those domains in areas such as professional judgement The development of professional judgement is a key outcome

of surgical training

Flexible and easy (intuitive) to use – PBAs have to fit a variety of specialties,

situations and personnel (see above) It is intended that their design will recognize this, whilst providing a consistency of standard and outcome The hospital environment, where many trainers do not have their own office space and distractions abound, is hostile to finding time and space to

•

•

•

meet and talk Most surgeons join the profession to perform surgery They acknowledge the need to train but appreciate the evaluation of training must be part and parcel of service delivery With these factors in mind we have tried to keep PBA s straightforward and sympathetic to the paucity of time in rapidly changing settings to learn complex tools

Able to adapt to new developments (open architecture) – Many innovations,

especially in social technology settings, have a lengthy gestation period From the beginning every effort has been made to try to ensure that the PBA’s architecture is sufficiently open to allow synergy with new developments and requirements

Driven by the trainee – The triggered nature of the PBA puts responsibility

into the hands of those who hold largest stake in seeing training happen – the trainees PBA require and enables the trainee to take the initiative and responsibility for her/his own training

Valid – Questions of validity (truth) may be addressed in several different

ways Does the implementation of the whole system make a valid improvement in the outcomes of training? Are the index procedures selected for assessments a valid choice? Is the internal structure of each assessment valid in terms of the measures of performance it proposes? A major problem in this area is the lack of previous measures of surgical competence It is impossible to make comparison with anything other than examination results, which only measure a limited area of intellectual competence Extensive efforts must be ongoing, within other constraints, to achieve detailed validation of index procedures and PBA

Reliable – PBA should be understood by all in the same way Efforts have

been made to link PBA closely to accepted practice so that a firm foundation

of agreement can be laid for the future

Usable – The circumstances in which PBAs are used dictate that this area

is of primary concern ‘It might be valid and reliable but can you use it

in a practical situation?’ Efforts have been made to ensure that PBAs can

be used in real life contexts within the constraints of time, user skills and attitudes

Holistic in approach – It was clear from early observations that many

problems encountered amongst trainees had their roots in the area of non- technical skills Elements of the PBA address these skills (and highlight them for assessors as well as trainees) It is hoped that more elements

of current non-technical advances will be incorporated into PBA in the future

Formative and summative – The notion of a summative assessment where

a trainer (possibly external) observes a trainee’s performance in a pass/ fail scenario was rejected at an early stage after two pilot studies On one hand there seemed to be insurmountable logistic and resource problems but more importantly, training in the workplace is an ongoing activity and assessment should resonate with its formative nature It was decided that all

•

•

•

•

•

•

•

workplace assessments should be formative, giving feedback to the trainee

to inform and guide her/his future performance It was noted, however, that such assessments would, as a whole, be a useful summary of the trainee’s ability to learn and progress The successful completion of a PBA is not seen as a licence to operate in that procedure but as a single component of a wider assessment of the trainee’s ability to learn operative procedures and perform them on a variety of patients with differing degrees of severity and complexity in their condition

Electronic application – If data are to be gathered from workplace-based

assessments then it must have an electronic application which would facilitate this Sadly the levels of IT ‘literacy’ encountered in pilots trials were highly variable and, more importantly, access to IT resources in NHS Trusts is extremely patchy PBAs have been developed in a paper-based format whilst maintaining the possibility of an easy transfer to a digital system

Selection of a Rating Scale

In the 1994 PBA, it was envisaged that the rater/assessor could be a scrub nurse, senior colleague or peer The rating of any element was made on the basis of how much evidence there was for the judgment For example, one element of the instrument asked about skin preparation, with three options: ‘Was it prepared aseptically/dry prior to draping procedure/ensure no pooling of antiseptic solutions below patient?’ (NB: the early version posed the questions in a very different way.)

The available scores were:

1 = no evidence whatsoever that the stage/task/activity has been completed

2 = some evidence

3 = ample evidence

This approach was taken because we were uncertain at that time whether such observations were possible and in particular, we wanted to compare the scores from professionals with differing interests (e.g., nurses and surgeons) and how much impact a training event had on the trainee’s behaviour in theatre

For the early versions of the later PBAs, we chose a similarly simple scale but from a different assessment viewpoint By this time we were not trying to measure the impact of training, we were attempting to capture a snapshot of the trainee’s behaviour in order to assess competence The rating scale chosen for this was:

0 = not assessed

1 = unsatisfactory

2 = satisfactory

•

Numbers were chosen initially with a view to producing an electronic version later We considered the use of a Likert scale and there was considerable debate

as to whether this would be beneficial in demonstrating degrees of progress that would have a motivational effect We also considered the inclusion of an extra column that could be marked if a trainee showed excellence at particular points (star quality) but eventually concluded that the simplest rating options would be the most effective

A number of factors influenced the choice of the simple scale

The first was that we needed to cater for the possibility that not all items would be assessed There could be no guarantee that the trainee would be able

to complete the whole procedure for a variety of reasons and to complete part of the assessment would be of great benefit to more junior trainees (mirroring actual training practice)

Secondly, it was never considered feasible, given the numbers of assessments involved and the variety of locations, that an independent assessor would be present

in theatre Even if they were, their independence would prevent them entering the sterile area and so limit their observations The consequence of this was that the detail of the observation would only be recorded by the assessor at the end of the procedure The more detailed the rating scale, the more likely the assessor might

be to enter an incorrect score, having remembered the performance inadequately Thirdly, the naturally competitive personality of surgical trainees suggests that there could be lengthy debates about whether their performance should score two

or three on a larger scale and this would introduce an unwanted variable (trainer personality) into the assessment process

The final change to the rating scale came after a meeting in which the PBA was discussed by individuals (surgeons, educators and administrators) who had not been part of the original design group One person in particular found it difficult

to grasp the nominative nature of the scores and insisted on trying to calculate a

minimum average score for the PBA To avoid such problems recurring, the scale was altered to:

N = not assessed

U = unsatisfactory

S = satisfactory

Since the acceptance of PBA by all specialties, some have insisted on changing

the scale from unsatisfactory to requiring further development The authors see no

advantage in this and some potential problems including the danger of increasing uncertainty through lack of definition

The inclusion of the global assessment at the end of the PBA was one of the elements acquired from the merger with the OpComp tool The inclusion of this domain enables a qualitative triangulation of the other domains which has proved extremely beneficial for the reasons of adding an element of overall professional judgement as described above

validity and Reliability of PBAs

The power of the PBA assessment rests in part on the fact that the PBA assesses the same competencies in a variety of procedures with a broad range of suitably qualified assessors An orthopaedic trainee will normally have at least eight trainers, in a series of six-month attachments, during her/his training In addition, she/he will operate in emergency situations, through rostering, with an even wider set of trainers, all of whom may act as assessors for a PBA

Internal Validity of PBA

The initial selection of PBA domains and elements came from two sources One was the original 20-element tool (Pitts and Ross 1994) the other was a series of Delphic groups involving surgeons within the orthopaedic community selected for their expertise as both trainers and surgeons At a later stage the PBA which related

to specific procedures was reviewed by a further series of individuals and groups These revisions were to establish that in a particular procedure all elements were easily observable in a particular procedure and so that examples of positive and negative descriptors, as well as negative-passive indicators (sins of omission) could be identified As a result, all PBAs have been validated against a standard worksheet of these descriptors for every element of every domain, an extract from which is shown as Table 3.4 The worksheet offers the opportunity to articulate specific examples (in italics) of generic competences

Validity of Index Procedures

Whilst the initial selection of index procedures was made by a small group, its work was corroborated using a further set of groups consisting of 50+ surgical trainers in all In this exercise the trainers were required to produce lists of index procedures (to the agreed criteria) on which they had achieved consensus After the outliers were removed from the group lists, a high degree of correlation was seen with the earlier Delphic group selection

A further triangulation of the selection of index procedures was made using the orthopaedic electronic logbook to check that all selected procedures were accessible to trainees in sufficient numbers (Pitts et al 2005) A final review of the procedures’ list was made using a further group of surgeons, during a south east training conference, who reviewed the list from the point of view of procedures that they felt they would, in their practice, be able to use to assess trainees

Reliability

Establishing the inter-rater reliability of the PBA tools proved extremely difficult within the time and budgetary constraints of the PBA Orthopaedic Competence Assessment Project (OCAP) project An early attempt at producing video material

Competences and

definitions

Positive behaviours (doing what should

be done)

negative behaviours (doing what shouldn’t be done)

negative – passive

behaviours (not doing

what should be done) Pre-operative

planning

Demonstrates

recognition of

anatomical and

pathological

abnormalities and

operative strategy

to deal with these

Articulates the realistic clinical findings against any investigative findings and achieves a balance between the two

Describes an operative plan without the full use

of the clinical and investigative material

Fails to take into account specific medical conditions that might limit the technical choices

Ability to make

reasoned choice

of appropriate

equipment,

materials or

devices (if

any) taking

into account

appropriate

investigations e.g.,

x-rays

Is able to draw, write or iterate a preoperative plan

Does not take into account investigative findings when planning or selecting the equipment

Fails to check the notes for relevant or unexpected findings

Takes the x-ray and any templates and plans the operation on paper checking both

AP and lateral

Does not consult the x-ray at all Makes all the decisions on the AP x-ray

Fails to check both AP and lateral x-rays and makes all the decisions

on the AP x-ray

Checks materials,

equipment

and device

requirements with

operating room

staff

Either personally visits or rings up the operating theatre to check on equipment availability

Delegates the task to

a more junior team member with no plans to check the instruction has been carried out

Fails to communicate with the theatre staff

Where applicable

ensures the

operation site is

marked

Personally marks the site

Delegates the task of marking the site to a junior doctor or nurse

Fails to check that the site has been marked

Checks patient

records

Ensures that the relevant information such as investigative findings are present

During the procedure asks theatre staff to look something up in the notes

Fails to check notes to ensure all information

is available that is needed

Table 3.4 validation worksheet example taken from T&O curriculum

(Pitts et al 2007)

for viewing by raters was abandoned due to the difficulty of obtaining sufficiently high quality footage of a lengthy procedure and persuading sufficient numbers

of surgical trainers to spend time scoring it Fortunately this area has now been revisited by a team at Sheffield (Beard, Purdie et al – see Chapter 4 in this volume)

innovation and Acceptability

The positioning of the PBA tool has, from its inception, been as a device ‘designed

by surgeons for surgeons’ The orthopaedic curriculum (OCAP) steering group has had some 22 members in its approximately six-year lifespan with all but one being practising surgeons This has resulted in a high degree of face validity We have further supplemented this with a number of audits in various aspects of the PBA (and curriculum) acceptance and adoption by the orthopaedic community and this is described below It has as yet not been possible to replicate this work

in other specialties

Baseline Survey

Prior to the launch of the curriculum materials into the orthopaedic trainee population in 2005 a small survey was conducted of trainee activity using trainees attending the annual British Orthopaedic Association (BOA) congress Amongst other results, the survey found the following:

10 per cent of respondents had no meeting with their trainer outside the operating theatre in their entire six month attachment;

40 per cent of respondents had no written aims or objectives (learning agreement) for their attachment;

55 per cent had no formal assessment of their operating skills during their attachment;

although the results fitted the expected picture the number of respondents was small (50) but it provided a baseline against which future progress might be measured

Acceptance Survey

In the process of introducing the PBA and other curriculum tools, a number of briefing meetings were held across the UK, with varying numbers attending At each of these meetings a survey was issued with questions relating to different tools, including the PBA Two questions were posed: 1 Is this a good idea? and

2 Will it work?

Whilst some doubts were expressed as to whether trainers would comply with the new system (or have time to do so) respondents clearly expressed the view

•

•

•

•

that it was a good idea and, to a lesser extent, that it would work, although all the outcomes tended to the positive

In addition to gathering broad response, the questionnaires highlighted areas of expected difficulty, many of which have proven to be valid

RITA Questionnaire

The Regional In service Training Assessment (RITA) has been an annual or bi-annual event for UK surgical trainees It is in the process of being replaced by the Annual Review of Competence Progression (ARCP) In October 2005, following the launch of OCAP in August of that year, a questionnaire was issued to all trainees and programme directors to be completed before the RITA The questionnaire asked factual questions about how many PBAs had been conducted, who triggered them and if none or few had been conducted, what the reasons were The primary purpose

of this tool was to find out what was happening in the field The secondary purpose was to send a clear message that the Specialist Advisory Committee (SAC) was taking note of progress and would (and did) investigate instances of non compliance

in a low key way The results have been invaluable in identifying areas where engagement has been weak and further intervention is necessary

Subsequently, an internet survey has been conducted since 2006, annually open to all T&O trainees contacted via their electronic logbook In January 2006 only 50 per cent of trainees had completed one or more PBA assessments but this has risen to 93 per cent by January 2008 (Boardman et al 2008) The work will be submitted for publication shortly as a longitudinal audit study

Latest Developments

PBA assessment tools are now embedded in all surgical curricula Their development continues in a number of areas; particularly in orthopaedics but also

in other specialties

Later Years of Training

Orthopaedic trainees often specialise further in the later years of training preparing for a career in a sub-specialty such as spine, joint replacement, hand surgery etc Debate is continuing as to whether there should be the same PBA assessment conducted on more difficult and specialised procedures or whether an ‘advanced’ PBA should be designed that would assess higher order surgical competencies

OCAP Online

The online version of the orthopaedic curriculum (OCAP Online) was launched

in August 2008 Details can be found on the website: <www.ocap.org.uk> The