Safer Surgery part 9 pot

The acceptability of the assessment tools to trainees and assessors is being evaluated during the study’s implementation, as the future direction of competency-based assessment in the UK

Our study views feasibility as a key assessment characteristic and it is formally considered within the study’s design Following each case, the supervising consultant completes a PBA or OSATS before giving feedback to the trainee We observe this process and record the time taken to complete the assessment form and the duration

of feedback The follow-up questionnaires which we distribute to trainees and assessors approximately one month after completing assessments, in the relevant specialty, include questions to address feasibility issues, e.g., time added to operating list, available room for feedback, ease of use of tools We have not directly addressed cost effectiveness in this study, although the data could inform future research

Are the Assessment Tools Acceptable to Stakeholders?

User acceptability is the extent to which an assessment tool or method is accepted

by the stakeholders involved in the assessment It is a crucial factor in the design and successful implementation of assessment programmes

The acceptability of the assessment tools to trainees and assessors is being evaluated during the study’s implementation, as the future direction of competency-based assessment in the UK will be influenced by the opinions of surgical trainees and trainers After each feedback session and in the follow-up questionnaires we address the acceptability of the assessments and the assessment tools

There is some overlap of user acceptability issues with feasibility and validity For example, an assessment tool which is very complex and time-consuming to complete is likely to have low face validity and low feasibility as well as having inferior acceptability However, we consider acceptability from the overall perspective of the people directly involved in the assessments We ask supervising consultants and trainees to rate their overall satisfaction with the assessment tools immediately after feedback The subsequent questionnaires provide an opportunity for trainees and assessors to express their views on the ease or difficulty of using the tools, their perspectives on the value of assessment and feedback and the impact of assessment on training and patient safety

Thematic Analysis of the Problem-solving Approach to the Study

implementation

We have used a number of themes below to illustrate the problem-solving approaches we have adopted during the study implementation We have identified lessons learnt and made some practical suggestions which may be of use to those undertaking similar work

1 Matching the Research Team to the Study Design

We have assembled a research team which has the skills required to evaluate assessment in the operating room The team includes surgeons, a research

coordinator with surgical experience and a psychometrician whose particular area of expertise is workplace-based assessment and generalizability theory All three independent assessors are practising surgeons with expertise in many of the surgical specialities, having received training in assessment and feedback through the ‘Training the Trainers’ course at the Royal College of Surgeons, and NOTSS training facilitated by the NOTSS research team, in Edinburgh

Lessons learnt and suggested learning points Whilst it is impossible to outline

all the skills and attributes required from a research team in this field, we consider the following to be essential:

expertise in surgical knowledge, skills, attributes and competence;

familiarity and confidence with working in the operating room environment;

firm research governance knowledge and ‘good clinical practice’ training; statistical expertise independent from the grass roots researchers;

diplomacy in negotiating the socio-political surgical frameworks;

tenacity towards recruitment of cases and engagement of trainees and trainers;

flexibility towards workload and research schedule;

consistent communication and organization between team members to use the team’s resources to their full potential

Our aim was to bring together a multidisciplinary team of researchers with a mix of skills and attributes to complement the design and implementation of our study

2 Engaging and Informing Clinicians

We are carrying out research in real time in the workplace However, surgical environments are both busy and rapidly changing One of the challenges we faced

in implementing the study was ensuring that we used a timely and appropriate method to inform the surgical teams involved We used the following methods of communication, often in combination, to disseminate the purpose and design of the research study:

email information packs;

written information packs;

presentations at departmental meetings;

face-to-face discussions

Our main aim in the advance communication was to familiarize staff with the study before we moved the research into their specialty This approach recognized that research and assessment in the operating room could be viewed by staff as

•

•

•

•

•

•

•

•

•

•

•

•

threatening or unnecessary, unless we clearly explained the aims of the study within the context of work-place based assessment and the surgical curricula The information packs include an overview of the study and examples of the assessment tools with guidance on their use for assessment and feedback As the study has rolled out, we have recognized the need to support email communication with meetings, written information and face-to-face discussion We have been able

to use formal meetings at times but this has often been constrained by practical considerations, for example the size, organizational structure and availability of the target audience Presentations at specialty meetings for surgeons and surgical trainees have proved useful in some specialties but impractical in others It has not been feasible to organize formal presentations within work time for scrub team and anaesthetic staff, which we have managed by arranging smaller ad hoc meetings Overall we have found that trainee and assessor engagement has been best achieved using face-to-face discussions in the field with supporting written information, having provided a background to the study by email communication and presentations where possible

Lessons Learnt and Suggested Learning Points

Identify the most effective way to communicate and disseminate your research to all interested parties

Consider your resources and the feasibility of your approach to communicate and disseminate the study

Be flexible and revise your approach to overcome time constraints and structural barriers in the workplace

3 Training of Assessors

Further to communicating and disseminating the study methodology, there was the need for us to familiarize and train staff involved in assessing and/or giving feedback We experienced wide variations in staff engagement and attitude towards the research, which challenged our ability to provide consistent training for all staff The breadth of the personnel involved in the study and the feasibility

of providing training within work time were also significant barriers to the study implementation

Staff for whom training was required included:

consultant surgeons: PBA and OSATS assessment tool and feedback

training;

consultant/senior anaesthetists: NOTSS training;

scrub nurses: NOTSS training.

We have made great efforts to train all assessors to ensure confidence in and credibility of the assessor ratings We have provided all assessors with information

•

•

•

•

•

•

packs, including the relevant assessment tools, guidance on their use and access to web-based training which we have usually supported with one to one familiarization before they have undertaken any assessments

We acknowledge that we have been unable to achieve an entirely level playing field for training assessors in the use of workplace assessment tools, which reflects the reality of the surgical workplace Consultant assessors differ in their educational interest and awareness, as well as their uptake of assessment training, including web-based training and the Training the Trainers course However, the study design includes several ‘quality measures’ of training Supervising consultants are asked at the time of giving feedback to trainees and in a subsequent questionnaire what type

of training they have received in assessment This will be incorporated in the data analysis Furthermore, our data analysis using generalizability theory will enable an examination of the variance in ratings for different assessor designations, for example

a comparison of NOTSS ratings between independent assessors, anaesthetists and scrub nurses Despite training all assessors in the appropriate use of the tools, we have observed some inconsistencies in the way the tools are used:

prompting trainees too readily;

being unable to allow trainees to lead the case within their level of

competence, taking over decision-making, or the surgical instruments; directing trainees to operate using their preferred surgical sequence and/or technique;

being reluctant to score competencies negatively (and/or give difficult feedback), particularly for senior trainees

We recognize that these training styles may influence ratings of trainee skills and behaviours For example, if a trainee is directed to operate using a different technique, they are unlikely to be as smooth in its delivery, affecting PBA/OSATS scores We found it very difficult to rate skills and behaviours where a trainer repeatedly intervened and in some cases we used ‘not applicable’ to rate these skills and behaviours

The most successful assessments and training opportunities are during cases where the consultant trainer permits the trainee to operate within their limits

of competence, and grants them the leadership to carry this out, prompting or intervening only when required or requested Supervisor training to this level is beyond the scope of our study, but is what will be required of trainers in the future

if training is to become more effective

Lessons Learnt and Suggested Learning Points

Providing training in the workplace requires flexibility and tenacity to ensure full coverage

Identify the most effective and suitable method for training different assessors

•

•

•

•

•

•

Consider consistency in the training approach you adopt, taking into account differences in staff engagement and attitude

Be prepared to abandon assessments or preclude assessors if an authentic assessment is compromised

4 Consenting and Recruitment of Patient Participants

To successfully recruit a surgical case requires the favourable alignment and accurate timing of many factors which are beyond our control These factors are often not confirmed until the day of the operation and the case cannot go ahead as part of the study if any one of them is missing:

an appropriate case;

a consenting patient;

a surgical/HDU bed;

a suitable and consenting surgical trainee;

an available consultant for providing supervision, assessment and feedback;

an independent assessor;

theatre staff for the list;

sufficient operating time for the case;

a suitable training list (some lists are for service provision only)

Furthermore, the ethical requirement of the study is that patients receive

a Patient Information Sheet 24 hours before they are approached for consent

to give them time to consider This requires us to provide patients with an Information Sheet before their admission, with the corollary that we need to identify suitable cases for recruitment at least three working days ahead to allow for postage time Our selection of cases in advance is affected by a number

of other last-minute changes such as lack of beds, insufficient operative time

or there being no available trainee for assessment This results in a number of cases which cannot be recruited as planned, with patient consent rarely being the determining factor

For the few cases in which patients declined to participate, we found that their decision was often surrounded by misconceptions regarding trainee involvement

in performing supervised elective surgery For some patients this was a reflection

of heightened concern for their surgery and a wish for a consultant to perform the operation For some others, there was simply an expectation that a consultant would be performing the operation We provided an open discussion of the training system, acknowledging the role of supervised operating in training

In some cases, this open discussion resulted in patients deciding to consent to participate

•

•

•

•

•

•

•

•

•

•

•

Lessons Learnt and Suggested Learning Points

Consider the complexity of consenting and recruiting patient participants for observational studies in the surgical environment and the resources required

Appreciate the patient’s perspective towards surgical systems and training, which will inform your consent process

Communicate at all levels to make full potential of the team’s resources and to maximize the recruitment of suitable cases

Consider contingency lists, so that if a surgical case ‘goes down’, there may

be other suitable cases for assessments

5 Surgical Trainees as Study ‘Participants’

Ethical approval for the study dictates written consent from patients as participants but not from trainees However, from a training perspective and during the implementation of the study, we have also considered surgical trainees to be study participants It could be argued that competency-based assessment of surgical skills is now a requirement of surgical training However, this does not extend

to the additional research conditions, including independent assessors, video and NOTSS assessments Our approach has been to seek verbal consent from surgical trainees before their involvement in the study We provide all trainees with an invitation letter before introducing the study into a new surgical specialty and give them an opportunity to discuss the research with the study coordinator, thereby ensuring participation without duress There have been a handful of trainees across the specialties, usually senior trainees approaching their CCT (certificate of completed training), who have declined to take part

We have experienced initial hesitancy from some trainees regarding their involvement in the study, often centred on misunderstanding the purpose of the study and concerns that research data could affect their training, for example in the event of a critical incident occurring One of the clear messages we have conveyed

is that the study is designed to assess the assessment tools themselves, in particular their validity and reliability across different surgeons, cases and specialties, and it is not assessing an individual’s level of surgical skill or competence The majority of trainees, who decided to participate, have increasingly engaged with assessments (See Theme 6, ‘Research versus Training Agenda’, for a fuller discussion.)

We have continued in our attempts to collect data to suit the statistical model employed for the study, the optimal data being different combination of trainees and assessors across the cases Some trainees operate more frequently, some senior surgical trainees perform complex procedures which are not covered by the study and some consultant lists have more index procedures These differences generate unbalanced data However, moving trainees across lists for assessments in the operating room proved unworkable Some trainees were resistant to movement, preferring training and assessment by their regular consultant We also realized

•

•

•

•

that assigning a particular trainee to perform a surgical case assessed their behaviour as ‘technical operators’, rather than reflecting the complete role of a holistic surgical practitioner For the same reasons, if there were late changes to the trainee covering a particular list, we decided to exclude the case to focus the study on authentic workplace assessments

Lessons Learnt and Suggested Learning Points

Consider the ethics of trainees as participants in observational surgical education studies

Engage trainees in the process of assessment by communicating effectively the research purpose and the role of their involvement

Respect the working surgical system in place: research needs to work with the surgical system rather than adapt the system to suit the research.

6 Research versus Training Agenda: A Dichotomy or Collaboration?

The study protocol includes the validation of assessment tools which are in current use in the workplace for surgical training (See the earlier background of this chapter for an overview of PBA and OSATS tools within surgical training programmes.) There are opportunities for educational research to form collaborations or conflicts with the training agenda, and this is illustrated by discussing our role as independent assessors in the study

Examples of collaboration Providing opportunities for training in

workplace-based assessment has moved beyond the research agenda to provide trainees and trainers with valuable, timely training on the tools which are integral to the new surgical curricula

The study has provided ring-fenced opportunities for training and assessment, which has resulted in the increasing engagement of trainees in the study Trainees have made comments such as, ‘I’m happy to take part; it means I get to operate’ and ‘It’ll guarantee some assessments for my portfolio’ Participation in the study has also encouraged a number of trainers and trainees to use the PBA tool for the first time We have been able to show trainers and trainees that suitable cases for workplace training can be identified opportunistically, and that the process

of assessment and feedback is feasible, adding little time to an operating list Highlighting the role of formative assessment in driving learning has encouraged appropriate use of the PBA within the curriculum For example, use of parts of the PBA for trainees not ready to complete the whole operation under supervision Our ‘field testing’ of the assessment tools has generated suggestions for tool modification which have been forwarded to the relevant bodies for consideration For example, carotid endarterectomy and caesarean section are performed under local or regional anaesthetic The trainee’s communication with the patient

•

•

•

therefore becomes an important assessment item which was not part of the original template

Examples of dichotomy All assessors score independently without conferring

or discussion until the assessment tools for the case are completed There is no

discussion of scoring between cases, either between independent assessors or

with trainers, which could have a convergent effect on assessor ratings However,

it is inevitable in the clinical setting for some discussion to take place with the presence of trained independent assessors who are facilitating cases for assessment purposes Assessment in the operating theatre is a relatively new training method, and the research team are seen to represent a body of expertise in assessment and feedback

We have found that our role as a complete observer in the independent

assessment of skills and behaviours in the operating room is both unrealistic and unworkable We suggest that our role within this study is more aligned to

observer-as-participant, part way along the continuum from complete observer, to

observer-as-participant, onto participant-as-observer, then to complete participant (Gold 1958) The process of assessment itself prompts discussion; for example, what constitutes a good surgical technique and why? Judgements on skills and behaviours are rated independently between assessors, although discussion surrounding the subjectivity of skills and behaviours is seen as a necessary outcome

of the study implementation

Another area of conflict we have recognized relates to the quantity and quality

of the research data The supervising consultant is often not present in the operating room until the case commences, which omits the pre-operative preparation section

of the PBA and equivalent sections on the OSATS Do we prompt consultants to

be present at the whole case to maximize data quantity or is the research question most accurately answered by the researchers assuming the observer role? Do we encourage consultants to allow trainees to lead the case, which provides the most authentic assessment of surgical performance or observe the real-life training situation?

We have found it challenging, particularly when new trainees and/or trainers become involved in the study, to provide sufficiently general information to support the assessments and the use of the assessment tools, without introducing specific prompts which would affect the assessment ratings and overall data quality

We have made compromises to uphold the research agenda, as from a training standpoint, independent assessor(s) would prompt trainers on the appropriate use of the assessment tools, such as to address the inconsistencies in tool use highlighted above in Theme 3

Lessons Learnt and Suggested Learning Points

Educational research can engender collaboration and conflicts between the research and training agendas

•

Consider your role as researchers in upholding the research agenda whilst drawing upon collaborations with the training agenda

Some discussions surrounding the assessment of skills and behaviours are

a necessary condition for implementing workplace assessments

7 Developments in Study Design and Methodology during Implementation

Developments to the study design and methodology have arisen to meet the requirements of the study aim and to promote future research directions It is only when the study design and methodology are subjected to field testing during their implementation that the full requirements of the study protocol can be realized These developments have been driven by the foresight of the research team and with guidance from the Studys Steering Committee External review of the original study protocol advised us to consider more than two assessment tools (originally PBA and OSATS) The development of rating non-technical skills in the operating room, supported by the literature evidence for the relationship of these skills to surgical skills and safety, stimulated the addition of NOTSS to the study protocol

We originally recruited patients from one specialty at a time because of the logistical difficulties in working in multiple specialties simultaneously However,

in order to improve recruitment and maximize the reliability data it has been essential for us to recruit from two, sometimes three specialties at any one time This has been achieved by securing funding for a further independent assessor who has also taken responsibility for coordination of cases within her own specialty of obstetrics and gynaecology An early ethics amendment approved the addition of obstetrics and gynaecology as a surgical specialty with the inclusion of four extra index procedures This has provided the opportunity to compare PBA with OSATS

in a specialty that uses OSATS as the current workplace assessment tool We needed to collect a larger dataset of assessments in this specialty compared to the others to allow a comparison of the tools However, recruiting cases in obstetrics and gynaecology has been very successful because it lends itself to providing large numbers of suitable cases (see Theme 8, ‘The Significance of Context’, for

a full discussion) The addition of this specialty has included a non-elective index

procedure, urgent caesarean, which provides an opportunity for video assessment and use of the NOTSS tool in urgent surgical cases

The completion of more than one assessment tool has a potential to confound each assessment The best case scenario is for each independent assessor to complete one tool per case However, the number of cases required

to obtain sufficient data to test each of the assessment tools would be very large, certainly unachievable within this study The final study protocol accepts that there will be completion of more than one tool by independent assessors (PBA or OSATS and NOTSS) We recognize that differences exist between the assessment tools which make their simultaneous use in assessment problematic For the PBA tool, there is an expectation that trainees should verbalize their

•

•

intentions throughout, also advised in the PBA validation document for training assessors The NOTSS tool has been designed for use in as naturalistic a setting

as possible, without prompting trainee communication or decision-making Our concern surrounding the completion of two tools was raised with the Steering Committee and the decision has been to minimize the completion of two tools For example, if there are two independent assessors observing a case, both complete the PBA or OSATS but only one completes a NOTSS assessment Where two tools have been completed, the impact of this can be considered using a post hoc analysis

Recruitment has been better in some specialties than in others In specialties where we realized that recruitment would not improve, we decided to move on rather than risk a training effect by remaining in the speciality Our intention is to return to specialties to capture another cohort of trainees next year, prioritizing the specialties in which further recruitment is needed

Lessons Learnt and Suggested Learning Points

Developments to study design may be required to meet the research aim or

to take advantage of new research directions

Preliminary fieldwork can require changes to be made to the study design and methodology, which may involve ethics amendments

A flexible team approach with good foresight encourages the negotiation

of study developments

8 The Significance of Context: Inter-specialty Differences

The individual surgical specialties have offered different advantages and disadvantages to the study implementation The obstetrics and gynaecology specialty has leant itself well to the study methodology Index cases occur frequently on operating lists, for example between three to four elective caesareans every weekday All the index cases are relatively short so it is feasible to obtain several trainee assessments per list The specialty uses a team consultant structure, with each trainee assigned to a team of three to four consultants, which enables each trainee to operate with several consultants each week Other specialties presented significant problems for study implementation Within orthopaedics, operating lists were often amended at short notice, giving insufficient time to inform patients about the study Staff structuring, with each consultant allocated a single trainee limited the combinations of trainees to trainers for assessment

We also found the culture of surgical assessment and training within each surgical specialty significantly different In obstetrics and gynaecology there is

an established culture for objective assessments of surgical competence and a requirement of consultants to provide assessment and feedback to trainees, which facilitated our introduction of the study The RCOG has phased in the use of

•

•

•