Ebook Observer performance methods for diagnostic imaging: Part 1

Part 1 book “Observer performance methods for diagnostic imaging” has contents: Preliminaries, the binary paradigm, modeling the binary task, the ratings paradigm, empirical AUC, binormal model, hypothesis testing, sample size estimation,… and other contents.

Observer Performance Methods

for Diagnostic Imaging

IMAGING IN MEDICAL DIAGNOSIS AND THERAPY Series Editors: Andrew Karellas and Bruce R Thomadsen

Quality and Safety in Radiotherapy

Todd Pawlicki, Peter B Dunscombe,

Arno J Mundt, and Pierre Scalliet, Editors

ISBN: 978-1-4398-0436-0

Adaptive Radiation Therapy

X Allen Li, Editor

ISBN: 978-1-4398-1634-9

Quantitative MRI in Cancer

Thomas E Yankeelov, David R Pickens,

and Ronald R Price, Editors

ISBN: 978-1-4398-2057-5

Informatics in Medical Imaging

George C Kagadis and Steve G Langer, Editors

Image-Guided Radiation Therapy

Daniel J Bourland, Editor

ISBN: 978-1-4398-0273-1

Targeted Molecular Imaging

Michael J Welch and William C Eckelman,

Editors

ISBN: 978-1-4398-4195-0

Proton and Carbon Ion Therapy

C.-M Charlie Ma and Tony Lomax, Editors

ISBN: 978-1-4398-1607-3

Physics of Mammographic Imaging

Mia K Markey, Editor

ISBN: 978-1-4398-7544-5

Physics of Thermal Therapy:

Fundamentals and Clinical Applications

Eduardo Moros, Editor

Image Processing in Radiation Therapy

Kristy Kay Brock, Editor ISBN: 978-1-4398-3017-8

Informatics in Radiation Oncology

George Starkschall and R Alfredo C Siochi, Editors

ISBN: 978-1-4398-2582-2

Cone Beam Computed Tomography

Chris C Shaw, Editor ISBN: 978-1-4398-4626-1

Computer-Aided Detection and Diagnosis in Medical Imaging

Qiang Li and Robert M Nishikawa, Editors ISBN: 978-1-4398-7176-8

Cardiovascular and Neurovascular Imaging: Physics and Technology

Carlo Cavedon and Stephen Rudin, Editors ISBN: 978-1-4398-9056-1

George Kagadis, Nancy L Ford, Dimitrios N Karnabatidis, and George K Loudos Editors ISBN: 978-1-4665-5568-6

Published titles

IMAGING IN MEDICAL DIAGNOSIS AND THERAPY Series Editors: Andrew Karellas and Bruce R Thomadsen

Published titles

Comprehensive Brachytherapy:

Physical and Clinical Aspects

Jack Venselaar, Dimos Baltas, Peter J Hoskin,

and Ali Soleimani-Meigooni, Editors

ISBN: 978-1-4398-4498-4

Handbook of Radioembolization:

Physics, Biology, Nuclear Medicine,

and Imaging

Alexander S Pasciak, PhD., Yong Bradley, MD.,

and J Mark McKinney, MD., Editors

Stereotactic Radiosurgery and

Stereotactic Body Radiation Therapy

Stanley H Benedict, David J Schlesinger,

Steven J Goetsch, and Brian D Kavanagh,

Editors

ISBN: 978-1-4398-4197-6

Physics of PET and SPECT Imaging

Magnus Dahlbom, Editor

ISBN: 978-1-4665-6013-0

Tomosynthesis Imaging

Ingrid Reiser and Stephen Glick, Editors

ISBN: 978-1-138-19965-1

Ultrasound Imaging and Therapy

Aaron Fenster and James C Lacefield, Editors ISBN: 978-1-4398-6628-3

Beam’s Eye View Imaging in Radiation Oncology

Ross I Berbeco, Ph.D., Editor ISBN: 978-1-4987-3634-3

Principles and Practice of Image-Guided Radiation Therapy

Dev P Chakraborty, Editor ISBN: 978-1-4822-1484-0

Observer Performance Methods

for Diagnostic Imaging

Foundations, Modeling, and Applications with

R-Based Examples

Dev P Chakraborty

CRC Press

Taylor & Francis Group

6000 Broken Sound Parkway NW, Suite 300

Boca Raton, FL 33487-2742

© 2018 by Taylor & Francis Group, LLC

CRC Press is an imprint of Taylor & Francis Group, an Informa business

No claim to original U.S Government works

Printed on acid-free paper

International Standard Book Number-13: 978-1-4822-1484-0 (Hardback)

This book contains information obtained from authentic and highly regarded sources Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials

or the consequences of their use The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint Except as permitted under U.S Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers.

For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://www copyright.com/) or contact the Copyright Clearance Center, Inc (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-

8400 CCC is a not-for-profit organization that provides licenses and registration for a variety of users For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for

identification and explanation without intent to infringe.

Library of Congress Cataloging-in-Publication Data

Names: Chakraborty, Dev P., author.

Title: Observer performance methods for diagnostic imaging : foundations,

modeling, and applications with R-based examples / Dev P Chakraborty.

Other titles: Imaging in medical diagnosis and therapy ; 29.

Description: Boca Raton, FL : CRC Press, Taylor & Francis Group, [2017] |

Series: Imaging in medical diagnosis and therapy ; 29

Identifiers: LCCN 2017031569| ISBN 9781482214840 (hardback ; alk paper) |

ISBN 1482214849 (hardback ; alk paper)

Subjects: LCSH: Diagnostic imaging–Data processing | R (Computer program

language) | Imaging systems in medicine | Receiver operating

characteristic curves.

Classification: LCC RC78.7.D53 C46 2017 | DDC 616.07/543–dc23

LC record available at https://lccn.loc.gov/2017031569

Visit the Taylor & Francis Web site at

http://www.taylorandfrancis.com

and the CRC Press Web site at

http://www.crcpress.com

Dedication

Dedicated to my paternal grandparents:

Dharani Nath (my “Dadu”) and Hiran Bala Devi (my “Thamma”)

1.3 Imaging device development and its clinical deployment 3

1.3.2 Quality control and image quality optimization 5

1.5 Why physical measures of image quality are not enough 8

1.9.1.1 Part A: The receiver operating characteristic (ROC) paradigm 131.9.1.2 Part B: The statistics of ROC analysis 13

References 15

PART A The receiver operating characteristic (ROC) paradigm

2.2 Decision versus truth: The fundamental 2 × 2 table of ROC analysis 21

x Contents

3.8 Inverse variation of sensitivity and specificity and the need for a single FOM 46

3.9.4 Properties of the equal variance binormal model ROC curve 50

Contents xi

4.4 Relation between ratings paradigm and the binary paradigm 67

4.6 A single “clinical” operating point from ratings data 68

4.8 Observer performance studies as laboratory simulations of clinical tasks 704.9 Discrete versus continuous ratings: The Miller study 71

6.2.2 Invariance of the binormal model to arbitrary monotone transformations 94

xii Contents

7.4 Estimating case sampling variability using the DeLong method 124

7.5 Estimating case sampling variability of AUC using the bootstrap 128

7.6 Estimating case sampling variability of AUC using the jackknife 131

Contents xiii

9.1.3 The dearth of numbers to analyze and a pivotal breakthrough 164

9.6.2 Meanings of variance components in the DBM model 171

9.7.3 Decision rules, p-value, and confidence intervals 177

9.10.2.3 Which analysis should one conduct: RRRC or FRRC? 189

10.2.4 Meaning of the covariance matrix in Equation 10.5 210

10.3.4 Decision rule, p-value, and confidence interval 220

11.3 Sample size estimation for random-reader random-cases 233

11.4 Dependence of statistical power on estimates of model parameters 23711.5 Formula for random-reader random-case (RRRC) sample size estimation 23811.6 Formula for fixed-reader random-case (FRRC) sample size estimation 23911.7 Formula for random-reader fixed-case (RRFC) sample size estimation 239

11.11 Prediction accuracy of sample size estimation method 248

References 254

PART C The free-response ROC (FROC) paradigm

12.4.4 The free-response receiver operating characteristic (FROC) plot 267

13.3.1 The semi-constrained property of the observed end-point of

13.4.2.1 Constrained property of the observed end-point of the

AFROC 286

14 Computation and meanings of empirical FROC FOM-statistics

14.6.1 Physical interpretation of area under AFROC 32714.6.2 Physical interpretation of area under wAFROC 327

References 329

Contents xvii

15.6.3 Resemblance of Kundel–Nodine model to CAD algorithms 33815.7 Analyzing simultaneously acquired eye-tracking and FROC data 340

15.7.3 Generalized ratings, figures of merit, agreement, and

17.3 Constrained end-point property of the RSM-predicted ROC curve 354

xviii Contents

17.4.4 Proper property of the RSM-predicted ROC curve 359

17.9.1 Lesion-classification performance and the 2AFC LKE task 38117.9.2 Significance of measuring search and lesion-classification

17.10 The FROC curve is a poor descriptor of search performance 38217.10.1 What is the “clinically relevant” portion of an operating

characteristic? 385

17.11.2 The binormal model is a special case of the RSM 387

17.11.3 Explanations of empirical observations regarding binormal parameters 391

17.11.3.1 Explanation for empirical observation b < 1 39117.11.3.2 Explanation of Swets et al observations 393

Contents xix

19.5 RSM versus PROPROC and CBM, and a serendipitous finding 425

19.5.1.2 Application of RSM/PROPROC/CBM to three datasets 433

19.5.2 Inter-correlations between different methods of estimating AUCs and

19.5.2.1 A digression on regression through the origin 435

19.5.3 Inter-correlations between RSM and CBM parameters 43819.5.4 Intra-correlations between RSM derived quantities 44019.5.4.1 Lesion classification performance versus AUC 44419.5.4.2 Summary of search and lesion classification performances

versus AUC and intra-correlations of RSM parameters and a

19.7.1 Relating an ROC effect size to a wAFROC effect size 446

References 452

PART D Selected advanced topics

20.7.2 Example: Check of Equation 36 in the Metz–Pan paper 471

xx Contents

20.7.4 The role of simulation testing in validating curve fitting software 473

21.5.1 Examples of bivariate normal probability integrals 49221.5.1.1 Examples of bivariate normal probability integrals 492

21.6.3.1 Exercise: Convert the above parameter values to (μ, σ) notation 496

References 516

23.4 Calibration, validation of simulator, and testing its NH behavior 531

23.4.2 Validation of simulator and testing its NH behavior 532

References 533

Index 535

Series Preface

Since their inception over a century ago, advances in the science and technology of medical ing and radiation therapy are more profound and rapid than ever before Further, the disciplines are increasingly cross-linked as imaging methods become more widely used to plan, guide, moni-tor, and assess treatments in radiation therapy Today, the technologies of medical imaging and radiation therapy are so complex and computer-driven that it is difficult for the people (physi-cians and technologists) responsible for their clinical use to know exactly what is happening at the point of care when a patient is being examined or treated The people best equipped to understand the technologies and their applications are medical physicists, and these individuals are assuming greater responsibilities in the clinical arena to ensure that what is intended for the patient is actu-ally delivered in a safe and effective manner

imag-The growing responsibilities of medical physicists in the clinical arenas of medical imaging and radiation therapy are not without their challenges, however Most medical physicists are knowl-edgeable in either radiation therapy or medical imaging and expert in one or a small number of areas within their disciplines They sustain their expertise in these areas by reading scientific arti-cles and attending scientific talks at meetings In contrast, their responsibilities increasingly extend beyond their specific areas of expertise To meet these responsibilities, medical physicists must periodically refresh their knowledge of advances in medical imaging or radiation therapy and they must be prepared to function at the intersection of these two fields To accomplish these objectives

is a challenge

At the 2007 annual meeting of the American Association of Physicists in Medicine in Minneapolis, this challenge was the topic of conversation during a lunch hosted by Taylor & Francis Publishers and involving a group of senior medical physicists (Arthur L Boyer, Joseph O Deasy, C.-M Charlie Ma, Todd A Pawlicki, Ervin B Podgorsak, Elke Reitzel, Anthony B Wolbarst, and Ellen D Yorke) The conclusion of this discussion was that a book series should be launched under the Taylor & Francis banner, with each volume in the series addressing a rapidly advancing area of medical imaging or radiation therapy of importance to medical physicists The aim would be for each volume to provide medical physicists with the information needed to understand technologies driving a rapid advance and their applications to safe and effective delivery of patient care

Each volume in the series is edited by one or more individuals with recognized expertise in the technological area encompassed by the book The editors are responsible for selecting the authors

of individual chapters and ensuring that the chapters are comprehensive and intelligible to one without such expertise The enthusiasm of volume editors and chapter authors has been grati-fying and reinforces the conclusion of the Minneapolis luncheon that this series of books addresses

some-a msome-ajor need of medicsome-al physicists

The series Imaging in Medical Diagnosis and Therapy would not have been possible without the encouragement and support of the series manager, Lu Han, of Taylor & Francis Publishers The edi-tors and authors, and most of all I, are indebted to his steady guidance of the entire project

William R Hendee

Founding Series Editor Rochester, MN

Foreword (Barnes)

Gary T Barnes, PhD, Professor Emeritus, Department

of Radiology, University of Alabama Birmingham

The stated objective of this book is to educate individuals who are interested in evaluating observer performance in diagnostic imaging It provides insight into the practical methodology of such studies The author assumes the reader is not a statistician or otherwise an expert in such evalua-tions, but is persistent and willing to work through the examples given The author also provides background and references on the evolution of observer performance evaluations from their intro-duction in the 1940s to the present, as well as insight on common pitfalls and misconceptions

I have benefited from knowing Dr Chakraborty for the past thirty-five years I’ve been aware of his contributions and also worked with him from 1981 to 1989 His accomplishments during this period were numerous and impressive—the first paper1 on digital tomosynthesis (1984); development of a digital road mapping system2 employing an early personal computer, digital frame grabber, and video tape recorder that was used clinically to assess the blood vessel patency in vascular surgery (1985); correction3 of x-ray image intensifier pin cushion distortion (1985); the first paper4 to apply FROC analysis to compare two clinical imaging modalities (1986) The breadth of these contributions is impressive and he was the first author on all the resultant papers There were many other areas where

he contributed and was a co-author Noteworthy is that commercial digital tomosynthesis units are now in clinical use and road mapping is a standard offering on digital fluoroscopy vascular units

As noted above, Dr Chakraborty was the first author on the first paper to employ FROC analysis

to a clinical comparison of two modalities—conventional screen-film and digital chest units The digital unit was an early prototype FROC methodology was chosen because nodule detection was the clinical problem being studied and conventional ROC analysis, or for that matter LROC, new at that time, did not appropriately address the problem of localization and did not (and does not) han-dle multiple lesions and responses in an image As a result of this effort, he became aware of the limi-tations of observer performance studies and subsequently published a number of seminal papers on the subject resulting in a valid figure of merit definition when two or more modalities are compared.When Dr Chakraborty has made suggestions to me in the past, I have listened and benefited from listening His performance in solving a wide breadth of technical and mathematical problems

is impressive This book will benefit individuals interested in observer performance evaluations in diagnostic medical imaging and provide additional insights to those that have worked in the field for many years

3 Chakraborty DP Image intensifier distortion correction Med Phys 1987;14(2):249–252.

4 Chakraborty DP, Breatnach ES, Yester MV, Soto B, Barnes GT, Fraser RG Digital and tional chest imaging: A modified ROC study of observer performance using simulated nod-

conven-ules Radiology 1986;158:35–39.

Foreword (Kundel)

Harold L Kundel, Emeritus Professor, Department of Radiology,

Perelman School of Medicine, University of Pennsylvania

“Medicine is, at its core, an uncertain science Every doctor makes mistakes in

diagnosis and treatment.”

—J Groopman 1

Diagnostic imaging is an important component of medical care It has been estimated that about 10% of physician encounters with patients over age 65 involve imaging.2 Human observers interpret all of the images and when the diagnostic information in the image is either obscure or ambiguous there is a chance for decision error Imaging technology also changes, sometimes rapidly; consider the introduction of computed tomography (CT) and magnetic resonance imaging (MRI) Each change in technology requires evaluation to decide whether it improves diagnostic performance and ultimately improves patient care Radiologists and radiological physicists are well aware of the need for the continuous evaluation of technological advances and long before Evidence Based Medicine became popular, they were measuring both the physical properties of images and the effect of those properties on observer performance

In fact, the first systematic study of diagnostic performance in radiology was done in the late 1940s, as part of a field study to determine which of four possible chest x-ray techniques was best for tuberculosis screening Unfortunately, variation in the detection of tuberculosis between observers was so large that it precluded any valid statistical comparison of the relative accuracy of the four x-ray techniques.3 The large inter-individual variability in performance implies diagnostic error

If two observers disagree, only one can be right Maybe both are wrong! The biostatistician for the tuberculosis screening study, J Yerushalmy, identified two major problems in the statistical analysis of the data First, the true diagnosis of each case was not known, and second, the large vari-ability between and within observers could not be accounted for by the statistical tools available

at that time for dealing with sensitivity and specificity, which were considered to be independent components of accuracy.4

The statistical toolbox was augmented in the late 1960s when L Lusted introduced the receiver operating characteristic (ROC) into diagnostic medicine.5 The ROC curve, which is a plot of true positive responses (sensitivity) against false positive responses (1—specificity), is based on the assumption that sensitivity and specificity are covariates The detection theory model also implies that sensory variables (ability to detect) and decision variables (bias toward negative or positive decisions) can be separated The methodology still requires that the investigator (not the observer) knows the true diagnosis, but it produces a single figure of merit for performance, the area under the ROC curve, that is not affected by the observer’s decision-making bias It is the decision bias, the tendency to use either strict or lenient criteria when making a decision about the presence or absence of disease, that is a major source of individual variability

The ideal ROC study consists of having several observers report on a carefully selected set of cases, normal and abnormal, in which there is a single abnormality, and then report on the cases again after a sufficient interval of time has elapsed to forget previous responses The ROC area and

xxviii Foreword (Kundel)

the measured variability within and between observers can be used in an appropriate statistical model to characterize or compare imaging modalities.6 An experiment that fits all of the model requirements can be done in the laboratory but in the real world, decision tasks and images are more complicated Observers may be asked to specify disease location or may be required to clas-sify as well as detect disease, for example, classify a tumor as benign or malignant Furthermore, some diseases, metastatic lung tumors, for example, may occupy multiple sites

The ROC methodology has been gradually refined to encompass some of the more realistic ations that are encountered in medical imaging In particular, software that accounts for case vari-ability and observer variability has been developed and is available on the Internet for general use.7 The author of this book, Dev Chakraborty, has been particularly instrumental in advancing the analytical capabilities of the ROC methodology for the task of locating abnormalities and for images with multiple abnormalities by developing the free-response ROC (FROC) methodology

situ-As opposed to most of the books with a primary statistical orientation, this book presents the technology evaluation methodology from the point of view of radiological physics and contrasts the purely physical evaluation of image quality with the determination of diagnostic outcome through the study of observer performance The reader is taken through the arguments with con-

crete examples illustrated by code in R, an open source statistical language There is a potential here

to make the powerful ROC method of technology evaluation available to a wide community in a format that is readily understandable

References

1 Groopman J How Doctors Think New York, NY: Houghton Mifflin Company; 2007.

2 Dodoo MS, Duszak R, Jr., Hughes DR Trends in the utilization of medical imaging from 2003

to 2011: Clinical encounters offer a complementary patient-centered focus J Am Coll Radiol

2013;10:507–512

3 Birkelo CC, Chamberlain WE, Phelps PS, Schools PE, Zacks D, Yerushalmy J Tuberculosis case

finding JAMA 1947;133:359–365.

4 Yerushalmy J Statistical problems in assessing methods of medical diagnosis with special

reference to x-ray techniques Public Health Rep 1947;62(40):1432–1449.

5 Lusted LB Introduction to Medical Decision Making Springfield, IL: Charles C Thomas; 1968.

6 Swets JA, Pickett RM Evaluation of Diagnostic Systems: Methods from Signal Detection Theory

New York, NY: Academic Press; 1982

7 Hillis SL, Berbaum KS, Metz CE Recent developments in the Dorfman-Berbaum-Metz

pro-cedure for multireader ROC study analysis Acad Radiol 2008;15:647–661.

People sometimes ask me what exactly it is I do for a living Here is my answer: radiologists make decisions based on their interpretations of medical images Contrary to folklore, they do make mis-

takes—missed lesions and/or false positive findings, some of which have serious consequences My

work involves modeling their interpretations and, based on these models, objectively estimating their performance This sentence is usually met with incredulity among my “hard-core” physicist friends

How can one model and objectively measure something as subjective as human interpretations? Physicists are used to analyzing “hard data” with instruments whose behaviors are well under-stood Every physicist knows that the angle a galvanometer moves is proportional to the product of the current and the magnetic field To put it mildly, the human brain is not as well understood The story of how I was plunged (unprepared) into this field is described in Chapter 9

My background is varied, more so than most scientists working in this field that is variously termed observer performance, receiver operating characteristics (ROC) analysis, model observers, psycho-physics, and statistical analysis of ROC data So here is an account of my evolution This should help

the reader appreciate where I am coming from and what to expect in my approach to the book.

I did my PhD under the late Prof Ronald Parks, of superconductivity fame I worked in an tal physics laboratory I got my hands dirty by learning how to use a machine shop to construct measure-ment apparatus My PhD thesis was forgettable.1 I obtained a post-doctoral fellowship at the University

experimen-of Pennsylvania, under Prexperimen-of Alan J Heeger, a future Nobel laureate, and even had a shared tion.2 I performed magneto-resistance measurements in the Francis Bitter National Magnet Laboratory

publica-at MIT In spite of my pedigreed background I realized thpublica-at I was really not good publica-at physics, or, stpublica-ated alternatively, the competition was too strong (the field was clogged with brilliant folks, and physicists were coming in by the hordes from the Soviet Union) In addition, I had the handicap of a student visa

So, I needed to change fields and go someplace that would help me obtain the proverbial green-card That

turned out to be the University of Alabama at Birmingham (UAB), where I moved in 1979

I worked on hyperthermia—basically heating tumors to about 45° Celsius, to effect preferential cancer-cell kill.3,4 My boss there was Prof Ivan Brezovich and we had a close relationship I still recall our daily noon trips to the UAB Faculty Club, which had the best pecan pie anywhere and which I have since searched for in vain I had to get my hands really dirty this time: besides machine shop work building various hyperthermia applicators, I helped use them on patients Since hyperthermia was then considered experimental therapy, we only got patients who had failed all conventional thera-pies; in short, they were very sick I recall one who went into shock during the therapy (i.e., heating) session, and Dr John Durante, then director of the Comprehensive Cancer Center at UAB, had to step in I also recall an African American patient with melanoma, rare among African Americans Another patient had cancer of the esophagus and Dr Brezovich designed a sausage-like tube that

xxx Preface

contained a concentric electrode and was otherwise filled with saline The patient endured the ible task of swallowing this tube so that we could pass current through the electrode and radially out-ward to the patient’s body, thereby preferentially heating the lining of the esophagus The experiment succeeded but the patient eventually died These experiences, however painful to watch, were what got me my next job When Prof Gary Barnes was looking for a junior faculty member, Ivan told him,

incred-“Dev is good with patients.” Apparently, that is all that it took to get me my first full-time job!

My work with Prof Gary Barnes was my entry into diagnostic radiological physics I was tunate to have Gary as a boss/mentor, for he was ideal in many respects—his patience and sense

for-of humor is legendary He still recalls with pride my early tomosynthesis work,5 which received a Certificate of Merit at RSNA The images were turned 90° compared to their usual presentation—I did not know the difference! So, the story is each radiologist would step up to see the poster, and turn his or her head 90° to see the images of the inner ear in the orientation they were used to seeing!

A particularly productive research venture, which yielded a technical note6 in radiology, was the Quantex project My readers are probably unfamiliar with this device; it was an early fluoroscopic image-processing machine With its myriad buttons, blinking lights, and IEEE-GPIB interface, allow-ing every function to be controlled by a computer, it was a dream machine as far as I was concerned I became aware of work by Gould et al.7 that used two Quantex video processors, one to perform low-pass filtering and the other to perform high-pass filtering, and subtracting the two image sequences to can-cel stationary anatomy, revealing only the iodinated blood vessels Unfortunately, each Quantex DS-30 box cost about $35,000 and our administrator would not approve buying another one So, I designed a system that used the only box we had, but added two high-quality VCRs, and built some electronics, all controlled by a Commodore PET computer I wish I had held on to that computer, as it belongs in a museum It had no hard drive—the BASIC and machine language code was saved to a cassette drive

A period of intense collaboration with Dr Jiri Vitek (an interventional neuroradiologist) resulted During the evenings, I would wheel my equipment into the angiography suite, place a phantom with which I could mimic iodine being injected into a blood vessel (basically a rubber tube concentric with a large diameter tube filled with water) To prevent damaging the image intensifier I occasionally blocked the x-rays at the tube with a lead plate Once, after finishing, I forgot to remove the lead plate To my horror, on the following day, when I was demonstrating my apparatus to Jiri, the technologist looked bewildered when, with a patient on the table, there was no image—I immediately rose to the occasion and nonchalantly removed the lead plate

This project was a personal success The first time I showed subtracted images, Jiri said, “This is DSA.” DSA is an acronym for digital subtraction angiography It was also my introduction to the

general principle that to do clinically meaningful research one has to collaborate with clinicians

Each time I thought the project was ready to be handed over to the technologists, Jiri would come

up with one more request A notable one was road-mapping When iodine is injected the vessels

“light up” (actually, they turn dark as iodine contrast material absorbs more x-rays than blood) Jiri asked me if I could capture a static image of the “lit up” vessels and superpose it on the live x-ray image The static vessel image would then serve as a road map, allowing the radiologist to negotiate the tip of the catheter into the correct blood vessel To do this I had to learn machine language on the Motorola 6502 processor, the heart of the Commodore PET computer This sped up a procedure (loading a look-up-table into a part of computer memory) by a factor of at least 100, making the whole idea feasible At that time, commercial machines did not have the road mapping capability When a GE digital system was purchased for the angiography suite, I was tasked to roll my cart into the suite to show the GE scientists how it was done This project started my love affair with com-puters, both at the software and the hardware levels To really understand image processing, I had

to implement it in software/hardware One learns by doing and this philosophy pervades this book

The next project was evaluation of Picker International’s prototype digital chest machine versus

a conventional machine by the same manufacturer Gary wanted me to do an “ROC” study I put ROC in quotes, as Gary was deeply suspicious of ROC methodology that simply requires a rat-

ing that there are one or more lesions somewhere in the image As Gary would say, “What do you mean by somewhere? If you see a lesion, why not point to it?” In fact, radiologists do, but location

information cannot be accounted for by the ROC paradigm Gary was also upset that his requests

of perceived lesions on an acrylic overlay The locations were scored as true positives and false tives according to how close they were to true lesions, and I could construct free-response ROC (FROC) curves and it appeared that in the lung field the two curves (digital and conventional) were similar, while in the mediastinum the digital modality was visibly better We submitted to radiology and fortunately Prof Charles E Metz reviewed the paper Dr Metz noted that it was an interesting study, but lacked any statistical analysis So, I delved into statistical analysis The only statistics I knew prior to this were limited to the Maxwell-Boltzmann, Fermi-Dirac, and Bose-Einstein distributions and the only statistics book I had in my possession was one by Bevington.8

posi-I had no idea what a p-value meant posi-In fact, posi-I was in a similar situation as most expected readers

of this book My education began with a book by Swets and Pickett,9 and I eventually managed to squeeze out confidence intervals and satisfy Charlie, and get the paper published.10

A turning point was a letter from Prof James Sorenson, author of a famous book on Nuclear Medicine,11 who was also interested in the FROC problem His letter struck an intellectual nerve and overnight I wrote the FROCFIT program (in FORTRAN) that produced excellent visual fits to our Picker study FROC data This resulted in a paper in Medical Physics.12 This was followed soon after by

my alternative FROC (AFROC) paper.13 In those days, getting my papers published was easy!Due to difficulty getting funding, my FROC work languished for a decade until I obtained fund-ing for an idea (differential ROC or DROC) that eventually turned out to have a fatal flaw The awful realization signaled a low point in my career, but I did eventually document, for posterity, the rea-sons why DROC was a bad idea.14 I used the grant funds to revisit the FROC problem A key inter-action with Prof Berbaum led to jackknife AFROC (JAFROC) analysis.15 Another key interaction, with Dr Claudia Mello-Thoms, resulted in a fuller understanding of the Kundel–Nodine model of diagnostic search yielding the radiological search model,16,17 a major component of this book

On a different note, apart from the interactions noted above, doing research in this field has been a solitary experience I do have a lot of collaborators at the “end-user” level, but not one at the methodol-ogy developer level Instead my FROC work has sparked US-based opposition, which has increased

in proportion to its growing acceptance outside the United States.18,19 With the passing of Swensson, Wagner, Metz, and Dorfman, I have been deprived of the four most qualified scientists who can appreci-ate my work Writing this book has also been a solitary experience It started as an edited book, but I rap-idly realized that few of the ROC experts would collaborate with me, which led me to convert it to a sole author book, a decision I have not regretted In all, I have spent about three years working on this book

I have had the privilege of learning from some of the best in this field: Prof Brezovich, who helped me transition from basic physics to medical physics; Prof Barnes, who helped me transi-tion to imaging physics; Prof Kundel, who was instrumental in hiring me at the University of Pennsylvania and taught me everything I know about image perception; and Prof Berbaum, a

friend and supporter, who once paid me the compliment that “you take on difficult projects.” Prof

Metz paid a similar public compliment at the 2001 Airlie Conference Center in Warrenton, VA at the Medical Image Perception Society meeting

Mr Xuetong Zhai, currently a graduate student in Bioengineering at the University of Pittsburgh, vided extensive programming and analytical help, especially in connection with the chapters on signifi-cance testing and designing a calibrated simulator On these chapters (9, 10, and 23) he is a co-author On his own initiative, he developed the RJafroc package, detailed in online chapter 25 This R package forms the software backbone of this book An essential component of the book is the Online Supplementary material As with any software, bug fixes and updates will be posted to the BitBucket website listed below Instructions on how to download material from BitBucket are on www.devchakraborty.com.Finally, I wish to acknowledge my wife Beatrice for believing in me and for putting up with me for over 26 years and my daughter Rimi, co-founder of minuvida (http://www.minuvida.com), a friend and confidant

pro-xxxii Preface

References

1 Chakraborty DP, Parks RD Resistance anomaly of the order-disorder system Fe3A Phys

Rev B 1978;18:6195–6198.

2 Chakraborty DP, Spal R, Denenstein AM, Lee K-B, Heeger AJ, Azbel MY Anomalous

magne-toresistance of Quasi one-dimensional Hg3-dAsF6 Phys Rev Lett 1979;43:1832–1835.

3 Chakraborty DP, Brezovich IA Error sources affecting thermocouple thermometry in RF

electromagnetic fields J Microw Power 1982;17(1):17–28.

4 Atkinson WJ, Brezovich IA, Chakraborty DP Usable frequencies in hyperthermia with

ther-mal seeds IEEE Trans Biomed Imaging 1984;BME-31(1):70–75.

5 Chakraborty DP, Yester MV, Barnes GT, Lakshminarayanan AV Self-masking subtraction

tomosynthesis Radiology 1984;150:225–229.

6 Chakraborty DP, Gupta KL, Barnes GT, Vitek JJ Digital subtraction angiography apparatus

Radiology 1985;157:547.

7 Gould RG, Lipton MJ, Mengers P, Dahlberg R Digital subtraction fluoroscopic

sys-tem with tandem video processing units Proceedings Volume 0273, Application of

Optical Instrumentation in Medicine IX Application of Optical Instrumentation in Medicine,

San Francisco, 1981 doi:10.1117/12.931794

8 Bevington PR, Robinson DK Data Reduction and Error Analysis Boston, MA: McGraw-Hill; 2003.

9 Swets JA, Pickett RM Evaluation of Diagnostic Systems: Methods from Signal Detection Theory

1st ed New York, NY: Academic Press; 1982, p 253

10 Chakraborty DP, Breatnach ES, Yester MV, Soto B, Barnes GT, Fraser RG Digital and tional chest imaging: A modified ROC study of observer performance using simulated nod-

conven-ules Radiology 1986;158:35–39.

11 Cherry SR, Sorenson JA, Phelps ME Physics in Nuclear Medicine 4th ed Orlando, FL: Elsevier,

Saunders; 2012, p 523

12 Chakraborty DP Maximum Likelihood analysis of free-response receiver operating

charac-teristic (FROC) data Med Phys 1989;16(4):561–568.

13 Chakraborty DP, Winter LHL Free-response methodology: Alternate analysis and a new

observer-performance experiment Radiology 1990;174:873–881.

14 Chakraborty DP Problems with the differential receiver operating characteristic (DROC) method Proceedings of the SPIE Medical Imaging 2004: Image Perception, Observer Performance, and Technology Assessment; San Diego, CA; 2004;5372:138–43

15 Chakraborty DP, Berbaum KS Observer studies involving detection and localization:

Modeling, analysis and validation Med Phys 2004;31(8):2313–2330.

16 Chakraborty DP ROC curves predicted by a model of visual search Phys Med Biol

2006;51:3463–3482

17 Chakraborty DP An alternate method for using a visual discrimination model (VDM) to

opti-mize softcopy display image quality J Soc Inf Disp 2006;14(10):921–926.

18 Hillis SL, Chakraborty DP, Orton CG ROC or FROC? It depends on the research question Med

Phys 2017;44:1603–1606.

19 Chakraborty DP, Nishikawa RM, Orton CG Due to potential concerns of bias and conflicts of interest, regulatory bodies should not do evaluation methodology research related to their

regulatory missions Med Phys 2017;44:4403–4406.

Note about Online Supplementary Resources

The following supplementary tools and materials are available online:

■ www.devchakraborty.com

■ www.expertcadanalytics.com

Preface xxxiii

The online supplementary material, outlined below, is available at

■ https://bitbucket.org/dpc10ster/onlinebookk21778, access to which will be granted upon e-mail request to the author at dpc10ster@gmail.com

■ https://cran.r-project.org/web/packages/RJafroc/

Chapter 1: Preliminaries

Online Appendix: Introduction to R/RStudio: Part I

Chapter 3: Modeling the binary task

Online Appendix 3.A: R code demonstration of sensitivity/specificity concepts

Online Appendix 3.B: Calculating a confidence interval

Online Appendix 3.C: Introduction to R/RStudio: Part II

Online Appendix 3.D: Plotting in R

Online Appendix 3.E: Getting help in R: Part I

Online Appendix 3.F: Getting help in R: Part II

Online Appendix 3.G: What if a package is missing

Online Appendix 3.H: Shaded distributions in R

Online Appendix 3.I: Numerical integration in R

Chapter 4: The ratings paradigm

Online Appendix 4.A: operating points from counts table

Chapter 5: Empirical AUC

Online Appendix 5.A: Calculating the Wilcoxon statistic

Chapter 6: Binormal model

Online Appendix 6.A: Equivalence of two and four parameter models

Online Appendix 6.B: Output of Eng website software

Online Appendix 6.C: Maximizing log likelihood

Online Appendix 6.D: Validating the fitting model

Online Appendix 6.E: Variance of AUC

Online Appendix 6.F: Transformations

Chapter 7: Sources of variability in AUC

Online Appendix 7.A: The bootstrap method in R

Online Appendix 7.B: The jackknife method in R

Online Appendix 7.C: A calibrated simulator for a single dataset

Online Appendix 7.D: Comparison of different methods of estimating variability

xxxiv Preface

Chapter 8: Hypothesis testing

Six sigma.pdf

Chapter 9: DBMH analysis

Online Appendix 9.A: The Satterthwaite degree of freedom

Online Appendix 9.B: Demonstration of significance testing formulae

Online Appendix 9.C: Text output file listing

Online Appendix 9.D: Excel ANOVA tables

Online Appendix 9.E: Code for validating DBMH analysis

Online Appendix 9.F: Simulators for validating fixed-reader and fixed-case analyses

Online Appendix 9.G: Code illustrating the meaning of pseudovalues

Online Appendix 9.H: Testing for interactions

Chapter 10: Obuchowski–Rockette–Hillis (ORH) analysis

Online Appendix 10.A: The DeLong method for estimating the covariance matrix

Online Appendix 10.B: Estimation of covariance matrix: single-reader multiple-treatmentOnline Appendix 10.C: Comparing DBMH and ORH methods for single-reader multiple-treatmentOnline Appendix 10.D: Minimal implementation of ORH method

Online Appendix 10.E: Proof of Eqn (10.64)

Online Appendix 10.F: Single-treatment multiple-reader analysis

Chapter 11: Sample size estimation

Online Appendix 11.A: Sample size formula using ORH approach

Online Appendix 11.B: ORH fixed-readers random-case (FRRC)

Online Appendix 11.C: ORH random-reader fixed-cases (RRFC)

Online Appendix 11.D: Details on effect-size specification

Chapter 12: The FROC paradigm

Online Appendix 12.A: Code used to generate the FROC plots

Online Appendix 12.B: CAMPI and cross correlation

Online Appendix 12.C: The Bunch transforms

Chapter 13: Empirical operating characteristics possible with FROC dataOnline Appendix 13.A: FROC versus AFROC

Chapter 14: Computation and meanings of empirical

FROC FOM-statistics and AUC measures

Online Appendix 14.A: Proof of Equivalence theorem for wAFROC

Online Appendix 14.B: Understanding the AFROC and wAFROC

Online Appendix 14.C: Summary of FROC FOMs

Online Appendix 14.D: Numerical demonstrations of FOM-statistic versus AUC equivalences

Preface xxxv

Chapter 15: Visual search paradigms

Appendix 15A: Eye-position data clustering

Appendix 15B: Figures-of-merit derived from FROC and eye-tracking data

Appendix 15C: Measuring agreement between pairs of figures-of-merit

Appendix 15D: Confidence intervals for FOMs and agreement

Chapter 16: The radiological search model (RSM)

Online Appendix 16.A.1: Demonstration of Poisson sampling

Online Appendix 16.B.1: Demonstration of binomial sampling #1

Online Appendix 16.C.1: Demonstration of binomial sampling #2

Chapter 17: Predictions of the RSM

Online Appendix 17.A: The error function

Online Appendix 17.B: Derivation of expression for TPF

Online Appendix 17.C: Expression for pdf of diseased cases

Online Appendix 17.D: RSM-predicted ROC & pdf curves

Online Appendix 17.E: Is FROC good?

Online Appendix 17.F: Binormal parameters for RSM-generated ROC datasets

Online Appendix 17.G: Explanations of Swets et al observations

Chapter 18: Analyzing FROC data

GEHealthcare-Education-TiP-App-Library_XR-Volume-Rad-Quicksteps.pdf

PowerComparison.xlsx

Chapter 19: Fitting RSM to FROC/ROC data and key findings

Online Appendix 19.A: Organization ofAllResults

Online Appendix 19.B: Inter-correlations

Online Appendix 19.C: Intra-correlations

Online Appendix 19.D: Sample size estimation for FROC studies

Chapter 20: Proper ROC models

Online Appendix 20.A: Viewing slopes and ROC curves

Online Appendix 20.B: Plotting PROPROC ROC curves

Online Appendix 20.C: Plotting CBM ROC curves

Online Appendix 20.D: Plotting bigamma model curves

Chapter 21: The bivariate binormal model

Online Appendix 21.A: Sampling the bivariate normal distribution

Online Appendix 21.B: Density of multivariate normal distribution

xxxvi Preface

Online Appendix 21.C: Multivariate integrals

Online Appendix 21.D: Running CORROC2 in a Mac environment

Online Appendix 21.E: Application to a real dataset: details

Chapter 22: Evaluating standalone CAD versus radiologists

Online Appendix 22.A: Random-reader fixed-case analysis

Online Appendix 22.B: Random-reader random-case analysis

Chapter 23: Validating CAD analysis

Online Appendix 23.A: Roe-Metz (RM) simulator for single modality CAD versus radiologistsOnline Appendix 23.B: Meanings of terms in RM simulator model

Online Appendix 23.C: Calibrating the RM simulator to a specific dataset

Online Appendix 23.D: Validation of the RM simulation and NH behavior

Online Appendix 23.E: Simulator calibration helper functions

Online Appendix 23.F: Code to perform NH testing

About the Author

Dev P Chakraborty received his PhD in physics in 1977 from the

University of Rochester, NY Following postdoctoral fellowships at the University of Pennsylvania (UPENN) and the University of Alabama

at Birmingham (UAB), since 1982 he has worked as a clinical tic imaging physicist He is American Board of Radiology certified in Diagnostic Radiological Physics and Medical Nuclear Physics (1987) He has held faculty positions at UAB (1982–1988), UPENN (1988–2002), and the University of Pittsburgh (2002–2016) At UPENN he supervised hos-pital imaging equipment quality control, resident physics instruction and conducted independent research He is an author on 78 peer-reviewed publications, the majority of which are first-authored He has received research funding from the Whittaker Foundation, the Office of Women’s Health, the FDA, the DOD, and has served as principal investigator on several NIH RO1 grants.His work has covered varied fields: hyperthermia, physical measures of image quality, feasibility

diagnos-of digital tomosynthesis for inner-ear imaging (his RSNA poster received a Certificate diagnos-of Merit), building a digital subtraction angiography apparatus for interventional neuro-angiography, and computerized analysis of mammography phantom images (CAMPI) He conducted (1986) the first free-response ROC study comparing Picker International’s prototype digital chest imaging device

to a conventional device In 1989, he coined the term AFROC to describe the currently widely used operating characteristic for analyzing free-response studies Since 2004 he has distributed JAFROC software for the analysis of free-response and ROC studies Over 107 publications have used this software and RJafroc, an enhanced R-version, which are being used in courses and PhD projects worldwide He is internationally recognized as an expert in observer performance methodology

He recently served as statistical consultant to General Electric on the evaluation of the VolumeRad chest tomosynthesis device

Dr Chakraborty’s overarching research interest has been measuring image quality, both at physical and at perceptual levels He showed, via CAMPI, that a widely used mammography QC phantom could be analyzed via an algorithm, achieving far greater precision than radiologic tech-nologists With the realization that wide variability (about 40%, 1996 study by Beam et al.) affects expert radiologist interpretations, over the past two decades Dr Chakraborty’s research has focused

on observer performance measurements, specifically the free-response paradigm, modeling and quantifying visual search performance, and developing associated statistical analysis He has pro-posed the radiological search model (RSM) that resolves several basic questions, dating to the 1960s about ROC curves Recently he has developed an RSM-based ROC curve-fitting method that yields important insights into what is limiting performance In 2016 Dr Chakraborty formed Expert

CAD Analytics, LLC, to pursue novel ideas to develop expert-level CAD that can be used as a first

reader; issues with current approaches to CAD are detailed in http://www.expertcadanalytics.com

Notation