New approaches to automated annotation of pathology level findings in medical images

17 3 An overview of the pathology-level medical image annotation system 19 3.1 Feature extraction... Existing works inpathological annotation often require large and fully annotated trai

New approaches to automated annotation of pathology-level findings in brain images

DINH THIEN ANH

Bachelor of Computing National University of Singapore

A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY

SCHOOL OF COMPUTING NATIONAL UNIVERSITY OF SINGAPORE

2013

First and foremost, I would like to express my deepest gratitude to my thesis advisor,

Dr Tze-Yun Leong, for her incisive guidance, encouragement, patience and immensesupport through out my Ph.D career And I have learned a lot from her She has alsoprovided me with an excellent research environment that is full of freedom Withouther help and belief, I would not have finished my dissertation

I am very grateful to have Dr Choie Cheio Tchoyoson Lim from National roscience Institute as my medical advisor Despite his extremely busy schedule, he

Neu-is always available to share with me hNeu-is valuable medical knowledge and insightfulfeedbacks for my research In addition, thanks to his generous reference, I have thehonor to receive the Singapore Millennium Scholarship for my graduate study

I am also much indebted to Dr Tomi Silander for being such an excellent mentorand for his inputs in my research He has helped me to overcome so many obstacles

in my research Together with Dr Tze Yun Leong, he has reviewed my thesis andprovided many thoughtful suggestions, which help me to improve this thesis tremen-dously I cannot thank him enough for his devotion And I have also benefited somuch from his wide knowledge and constructive advices

I am very fortunate to have several other mentors and collaborators along the way

I am thankful to Dr Chew Lim Tan for his financial support which funded me asResearch Assistant through the last year of my study I would like to thank Dr BoonChuan Pang and Dr Cheng Kiang Lee from National Neuroscience Institute for pro-viding me the traumatic brain injury dataset My sincere thank goes to Dr Tianxia

Gong for providing me the labelled training dataset and her valuable experience in themedical imaging field.

I wish to extend my thanks to Dr Dinh Truong Huy Nguyen, Dr Duc HiepChu, Thang Truong Duc, Dr Bolan Su, Quang Loc Le, Thuy Ngoc Le, Thanh TrungNguyen, Zhuoru Li, and many more great friends and colleagues through out the yearsfor their friendship, ideas, encouragement and support Without their accompanies, Iwould not have had that much fun in my life

My heartfelt gratitude goes to my fiance Ngoc Yen for her unconditional love,encouragement, patience, loyalty and for standing by me in both good and bad times.She has been virtually working as hard as me on this thesis I am completely amazed

at her willingness to proof read my writing countlessly She is truly a gift that I am

so blessed to have Thank you dear from the bottom of my heart and I am lookingforward to starting a family with you

Last but not least, I am extremely grateful to my parents for their unbounded loveand sacrifice; and my elder brother and sister-in-law for their encouragement and un-derstanding My parents have been giving me many wonderful opportunities in life

I am forever thankful to have such an amazing family, and there is no word that candescribe how much I love them The past six years have been a bumpy ride for me.And their love, care, sacrifice, support and encouragement have made it become mucheasier Thus, I owe this to my family

Table of Contents

1.1 Problem 3

1.2 Approaches and contributions 5

1.3 Problem formulation 8

1.4 Road map of this thesis 9

2 The medical domains 11 2.1 Ischemic stroke 11

2.1.1 Definition 11

2.1.2 Related work 13

2.2 Traumatic brain injury 14

2.2.1 Definition 14

2.2.2 Related work 16

2.3 Summary 17

3 An overview of the pathology-level medical image annotation system 19 3.1 Feature extraction 20

3.2 Modelling 21

3.3 Annotation 22

3.4 Evaluation metrics 22

3.5 Summary 23

4 Related work 27 4.1 Overview 27

4.1.1 Generative models vs Discriminative models 27

4.1.2 Ensemble learning 29

4.2 Feature extraction 31

4.2.1 Global features 31

4.2.2 Local features 32

4.3 Annotating natural images 33

4.3.1 Translation paradigm 34

4.3.2 Relevance Models 35

4.3.3 Other approaches 35

4.3.4 Discussion 36

4.4 Annotating medical images 38

4.4.1 Organ-level annotation 39

4.4.2 Pathology-level annotation 40

4.5 Summary 41

5 A generative model based approach 43 5.1 Introduction 43

5.2 Data 45

5.3 Image Processing Component 46

5.3.1 Automated lesion segmentation 47

5.3.2 Feature Extraction 48

5.4 Generative model 50

5.5 Content-based retrieval 56

5.6 Result 57

5.7 Discussion 58

5.8 Summary 59

6 A discriminative-model based approach 61 6.1 Introduction 62

6.2 Data 63

6.3 Methods 65

6.3.1 Feature extraction component 65

6.3.2 Classification system 66

6.4 Evaluation 74

6.4.1 Without global features 74

6.4.2 With global features 76

6.5 Discussion 77

6.6 Summary 78

7 Unsupervised classification by combining case-based classifiers 81 7.1 Introduction 81

7.2 Methods 83

7.2.1 System architecture 84

7.2.2 Gabor feature extraction 84

7.2.3 Sparse representation-based classifier 87

7.2.4 Ensemble of weak classifiers 89

7.3 Experiments 90

7.3.1 Materials 90

7.3.2 Experimental setup 92

7.3.3 Results 93

7.4 Summary 96

8 Automatic Traumatic Brain Injury prognosis 99 8.1 Introduction 100

8.2 Data 102

8.3 Method 102

8.3.1 Preprocessing and feature extraction 103

8.3.2 Classification of CT image slices 104

8.4 Evaluation 108

8.5 Discussion 109

8.6 Summary 110

9 Prototype implementation and informal evaluation 113 9.1 Implementation 113

9.1.1 GUI 116

9.1.2 Annotator 116

9.2 Informal evaluation 117

9.3 Summary 119

10 Conclusion 121 10.1 Summary 121

10.2 Proposed approaches and contributions 122

10.2.1 The generative model based approach 122

10.2.2 The discriminative model based approach 123

10.2.3 The unsupervised classification by combining case-based clas-sifiers 123

10.3 Future work 124

Medical image annotation aims to improve the e↵ectiveness and efficiency of based image retrieval In this work, we focus on automated pathology annotation thattries to identify potential pathologies, abnormalities and diseases from brain images.This is a challenging task because pathology annotation demands a deep understand-ing of the structural and functional changes induced by diseases Existing works inpathological annotation often require large and fully annotated training data, reliablesegmentation, and domain knowledge for hand-crafted feature extraction and selec-tion Since these prerequisites are not always feasible, they reduce the level of au-tomation, desirability, and practicality of the annotation systems

keyword-To mitigate the requirements of annotated training data and reliable segmentation,

we propose to use probabilistic generative models, since they support the integration ofexpert knowledge and e↵ectively handle the uncertainties inherent in the images andsegmentation However, when a priori knowledge is not available, these generativemodels are not able to achieve their best performance In this case, we suggest us-ing a discriminative model which incorporates an automated feature selection method

to tackle the problem Specifically, sparse group lasso provides a flexible selectionmechanism that helps to handle annotation problems without relying on the domainknowledge

The performance of existing annotation methods heavily depends on the quality

of hand-crafted features extracted from an automatic image segmentation To achievegood performance, constructing the system requires a considerable amount of man-ual work We propose to combine an unsupervised feature extraction technique with

a case-based classification in an ensemble learning framework to improve the ability and automation of the annotation systems The unsupervised nature of thisnon-parametric technique can significantly reduce the time and e↵ort for system cali-bration

adapt-To evaluate these approaches, we select two important neurological disorders - chemic stroke and traumatic brain injury, as illustrative domains because imaging find-ings of these diseases play significant roles in their diagnosis Despite the additionalchallenges due to the relaxation of the common prerequisites in existing systems, our

is-proposed frameworks still show reasonable performance An informal evaluation withexpert users has also demonstrated the practical promise of the proposed system.

Publications from the dissertation

research work

1 Automated predication of Glasgow Outcome Scale for Traumatic Brain Injury,

Lu, Boon Chuan Pang, C C Tchoyoson Lim, Cheng Kiang Lee, Chew Lim Tan,Tze-Yun Leong,

Proceedings of the 22nd International Conference on Pattern Recognition (ICPR2014),

Stockholm, Sweden August 2014 (To appear)

2 Unsupervised medical image classification by combining case-based classifiers,Thien Anh Dinh, Tomi Silander, Bolan Su, Tianxia Gong, Boon Chuan Pang, C

C Tchoyoson Lim, Chiang Kiang Lee, Chew Lim Tan, and Tze Yun Leong,Proceedings of the 14th World Congress on Health and Medical Informatics(MEDINFO 2013),

Copenhagen, Denmark August 2013

3 An automated pathological class level annotation system for volumetric brainimages,

Thien Anh Dinh, Tomi Silander, C C Tchoyoson Lim, and Tze Yun Leong,Proceedings of the American Medical Informatics Association Annual Sympo-sium (AMIA 2012),

Chicago, USA November 2012

4 A generative model based approach to retrieving ischemic stroke images,Thien Anh Dinh, Tomi Silander, C C Tchoyoson Lim, and Tze Yun Leong,Proceedings of the American Medical Informatics Association Annual Sympo-sium (AMIA 2011),

Washington D.C., USA October 2011

List of Tables

1.1 Summary of the proposed approaches The rationale of developingdi↵erent methods is to address di↵erent perspectives of the annotationproblem The generative model proposed in Chapter 5 will be the mostpowerful method if expert knowledge and prior knowledge are avail-able for the system This framework is capable of using weakly an-notated training data, handling variable size input data, and outputtingmany labels However, prior knowledge might not always be availablefor the inherently perceptual tasks of medical image pattern identifica-tion Therefore, generative model approach is not always an optimalchoice The unsupervised method proposed in Chapter 7 aims to im-prove the automation and reduce the amount of required manual workfor setting up the system Although it doesnt focus on performingbetter than the discriminative method proposed in Chapter 6, its ob-jective is to achieve a reasonable result as compared to less automatedmethods Hence, we employ a generic and straightforward featureextraction method (Gabor filter) and sparse representation-based clas-sifier While the proposed discriminative method in Chapter 6 yieldsbetter performance, it involves more preprocessing and calibration ef-forts (e.g., region based features, time-consuming feature selection

3.1 Summary of the proposed approaches for automatic pathological notation in brain images 24

an-5.1 An example of a feature vector for a single scan Notice the missing information for true lesions For the training data the TOAST class of

the stroke is also known 50

5.2 Precision and recall of the classifier for each subtype 58

5.3 Detail breakdown on the performance of classifiers 58

6.1 A volumetric CT brain scan with 19 slices 64

6.2 Preprocessing and segmentation process (a) original image, (b) im-age after skull removal (c) imim-age after normalization process, (d) seg-mented region 67

6.3 Extracted features for each potential region 68

6.4 Description of the global features 68

6.5 Precision and recall of classifiers using di↵erent feature selection tech-niques 75

6.6 Precision and recall of classifier for di↵erent and G 75

6.7 Confusion matrix 76

6.8 Average precision and recall of classifiers and their standard devia-tions after integrating global features 76

7.1 Images (a) and (c) are examples of extradural hematoma (EDH), while image (b) features subdural hematoma (SDH) 91

7.2 Average precision and recall for di↵erent methods The standard de-viations over several foldings listed in parenthesis 94

7.3 Average precision and recall of classifiers when varying the ensemble size and fixing the number of features at 1000 95

7.4 Average precisions and recalls of classifiers when varying number of features and fixing the number of classifiers at 50 95

7.5 Comparison of the proposed framework and Gong et al [38] 96

8.1 Glasgow outcome scale (GOS) 101

8.2 GOS prediction accuracy of di↵erent methods 109

List of Figures

1-1 An example of an ischemic stroke dataset 3

2-1 A CT image with ICH 14

2-2 A CT image with EDH 15

2-3 A CT image with SDH 15

3-1 An overview of the pathology-class image annotation system 20

5-1 The population of ischemic stroke dataset 45

5-2 System overview - These three components correspond to the feature extraction component, modelling component and annotation compo-nent in the general image annotation framework described in Chapter 3 46

5-3 Segmentations of ischemic lesion 49

5-4 A single lesion tracked through multiple slices of a DWI scan 49

5-5 The generative model describes the joint probabilities of scans S (see Table 5.1 for its structure) and tags (T1,T2, ,Tk) The random vari-able O denotes the TOAST subclass of the scan and it may take one of the values from the set {LAA, SVO, CE, non-stroke} C is a binary vector of size N denoting which lesions are real 51

5-6 KDE for P(sizei | ci,O) for di↵erent subtypes LAA, SVO, CE 54

5-7 KDE for P(zi | ci,O) for di↵erent subtypes LAA, SVO, CE 54

5-8 KDE of P(xi,yi | O, ci,zi) with zi = {12, 6, 20} after EM for di↵erent subtypes LAA, SVO, CE respectively 54

5-9 Retrieval component 56

6-1 The population of each TBI subtype in the experiment dataset 636-2 Overview of the classification system (or the modelling and annotationcomponents in the general framework described in Chapter 3) 696-3 Mapping from a variable set of features to a uniform-length vector 707-1 System architecture: The Gabor features of preprocessed images are

which are then combined to get the final class label In this ture, the preprocessing and Gabor feature extraction correspond to thefeature extraction component in the general framework Meanwhile,the weak classifiers and the ensemble system are the modeling andannotation components correspondingly 857-2 Values of the sparse coefficients x recovered from Algorithm 3 of asingle test image on our training data The number of nonzero coeffi-cients is only 5% of the total 887-3 Histogram of the ratios of nonzero coefficients over all coefficients ofall testing images 897-4 The population of each TBI subtype in the experiment dataset 918-1 Architecture of our proposed auto-prognosis system 1038-2 The workflow of construction training data for logistic regression model1079-1 An overview of the pathology annotation prototype system The dashedline indicates the preprocessing phase of the system The connectedline indicates the annotation phase of the system 1149-2 Five main components of the GUI: 1) Visualization of an upload im-age, 2) Classification result, 3) Decision stratification, 4) Control panel(Upload image, Predict, Save image and Next image) and 5) Visual-ization of related images 1159-3 Examples of the system when predictions are made for uploaded images118

architec-Chapter 1

Introduction

The importance of medical imaging in patient care has grown significantly in recentyears, especially in the diagnosis and treatment processes Advances in digital imag-ing technologies have led to a huge number of digital images being generated everyday There is an increasing need for efficiently accessing and retrieving relevant im-ages for teaching, research and diagnosis [111, 91, 56, 36] In a modern radiology de-partment, retrieval systems have become even more vital as evidence-based medicine

is widely adopted [88]

By indexing the correspondence of keywords and images, traditional text-basedimage retrieval (TBIR) systems can be used to solve the image search problem TBIRsystems, however, require laborious manual annotation of the images with the corre-sponding keywords Content-based retrieval (CBIR) systems, on the other hand, use

a query image to search for images with similar low-level visual features However,due to their reliance on the low-level features, CBIR systems are not suitable for an-swering abstract or high-level queries Automatic annotation technology combinesthe advantages of both TBIR and CBIR systems by first annotating the images withtheir semantic content, and then allowing the users to perform text-based search onthe image databases Semantic labels are automatically associated to the images bythe annotation system Generally, this association can also be considered as a classifi-

cation problem.

The are two major challenges in image annotation [79] The first challenge is thesemantic gap problem, which refers to the difficulty of deriving semantic informa-tion from low-level image features, e.g., texture and color The second challenge isthe lack of correspondence between the image regions and annotation keywords inthe training data Although many automatic image annotation techniques have beenproposed, most of the e↵orts are targeted at annotating natural images Since medicalimages and natural images (e.g., a Computed Tomography (CT) scan of a human brainand a natural scene image) are fundamentally di↵erent in terms of their appearance,dimensionality (e.g., 3-D data vs 2-D data), structure and content, existing techniquesfor annotating natural images are not always suitable for annotating medical images

As a result, automated annotation techniques for medical images need to be designeddi↵erently

There are two types of medical image annotation: organ-level annotation andpathology-level annotation Organ level annotation is the process of annotating gen-eral aspects of the images such as their modality (e.g., CT) or the anatomical structures

in an image (e.g., organ identification) Alternatively, pathology-level annotation cuses on the information about the pathologies, abnormalities and diseases shown inthe image For example, in the domain of ischemic stroke, each Magnetic ResonanceImaging (MRI) brain scan often needs to be categorized according to its subtype.Many past e↵orts mainly focused on organ-level annotation problems [19, 104, 24]

fo-In recent years, pathology annotation has gained more attention [3] because it notonly enables more semantic-based queries but also empowers clinical decision sup-port tools Nevertheless, pathology-level annotation is a usually a more challengingtask due to the uncertainty inherent in the abnormalities, the complexity of the patho-logical patterns exhibited in the images, and the unavailability of the training data

1.1 Problem

Before we formally define the pathology-level image annotation problems and theirchallenges, we will briefly describe the medical domain of ischemic stroke that serves

as a motivating illustration for tackling this class of problems

Figure 1-1: An example of an ischemic stroke dataset

Stroke is a major cause of death and permanent disability in the world An earlyclassification of the stroke subtype can improve both short-term and long-term treat-ment and management of patients Clinical imaging features, which are usually ac-quired by MRI, are important factors in the classification process Based on thesevisual findings, a patient can be classified into di↵erent ischemic stroke subtypes [64].Specifically, each MRI brain scan represents a 3-D scan of a patient’s head (Figure 1-1) In stroke imaging, clinical imaging features are the abnormalities which often arethe hyper intensive regions in the MRI images The objective of pathology annota-tion in ischemic stroke is to automatically annotate a 3-D MRI brain scan with itscorresponding pathology class For instance, when small scatter lesions are observed

in one vascular territory of a patient’s brain, the scan images can be annotated as the

large-artery atherosclerosis ischemic stroke subtype [57].

The following are the main challenges in pathology annotation in this domain:

1 Semantic gap: The mapping between low-level image features and high-levelimage semantics is challenging in pathology annotation due to the complex-ity of the pathology A subtle change in the image could indicate a di↵erentpathological class Uncertainty and noise in the medical image feature extrac-tion techniques would also adversely a↵ect the performance of the annotationsystem

For example, in ischemic stroke, patterns of lesions determine a pathology class

of an image Unsupervised or supervised feature extraction methods are oftenused to recognize abnormal lesions Although unsupervised methods are moreautonomous, they are not always able to di↵erentiate the lesions from noises.Alternatively, supervised methods (e.g., image segmentation), which are accu-rate for extracting abnormalities, are not always reliable due to the complexity

of the images [88] Manual calibrations are often required when new data rives

ar-2 Spatial relations of abnormal regions: Abnormal regions in medical images areoften found to be structured Relationships among the findings are critical forunderstanding the images It is common that the annotation of a certain disease

or disorder does not always depend on a single finding but rather depend on a set

of findings For example, to annotate the pathological class for ischemic strokeimages, the lesions need to be considered in groups They can be scattered indi↵erent slices of a brain scan

3 Domain knowledge: Expert knowledge plays an important role in the pathologyannotation system It can make up for the lack of training data and guide the sys-tem to obtain better results A natural question that arises concerns integratingsuch knowledge into the system At the same time, since it is not always easy

to extract and quantify domain knowledge, the requirement of prior knowledgecould become an obstacle.

4 Annotated training data: Although medical imaging data is abundant, tated data is very rare It is labour-intensive and time-consuming to (manually)annotate medical images Lack of annotated data poses challenges in apply-ing standard machine learning techniques to the annotation task For example,

anno-in our ischemic stroke dataset, the traanno-inanno-ing data is only labelled with ischemicstroke subtype while the exact locations of lesions are still unknown Classifica-tion techniques requiring a large training dataset are not practical since labelledtraining data are limited as compared to other domains

The above challenges are relevant not only to ischemic stroke but also to otherbrain diseases or disorders (such as Traumatic brain injury) and in di↵erent imagemodalities (such as CT, MRI or X-ray)

1.2 Approaches and contributions

The aim of this thesis is to propose solutions to the challenges of annotating medicalimages, especially of the brain, at the pathology level The annotation is guided byabnormalities found in the image An abnormality can be a tumor, a malignancy or alesion

Many methods have been proposed for pathology annotation of brain images.However, large sample size, fully annotated training data, reliable segmentation re-sults, and availability of domain knowledge for hand-crafted feature extraction andselection are often the prerequisites for these techniques Since these demands are ex-pensive and difficult to be fulfilled, they reduce the automation, robustness and practi-cality of the annotation systems In addition, most existing techniques can only handle2-D brain images

We propose three novel techniques (Table 1.1) based on the generative model, thediscriminative model and ensemble learning to address these problems Although theyare all solving the annotation problem, each of them addresses di↵erent requirements.

No single method is optimal for all the requirements They are summarized as follows:

• We present a generative model based framework to address the requirements

of large and annotated training data Since it supports integration of expert’sknowledge, the generative model is e↵ective in handling the uncertainty inher-ent in the medical images and image segmentation Empirical studies demon-strate that the proposed system can achieve good performance even when usinglimited training datasets

• When prior knowledge is inadequate, generative models cannot achieve theirbest performance In order to mitigate the dependency on domain knowledge,

we introduce a discriminative model based framework This combines the sparsegroup lasso feature selection technique, which allows simultaneous group selec-tion and feature selection, and the kernel-based discriminative method Withoutrelying on prior knowledge, the proposed framework can handle a large number

of candidate features and yield promising results in the annotation task

• The performance of existing annotation methods is significantly influenced bythe quality of hand-crafted features extracted from automatic image segmenta-tion Hence, system calibration is often required Consequently, the robustnessand automation of the annotation systems are compromised We propose tocombine an unsupervised feature extraction technique with case-based classi-fication in an ensemble learning framework to circumvent these manual pro-cesses Despite additional challenges in further automating some of the systemtasks, the proposed framework still performs reasonably as compared to existingmethods

We also extend the case-based classication framework with logistic regression

Method Requirements Main objective Strengths Generative based method 1 Expert knowledge

2 Hand-crafted features 1 Small dataset2 Complicated visual patterns 1 Weakly annotated trainingdata

2 The most limited training dataset

3 Spatial relations of mal regions

abnor-4 Integrates domain edge

knowl-5 Multi-class labeling Discriminative based method 1 Hand-crafted features

2 Feature selection process 1 Less domain knowledge2 Small dataset 1 Weakly annotated trainingdata

2 Small training dataset

3 Provides insight on tures of data

2 Reduces manual work 1 Automation2 Minimum manual

calibra-tion

Table 1.1: Summary of the proposed approaches The rationale of developing di↵erentmethods is to address di↵erent perspectives of the annotation problem The generativemodel proposed in Chapter 5 will be the most powerful method if expert knowledgeand prior knowledge are available for the system This framework is capable of us-ing weakly annotated training data, handling variable size input data, and outputtingmany labels However, prior knowledge might not always be available for the inher-ently perceptual tasks of medical image pattern identification Therefore, generativemodel approach is not always an optimal choice The unsupervised method proposed

in Chapter 7 aims to improve the automation and reduce the amount of required ual work for setting up the system Although it doesnt focus on performing betterthan the discriminative method proposed in Chapter 6, its objective is to achieve a rea-sonable result as compared to less automated methods Hence, we employ a genericand straightforward feature extraction method (Gabor filter) and sparse representation-based classifier While the proposed discriminative method in Chapter 6 yields betterperformance, it involves more preprocessing and calibration e↵orts (e.g., region basedfeatures, time-consuming feature selection process)

man-to support risk stratification of based on medical images Unlike existing proaches, which only work on 2-D images, this method can be extended todirectly handle 3-D brain images without requiring annotated training data

ap-In summary, our contributions are as follows

• We have carefully examined the challenges in pathology-level annotation

• We have proposed three novel approaches to existing unaddressed challenges ofpathology-level annotation problem

• Our work provides a solid step toward improving the automation and practicality

of existing pathology-level annotation systems

• The proposed methods are evaluated into two important neurological domains,ischemic stroke and traumatic brain injury

1.3 Problem formulation

We will now present a more systematic formulation of the problem(s) addressed cussions on the proposed solutions will be based on this formulation Suppose that weare given a brain image I which is known to exhibit a certain disease or disorder D

Dis-We know that disease D can have M di↵erent types The problem is bounded to brainimages which are acquired in 2-D or 3-D format (such as MRI, X-ray or CT brainimage)

Assume that in the disease or disorder D, a di↵erent subtype t can be categorized

by di↵erent visual patterns of abnormal findings in an image For example, an mal finding can be a malignant tumor or a lesion This is the core assumption whichenables our proposed system to perform properly For simplicity, we also assume thateach image exhibits only one subtype t of abnormality This assumption, while ap-plicable in many domains and tasks including those that we have examined, can berelaxed in future

abnor-Each image I have N abnormal findings abnor-Each image has a di↵erent number ofN(I) abnormal findings N(I) is influenced by a subtype t Spatial relationships amongthese abnormalities can be important factors in recognizing the disease or disorder Forexample, in ischemic stroke, depending on the locations of multiple small lesions, apatient can be categorized to di↵erent subtype of ischemic stroke

A feature extraction technique is applied to each image to identify K potentialfindings We use the word potential to indicate that an identified finding might not be

a “true” abnormal finding Rather a finding can be an artifact or noise resulting from

image acquisition process or a false signal picked up by the feature extraction from theimage K can be smaller or bigger than the original number N(I) For each finding,

we compute a set of real-value features representing the visual properties such as itsposition, size, texture, intensity and shape

Due to the expensive and time-consuming process of acquiring annotated trainingdata which indicates exactly true findings from a set of potential findings, such anno-tated data is often not available As a result, N(I) is unknown too Each image I andits corresponding disease subtype t are represented as a pair (I, t)

In summary, brain image annotation can be formulated as a classification lem with ambiguous features Ambiguous features are mainly due to a noisy set ofabnormal findings and partially annotated training data For example, in the task ofannotating ischemic stroke images with their ischemic stroke subtypes, each scan isrepresented by a variable set of abnormal findings Abnormal findings, which areextracted by automated segmentation techniques, consist of both true lesions and arti-facts The training data only consists of subtype t for each scan I

prob-In the scope of this thesis, we aim to address the ambiguities arising from featureextraction techniques and the lack of annotated training data We also address thegeneral lack of domain knowledge and limited training data size

1.4 Road map of this thesis

The following is a road map of the remaining chapters of this thesis

In Chapter 2, we discuss the domain knowledge and related work in ischemicstroke imaging and traumatic brain injury imaging

In Chapter 3, we present the general framework of a pathology-level image tation system

anno-In Chapter 4, we survey existing techniques in annotating natural and medicalimages Before going into the details of these techniques, we will briefly present the

strengths and weaknesses of three general machine learning approaches commonlyused for these tasks, including discriminative models, generative models and ensemblelearning We will then illustrate the common methods in annotating natural imagesand the challenges in applying to the medical imaging domain We will end the chapterwith a discussion of using existing techniques in annotating medical images and theirlimitations.

In Chapter 5, we discuss our generative model-based approach To evaluate themethod, we have conducted experiments on annotating the ischemic stroke subtypesfor MRI brain images

When prior knowledge is not available, the strengths of generative models not be easily exploited In Chapter 6, we demonstrate a discriminative model basedframework to address such challenges

can-The performance of annotation systems often relies on the results extracted fromautomated segmentation results Since it is difficult and time-consuming to calibratethis process, it limits the automation capability and robustness of the annotation sys-tem In Chapter 7, we propose to combine distributed representation with case-basedclassification method for this problem In Chapter 8, we extend the proposed approach

in Chapter 7 to handle 3-D medical images

In chapter 9, we present and evaluate a prototype of the pathology annotationsystem Feedback from doctors on the prototype is also discussed in this chapter.Finally, in Chapter 10, we summarize the achievements and limitations of thiswork We also discuss possible extensions and improvements for future research

Chapter 2

The medical domains

In this chapter, we will briefly discuss the definitions of ischemic stroke and traumaticbrain injury, which will serve as illustrated domains for our proposed approaches.They are two major neurological disorders where imaging findings are significant de-cision factors for diagnosis and treatment of these diseases

In addition, an overview of existing computerized methods for automated tion and classification in these two domains is also included in this chapter

annota-2.1 Ischemic stroke

2.1.1 Definition

Stroke is a sudden development of neurological damage It occurs in an acute andunexpected way Stroke can be classified into two main types: hemorrhage (15%)and ischemia (85%) In this work, we only consider ischemic stroke, which occurswhen blood vessels that carry blood to the brain become narrowed or occluded, caus-ing insufficient blood supply for normal cellular function Early classification of is-chemic stroke subtype is important for both short-term and long-term treatment andmanagement of ischemic stroke patients The Trial of ORG 10172 in Acute StrokeTreatment (TOAST) and Oxfordshire classification criteria [4] are the most widely

used formal systems for stroke subtype classification These systems rely on clinicalimaging features and ancillary diagnosis test results, and can be used to predict prog-nosis as well as the underlying ischemic stroke etiology The TOAST system consists

of five stroke subtypes: large artery atherothromboembolism (LAA), cardioembolism(CE), small vessel occlusion (SVO), stroke of other determined etiology, and stroke ofundetermined etiology Meanwhile, the Oxford stroke classification clinically subdi-vides cerebral infarction into four subtypes: total anterior circulation stroke (TACS),partial anterior circulation syndrome (PACS), posterior circulation syndrome (POCS)and lacunar syndrome (LACS)

Acute stroke imaging is crucial in the management of acute ischemic stroke ing can rule out nonvascular causes of the symptoms and define the extent of an area

Imag-of acute ischemic change Moreover, it can identify the cause Imag-of an acute ischemicprocess Computed tomography (CT), CT angiography (CTA), Magnetic resonanceimaging (MRI), di↵usion-weighted MRI (DWI), perfusion MRI and MR angiogra-phy (MRA) are popular imaging technologies for evaluation of patients with ischemicstroke’s symptoms

MRI, especially di↵usion-weighted MRI (DWI), has shown to be superior to otherimaging modalities such as CT in improving the accuracy of classifying ischemicstroke subtypes [64] DWI images reveal relative hyperintense signals in regions withpotential ischemic brain lesions Lee et al [64] confirm that the combination of DWIand MRI within 24 hours of hospitalization significantly improves the accuracy ofearly classification of ischemic stroke subtype In another study, Kang et al [57] findclose association of early DWI lesion patterns with TOAST stroke subtypes Theyhighlighted the associations between certain ischemic lesion patterns appearing onDWI and ischemic stroke causes

Inspired by these clinical findings, we wish to develop a computerized methodthat utilizes the visual patterns from CT and DWI MRI for classifying subtypes ofischemic stroke

2.1.2 Related work

Although ischemic stroke imaging has been a very active research field in recent years,the focus has been limited to attempts at segmentation of ischemic lesions from var-ious image modalities such as CT and MRI To the best of our knowledge, there hasnot been any work taking the next step to automating annotation of ischemic strokeimages The earliest attempt in an automated segmentation of stroke lesions fromDWI images is introduced by Martel et al [81] They introduce a method of using theadaptive thresholding algorithm with spatial constraints for segmentation Matesin

et al [82] apply seeded region growing algorithm and rule-based labeling to nize brain lesions from CT images Usinskas et al [105] introduce an unsupervisedclassifier to identify stroke and non-stroke regions However, thresholds have to bespecified separately Kabir et al [55] suggest taking advantage of di↵erent kinds ofanatomical information provided by di↵erent imaging modalities by using multipleMRI sequences such as T2, Flair and DWI [98] to recognize lesions from input im-ages However, the proposed method is only used for automatic segmentation of strokelesions

recog-In summary, previous works focused mainly on identifying lesions from 2-D MRIimages The attempt to move from identifying lesions to classifying 3-D MRI imagesinto stroke subtypes requires additional e↵ort because of two main reasons First, inthe case of 3-D MRI images, it is necessary to pre-select an important slice (the onewith abnormal lesions) for the system to annotate, but this pre-selection process is notapplicable in the case of 2-D images Second, stroke subtypes depend on multiplelesions from di↵erent slices Furthermore, existing works on ischemic stroke classi-ciation need substantial annotated training data, which is difficult to acquire In ourwork, we directly use labelled training data, i.e., images with only the subtype labels,but without the detailed annotated lesions or patterns

2.2 Traumatic brain injury

2.2.1 Definition

Traumatic brain injury (TBI) is a major cause of death and disability worldwide It

is usually caused by external mechanical forces such as falls, vehicle accidents andviolence CT is an important tool for diagnosis and assessment for this disease due toits ability of clearly indicating fresh bleeding and bony injury as well as its availability

in the clinical setting TBI brain damages include several types of hemorrhages (orhematoma): extradural hematoma (EDH), subdural hematoma (SDH), intracerebralhemorrhage (ICH), subarachnoid hemorrhage (SAH) and intraventricular hematoma(IVH)

Radiologists, trained specialists and senior residents usually categorize TBI byscrutinizing the CT images

Figure 2-1: A CT image with ICH

EDH (Figure 2-2) happens when blood from a rupture of blood vessels bleeds intothe space between the dura matter and the skull Dura matter is the hard membrane ofthe central nervous system On images produced by CT, EDH usually has a biconvexshape with a well defined margin since the expansion of hematoma often stops at theskull’s sutures where the dura matter is tightly closed to the skull

SDH (Figure 2-3) occurs when bloods build up in between of the dura matterand arachnoid matter which is the outer cover of the brain Bleeding mainly comes

Figure 2-2: A CT image with EDH.

Figure 2-3: A CT image with SDH

from veins crossing the subdural space SDH is often crescent-shaped with a concavesurface away from the skull However, in its early stage of development, it often has aconvex appearance Hence, it may cause confusion when di↵erentiating subdural andepidural hemorrhages EDH and SDH can happen simultaneously.

Unlike SDH and EDH, ICH (Figure 2-1) happens within the brain tissue ratherthan outside It is usually caused by a stretching and shearing injury On CT images,ICH can be recognized as a brighter tissue separated from the skull by brain tissue.The surrounding tissue of a bleed often appears darker than the rest of the brain on theimage due to its lower density

SAH is a bleeding in the area between the arachnoid membrane and the pia mattersurrounding the brain It can happen by itself or together with other intracerebral orextracerebral hematomas

IVH is a bleeding into the brain’s ventricular system where the cerebrospinal fluid

is produced and circulated towards the subarachnoid space

2.2.2 Related work

One of the earliest attempts to annotate CT brain images is from Cosic and garic [21] who propose a rule-based approach for labeling intracerebral brain hemor-rhage (ICH) on CT brain images Liao et al [70] implement a pathology-level imageclassifier for CT brain images using a decision tree-based approach Their trainingdata consists of 48 images classified into three hematoma types: epidural, subduraland intracerebral; the authors assumed that all the images have hemorrhage regions.Zhang and Wang [115] propose to classify normal and abnormal CT brain images

Lon-by using global image features such as intensity, shape, texture and symmetry Theyused the C4.5 decision tree algorithm and radial basis function neural network for theclassification

Obtaining annotated training data requires substantial e↵ort To address the lem of a lack of annotated training data, Gong et al [37] propose to use a traditional

prob-machine translation paradigm to find the correspondence between pathology labelsand the regions of interest They used an unsupervised approach to label the imageswith keywords extracted from the associated reports This work, however, does notaddress the case where the extracted regions of interest from segmentation are notreal hemorrhage regions In addition, this work only focus on classifying 2-D brainimages.

2.3 Summary

Ischemic stroke and traumatic brain injury are two important domains where medicalimaging is important Classifying a subtype of an ischemic stroke patient or cate-gorizing TBI patients is mainly performed by experienced radiologists Therefore,computerized systems that can automate such tasks are particularly helpful for juniorradiologists and medical students.ag

In ischemic stroke, existing computer-aided pattern identification techniques mainlyfocus on the image segmentation tasks Automated classification techniques still re-main very primitive To the best of our knowledge, our proposed approach in annotat-ing the ischemic stroke subtypes is the first in this domain Meanwhile, there are morecomputerized classification methods available in the TBI domain Yet, existing tech-niques usually depend on the availability of perfect segmentation results and annotatedtraining data, which are very time-consuming and labour-intensive to obtain Further-more, the majority of related e↵orts can only handle 2-D brain images Hence, thereare needs to improve the automation and practicality of existing annotation methods

Chapter 3

An overview of the pathology-level

medical image annotation system

In this chapter, we describe a general framework for the pathology-level annotation ofmedical images The framework supports automated or semi-automated annotation ofpathology-level information in input images The framework is able to handle images

in 2-D or 3-D format and from di↵erent modalities (such as CT, MRI, X-ray)

Annotated images can be directly used as a decision support tool or as source datafor content based image retrieval systems Thus, this framework is potentially usefulfor diagnosis, treatment, research and teaching purposes

One of the highlights of this framework is the ability to handle weakly labelleddataset, where annotated abnormal regions are not required As a result, the e↵ort foracquiring training data can be greatly reduced

The framework consists of three main components: feature extraction, modellingand annotation (Figure 3-1)

The life cycle of the annotation system can be divided into two di↵erent phases: 1)Training phase where training samples are used to construct modelling and annotationcomponents and 2) Annotating phase where the system is actually used for annotatingnew images

Training' (volumetric)' images'

Feature'extrac5on'

Figure 3-1: An overview of the pathology-class image annotation system

As shown in Figure 3-1, the connected and dashed boxes illustrate the training andannotating phases respectively

In the training phase, the training images are first processed by the feature tion component Then, the modeling component takes these extracted features as inputand applies machine learning techniques on them Once constructed, the model will

extrac-be used in the annotation component

In the annotating phase, the features from a new image are also extracted by thefeature extraction component before they are sent to the annotation component Onceannotated, the image is ready for the retrieval process

3.1 Feature extraction

Since an image is an unstructured array of pixels, extracting suitable visual featuresfrom these pixels is usually the first step of an image annotation system An appropri-ate representation of image significantly improves the performance of the annotationtechniques

Feature extraction techniques are often categorized into two categories: local ture and global feature Global features are directly calculated from the whole im-age while region-based feature extraction requires prior image segmentation Localmethods divide images into blocks/regions and a set of features is computed for eachblock/region.

fea-Depending on a feature extraction technique, the modelling component is required

to adapt to the format of input For example, global methods produce a length feature vectors independent of the input image while region-based techniquesprovide a variable size feature vectors according to the input image Direct application

uniform-of common machine learning techniques might not be possible for a region-basedmethod

Moreover, when the features are extracted from the volumetric images, the featurescan be extracted in a 2-D (slice based features) or a 3-D (scan based features) manner

3.2 Modelling

Once the images are represented using the local features or global features, higherlevel semantics can be learned from the training samples by applying machine learningalgorithms

In pathology annotation problems, labelled training datasets are either directly belled datasets or indirectly labelled datasets (also called as weakly labelled datasets)

la-In a directly labelled dataset, each region, represented by vector feature, is directlyannotated with one or multiple labels For example, a segmented region from an image

is manually labelled with its category Meanwhile, in an indirectly labelled dataset, alabel is assigned to an image or to a scan (multiple images) For example, an image

is extracted with multiple regions but only a single label is collectively assigned forthese regions In the directly labelled dataset, samples can be fed directly into thealgorithms to build a model However, more steps are involved when dealing with an

indirectly labelled dataset.

In this modelling phase, we have an option either to combine multiple labels into

a single model or to separate each label to a binary model

Since feature extraction can work in a supervised manner, it is able to integratefeature extraction and modelling into a single process

During retrieval, images assigned to input keywords earlier will be directly trieved The advantage of this approach is that there is no need for expensive in-formation retrieval processes such as image indexing or online matching However,

re-by ignoring the fact that an image can belong to multiple categories, many relevantimages can be missed from the retrieval process

In contrast, multi-label methods annotate an image / a volumetric image with tiple semantic concepts

mul-3.4 Evaluation metrics

Since the main motivation for automated image annotation is for image retrieval, it

is common to use the retrieval metrics to indicate the performance of the annotationsystem In most image annotation works, precision and recall are often used as theperformance metrics; they are defined via the process of retrieving test images with asingle keyword as follows:

cor-3.5 Summary

In this chapter, we have briefly described the general framework of pathological notation systems Feature extraction, modelling and annotation are three main com-ponents of the framework Each of these components supports multiple approaches

an-of operation, with di↵erent advantages and disadvantages Depending on the domainknowledge, training data, feature extraction methods, feature selection methods andthe annotation objective, di↵erent modelling approaches can be employed to achieveoptimal performance

Table 3.1 summarizes our proposed approaches for pathology annotation of brainimages Each of them addresses di↵erent requirements and functionalities in theframework However, the common highlights of these approaches are the abilities

to deal with imperfect segmentation results, un-annotated and limited training data.The objective is to improve the e↵ectiveness, automation, robustness, and practicality

of the annotation systems

Generative model(Chapter 5)

Discriminativemodel(Chapter 6)

Sparse model +Ensemblelearning(Chapter 7)

Table 3.1: Summary of the proposed approaches for automatic pathological annotation

The discriminative model (Chapter 6) addresses the scenario where prior edge is not available Using a supervised feature selection technique and kernel-baseddiscriminative method, it can handle the same challenges addressed by the generativemodel without relying on domain knowledge The main drawback is that its perfor-mance is sensitive to the feature selection process

knowl-When reliable segmentation results are difficult to obtain, the unsupervised featureextraction based approach (Chapter 7) can o↵er better performance In addition, due tothe unsupervised nature of the non-parametric technique, it can significantly improve