A computational image based guidance system for precision laparoscopy

16 Figure I.7 Clamping of right portal vein and right hepatic artery image taken from [8] .... Common hepatic duct The liver is supplied by the portal vein and the hepatic artery that d

A COMPUTATIONAL IMAGE-BASED GUIDANCE SYSTEM FOR PRECISION

LAPAROSCOPY

_

A Dissertation Presented to the Faculty of the Department of Computer Science

A COMPUTATIONAL IMAGE BASE

GUIDANCE SYSTEM FOR PRECISION

LAPAROSCOPY

_ Toan B Nguyen

APPROVED: _

Dr Nikolaos V Tsekos, Chairman _Dept of Computer Science, University of Houston _

Dr Ioannis Pavlidis _Dept of Computer Science, University of Houston _

Dr Ricardo Vilalta _Dept of Computer Science, University of Houston _

Dr Marc Garbey Center of Computational Surgery, Houston Methodist Hospital _ Dean, College of Natural Sciences and Mathematics

Acknowledgment

This dissertation would not have been possible without the generous funding from the Vietnam Education Foundation (VEF) and the National Science Foundation's Industry/University Cooperative Research Centers program [NSF 1060222 IUCRC Center CYBHOR]

I would like to express my gratitude to Dr Vo Van Toi for introducing me to the program

of Computational Surgery sponsored by VEF The memories and motivation from when I worked in the Department of Biomedical Engineering of Vietnam National University – International University have brought me this far

I would also like to thank the University of Houston for providing the Competitive Scholarship and the Department of Computer Science for assisting me with paperwork and procedures

The MR and CT scan images used in this dissertation would not have been available without the hard work of the colleagues at MITTE, Houston Methodist Hospital

The advisory committee, consisting of Dr Marc Garbey, Dr Nikolaos Tsekos, Dr Pavlidis Ioanis, and Dr Ricardo Vilalta has been excellent at giving me valuable feedback throughout the duration of this dissertation

I would also like to thank the people at the lab of Computational Surgery for their help and support

I was able to complete this dissertation under the guidance of my supervisors, Dr Marc Garbey and Dr Brian Dunkin Their critiques and opinions about my work and trust in

my abilities shall never be forgotten; I am grateful for the support they have provided me

in all these years

Finally, I would like to dedicate this dissertation to people who mean more to me than my life – my father, mother, brother, wife, and soon-to-be-born baby Their patience, love, support and guidance throughout my life has made me the person I am today

A COMPUTATIONAL IMAGE-BASED GUIDANCE SYSTEM FOR PRECISION

LAPAROSCOPY

_

An Abstract of a Dissertation

Presented to the Faculty of the Department of Computer Science

University of Houston _

In Partial Fulfillment

of the Requirements for the Degree Doctor of Philosophy

By Toan B Nguyen December 2016

Abstract

This dissertation presents our progress toward the goal of building a computational image-based guidance system for precision laparoscopy; in particular, laparoscopic liver resection

As we aim to keep our working goal as simple as possible, we have focused on the most important questions of laparoscopy - predicting the new location of tumors and resection plane after a liver maneuver during surgery Our approach was to build a mechanical model of the organ based on a pre-operative image and register it to intra-operative data

We proposed several practical and cost-effective methods to obtain the intra-operative data in the real procedure We integrated all of them into a framework on which we could develop new techniques without redoing everything

To test the system, we did an experiment with a porcine liver in a controlled setup: a wooden lever was used to elevate a part of the liver to access the posterior of the liver

We were able to confirm that our model has decent accuracy for tumor location (approximately 2 mm error) and resection plane (1% difference in remaining liver volume after resection) However, the overall shape of the liver and the fiducial markers still left a lot to be desired

For further corrections to the model, we also developed an algorithm to reconstruct the 3D surface of the liver utilizing Smart Trocars, a new surgical instrument recognition

system The algorithm had been verified by an experiment on a plastic model using the laparoscopic camera as a mean to obtain surface images This method had millimetric accuracy provided the angle between two endoscope views is not too small

In an effort to transit our research from porcine livers to human livers, in-vivo experiments had been conducted on cadavers From those studies, we found a new method that used a high-frequency ventilator to eliminate respiratory motion

The framework showed the potential to work on real organs in clinical settings Hence, the studies on cadavers needed to be continued to improve those techniques and complete the guidance system

Contents

I Motivation and background 1

1 Introduction 2

1.1 Laparoscopic surgery 2

1.2 Motivation 4

1.3 Dissertation overview 6

2 Background 8

2.1 Anatomy of the liver 9

2.2 Liver segmentation - Couinard system 13

2.3 Liver resection 15

2.4 Finite elasticity theory 20

2.5 Biomechanical properties of liver 28

2.6 Finite-element for finite elasticity 30

2.7 Image segmentation 33

2.8 Literature review 39

2.9 Dissertation proposal 43

II Developing of the image-based guidance system 44

3 Building the framework 45

3.1 Main workflow of the framework 45

3.2 Description of the components 48

4 Validation of the mechanics modeling framework 57

4.1 Problem statement and method 58

4.2 Experimental setup 60

4.3 Model and simulation 64

4.4 Validation and discussion 71

4.5 Conclusion 80

5 New method for going from 2D-endoscope to 3D-acquisition of surface landmarks by software 81

5.1 State of the art 82

5.2 Smart Trocar 85

5.3 Method 90

5.4 Verification experiment 94

5.5 In vivo experiment and discussion 99

5.6 Discussion 103

6 Transition to human liver 105

6.1 Experiments on cadavers 105

6.2 New method for respiratory motion correction 115

III Dissertation review 119

7 Conclusion and future work 120

7.1 Validation of the tissue mechanics 121

7.2 Liver surface reconstruction using Smart Trocars 123

7.3 Future works 124

7.4 List of publications 125

Bibliography 126

List of figures

Figure I.1 Laparoscopic surgery (image taken from [3]) 3

Figure I.2 Liver anatomy (image taken from [6]) 9

Figure I.3 Surface and bed of liver (image taken from [7]) 10

Figure I.4 Vascular supply (image taken from [6]) 11

Figure I.5 Liver segmented by portal and hepatic veins (figure take from [8]) 13

Figure I.6 Liver segmentation (image taken from [8]) 16

Figure I.7 Clamping of right portal vein and right hepatic artery (image taken from [8]) 18

Figure I.8 Body in reference and current configuration 21

Figure I.9 A Closed Active Contour Model (image taken from [9]) 34

Figure II.1 Workflow of the framework 47

Figure II.2 Run the snake method to segment the liver from the scans 50

Figure II.3 Guide user interface for manual method 50

Figure II.4 Apply anisotropic filter to the image to reduce noise 51

Figure II.5 Apply white top-hat transform to the image 51

Figure II.6 Apply adaptive threshold with Otsu‟s method 52

Figure II.7 Apply new mask which comes from erosion transform of the old mask 52

Figure II.8 Do open operation with ball structure to remove small objects 53

Figure II.9 Tumor (black) location changes after lifting with a surgical tool 58

Figure II.10 Resection plane (green) changes after lifting with a surgical tool 58

Figure II.11 X-ray image of the liver showing guide wire, simulated tumor, and vessels 61

Figure II.12 Porcine liver in position A - stays flat down 61

Figure II.13 Porcine liver in position B - elevated (a) Top down view (b) Side view 62

Figure II.14 Liver parenchyma and wire changes between the flat (a) and elevated (b) Position changes are visualized using the imaging modalities 64

Figure II.15 Segmentation results of (a) fake simulated tumor (b) guide wire (c) liver boundary (d) 2D-top view of tumor (green), guide wire (red) inside the liver (blue) 66

Figure II.16 Geometry of liver‟s simulation in COMSOL (a) without liver (b) with liver 67

Figure II.17 Stage 1 of the simulation: Liver shape changes from A to B‟ 69

Figure II.18 Stage 2 of the simulation: Liver shape changes from B‟ to B 70

Figure II.19 Tumor centroid, guide wire, and liver upper curve comparison on same 2D-cut in position B (a) linear-elastic model and (b) hyper-elastic model 72

Figure II.20 Tumor-centroid error based on mesh type: Linear-elastic model vs hyper-elastic model 74

Figure II.21 Tumor-centroid error based on mesh type: Young‟s modulus (high vs low) 75

Figure II.22 Tumor-centroid error based on mesh type: Poisson‟s ratio (high vs low) 75

Figure II.23 Tumor-centroid error based on mesh type: Friction (high vs low) 76

Figure II.24 Resection plane in (a) position A (b) position B using simulated model (c) position B using CT images 78

Figure II.25 Concept of the Smart Trocar 86

Figure II.26 Smart Trocar prototype with camera attached and colored marker 87

Figure II.27 Smart Trocar can obtain its position in 3d-coordinate space by capturing

trajectory of the crosses through the camera 88

Figure II.28 Setup of the concept 90

Figure II.29 Successive bijective transformations between the plane Ω and the image acquired by the laparoscope: first an homothety with scaling factor ξ to relate pixel size to physical size and then a rotation 91

Figure II.30 Experiment setup using a clamp to hold the laparoscope In right corner is 3D-printed model having the sharp of a half-sphere 94

Figure II.31 The laparoscope took images of the model in 4 positions 96

Figure II.32 Six interest points on the model surface needs to be calculated 97

Figure II.33 Liver position and landmarks at (a) full exhalation and (b) full inhalation 99

Figure II.34 Time dependence of distance between two markers (purple and blue in Figure II.33) in (a) x-direction and (b) in y-direction 101

Figure II.35 Location of Smart Trocars on the patient abdomen 107

Figure II.36 Ventilator machine‟s display 109

Figure II.37 Grid of markers on liver surface 110

Figure II.38 Two laparoscopic cameras were used at the same time 110

Figure II.39 The Smart Trocars were set up to record the trajectories and location of the instruments and the camera 111

Figure II.40 Fake tumors and stent were deployed into the liver 111

Figure II.41 The liver lift with a wooden stick 112

Figure II.42 CT scan of the cadaver‟s liver 113

Figure II.43 Simulation in COMSOL software 114

Figure II.44 Markers on the liver surface were tracked with software 116

Figure II.45 Trajectory of one marker: High-frequency vesus normal ventilator 117

Figure II.46 Marker displacement versus respiratory rate in case of HFV 117

List of table

Table 1 Segments to be removed in different types of liver resection 17 Table 2 State of the art for surface imaging system 84 Table 3 Result of experiment on rigid model 98 Table 4 Effect of perturbation on the results from the verification experiment reported in

the previous section 102

Part I

Laparoscopic surgery - also referred to as minimally invasive surgery (MIS) -

is a modern surgical technique in which small incisions are made in the abdominal wall, and plastic tubes called ports are placed through these incisions The laparoscope and other instruments can be pushed through those ports, allowing the surgeon to see and operate inside the patient„s abdomen and pelvis [2]

This technique can help to avoid large incisions used in traditional open surgery Therefore, it can reduce hemorrhaging and pain Smaller incision also leads to less postoperative scarring and faster recovery times

Figure I.1 Laparoscopic surgery (image taken from [3])

Laparoscopic surgery is now one of the most common surgical procedures performed in many parts of the world

1.2 Motivation

The motivation comes from personal and technical aspects

I am originally from Vietnam Liver disease has a high prevalence in Vietnam due to chronic infection with hepatitis viruses and high consumption of alcohol Chronic infection with hepatitis viruses, including hepatitis C virus (HCV), and hepatitis B virus (HBV) can lead to liver cancer and cirrhosis (liver damage) If it is not treated well, the risk of liver cancer considerably increases When long-term high-consumption of alcohol is combined with viral infections, the risk of cancer and cirrhosis can be doubled A nationwide study found that liver cancer is the most common cause of cancer death in Vietnam [29] In America, liver cancer is the 10th most common cancer, the 5th most common cause of cancer death among men, and the 8th most common cause of cancer death among women [4] Contributing to research which helps liver-cancer treatment feels like a natural way to give back to my original country and the country where I have been conducting this study

From a technical aspect, laparoscopic solid-organ tissue resections are challenging In open liver resection, most of the abdomen is exposed with the incision A trained surgeon can quickly identify, dissect and divide the veins and arteries with his/her eyes In laparoscopic liver resection, the surgeon can only look at the limited 2D-view from the laparoscope and operate through the trocars Therefore, extensive pre-operative imaging must be performed before the surgery so that the surgeon can plan out the procedure This works well until the actual resection begins and the coordinates in places are lost due to

the deformation of the liver Traditionally, accuracy is restored by rescanning the patient – a difficult task in the middle of complicated surgery Hence, there exists a need for dynamic, real-time tracking and recognition of internal structures and target tissues during a minimally invasive procedure

Working on a computational image-based guidance system for precision laparoscopy, which first target is laparoscopic liver resection (LLR), fulfills both of the aspects This system can help to increase favorable outcomes of minimally invasive surgery and thus support treatment of liver cancer in my country and America

This dissertation is divided into three major parts, each consisting of a set of chapters focused on a particular area of work An overview of the parts and chapters in this dissertation is provided here, highlighting the contributions made by this body of work to the field of the surgical navigation system

Part I: Motivation and Background

Chapter 1 provides introduction and motivation of the work in this dissertation Chapter 2 presents background information on the anatomy of the liver, liver resection, the mechanic's theory and basic modeling framework used to predict liver deformation The chapter concludes with a review of the literature in liver biomechanics modeling and describes the approach taken in this dissertation to address the problems in the literature

Part II: Developing of the image-based guidance system

Chapter 3 establishes the framework for the system and its components

Chapter 4 describes the work done to build a working validation of the model

It also gives detail about the experiment on a porcine liver-explant model and the simulation coming from it

Chapter 5 introduces the new method of reconstructing the liver‟s surface

using only a 2D-laparoscope and Smart Trocars

Chapter 6 is about the transition from the porcine liver to the human liver The

experiments on cadavers are presented here along with a new finding about

how to eliminate respiratory motion with the high-frequency ventilator

Part III: Dissertation review

Chapter 7 concludes the dissertation by giving summaries about the work

having done and indicating what needs to do in the future

Chapter 2

This chapter provides the necessary background information to develop the guidance system Section 2.1 provides a description of the anatomy of the liver It is followed by a classic definition of liver segmentation based on functional units of the liver in Section 2.2 Section 2.3 continues by explaining the type of liver resections and how our system can contribute to the surgical navigation field Section 2.4, 2.5, 2.6 present the basic math of the computational model: the finite elasticity theory, biomechanical properties of the liver and finite-element method Section 2.7 describes the image-processing technique to segment the liver from medical images to build the mesh for the model Section 2.8 provides literature review about this dissertation‟s topic Finally, Section 2.9 gives the conclusion about the contribution of this dissertation

2.1 Anatomy of the liver

An understanding of the liver organization is critical for the mechanical model development The liver is an organ of the digestive system, the biggest organ

in the human body and accounts for approximately 2% of adult body weight, 5% of children body weight The liver is fixed just under the diaphragm, in the top right portion of the abdominal cavity but protected by thorax skeleton and relates to chest more than abdomen [5]

Figure I.2 Liver anatomy (image taken from [6])

It is settled to the abdominal wall by those ligaments: the left and right triangular ligaments posteriorly, the left and right coronary ligaments and the falciform ligament, and located below the costal margin in the right subphrenic part

Figure I.3 Surface and bed of liver (image taken from [7])

It is placed directly antecedent to the inferior vena cava (IVC), attached to the IVC by the three main hepatic veins

Figure I.4 Vascular supply (image taken from [6])

1 Hepatic artery 2 Portal vein 3 Common hepatic duct

The liver is supplied by the portal vein and the hepatic artery that drains blood from abdominal organs to the liver The bile is provided by the common hepatic duct, which is the junction of the left hepatic duct and the right hepatic duct in the porta hepatis The portal vein, the hepatic artery, and the common bile duct are parts of the porta hepatis which is attached to the liver on its inferior surface

The liver is an exocrine gland which produces and discharges bile to pour into the duodenum, but it gets involved in many essential functions of the body

such as glucose, protein, lipoprotein metabolism, especially glycogen storage, regulation of blood sugar to eliminate poisons and supernumerary medicines out of the body through bile duct

The liver is reddish-brown, shiny, and slippery Although its density is relatively thick, it is easy to be crushed and broken when injured Also, when cut, it bleeds very much For dead people, the liver weighs 1.5 kg For live people, the liver weighs about 2.3 kg The liver has an average width of 28

cm, anterior - posterior surfaces of about 18 cm and the average height of 8

cm

2.2 Liver segmentation - Couinard system

The Couinaud classification of liver anatomy is the most commonly used segmentation of the liver It splits the liver into eight segments that function independently Each unit has its own vascular inflow, outflow, and biliary drainage

Figure I.5 Liver segmented by portal and hepatic veins (figure take from [8])

Middle hepatic vein separates the liver into left and right lobes from the IVC

to the gallbladder The right lobe is divided by the right hepatic vein into anterior and posterior segments The left lobe is divided by the left hepatic vein into medial and lateral parts Portal vein separates the liver into upper and lower segments The left and right portal veins grow up and down into the

center of each segment The segments are numbered from 1 to 8 as in Figure I.5

To remove a segment of the liver, parenchymal division follows the inter-lobar gap while keeping slightly to the right of the middle hepatic vein So the middle hepatic vein needs to be localized to make the right resection, and the model should show every segment of the liver based on veins‟ positions

2.3 Liver resection

The technique of liver resection has profited recently from an improved knowledge of both surgical and radiological liver anatomy Nowadays, it is possible to resect any single or multiple segments of the liver, using methods

to decrease blood loss during surgery and new procedures to divide liver parenchyma New research in intra-operative monitoring and general anesthesia have also reduced operative risks

Nevertheless, significant intra-operative or postoperative complications related

to the surgical procedure directly can still occur Therefore, following a by-step technique is essential to minimize these risks [6]

step-2.3.1 Types of liver resection:

The types of liver resection are based on Couinaud‟s classification Right liver resection and left liver resection are the most commonly executed procedures Any procedure that involves the resection of three or more segments is called

a major procedure In left liver resection, segments 2, 3, and 4 are cut out It can extend to involve segment 1, segment 5, segment 8 In right liver resection, segments 5, 6, 7, and 8 are extracted It can extend to involve segment 4 or segment 1, or both The last case is one of the largest liver resections, as only the right posterior lateral segment is left untouched

Figure I.6 Liver segmentation (image taken from [8])

1 Right liver 2 Left liver 3 Caudal surface 4 IVC (inferior vena cava)

In minor liver resection, only one liver segment or two adjacent segments are removed Each liver segment may be resected separately or in combination with an adjacent segment Bisegmentectomy 2 and 3 corresponds to the left lobectomy Resection of posterior segments is harder than resection of anterior segments (3, anterior 4, 5)

R: removed segments E: extendable removed segments

Table 1 Segments to be removed in different types of liver resection

In liver surgery, the key point is controlling the portal branch, hepatic vein, branches of the hepatic artery, and hepatic duct Therefore, liver anatomy and vascular supply need to be understood

Therefore, landmarks are required on the portal vein and its branches for the segmentation model The number and position of the markers might depend on which type of liver resection of the application

2.3.2 Open liver resection versus laparoscopic liver resection

In open liver resection, most of the liver is exposed with the incision A trained surgeon can quickly identify, dissect, and divide the veins and arteries with his eyes The need for navigation systems might only come from the difficulty in determining the left lobe and the right lobe as veins go under the surface of the liver Even so, the surgeon can still clamp the portal vein and the hepatic artery of one liver side It results almost instantaneously discoloration of the corresponding side of the liver and demarcation of both lobes of the liver The surgeon then can proceed to parenchymal division follows the differentiation Therefore, the navigation system is not useful in open surgery

Figure I.7 Clamping of right portal vein and right hepatic artery (image taken from

[8])

Laparoscopic liver resections (LLR) are replacing open liver resection with over 3000 cases performed worldwide for benign disease, malignancy, and hepatectomy [30] While minimally invasive surgery has been welcomed in other fields, there are several difficulties to make LLR widespread Mobilization and transection techniques in open resection do not work the same in LLR There is concern about not being able to control hemorrhaging and gas embolism In spite of that, the technological advances in laparoscopic devices, radiologic imaging, and patient demand have made the number of LLR cases increase recently

In laparoscopic liver resection, the field of view is limited The current navigation system for this kind of liver resection is video feedback which is a 2D-view Moreover, the surgeon cannot contact directly with the liver Some

of the methods to identify and separate the veins and arteries in open surgery cannot be applied here including clamping techniques Therefore, a 3D-navigation system is urgently needed for laparoscopic liver resection It will help the surgeon gain the whole view of the organ and its surroundings, the sense of depth, the inner structure of the liver, and essential features needed for laparoscopic liver resection

2.4 Finite elasticity theory

With the liver anatomy and resection described, this section introduces the mechanical modeling of liver tissue This section first outlines the kinematics that describes the equations relating strain tensors to displacement gradient [11, 12] We then present the stress equilibrium equation for a linear and hyper-elastic material These equations constitute the connection between stress and strain of the material We conclude with a discussion on the computational implementation of these mechanical models on commercial software

2.4.1 Kinematics

For the coordinates system, we consider the body, has been subjected to a deformation now in a deformed state, We refer the current state of the body as the current configuration and the initial, non-deformed state, as the reference configuration For these two configurations, two types of coordinate systems are used to analyze stresses and strains:

Material coordinates (X1, X2, X3) are used to link each material point from the reference configuration to the deformed configuration When the body deforms, the material coordinate axes deform in a similar manner Hence the coordinate values of a material point do not change during the deformation This coordinate system is called a Lagrangian description

Spatial coordinates (x1, x2, x3) are used to link the material point to a specific global position in space As the body deforms, this point can be associated with a different material point A spatial coordinate system is a fixed coordinate system in space This coordinate system is often called a Eulerian description

Figure I.8 Body in reference and current configuration

(image reproduced from [10])

The measure of strain requires a relationship between the material coordinates

of the material points in the reference configuration and the spatial coordinates

of material points in a deformed body Let the position of material points in the current configuration, X, is functions of their position in reference configuration, x The deformation can be represented by a mapping where x = (X)

Consider a line element dX in the reference configuration being deformed into

dx in the current configuration Then a deformation gradient tensor, F, relating

the line element in the non-deformed state to the line element in the current state is defined by the relationship dx = FdX, where

where J is the Jacobian of the deformation gradient tensor

The entire deformation, including the strain and rigid body motion, is represented by the deformation gradient tensor It can be decomposed according to the polar decomposition of the tensor as F = RU where R is an orthogonal rotation tensor, and U is a symmetric tensor, known as the right stretch tensor, representing the material strain independent of the rigid body rotation Because U is a positive and symmetric tensor, there exists a basis in which U is diagonal These are the principal direction of U and by definition, the positive diagonal components , , of U in the principal axes called the principal values of U, presenting the principal stretches In continuum mechanics, it is usually good to use rotation-independent measures of deformation since a pure rotation should not induce any stresses in a deformable body

We derive another tensor that is independent of the rigid body rotation, the right Cauchy-Green tensor, C, defined as:

(2.3)

In three-dimension, C is a 3 x 3 matrix and possesses 3 principal invariants under coordinate change:

( ) (2.4) ,( ) - (2.5)

(2.6)

They are usually used to express strain energy density functions When there

is no deformation or motion, the deformation gradient tensor will be equal to the identity tensor I Therefore, the right Cauchy-Green deformation tensor is

as well To get a strain tensor such that no motion or deformation yields the zero tensor, 0, the Green Strain tensor, E, was introduced, satisfying this property:

( ) (2.7)

3 Conservation of angular momentum indicates that the time rate of change

of momentum of the body must balance the applied moments

The Cauchy stress tensor refers to the force measured per unit area in the deformed configuration Another measurement of stress include P and S, respectively the 1st and 2nd Piola-Kirchhoff stress tensor defined by:

(2.11) (2.12)

At this equilibrium state, if the body undergoes an arbitrary, small, virtual displacement (not caused by prescribed external forces), the equation 2.9 can be rewritten as

∫ + ∫ = 0 (2.13)

2.4.3 Constitutive equation

In the previous section, the stress equilibrium equations and the kinematic equations have been expressed Therefore, a mathematical relationship between strains and the stresses must be established This relationship is called constitutive relation

The form of the constitutive relation is not the same for every material It is material-behavior specific Hence, it may be necessary to develop more than one constitutive relation for material if there is an interest in different conditions A class of constitutive relations can be theorized in which the stress is a function of the deformation gradient tensor ( ), where F are the components of the deformation gradient tensor, , represent components