The biomedical engineering handbook

(BQ) Part 1 The biomedical engineering handbook Medical devices and systems has contents: Advanced thermal image processing, functional infrared imaging in clinical applications, thermal imaging in surgery, infrared imaging applied to dentistry, infrared detectors and detector arrays,...and other contents.

Advanced Thermal Image Processing

Technical University of Lodz

28.1 Histogram-Based First Order Thermal Signatures 28-3 28.2 Second Order Statistical Parameters 28-5 28.3 Wavelet Transformation 28-8 28.4 Classification 28-9 28.5 Conclusion 28-12 References 28-12

Thermal imaging can be useful as an early diagnostic technique that can detect many diseases, such asfor example, breast cancer, malignant tumors, etc There are many different methods that describe imagefeatures A large group of methods is based on statistical parameters calculations Parameters like meanvalue, standard deviation, skewness, kurtosis, etc can be used to compare thermal images We considerboth the first and second order statistical parameters [1,2] The first order statistical parameters methodsuse image’s histogram (Figure 28.1) to compute signatures, while the second order statistical parametersare defined for so-called co-occurrence matrix of the image [2,11]

In medical applications, one of the principal features of the thermal image is its symmetry oftemperature patterns Thermal images are usually asymmetrical in pathological cases Any significantasymmetry can indicate a physiologic abnormality (Figure 28.2) This may be pathological (includingcancer, fibrocystic disease, an infection, or a vascular disease) or it might indicate an anatomical variant[4–6]

The next group of methods is based on image transformations, such as linear filtering, Fourier or waveletanalysis All these methods allow to regenerate an image which is processed or converted, and signaturesare defined in different domain, for example, frequency or scale domains Well-known Karhunen-Loevetransform is implemented in form of principle component analysis (PCA) PCA is a technique that

is usually used for reducing the dimensionality of multivariate data, preserving most of the variance[7–9]

Thermal image classification is a powerful tool for many medical diagnostic protocols, for example,during breast cancer screening [3,10,11] Figure 28.3 and Figure 28.4 show thermal images of a healthybreast and that with malignant tumor, respectively Among the variety of different image features, stat-istical thermal signatures (first and second order) have been already effectively used for classification ofimages represented by raw data [1,2,10,12] In another approach, the features obtained from wavelettransformation can also be used for successful classification

28-1

Temperature( ° C)

34.70

FIGURE 28.1 (See color insert following page 29-16.) ROI of thermal image and its histogram.

FIGURE 28.2 (See color insert.) Nonsymmetrical temperature distribution for pneumonia with corresponding X-ray image.

FIGURE 28.3 (See color insert.) Thermal image of the healthy breast.

Advanced Thermal Image Processing 28-3

FIGURE 28.4 (See color insert.) Thermal image of the breast with malignant tumor (left side).

It is possible to define many features for an image, and obviously, the selection and reduction areneeded Two approaches are applied, based on Fischer coefficient as well as by using minimization ofclassification error probability (POE) and average correlation coefficients (ACC), calculated for chosenfeatures [13] It can reduce the number of features to a few ones Features preprocessing which gen-erates new parameters after linear or nonlinear transformations can be the next step in the procedure

It allows to get less correlated and of the lower order data Two approaches are used, that is, PCAand linear discriminant analysis (LDA) [1,3,8] Finally, classification can be performed using differentartificial neural network (ANN), with or without additional hidden layers, and with different number

of neurons Alternatively, nearest neighbor classification (NNC) can also be employed for such imageprocessing

28.1 Histogram-Based First Order Thermal Signatures

An image is assumed to be a rectangular matrix of discretised data (pixels) pix [m, n], where m =

0, 1, , M, n = 0, 1, , N Each pixel takes a value from the range i ∈ 0, L − 1 The

histo-gram describes the frequency of existence of pixels of the same intensity in whole image or in theregion of interest (ROI) Formally, the histogram represents the distribution of the probability func-tion of the existence of the given intensity in an image and it is expressed using Kronecker deltafunction as:

Histogram gives global information on an image By converting histogram we can obtain some veryuseful image improvement, such as contrast enhancement [1,14] The first order statistical parameterscan be used to separate physiological and pathological breast thermal images The results for breast cancerscreening are presented below

Thirty-two healthy patients and ten patients with malignant tumors were analyzed using thermography.There were four images registered for every patient that represented each breast in direct and lateraldirection to the camera Histograms were created for these images and on the basis of statistical parameters,the following features were calculated: mean temperature, standard deviation, variance, skewness, andkurtosis Afterward, differences of parameter values for left and right breast were calculated The degree

of symmetry on the basis of these differences was then estimated (see Figure 28.5 and Figure 28.6)

0.08 0.07 0.06 0.05 0.04 0.03 0.02 0.01 0

Left side Right side

Temperature ( ° C) AS67-healthy breast

FIGURE 28.5 Histograms of healthy breast thermographs.

Advanced Thermal Image Processing 28-5

28 30 32 34 36 0

0.005 0.01 0.015 0.02 0.025 0.03 0.035

Temperature ( ° C)

Left side Right side jk33-malignant tumor in left breast

FIGURE 28.6 Histograms of breast thermographs with malignant tumor.

The mean temperature in the healthy group was estimated at the level 30.2± 1.8◦C in the directposition and 29.7± 1.9◦C in the lateral one The mean temperature was higher in 8 cases out of 10 inmalignant tumor group Moreover, 6 cases out of 32 in the healthy group with mean temperatures exceedednormal level Therefore, we have found that it is necessary to analyze symmetry between left and rightbreast Comparison of mean temperature was not sufficient to separate physiological and pathologicalimages Among analyzed parameters, skewness was the most promising for successful classification ofthermal images Absolute differences of skewness for left and right side was equal to 0.4± 0.3◦C infrontal position and 0.6± 0.4◦C in lateral one for the healthy group These differences were higher forimages in lateral position in all cases in the pathological group in comparison to the healthy patients’images

The image in Figure 28.4 confirm the evidence of asymmetry between left and right side for healthyand malignant tumor cases

Analyzing the first order statistical parameter let us conclude that it is quite hard to use them for theimage classification and detecting tumors In some cases, only mean temperature and the skewness let

us to separate and classify thermal images of breasts with and without malignant tumors Frontal andlateral positions were used during this investigation, but no significant difference of the obtained resultwas noticed

It is concluded that the first order parameters do not give the satisfactory results, and due to somephysiological changes of the breast, we could observe that these parameters do not allow separatingpatients with and without tumors That was the main reason, that the second order statistical parametersare used for the further investigations

28.2 Second Order Statistical Parameters

More advanced statistical information on thermal images can be derived from second order parameters.They are defined on so-called co-occurrence matrix Such a matrix represents the joint probability of two

pixels having i-th and j-th intensity at the different distances d, in the different directions Co-occurrence

matrix gives more information on intensity distribution over the whole image, and in this sense, it caneffectively be used to separate and classify thermal images

Let us assume that each pixel has eight neighbors lying in four directions: horizontal, vertical, diagonal

and antidiagonal Let us consider only the nearest neighbors, so the distance d= 1 (see Figure 28.7)

As an example let us take an image 4× 4 with 4 intensity levels given as (see Figure 28.8)

1 1

FIGURE 28.8 Example of 4 × 4 image with 4 discrete intensity levels.

Diagonal co-occurrence matrix, d =1

Pathology Physiology 20

40 60 80 100

Variance

0 120

FIGURE 28.9 Difference variance vs variance obtained from co-occurrence matrix for horizontal direction.

For horizontal direction the co-occurrence matrix takes a form:

The co-occurrence matrix is always square and diagonal with the dimension equal to the number

of intensity levels in the image After normalization, we get the matrix of the probabilities p(i, j).

Normalization is done by dividing the all elements by number of possible couple pixels for a given

direction of analysis For horizontal and vertical directions this number is equal to 2N (M − 1) and

2M (N − 1), while for diagonal directions it is 2(M − 1)(N − 1), respectively.

Second order parameters are presented by Equation 28.4 and Equation 28.6 As an example of paring thermal imagesµ x, µ y, σ x, σ y are mean values and standard deviations for the elements ofco-occurrence matrices that were calculated for horizontal and vertical directions, respectively, and theresults are presented in Figure 28.9 and Figure 28.10

com-Advanced Thermal Image Processing 28-7

Diagonal co-occurrence matrix, d =1

0 50 100 150 200 250

Variance

Pathology

FIGURE 28.10 Difference variance vs variance obtained from co-occurrence matrix for diagonal direction.

Second order statistical parameters can be used to discriminate the physiological and pathological cases,for example, breast cancers Most of them successfully discriminate healthy and malignant tumor cases.The protocol of the investigation assumes the symmetry analysis in the following way At first, the squareROI were chosen for analysis Then, co-occurrence matrixes were calculated for left and right breasts

to evaluate second order statistical parameters for different directions Only the neighboring pixels are

considered in these investigations (d = 1) Finally, the differences of the values of these parameters forleft and right sides were used for the image classification Figure 28.9 and Figure 28.10 illustrate that thedifferences of second order parameters for left and right sides are typically greater for pathological cases.Taking two parameters such as difference variance and variance allows successfully separating almost allhealthy and malignant tumor cases

transforma-As an example, the investigations of 10 patients with recognized breast tumors, as well as 30 healthypatients are presented All healthy and pathological cases were confirmed by other diagnostic methods,such as mammography, USG, biopsy, and so on We have used thermographic camera to take two imagesfor each breast: frontal and side ones Each patient was relaxing before the experiment for 15 min in aroom where temperature were stabilized at 20◦C.

Then the numerical procedure was used to calculate features of the images Figure 28.12 presents thepathological case where left breast has evidently higher temperature and the temperature distribution isvery asymmetric

One of the possible transformations uses wavelets realized by filtering as it has already been mentioned.The result of such processing is presented in Figure 28.13 As it is seen, the different filters are showingdifferent details of the image, that is, high pass filters allows to present gradient of the temperature, while

FIGURE 28.11 Wavelet transformation of an image.

FIGURE 28.12 (See color insert.) Example of thermal image with the tumor.

Advanced Thermal Image Processing 28-9

FIGURE 28.13 Result of wavelet transformation.

the low pass ones display the global temperature distribution and the energy of the signal understood asthe level of temperature

As a powerful tool, wavelet transformation gives the possibility of generating new features from differentsubimages, in different scales and subbands Filtering can be easily parameterized in the sense of varyingcutoff frequency, what provides the additional flexibility in the algorithm It denotes, that a lot of differentfeatures can be produced by this method Obviously, the selection of these features is necessary to obtainonly these ones, which are the most discriminative and weakly correlated [11]

28.4Classification

Artificial neural network are typically used for classification The selected image features are used as inputs

It means that the number of input nodes is equal to number of the features The number of neurons inthe first hidden layer can be equal or lower than the number of features in the classification, as shown inFigure 28.14 ANN can have user-defined next hidden layers, which allow additional nonlinear processing

of the input features As ANN is the nonlinear system, such technique allows to additional decorrelatingand data reduction, what in consequence improves the classification Such approach is known as nonlineardiscriminant analysis (NDA) [3,11]

It is well known that the training of ANN is the very important step in the entire protocol It is anmultivariable optimization problem typically based on back-propagation technique In general case it canlead to wrong solutions if the there is no single minimum of the error function That is why we needenough data during learning phase, and sometimes it is better or necessary to repeat training of ANN withdifferent initial values of the neuron weight coefficients

Classification can start from the raw data analysis As seen in Figure 28.15, two classes of imagescontaining (1) nonhealthy and (2) healthy cases are chosen In this example, selection of features reducesthe number of features to two, derived from the co-occurrence matrix (sum of squares and inverse

difference moment) [1,10,12] The third one is based on wavelet transformation — energy EHL forthe scale no.1 and high and low frequency subbands Raw data analysis (Figure 28.15) does not givesatisfactory feature separation and classification It is hard to separate clusters of features corresponding

to physiological and pathological images The distance in between them in the multivariate space is notlarge enough, which means that the probability of erroneous classification is still quite high

The next results were obtained by LDA (Figure 28.16), in which the most discriminative features (MDF)are generated [10,13] LDA creates typically new set with smaller number of features LDA is the linear

–2.62 –2.14

1.94

S(1,1)InvDfMom S(5,5)SumOfSqs

NavEnHL_S–1

FIGURE 28.15 Raw data analysis result.

transformation, and produces linearly separated features, which means that in general case it is also notpossible to separate them fully This case is illustrated in Figure 28.16

Nonlinear transformation of the features obtained by ANN with additional hidden layer is the mostpromising technique The output of this additional hidden layer creates the new smaller set of features,which are typically much better separable It can be simple verified by the larger value of Fisher coefficient

In our investigation, seven original features were selected using POE and ACC selection criteria The firstand second hidden layers contained only one and two neurons, respectively It denotes that the originalfeatures were reduced to two new ones The expectation that this approach allows better separation andclassification is now confirmed [2]

The results of NDA are presented in Figure 28.17 and Figure 28.18 Two new features are far away fromeach other on the feature space, both for frontal and side images It is even not very surprising that the

value of Fisher coefficient F = 2.7 for original features is now increased to F = 443.4 ANN together with

Advanced Thermal Image Processing 28-11

FIGURE 28.17 NDA classification results for side thermal images of the breast.

NDAf1 0.00

1.00

NDAf2

0.44 0.92

FIGURE 28.18 NDA classification results for frontal thermal images of the breast.

NDA have one more advantage Besides data reduction and decorrelating, it also allows implementingclassification realized by the last output layer of ANN

We faced one problem in the presented research The number of pathological cases was small, only ten

As it has been already mentioned it is difficult to get well diagnosed patients, especially in the young age[4–6] The research is still in progress, and we actually enlarging our image database day-by-day Thisresearch is a preliminary one, and shows the possibility of using advanced image processing tools Theeffectiveness of the screening procedure can be verified later, when we will have enough input data.Because of the above limits, the classification was carried out in the following simplified way: First wehave chosen equal number of physiological and pathological images, that is, 10+ 10 Every image from

Bronz: “2122_c028” — 2006/2/9 — 22:32 — page 12 — #12

TABLE 28.1 Errors of Classification

Frontal positions Side positions False negative False positive False negative False positive

Finally, calculation using NDA was performed with an additional hidden layer that consisted of twoneurons for generating two new more discriminative features The results in Table 28.1 confirm theeffectiveness of using both NN and ANN classifiers Although it was not evidently proved that NDAresults are better in the classification, we definitely conclude that ANN is a powerful tool for thermalimage processing during breast cancer screening Reducing false/positive errors of classification is themost important task for the future research

an image Actually, the study is being extended by choosing second order parameters for multivariate dataclassification The presented approach includes the PCA to reduce the dimensionality of the problem and

by selecting the eigen vectors it is possible to generate data, which represents the tumors more evidently.Breast cancer screening is a challenge today for medical engineering Breast temperature depends notonly due to some pathological changes, but it also varies in normal physiological situations, it is even aconsequence of emotional state of a patient It was a main reason that we are looking for more advancedmethods of thermal image processing, that could give satisfactory results

One of the possible alternative for such processing is ANN classification based on multidimensionalfeature domain, with use of modern transformations, such as wavelets The preliminary investigations arequite successful, and can be improved by increasing the number of samples taken for processing.Future research will concentrate around selection of features and adjusting wavelet transformationparameters to get the best classification We assume that the more satisfactory results can be obtained byusing features based on asymmetry between left and right sides of a patient It could help for one-sidecancerous lesion classification, what is the most typical pathological case and frequently happens today

References

[1] P Cichy, “Texture analysis of digital images, doctoral thesis, Technical University of Lodz, Institute of

Electronics, Lodz, 2000, in Polish.

Advanced Thermal Image Processing 28-13

[2] A Materka, M Strzelecki, R Lerski, and L Schad, “Evaluation of texture features of test objects for

magnetic resonance imaging,” Infotech Oulu Workshop on Texture Analysis in Machine Vision, June

1999, Oulu, Finland

[3] P Debiec, M Strzelecki, and A Materka, “Evaluation of texture generation methods based on

CNN and GMRF image texture models,” Proceedings of the International Conference on Signals and

Electronic Systems, 17–20 October 2000, Ustron, pp 187–192.

[4] E.Y.K Ng, L.N Ung, F.C Ng, and L.S.J Sim “Statistical analysis of healthy and malignant breast

thermography,” Journal of Medical Engineering and Technology, 25, 253–263, 2001.

[5] B.F.J Manly, Multivariate Statistical Method: A Primer London, Chapman & Hall, 1994.

[6] Michael Bennett,“Breast cancer screening using high-resolution digital thermography,” Total Health,

22, 44, 1985

[7] I.T Jolliffe, Principal Component Analysis New York, Springer-Verlag, 1986.

[8] B.F.J Manly, Multivariate Statistical Method: A Primer New York, London, Chapman & Hall, 1994 [9] D.R Causton, A Biologist’s Advanced Mathematics, London, Allen and Unwin, 1987.

[10] Schürman J Pattern Classification, New York, John Wiley & Sons, 1996.

[11] M Kociolek, A Materka, M Strzelecki, and P Szczypinski, “Discrete wavelet transform-derived

features for digital image texture analysis,” Proceedings of the International Conference on Signals and

Electronic Systems ICSES’2001, Lodz, 18–21 September 2001, pp 111–116.

[12] T Jakubowska, B Wiecek, M Wysocki, and C Drews-Peszynski, “Thermal signatures for breast

cancer screening comparative study,” Proceedings of the IEEE EMBS Conference, Cancun, Mexico,

17–21 September, 2003

[13] P Debiec, M Strzelecki, and A Materka, “Evaluation of texture generation methods based on

CNN and GMRF image texture models,” International Conference on Signals and Electronic Systems

ICSES’2000, Ustron, October 2000, pp 187–192.

[14] B Wiecek and S Zwolenik, “Thermal wave method — limits and potentialities of active

thermo-graphy in biology and medicine,” 2nd Joint EMBS-BMES Conference, 24th Annual International

Conference of the IEEE Engineering in Medicine and Biology Society, BMES-EMS 2002, Houston,

23–26 October, 2002

Biometrics: Face Recognition in Thermal Infrared

Facial Tissue Delineation Using a Bayesian Approach

29.3 Facial Feature Extraction in Thermal Infrared 29-6

Morphological Reconstruction of Superficial Blood Vessels

29.4 Conclusion 29-13 Acknowledgments 29-14 References 29-15

29.1 Introduction

Biometrics has received a lot of attention in the last few years both from the academic and businesscommunities It has emerged as a preferred alternative to traditional forms of identification, like cardIDs, which are not emedded into one’s physical characteristics Research into several biometric modalitiesincluding face, fingerprint, iris, and retina recognition has produced varying degrees of success [1] Facerecognition stands as the most appealing modality, since it is the natural mode of identification amonghumans and totally unobtrusive At the same time, however, it is one of the most challenging modalities[2] Faces are 3D objects with rich details that vary with orientation, illumination, age, and artifacts (e.g.,glasses)

Research into face recognition has been biased towards the visible spectrum for a variety of reasons.Among those is the availability and low cost of visible band cameras and the undeniable fact that facerecognition is one of the primary activities of the human visual system Machine recognition of humanfaces, however, has proved more problematic than the seemingly effortless face recognition performed

by humans The major culprit is light variability, which is prevalent in the visible spectrum due to thereflective nature of incident light in this band Secondary problems are associated to the difficulty ofdetecting facial disguises [3]

As a cure to the aforementioned problems, researchers have started investigating the use of thermalinfrared for face recognition purposes [4,5] Efforts were directed into solving three complementaryproblems: face detection, feature extraction, and classification (see Figure 29.1) [6,7] Face detection is a

29-1

Bronz: “2122_c029” — 2006/2/9 — 21:55 — page 2 — #2

prerequisite step, since a face cannot be recognized unless first it is detected in the scene Feature extractionfollows face detection and reduces the face to a succinct mathematical description — the feature vector.Classification operates upon the feature vector and matches the incoming face to one of the records kept

in the database

Many of the research efforts in thermal face recognition use the thermal infrared band as a way to see

in the dark or reduce the deleterious effect of light variability [8] Methodologically, they do not differvery much from face recognition algorithms in the visible band [9] In this chapter, we will present a novelapproach to the problem of thermal facial recognition that realizes the full potential of the thermal infraredband It consists of a statistical face detection and physiological feature extraction algorithm tailored tothermal phenomenology We will not elaborate on classification, since we do not have to add anythingnew in this part of the face recognition process Our goal is to promote a different way of thinking in areaswhere thermal infrared should be approached in a distinct manner with respect to other modalities

29.2 Face Detection in Thermal Infrared

We approach the face detection problem in thermal infrared from a statistical point of view Due to

its physiology, a typical human face consists of hot and cold parts Hot parts correspond to tissue areas

that are rich in vasculature (e.g., periorbital and forehead) Cold parts correspond to tissue areas thatfeature either sparse vasculature or multi-sided exposure to the environment (e.g., cheeks and nose).This casts the human face as a bimodal distribution entity The background can also be described by a

bimodal distribution It typically consists of walls (cold objects) and the upper part of the subject’s body dressed in clothes (hot objects) The bimodal distributions of the face and background vary over time, due

to thermophysiological processes and environmental noise respectively Therefore, the problem rendersitself naturally to the Bayesian framework, since we have a priori knowledge of the bimodal nature of thescene, which can be updated over time by the incoming evidence (new thermal frames from the camera)

29.2.1 Facial Tissue Delineation Using a Bayesian Approach

We consider an indoor area that is being monitored with a thermal infrared camera The objective is todetect and delineate a human face should this become available in the scene The segmented face datafeed the feature extraction and classification modules, which complete the face recognition process (seeFigure 29.1) Initially, we consider the face detection problem at the pixel level and label every pixel as

either facial skin s or background b pixel.

In more detail, we use a mixture of two Normal distributions to model facial skin temperature Thedominant mode is in the upper band of the values where usually about 60 to 70% of the probability mass

Biometrics 29-3

Background (including covered skin) Exposed skin

1800 1600 1400 1200 1000 800 600 400 200

resides The secondary mode is in the lower band of the values For subjects in climate controlled rooms

a typical temperature range for the dominant skin mode is∼(32–35)◦C, while for the secondary mode is

∼(28–30)◦C The latter may overlap with the background distribution since areas of the face like the noseand ears have temperatures similar to the environment (see Figure 29.2)

Similarly, we use a mixture of two Normal distributions to model background temperature Thedominant mode is in the lower band of the values where about 80% of the background probability massresides (typically, in the range∼[27 to 29]◦C) The secondary background mode has its mean somewherebetween the two modes of the skin distribution and variance large enough to cover almost the entire band.Therefore, the secondary background distribution includes some relatively high temperature values Theseare due to the fact that light clothes offer spots of high temperature (e.g., places where the clothes touchthe skin), which mimic the skin distribution but are not skin (see Figure 29.2)

For each pixel we have some prior distribution (information) available of whether this particular pixelrepresents skin(π(s)) or background (π(b) = 1 − π(s)) Then, the incoming pixel value represents the

data (likelihood) that will be used to update the prior to posterior distribution via the Bayes theorem

Based on the posterior distribution we will draw our inference of whether the specific pixel represents s

or b At the end, this posterior distribution will be used as the prior for the next incoming data point.

We callθ the parameter of interest, which takes two possible values (s and b) with some probability.

The prior distribution at time t consists of the probabilities of the two complementary events, s and b.

The incoming pixel value x t has a conditional distribution f (x t |θ), which depends on whether the

par-ticular pixel is skin (i.e.,θ = s) or background (i.e., θ = b) Based on our bimodal view of the skin and

background distributions we will have for the likelihood:

Bronz: “2122_c029” — 2006/2/9 — 21:55 — page 4 — #4

There are ten parameters that are involved in the conditional distribution of Equation 29.2, namely,

2 weights, 4 means, and 4 variances We will discuss later on, how to initialize and update these parameters.The prior distribution,π (t) (θ), will be combined with the likelihood, f (x t | θ), to provide (via the Bayes theorem) the posterior distribution p (t) (θ | x t ) Thus, we will have:

In the Bayesian philosophy the posterior distribution of the parameter of interest represents how the prior

information (distribution) is updated in light of new evidence (data) At every time point t our inference (on whether the particular pixel represents s or b ) will be based on the posterior distribution, p (t) (θ | x t ).

This posterior distribution will also be used to provide the prior distribution for the next stage Moreprecisely:

As part of the initialization, we need to specify two things: the prior distribution, π (1) (θ) and the

likelihood f (x1 | θ) Absence of information on whether a pixel is skin or background, leads us to adopt a noninformative prior distribution where a priori each pixel is equally likely to be s or b Thus,

we have:

π (1) (s) = 1

Regarding the likelihood, we need to calculate the initial values of the ten parameters involved in the

likelihood Equation 29.2 For that, we select N facial frames (off-line) from a variety of subjects This is

the so-called training set It is important for this set to be representative, that is, to include people of both

sexes, different ages, and with different physical characteristics We manually segment on all the N frames,

skin and, background areas (see Figure 29.3) The segmentation needs to be representative For example,

FIGURE 29.3 (See color insert following page 29-16.) Manual segmentation of skin (black rectangles) and

background (white rectangles) areas for initialization purposes.

Biometrics 29-5

in the case of skin areas, we need to include eyes, ears, nose and the other facial areas, not parts thereof

Out of the N frames the segmentation will yield Nsskin and Nbbackground pixels

For the skin distribution we have available at the initial state the pixels x1, x2, , x Ns, which are assumed

to be sampled from a mixture of two Normal distributions:

whereω s2 = 1 − ω s1 We estimate the mixture parametersω s i,µ s i, andσ2

s i using the Nsskin pixels of thetraining set via the EM algorithm Initially, we provide the EM algorithm with some crude estimates ofthe parameters of interest:ω (0) s i ,µ (0) s i , and(σ s (0) i )2 Then, we apply the following loop: For k = 0, 1, , we

for i = 1, 2 and j = 1, , Ns Then, we set k = k + 1 and repeat the loop The condition for terminating

the loop is:

|ω (k+1) s i − ω (k) s i | < ε i = 1, 2 (29.12)whereε is a small positive number (10−3, 10−4, .) We apply a similar EM process for determining the

initial parameters of the background distributions

29.2.1.2 Inference

Once a data point x t becomes available we are faced with the decision of whether the particular pixelrepresents skin or background In a decision theory framework this can be cast as testing the statisticalhypotheses for the parameter of interestθ:



H0:θ = s (i.e., the pixel represents skin)

H1:θ = b (i.e., the pixel represents background)

Bronz: “2122_c029” — 2006/2/9 — 21:55 — page 6 — #6

FIGURE 29.4 (See color insert.) Visualization of Bayesian segmentation on a subject: (a) original image; (b) mented image The nose has been erroneously segmented as background and a couple of hair patches have been erroneously marked as facial skin This is due to occasional overlapping between portions of the skin and background distributions The isolated nature of these mislabeled patches makes them easily correctable through post-processing.

seg-Figure 29.4 visualizes the results of Bayesian segmentation on a typical facial thermal image from theUniversity of Houston database

29.3 Facial Feature Extraction in Thermal Infrared

Once the face is delineated from the rest of the scene, one can extract the features necessary for classification

In contrast to the visible band, thermal infrared provides the capability to extract physiological features Inparticular, blood that flows in the major superficial vessels creates a substantial convective heat effect that

is manifested in thermal imagery Segmentation of the vascular network can provide the basis of a uniquefeature vector The topology and extent of the recovered vascular network depend on the genetic traitsand physical characteristics of the individual (e.g., facial skin fat deposits) Therefore, facial vasculature is

an identifying entity that endures through aging and superficial changes of appearance

The problem for vessel segmentation has been solved in other imaging modalities using a variety

of methods To the best of our knowledge, it is the first time that vessel segmentation is reported in thethermal imaging modality In our effort, we took into account methodologies used for vessel segmentation

in modalities other than thermal imaging (e.g., ultrasound and MRI) We can broadly categorize thesemethodologies as follows:

• Center line detection approaches

• Ridge-based approaches

• Region growing approaches

• Differential geometry-based approaches

• Matched filter approaches

• Mathematical morphology approaches

In the center line extraction scheme, the vasculature is developed by traversing the vessels’ center lines.The method employs thresholding and thinning of the image, followed by connected component analysis.Center line extraction techniques are used by References 10 and 11 They use graph search theory tofind out vessel center lines and then curvature features to reconstruct the vasculature In Reference 12,the authors detect center lines from images acquired from different angles As a result, they are able torepresent the 3D shape of the vessels

Ridge-based approaches convert a 2D gray scale image into a 3D surface by mapping the intensity

values along the z-axis Once the surface is generated, the local ridge points are those where the directional

gradient is the steepest The ridges are invariant to affine transforms and this property is exploited in

Tra-in Reference 19.

If a 2D image is mapped into a 3D surface, then the crest lines are the most salient features of thatsurface This gives rise to the differential geometry based methods for vessel segmentation The crest linesare obtained by linking the crest points in the image, which are the local maxima of the surface curvature

A prominent differential geometry-based method is the directional anisotropic diffusion (DAD), which

is discussed in Reference 20 It uses Gaussian convolution to remove the image noise This method is ageneralized form of the method reported in Reference 21 and uses the gradient as well as the minimumand maximum curvature information to differentiate the diffusion equation The method removes thenoise without blurring and is thus, very useful in edge enhancement procedures

The matched filtering approach uses a series of Gaussian kernels of different sizes and orientations InReference 22 the orientation of the Gaussian filters is chosen using the Hessian matrix A similar approach

is used in Reference 23 to enhance and detect vessels in real time These methods are similar to the Gaborfilters, which are extensively used in texture analysis

29.3.1 Morphological Reconstruction of Superficial Blood Vessels

In our method, first, we smooth the image to remove the unwanted noise and then, we apply the logical operators In thermal imagery of human tissue the major blood vessels do not have strong edges.Due to the thermal diffusion process the edges that we get have a sigmoid temperature profile A sigmoid

morpho-function can be written as y = 1/(1 + e −x ) (see Figure 29.5).

The method of anisotropic diffusion has proved very effective in handling sigmoid edges [24,25].Nonlinear anisotropic diffusion filters are iterative filters introduced by Perona et al [21] Greig et al.[26] used such filters to enhance MR images Sapiro et al [27] used a similar technique to perform edgepreserving smoothing of MR images Others have shown that diffusion filters can be used to enhance anddetect object edges within images [21,28]

1 0.9 0.8 0.7 0.6 0.5

0.4 0.3 0.2 0.1 0

In our case I (¯x, t) is the thermal image, ¯x refers to the spatial dimensions, and t to time The function

c (¯x, t) is a monotonically decreasing function of the image gradient magnitude and is called the diffusion function

Typical responses of the thermal image of the wrist to the Perona–Malik filters with diffusion functions c1

and c2respectively are shown in Figure 29.6 One can observe that for the same image gradient and value

of the k parameter a steeper slope is obtained for c1as compared to c2

The discrete version of the anisotropic diffusion filter of Equation 29.13 is as follows:

I t+1(x, y) = I t +1

4×[c N ,t (x, y)∇I N ,t (x, y) + c S,t (x, y)∇I S,t (x, y)

+ c E,t (x, y)∇I E,t (x, y) + c W ,t ∇I W ,t (x, y)] (29.17)The four diffusion coefficients and gradients in Equation 29.17 correspond to four directions (i.e., North,South, East, and West) with respect to location(x, y) Each diffusion coefficient and the corresponding

gradient are calculated in a similar manner For example, the coefficient along the north direction iscalculated as:

c N ,t (x, y) = exp −∇I

2

N ,t (x, y)

where∇I N ,t = I t (x, y + 1) − I t (x, y).

Figure 29.7a,b show the original thermal image of skin tissue (wrist in this case) and its perature surface plot respectively One can observe in the surface plot the noisy ridges formeddue to the hair Figure 29.7c shows the filtered skin image The hair has been removed fromthe surface, resulting into smoother ridges and heightened peaks in the temperature surface plot(Figure 29.7d)

tem-As these figures testify anisotropic diffusion is highly beneficial in improving the contrast in theimage and removing the noise This is in preparation of vessel segmentation through morphologicalmethods

29.3.1.1 Image Morphology

Image morphology is a way of analyzing imagery based on shapes It is rooted on set theory, the Minkowskioperators, and DeMorgan’s Laws The Minkowski operators are usually applied to binary images where it

Biometrics 29-9

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

0

Image gradient 1

Image morphology is a simple but effective tool for shape analysis and segmentation In retinotherapy

it has shown great results in localization of blood vessels in the retina Leandro et al [29] have usedmorphological operators for vessel delineation in the retina where the background intensity was veryclose to that of the blood vessels In our case, we have the blood vessels, which have a relatively lowcontrast compared to that of the surrounding tissue As per our hypothesis the blood vessel is a tubulelike structure running either along the length of the forearm or the face Thus, our problem is to segmenttubule structures from the image We employ for this purpose a top-hat segmentation method, which is acombination of erosion and dilation operations

Bronz: “2122_c029” — 2006/2/9 — 21:55 — page 10 — #10

32 30 28 26 24

30

25

100 200

200 150 100 50

100 200

FIGURE 29.7 (See color insert.) (a) Original thermal image of a wrist (b) Temperature surface plot of the original image (c) Diffused thermal image of the wrist (d) Temperature surface plot of the diffused image.

A 3 × 3 structuring element A 5 × 5 structuring element

FIGURE 29.8 Structuring elements.

Erosion and dilation are the two most basic operations in mathematical morphology Both of theseoperations take two pieces of data as input: an image to be eroded or dilated and a structuring element.The structuring element is similar to what we know as a kernel in the convolution operation There are avariety of structuring elements available but a simple 3× 3 square matrix is used more often Figure 29.8shows some commonly used structuring elements

The combination of erosion and dilation results into more advanced morphological operationssuch as:

• Opening

• Closing

• Skeletonization

Biometrics 29-11

FIGURE 29.9 Binary erosion by a 3 × 3 flat structuring element.

• White top hat segmentation

• Black top hat segmentation

In our application, we are interested only in image opening and top hat segmentation

1 Erosion: The name suggests that this operation erodes the boundary pixels of an image and thus, the

resultant image has a shrunk boundary Mathematically, it is defined as follows:

Therefore, erosion removes elements smaller than the structuring element Figure 29.9 shows a simpleerosion operation performed on a binary image Gray scale erosion with a flat structuring element generallydarkens the image Bright regions surrounded by dark regions shrink in size and dark regions surrounded

by bright regions grow in size Small bright spots in images disappear, as they are eroded away down to thesurrounding intensity value In contrast, small dark spots grow The effect is most pronounced at places

in the image where the intensity changes rapidly Regions of fairly uniform intensity are left more or lessunchanged except at their edges

2 Dilation: The name suggests that this operation gradually expands the boundary pixels of an image

and thus, the resultant image will have an enlarged boundary Dilation results in fusing small holes inthe boundary area, by enlarging the boundary pixels This is the equivalent of a smoothing function.Mathematically, it is defined as follows:

Therefore, dilation fills in gaps smaller than the structuring element Figure 29.10 shows a simple dilationoperation performed on a binary image Gray scale dilation with a flat structuring element generallybrightens the image Bright regions surrounded by dark regions grow in size and dark regions surrounded

by bright regions shrink in size Small dark spots in images disappear as they are “filled in” to the rounding intensity values In contrast, small bright spots will grow in size The effect is most pronounced

sur-at places in the image where the intensity changes rapidly Regions of fairly uniform intensity will belargely unchanged except at their edges

3 Opening: The basic effect of an opening operation is reminiscent of erosion, since it tends to remove

some of the foreground (bright) pixels at the edges However, it is less destructive than erosion Thesize and shape of the structuring element plays an important role in performing opening The operation

Bronz: “2122_c029” — 2006/2/9 — 21:55 — page 12 — #12

FIGURE 29.10 Binary dilation by a 3 × 3 flat structuring element.

preserves foreground regions that have a shape similar to its structuring element while erodes all otherregions of foreground pixels In mathematical terms opening can be written as:

A ◦ B = (A  B) ⊕ B or A ◦ B = ∪[(B) z | (B) z ⊆ A] (29.21)

While erosion can be used to eliminate small clumps of undesirable foreground pixels (e.g., “salt noise”)quite effectively, it has the disadvantage that it affects all foreground regions indiscriminately Openinggets around this by performing both erosion and dilation on the image The effect of opening can bevisualized quite easily Imagine taking the structuring element and sliding it inside each foregroundregion without changing its orientation All foreground pixels that can be covered by the structuringelement with the structuring element being entirely within the foreground region will be preserved.However, all foreground pixels which cannot be reached by the structuring element without parts of itmoving out of the foreground region will be eroded away After the opening has been carried out, the newboundaries of foreground regions will all be such that the structuring element fits inside them Therefore,further openings with the same element have no effect, a property known as idempotence The effect of

an opening on a binary image using a 3× 3 flat structuring element is illustrated in Figure 29.11

4 White Top Hat Segmentation: Many times gray scale images feature poor contrast For example, in our

case thermal imagery of human tissue has poor contrast around the vessels due to the thermal diffusionprocess As a result, image thresholding yields very poor results Top-hat segmentation is a morphologicaloperation that corrects this problem Top hat segmentation has two forms:

• White top hat segmentation

• Black top hat segmentation

The white top-hat segmentation process enhances the bright objects in the image, while the black top-hatsegmentation enhances the dark objects In our case, we are interested in enhancing the bright (hot) ridgelike structures corresponding to the blood vessels Therefore, we are interested only in the white top-hatsegmentation process Two methods have been introduced for performing the white top hat segmentation.The first one proposed in Reference 30 is based on image opening using a flat structuring element, whilethe second one proposed in Reference 31 uses H-dome transformation We have adopted the first methodwhere the image is first opened and then this opened image is subtracted from the original image Thisgives only the peaks in the image and thus enhances the maxima The step by step evolution of the originalimage toward the top-hat segmented image is shown in Figure 29.12 The simple functioning of top-hattransformation can be understood from the line profile plots in Figure 29.13

We have outlined a novel approach to the problem of face recognition in thermal infrared The cornerstones

of the approach are a Bayesian face detection method followed by a physiological feature extractor Theface detector capitalizes upon the bimodal temperature distribution of human skin and typical indoorbackgrounds The physiological feature extractor delineates the facial vascular network based on a whitetop hat segmentation preceded by anisotropic diffusion (see Figure 29.14) These novel tools can be used

Pixel location along the n-line

FIGURE 29.13 Image line profiles for (a) original, (b) opened, and (c) top-hat segmented image The profiles were taken along the line shown in Figure 29.8.

FIGURE 29.14 (See color insert.) Segmented facial images annotated with the facial vascular network (yellow lines)

as per the approach detailed in this chapter.

in combination with traditional classification methods to exploit the full potential of thermal infrared forface recognition — one of the fastest growing biometrics

Acknowledgments

Research related to the content of this chapter was supported by NSF grant #0313880 and ONR/DARPAgrant #N00014-03-1-0622 The views expressed in this chapter do not necessarily represent the views ofthe funding Agencies

Biometrics 29-15

References

[1] Jain, A., Bolle, R., Pankanti, S., and Jain, A.K., Biometrics: Personal Identification in Networked

Society, 1st ed., Kluwer Academic Publishers, 1999.

[2] Zhao, W., Chellapa, R., Phillips, P.J., and Rosenfeld, A., Face recognition: a literature survey, ACM

Computing Surveys (CSUR), 35, 399, 2003.

[3] Pavlidis, I and Symosek, P., The imaging issue in an automatic face/disguise detection system, in

Proceedings of the IEEE Workshop on Computer Vision Beyond the Visible Spectrum: Methods and Applications, Hilton Head Island, South Carolina, 2000, p 15.

[4] Wilder, J., Phillips, P., Jiang, C., and Wiener, S., Comparison of visible and infrared imagery for face

recognition, in Proceedings of the Second International Conference on Automatic Face and Gesture

Recognition, Killington, Vermont, 1996, p 182.

[5] Socolinsky, D and Selinger, A., A comparative analysis of face recognition performance with

vis-ible and thermal infrared imagery, in Proceedings of the 16th International Conference on Pattern

Recognition, Vol 4, Quebec, Canada, 2002, p 217.

[6] Srivastava, A., Liu, X., Thomasson, B., and Hesher, C., Spectral probability models for IR images

with applications to IR face recognition, in Proceedings of the IEEE Workshop on Computer Vision

Beyond the Visible Spectrum: Methods and Applications, Kauai, Hawaii, 2001.

[7] Buddharaju, P., Pavlidis, I., and Kakadiaris, I., Face recognition in the thermal infrared spectrum,

in Proceedings of the Joint IEEE Workshop on Object Tracking and Classification Beyond the Visible

Spectrum, Washington D.C., 2004.

[8] Socolinsky, D., Wolff, L., Neuheiser, J., and Evelenad, C., Illumination invariant face recognition

using thermal infrared imagery, in Proceedings of the IEEE Computer Society Conference on Computer

Vision and Pattern Recognition, Vol 1, Kauai, Hawaii, 2001, p 527.

[9] Cutler, R., Face recognition using infrared images and eigenfaces, http://www.cs.umd.edu/rgc/face/face.htm, 1996

[10] Niki, N., Kawata, Y., Satoh, H., and Kumazaki, T., 3D imaging of blood vessels using x-ray rotational

angiographic system, in Proceedings of the IEEE Nuclear Science Symposium and Medical Imaging

Conference, Vol 3, San Francisco, California, 1993, p 1873.

[11] Kawata, Y., Niki, N., and Kumazaki, T., Characteristics measurement for blood vessels diseases

detection based on cone-beam CT images, in Proceedings of the IEEE Nuclear Science Symposium

and Medical Imaging Conference, Vol 3, 1995, p 1660.

[12] Parker, D.L., Wu, J., and van Bree, R.E., Three-dimensional vascular reconstruction from projections:

a theoretical review, in Proceedings of the IEEE Engineering in Medicine and Biology Society Conference,

Vol 1, New Orleans, 1998, p 399

[13] Aylward, S., Pizer, S., Bullit, E., and Eberl, D., Intensity ridge and widths for tubular object

seg-mentation and registration, in Proceedings of the Workshop on Mathematical Methods in Biomedical

Image Analysis, 1996, p 131.

[14] Aylward, S and Bullitt, E., Analysis of parameter space of a metric for registering 3D vascular

images, MICCAI, 2001.

[15] Bullitt, E and Aylward, S.R., Analysis of time-varying images using 3D vascular models, Applied

Imagery Pattern Recognition Works, 2001, p 9.

[16] Jones, T and Metaxas, D., Image segmentation based on the integration of pixel affinity and

deformable models, in Proceedings of the IEEE Computer Society Conference on Computer Vision and

Pattern Recognition, Santa Barbara, California, 1998, p 330.

[17] Pednekar, A.S and Kakadiaris, I.A., Applications of virtual reality in surgery, in Proceedings

of the Indian Conference on Computer Vision, Graphics, and Image Processing, Bangalore, India,

2000, p 215

[18] O’Brien, J.F and Exquerra, N.F., Automated segmentation of coronary vessels in angiographic

image sequences utilizing temporal, spatial, and structural constraints, in Proceedings of the SPIE

Visualisation Conference in Biomedical Computing, 1994.

Bronz: “2122_c029” — 2006/2/9 — 21:55 — page 16 — #16

[19] Schmitt, H., Grass, M., Rasche, V., Schramm, O., Haehnel, S., and Sartor, K., An x-ray based method

for determination of the contrast agent propagation ub in 3D vessel structures, IEEE Transactions

on Medical Imaging, 21, 251, 2002.

[20] Krissian, K., Malandain, G., and Ayache, N., Directional Anisotropic Diffusion Applied toSegmentation of Vessels in 3D Images, Technical Report 3064, INRIA, 1996

[21] Perona, P and Malik, J., Scale-space and edge detection using anisotropic diffusion, IEEE

Transactions on Pattern Analysis and Machine Intelligence, 12, 629, 2003.

[22] Sato, Y., Nakjima, S., Shiraga, N., Atsumi, H., Yoshida, S., Kikker, T., Greig, G., and Kikinis, R., 3Dmultiscale line filter for segmentation and visualization of curvilinear structures in medical images,

IEEE Medical Image Analysis, 2, 143, 1998.

[23] Poli, R and Valli, G., An algorithm for real-time vessel enhancement and detection, Computer

Methods and Programming in Biomedicine, 52, 1, 1997.

[24] Acton, S.T., Bovik, A.C., and Crawford, M.M., Anisotropic diffusion pyramids for image

segment-ation, in Proceedings of the IEEE International Conference on Image Processing, Vol 3, Austin, Texas,

1994, p 478

[25] Acton, S.T., Edge enhancement of infrared imagery by way of the anisotropic diffusion pyramid, in

Proceedings of the IEEE International Conference on Image Processing, Vol 1, Laussane, Switzerland,

1996, p 865

[26] Gerig, G., Kubler, O., Kikinis, R., and Jolesz, F.A., Nonlinear anisotropic filtering of MRI data, IEEE

Transactions on Medical Imaging, 11, 221, 1992.

[27] Sapiro, G and Tannenbaum, A., Edge Preserving Geometric Smoothing of MRI Data, TechnicalReport, University of Minnesota, Department of Electrical Engineering, April 1994

[28] Nordstrom, N., Biased anisotropic diffusion — a unified generalization and diffusion approach to

edge detection, Image and Vision Computing, 8, 318, 1990.

[29] Leandro, J., Cesar, R.M Jr., and Jelinek, H., Blood vessel segmentation in retina: preliminary

assessment of the mathematical morphology and wavelet transform techniques, 14th Brazilian

Symposium on Computer Graphics and Image Processing, 2001, p 84.

[30] Dougherty, E.R., Euclidean gray-scale granulometries: representation and umbra inducement,

Journal of Mathematical Imaging and Vision, 1, 7, 1992.

[31] Vincent, L., Morphological grayscale reconstruction in image analysis: application and efficient

algorithms, IEEE Transactions on Image Processing, 2, 176, 1993.

Infrared Imaging for

Tissue Characterization and

National Institutes of Health

30.1 Near-Infrared Quantitative Imaging of Deep Tissue

Structure 30-2

Optical Properties of Biological Tissue • Measurable Quantities and Experimental Techniques • Models of Photon Migration in Tissue • RWT Applied to Quantitative Spectroscopy of the Breast • Quantitative Fluorescence Imaging and Spectroscopy • Future Directions

30.2 Infrared Thermal Monitoring of Disease Processes:

Clinical Study 30-15

Emissivity Corrected Temperature • Temperature Calibration • Clinical Study: Kaposi’s Sarcoma

Acknowledgments 30-20 References 30-21

Noninvasive imaging techniques are emerging into the forefront of medical diagnostics and treatmentmonitoring Both near- and mid-infrared imaging techniques have provided invaluable information inthe clinical setting

Near-infrared imaging in the spectrum of 700 to 1100 nm has been used to functionally monitordiseases processes including cancer and lymph node detection and optical biopsies Spectroscopic imagingmodalities have been shown to improve the diagnosis of tumors and add new knowledge about thephysiological properties of the tumor and surrounding tissues Particular emphasis should be placed onidentifying markers that predict the risk of precancerous lesions progressing to invasive cancers, therebyproviding new opportunities for cancer prevention This might be accomplished through the use ofmarkers as contrast agents for imaging using conventional techniques or through refinements of newertechnologies such as MRI or PET scanning The spectroscopic power of light, along with the revolution

in molecular characterization of disease processes has created a huge potential for in vivo optical imaging

and spectroscopy

In the infrared thermal waveband, information about blood circulation, local metabolism, sweat glandmalfunction, inflammation, and healing can be extracted Infrared thermal imaging has been increasingly

30-1

Bronz: “2122_c030” — 2006/2/9 — 21:55 — page 2 — #2

used for detection of cancers As this field evolves, abnormalities or changes in infrared images could beable to provide invaluable information to physicians caring for patients with a variety of disorders Thecurrent status of modern infrared imaging is that of a first line supplement to both clinical exams andcurrent imaging methods Using infrared imaging to detect breast pathology is based on the principlethat both metabolic and vascular activity in the tissue surrounding a new and developing tumor is usuallyhigher than in normal tissue Early cancer growth is dependent on increasing blood circulation by creatingnew blood vessels (angiogenesis) This process results in regional variations that can often be detected byinfrared imaging

Section 30.1 discusses near-infrared (NIR) imaging and its applications in imaging biological tissues.Infrared thermal imaging techniques, calibration and a current clinical trial of Kaposi’s sarcoma aredescribed in Section 30.2

30.1 Near-Infrared Quantitative Imaging of Deep Tissue

Structure

In vivo optical imaging has traditionally been limited to superficial tissue surfaces, directly or

endoscop-ically accessible, and to tissues with a biological window (e.g., along the optical axis of the eye) Thesemethods are based on geometric optics Most tissues scatter light so strongly, however, that for geometricoptics-based equipment to work, special techniques are needed to remove multiply scattered light (such

as pinholes in confocal imaging or interferometry in optical coherence microscopies) Even with thesespecial designs, high resolution optical imaging fails at depths of more than 1 mm below the tissue surface.Collimated visible or infrared (IR) light impinging upon thick tissue is scattered many times in a distance

of∼1 mm, so the analysis of light-tissue interactions requires theories based on the diffusive nature oflight propagation In contrast to x-ray and Positron Emission Tomography (PET), a complex underlyingtheoretical picture is needed to describe photon paths as a function of scattering and absorption properties

of the tissue

Approximately a decade ago, a new field called “Photon Migration” was born that seeks to characterizethe statistical physics of photon motion through turbid tissues The goal has been to image macro-scopic structures in 3D at greater depths within tissues and to provide reliable pathlength estimationsfor noninvasive spectral analysis of tissue changes Although geometrical optics fails to describe lightpropagation under these conditions, the statistical physics of strong, multiply scattered light providespowerful approaches to macroscopic imaging and subsurface detection and characterization Techniquesusing visible and NIR light offer a variety of functional imaging modalities, in addition to density imaging,while avoiding ionizing radiation hazards

In Section 30.1.1, optical properties of biological tissue will be discussed Section 30.1.2 is devoted todiffering methods of measurements Theoretical models for spectroscopy and imaging are discussed inSection 30.1.3 In Sections 30.1.4 and 30.1.5, two studies on breast imaging and the use of exogenousfluorescent markers will be presented as examples of NIR spectroscopy Finally, the future direction of thefield will be discussed in Section 30.1.6

30.1.1 Optical Properties of Biological Tissue

The difficulty of tissue optics is to define optical coefficients of tissue physiology and quantify their

changes to differentiate structures and functional status in vivo Light-tissue interactions dictate the way

that these parameters are defined The two main approaches are the wave and particle descriptions of lightpropagation The first leads to the use of Maxwell’s equations, and therefore quantifies the spatially varyingpermittivity as a measurable quantity For simplistic and historic reasons, the particle interpretation oflight has been mostly used (see section on models of photon migration) In photon transport theory, oneconsiders the behavior of discrete photons as they move through the tissue This motion is characterized

by absorption and scattering, and when interfaces (e.g., layers) are involved, refraction The absorption

Infrared Imaging for Tissue Characterization 30-3

tissue is strongly wavelength dependent and is due to chromophores and water Among the chromophores

in tissue, the dominant component is the hemoglobin in blood In Figure 30.1, hemoglobin absorption

is devided in to oxy- and deoxy-hemoglobin As seen in this figure, in the visible range (600–700 nm),the blood absorption is relatively high compared to absorption in the NIR By contrast, water absorption

is low in the visible and NIR regions and increases rapidly above approximately 950 nm Thus, forgreatest penetration of light in tissue, wavelengths in the 650–950 nm spectrum are used most often Thisregion of the light spectrum is called “the therapeutic window.” One should note that different spectra

of chromophores allow one to separate the contribution of varying functional species in tissue (e.g.,quantification of oxy- and deoxy-hemoglobin to study tissue oxygenation)

Similarly, scattering is characterized by a coefficient,µs, which is the inverse mean free path of photonsbetween scattering events The average size of the scattered photons in tissue, in proportion to thewavelength of the light, places the scattering in the Mie region In the Mie region, a scattering event does notresult in isotropic scattering angles [1,2] Instead, the scattering in tissue is biased in the forward direction.For example, by studying the development of neonatal skin, Saidi et al [3] were able to show thatthe principal sources of anisotropic scattering in muscle are collagen fibers The fibers were determined

to have a mean diameter of 2.2µm In addition to the Mie scattering from the fibers, there is isotropic

Rayleigh scattering due to the presence of much smaller scatterers such as organelles in cells

Anisotropic scattering is quantified in a coefficient, g , which is defined as the mean cosine of the scattering angle, where p (θ) is the probability of a particular scattering angle,

a function of wavelength, are more gradual and have smaller extremes Abnormal tissues such as tumors,fibro-adenomas, and cysts all have scattering properties that are different from normal tissue [6,7] Thus,the scattering coefficient of an inclusion may also be an important clue to disease diagnoses

Bronz: “2122_c030” — 2006/2/9 — 21:55 — page 4 — #4

Theories of photon migration are often based on isotropic scattering Therefore, one must find theappropriate scaling relationships that will allow use of an isotropic scattering model For the case ofdiffusion-like models (e.g., see Reference 8), it has been shown that one may use an isotropic scatteringmodel with a corrected scattering coefficient,µ

s, and obtain equivalent results where:

The corrected scattering coefficient is smaller than the actual scattering which corresponds to a greaterdistance between isotropic scattering events than would occur with anisotropic scattering For this reason,

µ

sis typically called the transport-corrected scattering coefficient

There are instances in which the spectroscopic signatures will not be sufficient for detection of disease.This can occur when the specific disease results in only very small changes to the tissue’s scattering andabsorption properties, or when the scattering and absorption properties are not unique to the disease.Although it is not clear what the limits of detectability are in relationship to diseased tissue properties, it

is clear that there will be cases for which optical techniques based on elastic absorption are inadequate

In such cases, another source of optical contrast, such as fluorescence, will be required to detect andlocate the disease Presence of fluorescent molecules in tissues can provide useful contrast mechanisms.Concentration of these endogenous fluorophores in the body can be related to functional and metabolicactivities, and therefore to the disease processes For example, the concentrations of fluorescent moleculessuch as collagen and NADH have been used to differentiate between normal and abnormal tissue [9].Advances in the molecular biology of disease processes, new immunohistopathological techniques, andthe development of fluorescently-labeled cell surface markers have led to a revolution in specific moleculardiagnosis of disease by histopathology, as well as in research on molecular origins of disease processes(e.g., using fluorescence microscopy in cell biology) As a result, an exceptional level of specificity is nowpossible due to the advances in the design of exogenous markers Molecules can now be tailor-made tobind only to specific receptor sites in the body These receptor sites may be antibodies or other biologicallyinteresting molecules Fluorophores may be bound to these engineered molecules and injected into thebody, where they will preferentially concentrate at specific sites of interest [10,11]

Furthermore, fluorescence may be used as a probe to measure environmental conditions in a particularlocality by capitalizing on changes in fluorophore lifetimes [12,13] Each fluorophore has a characteristiclifetime that quantifies the probability of a specific time delay between fluorophore excitation and emission

In practice, this lifetime may be modified by specific environmental factors such as temperature, pH, andconcentrations of substances such as oxygen In these cases, it is possible to quantify local concentrations ofspecific substances or specific environmental conditions by measuring the lifetime of fluorophores at thesite Whereas conventional fluorescence imaging is very sensitive to non-uniform fluorophore transportand distribution (e.g., blood does not transport molecules equally to all parts of the body), fluorescencelifetime imaging is insensitive to transport non-uniformity as long as a detectable quantity of fluorophores

is present in the site of interest Throughout the following sections, experimental techniques and differingmodels used to quantify these sources of optical contrast will be presented

30.1.2Measurable Quantities and Experimental Techniques

Three classes of measurable quantities prove to be of interest in transforming results of remote sensingmeasurements in tissue into useful physical information The first is the spatial distribution of light orthe intensity profile generated by photons re-emitted through a surface and measured as a function of theradial distance from the source and the detector when the medium is continually irradiated by a pointsource (often a laser) This type of measurement is called continuous wave (CW) The intensity, nominally,does not vary in time The second class is the temporal response to a very short pulse (∼picosecond) ofphotons impinging on the surface of the tissue This technique is called time-resolved and the temporalresponse is known as the time-of-flight (TOF) The third class is the frequency-domain technique inwhich an intensity-modulated laser beam illuminates the tissue In this case, the measured outputs are

Infrared Imaging for Tissue Characterization 30-5

the AC modulation amplitude and the phase shift of the detected signal These techniques could beimplemented in geometries with different arrangements of source(s) and detector(s); (a) in the reflectionmode, source(s) and detector(s) are placed at the same side of the tissue; (b) in the transmission mode,source(s) and detector(s) are located on opposite sides of the tissue In the latter, the source(s) anddetector(s) can move in tandem while scanning the tissue surface and detectors with lateral offsets alsocan be used; and (c) tomographic sampling often uses multiple sources and detectors placed around thecircumference of the target tissue

For CW measurements, the instrumentation is simple and requires only a set of light sources anddetectors In this technique, the only measurable quantity is the intensity of light, and, due to multiplescattering, strong pathlength dispersion occurs which results in a loss of localization and resolution.Hence, this technique is widely used for spectroscopic measurements of bulk tissue properties in whichthe tissue is considered to be homogeneous [14,15] However, CW techniques for imaging abnormaltargets that use only the coherent portion of light, and thereby reject photons with long paths, havealso been investigated Using the transillumination geometry, collimated detection is used to isolateun-scattered photons [16–18] Spatial filtering has been proposed which employs a lens to produce theFourier spectrum of the spatial distribution of light from which the high-order frequencies are removed.The resulting image is formed using only the photons with angles close to normal [19] Polarizationdiscrimination has been used to select those photons which undergo few scattering events and thereforepreserve a fraction of their initial polarization state, as opposed to those photons which experience multiplescattering resulting in complete randomization of their initial polarization state [20] Several investigatorshave used heterodyne detection which involves measuring the beat frequency generated by the spatialand temporal combination of a light beam and a frequency modulated reference beam Constructiveinterference occurs only for the coherent portion of the light [20–22] However, the potential of directimaging using CW techniques in very thick tissue (e.g., breast) has not been established On the other hand,use of models of photon migration implemented in inverse method based on backprojection techniqueshas shown promising results For example, Phillips Medical has used 256 optical fibers placed at theperiphery of a white conical shaped vessel The area of interest, in this case the breast, is suspended in thevessel, and surrounded by a matching fluid Three CW laser diodes sequentially illuminate the breast usingone fiber The detection is done simultaneously by 255 fibers It is now clear that CW imaging cannotprovide direct images with clinically acceptable resolution in thick tissue Attempts are underway to deviseinverse algorithms to separate the effects of scattering and absorption and therefore use this technique forquantitative spectroscopy as proposed by Phillips [23] However, until now, clinical application of CWtechniques in imaging has been limited by the mixture of scattering and absorption of light in the detectedsignal To overcome this problem, time-dependent measurement techniques have been investigated.Time-domain techniques involve the temporal resolution of photons traveling inside the tissue Thebasic idea is that photons with smaller pathlengths are those that arrive earlier to the detector In order

to discriminate between un-scattered or less scattered light and the majority of the photons, whichexperience a large number of multiple scattering, subnanosecond resolution is needed This short timegating of an imaging system requires the use of a variety of techniques involving ultra-fast phenomenaand/or fast detection systems Ultra-fast shuttering is performed using the Kerr effect The birefringence

in the Kerr cell, placed between two crossed polarizers, is induced using very short pulses Transmittedlight through the Kerr cell is recorded, and temporal resolution of a few picoseconds is achieved [19].When an impulse of light (∼picoseconds or hundreds of femtoseconds) is launched at the tissue surface,the whole temporal distribution of photon intensity can be recorded by a streak camera The streakcamera can achieve temporal resolution on the order of few picoseconds up to several nanosececondsdetection time This detection system has been widely used to assess the performance of breast imagingand neonatal brain activity [24,25] The time of flight recorded by the streak camera is the convolution ofthe pulsed laser source (in practice with a finite width) and the actual Temporal Point Spread Function(TPSF) of the diffuse photons Instead of using very short pulse lasers (e.g., Ti–Sapphire lasers), theadvent of pulse diode lasers with relatively larger pulse widths (100 to 400 psec) have reduced the cost oftime-domain imaging However, deconvolution of the incoming pulse and the detected TPSF have been a

Bronz: “2122_c030” — 2006/2/9 — 21:55 — page 6 — #6

greater issue Along with diode laser sources, several groups have also used time-correlated single photoncounting with photomultipliers for recording the TPSF [26,27] Fast time gating is also obtained by usingStimulated Raman Scattering This phenomenon is a nonlinear Raman interaction in some materials such

as hydrogen gas involving the amplification of photons with Stokes shift by a higher energy pump beam.The system operates by amplifying only the earliest arriving photons [28] Less widely used techniquessuch as second-harmonic generation [29], parametric amplification [30] and a variety of others have beenproposed for time-domain (see an excellent review in Reference 31)

For frequency-domain measurements, the requirement is to measure the DC amplitude, the AC litude, and the phase shift of the photon density wave For this purpose a CW light source is modulatedwith a given frequency (∼100 MHz) Lock-in Amplifiers and phase sensitive CCD camera have been used

amp-to record the amplitude and phase [32,33] Multiple sources at different wavelengths can be modulatedwith a single frequency or multiple frequencies [6,34] In the latter case a network analyzer is used toproduce modulation swept from several hundreds of MHz to up to 1 GHz

30.1.3 Models of Photon Migration in Tissue

Photon Migration theories in biomedical optics have been borrowed from other fields such as astrophysics,atmospheric science, and specifically from nuclear reactor engineering [35,36] The common properties

of these physical media and biological tissues are their characterization by elements of randomness inboth space and time Because of many difficulties surrounding the development of a theory based on adetailed picture of the microscopic processes involved in the interaction of light and matter, investigationsare often based on statistical theories These can take a variety of forms, ranging from quite detailedmultiple-scattering theories [36] to transport theory [37] However, the most widely used theory is thetime-dependent diffusion approximation to the transport equation:

∇ · (D ∇(r, t)) − µa(r, t) = 1

c

∂(r, t)

wherer and t are spatial and temporal variables, c is the speed of light in tissue, and D is the diffusion

coefficient related to the absorption and scattering coefficients as follows:

3[µa+ µ

The quantity(r, t) is called the fluence, defined as the power incident on an infinitesimal volume

element divided by its area Note that the equation does not incorporate any angular dependence, thereforeassuming an isotropic scattering However, for the use of the diffusion theory for anisotropic scattering,

the diffusion coefficient is expressed in terms of the transport-corrected scattering coefficient S (r, t)

is the source term The gradient of fluence, J (r, t), at the tissue surface is the measured flux of photons by

the detector:

For CW measurements, the time-dependence of the flux vanishes, and the source term can be seen

as the power impinging in its area For time-resolved measurements, the source term is a Dirac deltafunction describing a very short photon impulse Equation 30.3 has been solved analytically for differenttypes of measurements such as reflection and transmission modes assuming that the optical propertiesremain invariant through the tissue To incorporate the finite boundaries, the method of images has beenused In the simplest case, the boundary has been assumed to be perfectly absorbing which does nottake into account the difference between indices of refraction at the tissue–air interface For semi-infiniteand transillumination geometries, a set of theoretical expressions has been obtained for time-resolvedmeasurements [38]

Infrared Imaging for Tissue Characterization 30-7

The diffusion approximation equation in the frequency-domain is the Fourier transformation of thetime-domain with respect to time Fourier transformation applied to the time-dependent diffusionequation leads to a new equation:

∇ · (D ∇(r, ω)) −µa+iω

c



Here the time variable is replaced by the frequencyω This frequency is the modulation angular frequency

of the source In this model, the fluence can be seen as a complex number describing the amplitude andphase of the photon density wave, dumped with a DC component:

(r, ω) = AC(r, ω) + DC(r, 0) = IACexp(iθ) + DC(r, 0) (30.7)

In the RHS of Equation 30.7, the quantityθ is the phase shift of the diffusing wave For a nonabsorbing

medium, its wavelength is:

For imaging, where the goal is to distinguish between structures in tissue, the diffusion coefficient andthe absorption coefficient in Equation 30.3 and Equation 30.6 become spatial-dependent and are replaced

by D (r) and µa(r) For the cases that an abnormal region is embedded in otherwise homogeneous

tissue, perturbation methods based on Born approximation or Rytov approximation have been used (seeexcellent review in Reference 39) However, for the cases that the goal is to reconstruct the spectroscopicsignatures inside the tissue, no analytical solution exists For these cases, inverse algorithms are devised tomap the spatially varying optical properties Numerical methods such as finite-element or finite-differencemethods have been used to reconstruct images of breast, brain, and muscle [40–42] Furthermore, in thosecases that structural heterogeneity exists, a priori information from other image modalities such as MRIcan be used An example is given in Figure 30.2 Combining MRI and NIR imaging, rat cranium functionalimaging during changes in inhaled oxygen concentration was studied [43] Figure 30.2a,b correspond tothe MRI image and the corresponding constructed finite-element mesh Figure 30.2c,d correspond to theoxygen map of the brain with and without incorporation of MRI geometry and constraints

The use of MRI images has improved dramatically the resolution of the oxygen map The use of opticalfunctional imaging in conjunction with other imaging modalities has opened new possibilities in imagingand treating diseases at the bedside

The second theoretical framework used in tissue optics is the random walk theory (RWT) on a latticedeveloped at the National Institutes of Health [44,45] and historically precedes the use of the diffusionapproximation theory It has been shown that RWT may be used to derive an analytical solution for thedistribution of photon path-lengths in turbid media such as tissue [44] RWT models the diffusion-likemotion of photons in turbid media in a probabilistic manner Using RWT, an expression may be derivedfor the probability of a photon arriving at any point and time given a specific starting point and time.Tissue may be modeled as a 3D cubic lattice containing a finite inclusion, or region of interest, asshown in Figure 30.3 The medium has an absorbing boundary corresponding to the tissue surface, andthe lattice spacing is proportional to the mean photon scattering distance, 1/µ

s The behavior of photons

in the RWT model is described by three dimensionless parameters,ρ, n, µ, which are respectively the

radial distance, the number of steps, and the probability of absorption per lattice step In the RWT model,photons may move to one of the six nearest neighboring lattice points, each with probability 1/6 If the

number of steps, n, taken by a photon traveling between two points on the lattice is known, then the

length of the photon’s path is also known

160.000 155.000 150.000 145.000 140.000 135.000

450 400 350 300 250 200

100.000 80.000 60.000 40.000 20.000 0.000

100.0 80.0 60.0 40.0 20.0 0.0

100.0 80.0 60.0 40.0 20.0 0.0

0 25 50 75 100

280 270 260 250 230 240 220 210

70.000 80.000 60.000 40.000 50.000 30.000 20.000

70.000 80.000 60.000 40.000 50.000 30.000 20.000

0 20 40 60 90

90.0 60.0 40.0 20.0 0.0

105.000 80.000 60.000 40.000 20.000 0.000

0 20 40 60 90

90.0 60.0 40.0 20.0 0.0

110.000 80.000 60.000 40.000 20.000 0.000

110.000 80.000 60.000 40.000 20.000 0.000

0 20 40 60 90

90.0 60.0 40.0 20.0 0.0

90.0 60.0 40.0 20.0 0.0

90.0 60.0 40.0 20.0 0.0

0 20 40 60 90

0 2040 60 90

0 204060 90

100.000 80.000 60.000 40.000 20.000 0.000

0 25 50 75 100

100.000 80.000 60.000 40.000 20.000 0.000

0 25 50 75 100

Brain

Muscle

Bone

Hb total concentration [uM] Hb total concentration [uM]

Hb total concentration [uM]

Hb total concentration [uM]

Hb total concentration [uM] Oxygen saturation [%]

Hb total concentration [uM] Oxygen saturation [%]Oxygen saturation [%]

Oxygen saturation [%]

FIGURE 30.2 (See color inset following page 29-16.) Functional imaging of rat cranium during changes in inhaled

oxygen concentration: (a) MRI image; (b) creation of the mesh to distinguish different compartments in the brain; (c) map of hemoglobin concentration and oxygen saturation of the rat brain without structural constraints from MRI; (d) same as (c) with structural constraints including tissue heterogeneity In (c) and (d) the rows from top correspond

to 13, 8, and 0% (after death) oxygen inhaled (Courtesy of Dartmouth College.)

FIGURE 30.3 2D random walk lattice showing representative photon paths from an emitter to a specific site and then to a detector.

Infrared Imaging for Tissue Characterization 30-9

Random walk theory is useful in predicting the probability distribution of photon path lengths overdistances of at least five mean photon scattering distances The derivation of these probability distributions

is described in papers [44,45] For simplicity in this derivation, the tissue–air interface is considered to beperfectly absorbing; a photon arriving at this interface is counted as arriving at a detector on the tissuesurface The derivation uses the Central Limit Theorem and a Gaussian distribution around lattice points

to obtain a closed-form solution that is independent of the lattice structure

The dimensionless RWT parameters, ρ, n, and µ, described above, may be transformed to actual

parameters, in part, by using time, t , the speed of light in tissue, c, and distance traveled, r, as follows:

ρ → rµ√s

2, n → µ

sct , µ → µa

µ s

(30.9)

As stated previously, scattering in tissue is highly anisotropic Therefore, one must find the appropriatescaling relationships that will allow the use of an isotropic scattering model such as RWT Like diffusiontheory, for RWT [46], it has been shown that one may use an isotropic scattering model with a correctedscattering coefficient,µ

s, and obtain equivalent results The corrected scattering coefficient is smaller thanthe actual scattering that corresponds to a greater distance between isotropic scattering events than wouldoccur with anisotropic scattering RWT has been used to show how one would transition from the use of

µstoµ

sas the distance under considerable increases [47]

As an example, for a homogeneous slab into which a photon has been inserted, the probability, P, of a

photon arriving at a pointρ after n steps is [48]:

where L is the thickness of the slab The method of images has been used to take into account the

two boundaries of the slab Plotting Equation 30.10 yields a photon arrival curve as shown in Figure 30.4;Monte Carlo simulation data are overlaid In the next two sections the use of RWT for imaging will bepresented

30.1.4 RWT Applied to Quantitative Spectroscopy of the Breast

One important and yet extremely challenging areas to apply diffuse optical imaging of deep tissues is thehuman breast (see review article of Hawrysz and Sevick-Muraca [49]) It is clear that any new imaging

Bronz: “2122_c030” — 2006/2/9 — 21:55 — page 10 — #10

or spectroscopic modalities that can improve the diagnosis of breast tumors or can add new knowledgeabout the physiological properties of the breast and surrounding tissues will have a great significance inmedicine

Conventional transillumination using continuous wave (CW) light was used for breast screening severaldecades ago [50] However, because of the high scattering properties of tissue, this method resulted inpoor resolution In the late 1980s, time-resolved imaging techniques were proposed to enhance spatialresolution by detecting photons with very short time-of-flight within the tissue In this technique, a veryshort pulse, of∼picosecond duration, impinges upon the tissue Photons experience dispersion in theirpathlengths, resulting in temporal dispersion in their time-of-flight (TOF)

To evaluate the performance of time-resolved transillumination techniques, RWT on a lattice wasused The analysis of breast transillumination was based on the calculation of the point spread function(PSF) of time resolved photons as they visit differing sites at different planes inside a finite slab of

thickness L The PSF [51], is defined as the probability that a photon inserted into the tissue visits a given site, is detected at the nth step (i.e., a given time), and has the following rather complicated analytical

a +1

b

exp

s /√2) + 1 is dimensionless RWT thickness of the slabs, ¯s(s1, s2, s3) are the dimensionless

coordinates (see Equation 30.9) of any location for which the PSF is calculated Evaluation of time-resolvedimaging showed that strong scattering properties of tissues prevent direct imaging of abnormalities [52].Hence, devising theoretical constructs to separate the effects of the scattering from the absorption wasproposed, thus allowing one to map the optical coefficients as spectroscopic signatures of an abnormaltissue embedded in thick, otherwise normal tissue In this method, accurate quantification of the sizeand optical properties of the target becomes a critical requirement for the use of optical imaging atthe bedside RWT on a lattice has been used to analyze the time-dependent contrast observed in time-resolved transillumination experiments and deduce the size and optical properties of the target andthe surrounding tissue from these contrasts For the theoretical construction of contrast functions, twoquantities are needed First, the set of functions [51] defined previously Second, the set of functions [53]

defined as the probability that a photon is detected at the nth step (i.e., time) in a homogeneous medium

(Equation 30.10)[48]

To relate the contrast of the light intensity to the optical properties and location of abnormal targets

in the tissue, one can take advantage of some features of the theoretical framework One feature is thatthe early time response is most dependent on scattering perturbations, whereas the late time behavior ismost dependent on absorptive perturbations, thus allowing one to separate the influence of scattering andabsorption perturbations on the observed image contrast Increased scattering in the abnormal target ismodeled as a time delay Moreover, it was shown that the scattering contrast is proportional to the time-

derivative of the PSF, dW n /dn, divided by P n[53] The second interesting feature in RWT methodology