Murat Tekalp University of Rochester 53.1 Introduction53.2 ModelingIntra-Frame Observation Model • Multispectral Observa-tion Model •Multiframe Observation Model•RegularizationModels 53.
Trang 1A Murat Tekalp “Image and Video Restoration.”
2000 CRC Press LLC <http://www.engnetbase.com>.
Trang 2Image and Video Restoration
A Murat Tekalp
University of Rochester
53.1 Introduction53.2 ModelingIntra-Frame Observation Model • Multispectral Observa-tion Model •Multiframe Observation Model•RegularizationModels
53.3 Model Parameter EstimationBlur Identification •Estimation of Regularization Parameters
•Estimation of the Noise Variance
53.4 Intra-Frame RestorationBasic Regularized Restoration Methods•Restoration of Im- ages Recorded by Nonlinear Sensors•Restoration of Images Degraded by Random Blurs •Adaptive Restoration for Ring-ing Reduction •Blind Restoration (Deconvolution)•Restora-tion of Multispectral Images •Restoration of Space-VaryingBlurred Images
53.5 Multiframe Restoration and SuperresolutionMultiframe Restoration •Superresolution•Superresolutionwith Space-Varying Restoration
53.6 ConclusionReferences
53.1 Introduction
Digital images and video, acquired by still cameras, consumer camcorders, or even broadcast-qualityvideo cameras, are usually degraded by some amount of blur and noise In addition, most electroniccameras have limited spatial resolution determined by the characteristics of the sensor array Commoncauses of blur are out-of-focus, relative motion, and atmospheric turbulence Noise sources includefilm grain, thermal, electronic, and quantization noise Further, many image sensors and media haveknown nonlinear input-output characteristics which can be represented as point nonlinearities Thegoal of image and video (image sequence) restoration is to estimate each image (frame or field) as itwould appear without any degradations, by first modeling the degradation process, and then applying
an inverse procedure This is distinct from image enhancement techniques which are designed tomanipulate an image in order to produce more pleasing results to an observer without makinguse of particular degradation models On the other hand, superresolution refers to estimating animage at a resolution higher than that of the imaging sensor Image sequence filtering (restorationand superresolution) becomes especially important when still images from video are desired This
is because the blur and noise can become rather objectionable when observing a “freeze-frame”,although they may not be visible to the human eye at the usual frame rates Since many video signalsencountered in practice are interlaced, we address the cases of both progressive and interlaced video
Trang 3The problem of image restoration has sparked widespread interest in the signal processing nity over the past 20 or 30 years Because image restoration is essentially an ill-posed inverse problemwhich is also frequently encountered in various other disciplines such as geophysics, astronomy, med-ical imaging, and computer vision, the literature that is related to image restoration is abundant Aconcise discussion of early results can be found in the books by Andrews and Hunt [1] and Gonzalezand Woods [2] More recent developments are summarized in the book by Katsaggelos [3], and re-view papers by Meinel [4], Demoment [5], Sezan and Tekalp [6], and Kaufman and Tekalp [7] Mostrecently, printing high-quality still images from video sources has become an important applicationfor multi-frame restoration and superresolution methods An in-depth coverage of video filtering
commu-methods can be found in the book Digital Video Processing by Tekalp [8] This chapter summarizeskey results in digital image and video restoration
53.2 Modeling
Every image restoration/superresolution algorithm is based on an observation model, which relatesthe observed degraded image(s) to the desired “ideal” image, and possibly a regularization model,
which conveys the available a priori information about the ideal image The success of image
restora-tion and/or superresolurestora-tion depends on how good the assumed mathematical models fit the actualapplication
53.2.1 Intra-Frame Observation Model
Let the observed and ideal images be sampled on the same 2-D lattice3 Then, the observed blurred
and noisy image can be modeled as
whereg, f , and v denote vectors representing lexicographical ordering of the samples of the observed
image, ideal image, and a particular realization of the additive (random) noise process, respectively.The operatorD is called the blur operator The response of the image sensor to light intensity is
represented by the memoryless mappings(·), which is, in general, nonlinear (This nonlinearity has
often been ignored in the literature for algorithm development.)
The blur may be space-invariant or space-variant For space-invariant blurs,D becomes a
convo-lution operator, which has block-Toeplitz structure; and Eq (53.1) can be expressed, in scalar form,as
whered(m1, m2) and S ddenote the kernel and support of the operatorD, respectively The kernel
d(m1, m2) is the impulse response of the blurring system, often called the point spread function
(PSF) In case of space-variant blurs, the operatorD does not have a particular structure; and the
observation equation can be expressed as a superposition summation
whereS d (n1, n2) denotes the support of the PSF at the pixel location (n1, n2).
The noise is usually approximated by a zero-mean, white Gaussian random field which is additiveand independent of the image signal In fact, it has been generally accepted that more sophisticatednoise models do not, in general, lead to significantly improved restorations
Trang 453.2.2 Multispectral Observation Model
Multispectral images refer to image data with multiple spectral bands that exhibit inter-band relations An important class of multispectral images are color images with three spectral bands.Suppose we haveK spectral bands, each blurred by possibly a different PSF Then, the vector-matrix
cor-model (53.1) can be extended to multispectral modeling as
denoteN2K × 1 vectors representing the multispectral observed, ideal, and noise data, respectively,
stacked as composite vectors, and
is anN2K × N2K matrix representing the multispectral blur operator In most applications, D is
block diagonal, indicating no inter-band blurring
53.2.3 Multiframe Observation Model
Suppose a sequence of blurred and noisy imagesg k (n1, n2), k = 1, , L, corresponding to multiple
shots (from different angles) of a static scene sampled on a 2-D lattice or frames (fields) of videosampled (at different times) on a 3-D progressive (interlaced) lattice, is available Then, we may
be able to estimate a higher-resolution “ideal” still imagef (m1, m2) (corresponding to one of the
observed frames) sampled on a lattice, which has a higher sampling density than that of the inputlattice The main distinction between the multispectral and multiframe observation models is thathere the observed images are subject to sub-pixel shifts (motion), possibly space-varying, whichmakes high-resolution reconstruction possible In the case of video, we may also model blurring due
to motion within the aperture time to further sharpen images
To this effect, each observed image (frame or field) can be related to the desired high-resolutionideal still-image through the superposition summation [8]
where the support of the summation over the high-resolution grid(m1, m2) at a particular observed
pixel(n1, n2; k) depends on the motion trajectory connecting the pixel (n1, n2; k) to the ideal image,
the size of the support of the low-resolution sensor PSFh a (x1, x2) with respect to the high resolution
grid, and whether there is additional optical (out-of-focus, motion, etc.) blur Because the relativepositions of low- and high-resolution pixels in general vary by spatial coordinates, the discrete sensorPSF is space-varying The support of the space-varying PSF is indicated by the shaded area in Fig.53.1,where the rectangle depicted by solid lines shows the support of a low-resolution pixel over the high-resolution sensor array The shaded region corresponds to the area swept by the low-resolution pixeldue to motion during the aperture time [8]
Trang 5FIGURE 53.1: Illustration of the discrete system PSF.
Note that the model (53.5) is invalid in case of occlusion That is, each observed pixel(n1, n2; k)
can be expressed as a linear combination of several desired high-resolution pixels(m1, m2), provided
that(n1, n2; k) is connected to (m1, m2) by a motion trajectory We assume that occlusion regions can be detected a priori using a proper motion estimation/segmentation algorithm.
53.2.4 Regularization Models
Restoration is an ill-posed problem which can be regularized by modeling certain aspects of the desired
“ideal” image Images can be modeled as either 2-D deterministic sequences or random fields A priori information about the ideal image can then be used to define hard or soft constraints on the
solution In the deterministic case, images are usually assumed to be members of an appropriateHilbert space, such as a Euclidean space with the usual inner product and norm For example, in thecontext of set theoretic restoration, the solution can be restricted to be a member of a set consisting
of all images satisfying a certain smoothness criterion [9] On the other hand, constrained leastsquares (CLS) and Tikhonov-Miller regularization use quadratic functionals to impose smoothnessconstraints in an optimization framework
In the random case, models have been developed for the pdf of the ideal image in the context of
maximum a posteriori (MAP) image restoration For example, Trussell and Hunt [10] have proposed
a Gaussian distribution with space-varying mean and stationary covariance as a model for the pdf
of the image Geman and Geman [11] proposed a Gibbs distribution to model the pdf of the image.Alternatively, if the image is assumed to be a realization of a homogeneous Gauss-Markov randomprocess, then it can be statistically modeled through an autoregressive (AR) difference equation [12]
by Jeng and Woods [13]
53.3 Model Parameter Estimation
In this section, we discuss methods for estimating the parameters that are involved in the observationand regularization models for subsequent use in the restoration algorithms
Trang 653.3.1 Blur Identification
Blur identification refers to estimation of both the support and parameters of the PSF{d(n1, n2) : (n1, n2) ∈ S d} It is a crucial element of image restoration because the quality of restored images ishighly sensitive to errors in the PSF [14] An early approach to blur identification has been based onthe assumption that the original scene contains an ideal point source, and that its spread (hence thePSF) can be determined from the observed image Rosenfeld and Kak [15] show that the PSF canalso be determined from an ideal line source These approaches are of limited use in practice because
a scene, in general, does not contain an ideal point or line source and the observation noise may notallow the measurement of a useful spread
Models for certain types of PSF can be derived using principles of optics, if the source of theblur is known [7] For example, out-of-focus and motion blur PSF can be parameterized with a fewparameters Further, they are completely characterized by their zeros in the frequency-domain Powerspectrum and cepstrum (Fourier transform of the logarithm of the power spectrum) analysis methodshave been successfully applied in many cases to identify the location of these zero-crossings [16,17].Alternatively, Chang et al [18] proposed a bispectrum analysis method, which is motivated by thefact that bispectrum is not affected, in principle, by the observation noise However, the bispectralmethod requires much more data than the method based on the power spectrum Note that PSFs,which do not have zero crossings in the frequency domain (e.g., Gaussian PSF modeling atmosphericturbulence), cannot be identified by these techniques
Yet another approach for blur identification is the maximum likelihood (ML) estimation approach.The ML approach aims to find those parameter values (including, in principle, the observation noisevariance) that have most likely resulted in the observed image(s) Different implementations of the
ML image and blur identification are discussed under a unifying framework [19] Pavlovi´c andTekalp [20] propose a practical method to find the ML estimates of the parameters of a PSF based on
a continuous domain image formation model
In multi-frame image restoration, blur identification using more than one frame at a time becomespossible For example, the PSF of a possibly space-varying motion blur can be computed at eachpixel from an estimate of the frame-to-frame motion vector at that pixel, provided that the shutterspeed of the camera is known [21]
53.3.2 Estimation of Regularization Parameters
Regularization model parameters aim to strike a balance between the fidelity of the restored image tothe observed data and its smoothness Various methods exist to identify regularization parameters,such as parametric pdf models, parametric smoothness constraints, and AR image models Somerestoration methods require the knowledge of the power spectrum of the ideal image, which can beestimated, for example, from an AR model of the image The AR parameters can, in turn, be estimatedfrom the observed image by a least squares [22] or an ML technique [63] On the other hand,non-parametric spectral estimation is also possible through the application of periodogram-basedmethods to a prototype image [69,23] In the context of maximum a posteriori (MAP) methods, the a priori pdf is often modeled by a parametric pdf, such as a Gaussian [10] or a Gibbsian [11] Standardmethods for estimating these parameters do not exist Methods for estimating the regularizationparameter in the CLS, Tikhonov-Miller, and related formulations are discussed in [24]
53.3.3 Estimation of the Noise Variance
Almost all restoration algorithms assume that the observation noise is a zero-mean, white randomprocess that is uncorrelated with the image Then, the noise field is completely characterized by itsvariance, which is commonly estimated by the sample variance computed over a low-contrast local
Trang 7region of the observed image As we will see in the following section, the noise variance plays animportant role in defining constraints used in some of the restoration algorithms.
53.4 Intra-Frame Restoration
We start by first looking at some basic regularized restoration strategies, in the case of an LSI blur modelwith no pointwise nonlinearity The effect of the nonlinear mappings(.) is discussed in Section53.4.2.Methods that allow PSFs with a random components are summarized in Section53.4.3 Adaptiverestoration for ringing suppression and blind restoration are covered in Sections53.4.4and53.4.5,respectively Restoration of multispectral images and space-varying blurred images are addressed inSections53.4.6and53.4.7, respectively
53.4.1 Basic Regularized Restoration Methods
When the mappings(.) is ignored, it is evident from Eq (53.1) that image restoration reduces tosolving a set of simultaneous linear equations If the matrixD is nonsingular (i.e., D−1exists) andthe vectorg lies in the column space of D (i.e., there is no observation noise), then there exists a
unique solution which can be found by direct inversion (also known as inverse filtering) In practice,however, we almost always have an underdetermined (due to boundary truncation problem [14]) andinconsistent (due to observation noise) set of equations In this case, we resort to a minimum-normleast-squares solution A least squares (LS) solution (not unique when the columns ofD are linearly
dependent) minimizes the norm-square of the residual
It follows that the regularized inverse deviates from the pseudo-inverse at these frequencies whichleads to other types of artifacts, generally known as regularization artifacts [14] Various strategiesfor regularized inversion (and how to achieve the right amount of regularization) are discussed inthe following
Singular-Value Decomposition Method
The pseudo-inverseD+can be computed using the singular value decomposition (SVD) [1]
runs toR, the rank of D Under the assumption that D is block-circulant (corresponding to a
circular convolution), the PIS computed through Eq (53.8) is equivalent to the frequency domain
Trang 8whereD(u, v) denotes the frequency response of the blur This is because a block-circulant matrix
can be diagonalized by a 2-D discrete Fourier transformation (DFT) [2]
Regularization of the PIS can then be achieved by truncating the singular value expansion (53.8)
to eliminate all terms corresponding to smallλ i (which are responsible for the noise amplification)
at the expense of reduced resolution Truncation strategies are generally ad-hoc in the absence ofadditional information
Iterative Methods (Landweber Iterations)
Several image restoration algorithms are based on variations of the so-called Landweber tions [25,26,27,28,31,32]
itera-f k+1 = f k + RD T g − Df k (53.10)whereR is a matrix that controls the rate of convergence of the iterations There is no general way
to select the bestC matrix If the system (53.1) is nonsingular and consistent (hardly ever the case),the iterations (53.10) will converge to the solution If, on the other hand, (53.1) is underdeterminedand/or inconsistent, then (53.10) converges to a minimum-norm least squares solution (PIS) Thetheory of this and other closely related algorithms are discussed by Sanz and Huang [26] and Tom
et al [27] Kawata and Ichioka [28] are among the first to apply the Landweber-type iterations toimage restoration, which they refer to as “reblurring” method
Landweber-type iterative restoration methods can be regularized by appropriately terminatingthe iterations before convergence, since the closer we are to the pseudo-inverse, the more noiseamplification we have A termination rule can be defined on the basis of the norm of the residualimage signal [29] Alternatively, soft and/or hard constraints can be incorporated into iterations toachieve regularization The constrained iterations can be written as [30,31]
f k+1 = Chf k + RD T g − Df ki (53.11)whereC is a nonexpansive constraint operator, i.e., ||C(f1) − C(f2)|| ≤ ||f1− f2||, to guaranteethe convergence of the iterations Application of Eq (53.11) to image restoration has been extensivelystudied (see [31,32] and the references therein)
Constrained Least Squares Method
Regularized image restoration can be formulated as a constrained optimization problem, where
a functional||Q(f )||2of the image is minimized subject to the constraint||g − Df ||2= σ2 Here
σ2is a constant, which is usually set equal to the variance of the observation noise The constrainedleast squares (CLS) estimate minimizes the Lagrangian [34]
J CLS (f ) = ||Q(f )||2+ α||g − Df ||2− σ2
(53.12)whereα istheLagrangemultiplier TheoperatorQischosensuchthattheminimizationofEq.(53.12)enforces some desired property of the ideal image For instance, ifQ is selected as the Laplacian
operator, smoothness of the restored image is enforced The CLS estimate can be expressed, by takingthe derivative of Eq (53.12) and setting it equal to zero, as [1]
ˆf =D H D + γ Q H Q−1D H g (53.13)
Trang 9whereH stands for Hermitian (i.e., complex-conjugate and transpose) The parameterγ = 1
α (the
regularization parameter) must be such that the constraint||g − Df ||2= σ2is satisfied It is oftencomputed iteratively [2] A sufficient condition for the uniqueness of the CLS solution is thatQ−1
exists For space-invariant blurs, the CLS solution can be expressed in the frequency domain as [34]
ˆF(u, v) = |D(u, v)| D2∗+ γ |L(u, v)| (u, v) 2G(u, v) (53.14)
where∗denotes complex conjugation A closely related regularization method is the Tikhonov-Miller(T-M) regularization [33,35] T-M regularization has been applied to image restoration [31,32,36].Recently, neural network structures implementing the CLS or T-M image restoration have also beenproposed [37,38]
Linear Minimum Mean Square Error Method
The linear minimum mean square error (LMMSE) method finds the linear estimate whichminimizes the mean square error between the estimate and ideal image, using up to second orderstatistics of the ideal image Assuming that the ideal image can be modeled by a zero-mean homoge-neous random field and the blur is space-invariant, the LMMSE (Wiener) estimate, in the frequencydomain, is given by [8]
ˆF(u, v) = |D(u, v)| D2+ σ∗(u, v)2
v /|P (u, v)|2G(u, v) (53.15)whereσ2
v is the variance of the observation noise (assumed white) and|P (u, v)|2stands for thepower spectrum of the ideal image The power spectrum of the ideal image is usually estimated from
a prototype It can be easily seen that the CLS estimate (53.14) reduces to the Wiener estimate bysetting|L(u, v)|2= σ2
v /|P (u, v)|2andγ = 1.
A Kalman filter determines the causal (up to a fixed lag) LMMSE estimate recursively It is based
on a state-space representation of the image and observation models In the first step of Kalmanfiltering, a prediction of the present state is formed using an autoregressive (AR) image model andthe previous state of the system In the second step, the predictions are updated on the basis of theobserved image data to form the estimate of the present state Woods and Ingle [39] applied 2-Dreduced-update Kalman filter (RUKF) to image restoration, where the update is limited to only thosestate variables in a neighborhood of the present pixel The main assumption here is that a pixel isinsignificantly correlated with pixels outside a certain neighborhood about itself More recently, areduced-order model Kalman filtering (ROMKF), where the state vector is truncated to a size that is onthe order of the image model support has been proposed [40] Other Kalman filtering formulations,including higher-dimensional state-space models to reduce the effective size of the state vector, havebeen reviewed in [7] The complexity of higher-dimensional state-space model based formulations,however, limits their practical use
Maximum A posteriori Probability Method
The maximum a posteriori probability (MAP) restoration maximizes the a posteriori probability
density function (pdf)p(f |g), i.e., the likelihood of a realization of f being the ideal image given
the observed datag Through the application of the Bayes rule, we have
wherep(g|f ) is the conditional pdf of g given f (related to the pdf of the noise process) and p(f ) is
the a priori pdf of the ideal image We usually assume that the observation noise is Gaussian, leading
Trang 10Trussell and Hunt [10] used non-stationary a priori pdf models, and proposed a modified form of
the Picard iteration to solve the nonlinear maximization problem They suggested using the variance
of the residual signal as a criterion for convergence Geman and Geman [11] proposed using a Gibbs
random field model for the a priori pdf of the ideal image They used simulated annealing procedures
to maximize Eq (53.16) It should be noted that the MAP procedures usually require significantlymore computation compared to, for example, the CLS or Wiener solutions
Maximum Entropy Method
A number of maximum entropy (ME) approaches have been discussed in the literature, whichvary in the way that the ME principle is implemented A common feature of all these approaches,however, is their computational complexity Maximizing the entropy enforces smoothness of therestored image (In the absence of constraints, the entropy is highest for a constant-valued image).One important aspect of the ME approach is that the nonnegativity constraint is implicitly imposed
on the solution because the entropy is defined in terms of the logarithm of the intensity
Frieden was the first to apply the ME principle to image restoration [41] In his formulation, thesum of the entropy of the image and noise, given by
be viewed as another form of Tikhonov regularization (or constrained least squares formulation),where the entropy of the image