Tài liệu 25 Signal Recovery from Partial Information ppt

25 Signal Recovery from PartialProlate Spheroidal Wavefunctions 25.3 Least Squares Solutions Wiener Filtering •The Pseudoinverse Solution•RegularizationTechniques 25.4 Signal Recovery us

Podilchuk, C “Signal Recovery from Partial Information”

Digital Signal Processing Handbook

Ed Vijay K Madisetti and Douglas B Williams

Boca Raton: CRC Press LLC, 1999

25 Signal Recovery from Partial

Prolate Spheroidal Wavefunctions

25.3 Least Squares Solutions

Wiener Filtering •The Pseudoinverse Solution•RegularizationTechniques

25.4 Signal Recovery using Projection onto Convex Sets (POCS)

The POCS Framework

25.5 Row-Based Methods25.6 Block-Based Methods25.7 Image Restoration Using POCSReferences

25.1 Introduction

Signal recovery has been an active area of research for applications in many different scientific ciplines A central reason for exploring the feasibility of signal recovery is due to the limitationsimposed by a physical device on the amount of data one can record For example, for diffraction-limited systems, the finite aperture size of the lens constrains the amount of frequency informationthat can be captured The image degradation is due to attenuation of high frequency componentsresulting in a loss of details and other high frequency information In other words, the finite aperturesize of the lens acts like a lowpass filter on the input data In some cases, the quality of the recordedimage data can be improved by building a more costly recording device but many times the requiredcondition for acceptable data quality is physically unrealizable or too costly Other times signal re-covery may be necessary is for the recording of a unique event that cannot be reproduced under moreideal recording conditions

dis-Some of the earliest work on signal recovery includes the work by Sondhi [1] and Slepian [2] onrecovering images from motion blur and Helstrom [3] on least squares restoration A sampling ofsome of the signal recovery algorithms applied to different types of problems can be found in [4]–[21] Further reading includes the other sections in this book, Chapter 53, and the extended list ofreferences provided by all the authors

The simple signal degradation model described in the next section turns out to be a useful tation for many different problems encountered in practice Some examples that can be formulatedusing the general signal recovery paradigm include image restoration, image reconstruction, spectral

represen-estimation, and filter design We distinguish between image restoration, which pertains to image covery based on a measured distorted version of the original image, and image reconstruction, whichrefers most commonly to medical imaging where the image is reconstructed from a set of indirectmeasurements, usually projections For many of the signal recovery applications, it is desirable toextrapolate a signal outside of a known interval Extrapolating a signal in the spatial or temporaldomain could result in improved spectral resolution and applies to such problems as power spec-trum estimation, radio–astronomy, radar target detection, and geophysical exploration The dual

re-problem, extrapolating the signal in the frequency domain, also known as superresolution, results in

improved spatial or temporal resolution and is desirable in many image restoration problems Aswill be shown later, the standard inverse filtering techniques are not able to resolve the signal estimatebeyond the diffraction limit imposed by the physical measuring device

The observed signal is degraded from the original signal by both the measuring device as well asexternal conditions Besides the measured, distorted output signal we may have some additionalinformation about the following: the measuring system and external conditions, such as noise, as

well as some a priori knowledge about the desired signal to be restored or reconstructed In order

to produce a good estimate of the original signal, we should take advantage of all the availableinformation

Although the data recovery algorithms described here apply in general to any data type, we derivemost of the techniques based on two-dimensional input data for image processing applications Formost cases, it is straightforward to adapt the algorithms to other data types Examples of data recoverytechniques for different inputs are illustrated in the other sections in this book as well as Chapter 53for image restoration The material in this section requires some basic knowledge of linear algebra

as found in [22]

Section25.2presents the signal degradation model and formulates the signal recovery problem.The early attempts of signal recovery based on inverse filtering are presented in Section25.3 Theconcept of Projection Onto Convex Sets (POCS) described in Section25.4allows us to introduce a

priori knowledge about the original signal in the form of linear as well as nonlinear constraints into

the recovery algorithm Convex set theoretic formulations allow us to design recovery algorithmsthat are extremely flexible and powerful Sections25.5and25.6present some basic POCS-basedalgorithms and Section25.7presents a POCS-based algorithm for image restoration as well as someresults The sample algorithms presented here are not meant to be exhaustive and the reader isencouraged to read the other sections in this chapter as well as the references for more details

25.2 Formulation of the Signal Recovery Problem

Signal recovery can be viewed as an estimation process in which operations are performed on anobserved signal in order to estimate the ideal signal that would be observed if no degradation waspresent In order to design a signal recovery system effectively, it is necessary to characterize thedegradation effects of the physical measuring system The basic idea is to model the signal degradationeffects as accurately as possible and perform operations to undo the degradations and obtain a restoredsignal When the degradation cannot be modeled sufficiently, even the best recovery algorithms willnot yield satisfactory results For many applications, the degradation system is assumed to be linearand can be modeled as a Fredholm integral equation of the first kind expressed as

g(x) =

Z +∞

This is the general case for a one-dimensional signal wheref and g are the original and measured

signals, respectively,n represents noise, and h(x; a) is the impulse response or the response of the

measuring system to an impulse at coordinatea.1 A block diagram illustrating the general dimensional signal degradation system is shown in Fig.25.1 For image processing applications, wemodify this equation to the two-dimensional case, that is,

The degradation operatorh is commonly referred to as a point spread function (PSF) in imaging

applications because in optics,h is the measured response of an imaging system to a point of light.

FIGURE 25.1: Block diagram of the signal recovery problem

The Fourier transform of the point spread functionh(x, y) denoted as H(w x , w y ) is known as

the optical transfer function (OTF) and can be expressed as

ill-posed when a small change in the observed image, g, results in a large change in the solution, f

Most signal recovery problems in practice are ill-posed

The continuous version of the degradation system for two-dimensional signals formulated in

Eq (25.2) can be expressed in discrete form by replacing the continuous arguments with arrays ofsamples in two dimensions, that is,

It is convenient for image recovery purposes to represent the discrete formulation given in Eq (25.4)

as a system of linear equations expressed as

where g, f, and n are the lexicographic row-stacked versions of the discretized versions ofg, f , and

n in Eq (25.4) and H is the degradation matrix composed of the PSF.

This section presents an overview of some of the techniques proposed to estimate f when the

recovery problem can be modeled by Eq (25.5) If there is no external noise or measurement error

1 This corresponds to the case of a shift–varying impulse response.

and the set of equations is consistent, Eq (25.5) reduces to

For most systems, the degradation matrix H is highly structured and quite sparse The additive

noise term due to measurement errors and external and internal noise sources is represented by the

vector n At first glance, the solution to the signal recovery problem seems to be straightforward — find the inverse of the matrix H to solve for the unknown vector f It turns out that the solution is not

so simple because in practice the degradation operator is usually ill-conditioned or rank-deficientand the problem of inconsistencies or noise must be addressed Other problems that may ariseinclude computational complexity due to extremely large problem dimensions especially for imageprocessing applications The algorithms described here try to address these issues for the generalsignal recovery problem described by Eq (25.5)

25.2.1 Prolate Spheroidal Wavefunctions

We introduce the problem of signal recovery by examining a one-dimensional, linear, time-invariantsystem that can be expressed as

g(x) =

Z +T

whereg(x) is the observed signal, f (α) is the desired signal of finite support on the interval(−T , +T ),

andh(x) denotes the degradation operator Assuming that the degradation operator in this case is

an ideal lowpass filter,h can be described mathematically as

h(x) = sin(x)

For this particular case, it is possible to solve for the exact signalf (x) with prolate spheroidal

wavefunctions [23] The key to successfully solving forf lies in the fact that prolate spheroidal

wavefunctions are the eigenfunctions of the integral equation expressed by Eq (25.7) with Eq (25.8)

as the degradation operator This relationship is expressed as:

This allows the functionsg(x) and f (x) to be expressed as the series expansion,

whereψ Ln (x) are the prolate spheroidal functions truncated to the interval (−T , T ) The coefficients

c nandd nare given by

age,g(x), using prolate spheroidal wavefunctions The difficulties of signal recovery become more

apparent when we examine the simple diffraction-limited case in relation to prolate spheroidal functions as described in Eq (25.21) The finite aperture size of a diffraction-limited system translates

wave-to eigenvaluesλ nwhich exhibit a unit–step response; that is, the several largest eigenvalues are proximately one followed by a succession of eigenvalues that rapidly fall off to zero The solution given

ap-by Eq (25.21) will be extremely sensitive to noise for small eigenvaluesλ n Therefore, for the general

problem represented in vector–space by Eq (25.5), the degradation operatorH is ill-conditioned or

rank-deficient due to the small or zero-valued eigenvalues, and a simple inverse operation will notyield satisfactory results Many algorithms have been proposed to find a compromise between exactdeblurring and noise amplification These techniques include Wiener filtering and pseudo-inversefiltering We begin our overview of signal recovery techniques by examining some of the methodsthat fall under the category of optimization-based approaches

25.3 Least Squares Solutions

The earliest attempts toward signal recovery are based on the concept of inverting the degradationoperator to restore the desired signal Because in practical applications the system will often be ill-conditioned, several problems can arise Specifically, high detail signal information may be masked

by observation noise, or a small amount of observation noise may lead to an estimate that containsvery large false high frequency components Another potential problem with such an approach

is that for a rank-deficient degradation operator, the zero-valued eigenvalues cannot be inverted.Therefore, the general inverse filtering approach will not be able to resolve the desired signal beyondthe diffraction limit imposed by the measuring device In other words, referring to the vector–spacedescription, the data that has been nulled out by the zero-valued eigenvalues cannot be recovered

25.3.1 Wiener Filtering

Wiener filtering combines inverse filtering with a priori statistical knowledge about the noise and

unknown signal [24] in order to deal with the problems associated with an ill-conditioned system.The impulse response of the restoration filter is chosen to minimize the mean square error asdefined by

Ef= E

f − ˆf2



(25.22)

where ˆf denotes the estimate of the ideal signal f and E{·} denotes the expected value The Wiener

filter estimate is expressed as

H−1

W = R ff HT

HR ff HT + R nn

(25.23)

where R ff and R nn are the covariance matrices of f and n, respectively, and f and n are assumed to be

uncorrelated; that is,

The superscriptT in the above equations denotes transpose The Wiener filter can also be expressed

in the Fourier domain as

H−1W = H∗S ff

|H|2S ff + S nn (25.27)

whereS denotes the power spectral density, the superscript ∗ denotes the complex conjugate, and H

denotes the Fourier transform of H Note that when the noise power is zero, the Wiener filter reduces

to the inverse filter; that is,

H−1W = H−1. (25.28)

The Wiener filter approach for signal recovery assumes that the power spectra are known for theinput signal and the noise Also, this approach assumes that finding a least squares solution thatoptimizes Eq (25.22) is meaningful For the case of image processing, it has been shown, specifically

in the context of image compression, that the mean square error (mse) does not predict subjectiveimage quality [25] Many signal processing algorithms are based on the least squares paradigmbecause the solutions are tractable and, in practice, such approaches have produced some usefulresults However, in order to define a more meaningful optimization metric in the design of imageprocessing algorithms, we need to incorporate a human visual model into the algorithm design Inthe area of image coding, several coding schemes based on perceptual criteria have been shown toproduce improved results over schemes based on maximizing signal–to–noise ratio or minimizingmse [25] Likewise, the Wiener filtering approach will not necessarily produce an estimate thatmaximizes perceived image or signal quality Another limitation of the Wiener filter approach is that

the solution will not necessarily be consistent with any a priori knowledge about the desired signal

characteristics In addition, the Wiener filter approach does not resolve the desired signal beyondthe diffraction limit imposed by the measuring system For more details on Wiener filtering and thevarious applications, see other chapters in this book

25.3.2 The Pseudoinverse Solution

The Wiener filters attempt to minimize the noise amplification obtained in a direct inverse by viding a taper determined by the statistics of the signal and noise process under consideration Inpractice, the power spectra of the noise and desired signal might not be known Here we presentwhat is commonly referred to as the generalized inverse solution This will be the framework forsome of the signal recovery algorithms described later

pro-The pseudoinverse solution is an optimization approach that seeks to minimize the least squareserror as given by

En = nTn = g − Hf
T (g − Hf). (25.29)The least squares solution is not unique when the rank of theM ×N matrix H is r < N ≤ M In other

words, there are many solutions that satisfy Eq (25.29) However, the Moore-Penrose generalizedinverse or pseudoinverse [26] does provide a unique least squares solution based on determiningthe least squares solution with minimum norm For a consistent set of equations as described in

Eq (25.6), a solution is sought that minimizes the least squares estimation error; that is,

HTg = HTHf. (25.31)The generalized inverse solution, also known as the Moore-Penrose generalized inverse, pseudoin-verse, or least squares solution with minimum norm is defined as

f†=HTH−1

HTg = H†

where the dagger†denotes the pseudoinverse and the rank of H isr = N ≤ M.

For the case of an inconsistent set of equations as described in Eq (25.5), the pseudoinverse solutionbecomes

f†= H†g = H†Hf + H†n (25.33)

where f†is the minimum norm, least squares solution If the set of equations are overdetermined withrankr = N < M, H†H becomes an identity matrix of sizeN denoted as I Nand the pseudoinversesolution reduces to

where the productk H†kk H k is the condition number of H This quantity determines the relative

error in the estimate in terms of the ratio of the vector norm of the noise to the vector norm of theobserved image The condition number ofH is defined as

C H =k H†kk H k= σ1

whereσ1andσ Ndenote the largest and smallest singular values of the matrixH , respectively The

larger the condition number, the greater the sensitivity to noise perturbations A matrix with a largecondition number, typically greater than 100, results in an ill-conditioned system

The pseudoinverse solution is best described by diagonalizing the degradation matrix H using

singular value decomposition (SVD) [22] SVD provides a way to diagonalize any arbitraryM × N

matrix In this case, we wish to diagonalize H; that is,

where U is a unitary matrix composed of the orthonormal eigenvectors of HTH, V is a unitary matrix

composed of the orthonormal eigenvectors of HHT, and6 is a diagonal matrix composed of the

singular values of H The number of nonzero diagonal terms denotes the rank of H The degradation

matrix can be expressed in series form as

The series form of the pseudoinverse solution using SVD allows us to solve for the pseudoinversesolution using a sequential restoration algorithm expressed as

easy to implement for the case of a circulant degradation matrix H, where the unitary matrices in

Eq (25.37) reduce to the discrete Fourier transform (DFT)

25.3.3 Regularization Techniques

Smoothing and regularization techniques [27,28,29] have been proposed in an attempt to overcomethe problems associated with inverting ill-conditioned degradation operators for signal recovery.These methods attempt to force smoothness on the solution of a least squares error problem Theproblem can be formulated in two different ways One way of formulating the problem is

minimize:

subject to: 

g − HˆfT W(g − Hˆf) = e (25.44)

where S represents a smoothing matrix, W is an error weighting matrix, and e is a residual scalar

estimation error The error weighting matrix can be chosen as W = R−1

nn The smoothing matrix is

typically composed of the first or second order difference For this case, we wish to find the stationarypoint of the Lagrangian expression

F (ˆf, λ) = ˆf TSˆf+ λ

g − HˆfTW(g − Hˆf) − e



The solution is found by taking derivatives with respect to f andλ and setting them equal to zero.

The solution for a nonsingular overdetermined set of equations becomes

ˆf =HTWH+ 1

λS

−1

whereλ is chosen to satisfy the compromise between residual error and smoothness in the estimate.

Alternately, this problem can be formulated as

g − HˆfTW(g − Hˆf) (25.47)subject to:

ˆf =HTWH+ γ S−1HTWg. (25.50)

Note that for the two problem formulations, the results as given by Eq (25.46) and Eq (25.50) areidentical ifγ = 1/λ The shortcomings of such a regularization technique is that the smoothing

function S must be estimated and either the degree of smoothness,d, or the degree of error, e, must

be known to determineγ or λ.

Constrained restoration techniques have also been developed [30] to overcome the problem of

an ill-conditioned system Linear equality constraints and linear inequality constraints have beenenforced to yield one-step solutions similar to those described in this section All the techniquesdescribed thus far attempt to overcome the problem of noise corrupted data and ill-conditionedsystems by forcing some sort of taper on the inverse of the degradation operator The sampling ofalgorithms discussed thus far fall under the category of optimization techniques where the objectivefunction to be minimized is the least squares error Recovery algorithms that fall under the category of

optimization-based algorithms include maximum likelihood, maximum a posteriori, and maximum

A broad set of recovery algorithms has been proposed to conform to the general framework introduced

by the theory of projection onto convex sets (POCS) [31] The POCS framework enables one to define

an iterative recovery algorithm that can incorporate a number of linear as well as nonlinear constraints

that satisfy certain properties The more a priori information about the desired signal that one can

incorporate into the algorithm, the more effective the algorithm becomes In [21], POCS is presented

as a particular example of a much broader class of algorithms described as Set Theoretic Estimation.The author distinguishes between two basic approaches to a signal estimation or recovery problem:optimization-based approaches and set theoretic approaches The effectiveness of optimization-based approaches is highly dependent on defining a valid optimization criterion that, in practice,

is usually determined by computational tractability rather than how well it models the problem.The optimization-based approaches seek a unique solution based on some predefined optimizationcriterion The optimization-based approaches include the least squares techniques of the previous

section as well as maximum likelihood (ML), maximum a posteriori (MAP), and maximum entropy techniques Set theoretic estimation is based on the concept of finding a feasible solution, that is, a solution that is consistent with all the available a priori information Unlike the optimization-based

approaches which seek to find one optimum solution, the set theoretic approaches usually determineone of many possible feasible solutions Many problems in signal recovery can be approachedusing the set theoretic paradigm POCS has been one of the most extensively studied set theoreticapproaches in the literature due to its convergence properties and flexibility to handle a wide range

of signal characteristics We limit our discussion here to POCS–based algorithms The more generalcase of signal estimation using nonconvex as well as convex sets is covered in [21] The rest of thissection will focus on defining the POCS framework and describing several useful algorithms that fallinto this general category

25.4.1 The POCS Framework

A projection operator onto a closed convex set is an example of a nonlinear mapping that is easilyanalyzed and contains some very useful properties Such projection operators minimize error dis-tance and are nonexpansive These are two very important properties of ordinary linear orthogonalprojections onto closed linear manifolds (CLMs) The benefit of using POCS for signal restoration