New strategies for generating panoramic images for imperfect image series 4

Due to the parallax effect, the traditional image mosaicing pipeline whichuses single homography transformation model usually results in misalignments inthe stitched images.. As demonstr

Dual-Homography Image Stitching

3.1 Introduction

As presented in the previous chapters, although image mosaicing has received agreat deal of attention over the years, stitching a sequence of images that do notmeet the imaging assumptions (i.e an imperfect images series) is still a challengingproblem Due to the parallax effect, the traditional image mosaicing pipeline whichuses single homography transformation model usually results in misalignments inthe stitched images As previously discussed, these misalignments are then hiddenvia post-processing techniques, such as image blending or seam cutting, to varyingdegrees of success, often producing mosaics that appear seamless at a quick glance,only to reveal breaks and tears on closer inspection

In this chapter, we focus on one common type of scene that is particularlytroublesome for existing methods: a scene containing both a distant plane and a

illustration of this type of scene There are two main reasons that we choose this

Distant Plane

Ground Plane

Figure 3.1: A scene containing two dominant planes targeted by our mosaicingapproach

specific scene as our input series Firstly, this type of scene is commonly found

at tourist locations where panoramic imaging is desired When the photographer

is standing on a flat ground and targeting on the landscape far away, the takenimage series usually contains these two dominant planes The other reason is thatthe parallax of this type of scene is between this two continuous plane Withoutabrupt structures that cross the scene, it is possible to smoothly align these twoplanes respectively and leave the misalignment to the unnoticeable part of the jointregion

The work presented here describes how to address this type of panoramicscene before the post-processing stage by focusing on how to improve the imagealignment This is achieved by estimating two homographies per image-pair and

blending these homographies to align the images As demonstrated by our results,this approach can produce more seamless results than those obtained using currentstate-of-the-art methods that rely on a single homography per image pair While

a seemingly straight-forward idea, given that our scene has two dominate planes,there are several issues addressed in this chapter that must be considered to makethis approach work These issues include estimating the dual homographies frommatched points, weights for blending the homographies, extending the nonlinearwarping to concatenate adjacent images, and post-processing to blend the imagesand to reduce undesired curvilinear artifacts introduced by the nonlinear warping

is applied to the aligned result as a post-processing step which tries to removethe misalignment artifacts Unique to our system, since the dual-homographymay introduce curvilinear artifacts due to the interpolation of two homographies,

a content-aware straightening process is applied to achieve a visually appealing

result which remove these artifacts Based on the challenges of each step in ourdual-homography pipeline, the following of this chapter is organized as follows:Section3.3 describes our dual-homography estimation and blending for an inputimage pair and its extension for multiple image concatenation; Section 3.4 de-scribes post-processing procedures to reduce visual seams and rectify curvilinearartifacts; Section 3.5 demonstrates results obtained using our approach followed

by a discussion and summary of our work in Section3.6

3.3 Dual-Homography Alignment

We first describe how to compute the two homographies for a pair of ping images, and show how to use the dual-homographies to perform nonlinearalignment We then describe how to extend this warping to adjacent images in themosaic For the sake of simplicity, we assume that each image in our panoramahas at most two overlapping regions with images on either side of its borders

overlap-3.3.1 Dual-Homography Estimation

The idea of blending homographies is to find a smooth interpolating warping field

to register the pair of input images As we know, for the objects which are lying onthe same plane in the target scene, a single homography model can represent thetransformation between the planes Thus, to register two dominant planes insidethe scene with parallax, a natural solution is to compute a homography for eachplane and assign the ambiguous regions a smooth interpolating transformations.Following this idea, our approach models the relationship between two overlap-

ping images in our panorama using a blending of two homographes, Hg and Hd,

Distant plane homography Hd Ground plane homography Hg

Input image pair and clustered match points

Dual homography Hij = ω ij Hg+(1-ω ij)Hd

Weight map ωij

Figure 3.2: Work flow of our dual-homography computation Candidate pointsare first clustered into groups from which two homographies are estimated usingRANSAC A per-pixel weight map ωi jis computed to control the blending of thetwo homographies

expressed as:

respectively, and ωi j is a per pixel weight that controls the contribution of eachhomography at the pixel location (i, j)

As described in the Chapter2, our approach begins with a set of correspondingpoints matched between the input images We use SIFT features [42] to establishcorrespondences as done in [16] Since we are computing two homographies per

image pair, our first step is to cluster the correspondences into two groups Gg

and Gd based on their spatial location in the image This is done using K-meansclustering, with 2D seed points taken to be:

homography Hgand Hdis considered as the transform of its corresponding group

of features After RANSAC is performed, outlier SIFT features are removed to form

new groups G0g and G0d.

The two resulting homographies can partially align the image as shown inFigure3.2 The goal now is to assign a weight to each pixel location pi jto determinehow much of each homography should be used A natural choice is to weight thewarping based on spatial proximity, i.e.:

where dgand ddare the inverse squared distances (||·||−2

2 ) to the closest feature points

in the sets G0g and G0drespectively Figure 3.2shows an example of this weightingmap and results from this weighted dual-homography warping

Dual Homography vs Two Homographies with Explicit Segmentation Afterthe dual-homography stitching model is proposed, a straight-forward question can

be raised up: can we use two homographies as the transformation model for eachdominant plane, and give the scene a explicit segmentation? Figure 3.3 shows asynthesis case of a scene which contains exactly two explicit dominant planes Boththe dual homography result and the two homography result are provided Fromthe result we can see that, in ideal cases, since the accurate segmentation betweenthe two planes is known, the two homography result can achieve a fully alignmentfor each plane, while our dual homography method have mis-alignment artifacts

in the middle “vague” region of the scene This is because our dual homographymethod does not know the explicit segmentation of the scene However, thingschange when it comes to a more realistic case Figure 3.4 shows a synthesis ex-ample of a scene that contains stair structures between two dominant planes As

we can see that, these stair structures breaks the assumption of only two dominant

(a) Synthesized Input images in a scene with two planes

(b) Stitched result using two homographies with explicit segmentation

(c) Stitched result using dual-homography

Figure 3.3: A synthesized example of an ideal scene of two plains and its respondent results of two homographies with explicit segmentation method anddual-homography method

cor-(a) Synthesized Input images in a scene with two planes and stair structure in the middle

(b) Stitched result using two homographies with explicit segmentation

(c) Stitched result using dual-homography

Figure 3.4: A synthesized example of a more realistic scene of two plains with stairstructures in the middle and its correspondent results of two homographies withexplicit segmentation method and dual-homography method

Pros:

- Accurate registration in ideal case

- No curvilinear artifacts in ideal case

- Not geometrically correct

Two homographies with segmentation Dual Homography

Pros

- Accurate alignment in ideal case

- No curvilinear artifacts in ideal case

- Do not need the explicitly

segmentation

- Seamlessly blends in non-ideal case

Cons

- Need to perform explicit segmentation

- Breaks for more complex geometry

- Curvilinear artifacts

- Not geometrically correct

Table 3.1: Pros and cons of the compared two homographies with segmentationmethod and dual-homography method

planes In such case, the composite results of both two homography with explicitsegmentation result and our dual homography result will have misalignment at theoverlapping region However, for non-overlapping region, since the stair structure

do not belongs to either of the predominant planes, the two homographies withsegmentation method will have this noticeable breaking artifacts at the joint ofthe segmentation The dual-homography method on the other hand, provides asmooth interpolation in this region Table 3.1 has lists the pros and cons of thesetwo method From the table we can see that, although using two homographieswith segmentation has the advantages that it can achieve accurate alignment inideal case, it simply breaks when the underlying geometry is more complex Fur-thermore, another main drawback of the two homographies with segmentationmethod is that it requires explicit segmentation process Therefore, we believe thatour dual-homography method is more practiced

Figure 3.5: This figure illustrates how to concatenate multiple images into thevirtual image plane, IV using dual-homographies Refer to the text in Section 3.2for details.

3.3.2 Extending to Multiple Images

Unlike single homography approaches that can easily concatenate multiple graphies into a single matrix, our nonlinear warping requires more care Figure3.5

homo-is used to help illustrate the concatenation procedure using three overlapping ages I0, I1, and I2 that we wish to map to the virtual mosaic image plane IV I0 isused as the root image of the mosaic and undergoes no warping

im-The first dual-homography used to map I1 to I0 is denoted as H1→0, droppingthe subscripts, i j, for clarity This dual-homography is computed as previous-

ly described and is the only mapping required to place I1 in the virtual image:

i.e H1→V = H1→0

The task now is to add another dual-homography pair, H2→V, that maps I2

to I1 in the virtual image plane This is achieved by first computing the

dual-homography, H2→1, to aligned I2 to I1 This is computed as described in the priorsection Note that this estimation is done agnostic of the relationship between I1

and I0 However, in the virtual image, IV, image I1is expressed as H1→0(I1) Thus,

in the overlapped region between I2 and I1, we can relate I2 to the virtual image by

H1→0(H2→1(I2))

The question now is what do we do about the non-overlapped region of I2with

homogra-phy can be applied to points outside the overlapping region, our warping function

H1→0 is not defined for points outside I1 To ‘virtually extend’ the reach of H1→0,

we use a weighted blending of the boundary points of H1→V, resulting in:

whereΩB is the boundary of I1that overlaps with I2 The scalarζq = 1/kH2→1(p) −

qk2 is computed using the Euclidean distance from point p to all the boundary points q on ΩB Note that these weights are normalized such that P ζ q = 1

While this procedure has been illustrated using only three images, it is forward to derive a recursive expression from Eq 4 to concatenate further images

straight-3.4 Post-Processing the Mosaic

This section describes two post-processing techniques, seam blending and ening, that are used to enhance the aesthetic appearance of our final results.Seam-blending aims to reduce the visible seams between overlapping images, thismethod was previously described in Section 2.4.2 Content-aware straightening

straight-is unique to our mosaicing procedure and straight-is applied to reduce curvilinear effectsintroduced by the dual-homography warping

3.4.1 Global Straightening

The linear interpolation of the homographies makes the images warp in a quadraticfashion This can introduce a curved effect that may be aesthetically unappealingfor the final result

To remove this effect, we employ a content-aware warping method similar tothat used in [69, 71] In particular, our final virtual image, IV is tesselated into a

polygon mesh, where each unit, Q, is a quad This is done by firstly tessellating

each input image into a unit square mesh For each of the vertices inside thesemeshes, we compute the position that after dual-homography warping is applied.After that, for each image’s mesh we collect all the vertices that have a neighboringvertex lie in the other image (which means these vertices are along the seam cut).Then line up these vertices into a polyline Similarly, on the other side of theseam there would be another polyline Connect these two polylines, a strap-likepolygon is achieved At last, the two meshes are stitched together by triangulatingthis strap-like polygon An example of this polygonal mesh is shown in Figure3.6

Each polygon is composed of vertices, vi, that can be deformed to a new set of

polygons, Q0, with the following two distortion energies:

This energy attempts to constrain the deformation on all the points visuch that the

polygons, Q, only undergo a similarity transformation.

The other distortion energy is according to the observation in [16] that peoplerarely twist the camera relative to the horizon when shooting the panorama images

Hence, we give each point pair v1and v2in the vertical edge E a bending cost energy

Here, the weight µ(Q)

s allows this deformation to be flexible This weight is

Figure 3.6: Before and after the application of content-aware straightening.

determined by the underlying image content contained within each polygon, Q,

higher image energy to resist deformation, while polygons with lower energy can

be deformed more To enforce the vertical edge constraint, the weightµl we give

a comparatively large number which equals to 20 times of max{µ(Q)

s } to force thelines straight

This approach is similar to that proposed by [69, 71] for content-aware imageresizing, however, we have utilized it in a completely different manner, exploitingits ability to linearize the mesh using Equation3.8 This system is directly computedusing an over-determined linear solver Figure 3.6 shows examples of our resultbefore and after this straightening procedure

Closed-form solution for Equation 3.8 According to Equation3.6, for each

poly-gon, Q = {vi(xi, yi)|1 ≤ i ≤ n} (n = 3 for triangles and n = 4 for s) , it undergoes a similarity transformation s(·) which contains four parameters

quadrilateral-xs = [c d tx ty]T Inside these parameters, c and d are the general rotation andscale parameters and tx, tyrepresent the translations of x and y direction Thus, for

each vertex in Q, we have

y0 1

x0 n

y0 n

As we can see, AQis a 2n × 4 matrix which contains all the position information

of the vertices in Q, and y Q that is the desired target coordinates of the vertices

after warping Then the optimal xs can be substituted in closed form by solving

AQ xs = y0

Q in a least square sense: xs = (AT

QAQ)−1ATQ y0Q Now the error definitioncan be written as: