New strategies for generating panoramic images for imperfect image series 5

This led us to consider promoting the role of the seam-cut in a method we term seam-driven image stitching.. Instead of selecting homographies between two image pairs based on the best g

Seam-Driven Stitching

4.1 Introduction and Motivation

Our previous chapter focused on addressing image mosaicing in the face of imper-fect image images by allowing for more flexible warping This approach worked well for scenes with two dominate planes, however, there are many cases where a single homography can align large portions of the image, however, due to parallax there are still artifacts In such cases, artifact hiding using the post-processing step based on seam-cutting becomes very crucial Matter of fact, our observation is that at times the seam-cutting step may be the most crucial step This led us to consider promoting the role of the seam-cut in a method we term seam-driven image stitching Instead of selecting homographies between two image pairs based on the best geometric fit to matched feature points, a set of potential transforms are evaluated based on the perceptual quality of the resulting seam-cut The idea is that a potential homography may give a better perceptual seam-cut even though the transformation is considered sub-optimal Figure 4.1 shows an example We

(A) (B)

Figure 4.1: (A) A traditional image-stitching result where the images are first aligned using a geometric transform computed based on the best overall fit of 96 matched points followed by seam-cutting (B) Our seam-driven image stitching result where the geometric transform is less optimal in terms of geometric fit (only 26 matched points used), but the transform gives a perceptually better seam-cut The insets show the points used to compute the transforms as well as the seam-cut with our computed seam error

demonstrate that this approach can sometimes produce better results than current state-of-the-art methods, especially for challenging cases where the scene geometry

or imaging conditions are not ideal

4.2 Seam-Driven Image Stitching

Figure 4.2shows an example of the traditional image stitching pipeline (top) and our modification to perform seam-driven image stitching (bottom) In the tradi-tional method, feature points such as SIFT [42] are computed and matched between image pairs resulting in a set of feature correspondences Due to the non-planar scene geometry and erroneous matches, random sample consensus (RANSAC) [24]

is used to robustly estimate the homography with the best geometric fit Specif-ically, a set of homographies H1, H2, are randomly hypothesized and ranked based on their goodness-of-fit (measured in terms of consensus size) The best

Traditional image stitching

𝐻1 𝐻2 𝐻3

𝐻4 𝐻5

Feature

extraction and

matching Robust estimation to find

candidate homographies

Apply seam-cut Final result

Apply seam-cut to all

possible transformations

Choose H with

best geometric fit

Seam-driven image stitching

Final result

Choose result with best seam-cut

Feature

extraction and

matching

𝐻1 𝐻2 𝐻3

𝐻4 𝐻5

Robust estimation to find candidate homographies

Figure 4.2: A comparison between the traditional image stitching process and our seam-driven process

homography is used to warp and align the images Seam-cutting is then applied

to produce the final result

Our seam-driven approach operates in a similar fashion but evaluates and selects the homography differently This is achieved by modifying the hypothesize-and-test loop in RANSAC, whereby we generate candidate homographies in a manner that encourages diversity, and applying seam-cut to the aligned images by using all candidates Next, we evaluate the different seam-cuts based on our novel ranking metric that favors results with good perceptual quality The details to our approach are outlined in the following sections

4.2.1 Generating Homography Candidates

RANSAC generates model hypotheses by fitting a homography on randomly sam-pled minimal subsets of the data, where each subset contains four correspon-dences [16] For each hypothesis, the correspondences that are consistent with it (i.e the inliers of the hypothesis) are recorded This random sampling process is repeated n times and the hypothesis with the highest consensus size is chosen The number n rarely corresponds to an exhaustive sampling of all four-subsets

of the data Moreover, usually there is a dominant plane in the scene (e.g the background plane) that contributes a large number of similar candidate homogra-phies Traditional image-stitching methods favor the dominant plane as it gives the best consensus Since we desire more diverse homographies (to yield more varied seam-cuts), we use a simple filtering strategy to avoid resampling the dom-inant plane For each iteration, RANSAC instances each with a smaller n (50 in our implementation) is applied on the current matched points set Such a small replication number lowers the probability that the RANSAC process keep retriev-ing the dominant homography of the current matchretriev-ing point set After this, we remove the inliers of the dominant plane which are achieved by applying standard RANSAC with a comparatively large n (50 in our implementation) This process repeats until the number of points in current data set is less than a thresholdτ (10

in our system) This helps to find homographies that align other smaller planar structures which may result in better seam-cuts

Figure 4.3: An illustration of randomly generated homographies with respective seam cuts

4.2.2 Computing the Seam-Cut

We used the seam-cut method described in Section2.4.2to stitch each neighboring images For each image pair with the candidate homography, we denote I1and I2

as the overlapping regions Then the seam-cut is a global labeling problem on an MRF and can be solved by using the work in [13]

4.2.3 Evaluating the Cut

While the seam-cut energy minimizes image gradient between I1 and I2, it is inef-fective when used to rank the perceptual quality of different cuts We introduce

a more discriminating error measurement For each pixel, p, along the seam, we

low high

𝐸 = 9.706𝑒−4 𝐸 = 5.365𝑒−3 𝐸 = 6.150𝑒−3

Seam-cut result

Search for the most similar patch in the overlapping region of source images

s

-

s

-

s

I1

s

Figure 4.4: This illustrates how seams are evaluated For each pixel along the seam,

we search for the most similar patch about this pixel from either of two overlapping image and use the squared difference as the mismatch error

estimate an error value, E(p) by extracting a 17 × 17 patch, P, centered at p, and searching for its most similar patch in either I1or I2 This can be expressed as:

E(p)= min

S i∈I 1 ,I 2

where Si represents all image patches in the overlapping regions of I1 and I2 The idea here is that if the patches along the seam resemble a patch found in either

I1 or I2, the patch about the seam is perceptually plausible If a patch along the seam cannot be found in either source images, it is more likely to be an artifact and therefore assigned a larger error The total error along a seam of size m, is computed as E= Pn

pE(p)/m

4.3 Results and Discussion

low

high

𝐸 = 9.706𝑒−4 𝐸 = 5.365𝑒−3 𝐸 = 6.150𝑒−3

Search for the most similar patch in the overlapping region of source images

s

-

s

-

s

Figure 4.5: This comparison shows a hot map of energy from different cuts using our error metric The accumulated seam error, E is used to rank the cuts

Figure 4.4 and Figure 4.5 shows a diagram of this procedure as well as an example with three different cuts with the corresponding per pixel error, E(p) shown as a hot map and the total error E The minimum error is indicative of our perceptual ranking

4.3 Results and Discussion

Figure 4.6 shows our results and those obtained using and Photoshop CS6 on a panorama sequence of four input images The images contain parallax and there-fore are difficult to align Photoshop uses a traditional image-stitching pipeline that selects a best fit transform There are still noticeable artifacts even after seam-cutting In contrast, our seam-driven strategy is able to obtain perceptually better results (also see supplemental material) Noticeably, inside our results there exist

artifacts which are not geometrically corrected (e.g duplicated mountain texture

in the first example and omitting trees in the second example) However, these artifacts are perceptually not noticeable Compare to the result generated by Pho-toshop [50], our results are more visually appealing Detailed intermediate results

of Figure4.6are provided in AppendixA

This work have present a new sight of image mosaicing approaches by let the post-processing technique determine the transformation As demonstrated, our seam driven image stitching work can generate better result for imperfect shot images Limitations and potential future work related to this seam-drive image stitching method are presented in Chapter6

Our result

Photoshop result

Our result

Photoshop result

Figure 4.6: Comparison of panoramas constructed based on a traditional image stitching pipeline (Adobe Photoshop CS6) and our seam-driven approach There are noticeable visual artifacts in the Photoshop’s results, while our results appear more seamless