New strategies for generating panoramic images for imperfect image series 2

Given a series of photographs of a real world scene taken from different but overlapping view points, image stitching techniques attempt to geometrically align the image series and then

Constructing panoramic images from a series of overlapping input images is well-studied topic in the computer vision and computer graphics research communities The work has reached a level of maturity that there are now several commercially successful editing tools like Adobe Photoshop [50] and smart phone applications such as AutoStitch [1] as well as other established academic tools such as Microsoft ICE [2] The successful proliferation of these image stitching tools may lead to the impression that image stitching is solved, but, in fact, many tools fail to give convincing results when given non-ideal data

The goal of image stitching can be described as follows Given a series of photographs of a real world scene taken from different but overlapping view points, image stitching techniques attempt to geometrically align the image series and then map them onto a common canvas resulting in single wide-field-of-view panorama (see Figure1.1) This provides a way to capture very wide-field-of-views without

CHAPTER 1 Introduction

Figure 1.1: An example of image mosaicing technique The panorama is

construct-ed by using the four images on the top

the need for specialized lenses and high resolution single image capture

When constructing panorama images, not any arbitrary pair of images with overlapping view points are able to be stitched together Traditional image stitching methods have certain requirement for the input images The vast majority of methods rely on perspective planar transformations (also called homographies)

to align the images This type of transformations is valid under two restrictive imaging conditions: one is that the camera must be strictly rotated about its center

of projection; the other condition is that the target scene is planar or far away enough that it can be treated as planar When these two conditions are violated, the images cannot always be stitched due to the parallax effect Figure 1.2 gives

an illustration of these scenarios From Figure1.2(a, b) we can see that all target objects can be projected onto a single virtual plane, while Figure 1.2(c,d) show

(a) Camera is rotated along its projective center

(c) Parallax effect (project object onto

different position)

(b) All objects are lying on the same plane

ν

(d) An example of image pairs with parallax

Figure 1.2: An illustration of traditional photo taking assumption for constructing panorama (a) and (b) show two valid assumptions for image mosaicing We can see that any point on the object can be projected onto a unique position on the virtual plane ν Intuitively, each single image the camera takes can be considered

as partial content on the virtual plane On the contrary, as illustrated in (c), the object is projected onto different positions on ν due to the moving of camera Thus, although the images are targeting on the same objects, they cannot be fully registered due to the parallax (d) shows a real example of parallax effect

an example of the parallax effect Both of the two input image assumptions are not easy to achieve by using a hand-held camera in daily imaging As a result, noticeable artifacts usually exist in the stitched results

In this thesis, we use the term “imperfect image series” to indicate an input image series which was not taken under ideal assumptions In practice, most input series are imperfect As a result, image stitching approaches strive to provide the most optimal alignment, where the notion of goodness is measured by how well

CHAPTER 1 Introduction

matched feature points between overlapping images align Since imperfect im-age series violate the imaging assumptions, even under the geometrically optimal alignment there will still be misaligned regions resulting in undesirable visual arti-facts This problem is so well known that virtually all image mosaicing techniques employ a post-processing step to hide the artifacts after transforming the images This post-processing step is either in the form of image blending or most recently performed by estimating a seam-cut between overlapping images that minimizes perceptual artifacts

In fact, image mosaicing algorithms have become so reliant on this post-processing stage that it is now accepted that the panoramic construction process is

a two step procedure: i.e first, perform alignment; second, hide misalignment arti-facts It is worth noting that under this two step strategy, the goal is not to produce

a geometrically accurate panorama, but instead to produce a perceptually seamless panoramic image While the post-processing is critical in producing a final output, these techniques are limited and cannot always remove all visual artifacts The aim

of the work in this thesis is to revisit the traditional image mosaicing pipeline and develop new strategies for stitching imperfect image series which provide more visually appealing result than the existing works

Research work on image mosaicing for consumer level camera arose in mid 1990s [44,

65,20], and get further development in the past decade [70,15,16,61] As we dis-cussed in Section1.1, for a typical image mosaicing pipeline, two distinctive steps are usually applied in tandem to generate the final result: The first step is to

com-Step1: Register and warp images Step2: Seam-cut to provide smooth stitches Final Result Input Images

Image Mosaicing Pipeline

Seam-cut result Final result

Figure 1.3: An illustration of traditional image mosaicing pipeline

pute the geometry transformation to align the images and the second step is to apply a post-processing to achieve a seamless composite result Figure1.3 shows

an illustration of the traditional image mosaicing pipeline In the first step, clas-sical image mosaicing approaches use a perspective planar transform, as called a homography, as the transformation between each pair of image This is because the camera motion can be fully modeled by using a series of 3 × 3 matrices when the input images are taken under the ideal assumptions A homography can be estimated by using a set of reliable registering correspondences inside image pairs The purpose of the post-processing step is to hide the misalignment artifacts To achieve such goal, blending techniques [17, 16, 49, 36] that provide smooth tran-sition between image pairs have been applied in early days In recent years, a seam-cut approach [38,3], which stitches the images by finding the least noticeable seam between image pairs has been adopted in many works [50, 2] Figure 1.4 shows an example of traditional image mosaicing We can see that the seam-cut approach is able to remove most of the artifacts when the overlapped images are misaligned Chapter 2provides technical details of some of these techniques for traditional image mosaicing systems

CHAPTER 1 Introduction

Step1: Register and warp images Step2: Seam-cut to provide smooth stitches Final Result Input Images

Image Mosaicing Pipeline

Seam-cut result Final result

Figure 1.4: An example result of image stitching result using traditional image mosaicing pipeline

As we can see, traditional image mosaicing system which is established on the homography model is not tailored for imperfect image series stitching We believe this system can be further ameliorated to generate better stitched result for imperfect images

The goal of our work is to attempt to construct visually plausible panoramas from input images that violate the conventional imaging assumption That is, we want to improve upon the current image mosaicing processing given an imperfect

image series While our work cannot correct all possible imperfect image series,

we have examined the current image stitching pipeline and found several places where contributions can be made Specifically, we identified the following issues

to address:

• Virtually all state-of-the-art image stitching softwares [1,50,2] use 3 × 3 ho-mographies to model the transformations between pairs of the input images

We examine how allowing a more flexible warping method can improve the results for particular scenes

• Traditional image mosaicing first computes an alignment based on the best geometrical fit of match points However, given that the goal of image mosaic-ing is to produce a perceptually seamless result over a geometrically correct result for an imperfect image series, we consider how promoting the role of the seam-cut step may be beneficial in selecting he alignment, i.e a sub-optimal geometric alignment may provide a better perceptual seam-cut

• In some cases, an imperfect input series cannot be stitched together in an automatic fashion without some noticeable artifacts In such cases, we want

to explore how to design an editing framework to expediate users in making manual corrections

In this thesis, three distinct works are proposed which aim to provide solutions to the issues in Section1.3 These three works correspond to Chapters 3,4, 5in this thesis Figure 1.5 shows an illustration on which part of the conventional image

CHAPTER 1 Introduction

Dual-Homography Warping

Using dual-homography to

provide better registering

and alignment result

Seam Driven Stitching

Exploit the performance

of the seam-cut to find a better warp

Interactive Correction

Enable user to mani-pulate the result after the automatic system

Step1: Register and warp images

Step2: Seam-cut to provide smooth stitches Final Result Input Images

Image Mosaicing Pipeline

Figure 1.5: An illustration of our works with corresponding targeting part of the traditional image mosaicing pipeline

mosaicing pipeline each work targets Specifically, the contributions of these works can be summarized as follows:

• A dual-homography framework is proposed that focuses on the imperfect image series stitching problem in the image alignment stage A new registra-tion model is developed for the case which the target scene has two dominant planes A smoothly varying homography interpolation method is developed

to achieve more accurate alignment between the image pairs and is extended further to multiple images Results show our framework can generate more visually appealing results than existing commercial softwares This work has been published in CVPR’2011 [27]

• A seam-driven image stitching system is introduced that targets how the ge-ometric transformation is selected In particular, instead of selecting

homo-graphies based on the best geometric fit of matched feature points, potential transforms are evaluated based on the perceptual quality of the resulting seam-cut Along with the seam-evaluation pipeline, we propose a simple, yet effective, method to evaluate the seam cuts produced with different trans-forms We demonstrate that this method can produce better results than current state-of-the-art methods This work has been published as a short paper in EuroGraphics’2013 [34]

• Finally, a software for interactive editing of flawed stitched panoramas is proposed Specifically, we have developed a framework that allows the user

to locally correct the visual artifacts arising from both alignment and/or a bad seam-cut Our tool allows the user to locally recompute the seam-cuts based

on simple user markup In addition, we provide a content-aware local warp tool that helps the user find the best matching for two overlapping layers while warping We demonstrate that these editing tools can significantly speed up the manual editing time of flawed panoramas over conventional image-editing tools This work has been published as a technical brief in SIGGRAPH ASIA’2012 [26]

The rest of this thesis is organized as follows: Chapter 2 provides background

on traditional image mosaicing techniques and gives details to those most related

to our works Chapter 3 presents our dual-homography work targeting more flexible geometric alignment Chapter 4 presents the seam-driven image stitching pipeline and Chapter 5 proposes our interactive tools for post correction of the

CHAPTER 1 Introduction

panorama Finally, Chapter 6 concludes the thesis with a discussions on future research directions