New strategies for generating panoramic images for imperfect image series 1

In fact, many existing image stitching tools depend critically on post-processing routines to hide misalignment artifacts.. We demonstrate several cases where this seam-driven image stit

NEW STRATEGIES FOR GENERATING PANORAMIC IMAGES

FOR IMPERFECT IMAGE SERIES

GAO JUNHONG

NATIONAL UNIVERSITY OF SINGAPORE

2013

NEW STRATEGIES FOR GENERATING PANORAMIC IMAGES

FOR IMPERFECT IMAGE SERIES

GAO JUNHONG (B.Sc., Sichuan University of China, 2008)

A THESIS SUBMITTED FOR THE DEGREE OF

Doctor of Philosophy

in

SCHOOL OF COMPUTING

NATIONAL UNIVERSITY OF SINGAPORE

SINGAPORE, 2013

Declaration

I hereby declare that this thesis is my original work and it has been written by me in its entirety I have duly acknowledged all the sources of information which have been used in the thesis

This thesis has also not been submitted for any degree in

any university previously

To my parents, my wife and my coming son.

First of all, I own my deepest gratitude to my advisor Dr Michael

S Brown for his patient guidance during my Ph.D candidature I

am, and will always be, thankful for all of his encouraging advice, shared experiences and technical/financial support All the things that

he taught me I’ll treasure for the rest of my career

I am thankful to my co-authors: Seon Joo Kim, Tat-Jun Chin and Li Yu for their great contribution to my research work, for all their valuable suggestions and auxiliary works that contributed to this thesis It is always my pleasure to collaborate with these brilliant people and to work on exiting research topics together

I am grateful to my colleagues in the Computer Vision Lab: Lin Haiting, Deng Fanbo, Cheng Dongliang, Cheng Yuan etc , for every colorful day

I spent at the National University of Singapore (NUS) I really enjoyed the life in Singapore with these warmhearted and reliable friends

At last, I would like to thank my parents for their unconditional support and encouragement A special thank to my father, Gao Rongting, who has contributed at least two sets of photo data to this thesis I owe my deep appreciation to my wife, Zhang Lingyan, for her sincere love and all the wonderful things she has done for me May this thesis be my best present for her and our coming baby

The success of commercial image stitching tools often leads to the im-pression that image stitching is a ”solved problem” The reality, how-ever, is that many tools give unconvincing results when the input data violates fairly rigid imaging assumptions; the main two assumptions being that the photos correspond to views that differ purely by rotation

or that the imaged scene is planar In fact, many existing image stitching tools depend critically on post-processing routines to hide misalignment artifacts As a result, the defacto image-stitching pipeline involves two distinct steps: 1) a geometric alignment step; and 2) a post-processing step that computes optimal seam-cuts between overlapping images to hide misalignment artifacts

In this thesis, we re-examine this established panoramic image con-struction pipeline and introduce three strategies that are useful when the input images are less than ideal First, we introduce a method to compute a more flexible geometric alignment when the imaged scene contains two dominant planes We show that this method, termed dual-homography alignment, is able to stitch photographs that current state-of-the-art methods cannot Second, we show that instead of selecting an alignment based on the geometric fit of matched points between images,

we can use the seam-cut step to evaluate several possible geometric

transformations and select the alignment that gives the most “percep-tually optimal” seam We demonstrate several cases where this seam-driven image stitching is able to produce better results than existing methods Lastly, for examples when the automatic image stitching pro-cedure fails, we introduce an interactive method that provides the user tools to manually edit the seam-cuts and/or geometric alignment in a local manner These three works collectively provide solutions for im-age stitching using imperfect input imim-ages while targeting on different parts of the image stitching pipeline

List of Figures vii

List of Tables ix

1 Introduction 1 1.1 Motivation 1

1.2 Current Image Mosaicing Process 4

1.3 Objectives 6

1.4 Contributions 7

1.5 Road Map 9

2 Background and Preliminaries 11 2.1 Overview 11

2.2 Feature Registering 12

2.2.1 SIFT Feature Matching 14

2.2.2 RANSAC Process 14

2.3 Transformation Estimation 17

2.3.1 Homography Model Estimation 17

2.3.2 Rotation Model Estimation 18

2.3.3 Cylinder Mapping 19

2.4 Post-Processing Techniques 21

2.4.1 Blending 21

2.4.2 Seam-cut 22

2.5 Challenges for Imperfect Image Stitching and Its Related Works 27

2.6 Summary 31

i

3 Dual-Homography Image Stitching 33

3.1 Introduction 33

3.2 System Overview 35

3.3 Dual-Homography Alignment 36

3.3.1 Dual-Homography Estimation 36

3.3.2 Extending to Multiple Images 43

3.4 Post-Processing the Mosaic 45

3.4.1 Global Straightening 45

3.5 Results 50

3.6 Summary 55

4 Seam-Driven Stitching 57 4.1 Introduction and Motivation 57

4.2 Seam-Driven Image Stitching 58

4.2.1 Generating Homography Candidates 60

4.2.2 Computing the Seam-Cut 61

4.2.3 Evaluating the Cut 61

4.3 Results and Discussion 63

5 Interactive Post-Processing Tools for Correction 67 5.1 Introduction 67

5.2 System 69

5.2.1 Pipeline 69

5.2.2 Local seam-editing 70

5.2.3 Content-aware snapping 72

5.3 Results 74

5.4 Summary 78

6 Conclusion 81 6.1 Assessment 81

6.2 Discussion and Limitations 84

6.3 Future Work 85

A Intermediate Result of Seam Driven Stitching Approach 87

ii

B Interface of the Interactive Panorama Correction Tool 91

C Input Images for All Examples used in This Thesis 95

iii

iv

List of Figures

1.1 An example of panorama construction 2

1.2 An illustration of traditional photo taking assumption for construct-ing panorama 3

1.3 An illustration of traditional image mosaicing pipeline 5

1.4 An example result of image stitching result using traditional image mosaicing pipeline 6

1.5 An illustration of our works with corresponding targeting part of the traditional image mosaicing pipeline 8

2.1 A comparison of the performance of different feature descriptors 13

2.2 An example of SIFT feature detection and RANSAC filtering 15

2.3 An example of homography warping and cylinder warping 20

2.4 An example of post-processing techniques of image mosaicing Multi-band blending and seam-cut results are compared 23

2.5 An illustration of seam-cut theory 24

2.6 Seam cut example with blending 25

2.7 An illustration of graph cut labeling 27

3.1 A scene containing two dominant planes targeted by our mosaicing approach 34

3.2 Work flow of our dual-homography computation 37

3.3 A synthesized example of an ideal scene of two plains and its cor-respondent results of two homographies with explicit segmentation method and dual-homography method 40

v

LIST OF FIGURES

3.4 A synthesized example of a more realistic scene of two plains with stair structures in the middle and its correspondent results of

t-wo homographies with explicit segmentation method and

dual-homography method 41

3.5 Illustration of concatenate multiple images 43

3.6 An example of content-aware straightening 47

3.7 An example where the user can specify regions that should remain intact in the straightening process 50

3.8 A failure case of dual-homography 51

3.9 A comparison of our dual-homography stitching approach with commercial softwares ]1 52

3.10 A comparison of our dual-homography stitching approach with commercial softwares ]2 53

3.11 A comparison of our dual-homography stitching approach with commercial softwares ]3 54

4.1 An illustration of our approach and traditional image stitching ap-proach 58

4.2 A comparison between the traditional image stitching process and our seam-driven process 59

4.3 An illustration of randomly generated homographies with respec-tive seam cuts 61

4.4 An illustration of seam-cut evaluating 62

4.5 A comparison of energy for different seam cut 63

4.6 Comparison of panoramas constructed based on a traditional image stitching pipeline (Adobe Photoshop CS6) and our method 65

5.1 An example of the usage of our interactive tool 68

5.2 Overview of our approach 69

5.3 Illustration of seam adjustment 71

5.4 Illustration of content-aware warping 72

5.5 A comparison example with other softwares 74

5.6 Example results generated by our interactive approach(]1) 75

5.7 Example results generated by our interactive approach(]2) 76

vi

LIST OF FIGURES

5.8 Example results generated by our interactive approach(]3) 77

5.9 User-study result of comparing our panorama correction tool and Photoshop CS5 78

5.10 An example where Photoshop (and AutoStitch and ICE) fails to generate a panoramic image 79

A.1 Input images used in Chapter 4 87

A.2 Intermediate results for image set]1 88

A.3 Intermediate results for image set]2 89

B.1 The control dialog for the interactive panorama correction tools 91

B.2 An example of the display window in seam-editing mode 92

B.3 An example of the display window in warp mode 93

C.1 Input image series for Figure 3.9 95

C.2 Input image series for Figure 3.10 95

C.3 Input image series for Figure 3.11 96

C.4 Input image series for Figure 5.1 96

C.5 Input image series for Figure 5.6 96

C.6 Input image series for Figure 5.7 96

C.7 Input image series for Figure 5.8 97

vii

LIST OF FIGURES

viii

List of Tables

2.1 An illustration of different types of 2D motions 18

3.1 Pros and cons of the compared two homographies with

segmenta-tion method and dual-homography method 42

5.1 Processing time of the examples in Figure 5.6, 5.7, 5.8 75

ix

LIST OF TABLES

x