Ellips fitting for computer vision

Ellipse Fitting for Computer Vision Implementation and Applications Kenichi Kanatani Yasuyuki Sugaya Yasushi Kanazawa Series Editors: Gérard Medioni, University of Southern California Sv

Ellipse Fitting for Computer Vision

Implementation and Applications

Kenichi Kanatani Yasuyuki Sugaya Yasushi Kanazawa

Series Editors: Gérard Medioni, University of Southern California

Sven Dickinson, University of Toronto

Ellipse Fitting for Computer Vision

Implementation and Applications

Kenichi Kanatani, Okayama University

Yasuyuki Sugaya, Toyohashi University of Technology

Yasushi Kanazawa, Toyohashi University of Technology

Because circular objects are projected to ellipses in images, ellipse fitting is a first step for 3-D analysis

of circular objects in computer vision applications For this reason, the study of ellipse fitting began as

soon as computers came into use for image analysis in the 1970s, but it is only recently that optimal

computation techniques based on the statistical properties of noise were established These include

renormalization (1993), which was then improved as FNS (2000) and HEIV (2000) Later, further

improvements, called hyperaccurate correction (2006), HyperLS (2009), and hyper-renormalization

(2012), were presented Today, these are regarded as the most accurate fitting methods among all known

techniques This book describes these algorithms as well implementation details and applications to 3-D

scene analysis

We also present general mathematical theories of statistical optimization underlying all ellipse fitting algorithms, including rigorous covariance and bias analyses and the theoretical accuracy limit

The results can be directly applied to other computer vision tasks including computing fundamental

matrices and homographies between images This book can serve not simply as a reference of ellipse

fitting algorithms for researchers, but also as learning material for beginners who want to start computer

vision research The sample program codes are downloadable from the website: https://sites.google

com/a/morganclaypool.com/ellipse-fitting-for-computer-vision-implementation-and-applications

store.morganclaypool.com

About SYNTHESIS

This volume is a printed version of a work that appears in the Synthesis

Digital Library of Engineering and Computer Science Synthesis

books provide concise, original presentations of important research and

development topics, published quickly, in digital and print formats

Ellipse Fitting for Computer Vision

Implementation and Applications

Synthesis Lectures on

Computer Vision

Editors

Gérard Medioni, University of Southern California

Sven Dickinson, University of Toronto

Synthesis Lectures on Computer Visionis edited by Gérard Medioni of the University of SouthernCalifornia and Sven Dickinson of the University of Toronto e series publishes 50- to 150 pagepublications on topics pertaining to computer vision and pattern recognition e scope will largelyfollow the purview of premier computer science conferences, such as ICCV, CVPR, and ECCV.Potential topics include, but not are limited to:

• Applications and Case Studies for Computer Vision

• Color, Illumination, and Texture

• Computational Photography and Video

• Early and Biologically-inspired Vision

• Face and Gesture Analysis

• Illumination and Reflectance Modeling

• Image-Based Modeling

• Image and Video Retrieval

• Medical Image Analysis

• Motion and Tracking

• Object Detection, Recognition, and Categorization

• Segmentation and Grouping

• Sensors

• Shape-from-X

• Stereo and Structure from Motion

• Shape Representation and Matching

• Statistical Methods and Learning

• Performance Evaluation

• Video Analysis and Event Recognition

Ellipse Fitting for Computer Vision: Implementation and Applications

Kenichi Kanatani, Yasuyuki Sugaya, and Yasushi Kanazawa

Camera Networks: e Acquisition and Analysis of Videos over Wide Areas

Amit K Roy-Chowdhury and Bi Song

2012

Deformable Surface 3D Reconstruction from Monocular Images

Mathieu Salzmann and Pascal Fua

2010

Boosting-Based Face Detection and Adaptation

Cha Zhang and Zhengyou Zhang

2010

Image-Based Modeling of Plants and Trees

Sing Bing Kang and Long Quan

2009

Copyright © 2016 by Morgan & Claypool

All rights reserved No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means—electronic, mechanical, photocopy, recording, or any other except for brief quotations

in printed reviews, without the prior permission of the publisher.

Ellipse Fitting for Computer Vision: Implementation and Applications

Kenichi Kanatani, Yasuyuki Sugaya, and Yasushi Kanazawa

www.morganclaypool.com

ISBN: 9781627054584 paperback

ISBN: 9781627054980 ebook

DOI 10.2200/S00713ED1V01Y201603COV008

A Publication in the Morgan & Claypool Publishers series

SYNTHESIS LECTURES ON COMPUTER VISION

Lecture #8

Series Editors: Gérard Medioni,University of Southern California

Sven Dickinson,University of Toronto

Series ISSN

Print 2153-1056 Electronic 2153-1064

Ellipse Fitting for

Toyohashi University of Technology, Toyohashi, Aichi, Japan

SYNTHESIS LECTURES ON COMPUTER VISION #8

C

M

Because circular objects are projected to ellipses in images, ellipse fitting is a first step for

3-D analysis of circular objects in computer vision applications For this reason, the study of lipse fitting began as soon as computers came into use for image analysis in the 1970s, but it

el-is only recently that optimal computation techniques based on the statel-istical properties of noel-isewere established ese includerenormalization (1993), which was then improved as FNS (2000)

andHEIV (2000) Later, further improvements, called hyperaccurate correction (2006), HyperLS

(2009), andhyper-renormalization (2012), were presented Today, these are regarded as the most

accurate fitting methods among all known techniques is book describes these algorithms aswell implementation details and applications to 3-D scene analysis

We also present general mathematical theories of statistical optimization underlying all lipse fitting algorithms, including rigorous covariance and bias analyses and the theoretical accu-racy limit e results can be directly applied to other computer vision tasks including computingfundamental matrices and homographies between images

el-is book can serve not simply as a reference of ellipse fitting algorithmsfor researchers, but also as learning material for beginners who want to start com-puter vision research e sample program codes are downloadable from the web-site: https://sites.google.com/a/morganclaypool.com/ellipse-fitting-for-computer-vision-implementation-and-applications/

KEYWORDS

geometric distance minimization, hyperaccurate correction, HyperLS,

hyper-renormalization, iterative reweight, KCR lower bound, maximum likelihood,

renor-malization, robust fitting, Sampson error, statistical error analysis, Taubin method

vii Contents

Preface xi

1 Introduction 1

1.1 Ellipse Fitting 1

1.2 Representation of Ellipses 2

1.3 Least Squares Approach 3

1.4 Noise and Covariance Matrices 4

1.5 Ellipse Fitting Approaches 6

1.6 Supplemental Note 6

2 Algebraic Fitting 11

2.1 Iterative Reweight and Least Squares 11

2.2 Renormalization and the Taubin Method 12

2.3 Hyper-renormalization and HyperLS 13

2.4 Summary 15

2.5 Supplemental Note 16

3 Geometric Fitting 19

3.1 Geometric Distance and Sampson Error 19

3.2 FNS 20

3.3 Geometric Distance Minimization 21

3.4 Hyperaccurate Correction 23

3.5 Derivations 24

3.6 Supplemental Note 28

4 Robust Fitting 31

4.1 Outlier Removal 31

4.2 Ellipse-specific Fitting 32

4.3 Supplemental Note 34

5 Ellipse-based 3-D Computation 37

5.1 Intersections of Ellipses 37

5.2 Ellipse Centers, Tangents, and Perpendiculars 38

5.3 Perspective Projection and Camera Rotation 40

5.4 3-D Reconstruction of the Supporting Plane 43

5.5 Projected Center of Circle 44

5.6 Front Image of the Circle 45

5.7 Derivations 47

5.8 Supplemental Note 50

6 Experiments and Examples 55

6.1 Ellipse Fitting Examples 55

6.2 Statistical Accuracy Comparison 56

6.3 Real Image Examples 1 59

6.4 Robust Fitting 59

6.5 Ellipse-specific Methods 59

6.6 Real Image Examples 2 61

6.7 Ellipse-based 3-D Computation Examples 62

6.8 Supplemental Note 64

7 Extension and Generalization 67

7.1 Fundamental Matrix computation 67

7.1.1 Formulation 67

7.1.2 Rank Constraint 70

7.1.3 Outlier Removal 71

7.2 Homography Computation 72

7.2.1 Formulation 72

7.2.2 Outlier Removal 76

7.3 Supplemental Note 77

8 Accuracy of Algebraic Fitting 79

8.1 Error Analysis 79

8.2 Covariance and Bias 80

8.3 Bias Elimination and Hyper-renormalization 82

8.4 Derivations 83

8.5 Supplemental Note 90

9 Maximum Likelihood and Geometric Fitting 93

9.1 Maximum Likelihood and Sampson Error 93

9.2 Error Analysis 94

9.3 Bias Analysis and Hyperaccurate Correction 96

9.4 Derivations 96

9.5 Supplemental Note 101

10 eoretical Accuracy Limit 103

10.1 KCR Lower Bound 103

10.2 Derivation of the KCR Lower Bound 104

10.3 Expression of the KCR Lower Bound 107

10.4 Supplemental Note 108

Answers 111

Bibliography 119

Authors’ Biographies 125

Index 127

Because circular objects are projected to ellipses in images, ellipse fitting is a first step for 3-D ysis of circular objects in computer vision applications For this reason, the study of ellipse fittingbegan as soon as computers came into use for image analysis in the 1970s e basic principle was

anal-to compute the parameters so that the sum of squares of expressions that should ideally be zero

is minimized, which is today calledleast squares or algebraic distance minimization In the 1990s,

the notion of optimal computation based on the statistical properties of noise was introduced

by researchers including the authors e first notable example was the authors’renormalization

(1993), which was then improved asFNS (2000) and HEIV (2000) by researchers in Australia

and the U.S Later, further improvements, calledhyperaccurate correction (2006), HyperLS (2009),

andhyper-renormalization (2012), were presented by the authors Today, these are regarded as the

most accurate fitting methods among all known techniques is book describes these algorithms

as well as underlying theories, implementation details, and applications to 3-D scene analysis.Most textbooks on computer vision begin with mathematical fundamentals followed by theresulting computational procedures is book, in contrast,immediately describes computational

procedures after a short statement of the purpose and the principle e theoretical background

is briefly explained asComments us, readers need not worry about mathematical details, which

often annoy those who only want to build their vision systems Rigorous derivations and detailedjustifications are given later in separate sections, but they can be skipped if the interest is not intheories Sample program codes of the authors are provided via the website¹of the publisher Atthe end of each chapter is given a section calledSupplemental Note, describing historical back-

grounds, related issues, and reference literature

Chapters1 4specifically describe ellipse fitting algorithms Chapter5discusses 3-D ysis of circular objects in the scene extracted by ellipse fitting In Chapter6, performance com-parison experiments are conducted among the methods described in Chapters1 4 Also, somereal image applications of the 3-D analysis of Chapter5are shown In Chapter7, we point outhow procedures of ellipse fitting can straightforwardly be extended to fundamental matrix andhomography computation, which play a central role in 3-D analysis by computer vision Chap-ters8and9give general mathematical theories of statistical optimization underlying all ellipsefitting algorithms Finally, Chapter10gives a rigorous analysis of the theoretical accuracy limit.However, beginners and practice-oriented readers can skip these last three chapters

anal-e authors used the materials in this book as student projects for introductory computervision research at Okayama University, Japan, and Toyohashi University of Technology, Japan

By implementing the algorithms themselves, students can learn basic programming know-hows

¹https://sites.google.com/a/morganclaypool.com/ellipse-fitting-for-computer-vision-implementation-and-applications/

xii PREFACE

and also understand the theoretical background of vision computation as their interest deepens

We are hoping that this book can serve not simply as a reference of ellipse fitting algorithms forresearchers, but also as learning material for beginners who want to start computer vision research

e theories in this book are the fruit of the authors’ collaborations and interactions withtheir colleagues and friends for many years e authors thank Takayuki Okatani of TohokuUniversity, Japan, Mike Brooks and Wojciech Chojnacki of the University of Adelaide, Aus-tralia, Peter Meer of Rutgers University, U.S., Wolfgang Förstner, of the University of Bonn,Germany, Michael Felsberg of Linköping University, Sweden, Rudolf Mester of the University

of Frankfurt, Germany, Prasanna Rangarajan of Southern Methodist University, U.S., Ali Sharadqah of University of East Carolina, U.S., and Alexander Kukush of the University of Kiev,Ukraine Special thanks are to (late) Professor Nikolai Chernov of the University of Alabama

Al-at Birmingham, U.S., without whose inspirAl-ation and assistance this work would not have beenpossible

Kenichi Kanatani, Yasuyuki Sugaya, and Yasushi Kanazawa

March 2016

1.1 ELLIPSE FITTING

Ellipse fitting means fitting an ellipse equation to points extracted from an image is is one

of the fundamental tasks of computer vision for various reasons First, we observe many circularobjects in man-made scenes indoors and outdoors, and a circle is projected as an ellipse in cameraimages If we extract elliptic segments, say, by an edge detection filter, and fit an ellipse equation

to them, we can compute the 3-D position of the circular object in the scene (we will discusssuch applications in Chapter5) Figure1.1a shows edges extracted from an indoor scene, using

an edge detection filter is scene contains many elliptic arcs, as indicated there Figure1.1bshows ellipses fitted to them superimposed on the original image We observe that fitted ellipsesare not necessarily exact object shapes, in particular when the observed arc is only a small part ofthe circumference or when it is continuously connected to a non-elliptic segment (we will discussthis issue in Chapter4)

Figure 1.1: (a) An edge image and selected elliptic arcs (b) Ellipses are fitted to the arcs in (a) and superimposed

on the original image.

2 1 INTRODUCTION

Ellipse fitting is also used for detecting not only circular objects in the image but also jects of approximately elliptic shape, e.g., human faces An important application of ellipse fitting

ob-iscamera calibration for determining the position and internal parameters of a camera by taking

images of a reference pattern, for which circles are often used for the ease of image processing.Ellipse fitting is also important as a mathematical prototype of various geometric estimation prob-lems for computer vision Typical problems include the computation of fundamental matrices andhomographies (we will briefly describe these in Chapter7)

1.2 REPRESENTATION OF ELLIPSES

e equation of an ellipse has the form

Ax2C 2Bxy C Cy2C 2f0.Dx C Ey/ C f02F D 0; (1.1)wheref0 is a constant for adjusting the scale eoretically, it can be 1, but for finite-lengthnumerical computation it should be chosen so thatx=f0 andy=f0 have approximately the order

of 1; this increases the numerical accuracy, avoiding the loss of significant digits In view of this,

we take the origin of the imagexy coordinate system at the center of the image, rather than theupper-left corner as is customarily done, and takef0to be the length of the side of a square which

we assume to contain the ellipse to be extracted For example, if we know that an ellipse exists in

a600  600pixel region, we letf0 = 600 Since Eq (1.1) has scale indeterminacy, i.e., the sameellipse is represented ifA,B,C,D,E, andFare simultaneously multiplied by a nonzero constant,

we need some kind of normalization Various types of normalizations have been considered in thepast, including

1.3 LEAST SQUARES APPROACH 3

If we define the 6-D vectors

ξD

0BBBB

@

x22xy

y22f0x2f0y

f02

1CCCCA

0BBBB

@

ABCDEF

1CCCCA

Eq (1.4) can be written as

where and hereafter we denote the inner product of vectors a and b by.a;b/ Since the vector θ in

Eq (1.9) has scale indeterminacy, it must be normalized in correspondence with Eqs (1.2)–(1.7).Note that the left sides of Eqs (1.2)–(1.7) can be seen as quadratic forms inA, ,F; Eqs (1.2)and (1.3) are linear equations, but we may regard them asF2= 1 and.A C C /2 = 1, respectively.Hence, Eqs (1.2)–(1.7) are all written in the form

for some normalization matrix N e use of Eq (1.4) corresponds to N = I (the identity), in

which case Eq (1.10) is simplykθk= 1, i.e., normalization to unit norm

1.3 LEAST SQUARES APPROACH

Fitting an ellipse in the form of Eq (1.1) to a sequence of points.x1; y1/, ,.xN; yN/in thepresence of noise (Fig.1.2) is to findA,B,C,D,E, andF such that

Ax2˛C 2Bx˛y˛C Cy˛2C 2f0.Dx˛C Ey˛/ C f02F  0; ˛ D 1; :::; N: (1.11)

If we write ξ˛for the value obtained by replacingxandyin the 6-D vector ξ of Eq (1.8) byx˛

andy˛, respectively, Eq (1.11) can be equivalently written as

.ξ˛;θ/  0; ˛ D 1; :::; N: (1.12)

(x , y )α α

Figure 1.2: Fitting an ellipse to a noisy point sequence.

4 1 INTRODUCTION

Our task is to compute such a unit vector θ e simplest and the most naive method is the

followingleast squares.

Procedure 1.1 (Least squares)

1 Compute the6  6matrix

M D 1N

and return the unit eigenvector θ for the smallest eigenvalue

Comments.is is a straightforward generalization of line fitting to a point sequence (,!lem1.1); we minimize the sum of the squares

subject tokθk= 1 As is well known in linear algebra, the minimum of this quadratic form in

θ is given by the unit eigenvector θ of M for the smallest eigenvalue Equation (1.15) is oftencalled thealgebraic distance, and Procedure1.1is also known asalgebraic distance minimization It

is sometimes calledDLT (direct linear transformation) Since the computation is very easy and the

solution is immediately obtained, this method has been widely used However, when the inputpoint sequence covers only a small part of the ellipse circumference, it often produces a small andflat ellipse very different from the true shape (we will see such examples in Chapter6) Still, this

is a prototype of all existing ellipse fitting algorithms How we can improve this method is themain theme of this book

1.4 NOISE AND COVARIANCE MATRICES

e reason for the poor accuracy of Procedure1.1 is that the properties of image noise are notconsidered; for accurate fitting, we need to take the statistical properties of noise into consider-ation Suppose the datax˛ andy˛ are disturbed from their true values Nx˛ and Ny˛ by
x˛ and

y˛:

x˛ D Nx˛C
x˛; y˛ D Ny˛C
y˛: (1.16)

Substituting this into ξ˛, we can write

ξ˛D Nξ˛C
1ξ˛C
2ξ˛; (1.17)

1.4 NOISE AND COVARIANCE MATRICES 5

whereξN˛ is the value of ξ˛ obtained by replacingx˛ andy˛ by their true values Nx˛ and Ny˛, spectively, while
1ξ˛and
2ξ˛are, respectively, the first-order noise term (the linear expression

re-in
x˛ and
y˛) and the second-order noise term (the quadratic expression in
x˛ and
y˛).From Eq (1.8), we obtain the following expressions:

1ξ˛D

0BBBB

;
2ξ˛ D

0BBBB

@

x˛22
x˛
y˛

y˛2000

1CCCCA: (1.18)

We regard the noise terms
x˛and
y˛as random variables and define the covariance matrix of

ξ˛by

V Œξ˛ D EŒ
1ξ˛
1ξ>˛; (1.19)whereEŒ
denotes expectation over the noise distribution If we assume that
x˛and
y˛aresampled from independent Gaussian distributions of mean 0 and standard deviation, we obtain

EŒ
x˛ D EŒ
y˛ D 0; EŒ
x˛2 D EŒ
y˛2 D 2; EŒ
x˛
y˛ D 0: (1.20)Substituting Eq (1.18) and using this relationship, we obtain the covariance matrix in Eq (1.19)

in the following form:

V Œξ˛ D 2V0Œξ˛; V0Œξ˛ D 4

0BBBB

Since all the elements ofV Œξ˛have the multiple2, we factor it out and callV0Œξ˛thenormalized covariance matrix We also call the standard deviation thenoise level e diagonal elements of

the covariance matrixV Œξ˛indicate the noise susceptibility of each component of ξ˛, and theoff-diagonal elements measure their pair-wise correlation

e covariance matrix of Eq (1.19) is defined in terms of the first-order noise term
1ξ˛

alone It is known that incorporation of the second-order term
2ξ˛ has little influence overfinal results is is because
2ξ˛is very small as compared with
1ξ˛ Note that the elements

ofV0Œξ˛in Eq (1.21) contain true values Nx˛ and Ny˛ ey are replaced by observed valuesx˛

andy˛ in actual computation It is known that this replacement has practically no effect in thefinal results

6 1 INTRODUCTION

1.5 ELLIPSE FITTING APPROACHES

In the following chapters, we describe typical ellipse fitting methods that incorporate the abovenoise properties We will see that all the methods we considerdo not require knowledge of the noise level, which is very difficult to estimate in real problems e qualitative properties of noise areall encoded in the normalized covariance matrixV0Œξ˛, which gives sufficient information fordesigning high accuracy fitting schemes In general terms, there exist two approaches for ellipsefittings: algebraic and geometric

Algebraic methods: We solves some algebraic equation for computing θ e resulting solution

may or may not minimize some cost function In other words, the equation need not havethe form ofrθJ = 0 for some cost functionJ Rather, we can modify the equation in any

way so that the resulting solution θ is as close to its true value θNas possible us, our task

is to finda good equation to solve To this end, we need detailed statistical error analysis.

Geometric methods: We minimize some cost functionJ Hence, the solution is uniquely mined once the costJ is defined us, our task is to finda good cost to minimize For this,

deter-we need to consider the geometry of the ellipse and the data points We also need to devise

a convenient minimization algorithm, since minimization of a given cost is not always easy

e meaning of these two approaches will be better understood by seeing the actual proceduresdescribed in the subsequent chapters ere are, however, a lot of overlaps between the two ap-proaches

1.6 SUPPLEMENTAL NOTE

e study of ellipse fitting began as soon as computers came into use for image analysis in the1970s Since then, numerous fitting techniques have been proposed, and even today new methodsappear one after another Since we are unable cite all the literature, we mention only some ofthe earlest work:Albano[1974],Bookstein[1979],Cooper and Yalabik[1979],Gnanadesikan

[1977],Nakagawa and Rosenfeld[1979],Paton[1970] Beside fitting an ellipse to data points,

a voting scheme calledHough transform for accumulating evidences in the parameter space was

also studied as a means of ellipse fitting [Davis,1989]

In the 1990s, a paradigm shift occurred It was first thought that the purpose of ellipsefitting was to find an ellipse thatapproximately passes near observed points However, some re-

searchers, including the authors, turned their attention to finding an ellipse thatexactly passes through the true points that would be observed in the absence of noise us, the problem turned

to astatistical problem for estimating the true points subject to the constraint that they are on some ellipse It follows that the goodness of the fitted ellipse is measured not by how close it is to the

observed points but by how close it is to thetrue shape is type of paradigm shift has also

oc-curred in other problems including fundamental matrix and homography computation for 3-D

observations and the vector θ =.i/of unknowns en, all techniques and analysis for ellipsefitting can apply Evidently, Eq (1.22) includes all polynomial curves in 2-D and all polynomialsurfaces in 3-D, but all algebraic functions also can be expressed in the form of Eq (1.22) aftercanceling denominators For general nonlinear surfaces, too, we can usually write the equation

in the form of Eq (1.22) after an appropriate reparameterization, as long as the problem is toestimate the “coefficients” of linear/nonlinear terms Terms that are not multiplied by unknowncoefficients are regarded as being multiplied by 1, which is also regarded as an unknown eresulting set of coefficients can be viewed as a vector of unknown magnitude, or a “homogeneousvector,” and we can write the equation in the form.ξ;θ/= 0 us, the theory in this book haswide applicability beyond ellipse fitting

In Eq (1.1), we introduce the scaling constantf0 to makex=f0 andy=f0have the order

of 1, and this also make the vector ξ in Eq (1.8) have magnitude O.f02/so that ξ=f02 is proximately a unit vector e necessities and effects of such scaling for numerical computation

ap-is dap-iscussed byHartley[1997] in relation to fundamental matrix computation (we will discussthis in Chapter7), which is also a fundamental problem of computer vision and has the samemathematical structure as ellipse fitting In this book, we introduce the scaling constantf0 andtake the coordinate origin at the center of the image based on the same reasoning

e normalization using Eq (1.2) was adopted by Albano[1974], Cooper and Yalabik

[1979], andRosin[1993] Many authors used Eq (1.4), but some authors preferred Eq (1.5)[Gnanadesikan,1977] e use of Eq (1.6) was proposed byBookstein[1979], who argued that

it leads to “coordinate invariance” in the sense that the ellipse fitted by least squares after the ordinate system is translated and rotated is the same as the originally fitted ellipse translated androtated accordingly In this respect, Eqs (1.3) and (1.7) also have that invariance e normaliza-tion using Eq (1.7) was proposed byFitzgibbon et al.[1999] so that the resulting least-squares

co-fit is guaranteed to be an ellipse, while other equations can theoretically produce a parabola or ahyperbola (we will discuss this in Chapter4)

Today, we need not worry about the coordinate invariance, which is a concern of the past

As long as we regard ellipse fitting as statistical estimation and use Eq (1.10) for normalization,all statistically meaningful methods are automatically invariant to the choice of the coordinate

system is is because the normalization matrix N in Eq (1.10) is defined as a function of thecovariance matrixV Œξ˛of Eq (1.19) If we change the coordinate system, e.g., adding transla-tion, rotation, and other arbitrary mapping, the covariance matrixV Œξ defined by Eq (1.19)

8 1 INTRODUCTION

also changes, and the fitted ellipse after the coordinate change using the transformed covariancematrix is the same as the ellipse fitted in the original coordinate system using the original covari-ance matrix and transformed afterwards

e fact that the all the normalization equations of Eqs (1.2)–(1.7) are written in the form

of Eq (1.10) poses an interesting question: What N is the “best,” if we are to minimize the

algebraic distance of Eq (1.15) subject to.θ;N θ/= 1? is problem was studied by the authors’group [Kanatani and Rangarajan,2011,Kanatani et al.,2011,Rangarajan and Kanatani,2009],

and the matrix N that gives rise to the highest accuracy was found after a detailed error analysis.

e method was namedHyperLS (this will be described in the next chapter).

Minimizing the sum of squares in the form of Eq (1.15) is a natural idea, but ers may wonder why we minimizePN

read-˛D1.ξ˛;θ/2 Why not minimize, say, the absolute sum

PN

˛D1j.ξ˛;θ/jor the maximum maxN

˛D1j.ξ˛;θ/j? is is the issue of the choice of thenorm A

class of criteria, calledLp-norms, exist for measuring the magnitude of ann-D vector x =.xi/:

kxk0  jfij jxij ¤ 0gj; (1.27)where the right side means the number of nonzero components is is called theL0-norm, or

theHamming distance.

Least squares for minimizing theL2-norm is the most widely used approach for cal optimization for two reasons One is the computational simplicity: differentiation of a sum

dis-matically ignored Estimation methods that are not very susceptible to outliers are said to be

robust (we will discuss robust ellipse fitting in Chapter4) For this reason,L1-minimization isfrequently used in some computer vision applications

PROBLEMS

1.1. Fitting a line in the formn1x C n2y C n3f0 = 0 to points.x˛; y˛/,˛= 1, ,N, can be

viewed as the problem for computing n =.ni/, which can be normalized to a unit vector,such that.ξ˛;n/ 0,˛= 1, ,N, with

(1) Write down the least-squares procedure for this computation

(2) If the noise terms
x˛and
y˛are sampled from independent Gaussian tions of mean 0 and variance2, how is their covariance matrixV Œξ˛defined?

2.1 ITERATIVE REWEIGHT AND LEAST SQUARES

e followingiterative reweight is an old and well-known method:

Procedure 2.1 (Iterative reweight)

where ξ˛is the vector defined in Eq (1.8) for the˛th point

3 Solve the eigenvalue problem

and compute the unit eigenvector θ for the smallest eigenvalue

4 If θθ0 up to sign, return θ and stop Else, update

W˛

1.θ; V0Œξ˛θ/; θ0 θ; (2.3)and go back to Step2

Comments. is method is motivated to minimize the weighted sum of squares

12 2 ALGEBRAIC FITTING

and this quadratic form is minimized by the unit eigenvector of M for the smallest eigenvalue.

According to the theory of statistics, the weightsW˛are optimal if they are inversely proportional

to the variance of each term, being small for uncertain terms and large for certain terms Since

.ξ˛;θ/= Nξ˛;θ/ C
1ξ˛;θ/ C
2ξ˛;θ/and Nξ˛;θ/= 0, the variance is

EŒ.ξ˛;θ/2 D EŒ.θ;
1ξ˛
1ξ˛>θ/ D θ; EŒ
1ξ˛
1ξ˛>θ/ D 2.θ; V0Œξ˛θ/; (2.5)omitting higher-order noise terms us, we should letW˛=1=.θ; V0Œξ˛θ/, but θ is not known

yet So, we instead use the weightsW˛ determined in the preceding iteration to compute θ and

update the weights as in Eq (2.3) Let us call the solution θ computed in the initial iteration the

initial solution Initially, we letW˛= 1, so Eq (2.4) implies that we are minimizingPN

˛D1.ξ˛;θ/2

In other words, the initial solution is the least-squares solution, from which the iterations start

e phrase “up to sign” in Step4reflects the fact that eigenvectors have sign indeterminacy So,

we align θ and θ0by reversing the sign θ θwhenever.θ;θ0/ <0

2.2 RENORMALIZATION AND THE TAUBIN METHOD

It is well known that the iterative reweight method has a large bias, in particular when the inputelliptic arc is short As a result, we often obtain a smaller ellipse than expected e following

renormalization was introduced to remedy this.

2.3 HYPER-RENORMALIZATION AND HYPERLS 13

Comments.is method was motivated as follows Since the true valuesξN˛satisfy Nξ˛;θ/= 0, wecan immediately see thatM θN =0, whereMN is the true value of the matrix M in Eq (2.6) defined

by the true valuesξN˛ us, if we knowMN , the solution θ is obtained as its unit eigenvector for

eigenvalue 0 However,MN is unknown, so we estimate it e expectation of M is

W˛= 1, the initial solution minimizesPN

˛D1.ξ˛;θ/2subject to.θ;PN

˛D1V0Œξ˛

θ/= constant

is is known as theTaubin method (,!Problem2.1)

Standard numerical tools for solving the generalized eigenvalue problem in the form of

Eq (2.7) assume that N is positive definite (some eigenvalues are negative) However, Eq (1.21)implies that the sixth column and the sixth row of the matrixV0Œξ˛all consist of zero, so N is

not positive definite On the other hand, Eq (2.7) is equivalently written as

2.3 HYPER-RENORMALIZATION AND HYPERLS

According to experiments, the accuracy of the Taubin method is higher than iterative reweight,and renormalization has even higher accuracy e accuracy can be further improved by the fol-lowinghyper-renormalization.

Procedure 2.3 (Hyper-renormalization)

1 Let θ =0andW = 1,˛= 1, ,N

14 2 ALGEBRAIC FITTING

2 Compute the6  6matrices

M D 1N

; (2.11)

whereSŒ
is the symmetrization operator (SŒA=.ACA>/=2), and e is the vector

eD 1; 0; 1; 0; 0; 0/>: (2.12)

e matrix M5 is the pseudoinverse of M of truncated rank 5.

3 Solve the generalized eigenvalue problem

Comments.is method was obtained in an effort to improve renormalization by modifying the

matrix N in Eq (2.7) so that the resulting solution θ has the highest accuracy It has been found

by rigorous error analysis that the choice of N in Eq (2.11) attains the highest accuracy (thederivation of Eq (2.11) will be given in Chapter8) e vector e in Eq (2.12) is defined in such

a way that

holds for the second-order noise term
2ξ˛ From Eqs (1.18) and (1.20), we obtain Eq (2.12)

e pseudoinverse M5 of truncated rank 5 is computed from the eigenvalues1


 6of

M and the corresponding unit eigenvectors θ1, , θ6 of M in the form

M5 D 1 θ1θ>1 C


C1 θ5θ5>; (2.16)

2.4 SUMMARY 15

where the term for6and θ6is removed e matrix N in Eq (2.11) is not positive definite, but

we can use a standard numerical tool by rewriting Eq (2.13) in the form of Eq (2.10) and puting the unit generalized eigenvector for the generalized eigenvalue1=of the largest absolutevalue e initial solution minimizesPN

com-˛D1.ξ˛;θ/2subject to.θ;N θ/= constant for the matrix

Nobtained by lettingW˛= 1 in Eq (2.11) is corresponds to the method calledHyperLS (,!

Problem2.2)

2.4 SUMMARY

We have seen that all the above methods compute the θ that satisfies

where the matrices M and N are defined in terms of the observations ξ˛ and the unknown θ.

Different choices of them lead to different methods:



V0Œξ˛ C 2SŒξ˛e>1

.ξ˛;M5ξ˛/V0Œξ˛ C 2SŒV0Œξ˛M5ξ˛ξ˛>

:

(hyper-renormalization)

(2.19)

16 2 ALGEBRAIC FITTING

For least squares, Taubin, and HyperLS, the matrices M and N do not contain the unknown

θ, so Eq (2.17) is a generalized eigenvalue problem, which can be directly solved without ations For other methods (iterative reweight, renormalization, and hyper-renormalization), the

iter-unknown θ is contained in the denominators in the expressions of M and N Letting the part that contains θ be1=W˛, we computeW˛using the value of θ obtained in the preceding iteration

and solve the generalized eigenvalue problem in the form of Eq (2.17) We then use the resulting

θto updateW˛ and repeat this process

According to experiments, HyperLS has comparable accuracy to renormalization, andhyper-renormalization has even higher accuracy Since the hyper-renormalization iteration startsfrom the HyperLS solution, the convergence is very fast; usually, three to four iterations aresufficient Numerical comparison of the accuracy and efficiency of these methods is shown inChapter6

2.5 SUPPLEMENTAL NOTE

e iterative reweight was first presented bySampson[1982] e renormalization scheme wasproposed by Kanatani [1993b]; the details are discussed in Kanatani [1996] e method ofTaubin was proposed byTaubin[1991], but his derivation was rather heuristic without consider-ing the statistical properties of image noise As mentioned in the Supplemental Note of Chapter1

(page6), the HyperLS was obtained in an effort to improve the algebraic distance minimization

al.,2011,Rangarajan and Kanatani,2009] turned out to be the value of N in Eq (2.11) withW˛

= 1 is HyperLS method was then generalized to the iterative hyper-renormalization schemealso by the authors’ group [Kanatani et al.,2012,2014]

e fact that all known algebraic methods can be written in the form of Eq (2.17) waspointed out inKanatani et al.[2014] Since noise is regarded as random, the observed data are

random variables having probability distributions Hence, the computed value θ is also a random variable having its probability distribution e choice of the matrices M and N affects the probability distribution of the computed θ According to the detailed error analysis ofKanatani

et al.[2014] (which we will discuss in Chapter8), the choice of M controls the covariance of θ

(Fig.2.1a), while the choice of N controls its bias (Fig.2.1b) If we choose M to be the second

row of Eq (2.18), the covariance matrix of θ attains the theoretical accuracy limit called the KCR lower bound (this will be discussed in Chapter10) except forO.4/terms If we choose N to

be the fifth row of Eq (2.19), the bias of θ is 0 except forO.4/terms (this will be shown inChapter8) Hence, the accuracy of hyper-renormalization cannot be improved any further exceptfor higher order terms in It indeed fits a very accurate ellipse, as will be demonstrated in theexperiments in Chapter6

2.1. Write down the procedure of the Taubin method.

2.2. Write down the HyperLS procedure

C H A P T E R 3

Geometric Fitting

We consider geometric fitting, i.e., computing an ellipse that passes near data points as closely

as possible We first show that the closeness to the data points is measured by a function calledthe “Sampson error.” en, we give a computational procedure, called “FNS,” that minimizes it.Next, we describe a procedure for exactly minimizing the sum of squares from the data points,called the “geometric distance,” iteratively using the FNS procedure Finally, we show how theaccuracy can be further improved by a scheme called “hyperaccurate correction.”

3.1 GEOMETRIC DISTANCE AND SAMPSON ERROR

Geometric fitting refers to computing the ellipse that minimizes the sum of squares of the distance

d˛from observed points.x˛; y˛/to the ellipse (Fig.3.1) Let Nx˛; Ny˛/be the point on the ellipseclosest to.x˛; y˛/ We compute the ellipse that minimizes

S D 1N

N

X

˛D1

.x˛ Nx˛/2C y˛ Ny˛/2

D 1N

N

X

˛D1

d˛2; (3.1)

which is known as thegeometric distance Strictly speaking, it should be called the “square

geo-metric distance,” because it has the dimension of square length However, the term “geogeo-metricdistance” is commonly used for simplicity

Here, the notations Nx˛and Ny˛are used in a slightly different sense from the previous ters In the geometric fitting context, they are thevariables for estimating the true values ofx˛

chap-(x , y )α α

(x , y )α α

Figure 3.1: We fit an ellipse such that the geometric distance, i.e., the sum of the square distances of the data points to the ellipse, is minimized.

20 3 GEOMETRIC FITTING

andy˛, rather than their true values themselves We use this convention for avoiding overly troducing new symbols is also applies to Eq (1.16), i.e., we now regard
x˛ and
y˛as the

in-variables for estimating the discrepancy between observed and estimated positions, rather than

stochastic quantities Hence, Eq (3.1) is also written asS =.1=N /PN

and compute the unit eigenvector θ for the smallest eigenvalue

5 If θθ0 up to sign, return θ and stop Else , update

W˛

1.θ; V0Œξ˛θ/; θ0 θ; (3.7)and go back to Step2

3.3 GEOMETRIC DISTANCE MINIMIZATION 21

Comments.is scheme solvesrθJ = 0for the Sampson errorJ, whererθJ is the gradient

ofJ, i.e., the vector whose ith component is @J =@i Differentiating Eq (3.3), we obtain thefollowing expression (,!Problem3.1):

rθJ D 2.M L/θD 2Xθ: (3.8)

Here, M , L, and X are the matrices given by Eqs (3.4) and (3.5) After the iterations haveconverged,= 0 holds (see Proposition3.5below) Hence, Eq (3.8) implies that we obtain the

value θ that satisfiesrθJ =0 Since we start with θ =0, the matrix L in Eq (3.4) is initially the

zero matrix O (all elements are 0), so Eq (3.6) reduces to M θ = us, the FNS iterationsstart from the least-squares solution

3.3 GEOMETRIC DISTANCE MINIMIZATION

Once we have computed θ by the above FNS, we can iteratively modify it so that it strictly

minimizes the geometric distance of Eq (3.1) To be specific, we modify the data ξ˛, using the

current solution θ, and minimize the resulting Sampson error to obtain the new solution θ After

a few iterations, the Sampson error coincides with the geometric distance e procedure goes asfollows

Procedure 3.2 (Geometric distance minimization)

1 LetJ0=1(a sufficiently large number), Ox˛=x˛, Oy˛ =y˛, and Qx˛= Qy˛ = 0,˛= 1, ,N

2 Compute the normalized covariance matrixV0Œ Oξ˛obtained by replacing Nx˛ and Ny˛ in thedefinition ofV0Œξ˛in Eq (1.21) by Ox˛and Oy˛, respectively

3 Computing the following modified data ξ˛:

ξ˛D

0BBBBB

f02

1CCCCCA

X

˛D1

Qx˛2C Qy˛2/: (3.13)

IfJ J0, return θ and stop Else, letJ0 J, and go back to Step2

Comments.Since Ox˛ =x˛, Oy˛ = y˛, and Qx˛ = Qy˛ = 0 in the initial iteration, the value ξ

where

Qx˛ D x˛ Ox˛; Qy˛ D y˛ Oy˛: (3.15)

In the next iteration, we regard the corrected values Ox˛; Oy˛/ as the input data and minimize

Eq (3.14) Let OOx˛; OOy˛/be the resulting solution Ignoring high-order small terms in Ox˛ Nx˛

and Oy˛ Ny˛ and rewriting Eq (3.14), we obtain the modified Sampson errorJ in Eq (3.10)(see Proposition 3.7below) We minimize this, regarding OOx˛; OOy˛/as Ox˛; Oy˛/, and repeat thesame process Since the current Ox˛; Oy˛/are the best approximation of Nx˛; Ny˛/, Eq (3.15) impliesthat Qx2

˛C Qy2

˛is the corresponding approximation of Nx˛ x˛/2

C Ny˛ y˛/2 So, we use (3.13)

to evaluate the geometric distance S Since the ignored high order terms decrease after each

iteration, we obtain in the end the value θ that minimizes the geometric distanceS, and Eq (3.13)coincides withS According to experiments, however, the correction of θ by this procedure is very

small: the three or four significant digits are unchanged in typical problems; the corrected ellipse

is indistinguishable from the original one when displayed or plotted us, we can practicallyidentify FNS with a method to minimize the geometric distance

3.4 HYPERACCURATE CORRECTION 23

3.4 HYPERACCURATE CORRECTION

e geometric method, whether it is geometric distance minimization or Sampson error imization, is known to have small statistical bias, meaning that the expectationEŒθ does notcompletely agree with the true valueθN We express the computed solutionθOin the form

min-O

θD NθC
1θC
2θC


; (3.16)where
kθis akth order expression in
x˛ and
y˛ Since
1θis linear in
x˛and
y˛, wehaveEŒ
1θ=0 However, we haveEŒ
2θ ¤ 0in general If we are able to evaluateEŒ
2θin

an analytic form by doing error analysis, it is expected that the subtraction

Q

will result in a higher accuracy thanθO; its expectation isEŒ Qθ=θNC O.4/, where is the noiselevel Note that
3θis a third-order expression in
x˛and
y˛, soEŒ
3θ=0 is operationfor increasing accuracy by subtracting EŒ
2θ is called hyperaccurate correction e procedure

where M is the value of the matrix M in Eq (3.4) after the FNS iterations have converged

3 Compute the correction term

whereW˛ is the value ofW˛ in Eq (3.7) after the FNS iterations have converged, and e

is the vector in Eq (2.12) e matrix M5 is the pseudoinverse of M of truncated rank 5

given by Eq (2.16)

4 Correct θ into

whereNŒ
denotes normalization to unit norm (NŒa= a=kak)

Comments.e derivation of Eq (3.19) is given in Chapter9 According to experiments, theaccuracy of geometric distance minimization is higher than renormalization but lower than hyper-renormalization It is known, however, that after the above hyperaccurate correction the accuracyimproves and is comparable to hyper-renormalization, as will be demonstrated in Chapter6

us, k is the unit vector orthogonal to thexyplane, and P k v for any vector v is the projection

of v onto thexy plane, replacing the third component of v by 0 Evidently, P k k=0

Proposition 3.4 (Distance to ellipse) e square distanced˛2 of point.x˛; y˛/from the ellipse of

Eq ( 3.22 ) is written, up to high-order small noise terms, in the form

d˛2 f

2 0

4

.x˛;Qx˛/.Qx˛;P k Qx˛/: (3.24)

Proof Let Nx˛; Ny˛/ be the closest point on the ellipse from.x˛; y˛/ LetxN˛ and x˛ be theirvector representations as in Eq (3.21) If we write

the distanced˛between the two points isf0k
x˛k We compute the value
x˛that minimizes

k
x˛k2 Since the pointxN˛ is on the ellipse,

.x˛
x˛;Q.x˛
x˛// D 0 (3.26)holds Expanding the left side and ignoring second-order terms in
x˛, we obtain

.Qx˛;
x˛/ D 1

2.x˛;Qx˛/: (3.27)

3.5 DERIVATIONS 25

Since the third components of x˛ and xN˛ are both 1, the third component of
x˛ is 0 isconstraint is written as k;
x˛/= 0, using the vector k in Eq (3.23) Introducing Lagrangemultipliers, we differentiate

k
x˛k2 ˛

.Qx˛;
x˛/ 1

2.x˛;Qx˛/



.k;
x˛/; (3.28)with respect to
x˛ Letting the result be0, we obtain

Multiplying P k on both sides and noting that P k
x˛=
x˛ and P k k=0, we obtain

x˛D 2˛P k Qx˛: (3.30)Substituting this into Eq (3.27), we have

.Qx˛;˛

2 P k Qx˛/ D 1

2.x˛;Qx˛/; (3.31)from which˛is given in the form

where we have noted that P2

k = P k from the definition of P k We have also used the identity

kP k Qx˛k2 D P k Qx˛;P k Qx˛/ D Qx˛;P k2Qx˛/ D Qx˛;P k Qx˛/: (3.35)Hence,d˛2=f02k
x˛k2is expressed in the form of Eq (3.24) 

From the definition of the vectors ξ and θ and the matrices Q and V0Œξ˛ ( Nx˛; Ny˛/ in

Eq (1.21) are replaced by.x˛; y˛/), we can confirm after substitution and expansion that thefollowing identities hold:

.x˛;Qx˛/ D 1

f2.ξ˛;θ/; .Qx˛;P k Qx˛/ D 1

4f2.θ; V0Œξ˛θ/: (3.36)