Convergence of Gauss Newtons method

Convergence of Gauss–Newtons methodand uniqueness of the solution School of Mathematics and Computational Sciences, University of Petroleum, Dongying 257061, Shandong Province, PR China

Convergence of Gauss–Newton
s method

and uniqueness of the solution

School of Mathematics and Computational Sciences, University of Petroleum,

Dongying 257061, Shandong Province, PR China

Abstract

In this paper, we study the convergence of Gauss–Newton
s method for nonlinear least squares problems Under the hypothesis that derivative satisfies some kinds of weak Lipschitz condition, we obtain the exact estimates of the radii of convergence ball

of Gauss–Newton
s method and the uniqueness ball of the solution New results can be used to determinate approximation zero of Gauss–Newton
s method

2005 Published by Elsevier Inc

Keywords: Gauss–Newton
s method; Lipschitz condition; Convergence ball; Uniqueness ball

1 Introduction

We consider the nonlinear least squares problems:

min FðxÞ :¼1

0096-3003/$ - see front matter
2005 Published by Elsevier Inc.

doi:10.1016/j.amc.2004.12.055

*

Corresponding author.

E-mail address: cjh_maths@yahoo.com.cn (J Chen).

www.elsevier.com/locate/amc

where f(x): Rn! Rm

re-fers to the 2-norm

Gauss–Newton
s method, defined by

xnþ1¼ xn fh 0ðxnÞTf0ðxnÞi1

Gauss–New-ton
s method, analyzes for local and rate of convergence properties are mostly restricted in quality[1,6,15,16], only considering the existence neighborhood of the convergence, but it cannot make us clearly see how big the radius of the

ball with radius r and center x, and let Bðx; rÞ denote its closure

estimate for the convergence ball of Newton
s method Under the hypothesis that f0(x) satisfies the some kind Lipschitz condition

kf0ðx Þ1ðf0ðxÞ  f0ðxsÞÞk 6

Z qðxÞ

sqðxÞ

In this paper, we consider the convergence of the Gauss–Newton
s method Under more general conditions, we obtain the convergence domain

of the Gauss–Newton
s method and the uniqueness domain of the solution Further, we can prove the optimality of the estimation of the radii New results can be used to determinate approximation zero of Gauss–Newton
s method

2 Special and generalized Lipschitz condition

The condition on the function f(x)

condi-tion in the ball B(x*, r) with constant L Sometimes if it is only required to satisfy

we call it the center Lipschitz condition in the ball B(x*, r) with constant L Furthermore, the L in the Lipschitz condition need not be a constant, but

a positive integrable function, If this is the case, then(2.1)or(2.2)is replaced by

kf ðxÞ  f ðxsÞk 6

Z qðxÞ

sqðxÞ

or

kf ðxÞ  f ðx Þk 6

Z qðxÞ

0

condi-tion
is referred to as having the L average

inverse of matrix A, and if A has full rank (namely: rank (A) = min(m, n) = n) then A= (ATA)1AT

Now, we give some Lemmas

rank(A) = rank(B), then

yk

and if rank(B) = rank(A) = min(m,n), we can obtain

kBy Ayk 6

ffiffiffi 2

p

kAyk2kEk

n, then rank(B) = n

Lemma 2.3 Let

hðtÞ ¼ 1

ta

Z t

0

where L(u) is a positive integrable function and nondecreasing monotonically in [0,r] Then h(t) is nondecreasing with respect to t

Proof In fact, by the monotonicity of L, a P 1, we obtain

hðt2Þ  hðt1Þ ¼ 1

ta

Z t 2

0

ta

Z t 2

0

LðuÞua1du

ta

Z t2

t 1

ta1

ta

0

LðuÞua1du

ta

Z t 2

t 1

ta1

ta

0

ua1du

¼ Lðt1Þ 1

ta

Z t 2

t 0

ta

Z t 1

0

ua1du¼ 0

for 0 < t1 < t2 Thus hðtÞ ¼1

t a

Rt

0LðuÞua1du is nondecreasing with respect to t

Lemma 2.4 Suppose that

gðtÞ ¼1

t

Z t

0

where L(u) is a positive integrable function in [0,r] Then g(t) is increasing mono-tonically with respect to t

Proof In fact, by the positivity of L, we have

gðt2Þ  gðt1Þ ¼ 1

t2

Z t 2

0

LðuÞðt2 uÞdu 1

t1

Z t 1

0

LðuÞðt1 uÞdu

¼

Z t 2

t 1

t2

Z t 2

t 1

t2

t1

0

LðuÞudu

P

Z t 2

t 1

LðuÞdu 

Z t 2

t 1

t21

t1

0

LðuÞudu

t11

t2

0

LðuÞudu > 0 for 0 < t1< t2 Thus g(t) is increasing with respect to t h

3 Convergence ball of Gauss–Newton
s method

Theorem 3.1 Suppose x* satisfies(1.1), f has a continuous derivative in B(x*,r)

average

ðf0ðxÞ  f0ðxsÞÞ

Z qðxÞ

sqðxÞ

where xs= x* + s(x x*), q(x) = k x  x*k, and L is nondecreasing Let r > 0 satisfy

bRr

rð1  bRr

ffiffiffi 2

p

cb2Rr

rð1  bRr

Rqðx 0 Þ

qðx0Þ2 1 bRqðx 0 Þ

þ

ffiffiffi 2

p

cb2Rqðx 0 Þ

qðx0Þð1  bRqðx 0 Þ

where

and

Rqðx 0 Þ

qðx0Þ 1  bRqðx 0 Þ

ffiffiffi 2

p

cb2Rqðx 0 Þ

qðx0Þð1  bRqðx 0 Þ

is less than 1 Moreover, if c = 0, then

Lemma 2.3, we have

Rqðx 0 Þ

qðx0Þ2 1 bRqðx 0 Þ

ffiffiffi 2

p

cb2Rqðx 0 Þ

qðx0Þ2 1 bRqðx 0 Þ

< bRr

r2 1 bRr

0LðuÞdu qðx0Þ þ

ffiffiffi 2

p

cb2Rr

r2ð1  bRr

0LðuÞduÞqðx0Þ

6 kx0 x k

f0ðx ÞTf0ðx Þ1f0ðx ÞT

Z qðxÞ

0

LðuÞdu

0

LðuÞdu < 1; 8x 2 Bðx ; rÞ

By Lemma (2.1) and (2.2), we know"x2 B(x*, r), f0(x) has full rank and

f0ðxÞTf0ðxÞ

f0ðxÞT

1 bRqðxÞ

; rÞ;

f0ðxÞTf0ðxÞ

f0ðxÞT ½f0ðx ÞTf0ðx Þ1f0ðx ÞT

6

ffiffiffi 2

p

b2RqðxÞ

qðxÞ0ð1  bRqðxÞ

; rÞ

xnþ1 x ¼ xn x  fh 0ðxnÞTf0ðxnÞi1

f0ðxnÞTfðxnÞ

¼ fh 0ðxnÞTf0ðxnÞi1

f0ðxnÞT½f0ðxnÞðxn x Þ  f ðxnÞ þ f ðx Þ

þ fh 0ðx ÞTf0ðx Þi1

fðx ÞTf0ðx Þ  ½f0ðxnÞTf0ðxnÞ1f0ðxnÞTfðx Þ

Hence,

kxnþ1 x k 6 k½f0ðxnÞTf0ðxnÞ1f0ðxnÞTk

Z 1

0

½f0ðxnÞ  f0ðx þ sðxn x Þ

 ðx  xnÞdsk þ k½f0ðx ÞTf0ðx Þ1f0ðx ÞT

 ½f0ðxnÞTf0ðxnÞ1f0ðxnÞT

kf0ðx Þk

1 bRqðx n Þ

0

Z qðx n Þ

sqðx n

LðuÞduqðxnÞds

þ

ffiffiffi 2

p

cb2Rqðx n Þ

1 bRqðx n Þ

6 bRqðx n Þ

1 bRqðx n Þ

ffiffiffi 2

p

cb2Rqðx n Þ

1 bRqðx n Þ

Taking n = 0 above, we obtain kx1 x*k 6 qk x0 x*k < kx0 x*k.

q(xn) =kxn x*k decreases monotonically Therefore, for all n = 0,1, , we have

Rqðx n Þ

qðxnÞ2 1 bRqðx n Þ

þ

ffiffiffi 2

p

cb2Rqðx n Þ

qðxnÞ 1  bRqðx n Þ

qðx0Þ2 1 bRqðx 0 Þ

þ

ffiffiffi 2

p

cb2Rqðx 0 Þ

qðx0Þ 1  bRqðx n Þ

and if c = 0, we obtain

Rqðx 0 Þ

qðx0Þ2 1 bRqðx 0 Þ

qðx0Þkxn x

k2

4 The uniqueness ball for the optimal solution

Theorem 4.1 Suppose x* satisfies(1.1), f has a continuous derivative in B(x*,r)

L average

kf0ðxÞ  f0ðx Þk 6

Z qðxÞ

0

where q(x) =kx  x*k, and L is nondecreasing Let r > 0 satisfy

b

r

0

r

Z r

0

where c,b hold in(3.4), and

b0¼hf0ðx ÞTf0ðx Þi1

Then Eq.(1.1)has a unique solution x* in B(x*,r).

f0ðx ÞTf0ðx Þ

Hence

x0 x ¼ x0 x  fh 0ðx ÞTf0ðx Þi1

f0ðx0ÞTfðx0Þ

¼ fh 0ðx ÞTf0ðx Þi1

f0ðx ÞT½f0ðx Þðx0 x Þ  f ðxoÞ þ f ðx0Þ

þ fh 0ðx ÞTf0ðx Þi1

ðf0ðx ÞT f0ðx0ÞTÞf ðx0Þ

¼ fh 0ðx ÞTf0ðx Þi1

f0ðx ÞT

0

f0ðx Þ  f0ðx þ sðx0 x ÞÞ

 ðx0 x Þds þ fh 0ðx ÞTf0ðx Þi1

f0ðx ÞT f0ðx0ÞT

fðx0Þ;

kx0 x k 6 hf0ðx ÞTf0ðx Þi1

f0ðx ÞT

0

k f½ 0ðx Þ

f0ðx þ sðx0 x ÞÞkkðx0 x Þkds

þhf0ðx ÞTf0ðx Þi1

kf0ðx ÞT f0ðx0ÞTkkf ðx0Þk

0

Z sqðx 0 Þ

0

LðuÞduqðx0Þds þ cb0

Z qðx 0 Þ

0

LðuÞdu

¼ b

Z qðx 0 Þ

0

LðuÞðqðx0Þ  uÞdu þ cb0

Z qðx 0 Þ

0

LðuÞdu

t

Rt

kx0 x k 6 b

Z qðx 0 Þ

0

LðuÞðqðx0Þ  uÞdu þ cb0

Z qðx 0 Þ

0

LðuÞdu

6 bqðx0Þ qðx0Þ

Z qðx 0 Þ

0

LðuÞðqðx0Þ  uÞdu þcb0qðx0Þ

qðx0Þ

Z qðx 0 Þ

0

LðuÞdu

<bqðx0Þ r

Z r

0

LðuÞðr  uÞdu þcb0qðx0Þ

r

0

LðuÞdu

6 kx0 x k

This is in contradiction with the assumption Thus, it follows that

5 The optimality of the estimation of the radius

Theorem 3.1 Then the given value r of the convergence ball is the best possible Proof We notice that when r is determined by equality

bRr

0LðuÞudu

rð1  bRr

ffiffiffi 2

p

cb2Rr

0LðuÞudu rð1  bRr

ball such that Gauss–Newton
s method fails In fact, the following is an exam-ple on the scaled case:

 x þ bRxx

0 ðx  x  uÞLðuÞdu; x 6x < x þ r;

x  x þ bRx x

0 ðx  x þ uÞLðuÞdu; x  r 6 x < x ;

8

<

Theorem 4.1 Then the given value r of the convergence ball is the best possible Proof We notice that when r is determined by equality

b

r

0

r

Z r

0

ball such that Fðx0Þ ¼ min1

2fðxÞTfðxÞ An example of this is (5.2) in which

6 Corollaries of the main results

In the study of the Gauss–Newton
s method (or Newton
s method), the assumption that the derivative is Lipschitz continuous is considered tradi-tional Combining Theorems 3.1 and 4.1 with Theorems 5.1 and 5.2, and tak-ing L as a constant, the followtak-ing two corollaries are obtained directly

B(x*,r) f0(x*) has full rank, and f0satisfies the radius Lipschitz condition:

kf0ðxÞ  f0ðxsÞk 6 ð1  sÞLkxs x k; 8x 2 Bðx ; rÞ; 0 6 s 6 1; ð6:1Þ

ffiffiffi 2

p

Lb2c

k 2ð1  Lbkx0 x kÞþ

ffiffiffi 2

p

Lb2c

the inequality(3.5)holds If c = 0, then(3.6)follows Moreover, the given r is the best possible

B(x*,r) f0(x*) has full rank, and f0 satisfies the center Lipschitz condition:

where L is positive number and

r¼2ð1  Lb0cÞ

B(x*,r) Moreover, the given r is the best possible

Theorems 5.1 and 5.2, we obtain the following two corollaries

Corollary 6.3 Suppose f(x*) = 0, f has a continuous derivative in B(x*, r) f0(x*) has full rank, and f0 satisfies the radius Lipschitz condition:

kf0ðxÞ  f0ðxsÞk 6

Z qðxÞ

sqðxÞ

satisfy

bRr

r 1 bRr

holds in(3.4) For

q¼ b

Rqðx 0 Þ

qðx0Þ 1  bRqðx 0 Þ

the inequality(3.6)holds Moreover, the given r is the best possible

Corollary 6.4 Suppose f(x*) = 0, f has a continuous derivative in B(x*, r) f0(x*) has full rank, and f0 satisfies the center Lipschitz condition:

kf0ðxÞ  f0ðx Þk 6

Z qðxÞ

0

where q(x) =kx  x*k, and L is nondecreasing Let r > 0 satisfy

b

r

0

where b holds in(3.4) Then Eq.(1.1)has a unique solution x* in B(x*, r) More-over, the given r is the best possible

Corollaries (6.1), (6.2) and (6.3), (6.4) generalize the results which are

Using Theorems (3.1) and (4.1), we also derive some new properties in es-sence about the convergence of Newton
s method and the uniqueness of the solutions of the equation In the following example, let c be a positive number Example 1 Taking

1

ð1  ckx  x kÞ2

1

we have

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi bð9b þ 8Þ p

and

k

If the right-hand side in(6.9)is replaced by

1

7 Convergence under weaker Lipschitz condition

Theorem 7.1 Suppose x* satisfies(1.1), f(x*) = 0, f has a continuous derivative

in B(x*,r) f0(x*) has full rank, and f0satisfies the radius Lipschitz condition with

L average

kf0ðxÞyðf0ðxÞ  f0ðxsÞÞk 6

Z qðxÞ

sqðxÞ

ð7:1Þ

where xs= x* + s(x x*), q(x) = kx  x*, f0(x)= [f0(x)Tf0(x)]1f0(x)T, and L

is nondecreasing Let r > 0 satisfy

Rr

where

Rqðx 0 Þ

is less than 1

2.3, we have

Rqðx 0 Þ

qðx0Þ2 qðx0Þ <

Rs

k

From the condition(7.2), we know"x 2 B(x*, r), f0(x) has full rank

Now if xn2 B(x*,r), we have by(1.2)

xnþ1 x ¼ xn x  fh 0ðxnÞTf0ðxnÞi1

f0ðxnÞTfðxnÞ

¼ fh 0ðxnÞTf0ðxnÞi1

f0ðxnÞT½f0ðxnÞðxn x Þ  f ðxnÞ þ f ðx Þ

¼ fh 0ðxnÞTf0ðxnÞi1

f0ðxnÞT



0

½f0ðxnÞ  f0ðx þ sðxn x ÞÞðx  xnÞds

Hence

kxnþ1 x k 6

0

f0ðxnÞTf0ðxnÞ

f0ðxnÞT½f0ðxnÞ

f0ðx þ sðxn x ÞÞkkx  xnkds

6

0

Z qðx n Þ

sqðx n Þ

LðuÞduqðxnÞds ¼

Z qðx n Þ

0

LðuÞudu

q(xn) =kxn x*k decreases monotonically Therefore, for all n = 0,1, , , we have

kxnþ1 x k 6

Rqðx n Þ

qðxnÞ2 qðxnÞ

26

Rqðx 0 Þ

qðx0Þ2 qðxnÞ

2

qðx0Þkxn x

k2

For the condition (7.1), if f(x): Rn! Rn(namely: m = n), taking f0(x)=

f0(x*)1, Theorem 7.1 is obtained by Wang ([14], Theorem 3.1)

Now we give the uniqueness ball for the optimal solution

Theorem 7.2 Suppose x* satisfies(1.1), f(x*) = 0, f has a continuous derivative

in B(x*,r) f0(x*) has full rank, and f0satisfies the center Lipschitz condition with

L average

kf0ðx Þyðf0ðxÞ  f0ðx ÞÞk 6

Z qðxÞ

where q(x) =kx  x*k, f0(x*)= [f0(x*)Tf0(x*)]1f0(x*)T, and L is positive inte-grable function Let r > 0 satisfy

1

r

0

f0ðx ÞTf0ðx Þ

Hence

x0 x ¼ x0 x  ½f0ðx ÞTf0ðx Þ1f0ðx0ÞTfðx0Þ

¼ ½f0ðx ÞTf0ðx Þ1f0ðx ÞT½f0ðx Þðx0 x Þ  f ðx0Þ þ f ðx Þ

¼ ½f0ðx ÞTf0ðx Þ1f0ðx ÞT

0

½f0ðx Þ

 f0ðx þ sðx0 x ÞÞðx0 x Þds;

kx0 x k 6

0

k½f0ðx ÞTf0ðx Þ1f0ðx ÞT½f0ðx Þ

 f0ðx þ sðx0 x ÞÞkkx0 x kds

6

0

Z sqðx 0 Þ

0

LðuÞduqðx0Þds ¼

Z qðx 0 Þ

0

LðuÞðqðx0Þ  uÞdu

t

Rt

kx0 x k 6

Z qðx 0 Þ

0

LðuÞðqðx0Þ  uÞdu

6 qðx0Þ qðx0Þ

Z qðx 0 Þ

0

LðuÞðqðx0Þ  uÞdu

<qðx0Þ r

0

LðuÞðr  uÞdu 6 kx0 x k

Analogous to Section 5, we can obtain the following results

Theorem 7.3 Suppose that the equality sign holds in the inequality(7.3) in the Theorem 7.1 Then the given value r of the convergence ball is the best possible Furthermore, r only depends on L, but is independent of f

Proof We notice that when r is determined by equality

Rr

0LðuÞudu

there exits f satisfying(7.1)and(7.2)in B(x*,r) and x0on the boundary of the closed ball such that Gauss–Newton
s method fails In fact, the following is an example on the scaled case:

3

2xþL

4x2; 0 6 x 6 r;

3

2xL

4x2; r 6 x < 0;

(

ð7:10Þ where L is a positive constant and x0= r, xn= (1)n

r

Theorem (7.2) Then the given value r of the convergence ball is the best possible Furthermore, r only depends on L, but is independent of f

Proof We notice that when r is determined by equality

1

r

0

ball such that Fðx Þ ¼ min1

2fðxÞTfðxÞ An example of this is (5.2) in which

Combining Theorems (7.1) and (7.2) with Theorems (7.3) and (7.4), and tak-ing L as a constant, the followtak-ing Corollaries are obtained directly

Corollary 7.5 Suppose x* satisfies(1.1), f(x*) = 0, f has a continuous derivative

in B(x*, r) f0(x*) has full rank, and f0satisfies(7.2)and

kf0ðxÞyðf0ðxÞ  f0ðxÞsÞÞk 6 ð1  sÞLkx  x k; 8x 2 Bðx ; rÞ; ð7:12Þ where xs= x* + s(x x*), f0(x)= [f0(x)T f0(x)]T, L is positive number and

and for

q¼Lkx0 x

k

the inequality(7.4)holds Moreover, the given r is the best possible

Corollary 7.6 Suppose x* satisfies(1.1), f(x*) = 0, f has a continuous derivative

in B(x*,r) f0(x*) has full rank, and f0satisfies the center Lipschitz condition:

kf0ðx Þyðf0ðxÞ  f0ðx ÞÞk 6 Lkx  x k; 8x 2 Bðx ; rÞ; ð7:14Þ where f0(x*) = [f0(x*)Tf0(x*)]1f0(x*)T, L is positive number and r = 2/L Then

possible and is independent of the f

Taking

we obtain the following corollaries

Corollary 7.7 Suppose x* satisfies(1.1), f(x*) = 0, has a continuous derivative

in B(x*, r), f0(x*) has full rank, and f0satisfies(7.2)and

kf0ðxÞyðf0ðxÞ  f0ðxsÞÞk

ð1  ckx  x kÞ2

1 ð1  sckx  x kÞ2; 8x 2 Bðx

where xs= x* + s(x x*), f0(x)= [f0(x)T f0(x)]1f0(x)T, c is positive number

5

p

all x02 B(x*, r) and for

k

the inequality(7.4)holds Moreover, the given r is the best possible and is inde-pendent of the f

Corollary 7.8 Suppose x* satisfies(1.1), f(x*) = 0, f has a continuous derivative

in B(x*,r), f0(x*) has full rank, and f0satisfies the center Lipschitz condition:

kf0ðx Þyðf0ðxÞ  f0ðx ÞÞk 6 1

where f0(x*)=[f0(x*)Tf0(x*)]1f0(x*)T, c is positive number and r = 1/2c Then

possible and is independent of the f

For the conditions(7.6), (7.14) and (7.18), if f(x): Rn! Rn(namely: m = n), taking f0(x*)= f0(x*)1, Theorems 7.2 and 7.4 with Corollaries (7.6) and (7.8)

(6.4), so Theorems (7.2) and (7.4) with Corollaries (7.6) and (7.8) are more general

Remark In this section, we used the f(x*) = 0 and the properties of pseudo-inverse of f0(x) in an essential way We expect analogous results to hold for the situation of f(x*)50 under weaker Lipschitz conditions of f0(x)[3,4,11,13,14], but cannot prove those

Using Theorems (7.1) and (7.2), we also derive some new properties in essence about the convergence of Newton
s method and the uniqueness of the solutions of the equation In the following example, let c be a positive number

Example 2 Taking

c

ð1  ckx0 x kÞ2

c

we have

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 4cþ c2

p

and

k

If the right-hand side in(7.18)is replaced by

c

then

8 Applications to determination of an approximation zero

To study the property of quadratic convergence of Newton
s method and

approximation zeros of Newton
s method With the Smale
s studies, we can propose a new definition of the approximation zeros for the Gauss–Newton
s method This definition is as follows

To study the property of quadratic convergence of Newton
s method and

approximation zeros of Newton
s method With the Smale
s studies,...

2fxịTfxị An example of this is (5.2) in which

6 Corollaries of the main results

In the study of the Gauss? ??Newton
s method (or Newton
s method) , the assumption that the... (4.1), we also derive some new properties in es-sence about the convergence of Newton
s method and the uniqueness of the solutions of the equation In the following example, let c be a positive