Assessing the feasibility of increasing spatial resolution of remotely sensed image using HNN super-resolution mapping combined with a forward model

The soft-classified land cover proportions obtained using a fuzzy c-means classification are then used as input data for a Hopfield neural netw ork (HNN) to predi[r]

VNƯ Jo u rn al o f Science, E a rth Sciences 28 (2012) 264-275

Assessing the feasibility o f increasing spatial resolution o f remotely sensed image using HNN super-resolution mapping

combined with a forward model

Nguyen Quang Minh*

Faculty o f Surveying and Mapping, Hanoi University o f M ining and Geology

Received 03 September 2012 Revised 24 September 2012; accepted 15 October 2012

A bstract Spatial resolution o f land covers from remotely sensed images can be increased using super-resolution m apping techniques for soft-classified land cover proportions A further development o f super-resolution mapping techniques is downscaling the original remotely sensed image usmg super-resolution mapping techniques with a forward model In this paper, the model for increasing spatial resolution o f remote sensing multispectral image is tested with real SPO T 5 imagery for an area in Bac Giang Province, Vietnam in order to evaluate the feasibility o f application o f this model to the real imagery The soft-classified land cover proportions obtained using a fuzzy c-means classification are then used as input data for a Hopfield neural netw ork (HNN) to predict the m ultispecừal images at sub-pixel spatial resolution Visually, the resulted image is compared with a SPOT 5 panchromatic image acquired at the same time with the multispecừal data The predicted image is apparently sharper than the original coarse spatial resolution image

Keywords: Hopfield neural network optimisation, soft classification, image downscaling, forward

model

and B ishop [2] A lthough w idely applied in Spatial resolution o f im age and photos can im age processing, these approaches are hardly

be increased by the super-resolution algorithm s applicable for super-resolution o f rem otely

In the im age processing context, im age super- sensed m ultispectral (M S) im agery because o f resolution com m only refers to the process o f the lack o f a sequence o f im ages o f the scene at using a set o f cross-correlated coarse spatial the same or sim ilar tim es The only feasible resolution im ages o f the sam e scene to obtain a application o f the super-resolution approaches single higher spatial resolution im age T here are using im age sequences is for hyperspectral num erous studies on such super-resolution im agery [3] F or o ther com m on m ultispectral

rem otely sensed im agery, only few m ethods for

* Tel-84-982721243 in cre asin g th e sp a tia l re so lu tio n to sub-pixel E-mail: nguyenquangminh@humg.edu.vn

2 64

N.Q Mirth / V N U Journal of Science, Earth Sciences 28 (2012) 264-275 265

level have been proposed such as a Point

Spread Function-derived convolution filter [4],

segm entation technique [5], and geostatistical

m ethod [6],

Sub-pixel spatial resolution land cover

m aps can be predicted using super-resolution

m apping techniques The input data for super­

resolution m apping are com m only the land

cover proportions estim ated by soft-

classification [7], T here is a list o f super­

resolution m apping techniques have been

inừoduced including spatial dependence

m axim isation [8], linear optim isation

techniques [9], H opfield neural netw ork (H NN )

optim isation [10], tw o-point histogram

optim isation [1 1], genetic algorithm s [12] and

feed-forw ard neural netw orks [13], The

supplem entary data are also supplied to H N N to

produce m ore accurate sub-pixel land cover

m aps such as m ultiple sub-pixel shifted im age

[14], fused and panchrom atic (PA N ) im agery

[15,16] These latter approaches produce a

synthetic M S or PA N im age as an interm ediate

step for super-resolution m apping based on a

forw ard m odel and then these im ages are

com pared w ith the predicted and observed MS

or PAN im ages to produce an accurate sub­

pixel image classification

The creation o f the predicted M S an d then

PA N im age b y a forw ard m odel suggested a

possibility to im plem ent a super-resolution for

the M S image A m ethod for increasing the

spatial resolution o f the original M S image IS

inừoduced by N guyen Q uang M inh et al [17]

T he new m odel is based on the H NN super­ resolution m apping technique from unsupervised soft-classification com bined w ith

a forw ard m odel using local end-m em ber spectra [15,16] T he m ethod is exam ined w ith a degraded rem ote sensing im age and both visual and statistical evaluations show n a good result

H ow ever, there still exist som e concerns about the feasibility o f the m odel because it is not tested in a m ore com plicated landscape w ith different kinds o f land cover features w hich are varying in sizes and shapes as well as specfral characteristics This paper, therefore, is to

im plem ent the test o f the algorithm in a com plicated landscape

2 G eneral m odel

T he proposed m odel is an extension o f the super-resolution m apping approach based on

H N N optim isation T he prediction o f a MS

im age at the sub-pixel spatial resolution is based on a forw ard m odel w ith local specfra as

w as used in N guyen et al., 2006 [15], In addition to the goal functions and the proportion con sừ ain t o f the H N N for super- resolution m apping, a reflectance constraint is used to retain the brighừiess values o f the original M S im age

266 N.Q M in h / V N U Journal of Science, Earth Sciences 28 (2012) 264-275

Spatial SR MS image (2 0 m) convolution

Figure 1 General model for super-resolution MS imagery prediction

Synthetic MS image (4 0 m )

Figure 1 presents the H N N sub-pixel MS

im age prediction algorithm T he procedure is as

follow s: From the M S im ages at the original

M S spatial resolution, land cover area

proportion images are predicted using a soft-

classifier A set o f local end-m em ber specừ a

v a lu e s is calc u la te d b a se d o n th e e stim a te d land

cover proportions and the original M S image

L and cover proportions are then used to

constrain the H N N for super-resolution

m apping w ith a zoom factor z to produce the

land cover map at the sub-pixel spatial

resolution From the super-resolution land

cover m ap at the first iteration, an estim ated MS

im age (at the sub-pixel spatial resolution) is

then produced using a forw ard m odel and the

estim ated local end-m em ber spectra The

estim ated M S image is then convolved spatially

to create a synthetic M S im age at the coarse

spatial resolution o f the original image Follow ing a com parison o f the observed and synthetic M S im ages, an error value is produced to retain the brightness value o f the pixels o f the original M S image The process is repeated until the energy function o f the HNN

is m inim ised and the synthetic M S im age is generated

A dem onsfration o f the algorithm for an image o f 2x2 pixels can be d escn bed in Figure

2 Firstly, the soft-classification predicts land cover proportion as in Figure lb from the MS specừal im age as in Figure la There are two land covers in this im age called Class A and Class B From the land cover proportions in Figure lb , the land cover classes at sub-pixel level are predicted as in Figure Ic w here a pixel

is divided into 4 x 4 sub-pixels and the 2x2 pixels im age is super-resolved to 16x16 pixels

N.Q M in h Ị V N U Journal of Science, Earth Sciences 28 (20Ĩ2) 264-275 267

land cover image o f Class A and Class B The

brightness o f the new 16x16 pixels image is

predicted using end-m em ber spectra (standard

brightness for the C lass A and B in this area o f

the image) For exam ple, the brightness o f the

pure pixel o f Class A is 35 and Class B is 50

and it is possible to produce a new spectra! image by assigning all the sub-pixels belonging

to Class A the brightness value o f 35 and the sub-pixels o f C lass B the brighừiess value o f 50

as in Figure Id

100% land coverA

50% land

c o v erA 50% land

c o v er B

62.5% land cover A 37.5% land cover B 100% land coverB

(b) Figure 1 Creation o f !6><16 pixels image from 2x2 pixels image

2.Ỉ Soft-classification f o r super-resolution

m apping o f M S im agery

Soft-classification is an intennediate step in

the sub-pixel M S im age predictio n process The

prediction o f the M S im age based on super­

resolution m apping requires land cover

proportions w hich are obtained from soft-

classi fication as input data C onventionally,

there m ust be a set o f training data for m ost o f

the soft-classifiers A ccordingly, it is necessary

to have som e prior infom iation about the

specừal distribution o f land cover classes in the

M S bands, although training data are not

alw ays available for the im age A nd som etim es,

there is a requirem ent o f increasing the spatial

resolution o f the im age w itho ut concerning the

lan d cover classes in the im age scene In these

cases, the algorithm can be im plem ented w ith

unsupervised soft-classified land cover

proportions such as fuzzy c-means classifier [18]

Supervised soft-classifiers could also be

used, such as B ayesian, neural netw ork or k 'N N

classifiers H ow ever, the training data for these

soft-classification techniques should be

obtained from the unsupervised classifications

In the research im plem ented by N guyen Q uang

M inh et al [17], a test for algorithm w as

im plem ented w ith a set o f degraded MS im age and soft-classified land cover proportions w as obtained using k-N N classification using a fraining data set ex ừ acted from unsupervised Interactive Self-O rganising D ata (ISO D A TA ) classifier In this case, training data w ere clustered in the reference image In this experim ent, a fuzzy c-m eans classification is used to predict land cover proportions o f a real SPO T im age to produce super-resolved specừ al image o f different spatial resolutions to evaluate the algorithm

2.2 F orw ard m odel a nd end-m em ber spectra

A fter the first iteration o f the H N N algorithm , once the sub-pixel classification is obtained, a forw ard m odel is used to p ro duce a sub-pixel M S im age from the sub-pixel land cover classes The brighừiess value (e.g., reflectance, radiance, digital num ber) o f a sub­

pixel {m,n) o f a spectral band 5 can be predicted as

268 N.Q M in h / V N U journal o f Science, Earth Sciences 28 (2012) 264-275

w here Ve is the output neuron o f the class e and

Ss e is th e end-m em ber spectra o f the land cover

cla ss e for a spectral band s A s presented in

N guyen et al., 2006 [15], the end-m em ber

specfra vector Ss (Ss = [5^ o f the

original pixel (x,y) o f the specừ al band A’ can be

e s tim a te d lo c a lly u s in g th e p re d ic te d la n d c o v e r

class proportions and the M S im age at the

original coarse spatial resolution as

S ,= iP ^ W P )~ 'w P ^ R ,, (2)

where p is a m aừix o f land cover proportions wiửi

r ^ x -\) { y - \)

and w is the m atrix o f w eights w ith

w =

3 E xp erim en t condition 5.7 Data

T he experim ent in N guyen Q uang M inh et

al [17] is conducted in an area having many large objects w ith linear boundaries It m ay lead

to a concern that th e algorithm proposed in this

p aper is able to w o rk w ell only w ith some specific landscapes Therefore, a second data set is used for testin g the algorithm in a m ore com plicated landscape This im age was obtained in B ac G iang Province, Vietnam

T he SPO T 5 im age used in this test was acquired in A ug ust 2011 w ith the spatial resolution o f 10m and four spectral bands (Figure 2) T he test im age is registered to

W G S-1984 U TM m ap projection in Zone 48N and the location is at 21°17’53.65"N ,

1 0 6 °H 7 6 4 " E T he test im age covers 1 square kilom etre area o f 102x102 pixels To evaluate the results o f increasing the spatial resolution algorithm , a 2.5m spatial resolution panchrom atic im age acquired at the sam e time was used (Figure 3b)

N.Q M in h / V N U journal of Science, Earth Sciences 28 (2012) 264-275 269

Figure 2 SPOT 5 image in Bac Giang Province, Viettiam: (a) Band 1, (b) Band 2, (c) Band 3 and (d) Band 4.

3.2 Soft-classification

The land cover proportions are estim ated

from 10m spatial resolution SPO T 5 im age

using fuzzy c-m eans classifiers so it is not

necessary to have prior understanding about

land cover classes in the area The soft- classified land cover proportions o f five land cover classes and six land cover classes are obtained as in Figure 3c and Figure 3d, respectively

Figure 3 (a) original image, (b) Panchromatic image at 2.5m spatial resolution and land cover proportions from

fiizzy c-means classification: (c) 5 land cover classes and (d) 6 land cover classes

270 N.Q M in h / V N U Journal o f Science, Earth Sciences 28 (2012) 264-275

4 R esults and discussions

4.1 R esuits

The m eth o d o f increasing the M S image

using H N N an d forw ard m odel w as applied to

super-resolve the 10m SPO T 5 M S im ages to

p red ict M S im age at spatial resolutions o f 5m

(zoom factor o f 2), 3.3m (zoom factor o f 3) and

2.5m (zoom factor o f 4) The predicted soft- classified proportions w ere used to consừain

the H N N w ith w eighting factors o f kị = 100, k:

=1 0 0, k i = 100 and 100 to predict the sub­ pixel land cover and then the M S image The false colours com positions using B and 1, Band

2 and B and 4 as R ed, G reen and Blue are show n in Figure 4

Figure 4 Super-resolution o f 10m SPOT 5 multispectral image: (a) 5m super-resolved image using 5 land cover classes, (b) 3.3m super-resolution image using 5 land cover classes, (c) 2.5m super-resolution image using 5 land cover classes, (d) 5m super-resolution image using 6 land cover classes, (e) 3.3m super-resolution image using 6

land cover classes, and (Í) 2.5m super-resolution image o f 6 land cover classes

N.Q M in h / V N l ỉ Journal o f Science, Earth Sciences 28 (2012) 264-275 271

4.2 Evaluation

In the case o f degraded image as in N guyen

Q uang Minh et al [17], the dow nscaled

imagery was com pared with the reference

m ultispectral image at finer spatial resolution to

obtain visual and qualitative evaluations In this

experim ent, the finer m ultispectral im agery is

not available for the full assessm ent Therefore,

the predicted m ultispectral im age at finer

resolution can be only com pared w ith 2.5m

panchrom atic im age for a visual evaluation

The visual com parison o f the super­

resolved image from the real SPOT 5 data with

the panchrom atic im age (Figure 3b) also shows

an im provem ent in sharpness o f the results The

objects in Figure 4(a-f) are sharpened and look

clearer that o f original im age ( r ig u r e 3a)

A lthough the landscape o f im age area is com plicated w ith sm all and linear features such

as houses and roads, the im provem ent o f the algorithm can be seen in the b ou ndaries o f between the objects Figure 5 show s the

im provem ent o f the algorithm for increasing the spatial resolution o f M S im age using H N N with

a forw ard m odel to the original im age The boundaries o f ponds in the centre o f the original

M S image (Figure 5a) are blurred and fragm ented because o f the m ixing o f the land categories in these boundary pixels In the predicted M S im age using HNN and a forw ard

m odel (Figure 5b), these boundaries are clear and look m ore sim ilar to the real ponds in panchrom atic im age (Figure 5c)

Figure 5 Some land cover features in (a) original MS image (false colour composite), (b) increased resolution

MS image to 2.5m spatial resolution from 5 land cover class proportions (false colour composite)

and (c) panchromatic image

F or sm all objects such as houses and roads,

there are few objects w hich is not clearly seen

in the original im age can be recognised in the

super-resolved image In Figure 6a (com posite

image using Band 1, Band 2 and Band 4 o f the

original MS im age), the road is difficult to be

recognised because it is fragm ented due to the

m ixed pixels effect U sing the H NN super

resolution m apping and then the forw ard m odel

as in Figure 6b (com posite im age u sing B and 1, Band 2 and B and 4 o f the 2.5m spatial resolution increased im age), it is possible to recreate the road sim ilar to the shape o f the real road shown in the panchrom atic im age Figure

6c

For the sm all features such as a group o f houses in Figure 6c, the perform ance o f the algorithm is not as good as that for the road

272 N.Q M inh / V N U Journal of Science, Earth Sciences 28 ( 2 0 W 264-275

However, the new ly proposed algorithm still

show s some im provem ent in defining clear

boundaries o f these features T his m ay be

because o f the soft-classifier cannot define the

houses as a separate class T his problem m ay be

resolved by increasing the num ber o f classes for fuzzy c-m eans classifier or using prior inform ation on these classes in supervised soft- classifiers

Figure 6 Some land cover features such as roads and houses in (a) original image, (b) spatial resolution

increased image (false colour composite) and (c) panchrom atic image

4.3 D iscussions

The effect o f zoom factor to spatial

resolution increasing algorithm can be seen in

Figure 8 C om paring the im age created by

H N N using zoom factor o f 2 (Figure 4a), with

the image created w ith zoom factor o f 3 (Figure

4b) and 4 (Figure 4c), it is possible to see that

w hen the zoom factor increases, the boundaries betw een the features are sm oother The boundaries betw een the ponds and the surrounding features are fragm ented in Figure

8 a and F igure 8 b and look sm oother and clearer in F igure 8 c

Figure 8 Effect o f zoom factor to spatial resolution increasing algorithm: (a) zoom factor of 2, (b) zoom factor

o f 3 and (c) zoom factor o f 4

In spite o f increasing the spatial resolution

o f the rem otely sensed M S im ages, the

proposed m ethod has a problem w ith pixels that

belong to the sam e class (referred to as pure pixels in this paper) A lthough the problem can

be partly solved by increasing the num ber o f

N.Q M in h / V N U Journal of Science, Earth Sciences 28 (2012) 264-275 273

classes so there m ore m ixed pixels or dividing a

class into sub-classes so several pure pixels are

defined as m ixed pixels, there w ill still exist

pure pixels The effect o f num ber o f land cover

classes can be seen in Figure 9 The ponds in

Figure 9 a apparently sm aller than those in

Figure 9.b due to som e “pure pixels” in the boundaries w ere re-classified as m ixed pixels

w hen the num ber o f classes increased from 5 to

6 These m ixed pixels are then super-resolved

to produce different boundaries betw een the same features in the tw o im ages

Figure 9 (a) Result from HNN using 5 land cover classes, and (b) result from HNN using 6 land cover classes

B ecause the H N N super-resolution m ethod

w orks only on m ixed pixels, w hich are usually

located across the border betw een different

classes, it is suggested that the m ethod is

suitable for the super-resolution o f im ages o f

large objects, for exam ple the agricultural

scenes In these im ages, spatial variation is

hom ogeneous w ithin the land p arcels and

super-resolution based on the spatial clustering

goal functions o f the H N N can the agricultural

field boundaries or increasing the sharpness o f

linear features such as roads or canals

As m entioned above, the use o f

unsupervised classification can reduce the

eư o rs in land cover class p roportion prediction

Furtherm ore, the use o f unsupervised

classification facilitates the autom ation o f the

spatial resolution increasing process because

the class p roportions can b e obtained w ithout

fraining data and w ithout a fraining step

Through the choice o f the num ber o f classes, the user can control the effect o f the super­ resolution algorithm on the resulting sub-pixel

M S im ages W hen the num ber o f classes is changed, the num ber o f m ixed pixels m ay be changed as a result

A draw back o f the H N N super-resolution procedure is the subjective choice o f the param eters for the goal functions, the proportion co n sừ ain t and the m ulti-class constraint [15] B y em pirical investigation, the values o f the param eters should retain an equal effect betw een the con sừ ain ts and the goal functions in the optim isation process For exam ple, the em pirical investigation in this paper show s th at the values o f these param eters

in this paper w ere sim ilar and around the value

o f 100 T he finding is also obtained from the

N guyen Q uang M inh et al [17]