Finding interesting photos in albums using visual attention

74 4.8 Attention values for di erent image ategories... Immediate attention Passive attention Active attention Sensorial attention Intellectual attention Attention Types Derived att

ALBUMS USING VISUAL ATTENTION

KARTHIKEYAN VAIAPURY

2007

ALBUMS USING VISUAL ATTENTION

KARTHIKEYAN VAIAPURY

B.Te Te gy), Bharathidasan University, India

2007

Iwould like to express my and heartful thanksto myadvisorProf.

Mohan S Kankanhalli of the Department of Computer at hool

of Computing, National University of Singapore, for his immense

I thank almighty for his divine and strength to nish

this h work on time

I would like to render my thanks to hool of Computing for

givingme t opportunityto pursuehigherstudies at NUS

IwouldalsoliketothankmyseniorsPradeep,Vivek, ShivaandWilliam

fortheir t support

Iamhighlyindebtfultomyparents, sistersandfriendsfortheirsupport

and

knowledgements 3

1.1 Intro and kgroundto 10

1.2 Interestingness 14

1.2.1 Attention 16

1.3 Motivation 19

1.4 Problem Statement and e 21

1.5 Overviewof thesis 22

2 Related Works 24 2.1 VisualAttention models 25

2.1.1 Bottom-Up model 26

2.1.2 Top-Down Model 29

2.1.3 Hybrid: Bottom-Up +Top-Down Model 30

2.1.4 of Attention models 31

2.2 Interestingness 32

2.4 Non-Iden 37

2.5 General 38

2.6 The Itti-Ko h Visualattention model 39

2.7 BayesianSurprisetheory 46

2.8 Pseudo Relev F k Algorithm 47

2.9 SIFTmethod 48

3 Attention-based Interestingness 50 3.1 Overviewof theFramework 50

3.2 Feature pro 53

3.2.1 Itti-Ko h Based Attention 53

3.2.2 F Based Attention 56

3.2.3 SIFTbased attention 58

3.2.4 Groupbased attention 58

3.3 NonIden 59

3.4 Query Analysisand Retrieval(Relev F k) 60

4 Implementation and Results 65 4.1 Software 66

4.1.1 ImplementationPlatform 67

4.2 aboutin 67

4.3 Experimental Resultsand 68

4.3.1 Illustrationof attention values 71

4.3.2 Illustrationofattentionfeature with exam-ples 74

4.4 P 79

4.5 User Study 81

5 and Future Work 91

5.1 Summary 91

5.2 forFuture work 93

1.1 Types of attention (Sour e: James [28 ℄) 15

1.2 Relationship betweenInterestingness and Attention 16

1.3 Human Brain (Soure: Edgingtonet al [9℄) 17

2.1 Attention e Attributes(Sour e: Wolfe et al [70 ℄) 26

2.2 Itti-Ko Model [Sour e: Walther [66, 67 ℄℄ 40

2.3 GaussianPyramid 41

3.1 Proposed FrameworkforEChr AttentionSystem: nd-ing interesting Images 51

3.2 GUI of the system 54

3.3 F e dete and position weights 57

4.1 Software eof Attention System 66

4.2 Attention system: Initial Round 69

4.3 Attention system: Seond Round 69

4.4 Attention system: Third Round 70

4.5 Attention value graph 70

4.6 plotted on Sample ImagesDataset 73

4.7 Example map 74

4.8 Attention values for dierent image ategories 76

4.10 F e oordinate position 78

4.11 Group photo sample image 78

4.12 SIFT points of an image 79

4.13 Average time for omputation Vs Attention methods 80

4.14 Runtime for omputation of RFB 80

4.15 Results: User Study EChr Attention System Vs 84 4.16 Results: User Study EChr Attention with and without RFB 85

4.17 NIDsand Non-NIDs using SIFT 87

4.18 NID for sample imageset 89

4.19 NID matrix 89

4.20 NID Imagesin our dataset 90

2.1 A omparative table the state of the art 30

2.2 A omparative table the state of the art -

interest-ingness 31

4.1 Attention values for a sample of 10 images from ourdataset 71

4.2 User Study Results: PartI 82

4.3 User Study Results: PartII 84

4.4 SIFT for NIDS : Sample Images 88

A is an extended t, in prose or verse, of events,

sometimes legendarymaterial,presentedina order

and withoutauthorialinterpretation or t Due to adv in

sen-sor pro and storage hnologies, there has been a transition from

arethe haremediatedeitherintheformofvideos orphotos

and sometimes even do ts pro b the authors or others The

digitally in various phasesduring the lifetime of organizations or

individualsorgroups bebroadly intopersonal

harethemediated ofpersonallivesand

organi-zational harethemediated hival ofinstitutions

and orations Dueto people's interest inrememberingorgainingfrom

role inourlifeaswellasin manyasp of so y Personal

ilymembers theirlifeand events has wedding,birthday,

vo The reasons behind the sustained growth of family

media

1 Availabilityof hnologythatisneededfor andstoringthe

exp

2 Aordable of heaperdigital

Ingeneral,asstatedb Kimetal an system following

asp [32 ℄

1 datausingmultiplesensors,

2 Supporting htags for and presentationof appropriate

infor-mation,

3 Providing to dataat multiplelevels ofgranularityand

tion

In data using multiple sensors is an ongoing

pro-in most families Indeed, hers also have shown keen interest in

an individual's whole life and then mining the important events

[12 , 14 , 13 ℄ Thus the family needs to supportfor h

group-ings and providing to these groupings via web at multiple levels of

granularity and using appropriate hanisms The

-mediaof good quality(e.g 10 megapixels resolution) h are

sub-sequently stored in its onboard memory or memory reader and

thendownloaded to a digitalmedia album thatpossibly resideson a

be viewed, pro or exported to multiple image le formats In

oldendays,usinganalog photoswere with lmrolls

andthenpro ina h Theadvantagesofmedia with

digital overanalog betterresolutionandeasy

of media Withthemedia b analog the

 itdoesnotsupporteasy makingof multiple of media

 itdoesnothavetheabilitytoprovideinstant kof

quality

2 Media annotation and representation: Thestored digitalimages need

to beannotated h is theextension of interpretation Annotation

is a e form of metadata that assists users in the reuse and

ositionofmedia Ithelpstoidentify informationofthe

andmakesdatamore manageableto identifyand explorethe

available Metadata makes media hives more

3 Media querying: Posingthe h ts that helpin

eÆ-t hing isa problemfor large

4 Media retrieval: Media retrieval is the pro that enables users to

media Content-based retrieval is an entire area of

multipleusersto fromthesame is hallenginginterms

5 Media presentation: Customizing the presentation through intuitive

in thereby ofappropriate information to all

typesof users The key idea is to minimizeinformationoverloadand

present onlyrelevantinformation

6 Media sharing: Sharing of media to multiple users dierent

7 Mediase Providing mediaat dierent levelsfordierent

people at multiplelevels ofgranularitiesis Thisis b

dierent people have dierent trust levels establishedwith theowner

ofthe

As Gray [15 ℄ has we are on the verge of realizing Bush,

Bab-bage and Turing visions to develop a system that organizes

indexes,digests,evaluatesand summarizesinformation Ithasbeenargued

that organization isa humanneed - \Even ifimproved h means

we always ndtheinformationweneed,wemay tinueto organizeit

for other reasons to support serendipitous browsing and provide

the of puttingourthingsinorder"[63 ℄

Indeed,whenthereislargepileofphotos,peopleprefertoseeimagesthat

seeminterestingto themthanto tediously handretrieve imagesbased

on tentbasedretrieval Forexample,whenpeopleremember

theylookintoentiretemporal ofimages,sayamarriageeventatsome

timeperiod,andat astep onelevelfurther, theyprefertosee\interesting"

fromperson topersonduetotheirindividualinterests, text, exp

and

1.2 Interestingness

Interestingness is thepower of orholdingone's attention Based

on one's intention, one willpay attention Intention is anob e or goal

a person is willingto Usually, attention is driven b intention

hinturnisdrivenb interestingness Assumingthatintentionisrelated

tointerestingness, wede neinterestingnessasan entitythatarises from

1 interpretation and exp

2 surprise,

3 beauty,

5 desirability

The items 2,3,4 and 5 are based on how one interprets and his/her

mulation ofexp asembodied inthehuman system It

be thatnotalltypesofattentionareasso withinterestingness

(refer gure1.1)

Asstatedin[28 ℄, attention be into followingsixtypes:

 Sensorial Attention: It refers to the attention towards ob that

makes reasonablesenseto a person

 Intelle Attention: Theattentiontowardsrepresentedob that

isknownto aperson is asIn attention

Immediate attention

Passive attention

Active attention

Sensorial

attention

Intellectual attention

Attention Types

Derived attention

Figure1.1: Types of attention (Sour e: James [28 ℄)

 Immediate Attention: It is the attention that is drawn within short

timeperiod

 Derived Attention: When the orstimulusis interestinginitself,

itsinterestto asso withsome otherimmediatelyinterestingthing

isknownasderivedattention

 Passive attention: It is the attention that arises from non-voluntary,

eortless

 A attention: The attention that arises from voluntary is

as e attention Intention is tralto of voluntary

[35 ℄

In thisworkisinspiredb the that trolof

wouldbepossibleifattentiontointentionis binedasasingle hanism

[35 ℄ Attentionisthepro of ely tratingon partsof

theenvironment etoothers Intentionistheunderlyingsp

pur-posethatmayleadtoan inorderto thegoal In one

wayof intention beprobablyattendingtowardsob

ofrelevanttype Therelationshipbetweeninterestingnessandattention

beseenin gure1.2 Asone seefrom gure1.2,thereexistsrelationship

Figure1.2: Relationship betweenInterestingness and Attention

betweendesirability, beautyand eattention attention

is paid voluntarily Similarly, surprise has the ability to make us

at-tendimmediately, itisrelatedto immediateattention Also, sensorial

and in attention arise from one's interpretation and thinking that

dependson one's kgroundandexp interpretationandexp

isrelated to sensorialand in attention

interestingob are often attendedb human beings[28 ℄, now

we would provide a brief intro to the phenomenon of attention in

1.2.1

1.2.1 Attention

Attention is the e pro in h a person trates on some

featuresoftheenvironmenttotherelative oftheothers It also

powers) upon anob b or observingor listening[39 , 40 ℄

to[28 ℄,\itisviewedasthe takingpossessionby themind in ar

and vivid form, of one out of what seem several simultaneously possible

ob-je or trains of thought It implies withdrawal from some things in order

to deal ee withothers" Thesmallpart of visual

informa-tion hes shorttermmemory and visualawareness providing theability

to investigate [30 ℄ It has survival value to keep an eye on every

thing that is happening even if is lost This trade-o is a part

of the phenomenon of attention [61 ℄ Attention in images is based on the

that realimages often tainvast areas of t datafrom the

persp e of ifwe attend to therelevant parts, the

image beinterpretedmore klyusingless Thepro b

h people attend ob based on theirown interestis attention

Ingeneral, what ob isattendedto and where theattention islikelyto

be is trolled b the ventral and dorsal pathway resp ely [23 ℄ The

dorsalandventralpathwayofthehumanbrain itsroleinattention

pro isshown in ... Usually, attention is driven b intention

hinturnisdrivenb interestingness Assumingthatintentionisrelated

tointerestingness, wedeneinterestingnessasan entitythatarises from

1 interpretation... versionoftheimagerst[20 ℄,

5 Meeting Indexing: Estimatingheadpose gazing [18 ℄ and

6 Dete Visual Eventsin Underwater Video: Findingvisualevents

usingan bottom-upattention model [9 ,11 ,... eknowledgeabout

theexistingvisualattention models

2 Interestingness-Aftergettingadequateknowledgefromtheexisting

visualattention models,westudied interestingness his primarily

basedon