Interactive and multiorgan based plant species identification =177

For this, we first focus on improving the recognition accuracy the plants of interest even in complex background images; Propose a fusion scheme in multiple organ plant identification; Dev

HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY

NGUYEN THI THANH NHAN

INTERACTIVE AND MULTI-ORGAN BASED

PLANT SPECIES IDENTIFICATION

Major: Computer science

Code: 9480101

THESIS ABSTRACT COMPUTER SCIENCE

Hanoi − 2020

This dissertation is completed at:

Hanoi University of science and technology

Supervisors:

1 Assoc Prof Dr Le Thi Lan

2 Prof Dr Hoang Van Sam

Reviewer 3: Assoc Prof Dr Pham Van Cuong

The dissertation will be defended before approval committee

at Hanoi University of Science and Technology:

Time 13h30, date 06 month 05 year 2020

The dissertation can be found at

1 Ta Quang Buu Library

2 Vietnam National Library

is considered as important key to assess flora knowledge Nowadays, the availability

of relevant technologies (e.g digital cameras and mobile devices), images datasets and

advance techniques in image processing and pattern recognition let the idea of mated plants/species identification become reality The automatic plant identification

can be defined as a process of determining the name of species based on their observedimages

Objective

The main aim of this thesis is to overcome the second limitation of the automaticplant identification (low recognition accuracy) by proposing novel and robust methodsfor plant recognition For this, we first focus on improving the recognition accuracy

the plants of interest even in complex background images;

Propose a fusion scheme in multiple organ plant identification;

Develop image-based plant search module in Vietnamese medicinal plant retrievalapplication

The dissertation has three main contributions as follows:

Contribution 1: A complex background leaf-based plant identification method

has been proposed The proposed method combines the advantages of inter-

active segmentation that helps to determine the leaf region with very few user

interactions and the representative power of Kernel Descriptor (KDES)

Contribution 2: One fusion scheme for two-organ based plant identification

has been introduced The fusion is an integration between a product rule and aclassification-based approach

Contribution 3: Finally, an image-based plant search module has been devel-

oped and deployed in Vietnamese medicinal plant retrieval application namedVnMed

Dissertation outline

Introduction: This section describes the main motivations and objectives of thestudy We also present critical points the research’s context, constraints and

challenges Additionally, the general frame-work and main contributions of thedissertation are also presented

Chapter 1: A Literature Review: This chapter mainly surveys existing works andapproaches proposed for automatic plant identification

Chapter 2: In this chapter, a method for plant identification based on leaf image

is proposed In the proposed method, to extract leaf region from images, we

proposed to apply interactive segmentation Then, the improved KDES (Kernel

DEScriptor) is employ to extract leaf characteristic

Chapter 3: This chapter focuses on multi-organ plant identification We have

proposed different strategy for determining the result of multi-organ identification

based on those of single-organ ones

CHAPTER 1

LITERATURE REVIEW

1.1 Plant identification from images of single organ

There are a large number of automatic plant identification methods Amongdifferent organs of the plant, leaf is the most widely used [4] because leaf usually exists

in a whole year The identification results on leafscan often give the best results when

compared with other organs [5] The popular organ is flower because its appearances(e.g., color, shape, texture) are highly distinguishing [6] In addition, other organs are

used to identify plant such as fruit, stem and branch There are two main approaches

for the plant identification based on image of the plant organs The first one uses thehand-designed feature-based while the second one employs the deep learning method

Hand-designed feature-based method consists of main stages: training and test-

ing Each stage consists of four main components: image acquisition, preprocessing,feature extraction and classification [7] Feature extraction can be considered the most

is no single feature strong enough to separate all categories

The second employs deep learning methods Recently, learning feature

are very simple to extract for lines, curves, or blobs in the input image This infor-

1.2 Plant identification from images of multiple organs

The fact that the state-of-the-art results of the plant identification using a singleorgan are still far from practical requirements Recently, the plant identification movesfrom one sole organ to multi-organ More researches have been dedicated to plant

Multi-organ plant identification can be divided into two groups: The first group

is interested in organs of the plant, the second group does not care about the organs

of the plant In the first approach each organ will be trained separately In the second

approach, all images will be trained together, regardless of which organ they belong

Early or late fusion techniques will be used to combine the results

1.3 Plant data collection and identification systems

There are a number of image-based plant identification applications deployed

on mobile devices such as Pl@ntNet, iNaturalist, iSpot, Leafsnap, FlowerChecker,PlantSnapp, Plantifier, etc [9, 10] These applications often provide three main func-tions that are exploring, identifying, collecting data

Plant identification function and data collection function are two functions that

support each other When a plant identification function obtains highly accuracy, it

will attract more people to use the system and collect more data Collected data will

be diversity and more species Then, these data are used to retrain the system Onewell-known problem in classification is overfitting In order to avoid this, images of thesame species should be diverse and be taken under different protocols This shows the

role of crowdsourcing system

CHAPTER 2

LEAF-BASED PLANT IDENTIFICATION METHOD

BASED ON KERNEL DESCRIPTOR

2.1 The framework of lead-based plant identification method

The framework of leaf-based plant identification is illustrated in Figure 2.1 Themethod consists of three main modules that are image preprocessing, feature extractionand classification

Figure 2.1 The complex background leaf image plant identification framework

lines in and out of an interesting leaf Then, watershed method is used for image

segmentation[11] From the returned results of watershed segmentation, the users can

select the interested leaf region in the third step Finally, leaf-shape is normalized

2.3 Feature extraction

Kernel descriptor (KDES) has been proposed firstly by Liefeng Bo et al [12], allows

to capture various visual attributes of the pixel (e.g using gradient, color and binary

Fingure 2.5 An example of the uniform patch in the original KDES and the adaptive

patch in our method (a,b) two images of the same leaf with different sizes are divided

using uniform patch; (b,c): two images of the same leaf with different sizes are divided

using adaptive patch

shape) and to learn compact features from match kernels via kernel approximation In

[13], Nguyen et al proposed three improvements in KDES extraction Being similar

to KDES, the improved KDES is extracted through 3 steps: pixel level feature tion, patch-level feature extraction, and image-level feature extraction We proposedimproved KDES for feature extraction

extrac-a) Pixel-level features extraction

˜

θ z( ) = [sin θ z cos θ z( ( )) ( ( ))] (2 8).b) Patch-level features extraction

Generate a set of patches from image with adaptive size

In this task, we create patches with adaptive size instead of fixed size We make

an adaptive patch size in order to get a similar number of patches along both horizontal

axis and vertical axis Figure 2.5 describes an example of the uniform patch in the

original KDES and the adaptive patch in our method

Compute patch-level feature

Patch-level features are computed based on the idea of the kernel method Derived

from match kernel representing the similarity of two patches, we can extract feature

vector for the patch using approximative patch-level feature map, given a designed

Fingure 2.7 Construction of image-level eature concatenating feature vectors of cells

in layers of hand pyramid structure

patch level match kernel function The approximative feature over image patch P is

c) Image-level features extraction

Once patch-level features are computed for each patch, the remaining work iscomputing a feature vector representing the whole image To do this, a spatial pyramidstructure dividing the image into cells using horizontal and vertical lines at several

layers Then we compute the feature vector for each cell of the pyramid structure and

concatenate them into a final descriptor The feature map on the pyramid structureis:

2.4 Experimental results

2.4.1 Datasets

We conduct experiments on the following public datasets:

Subset of ImageCLEF 2013 dataset: 5,540 and 1,660 leaf images of 80 species

of ImageCLEF 2013 for training and testing respectively

Flavia dataset 1,907: leaf images on a simple background of 32 species

LifeCLEF 2015 dataset: The Table 2.1 shows detail leaf/leafscan dataset

Table 2.1 Leaf/leafscan dataset of LifeCLEF 2015

Leaf LeafscanTraining 13,367 12,605Testing 2,690 221

Number of species 899 351

2.4.2 Experimental results

Results on ImageCLEF 2013 dataset

The results are shown in Table 2.2 The results show that our improvements on

kernel descriptor extraction make a significant increase of the performance on both in-teractive and automatic segmented images Moreover, the proposed method obtains thebest result On the same dataset, improved KDES outperformed the original KDES

On the same KDES method, interactive segmentation allows to improve significantlythe accuracy

Table 2.2 Accuracy obtained in six experiments on ImageCLEF 2013 dataset

Improved KDES with Interactive segmentation 71.5

Original KDES with Interactive segmentation 63.4

Improved KDES with no segmentation 43.68

Original KDES with no segmentation 43.25

Improved KDES with Automatic segmentation 42.3

Original KDES with Automatic segmentation 35.5

Results on Flavia dataset

The accuracy is 99.06% We compare with other methods on Flavia dataset Theresults are as follows in Table 2.4, the our method is the best, it improve in range [0.36,

6.86]% than other results The results are very high with a simple image dataset.Results on LifeCLEF 2015 dataset

Table 2.4 Comparison of the improved KDES + SVM with the state-of-the-art hand-designed

features-based methods on Flavia datasetMethods Feature, classification method Accuracy(%)Proposed method Improved KDES; SVM 99.06

[15] CT,HU, moments, GF, GLCM; NFC 97.60

[17] GIST features (486), (PCA=40%); cosine KNN 98.7

[19] Geometrical features, invariant moments; RBPNN 94.1

[20] Geometrical features, vein features; SVM 92.2

Figure 2.14 Detailed scores obtained for Leaf Scan [1], our team’s name is Mica

Zenith, QUT RV, Sabanki Okan and Ecouan [1] The proposed method for Leafscan

obtains the second place with the score is0.737 while the score of the first place team

This chapter presents the proposed method for complex background leaf-based

plant identification The obtained results show that the combination of improved

KDES and interactive image segmentation in the proposed method outperform the

original KDES and different state of the art hand-designed feature-based methods

gle organ do not have enough information for the identification task due to the largeinter-class similarity and large intra-class variation Therefore, this chapter aims at

proposing a fusion technique for multi-organ plant identification Without lost of

erality, we present and evaluate fusion schemes for each pair of organs The proposed

classi-Figure 3.2 The framework for multi-organ plant identification

3.2 The proposed fusion scheme RHF

si(Ik) is the confidence score of the plant speciesi−th when using image of organ

k noted Ik as a query for single organ plant identification, where 1 ≤ i ≤ C,

1 ≤k ≤N;

c: the predicted class of the species for the query q

Basic combination techniques

The three rules that are widely used in basic combination techniques are max, sum

and product rules Using these rules, the class cof the queryq is defined as follows:Max rule

c = arg max

i max

k N =1 si(Ik) (3.1)Sum rule

for each species i th− : one for positive denotedPpos(i, q ) and one for negative denoted

Pneg(i, q ) respectively The list of plants that are ranked by si( ) is determined whereq

si( ) is the confidence score of the plant speciesq i−th obtained for the query :q

si( ) = q Ppos(i, q ) (3.4)

Figure 3.3 Explanation for positive and negative samples.

The class c is predicted as follows, where 1 ≤ ≤i C

c = arg max

Robust Hybrid Fusion (RHF)

The above classification-based approach can loose distribution characteristics for

each species because all positive and negative samples of all species are merged and

represented in a metric space only Therefore, we build each species an SVM model

based on its positive and negative samples When we input a pair of organs to ourmodel, we will know the probability that this pair belongs to each species by these

SVM classifiers Then we combine this probability with the confidence score of each

organ As far as we know, q is the query of a pair of two image organs, and si(Ik) is

i-th species confidence score for image Ik Let us denote s i( ) are the confidence scoreq

of a query for -th plant species computed by SVM model The robust hybrid fusionq i

model is formed as follows:

3.3 The choice of classification model for single organ plant

Fingure 3.10 Single organ plant identification

In our experiments, we use two schemes for the network weights that are trained on ImageNet dataset and fine tune the chosen networks with the workingdataset

Flower Leaf Entire Branch TotalCNN Training 1650 1930 825 1388 5793SVM Input 986 1164 495 833 3478

Total 3309 3870 1661 2774 11614

Species number = 50

Table 3.3 Single organ plant identification accuracies (%) with two schemes:

(1) A CNN for each organ; (2) A CNN for all organs

3.4.2 Single organ plant identification results

The results obtained for the two proposed schemes with three networks are shown

in Table 3.3 We can observe that GoogLeNet obtained better results than that of

tion, with AlexNet, the best performance for single organ is 73.0% for flower images,

whereas by applying the proposed RHF, the accuracy rate of a combination between

leaf-flower images dramatically increases by 16.8% to 89.8% When applying ResNet,

the combination of leaf and flower (Le-Fl) improves +17% over the single organ and+13.6% when applying GoogLeNet Not only the leaf-flower pair but in all six pairs ofmulti-organs combination, RHF also retain the high performances Almost the other

fusion performances are also higher than those of single organ

Comparison to MCDCNN (Multi Column Deep Convolutional NeuralNetworks)

To show the effectiveness of the proposed fusion scheme, we compare its

mance with the performance of MCDCNN [24] The obtained results on the same

dataset in Table 3.7 show that the proposed method outperforms MCDCNN in all