Apply incremental learning for daily activity recognition using low lever sensor data

Daily activity recognition is an important task of many applications, especially in an environment like smart home.. 7 3 Framework for Activity Recognition in the Home Environment 10 3.1

VIETNAM NATIONAL UNIVERSITY, HANOI

UNIVERSITY OF ENGINEERING AND TECHNOLOGY

VIETNAM NATIONAL UNIVERSITY, HANOI

UNIVERSITY OF ENGINEERING AND TECHNOLOGY

A thesis submitted in fulfillment of the requirements for the degree ofMaster of Computer Science

Supervisor:

ORIGINALITY STATEMENT

‘I hereby declare that this submission is my own work and to the best of myknowledge it contains no materials previously published or written by anotherperson, or substantial proportions of material which have been accepted for theaward of any other degree or diploma at University of Engineering and Technology

or any other educational institution, except where due acknowledgement is made

in the thesis I also declare that the intellectual content of this thesis is theproduct of my own work, except to the extent that assistance from others in theproject’s design and conception or in style, presentation and linguistic expression

is acknowledged.’

Signed

ii

Daily activity recognition is an important task of many applications, especially

in an environment like smart home The system needs to be equipped with therecognition ability so that it can automatically adapt to resident’s preferences Sofar, a great deal number of learning models has been proposed to classify activities.However, most of them only work well in off-line manner when training data is known

in advance It is known that people’s living habits change over time, therefore alearning technique that should learn new knowledge when there is new data in greatdemand

In this thesis, we improve an existing incremental learning model to solve thisproblem The model, which is traditionally used in unsupervised learning, is ex-tended for classification problems Incremental learning strategy by evaluating theerror classification is applied when deciding whether a new class is generated A sep-arated weighted Euclid is used to measure the distance because the large variance ofinformation contains in feature vectors To avoid constant allocation of new nodes

in the overlapping region, an adaptive insertion constraint is added Finally, anexperiment is carried to assess its performance The results show that the proposedmethod is better than the previous one The proposed method can be integratedinto a smart system, which then pro-actively adapts itself to the changes of humandaily activity pattern

I would like to express my respect and appreciation to my supervisor, AssociateProfessor Bui The Duy He is the person to guide me the approach of the thesis’sproblem He has given me a lot of supports during the progress of my thesis includinghow to do a better experiment and to write a good thesis Next, I would like tothank Dr Vu Thi Hong Nhan for her valuable discussions about the details aspects

of my thesis, and also for her recommendations for providing a background of myresearch I also want to thank for my colleagues and friends in Human MachineLaboratory for their friendly and willingness to help me during the time I studied

ii

1 Multi-Agent Architecture For Smart Home, Viet Cuong Ta, Thi Hong Nhan

Vu, The Duy Bui, The 2012 International Conference on Convergence TechnologyJanuary 26-28, Ho Chi Minh, Vietnam

2 A Breadth-First Search Based Algorithm for Mining Frequent Movements FromSpatiotemporal Databases Thi Hong Nhan Vu, The Duy Bui, Quang Hiep Vu, VietCuong Ta, The 2012 International Conference on Convergence Technology January26-28, Ho Chi Minh, Vietnam

3 Online learning model for daily activity recognition, Viet Cuong Ta, The DuyBui, Thi Hong Nhan Vu, Thi Nhat Thanh Nguyen, Proceedings of The Third In-ternational Workshop on Empathic Computing (IWEC 2012)

iii

Table of Contents

1.1 Overview 2

1.2 Our Works 3

1.3 Thesis Outline 4

2 Related Work 5 2.1 Daily Activity Recognition in Smart Home System 5

2.2 Daily Activity Recognition approach 6

2.3 Incremental learning model 7

3 Framework for Activity Recognition in the Home Environment 10 3.1 Data Acquisition 10

3.2 Data annotation 12

3.3 Feature extraction 12

3.4 Segmentation 14

4 Growing Neural Gas model 15 4.1 GNG structure 15

4.1.1 Competitive Hebbian Learning 17

4.1.2 Weights Adapting 18

4.1.3 New Node Insertion 19

4.2 Using GNG network for supervised learning 23

4.2.1 Separated Weighted Euclid 24

4.2.2 Reduce Overlapping Regions by Using a Local Error Threshold 25 5 Radial Basic Function network 27 5.1 Standard Radial Basic Function 27

5.2 Incremental Radial Basic Function 29

iv

TABLE OF CONTENTS v

7.1 Summary 357.2 Future works 36

List of Figures

3.1 Framework for activity recognition 113.2 An example of markov model for observing an activity Each statehas a normal distribution of time 13

4.1 An example of Voronoi tesselation of the space The lines representthe boundary of the subspaces and the nodes denote the centres ofthe subspaces The dot line represents the edges 164.2 An example of updating weight in GNG The bold lines representthe boundary of the subspaces and the thin lines nodes denote theconnections between the nodes The dot lines represents directionwhich the nodes is moved 19

5.1 The structure of a Radial Basic Function network 28

vi

List of Tables

3.1 An example of sensor events in data stream 12

6.1 Summary of datasets 326.2 Accuracy of the four models 326.3 The F-measure score of four models in the third training phase Thenumber in the brackets is the number of training samples of eachclass The activity labels are arranged in the decreasing order of thenumber of training samples 346.4 The confusion matrix of the incremental RBF model between activityLeave Home and Read 346.5 The confusion matrix of GNG model between activity Leave Homeand Read 346.6 The confusion matrix of new GNG model between activity LeaveHome and Read 34

1

Understanding the users’ activities is one of the key features of a smart homesystem The users’ activities represent the basic information of the environmentwhere the smart home system operates From these information, the system decidesappropriate actions which are most beneficial the users For example, in a homeautomation application, the system will carry out decision to control device in thehouse more accurately if it knows what the user is doing Regularly activities whichare performed in the home can be divided into two types One type of activities ischaracterized as simple gestures of some part of the body such as running or walking.The other one is the set of actions which become a pattern in users’ daily routines.Some examples of this type are reading, cooking or watching television Recognizingthese activities will provide useful information for understanding the environmentcontext In our thesis, we focus on the problem of recognizing the second types ofthe activities.

The most common data source in the smart home system is low-level sensorinformation (Cook et al., 2009) Those data comes from low level sensors Low levelsensors are prefer in the home environment because they are not expensive, easy to

2

1.2 Our Works 3

install and almost transparency in the home environment There have been manypropose approaches for recognizing activities from low level sensory data, includingNaive Bayes (Tapia et al., 2004), neural networks (Chen et al., 2010), Markov Modeland Conditional Random Field (CRF) (Van Kasteren et al., 2008)

One of the properties of daily activities in the home environment is its variationthrough time The activity patterns is usually different after period of time, becausethe user’s habits are change in real life There are many factors which can have affect

of the user’s habits For example, season factors make the daily schedule of the userchange The user may wake up late in the winter than in summer For this reason,learning activity pattern in smart home system is considered as a long-life learningwhere the new knowledge should be acquired during the entire life of the system

To deal with this problem, previous approach must have to train the models againwhen there are changes in activities patterns However, this is a resource consumingprocess

In our works, we apply incremental learning model into the problem of daily activityrecognition to deal with the above problem The incremental learning models arebased on the Growing Neural Gas (GNG) network (Fritzke, 1995) The GNG net-work is a type of neural networks, but it is traditionally used only for unsupervisedproblems such as clustering or vector quantization One of its advantages is that itmaintains the relation between old knowledge and new knowledge This property

is suitable with the variation of daily activities over long period of time When achange occurs, it can easily adapt its structure to the new one but also can refer back

to the old knowledge, i.e activities happened in the past Therefore, the activitiesare recognized well without the need to train the model from scratch

We extend the growing neural gas model for activity recognition in two ways.The first approach is to use multiple GNG networks For each activity class, wetrain a separated network using samples of the class Technically, the weak point

of the traditional growing neural is its network size constantly grows based on theerror evaluation for new sample insertion This learning strategy encounters theperformance problem in case classes overlap We propose the use of another metricfor measure distance in the GNG network and a constraint to the growing condition

1.3 Thesis Outline 4

with the purpose to make the models more balance between each class

The second approach is to create a Radial Basic Function (RBF) network (Moody

& Darken, 1989) from the GNG network This approach is similar to the methodproposed in (Fritzke, 1993) The RBF network receives the GNG network as its hid-den layer Its output layer is a perceptron layer which can be trained incrementally

by adapting the weight with a simple version of gradient descent

We carry out an experiment to compare both models Performance is evaluated

on real daily activity dataset The experimental results show that the task of ognizing daily activity can achieve good result with incremental learning approach

of the RBF network based on the GNG network is presented An experiment withreal data set is presented in Chapter 6 In Chapter 7, we discuss the summary ofour thesis and future works

Chapter 2

Related Work

In this chapter, we present some backgrounds of daily activity recognition problemrelated to the scene of smart home system in Section 2.1 Next, some proposed ap-proaches for recognizing the activities is introduced Then, we review some existingincremental learning method for both supervised and unsupervised approaches

System

In recent years, there are many researches in building a system for smart homeenvironment In these systems, the activity recognition is considered as a part ofthe context recognition module While the context recognition module is refer

to understand a wide range of knowledge in the intelligent environment, the mainpurpose of activity recognition is to extract the information of what the resident isdoing Based on the observed activities, the system can decide supporting decisions

or make future predictions Therefore, the type of activities which are needed torecognize can be varied depends on the main purpose of the system For example,the University of Essex’s intelligent dormitory (Doctor et al., 2005) creates a smartapartment which can support daily residents’ activities The system observes theresident activities such as work, entertaining and so on Then it uses this information

to make decisions like controlling the selected devices in the apartment The GatorTech project (Helal et al., 2005) creates assistive environments that can use formonitoring purposes and other intervention services Instead of identifying whichactions the user perform, it uses actuators - physical devices which residents can

5

2.2 Daily Activity Recognition approach 6

interact Each actuator has a certain intentional effect on a domain The system willdetermine appropriate behaviours for a given context by identifying the intentionaleffects from the actuator The MavHome project (Youngblood et al., 2005) built anenvironment which can act an intelligent agent The system observes daily resident’sactivities by sensors From these activities, frequent patterns of residents’ activitiesare extracted and used to predict the future activities Then, a control policy can belearned to minimize the user interactions with the environment and provide safetyconditions in the environment

The task of recognizing activities depends on which sensory data the system canperceive from the real world There are a variety of sensor types in a smart homesystem Sensors can be used for position measurement, detecting motion, detect-ing sound, reading temperature These sensors usually create many data streams(Madden & Franklin, 2002), which have to be combined in order to produce moreuseful information Besides that, other types of devices produce another sources

of data for activity recognition, such as computers/personal digital assistant(PDA)(M¨uhlenbrock et al., 2004), cameras (L¨uhr et al., 2007) and microphones (Brdiczka

et al., 2005) Although using perception such as cameras and microphones in thesmart home environment brings more information of the context, it also comes withmore challenge in how to process a large amount of data in real-time (L¨uhr et al.,2007) and protect the resident privacy On the other hand, using low level sensorsoffers low cost in building the sensor network and providing a robust systems Thedata generated by low level sensors is quite easy to collect and process in comparison

to other types of device like camera Because of these reasons, low level sensors areprefer in the task of activity recognition in smart environment

There are many proposed models for recognizing daily activities In Tapia et al.,they built an activity recognition model based on Naive Bayes classifiers Theclassifiers use low level sensors These sensors are placed around the environmentfor detecting motion and picking objects The classifiers use temporal features fromthe sensor data stream, which are the activation of a particular sensor and the order

of activation between two sensors There are 35 activity labels for classifying Theresults are mixed and depended on each class With a fair amount of data, the Naive

2.3 Incremental learning model 7

Bayes classifiers can reach the accuracy performance in range of 36% to 89%

In Wren and Tapia (Wren & Tapia, 2006), a multilayer model is used to detectwalking movements from passive infrared motion detectors placed in the environ-ment At level 1, a Naive Bayesian model classifies the binary motion activationevents from data streams into basic movements Some examples of these basicmovements are entering, leaving, turning-top-right At level 2, more context generalactivities, which are chatting, meeting and visiting, are recognized by discrete out-put Hidden Markov Models (HMMs) The HMMs is build for each activity usingobservations in discrete movement detection from level 1

In Van Kasteren et al (Van Kasteren et al., 2008), they proposed an approach

of using Conditional Random Field (CRF) The CRF model is used in the form

of linear-chain which is similar to the HMM but has undirected edges Then themodel is trained by maximizing the conditional probability between the observablevariables and the hidden nodes With a set of 8 activities labels, the CRF reachs70.8% accuracy

In Chen et al (Chen et al., 2010), they proposed a framework for activityrecognition including feature extraction, feature selection and predicting models.The framework use the four popular machine learning methods including BayesBelief Networks, Artificial Neural Networks, Sequential Minimal Optimization andBoosting The experiments are carried out in a number of data sets which are varied

in the setting of the home environment, size of the data and number of activity labels.The highest accuracy is around 90% and can be achieved with Multilayer Perceptron

or LogistBoost

The approach of incremental learning is aim to handle the problem of learning newknowledge while maintaining the previous one (Joshi & Kulkarni, 2012) Incrementallearning is useful in the tasks where input data comes from a non-stationary inputdistribution Throughout the learning process, the decision areas can be change fromdiscrete overlaps into continuous overlaps and decision boundaries can be affected

by new input patterns In an off-line manner, for example artificial neural networks,the model is not flexible enough to deal with those changes Later input patterns canmake the model forget the preserved patterns completely Moreover,in real system

2.3 Incremental learning model 8

such as smart environment, a requirement for learning model is to learn throughoutthe entire life time of the system Training the model again and again could be aresource consuming process Using incremental approach can be used to solve theseproblems

There have been a number of proposed incremental learning for unsupervisedlearning and supervised learning In unsupervised approach, the well known clus-tering algorithm k-means (Macqueen, 1967) can be learned on-line by a weightupdate rule Instead of running the algorithm until it converges, the centroids areupdated continuously and incrementally based on input patterns The winner takeall (WTA) competitive learning is employed for choosing the updated centroid Anartificial neural network, namely Self-Organized Map (SOM) (Kohonen, 1989), ispresented as a unsupervised learning method using incremental approach The net-work contains a layer of neurons, with connections in 2D space It processes theinput pattern one-by-one and maps clusters from high dimensional space to twodimensional space However, the network has fixed structure, including number ofneurons and connections More flexible networks with similar approach are GrowingCell Structures (Fritzke, 1993) and Growing Neural Gas (Fritzke, 1995)

In supervised learning, there are several efforts to adapt the exist off-line method

to use in incremental training Multilayer perceptron network which is trained byerror back-propagation represents the previously learned pattern by the weight ofconnections There are several approaches tries to adapt the weights for a newpattern without forgetting the stored ones (Kruschke, 1992) With the inspiring ofadaptive boost algorithm(AdaBoost) (Freund & Schapire, 1995), Polikar et al pro-posed an incremental learning method, namely Learn++ It uses the MultiLayerPerceptron network as a week learner to generate a number of weak hypotheses.Learn++ then combines them through weighted majority voting The main differ-ence between AdaBoost and Learn++ is their distribution update rule Instead ofoptimizing for classifier accuracy as in AdaBoost, Learn++ distribution update ruleoptimizes for learning new data

Radial Basic Function (RBF) networks (Moody & Darken, 1989) combine a localrepresentation of the input space and a perceptron layer to fit the input pattern withits target However, its local representation of the input space has a fixed number

of nodes and it is hard to determine this number Hamker proposed a method toinsert new nodes into the RBF network (Hamker, 2001) Using this approach, theRBF network is capable for learning overtime

2.3 Incremental learning model 9

Another approach is Fuzzy ARTMAP (Carpenter et al., 1992) which based onAdaptive Resonance Theory (ART) networks The ART networks use a similaritycriterion to match the new pattern with the stored ones If there is no appropriatedpattern, a new node is allocated for representing the new pattern in the network.The Fuzzy ARTMAP system includes an ART networks for input patterns andanother one for theirs label Two networks are connected by an associative memorywith WTA rule for learning and classifying

Chapter 3

Framework for Activity

Recognition in the Home

Environment

In this chapter, we present the framework of daily activity recognition using low levelsensor data in the home environment Its main purpose is to map the data from theenvironment to a sequence of activities We adapt the proposed framework in Chen

et al to emphasize the life-long learning properties of the framework Figure 3.1illustrates the steps in an activity recognition module using low-level sensor data.The information of the surrounding environment comes under a stream of sensorevents Then, the stream can be annotated and split into labelled samples Thelabelled samples are then provided to the incremental learning model for training

In online recognition, the stream is segmented into separated sequences of sensorevents Each separated sequence is then classified by the learned model By applyingincremental learning, the model is trained continuously over the life span of thesystem whenever there is new labelled data

In the data acquisition phase, sensors are implemented around the home The sors are low level, which are differ from cameras or microphones They continuouslymonitor the environment for some specific information When there is any change

sen-in the sensor’s state, it will create a signal and send the signal to the system For

10

3.1 Data Acquisition 11

Figure 3.1: Framework for activity recognition

examples, a low level motion sensor is used to detect the movement near its place

It will remain silent when there is not any movement If there is a motion around

it, and it will create a ”ON” signal and sends the signal back to the system Then,

if it does not detect any motion, it will create another ”OFF” signal and then backs

to the silent state There are many types of information in the home environmentthat can be monitored by state change sensor such as door, picking object or tem-perature Choosing the types of sensor are included totally depends on the system’sdesign

Compare to other receivers like cameras, state change sensors are transparency,low cost, reliable and easy to install However, in the domain of daily activitiesrecognition, they make the problem more challenge Each sensor provides a tinypiece of the context’s information It must be combined many pieces to get infor-mation in a higher abstract level For example, a motion sensor can tell exactly the

”MOVING” action of a user in a specific area On the other hand, if the actionneeds to recognize in the set of ”WORKING” or ”READING”, then it is hardly

to determine the appropriate label from only one motion sensor Moreover, it ispossible to have a great level of variance with low level sensor data Therefore, it isusually required more samples for training

The data from each low level sensor is collected and combined into a stream ofdata at a central server Each sensor’s signal is considered as an event in the stream

3.2 Data annotation 12

Table 3.1: An example of sensor events in data stream

Date Time Sensor Value2009-10-16 08:50:00 M026 ON2009-10-16 08:50:01 M026 OFF2009-10-16 08:50:02 M028 ON2009-10-16 08:50:13 M026 OFF2009-10-16 08:50:17 M026 OFF

An example of event stream (Cook & Schmitter-Edgecombe, 2009) is given in table3.1 Each event has a time stamp, the name of the sensor and its value There areseveral platforms for collecting and processing the data stream in the smart homesystem such as Motes platform (Anastasi et al., 2004) or OSGI platform (Helal et al.,2005)

For the purpose of training model, the stream data needs to be segmented andlabelled into separated activities In a smart home system, data can be annotateddirectly by using devices such as Personal Digital Assistant (Tapia et al., 2004)

An alternative way is labelling the stream data indirectly through a post processingstep The users can label their activities with a visualization tool (Chen et al., 2010).Because the data is captured from various sensors and span over a long time, it isdifficult to determine the right starting and ending point of an activity After thisstep, each sensor event in the stream data is added with an optional action label

Feature extraction can affect the performance of the learning model greatly Because

of the numerous sensors can be placed in the system, it should be able to deal withthe great variance of sensor events in an activity’s label Moreover, it is difficult

to create feature vectors that can include all the spatial, temporal and temporal information in the stream of events There are some frameworks that can

spatial-be employed to extract useful temporal information from the data (Gottfried et al.,2006)

3.3 Feature extraction 13

Figure 3.2: An example of markov model for observing an activity Each state has

a normal distribution of time

Temporal information can be used with Hidden Markov models for building amore accurate model for activity recognition (Rashidi & Cook, 2010) The modelsrepresent each state as a sensor in the environment and associate it with a normaldistribution over time (Figure 3.2) However, this approach is only suitable if thesensor events of the activity has small variance and does not change over time

A simpler approach for extracting temporal information is to use only low-orderbinary relationships (Tapia et al., 2004) The relationship between a sensor activa-tion before another sensor is included to the feature vector Because it is possiblethat the system have many sensors, this approach can result is a very large dimension

of feature vector

Because the daily activities in the home environment usually happen around aspecific area, the motion sensors can produce good features for differing the activityclasses In Chen et al., the activated motion sensors during the period of an activity

is included into the feature vector The feature vector also uses high contextualinformation such as day of week, time of day, previous activitiy, next activity andenergy consumption This approach achieves a high accuracy with learning modelssuch as multilayer perceptron or decision tree

In our approach, we use a similar method to extract feature for training withonline learning model A feature vector of an activity sample includes:

• Length of the activity in second

• Part of the day when the activity happens It is divided into six separated

3.4 Segmentation 14

parts: morning, noon, afternoon, evening, night, latenight,

• Day of week

• Previous performed activity

• Number of motion sensors are activated during the activity’s period, and thenumber of value ”ON”

• A list of motion sensors which are activated

• Energy consumption

In a real application of activity recognition, the stream of sensor events is required

to be segmented into separated subsequences before the recognition models canclassify them In Rashidi and Cook (Rashidi & Cook, 2009), they mine the datastream to discover the trigger patterns and use these patterns to determine thestart of an activity However, the trigger patterns are not always clear enough todiscover because daily activities are often change and overlapped with each other

An alternative approach is to use sliding time windows (Rashidi et al., 2011) Thedata stream is split into fixed-time windows and then classify them into one of theactivity classes Although this method can generate a sequence of activity labelsfrom data stream effectively, it may have difficulty when the time window overlapswith more than one activity

Chapter 4

Growing Neural Gas model

In this chapter, we present an incremental learning model which is improved fromGrowing Neural Gas (GNG) networks (Fritzke, 1995) The GNG network is anunsupervised learning and has been applied into vector quantization, clustering,and interpolation Its structure is similar to SOM (Kohonen, 1989) and Neural Gas(Martinetz & Schulten, 1991) The network produces a local representation of theinput space which is learned incrementally In the first section, we introduce thestructure and learning rule of the network in unsupervised learning Then, we extendits structure into the supervised problems with some modifications for improving theperformance

Traditionally, GNG is used in unsupervised learning for vector quantization Thestructure of GNG is a network which contains nodes and edges, like SOM network.The network represents a mapping from the input space to a lower dimension space.During learning process, the network divides the input space into subspaces, known

as Voronoi polyhedra The Voronoi polyhedras together create the Voronoi tion of the input space (as can be seen in Figure 4.1) Each node becomes a center

tessela-of these Voronoi polyhedras An input vector is matched to a polyhedra if the hedra’s center is closest to it The edge between a pair of nodes indicates that thetwo correspondent polyhedras are adjacent

poly-Let n be the dimension of the input space and V = Rn is the input space, thenthe GNG model is a network G consists of:

15