MẠNG NEURAL TRONG hệ THỐNG điều KHIỂN XE lăn CHO NGƯỜI tàn tật NẶNG sử DỤNG điện não (EEG) và CAMER tt tieng anh

In this thesis, author researched and analyzed three EEG signal pre-processing methods as Fourier transform, Wavelet transform and HHT transform, converting EEG signal to 5 basic EEG wav

MINISTRY OF EDUCATION AND TRAINING MINISTRY OF TRANSPORT

HO CHI MINH CITY UNIVERSITY OF TRANSPORT

LÂM QUANG CHUYÊN

NEURAL NETWORK IN THE WHEELCHAIR CONTROL SYSTEM FOR SEVERE DISABILIED PEOPLE USING EEG SIGNAL AND CAMERA

FIELD OF STUDY

TECHNIQUE OF CONTROL AND AUTOMATION

CODE: 9520216

SUPERVISORS:

1 Assoc Prof., Dr NGUYỄN HỮU KHƯƠNG

2 Assoc Prof., Dr VÕ CÔNG PHƯƠNG

HO CHI MINH CITY– 03/2020

Trang 1

ABSTRACT

Nowadays, EEG signal, one of the most important field was interested by science researchers, the main purpose of research is to support application devlepments, diagnose and find out pathological

of human as stress, depression, epileptic, alzheimer, brain trauma…, however, in the field of automatic control serving for human, especially for disabilities people, has not been studied so much For long time ago, recording and processing the EEGs or ECGs signal was the work of neurologists or cardiologists, nowadays, with the development of modern signal processing and analysis tools such

as neural networks and AI systems, such signals can be processed to meet the other needs, such as the control system support human acitivites.The goal of this thesis is to build a control system, which support some basic human activities through EEG signal For example, wheelchair equipement control for disabled people, meet today’s pressing social needs

In this thesis, author researched and analyzed three EEG signal pre-processing methods as Fourier transform, Wavelet transform and HHT transform, converting EEG signal to 5 basic EEG waves (Delta, Theta, Alpha, Beta, Gamma), and then using data clustering technical before put them into input layer of multilayer neural network The neural network was test from singlelayer to multilayer (3 layer) Author combined the EEG signal processing system with HHT pre-processing and image processing using multi neural network to control the wheelchair model with accurate rate 92.4% for group 20 students, this shows the successful in the practical of the thesis

Trang 2

THE THESIS STRUCTURE

Chapter 1: Overview – Presenting of the research situation of

EEG signal in domestic and internation, presenting of results have been researched and published, anayzing unsolved and limited problems that thesis needs to solve, in addition, the thesis also presentate the aimed and scope research, the contributions of thesis in science and reality life

Chapter 2: Theoretical basis – Presenting of basic knowledge

relatived to thesis as: Fourier Transform, STFT, Wavelet Transform, HHT…, clustering data and then classifying EEG signal partterns by multilayer neural network

Chapter 3: Model Contruction – Presenting how to construct

multilayer neural network, this process was conducted from singlelayer (detect 2 signal partterns) to multilayer (detect 5 signal partterns), in addition, author also discusses between EEG signals and image processing via camera in combination Each result has published on international journals or international conference

Chapter 4: Constructing software and hardware to control the wheelchair model – This chapter introduced the functions of sotfware,

the image processing to detect the eye direction in combined with EEG signals processing to achived the final result

This chapter also present the experimental results had been performed by student in Ho Chi Minh City Industry and Trade College (HITU), comparison the experimental results between 2 processes (image and EEG signal processing), and then the results has been combined with 2 processes

Chapter 5: Conclusions and recommendations – this chapter

present the results have been achived compared to thesis requirements and offering the solutions to develop EEG field more and more completely

Trang 3

CHAPTER 1 OVERVIEW OF EEG SIGNAL

1.1 Researching situation in domestic

In present, the previous researchs in EEG field mainly focused on motor activities, eye blink and movement of head… to detecting and classifying these activities could be performed by amplitude threshold method There aren’t topics now related in processing and classifying the difference EEG signal partterns corresponding to difference image types observation yet, this couldn’t be resolved by amplitude threshold method but must be resolved by extracted these signal partterns into specific features via multilayer neural network

1.2 International research situation

The research works were published at international paper mainly focused on diagnosing epilepsy sleep disorders, coma and brain death, stress, depresssion pathological… in automation field as spelling, eye blink, head movement, mental arithmetic… this were performed by offline, no mainly focus on resolving in realtime and in control automation field

1.3 The content of thesis

At first, author constructed the multilayer neural network base on raw EEG data from San Diego University (UCSD), to identify and classify 5 EEG signal partterns corresponding to 5 difference image types (human, city, landscape, flower and animal), after determining the feasibility of the multilayer neural network, author conducted on realtime EEG data combined with camera to increase efficiency, this has been performed by student of HITU, in the implementation process, the thesis extracted the feature signal by Hibert Huang Transform (HHT), clustering data and then using multilayer neural network, making system work more efficienty and avoiding

“overfitting” problem

Trang 4

1.4 The purpose of the research

Finding out the way collect 5 EEG signal partterns easily and effectively

Using math tools to transform the EEG signal partterns into specified extract features, using K-Mean technique to cluster EEG signal data before putting in multilayer neural network to classify these partterns

Combining EEG signals and camera processing aim to identified and classified process more easily and effectively

1.5 Object and scope of the research

The main object in thesis is to use multilayer neural network to classify 5 EEG signal partterns into control commands corresponding

to 5 commands to control the wheelchair as: Forward, Turn right, Turn left, Reverse and Stop In thesis also mention to image processing to detect the eye direction aim to help the system work more easily and effectivly However, the thesis don’t focus more on image processing, but on EEG signal processing

1.6 The contribution of thesis

1.6.1 The contribution about theory

Find out the observation board which easy to use to collect data, combine scientifically between feature extraction algorithm and cluster data before putting into neural network to classify data partterns

1.6.2 Practical contribution

The experimental result of thesis show that the EEG signal partterns classification through eye observation (with differnce image partterns), for people who has mind and eyes as normal people could absolutely performance

CHAPTER 2 - THEORETICAL BASIS

2.1 EEG signal and its characterizations

Delta wave (0 – 3 Hz), the highest amplitude as figure 2.1, it often appear at the child up to 1 year old and adult when sleeping, well sleep

It represents the grey matter of the brain This wave usually appears everywhere on the scalp

Trang 5

Figure 2.1 The Delta wave

The Theta wave (3 – 7 Hz), it appear when the eye are closed and the mind is in a relax state as Figure 2.2, it apperas in adult or when awake in the elderly and often appears in the temples

Figure 2.2 The Theta wave

The Alpha wave (7 – 13 Hz), it appear often in elder as Figure 2.2, Alpha wave usually appear on both sides of scalp but having an uneven amplitude, this waves appear when the eye are closed (a state

of relaxation) and often disappear when the eyes are opened or under stress

Figure 2.3 The Alpha wave

The Beta waves (13 – 30 Hz), with low amplitute as Figure 2.4, This waves usually appears in patients who are often in a state of alertness, prevention, and anxiety… this waves are distributed symmetrically on both sides and most clearly at the front, it usually appear in front and at the top of the cerebral cortex, the amplitude is less than 30uV

Figure 2.4 The Beta wave

Trang 6

The Gamma wave (30 – 45 Hz), it often referred to as fast Beta wave This wave usually has low amplitude and rarely appears, but the discovery of this wave plays a important role in identifying neurological diseases that occur in the center of the cerebral cortex

Figure 2.5 The Gamma wave 2.2 The electrode positions on the scalp

The brain is one of the largest and most complex organs in the human body, it is made up of over 100 billion nerves, communicating with 1,000 billion synapses The electrode positions are mounted on the scalp according to international standard 10/20 as Figure 2.6

Figure 2.6 Electrode positions according to international

standard 10/20

CHAPTER 3 CONTROL MODEL CONTRUCTION

To begin the research process, author used the database provided

on the prestigious University of San Diego (UCSD) website of the USA, ranked 38 in the world in 2018, this data was obtained from participants when looked at 5 different image objects (human, city, landscape, flower and animal), with 8 color bits and size (256 pixels

Trang 7

wide and 384 pixels high), the total number of samples is 21,000

In this chapter, author present to built the model from simple to complex step by step, and then evaluate the experimental results on database provided by the University of San Diego (UCSD), and from

80 students from HITU to make clarify the contribution and scientific meaning of the thesis

3.1 Single-Layer neural network model

At first, author built a single neural network to separate two parttern of signals (animals and not animals), the purpose of this study

is to evaluate whether the neural network meets the classification requirements, thesis used Matlab software for this experimental process

The features of EEG database is extracted by Wavelet transform (Mexico hat) and used a single neural network to identify The system model is shown in Figure 3.1, this model included of 2 stages: stage 1: preprocessing raw data signals and then synthesize into 5 basic EEG signals Delta, Theta, Alpha, Beta and Gamma Stage 2: builting a single neural network with 5 inputs corresponding to 5 basic EEG signals: Delta, Theta, Alpha, Beta, Gamma and one output to determine the clasified results

Figure 3.1 Single-Layer neural network model

The training process is performed on the database with the following parameters:

Trang 8

 Initial random weight in the range from -0.5 to 0.5

 The error threshold is 1x10-5 based on MSE (Mean Square Error)

 The maximun number loop: 5.000

Experimental results of identification on the database are shown

in Table 3.1

Table 3.1 Experimental results on database

Image types Animal/Landscape Identification

Rate

Trang 9

Image types Animal/Landscape Identification

Rate

From the experimental results in Table 3.1, author found that the average accuracy rate of the identification results on the database was 91.15%

3.2 Multi-Layer neural network model

3.2.1 System Model

Based on the results achieved from the single neural network model, author continued to develop a multi neural network model with the results of classifying 5 EEG signal partterns corresponding to 5 control signals with accurate rate 93.57% Table 3.2 describes the result of 05 control commands corresponding to the equivatent image types

This model uses Wavelet transform to noise signals and extract features, then using K-mean algorithm to cluster the characteristics of the signal partterns and then put into the multi-layer neural network to classify, in this model, author chooses 10 channels to reduce processing time and enhance performane System model is shown in Figure 3.2

Figure 3.2 Multi-Layer neural network system model

Trang 10

3-layer neural network model is shown in Figure 3.3

The first layer contains five nodes which are Delta, Theta, Alpha, Beta and Gamma, these classes is called the input layer

The second layer is the hidden layer, the number of nodes in the hidden layer is set to 5, 10, 15, 20, 25, 30, 35, 40, 45 and 50

The output layer contains a node, the result of this node is used

to classify the EEG signal The activation function used in this model

is hyperbolic tangent, the value of the output is in the range [-1, 1]

Figure 3.3 Multi-Layer neural network model

Before using the model, the neural network needs to training stage The training algorithm is shown in figure 3.4, the model used backpropagation algorithm

Trang 11

Figure 3.4 Neural network training algorithm

3.2.2 Experimental results with multi-layer neural network model

The dataset consists of 21,000 samples divided into subsets for training (70%), validation (15%) and testing (15%) The system uses Matlab and EEGLab tools for the testing process, the neural network

is divided into two test stages

The training stage is performed on the training database, using structures with different hidden nodes in the hidden layer of neural network, with the following parameters as follows:

 Learning rate: 0.7

 Initial random weight in the range from 0 to 1

 The error threshold is 1x10-5 based on RMSE (Root Mean

Initial random weight

Get value Delta, Theta, Alpha, Beta, Gamma Calculate the value of nodes in hidden layer Calculate the ouput value of nodes in the hidden layer

Calculate the input value of the output nodes Calculate the output value of the output nodes Calculate the error of the output layer Calculate the error of the hidden layer Calculate the error of the system Error system <= threshold value?

End

Update weight True

False

Trang 12

Square Error)

 The maximun number loop: 5.000

The accuracy of classification is measured by the ratio of result of wrong classification to the total number of samples follow formula (3.1)

Minimum Error

Accurate Rate

Table 3.3 Matrix confusion of classification results

To provide a more intuitive and easy-to-understand about result

of predictive quality, the following formulas are used for effective quality test

Accuracy (AC) is an accurate prediction rate It is determined using the formula (3.2)

(3.3)

Where, True Possitive (TP) refers to the correct database that are correctly classified to be true True Negative (TN) refers to incorrect database that are incorrectly classified to be false False Possitive (FP) refers to incorrect database that are incorrectly classified to be true False Negative (FN) refers to incorrectly categorized database to be false

The rate of identification with 40 hidden nodes in the hidden layer

Trang 14

Human 93,2% 93,4% 6,6% 6,8% 93,3% 93,4%

Flower 93,8% 93,9% 6,1% 6,2% 93,9% 93,9% These results are also compared with previous studies such as determining EEG based on winking with 15,360 samples and reaching 90.85%, with decision tree reach the rate 85%, based on eye movement by 2 experiments with 3,600 samples and 8,320 samples reaching acuracy rate 85%

3.3 Design of synthetic model for signal processing

This model was developed from a multi-layer neural network model in section 3.2 Beside of identifying the EEG signal, it also combined the user's eye direction signal through the camera, this model focuses on the items as below:

 Converting EEG signals using Hilbert Huang Transform (HHT) method to reduce noise signal because HHT conforms to EEG signal and for better results than other transform methods

 Eye direction recognition based on the user's face image combined with EEG signal recognition to improve the effectiveness

Trang 15

Figure 3.5 Synthetic system model

 The first block is the EEG signal recognition block to extract

5 features

 The second block is the eye direction recognition block by recognizing the eyes and eyebrows from the user's face image to extract 4 features

 The third block is a multi-layer neural network with 9 input nodes (4 for camera and 5 for EEG), 5 output nodes are classified equivalent 5 control commands as “FORWARD”, “TURN RIGHT”,

“TURN LEFT”, “REVERSE” and “STOP”

3.3.1 EEG signal recognition block

This block behaves like the model in section 3.2, at first, selecting useful information channels and remove channels with redundant information, next, using HHT to extract feature and eliminate noise signals, and then, use the K-Means algorithm to cluster data

3.3.2 Identifing eye direction signal

Face images are received from the camera and are cropped in the area of eye information to reduce processing time, after that, the face image is converted into a binary image that satisfies the requirements

Face image

Converting to binary image

Detecting eye and eyebrow

Rating between eye and eyebrow

Extracting 4 features

EEG signals

Selecting channels

Hibert Huang Transform

Clustering

Extracting 5 features

MULTILAYER NEURAL NETWORK