ĐỒ ÁN BÁO CÁO FINAL PROJECT ĐHBK

Dự án cuối cùng: Kiểm soát EEG thiết bị điện sử dụng thuật toán mạng thần kinh trên STM32,In the process to implement the project certain inevitable mistakes, we hope to receive your comments. Thank you parents, friends,etc. we have been the spiritual source of motivation for us to complete the subject. Finally, congratulations to the relationship between HCMUTE and RMUTL becomes better and better.

Firstly, we would like to thank to teachers, everyone in Ho Chi Minh City University of Technical and Education (HCMUTE) has created best conditions for us the opportunity to exchange schooling at Rajamangala University of Technology Lanna (RMUTL) Thanks to the enthusiastic assistance of RMUTL teachers, especially Dr Nopadon Maneetien and many Thai friends They help us to complete successfully project

“EEG Control Electrical Devices Using Neural Network Algorithm On STM32” In the process to implement the project certain inevitable mistakes, we hope to receive your comments Thank you parents, friends,etc we have been the spiritual source of motivation for us to complete the subject Finally, congratulations to the relationship between HCMUTE and RMUTL becomes better and better

Comment

………

………

………

………

………

………

………

………

………

………

Chiang Mai ……… Group of students: Âu Văn Bằng

Nguyễn Đức Tài

Chapter 1: Introduction 1

1.1 The origin and significance of the project 1

1.2 The purpose of the project 1

1.3 The scope of the project 1

1.4 The benefits of the system 2

Chapter 2: The theory and research 3

2.1 The principle of project 3

2.2 Theory of project 3

2.2.1 Overview of ARM 3

2.2.2 EEG and Brainwaves 6

2.2.3 Neurosky mind wave mobile 8

2.2.4 Classification algorithms 9

2.2.5 LCD (Liquid Crystal Display) 16x2 10

2.2.6 Relay 13

2.2.7 OPTO PC817 13

2.2.8 Module Bluetooth HC05 15

2.2.9 ESP 8266 16

Chapter 3: Design 18

3.1 Introduction 18

3.2 Block diagram 18

3.3 Design 19

3.3.1 Brainwave block 19

3.3.2 Webserver block 19

3.3.3 Bluetooth block 20

3.3.4 ESP8266 Block 21

3.3.5 Display Block 21

3.3.6 Control Device Block 22

3.3.7 Center processing block 24

3.3.8 Power block 26

Chapter 4: Implementation 27

4.1 Make PCB 27

4.1.1 Required components 27

4.1.2 Draw PCB 28

4.1.3 Completed circuit 29

4.2 Flow chart 29

4.3 KeilC 5 32

4.4 Web server interface 32

4.4.1 The requirements of the web server interface 32

4.4.2 Programs to build webserver 32

4.5 Test 35

Chapter 5: Results and Future Works 36

5.1 Results 36

5.2 Future Works 36

References 37

Addendum 38

List of Figures

Figure 2.1 Hardware block diagram 5

Figure 2.2 Neurosky Mindwave mobile 9

Figure 2.3 LCD 16x2 11

Figure 2.4 Timing chart LCD 12

Figure 2.5 Relay 5VDC 13

Figure 2.6 PC817 14

Figure 2.7 Module HC05 15

Figure 2.8 Module ESP8266 16

Figure 3.1 Diagram block 18

Figure 3.2 Bluetooth block 20

Figure 3.3 ESP8266 Block 21

Figure 3.4 Display Block 22

Figure 3.5 Control Device Block 22

Figure 3.6 Process block 25

Figure 4.1 PCB 28

Figure 4.2 3D 28

Figure 4.3 Completed circuit 29

Figure 4.4 Main flow chart 30

Figure 4.5 Neural network flow chart 31

Figure 4.6 Icon keilc 5 32

Figure 4.7 Start xampp for run web offline 33

Figure 4.8 Create new file 33

Figure 4.9 Selcet the PHP programming language 34

Figure 4.10 Save file path 34

Figure 4.11 Web interface completed 35

List of Tables

Table 2.1 Pin LCD 11

Table 2.2 Commands LCD 12

Table 2.3 Feature of PC817 14

Table 4.1 Components 27

Chapter 1: Introduction

1.1 The origin and significance of the project

These days, there is a rapid increasing of the global economic development as well

as technology explosion, people tend to enhance the standard of living based on hi-techdevices and smart inventions, this leads to scientists and technology specialists mustpropose projects as well as look for the new approaches for meeting people’s demand As

a result, IoT-Internet of Things, which is commonly used a term in recent year, sets asignal for the 4th industrial revolution In fact, there is a vast range of invented smartsystems which support the general public in many fields such as manufacturing,agriculture and health care Also, people are able to operate those systems in numerousways and one of them is a Mind-Controlled method, which is a technology formanipulating objects via brainwaves We find it’s interesting between IoT and this oneplaying a crucial role in life quality, therefore, we choose the project called “Controllinginternet-connected devices through brainwaves”

“Controlling internet-connected devices through brainwaves” is capable ofsupporting users control electronics devices in the house without moving the body.Especially, this is a light point for paralysis victims can take care themselves, even thoughnobody around them

Implementing this tool include a lot of methodologies such as FPGA, ARMmicrocontrollers, etc In this project, we use ARM microcontrollers to create this one

1.2 The purpose of the project

Understand how to operate the module to read the brainwaves and then handlesignals in order to return

Construct a web server to update the control signals as well as allow users tocontrol the system by pushing buttons on the screen

Research and build controlling internet-connected devices through brainwaves with

3 modes-controlled, namely brainwaves, website and Android app

1.3 The scope of the project

 Control on/off two devices

 Using mind wave mobile

 Using power 3.3 and 5v

1.4 The benefits of the system

 Support those who is paralysis

 Control robot, computer and other devices by brainwaves

Chapter 2: The theory and research

2.1 The principle of project

The principle of project are based on the brainwave-receive, analysis, classification,control and sending data to web server There are numerous classification algorithm forEEG such as Neural Networks, Nonlinear Bayesian classifiers, … However, NeuralNetworks are the most popular algorithm for its high performance

In the contrast, as the requirement of the subject's brain waves handling neuralnetwork algorithms need an ARM kit with relative speed to run well and test Kit needs anergonomic design with easy usage and low cost Therefore, we choose the Kit, namelySTM32F407VGT6 for this subject

 Introduce STM32F407VGT6

The STM32F4DISCOVERY Discovery kit allows users to easily developapplications with the STM32F407 high performance microcontroller with ARM®Cortex®-M4 32-bit core It includes everything required either for beginners or forexperienced users to get quickly started

Based on the STM32F407VGT6, it includes an ST-LINK/V2 or ST-LINK/V2-Aembedded debug tool, two ST MEMS digital accelerometers, a digital microphone, oneaudio DAC with integrated class D speaker driver, LEDs and push buttons and an USB

OTG micro-AB connector To expand the functionality of the STM32F4DISCOVERYDiscovery kit with the Ethernet connectivity, LCD display and more, visit thewww.st.com/stm32f4dis-expansion webpage The STM32F4DISCOVERY Discovery kitcomes with the STM32 comprehensive software HAL library, together with variouspackaged software examples.

 Features

 STM32F407VGT6 microcontroller featuring 32-bit ARM® Cortex®-M4with FPU core, 1-Mbyte Flash memory, 192-Kbyte RAM in an LQFP100package

 On-board ST-LINK/V2 on STM32F4DISCOVERY (old reference) orST-LINK/V2-A on STM32F407G-DISC1 (new order code)

 USB ST-LINK with re-enumeration capability and three differentinterfaces:

 Debug port

 Virtual Com port (with new order code only)

 Mass storage (with new order code only)

 Board power supply: through USB bus or from an external 5V supplyvoltage

 External application power supply: 3V and 5V

 LIS302DL or LIS3DSH ST MEMS 3-axis accelerometer

 MP45DT02 ST-MEMS audio sensor omnidirectional digital microphone

 CS43L22 audio DAC with integrated class D speaker driver

 Eight LEDs:

 LD1 (red/green) for USB communication

 LD2 (red) for 3.3V power on

 Four user LEDs, LD3 (orange), LD4 (green), LD5 (red) and LD6(blue)

 2 USB OTG LEDs LD7 (green) VBUS and LD8 (red) over-current

 USB OTG FS with micro-AB connector.

 Extension header for all LQFP100 I/O for

 Quick connection to prototyping board and easy probing

 Comprehensive free software including a variety of examples, part ofSTM32CubeF4 package or STSW-STM32068 to use legacy standardlibraries

 Hardware and layout

The STM32F4DISCOVERY is designed around the STM32F407VGT6microcontroller in a 100-pin LQFP package

Figure 2.1 illustrates the connections between the STM32F407VGT6 and itsperipherals (STLINK/V2 or ST-LINK/V2-A, pushbutton, LED, Audio DAC, USB, STMEMS accelerometer, ST MEMS microphone, and connectors)

Figure 2.1 Hardware block diagram

 Software and programming language

There is a variety of different programming languages, the most popularprogramming language is C Thus, manufacturers have fabricated kit for ARM C languageprogramming.

Additionally, the programming software also has a lot of choice such as EWARM,MDK-ARM, TrueSTUDIO, tasking, etc We decide to choose MDK-ARM because itgains a popularity of market as well as easy using and friendly interface with users

2.2.2 EEG and Brainwaves

 Brainwaves

At the root of all our thoughts, emotions and behavior is the communicationbetween neurons within our brains Brainwaves are produced by synchronized electricalpulses from masses of neurons communicating with each other

Brainwaves are detected using sensors placed on the scalp They are divided intobandwidths to describe their functions (below), but are best thought of as a continuous

It is a handy analogy to think of Brainwaves as musical notes the low frequencywaves are like a deeply penetrating drum beat, while the higher frequency brainwaves aremore like a subtle high pitched flute Like a symphony, the higher and lower frequencieslink and cohere with each other through harmonics

Our brainwaves change according to what we’re doing and feeling When slowerbrainwaves are dominant we can feel tired, slow, sluggish, or dreamy The higherfrequencies are dominant when we feel wired, or hyper-alert

The descriptions that follow are only broadly descriptions in practice things are farmore complex, and brainwaves reflect different aspects when they occur in differentlocations in the brain

Brainwave speed is measured in Hertz (cycles per second) and they are divided intobands delineating slow, moderate, and fast waves

Infra-Low brainwaves (frequency <0.5HZ) (also known as Slow CorticalPotentials), are thought to be the basic cortical rhythms that underlie our higher brainfunctions Very little is known about infra-low brainwaves Their slow nature make themdifficult to detect and accurately measure, so few studies have been done They appear totake a major role in brain timing and network function

Delta Waves (frequency 0.5 to 3 HZ), the slowest but loudest brainwaves Deltabrainwaves are slow, loud brainwaves (low frequency and deeply penetrating, like a drumbeat) They are generated in deepest meditation and dreamless sleep Delta waves suspendexternal awareness and are the source of empathy Healing and regeneration are stimulated

in this state, and that is why deep restorative sleep is so essential to the healing process

Theta brainwaves (frequency 3 to 8 HZ), occur in sleep and are also dominant indeep meditation It occur most often in sleep but are also dominant in deep meditation Itacts as our gateway to learning and memory In theta, our senses are withdrawn from theexternal world and focused on signals originating from within It is that twilight statewhich we normally only experience fleetingly as we wake or drift off to sleep In theta weare in a dream, vivid imagery, intuition and information beyond our normal consciousawareness It’s where we hold our “stuff”, our fears, troubled history, and nightmares

Alpha brainwaves (frequency 8 to 12 HZ), occur during quietly flowing thoughts,but not quite meditation It are dominant during quietly flowing thoughts, and in somemeditative states Alpha is “the power of now”, being here, in the present Alpha is theresting state for the brain Alpha waves aid overall mental coordination, calmness,alertness, mind/body integration and learning.

Beta brainwaves (frequency 12 to 38 HZ), are present in our normal waking state ofconsciousness It dominate our normal waking state of consciousness when attention isdirected towards cognitive tasks and the outside world Beta is a “fast” activity, presentwhen we are alert, attentive, engaged in problem solving, judgment, decision making, andengaged in focused mental activity Beta brainwaves are further divided into three bands.Low Beta (Beta1, 12-15Hz) can be thought of as a fast idle, or musing Beta (Beta2, 15-22Hz) is high engagement or actively figuring something out Hight Beta (Beta3, 22-38Hz) is highly complex thought, integrating new experiences, high anxiety, orexcitement Continual high frequency processing is not a very efficient way to run thebrain, as it takes a tremendous amount of energy

Gamma brainwaves (frequency 38 to 42 HZ), are the fastest of brain waves andrelate to simultaneous processing of information from different brain areas Gammabrainwaves are the fastest of brain waves (high frequency, like a flute), and relate tosimultaneous processing of information from different brain areas It passes informationrapidly, and as the most subtle of the brainwave frequencies, the mind has to be quiet toaccess it Gamma was dismissed as “spare brain noise” until researchers discovered it washighly active when in states of universal love, altruism, and the “higher virtues” Gamma

is also above the frequency of neuronal firing, so how it is generated remains a mystery It

is speculated that Gamma rhythms modulate perception and consciousness, and that agreater presence of Gamma relates to expanded consciousness and spiritual emergence

2.2.3 Neurosky mind wave mobile

 NeuroSky Technology

Your brain is constantly producing electrical signals while it operates, as the

scale, they produce a range of frequencies that scientists have found relate to particularmental states For example, a sleeping person’s brain produces an abundance of deltawaves, whereas an alert and awake person concentrating hard on something will producefar more beta waves.

The Mindwave headset picks up your brain’s electrical activity and divides thesignal by frequency into various types of waves, allowing it to infer your mental state Forthe most of the non-scientific apps however, it primarily reads how relaxed (as measured

by alpha/theta waves) or concentrated (as measured by beta/gamma waves) you are

Unfortunately your body makes a lot of other electrical noise, in addition to theactivity coming from your brain For this reason there is a “reference” contact, in the form

of a clip that attaches to your earlobe, which allows the headset to filter out non-brainrelated electrical activity

 Hardware

It connects via bluetooth to the device of your choice, and works with most modernoperating systems (Windows XP or newer, Mac OS X 10.6.5 or newer) and mobiledevices running android or IOS It’s battery life is rated at 8-10 hours with a single AAAbattery

Figure 2.2 Neurosky Mindwave mobile

2.2.4 Classification algorithms

 Neural Networks

Neural Networks (NN) is the category of classifiers mostly used in BCI research.Let us recall that a NN is an assembly of several artificial neurons which enables toproduce nonlinear decision boundaries This section first describes the most widely used

NN for BCI, which is the multilayer Perceptron (MLP)

An MLP is composed of several layers of neurons: an input layer, possibly one orseveral hidden layers, and an output layer Each neuron’s input is connected with theoutput of the previous layer’s neurons where as the neurons of the output layer determinethe class of the input feature vector

Neural Networks and thus MLP, are universal approximators, i.e., when composed

of enough neurons and layers, they can approximate any continuous function Added tothe fact that they can classify any number of classes, this makes NN very flexibleclassifiers that can adapt to a great variety of problems Consequently, MLP, which are themost popular NN used in classification, have been applied to almost all BCI problemssuch as binary or multiclass, synchronous or asynchronous BCI However, the fact thatMLP are universal approximators makes these classifiers sensitive to overtraining,especially with such noisy and non-stationary data as EEG Therefore, careful architectureselection and regularization is required

A multilayer Perceptron without hidden layers is known as a perceptron.Interestingly enough, a perceptron is equivalent to LDA and, as such, has been sometimesused for BCI applications

 Nonlinear Bayesian classifiers

This section introduces Bayesian classifiers used for BCI: Bayes quadratic Allthese classifiers produce nonlinear decision boundaries Furthermore, they are generative,which enables them to perform more efficient rejection of uncertain samples thandiscriminative classifiers However, these classifiers is not as widespread as NeuralNetworks in BCI applications

Bayesian classification aims at assigning to a feature vector the class it belongs towith the highest probability The Bayes rule is used to compute the so-called a posterioriprobability that a feature vector has of belonging to a given class Using the MAP(Maximum A Posteriori) rule and these probabilities, the class of this feature vector can beestimated.

Bayes quadratic consists in assuming a different normal distribution of data Thisleads to quadratic decision boundaries, which explains the name of the classifier Thisclassifier is not widely used for BCI

2.2.5 LCD (Liquid Crystal Display) 16x2

 Introduction

There are many types of LCD with multiple shapes and sizes vary from a fewletters to dozens of characters, from one to several tens restaurant row 16x2 LCD means 2rows and each row has 16 characters

4 RS Input Register Select H: data signal, L: instruction signal

Figure 2.4 Timing chart LCD

 Current activities at 5VDC: 70mA

 Minimum current, Voltage Minimum: 100 mA, 5VDC

 Maximum voltage: 250VAC/30VDC

 Maximum current: 15A.

2.2.7 OPTO PC817

Opto is not only the optical coupling (also called OPTO) but also a semiconductormade up of an optical transmitter and an optical sensor integrated in one block ofsemiconductor Optical transmitter is a light emitting diode which emits light stimulus forphotoconductor sensors Optical sensors are also photo transistor

Opto is used as a separator between different blocks of power capacity like cubiccapacity or small capacity with large voltage blocks It can used to prevent noises for theH-bridge circuits, output PLCs or microcontrollers and anti-jamming measuring devices

The principle of OPTO led glow when there is existed a current in the circuit, thenphoto diodes (or photo transistor) is opened allowing the electricity past through

Figure 2.6 PC817 Table 2.3 Feature of PC817

2.2.8 Module Bluetooth HC05

HC05 is a bluetooth module with AT command interface, used to connect bluetoothmicrocontrollers with a device that has bluetooth waves

 PIO (Programmable Input/Output) control.

 UART interface with programmable baud rate

 With integrated antenna

 With edge connector

 Software features

 Slave default Baud rate: 9600, Data bits: 8, Stop bit: 1, Parity: No parity

 Auto connect to the last device on power as default

 Permit pairing device to connect as default

 Auto pairing PINCODE: “1234” as default

 Work modes

HC-05 embedded Bluetooth serial communication module (can be short formodule) has two work modes: order-response work mode and automatic connection workmode And there are three work roles (Master, Slave and Loopback) at the automaticconnection work mode When the module is at the automatic connection work mode, itwill follow the default way set lastly to transmit the data automatically

When the module is at the order-response work mode, user can send the ATcommand to the module to set the control parameters and sent control order The workmode of module can be switched by controlling the module PIN (GPIO11) input level

Figure 2.8 Module ESP8266

 Feature

 Power supply: 3.3VDC

 3.3VDC UART interface

 Support 802.11 b/g/n

 WIFI 2.4 GHz, support WPA/WPA2

 There are three modes of operation as a client, access point, both shows

 Supporting both TCP and UDP protocols are

 UART Baud rate can be selected: 1200, 2400, 4800, 9600, 19200, 38400,

Chapter 3: Design

3.1 Introduction

Project “EEG Control Electrical Devices Using Neural Network Algorithm OnSTM32” In this chapter, we need to do the following tasks design the system blockdiagram, design schematic, calculate electronic components and select the power supply

3.2 Block diagram

From the target of the project “EEG Control Electrical Devices Using NeuralNetwork Algorithm On STM32” we have the following block diagram

PROCESSING CENTER

CONTROL DEVICE

LCD DISPLAY BRAINWAVES

 Power block: power supply function for the entire system

 Brainwaves block: measure brainwave and transmit data to the central processor viabluetooth

 Web server block: receive data from the central processing unit and send controlcommands to the central processor and other connected devices

 Bluetooth block: function the connection between the central processing block andthe brainwaves block

 ESP8266 block: function connection between the central processing block and webserver

 Control device block: function to receive control signals from the centralprocessing unit for controlling the devices.

 Center processing block: receive data from brainwaves, computation, analysis givecontrol commands to send out signals to the web and device drivers

 Use device have one electrode

o Advantage: low cost, easy to use

o Disadvantage: accuracy about 70%

 Use device have more electrode

o Advantage: high-precision signal

o Disadvantage: expensive

 We choose devices having one electrode, in particular mobile devicesmindwaves NeuroSky because if the group can handle data from mobilemindwaves we can do it with other devices

3.3.2 Webserver block

There are functions to receive data from the center processing unit and send controlcommands to the central processor and other devices

 Options

 Use web support for IoT

o Advantage: simple, register an account can be used

o Disadvantage: can’t change the user interface

 Design web server

o Advantage: build interface preferences, easier data management

o Disadvantage: know web programming language

 We choose design web server Because we want to build a separate interface,simple and user-friendly.

o Advantage: simple, can activity two mode MASTER or SLAVE

o Disadvantage: relatively high cost

 Use HC-06

o Advantage: easy to use

o Disadvantage: only working in SLAVE mode

communication between STM32F4-discovery and HC-05 Let say V1 is the voltage of the

Tx pins STM32F4 kit, we have: V1 = 5V Let say V2 is the voltage received on the HC-05

3.3.4 ESP8266 Block

This is a functional connection between the center processing block and web serverblock

 Options

 Use module ESP8266-v1

o Advantage: simple, low cost

o Disadvantage: small memory, few GPIO

 Use module ESP8266-v12

o Advantage: big memory, much GPIO

o Disadvantage: expensive

 We choose module ESP8266-v1 The main wifi block connection has only twoblocks then we do not need large memory and multiple GPIO

Figure 3.11 ESP8266 Block

o Advantage: display the characters, complex symbols and low cost

o Disadvantage: display few characters

 The system must not display too much characters so we choose LCD 16x2

Figure 3.12 Display Block

3.3.6 Control Device Block

Figure 3.13 Control Device Block

We uses relay for switching high powered devices

 Calculations

Control Device Block has two arms and control two devices

 Arm 1: Control the lamp 220VAC-60W

We have: load of lamp: IL =

to 150mA lines (datasheet)

To protect the transistor we need a diode, this diode must be able to withstand large5VDC voltages and currents greater than 100mA So, we use diode 1N4001.

To protect the microcontroller, we using opto-isolated microcontrollers and powerblocks There are many types of opto, the group uses PC817 opto because it's popular andcheaper

The 2SC1815 Transistor has many types with DC current gain ( β ) differently.

We choose design with type 0 DC current gain as low as 100

= 100 (Ω) and R)

 Arm 2: Control the fan 220VAC-45W

We have: load of fan: IL =

To protect the transistor we need a diode, this diode must be able to withstand large