Neural Network Toolbox in Matlab

Orlando De Jesús of Oklahoma State University for his excellent work in developing and programming the dynamic training algorithms described in Chapter 6, “Dynamic Networks,” and in prog

Neural Network Toolbox™ 6

User’s Guide

Howard Demuth

Mark Beale

Martin Hagan

How to Contact The MathWorks

comp.soft-sys.matlab Newsgroup

www.mathworks.com/contact_TS.html Technical support

suggest@mathworks.com Product enhancement suggestions

bugs@mathworks.com Bug reports

doc@mathworks.com Documentation error reports

service@mathworks.com Order status, license renewals, passcodes

info@mathworks.com Sales, pricing, and general information

508-647-7000 (Phone)

508-647-7001 (Fax)

The MathWorks, Inc

3 Apple Hill Drive

Natick, MA 01760-2098

For contact information about worldwide offices, see the MathWorks Web site.

Neural Network Toolbox™ User’s Guide

© COPYRIGHT 1992–2009 by The MathWorks, Inc.

The software described in this document is furnished under a license agreement The software may be used

or copied only under the terms of the license agreement No part of this manual may be photocopied or duced in any form without prior written consent from The MathWorks, Inc.

repro-FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation by, for, or through the federal government of the United States By accepting delivery of the Program or Documentation, the government hereby agrees that this software or documentation qualifies as commercial computer software or commercial computer software documentation as such terms are used or defined in FAR 12.212, DFARS Part 227.72, and DFARS 252.227-7014 Accordingly, the terms and conditions of this Agreement and only those rights specified in this Agreement, shall pertain to and govern the use, modification, reproduction, release, performance, display, and disclosure of the Program and Documentation

by the federal government (or other entity acquiring for or through the federal government) and shall supersede any conflicting contractual terms or conditions If this License fails to meet the government's needs or is inconsistent in any respect with federal procurement law, the government agrees to return the Program and Documentation, unused, to The MathWorks, Inc

Trademarks

MATLAB and Simulink are registered trademarks of The MathWorks, Inc See

www.mathworks.com/trademarks for a list of additional trademarks Other product or brand names may be trademarks or registered trademarks of their respective holders.

Patents

The MathWorks products are protected by one or more U.S patents Please see

www.mathworks.com/patents for more information.

Revision History

June 1992 First printing

April 1993 Second printing

January 1997 Third printing

July 1997 Fourth printing

January 1998 Fifth printing Revised for Version 3 (Release 11) September 2000 Sixth printing Revised for Version 4 (Release 12) June 2001 Seventh printing Minor revisions (Release 12.1)

July 2002 Online only Minor revisions (Release 13)

January 2003 Online only Minor revisions (Release 13SP1)

June 2004 Online only Revised for Version 4.0.3 (Release 14) October 2004 Online only Revised for Version 4.0.4 (Release 14SP1) October 2004 Eighth printing Revised for Version 4.0.4

March 2005 Online only Revised for Version 4.0.5 (Release 14SP2) March 2006 Online only Revised for Version 5.0 (Release 2006a) September 2006 Ninth printing Minor revisions (Release 2006b)

March 2007 Online only Minor revisions (Release 2007a)

September 2007 Online only Revised for Version 5.1 (Release 2007b) March 2008 Online only Revised for Version 6.0 (Release 2008a) October 2008 Online only Revised for Version 6.0.1 (Release 2008b) March 2009 Online only Revised for Version 6.0.2 (Release 2009a)

The authors would like to thank the following people:

Joe Hicklin of The MathWorks™ for getting Howard into neural network research years ago at the University of Idaho, for encouraging Howard and Mark to write the toolbox, for providing crucial help in getting the first toolbox Version 1.0 out the door, for continuing to help with the toolbox in many ways, and for being such a good friend

Roy Lurie of The MathWorks for his continued enthusiasm for the possibilities

for Neural Network Toolbox™ software

Mary Ann Freeman for general support and for her leadership of a great team of

people we enjoy working with

Rakesh Kumar for cheerfully providing technical and practical help,

encouragement, ideas and always going the extra mile for us

Sarah Lemaire for facilitating our documentation work.

Tara Scott and Stephen Vanreusal for help with testing.

Orlando De Jesús of Oklahoma State University for his excellent work in developing and programming the dynamic training algorithms described in Chapter 6, “Dynamic Networks,” and in programming the neural network controllers described in Chapter 7, “Control Systems.”

Martin Hagan , Howard Demuth, and Mark Beale for permission to include

various problems, demonstrations, and other material from Neural Network

Design, January, 1996.

Neural Network Toolbox™ Design Book

The developers of the Neural Network Toolbox™ software have written a

textbook, Neural Network Design (Hagan, Demuth, and Beale, ISBN

0-9717321-0-8) The book presents the theory of neural networks, discusses their design and application, and makes considerable use of the MATLAB®

environment and Neural Network Toolbox software Demonstration programs from the book are used in various chapters of this user’s guide (You can find all the book demonstration programs in the Neural Network Toolbox software

by typing nnd.)This book can be obtained from John Stovall at (303) 492-3648, or by e-mail at

John.Stovall@colorado.edu

The Neural Network Design textbook includes:

• An Instructor’s Manual for those who adopt the book for a class

• Transparency Masters for class use

If you are teaching a class and want an Instructor’s Manual (with solutions to the book exercises), contact John Stovall at (303) 492-3648, or by e-mail at

John.Stovall@colorado.edu

To look at sample chapters of the book and to obtain Transparency Masters, go directly to the Neural Network Design page at

http://hagan.okstate.edu/nnd.html

From this link, you can obtain sample book chapters in PDF format and you

can download the Transparency Masters by clicking Transparency Masters

(3.6MB)

You can get the Transparency Masters in PowerPoint or PDF format

Contents 1

Getting Started

Product Overview 1-2

Using the Documentation 1-3

Applications for Neural Network Toolbox™ Software 1-4

Applications in This Toolbox 1-4

Business Applications 1-4

Fitting a Function 1-7

Defining a Problem 1-7

Using Command-Line Functions 1-7

Using the Neural Network Toolbox™ Fitting Tool GUI 1-13

Recognizing Patterns 1-24

Defining a Problem 1-24

Using Command-Line Functions 1-25

Using the Neural Network Toolbox™ Pattern

Recognition Tool GUI 1-31

Clustering Data 1-42

Defining a Problem 1-42

Using Command-Line Functions 1-43

Using the Neural Network Toolbox™ Clustering Tool GUI 1-47

Network Architectures 2-8

A Layer of Neurons 2-8

Multiple Layers of Neurons 2-10

Input and Output Processing Functions 2-12

Data Structures 2-14

Simulation with Concurrent Inputs in a Static Network 2-14

Simulation with Sequential Inputs in a Dynamic Network 2-15

Simulation with Concurrent Inputs in a Dynamic Network 2-17

Outliers and the Normalized Perceptron Rule 3-23

Graphical User Interface 3-25

Introduction to the GUI 3-25 Create a Perceptron Network (nntool) 3-25 Train the Perceptron 3-29 Export Perceptron Results to Workspace 3-31 Clear Network/Data Window 3-32 Importing from the Command Line 3-32 Save a Variable to a File and Load It Later 3-33

Creating a Linear Neuron (newlin) 4-4

Least Mean Square Error 4-8

Linear System Design (newlind) 4-9

Linear Networks with Delays 4-10

Tapped Delay Line 4-10 Linear Filter 4-10

Linearly Dependent Vectors 4-18

Too Large a Learning Rate 4-19

Variable Learning Rate (traingda, traingdx) 5-19

Resilient Backpropagation (trainrp) 5-21

Conjugate Gradient Algorithms 5-22

Line Search Routines 5-26

Quasi-Newton Algorithms 5-29

Levenberg-Marquardt (trainlm) 5-30

Reduced Memory Levenberg-Marquardt (trainlm) 5-32

Speed and Memory Comparison 5-34

Summary 5-50

Improving Generalization 5-52

Early Stopping 5-53

Index Data Division (divideind) 5-54

Random Data Division (dividerand) 5-54

Block Data Division (divideblock) 5-54

Interleaved Data Division (dividerand) 5-55 Regularization 5-55

Summary and Discussion of Early Stopping

and Regularization 5-58

Preprocessing and Postprocessing 5-61

Min and Max (mapminmax) 5-62 Mean and Stand Dev (mapstd) 5-63 Principal Component Analysis (processpca) 5-64 Processing Unknown Inputs (fixunknowns) 5-65 Representing Unknown or Don’t Care Targets 5-66 Posttraining Analysis (postreg) 5-66

Sample Training Session 5-68

Limitations and Cautions 5-71

Focused Time-Delay Neural Network (newfftd) 6-11

Distributed Time-Delay Neural Network (newdtdnn) 6-15

Using the NN Predictive Controller Block 7-7

NARMA-L2 (Feedback Linearization) Control 7-16

Identification of the NARMA-L2 Model 7-16

NARMA-L2 Controller 7-18

Using the NARMA-L2 Controller Block 7-20

Model Reference Control 7-25

Using the Model Reference Controller Block 7-27

Importing and Exporting 7-33

Importing and Exporting Networks 7-33

Importing and Exporting Training Data 7-37

8

Radial Basis Networks

Introduction 8-2

Important Radial Basis Functions 8-2

Radial Basis Functions 8-3

Neuron Model 8-3

Network Architecture 8-4

Exact Design (newrbe) 8-5

More Efficient Design (newrb) 8-7

Demonstrations 8-8

Probabilistic Neural Networks 8-9

Network Architecture 8-9

Design (newpnn) 8-10

Generalized Regression Networks 8-12

Network Architecture 8-12 Design (newgrnn) 8-14

Self-Organizing Feature Maps 9-9

Topologies (gridtop, hextop, randtop) 9-10 Distance Functions (dist, linkdist, mandist, boxdist) 9-14 Architecture 9-17 Creating a Self-Organizing MAP Neural Network (newsom) 9-18 Training (learnsomb) 9-19 Examples 9-22

10

Adaptive Filters and Adaptive Training

Introduction 10-2

Important Adaptive Functions 10-2

Linear Neuron Model 10-3

Adaptive Linear Network Architecture 10-4

Single ADALINE (newlin) 10-4

Least Mean Square Error 10-7

LMS Algorithm (learnwh) 10-8

Adaptive Filtering (adapt) 10-9

Tapped Delay Line 10-9

Adaptive Filter 10-9

Adaptive Filter Example 10-10

Prediction Example 10-13

Noise Cancellation Example 10-14

Multiple Neuron Adaptive Filters 10-16

Thoughts and Conclusions 11-6

Applin2: Adaptive Prediction 11-7

Problem Definition 11-7

Network Initialization 11-8 Network Training 11-8 Network Testing 11-8 Thoughts and Conclusions 11-10

Appelm1: Amplitude Detection 11-11

Problem Definition 11-11 Network Initialization 11-11 Network Training 11-12 Network Testing 11-12 Network Generalization 11-13 Improving Performance 11-14

Appcr1: Character Recognition 11-15

Problem Statement 11-15 Neural Network 11-16 System Performance 11-19

12

Advanced Topics

Custom Networks 12-2

Custom Network 12-2 Network Definition 12-3 Network Behavior 12-13

Additional Toolbox Functions 12-16

Custom Functions 12-17

Important Recurrent Network Functions 13-2

Elman Networks 13-3

Architecture 13-3

Creating an Elman Network (newelm) 13-4

Training an Elman Network 13-5

Function Reference

Analysis Functions 15-3

Distance Functions 15-4

Graphical Interface Functions 15-5

Layer Initialization Functions 15-6

Learning Functions 15-7

Line Search Functions 15-8

Net Input Functions 15-9

Network Initialization Function 15-10

Network Use Functions 15-11

New Networks Functions 15-12

Mathematical Notation for Equations and Figures A-2

Basic Concepts A-2

Language A-2

Weight Matrices A-2

Bias Elements and Vectors A-2

Time and Iteration A-2

Layer Notation A-3

Figure and Equation Examples A-3

Mathematics and Code Equivalents A-4

B

Demonstrations and Applications

Tables of Demonstrations and Applications B-2

Chapter 2, “Neuron Model and Network Architectures” B-2

Chapter 3, “Perceptrons” B-2

Chapter 4, “Linear Filters” B-3

Chapter 5, “Backpropagation” B-3 Chapter 8, “Radial Basis Networks” B-4

Chapter 9, “Self-Organizing and Learning

Vector Quantization Nets” B-4 Chapter 10, “Adaptive Filters and Adaptive Training” B-4 Chapter 11, “Applications” B-5 Chapter 13, “Historical Networks” B-5

Block Generation C-5

Example C-5 Exercises C-7

D

Code Notes

Dimensions D-2

Getting Started

Product Overview (p 1-2)

Using the Documentation (p 1-3)

Applications for Neural Network Toolbox™ Software (p 1-4)

Fitting a Function (p 1-7)

Recognizing Patterns (p 1-24)

Clustering Data (p 1-42)

Typically, neural networks are adjusted, or trained, so that a particular input leads to a specific target output The next figure illustrates such a situation There, the network is adjusted, based on a comparison of the output and the target, until the network output matches the target Typically, many such input/target pairs are needed to train a network.

Neural networks have been trained to perform complex functions in various fields, including pattern recognition, identification, classification, speech, vision, and control systems

Neural networks can also be trained to solve problems that are difficult for

Neural Network including connections (called weights) between neurons

Target

Adjust weights

Compare

Using the Documentation

1-3

Using the Documentation

The neuron model and the architecture of a neural network describe how a network transforms its input into an output This transformation can be viewed as a computation

This first chapter gives you an overview of the Neural Network Toolbox™

product and introduces you to the following tasks:

• Training a neural network to fit a function

• Training a neural network to recognize patterns

• Training a neural network to cluster data

These next two chapters explain the computations that are done and pave the way for an understanding of training methods for the networks You should read them before advancing to later topics:

• Chapter 2, “Neuron Model and Network Architectures,” presents the

fundamentals of the neuron model, the architectures of neural networks It also discusses the notation used in this toolbox

• Chapter 3, “Perceptrons,” explains how to create and train simple networks

It also introduces a graphical user interface (GUI) that you can use to solve problems without a lot of coding

1 Getting Started

Applications for Neural Network Toolbox™ Software

Applications in This Toolbox

Chapter 7, “Control Systems” describes three practical neural network control system applications, including neural network model predictive control, model reference adaptive control, and a feedback linearization controller

Chapter 11, “Applications” describes other neural network applications

Business Applications

The 1988 DARPA Neural Network Study [DARP88] lists various neural

network applications, beginning in about 1984 with the adaptive channel equalizer This device, which is an outstanding commercial success, is a single- neuron network used in long-distance telephone systems to stabilize voice signals The DARPA report goes on to list other commercial applications, including a small word recognizer, a process monitor, a sonar classifier, and a risk analysis system

Neural networks have been applied in many other fields since the DARPA report was written, as described in the next table

Industry Business Applications

Aerospace High-performance aircraft autopilot, flight path

simulation, aircraft control systems, autopilot enhancements, aircraft component simulation, and aircraft component fault detection

Automotive Automobile automatic guidance system, and

warranty activity analysis

Applications for Neural Network Toolbox™ Software

1-5

Defense Weapon steering, target tracking, object

discrimination, facial recognition, new kinds of sensors, sonar, radar and image signal processing including data compression, feature extraction and noise suppression, and signal/image identification

Electronics Code sequence prediction, integrated circuit chip

layout, process control, chip failure analysis, machine vision, voice synthesis, and nonlinear modeling

Entertainment Animation, special effects, and market forecasting

Financial Real estate appraisal, loan advising, mortgage

screening, corporate bond rating, credit-line use analysis, credit card activity tracking, portfolio trading program, corporate financial analysis, and currency price prediction

Industrial Prediction of industrial processes, such as the

output gases of furnaces, replacing complex and costly equipment used for this purpose in the pastInsurance Policy application evaluation and product

optimizationManufacturing Manufacturing process control, product design

and analysis, process and machine diagnosis, real-time particle identification, visual quality inspection systems, beer testing, welding quality analysis, paper quality prediction, computer-chip quality analysis, analysis of grinding operations, chemical product design analysis, machine maintenance analysis, project bidding, planning and management, and dynamic modeling of chemical process system

Industry Business Applications

1 Getting Started

Medical Breast cancer cell analysis, EEG and ECG

analysis, prosthesis design, optimization of transplant times, hospital expense reduction, hospital quality improvement, and

emergency-room test advisementOil and gas Exploration

Robotics Trajectory control, forklift robot, manipulator

controllers, and vision systemsSpeech Speech recognition, speech compression, vowel

classification, and text-to-speech synthesisSecurities Market analysis, automatic bond rating, and

stock trading advisory systemsTelecommunications Image and data compression, automated

information services, real-time translation of spoken language, and customer payment processing systems

Transportation Truck brake diagnosis systems, vehicle

scheduling, and routing systems

Industry Business Applications

[HaRu78] You want to design a network that can predict the value of a house (in $1000s), given 13 pieces of geographical and real estate information You have a total of 506 example homes for which you have those 13 items of data and their associated market values.

You can solve this problem in three ways:

• Use a command-line function, as described in “Using Command-Line

inputs = [0 1 0 1; 0 0 1 1];

targets = [0 0 0 1];

The next section demonstrates how to train a network from the command line, after you have defined the problem This example uses the housing data set provided with the toolbox

Using Command-Line Functions

1 Load the data, consisting of input vectors and target vectors, as follows:

load house_dataset

2 Create a network For this example, you use a feed-forward network with the default tan-sigmoid transfer function in the hidden layer and linear

1 Getting Started

transfer function in the output layer This structure is useful for function approximation (or regression) problems Use 20 neurons (somewhat arbitrary) in one hidden layer The network has one output neuron, because there is only one target value associated with each input vector

net = newfit(houseInputs,houseTargets,20);

Note More neurons require more computation, but they allow the network to

solve more complicated problems More layers require more computation, but their use might result in the network solving complex problems more

efficiently

3 Train the network The network uses the default Levenberg-Marquardt algorithm for training The application randomly divides input vectors and target vectors into three sets as follows:

- 60% are used for training.

- 20% are used to validate that the network is generalizing and to stop

training before overfitting

- The last 20% are used as a completely independent test of network

Fitting a Function

1-9

This example used the train function All the input vectors to the network appear at once in a batch Alternatively, you can present the input vectors one at a time using the adapt function “Training Styles” on page 2-20

describes the two training approaches

This training stopped when the validation error increased for six iterations,

which occurred at iteration 23 If you click Performance in the training

window, a plot of the training errors, validation errors, and test errors

appears, as shown in the following figure In this example, the result is

reasonable because of the following considerations:

1 Getting Started

- The final mean-square error is small.

- The test set error and the validation set error have similar characteristics.

- No significant overfitting has occurred by iteration 17 (where the best

validation performance occurs)

4 Perform some analysis of the network response If you click Regression in

the training window, you can perform a linear regression between the network outputs and the corresponding targets

Fitting a Function

1-11

The output tracks the targets very well for training, testing, and validation,

and the R-value is over 0.95 for the total response If even more accurate

results were required, you could try any of these approaches:

• Reset the initial network weights and biases to new values with init and

train again

• Increase the number of hidden neurons.

• Increase the number of training vectors.

To get more experience in command-line operations, try some of these tasks:

• During training, open a plot window (such as the regression plot), and watch

it animate

• Plot from the command line with functions such as plotfit,

plotregression, plottrainstate and plotperform (For more information

on using these functions, see their reference pages.)

Fitting a Function

1-13

Using the Neural Network Fitting Tool GUI

1 Open the Neural Network Fitting Tool with this command:

nftool

1 Getting Started

2 Click Next to proceed.

3 Click Load Example Data Set in the Select Data window The Fitting Data

Set Chooser window opens

Fitting a Function

1-15

4 Select Simple Fitting Problem, and click Import This brings you back to

the Select Data window

Fitting a Function

1-17

6 Click Next.

The number of hidden neurons is set to 20 You can change this value in

another run if you want You might want to change this number if the

network does not perform as well as you expect

1 Getting Started

7 Click Next.

Fitting a Function

1-19

8 Click Train.

This time the training continued for the maximum of 1000 iterations

9 Under Plots, click Regression.

For this simple fitting problem, the fit is almost perfect for training, testing, and validation data