Machine learning for OpenCV

A Taste of Machine Learning Getting started with machine learning Problems that machine learning can solve Getting started with Python Getting started with OpenCV Installation Getting th

Machine Learning for OpenCV

A practical introduction to the world of machine learning and image processing using OpenCV and Python

Michael Beyeler

BIRMINGHAM - MUMBAI

Machine Learning for OpenCV

Copyright © 2017 Packt Publishing

All rights reserved No part of this book may be reproduced, stored in a

retrieval system, or transmitted in any form or by any means, without theprior written permission of the publisher, except in the case of brief

quotations embedded in critical articles or reviews

Every effort has been made in the preparation of this book to ensure the

accuracy of the information presented However, the information contained inthis book is sold without warranty, either express or implied Neither theauthor, nor Packt Publishing, and its dealers and distributors will be heldliable for any damages caused or alleged to be caused directly or indirectly bythis book

Packt Publishing has endeavored to provide trademark information about all

of the companies and products mentioned in this book by the appropriate use

of capitals However, Packt Publishing cannot guarantee the accuracy of thisinformation

First published: July 2017

Production reference: 1130717

Published by Packt Publishing Ltd.

Livery Place

Tejal Daruwale Soni

Content Development Editor

Over the last few years, our machines have slowly but surely learned how tosee for themselves We now take it for granted that our cameras detect ourfaces in pictures that we take, and that social media apps can even recognize

us and our friends in the photos that we upload from these cameras Over thenext few years we will experience even more radical transformation Beforelong, cars will be driving themselves, our cellphones will be able to read andtranslate a sign in any language for us, and our x-rays and other medicalimages will be read and analyzed by powerful algorithms that will be able toaccurately suggest a medical diagnosis, and even recommend effective

treatments

These transformations are driven by an explosive combination of increasedcomputing power, masses of image data, and a set of clever ideas taken frommath, statistics, and computer science This rapidly growing intersection that

is machine learning has taken off, affecting many of our day-to-day

interactions with the world, and with each other One of the most remarkablefeatures of the current machine learning paradigm-shift in computer vision isthat it relies to a large extent on software tools that are freely available anddeveloped by large groups of volunteers, hobbyists, scientists, and engineers

in open source communities This means that, in principle, the barriers toentry are also lower than ever: anyone who is interested in putting their mind

to it can harness machine learning for image processing

However, just like in a garden with many forking paths, the wealth of toolsand ideas, and the rapid development of these ideas, underscores the need for

a guide who can show you the way, and orient you in the right direction Ihave some good news for you: having picked up this book, you are in thegood hands of my colleague and collaborator Dr Michael Beyeler as yourguide With his broad range of expertise, Michael is both a hard-nosed

engineer, computer scientist, and neuroscientist, as well as a prolific opensource software developer He has not only taught robots how to see andnavigate through complex environments, and computers how to model brain

activity, but he also regularly teaches humans how to use programming tosolve a variety of different machine learning and image processing problems.This means that you will get to benefit not only from the sure-handed rigor ofhis expertise and experience, but also that you will get to enjoy his

thoughtfulness in teaching the ideas in his book, as well as a good dose of hissense of humor

The second piece of good news is that this going to be an exhilarating trip.There's nothing that matches the thrill of understanding that comes from

putting together the pieces of the puzzle that go into solving a problem incomputer vision and machine learning with code and data As Richard

Feynman put it: "What I cannot create, I do not understand" So, get ready toget your hands dirty (so to speak) with the code and data in the (open source!)code examples that accompany this book, and to get creative Understandingwill surely follow

Ariel Rokem

Data Scientist, The University of Washington eScience Institute

About the Author

Michael Beyeler is a Postdoctoral Fellow in Neuroengineering and Data

Science at the University of Washington, where he is working on

computational models of bionic vision in order to improve the perceptualexperience of blind patients implanted with a retinal prosthesis (bionic eye).His work lies at the intersection of neuroscience, computer engineering,

computer vision, and machine learning Michael is the author of OpenCV

with Python Blueprints by Packt Publishing, 2015, a practical guide for

building advanced computer vision projects He is also an active contributor

to several open source software projects, and has professional programmingexperience in Python, C/C++, CUDA, MATLAB, and Android

Michael received a PhD in computer science from the University of

California, Irvine as well as a MSc in biomedical engineering and a BSc inelectrical engineering from ETH Zurich, Switzerland When he is not

"nerding out" on brains, he can be found on top of a snowy mountain, in front

of a live band, or behind the piano

About the Reviewers

Vipul Sharma is a Software Engineer at a startup in Bangalore, India He

studied engineering in Information Technology at Jabalpur Engineering

College (2016) He is an ardent Python fan and loves building projects oncomputer vision in his spare time He is an open source enthusiast and huntsfor interesting projects to contribute to He is passionate about learning andstrives to better himself as a developer He writes blogs on his side projects at

http://vipul.xyz He also publishes his code at http://github.com/vipul-sharma20

Rahul Kavi works as a research scientist in Silicon Valley He holds a

Master's and PhD degree in computer science from West Virginia University.Rahul has worked on researching and optimizing computer vision

applications for a wide variety of platforms and applications He has alsocontributed to the machine learning module in OpenCV He has written

computer vision and machine learning software for prize-winning robots forNASA's 2015 and 2016 Centennial Challenges: Sample Return Robot (1stprize) Rahul's research has been published in conference papers and journals

At www.PacktPub.com, you can also read a collection of free technical articles,sign up for a range of free newsletters and receive exclusive discounts andoffers on Packt books and eBooks.

https://www.packtpub.com/mapt

Get the most in-demand software skills with Mapt Mapt gives you full

access to all Packt books and video courses, as well as industry-leading tools

to help you plan your personal development and advance your career

Customer Feedback

Thanks for purchasing this Packt book At Packt, quality is at the heart of oureditorial process To help us improve, please leave us an honest review onthis book's Amazon page at https://www.amazon.com/dp/1783980281

If you'd like to join our team of regular reviewers, you can e-mail us at

customerreviews@packtpub.com We award our regular reviewers with free eBooksand videos in exchange for their valuable feedback Help us be relentless inimproving our products!

To my loving wife, who continues to support me in all my

endeavors ; no matter how grand, silly, or nerdy they may be.

Table of Contents

Preface

What this book covers

What you need for this book

Who this book is for

1 A Taste of Machine Learning

Getting started with machine learning

Problems that machine learning can solve

Getting started with Python

Getting started with OpenCV

Installation

Getting the latest code for this book Getting to grips with Python's Anaconda distribution Installing OpenCV in a conda environment

Verifying the installation Getting a glimpse of OpenCV's ML module Summary

2 Working with Data in OpenCV and Python

Understanding the machine learning workflow

Dealing with data using OpenCV and Python

Starting a new IPython or Jupyter session Dealing with data using Python's NumPy package Importing NumPy

Understanding NumPy arrays Accessing single array elements by indexing Creating multidimensional arrays

Loading external datasets in Python Visualizing the data using Matplotlib

Importing Matplotlib Producing a simple plot Visualizing data from an external dataset Dealing with data using OpenCV's TrainData container in C++

Summary

3 First Steps in Supervised Learning

Understanding supervised learning

Having a look at supervised learning in OpenCV Measuring model performance with scoring functions Scoring classifiers using accuracy, precision, and recall Scoring regressors using mean squared error, explained variance, and R squa red

Using classification models to predict class labels

Understanding the k-NN algorithm Implementing k-NN in OpenCV Generating the training data Training the classifier Predicting the label of a new data point Using regression models to predict continuous outcomes

Understanding linear regression Using linear regression to predict Boston housing prices Loading the dataset

Training the model Testing the model Applying Lasso and ridge regression Classifying iris species using logistic regression

Understanding logistic regression Loading the training data Making it a binary classification problem Inspecting the data

Splitting the data into training and test sets Training the classifier

Testing the classifier Summary

4 Representing Data and Engineering Features

Understanding feature engineering

Preprocessing data

Standardizing features

Normalizing features Scaling features to a range Binarizing features Handling the missing data Understanding dimensionality reduction

Implementing Principal Component Analysis (PCA) in OpenCV Implementing Independent Component Analysis (ICA)

Implementing Non-negative Matrix Factorization (NMF) Representing categorical variables

Representing text features

Representing images

Using color spaces Encoding images in RGB space Encoding images in HSV and HLS space Detecting corners in images

Using the Scale-Invariant Feature Transform (SIFT) Using Speeded Up Robust Features (SURF)

Summary

5 Using Decision Trees to Make a Medical Diagnosis

Understanding decision trees

Building our first decision tree Understanding the task by understanding the data Preprocessing the data

Constructing the tree Visualizing a trained decision tree Investigating the inner workings of a decision tree Rating the importance of features

Understanding the decision rules Controlling the complexity of decision trees Using decision trees to diagnose breast cancer

Loading the dataset Building the decision tree Using decision trees for regression

Summary

6 Detecting Pedestrians with Support Vector Machines

Understanding linear support vector machines

Learning optimal decision boundaries Implementing our first support vector machine

Generating the dataset Visualizing the dataset Preprocessing the dataset Building the support vector machine Visualizing the decision boundary Dealing with nonlinear decision boundaries

Understanding the kernel trick Knowing our kernels

Implementing nonlinear support vector machines Detecting pedestrians in the wild

Obtaining the dataset Taking a glimpse at the histogram of oriented gradients (HOG) Generating negatives

Implementing the support vector machine Bootstrapping the model

Detecting pedestrians in a larger image Further improving the model

Summary

7 Implementing a Spam Filter with Bayesian Learning

Understanding Bayesian inference

Taking a short detour on probability theory Understanding Bayes' theorem

Understanding the naive Bayes classifier Implementing your first Bayesian classifier

Creating a toy dataset Classifying the data with a normal Bayes classifier Classifying the data with a naive Bayes classifier Visualizing conditional probabilities

Classifying emails using the naive Bayes classifier

Loading the dataset Building a data matrix using Pandas Preprocessing the data

Training a normal Bayes classifier Training on the full dataset Using n-grams to improve the result Using tf-idf to improve the result Summary

8 Discovering Hidden Structures with Unsupervised Learning

Understanding unsupervised learning

Understanding k-means clustering

Implementing our first k-means example Understanding expectation-maximization

Implementing our own expectation-maximization solution Knowing the limitations of expectation-maximization First caveat: No guarantee of finding the global optimum Second caveat: We must select the number of clusters beforehand Third caveat: Cluster boundaries are linear

Fourth caveat: k-means is slow for a large number of samples Compressing color spaces using k-means

Visualizing the true-color palette Reducing the color palette using k-means Classifying handwritten digits using k-means

Loading the dataset Running k-means Organizing clusters as a hierarchical tree

Understanding hierarchical clustering Implementing agglomerative hierarchical clustering Summary

9 Using Deep Learning to Classify Handwritten Digits

Understanding the McCulloch-Pitts neuron

Understanding the perceptron

Implementing your first perceptron

Generating a toy dataset Fitting the perceptron to data Evaluating the perceptron classifier Applying the perceptron to data that is not linearly separable Understanding multilayer perceptrons

Understanding gradient descent Training multi-layer perceptrons with backpropagation Implementing a multilayer perceptron in OpenCV Preprocessing the data

Creating an MLP classifier in OpenCV Customizing the MLP classifier Training and testing the MLP classifier Getting acquainted with deep learning

Getting acquainted with Keras Classifying handwritten digits

Loading the MNIST dataset Preprocessing the MNIST dataset Training an MLP using OpenCV Training a deep neural net using Keras Preprocessing the MNIST dataset Creating a convolutional neural network Fitting the model

Summary

10 Combining Different Algorithms into an Ensemble

Understanding ensemble methods

Understanding averaging ensembles Implementing a bagging classifier Implementing a bagging regressor Understanding boosting ensembles Implementing a boosting classifier Implementing a boosting regressor Understanding stacking ensembles Combining decision trees into a random forest

Understanding the shortcomings of decision trees Implementing our first random forest

Implementing a random forest with scikit-learn Implementing extremely randomized trees Using random forests for face recognition

Loading the dataset Preprocessing the dataset Training and testing the random forest Implementing AdaBoost

Implementing AdaBoost in OpenCV Implementing AdaBoost in scikit-learn Combining different models into a voting classifier Understanding different voting schemes Implementing a voting classifier

Evaluating a model in the right way Selecting the best model

Implementing a simple grid search Understanding the value of a validation set Combining grid search with cross-validation Combining grid search with nested cross-validation Scoring models using different evaluation metrics

Choosing the right classification metric Choosing the right regression metric Chaining algorithms together to form a pipeline

Implementing pipelines in scikit-learn Using pipelines in grid searches Summary

12 Wrapping Up

Approaching a machine learning problem

Building your own estimator

Writing your own OpenCV-based classifier in C++ Writing your own scikit-learn-based classifier in Python Where to go from here?

Summary

I'm glad you're here It's about time we talked about machine learning

Machine learning is no longer just a buzzword, it is all around us: from

protecting your email, to automatically tagging friends in pictures, to

predicting what movies you like As a subfield of data science, machine

learning enables computers to learn through experience: to make predictionsabout the future using collected data from the past

And the amount of data to be analyzed is enormous! Current estimates put thedaily amount of produced data at 2.5 exabytes (or roughly 1 billion

gigabytes) Can you believe it? This would be enough data to fill up 10

million blu-ray discs, or amount to 90 years of HD video In order to dealwith this vast amount of data, companies such as Google, Amazon,

Microsoft, and Facebook have been heavily investing in the development ofdata science platforms that allow us to benefit from machine learning

wherever we go scaling from your mobile phone application all the way tosupercomputers connected through the cloud

In other words: this is the time to invest in machine learning And if it is yourwish to become a machine learning practitioner, too then this book is foryou!

But fret not: your application does not need to be as large-scale or influential

as the above examples in order to benefit from machine learning Everyonestarts small Thus, the first step of this book is to introduce you to the

essential concepts of statistical learning, such as classification and regression,with the help of simple and intuitive examples If you have already studiedmachine learning theory in detail, this book will show you how to put yourknowledge into practice Oh, and don't worry if you are completely new tothe field of machine learning all you need is the willingness to learn

Once we have covered all the basic concepts, we will start exploring various

algorithms such as decision trees, support vector machines, and Bayesiannetworks, and learn how to combine them with other OpenCV functionality.Along the way, you will learn how to understand the task by understandingthe data and how to build fully functioning machine learning pipelines.

As the book progresses, so will your machine learning skills, until you areready to take on today's hottest topic in the field: deep learning Combinedwith the trained skill of knowing how to select the right tool for the task, wewill make sure you get comfortable with all relevant machine learning

What this book covers

Chapter 1, A Taste of Machine Learning, will gently introduce you to the

different subfields of machine learning, and explain how to install OpenCVand other essential tools in the Python Anaconda environment

Chapter 2, Working with Data in OpenCV and Python, will show you what a

typical machine learning workflow looks like, and where data comes in toplay I will explain the difference between training and test data, and showyou how to load, store, manipulate, and visualize data with OpenCV andPython

Chapter 3, First Steps in Supervised Learning, will introduce you to the topic of

supervised learning by reviewing some core concepts, such as classificationand regression You will learn how to implement a simple machine learningalgorithm in OpenCV, how to make predictions about the data, and how toevaluate your model

Chapter 4, Representing Data and Engineering Features, will teach you how to

get a feel for some common and well-known machine learning datasets andhow to extract the interesting stuff from your raw data

Chapter 5, Using Decision Trees to Make a Medical Diagnosis, will show you

how to build decision trees in OpenCV, and use them in a variety of

classification and regression problems

Chapter 6, Detecting Pedestrians with Support Vector Machines, will explain

how to build support vector machines in OpenCV, and how to apply them todetect pedestrians in images

Chapter 7, Implementing a Spam Filter with Bayesian Learning, will introduce

you to probability theory, and show you how you can use Bayesian inference

to classify emails as spam or not

Chapter 8, Discovering Hidden Structures with Unsupervised Learning, will

talk about unsupervised learning algorithms such as k-means clustering andExpectation-Maximization, and show you how they can be used to extracthidden structures in simple, unlabeled datasets

Chapter 9, Using Deep Learning to Classify Handwritten Digits, will introduce

you to the exciting field of deep learning Starting with the perceptron andmulti-layer perceptrons, you will learn how to build deep neural networks inorder to classify handwritten digits from the extensive MNIST database

Chapter 10, Combining Different Algorithms into an Ensemble, will show you

how to effectively combine multiple algorithms into an ensemble in order toovercome the weaknesses of individual learners, resulting in more accurateand reliable predictions

Chapter 11, Selecting the Right Model with Hyper-Parameter Tuning, will

introduce you to the concept of model selection, which allows you to

compare different machine learning algorithms in order to select the right toolfor the task at hand

Chapter 12, Wrapping Up, will conclude the book by giving you some useful

tips on how to approach future machine learning problems on your own, andwhere to find information on more advanced topics

What you need for this book

You will need a computer, Python Anaconda, and enthusiasm Lots of

enthusiasm Why Python?, you may ask The answer is simple: it has become

the de facto language of data science, thanks to its great number of opensource libraries and tools to process and interact with data

One of these tools is the Python Anaconda distribution, which provides allthe scientific computing libraries we could possibly ask for, such as NumPy,SciPy, Matplotlib, Scikit-Learn, and Pandas In addition, installing OpenCV

is essentially a one-liner No more flipping switches in cc make or compilingfrom scratch! We will talk about how to install Python Anaconda in Chapter 1,

A Taste of Machine Learning.

If you have mostly been using OpenCV in combination with C++, that's fine.But, at least for the purpose of this book, I would strongly suggest that youswitch to Python C++ is fine when your task is to develop high-performancecode or real-time applications But when it comes to picking up a new skill, Ibelieve Python to be a fundamentally better choice of language, because youcan do more by typing less Rather than getting annoyed by the syntacticsubtleties of C++, or wasting hours trying to convert data from one formatinto another, Python will help you concentrate on the topic at hand: to

become an expert in machine learning

Who this book is for

Throughout the book, I will assume that you already have a basic knowledge

of OpenCV and Python, but that there is always room to learn more

In this book, you will find a number of text styles that distinguish betweendifferent kinds of information Here are some examples of these styles and anexplanation of their meaning Code words in text, database table names,

folder names, filenames, file extensions, pathnames, dummy URLs, userinput, and Twitter handles are shown as follows: "In Python, we can create alist of integers by using the list() command." A block of code is set using theIPython notation, marking user input with In [X], line continuations with andcorresponding output with Out[X]:

"Clicking the Next button moves you to the next screen."

Warnings or important notes appear in a box like this.

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome Let us know what you thinkabout this book what you liked or disliked Reader feedback is important for

us as it helps us develop titles that you will really get the most out of

To send us general feedback, simply email feedback@packtpub.com, and mentionthe book's title in the subject of your message

If there is a topic that you have expertise in and you are interested in eitherwriting or contributing to a book, see our author guide at www.packtpub.com/author

s

Customer support

Now that you are the proud owner of a Packt book, we have a number ofthings to help you to get the most from your purchase

Downloading the example code

You can download the latest version of the example code files for this bookfrom GitHub: http://github.com/mbeyeler/opencv-machine-learning All code is releasedunder the MIT software license, so you are free to use, adapt, and share thecode as you see fit There you will also be able to explore the source code bybrowsing through the different Jupyter notebooks

If you get stuck or have questions about the source code, you are welcome topost in our web forum: https://groups.google.com/d/forum/machine-learning-for-opencv.Chances are, someone else has already shared a solution to your specificproblem

Alternatively, you can download the original code files from the date of

publication by visiting your account at http://www.packtpub.com If you purchasedthis book elsewhere, you can visit http://www.packtpub.com/support and register tohave the files e-mailed directly to you

You can download the code files by following these steps:

1 Log in or register to our website using your e-mail address and

password

2 Hover the mouse pointer on the SUPPORT tab at the top

3 Click on Code Downloads & Errata

4 Enter the name of the book in the Search box

5 Select the book for which you're looking to download the code files

6 Choose from the drop-down menu where you purchased this book from

7 Click on Code Download

You can also download the code files by clicking on the Code Files button on

the book's webpage at the Packt Publishing website This page can be

accessed by entering the book's name in the Search box Please note that youneed to be logged in to your Packt account

Once the file is downloaded, please make sure that you unzip or extract thefolder using the latest version of:

WinRAR / 7-Zip for Windows

Zipeg / iZip / UnRarX for Mac

7-Zip / PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktP ublishing/Machine-Learning-For-OpenCV We also have other code bundles from ourrich catalog of books and videos available at https://github.com/PacktPublishing/.Check them out!

Although we have taken every care to ensure the accuracy of our content,mistakes do happen If you find a mistake in one of our books maybe a

mistake in the text or the code we would be grateful if you could report this

to us By doing so, you can save other readers from frustration and help usimprove subsequent versions of this book If you find any errata, please

report them by visiting http://www.packtpub.com/submit-errata, selecting your book,clicking on the Errata Submission Form link, and entering the details of yourerrata Once your errata are verified, your submission will be accepted andthe errata will be uploaded to our website or added to any list of existingerrata under the Errata section of that title

To view the previously submitted errata, go to https://www.packtpub.com/books/conten t/support and enter the name of the book in the search field The required

information will appear under the Errata section

Piracy of copyrighted material on the Internet is an ongoing problem acrossall media At Packt, we take the protection of our copyright and licenses veryseriously If you come across any illegal copies of our works in any form onthe Internet, please provide us with the location address or website nameimmediately so that we can pursue a remedy

Please contact us at copyright@packtpub.com with a link to the suspected piratedmaterial

We appreciate your help in protecting our authors and our ability to bring youvaluable content

If you have a problem with any aspect of this book, you can contact us at

questions@packtpub.com, and we will do our best to address the problem

A Taste of Machine Learning

I am writing a new line with double spaces

So, you have decided to enter the field of machine learning That's great!

Nowadays, machine learning is all around us from protecting our email, toautomatically tagging our friends in pictures, to predicting what movies we

like As a form of artificial intelligence, machine learning enables computers

to learn through experience: to make predictions about the future using

collected data from the past On top of that, computer vision is one of

today's most exciting application fields of machine learning, with deep

learning and convolutional neural networks driving innovative systems such

as self-driving cars and Google's DeepMind

However, fret not; your application does not need to be as large-scale or

world-changing as the previous examples in order to benefit from machinelearning In this chapter, we will talk about why machine learning has

become so popular and discuss the kinds of problems that it can solve Wewill then introduce the tools that we need in order to solve machine learningproblems using OpenCV Throughout the book, I will assume that you

already have a basic knowledge of OpenCV and Python, but that there isalways room to learn more

Are you ready then? Let's go!

Getting started with machine

learning

Machine learning has been around for at least 60 years Growing out of thequest for artificial intelligence, early machine learning systems used hand-coded rules of if else statements to process data and make decisions Think

of a spam filter whose job is to parse incoming emails and move unwantedmessages to a spam folder:

Spam filter

We could come up with a blacklist of words that, whenever they show up in amessage, would mark an email as spam This is a simple example of a hand-

coded expert system (We will build a smarter one in Chapter 7, Implementing

a Spam Filter with Bayesian Learning.)

We can think of these expert decision rules to become arbitrarily complicated

if we are allowed to combine and nest them in what is known as a decision

tree (Chapter 5, Using Decision Trees to Make a Medical Diagnosis) Then, it

becomes possible to make more informed decisions that involve a series ofdecision steps, as shown in the following image: