mobile image processing part 3

Mobile image processing – Part 3„ Mobile application development on Android „ Project 1: “Hello Image”, reading/writing/modifying images „ Project 2: “Viewfinder EE368”, processing viewf

Mobile image processing – Part 3

„ Mobile application development on Android

„ Project 1: “Hello Image”, reading/writing/modifying images

„ Project 2: “Viewfinder EE368”, processing viewfinder frames

„ Project 3: “CVCamera”, utilizing OpenCV functions

„ Mobile application examples

Mobile feature tracking

Mobile book spine recognition

Mobile book spine recognition

Server

Mobile Device

TrackCamera Pose

Check Geometry

Send Response

Time

Wireless Network

Viewfinder Frames

Signal Processing and Linear Systems

Extract FeaturesSegment

Spines

Jon’s Java Imaging Library (JJIL)

„ Publicly available source code

http://code.google.com/p/jjil/

„ Collection of common image processing functions

„ Images are processed through Sequences

Rgb 3x3Average

Rgb SubSample

Sequence RgbImage

RgbImage

Sequence.Push

JJIL: 3x3 averaging filter

Rgb3x3Average

JJIL: Horizontal differences

RgbAvgGray Gray8HorizSimpleEdge

Gray8DeblurHorizHalftone Gray8DetectHaarMultiScale Gray8Fft

Gray8GaussDeblurHoriz Gray8GaussHoriz

Gray8GaussSmoothVert Gray8Gray32

Gray8Hist Gray8HistEq Gray8HistMatch Gray8HorizSimpleEdge Gray8HorizSum

Gray8HorizVar Gray8HorizVertContrast Gray8InverseFilter

Gray8LinComb

Gray8Peak3x3 Gray8Reduce Gray8Shrink LinefitHough Rgb3x3Average RgbAvgGray RgbClip

RgbCrop RgbDimMask RgbHorizGaussSmooth RgbHsv

RgbMaskedAbsDiff RgbMaxContrast2Gray RgbMaxDiff

RgbMeanStdDev RgbMinDiff

RgbSelectGray RgbSubSample

…

“CVCamera” project

„ Goals of this project

z Learn how to incorporate C/C++ code into an Android program

z Learn how to utilize OpenCV library functions

z Learn how to draw feature keypoints on viewfinder frames

„ Step-by-step instructions available in Tutorial #2:

z http://ee368.stanford.edu/Android

OpenCV: Open Source C omputer V ision

http://opencv.willowgarage.com/wiki/Welcome

OpenCV for Android

„ Port of OpenCV to Android framework

z Over 2K optimized algorithms written in C/C++

z Compiled with customized Android NDK (Crystax r4)

z Enables popular CV functions to be used on mobile images/videos

„ Differences from regular Android programming

z Requires writing C/C++ code

z Requires writing Java Native Interface (JNI) wrappers

z Requires using Android NDK in addition to Android SDK

„ Class tutorial #2 helps you make the transition

z How to set up OpenCV programming environment

z How to write Android apps that call OpenCV functions

z How to integrate NDK compilation into Eclipse IDE

Project structure for “CVCamera”

CVCamera Activity (Java)

Processor (C++)

CVCamera

JNI (Java)

Manages user interface and program execution

Processes a “pool”

of images

Interfaces with C/C++ code

OpenCV Library

CVCamera class: menu options

public boolean onCreateOptionsMenu(Menu menu) {

CVCamera class: calling feature extractors

public boolean onOptionsItemSelected(MenuItem item) {

Project structure for “CVCamera”

CVCamera Activity (Java)

Processor (C++)

CVCamera

JNI (Java)

Manages user interface and program execution

Processes a “pool”

of images

Interfaces with C/C++ code

OpenCV Library

CVCamera JNI class: interface to C/C++ code

public final static native int DETECT_FAST_get();

public final static native int DETECT_STAR_get();

public final static native int DETECT_SURF_get();

public final static native int DETECT_MSER_get();

public final static native long new_Processor();

public final static native void delete_Processor(long jarg1);

Each function prototype corresponds to an external C/C++ function

Load OpenCV and CVCamera

libraries built by the NDK

Project structure for “CVCamera”

CVCamera Activity (Java)

Processor (C++)

CVCamera

JNI (Java)

Manages user interface and program execution

Processes a “pool”

of images

Interfaces with C/C++ code

OpenCV Library

Processor class: initialize feature detectors

void detectAndDrawFeatures (int idx, image_pool* pool, int feature_type);

};

Instantiation list

Different feature extractors stored

as member objects

Processor class: detect and draw keypoints

// Detect feature keypoints

Mat grey_im = pool->getGrey(idx);

Mat color_im = pool->getImage(idx); my_keypoints.clear();

my_mserd->detect(grey_im,

my_keypoints);

// Draw feature keypoints

vector<KeyPoint>::const_iterator it; for (it = my_keypoints.begin();

it != my_keypoints.end(); ++it) {

// Draw black circle

CVCamera demo

http://ee368.stanford.edu/Android

Communications between phone and server

Phone

Server HTTP Post (e.g., Image, Features)

HTTP Get (e.g., Identification)

Wireless Network

Apache PHP MATLAB / C

HTTP file upload

URL connectURL = new URL( web address );

HttpURLConnection conn = connectURL.openConnection();

conn.setRequestMethod(“ POST ”);

DataOutputStream dos = new DataOutputStream(conn.getOutputStream()); FileInputStream fis = new FileInputStream( file location );

int bufferSize = 1024 ;

byte[] buffer = new byte[bufferSize];

int bytesRead = fileInputStream.read(buffer, 0 , bufferSize);

while (bytesRead > 0 ) {

dos.write(buffer, 0 , bufferSize);

int bytesAvailable = fis.available();

bufferSize = Math.min(bytesAvailable, bufferSize);

bytesRead = fis.read(buffer, 0 , bufferSize);

}

dos.flush();

dos.close();

fis.close();

Class Project: Mobile Panorama Creation

Class Project: Mobile Face Recognition

G Davo K Sriadibhatla, and X Chao, Spring 2010

http://ee368.stanford.edu/Project_10

„ Android-related office hours

Wednesday, 3:30pm – 5:00pm, SCIEN Lab

„ Reference book

Meier, Professional Android 2 Application Development