Aerial Vehicles Part 11 pot

Pure translation models were then used to recover the camera trajectory from images of a horizontal planar area, and they were found to be especially suitable for the estimation of the h

UAV were discussed Long endurance UAV flights require a number of aspects to be taken into consideration during the design phase In section two an overview of potential renewable power sources for long endurance UAVs were presented It was shown how a hybrid combination of photovoltaic cells and Li-Po batteries can fulfil the requirements of a long endurance UAV power source Fuel cell power sources are attractive power sources for shorter duration UAV flights

Section three showed by coupling the low cost inertial navigation system to a suitable control system how a complete navigation solution can be provided for long endurance UAV flights A number of control techniques are discussed enabling the construction of autopilots for autonomous flight applications

The field of image processing is a rapidly developing field Since imaging sensors are ideal UAV payload sensors, advances in image processing directly benefits many UAV applications A number of sensor payload design consideration are discussed with regard

to long endurance UAV missions

In an overview paper in 2007, Kenzo Nonami (Nonami, 2007) indicated the following aspects as important future research areas for UAV civilian use:

• Formation flight control (for data relay, in-air refuelling, observation work) with a possible flight control accuracy in the cm-order;

• Integrated hierarchical control in order to fly different classes of UAVs simultaneously

An example sited is that of coordinating various sizes of UAVs and MAVs with a larger supervisory UAV;

• High altitude flight, e.g flights in the stratosphere for scientific observation missions;

• High precision trajectory following flight;

• All weather flight;

Adams, J.A.; Cooper, J.L.; Goodrich, M.A.; Humphrey, C.; Quigley, M.; Buss, B.G & Morse,

B.S (2007); Camera-equipped mini UAVs for wilderness search support: Task analysis and lessons from field trials, Byuhcmi Technical Report 2007-1

Awan, S.H (2007); UAV applications in atmospheric research, MSc Aerodynamics, Cranefield

University School of Engineering

Barbir, F (2005); PEM Fuel Cells: Theory and Practice (Sustainable World Series); Academic

Press; ISBN-10: 0120781425, 978-0120781423

Barrows, G.L (2002); Future Visual Microsensors for Mini/Micro-UAV applications,

Proceedings of the Workshop on Cellular Neural Networks and their Applications

Bar-Shalom, Y & Li, X (2001); Estimation with Applications to Tracking and Navigation;

John Wiley & Sons; ISBN – 047141655X; Hoboken, NJ, USA

Bendeaa, H.; Chiabrandoa, F; Tonolob, F.G & Marenchino, D (2007); Mapping of

archaeological areas using a low-cost UAV - The Augusta Bagiennorum test site,

Proceedings of the 21st International CIPA Symposium, 01-06 October 2007, Athens,

Greece

Bird, R E & Halstrom, R L (1981); A Simplified Clear Sky Model for Direct and Diffuse

Insolation on Horizontal Surfaces; Solar Energy Research Institute;

SERI/TR-642-761

Blakelock, J.H (1991); Automatic Control of Aircraft and Missiles – Second Edition; Wiley

InterScience; ISBN – 0471506516; Hoboken, NJ, USA

Britting, K.R.( 1971); Inertial Navigation Systems Analysis; Wiley Interscience; ISBN –

047110485X; Hoboken, NJ, USA

Caballero, F.; Merino, L.; Ferruz, L & Ollero, A (2005); A visual odometer without 3D

reconstruction for aerial vehicles Applications to building inspection, Proceedings of the 2005 IEEE International Conference on Robotics and Automation Barcelona, Spain Casbeer, D.W & Li,S.M.(2005); Forest fire monitoring with multiple small UAVs, Proceedings

of the 2005 American Control Conference, June 8-10, 2005, Portland, USA, pp.3530-3535

Conway, B E (1999); Electrochemical Supercapacitors: Scientific Fundamentals and

Technological Applications; Springer; ISBN-10: 0306457369, 978-0306457364

Coradeschi, S.; Karlsson, L & Nordberg, K (1999); Integration of vision and

decision-making in an autonomous airborne vehicle for traffic surveillance, Computer Vision systems, Springer, Berlin/Heidelberg

Cramer, M (2001); On the use of Direct Georeferencing in Airborne Photogrammetry,

Proceedings of the 3rd International Symposium on Mobile Mapping Technology, January

2001, Cairo

Dahlgren, J.W (2006); Real Options and Value Driven in Spiral Development, Proceedings of

the CCRTS 2006, March 31, 2006

Department of Defence (2005); Unmanned Aircraft Systems Roadmap: 2005 – 2030,

Doherty, P (2004); Advanced Research with Autonomous Unmanned Aerial Vehicles,

Proceedings on the 9th International Conference on Principles of Knowledge Representation and Reasoning

Eisenbeiss, H.& Zhang, L (2006); Comparison of DSMs generated from mini UAV imagery

and terrestrial laser scanner in a cultural heritage application, Proceedings of IAPRS

vol 36, part 5, Dresden 25-27 September 2006

Eisenbeiss, H.; Lambers, K & Sauerbier, M (2005); Photogrammetric recording of the

archaeological site of Pinchango Alto (Palpa, Peru) using a mini helicopter (UAV),

Proceedings of the 33rd CAA Conference, Tomar, Portugal, 21-24 March 2005

Farrell, J.A & Bath, M (1999); The Global Positioning System & Inertial Navigation;

McGraw Hill; ISBN – 0-07-022045-X; New York, USA

Farrell, J.A (2008); Aided Navigation – GPS with High Rate Sensors; McGraw Hill; ISBN –

978-0-07-149329-1; New York, USA

Fisher, R.B.; Dawson-Howe, K.; Fitzgibbon, A.; Robertson, C.& Trucco, E (2005); Dictionary

of Computer Vision and Image Processing, John Wiley & Sons, Ltd, England, 2005 Franklin, G.F & Powell, J.D & Emami-Naeini, A (2006): Feedback Control of Dynamic Systems

(Fifth edition); Addison-Wesley, ISBN - 978-0131499300, Reading, MA

Groves, P.D (2008); Principles of GNSS, Inertial and Multisensor Integrated Navigation Systems;

Artech House; ISBN – 1-580-53255-1; Boston, USA

Heintz, F & Doherty, P (2004); A distributed architecture for UAV experimentation, In the

Proceedings of the AAAI Workshop on Anchoring Symbols to Sensor Data, 2004

Horcher, A & Visser, R.J.M (2004); Unmanned aerial vehicles: Applications for natural

resource management and monitoring, Proceedings of the Council Of Forest Engineering, 2004

Hornberg, A.(2006); Handbook of Machine Vision, Wiley-VCH, 2006, ISBN 3-527-40584-4

Hruska, R.C.; Lancaster, G.D.; Jerry H.; Harbour, L & Cherry, S.J (2005); Small

UAV-Acquired, High-resolution, Georeferenced Still Imagery, Proceedings of the Wildlife Society 12th Annual Conference, Baltimore, Maryland,28-30 May 2005

Joshi, R.; Bose, A & Breneman, S (2002); Efficient development of UAV electronics using

software frameworks, Proceedings of the AIAA 1st Conference and Workshop on Unmanned Aerospace Vehicles, Portsmouth, Virginia, May 20-23, 2002

Kahn, A.D (2001); The Design and Development of a Modular Avionics System, Master

Thesis, Georgia Institute of Technology, April 2001

Karr, D.A.; Rodrigues, C.; Krishnamurthy, Y.; Pyarali, I.; Loyall, J.P & Schantz, R.E.(2001);

Application of the QuO Quality-of-Service Framework to a Distributed Video

Application, Proceedings of the Third International Symposium on Distributed Objects and Applications (DOA'01), pp 299, 2001

Kim, Z.W (2005); Realtime road detection by learning from one example, Proceedings of the

Seventh IEEE Workshops on Application of Computer Vision, 2005 vol.1, pp.455-460, 5-7

Jan 2005

Kontitsis, M.; Valavanis, K.P & Tsourveloudis, N (2004); A UAV Vision System for

Airborne Surveillance, Proceedings of IEEE International Conference on ICRA '04, pp

77-83, 26 April-1 May 2004

Legat, K & Skaloud, J (2006); Reliability of direct georeferencing - A case study on practical

problems and solutions, Report to EuroSDR Commission, 2006

Lewis, F.L & Xie, L & Popa, D (2008); Optimal and Robust Estimation with an Introduction

to Stochastic Control Theory; Taylor & Francis Group LLC; ISBN – 0-849-39008-7; Boca Raton, FL, USA

Li, Y.; Atmosukarto, U.; Kobashi, M.; Yuen, J & Shapiro, L.G (2005); Object and event

recognition for aerial surveillance, Proceedings- SPIE the international society for optical engineering, 2005, vol 5781, pp 139-149

Luque, A & Hegedus, S (2003); Handbook of Photovoltaic Science and Engineering; John Wiley

& Sons; ISBN 0471491969, 9780471491965; Hoboken, NJ, USA

Maybeck, P.S (1994 - reprint); Stochastics Models, Estimation, and Control - Volume 1; Navtech

Books & Software Store;

McRuer, D & Ashkenas, I & Graham, D (1973); Aircraft Dynamics and Automatic Control;

Princeton University Press; ISBN – 0691024405; Princeton, NJ, USA

Merino, L & Ollero, A (2002); Computer vision techniques for fire monitoring using aerial

images, Proceedings of IECON 02 (28th Annual Conference of the Industrial Electronics Society), vol.3, pp.2214-2218, 5-8 Nov 2002

Merino, L.; Caballero, F.; Martinez-de Dios, J.R.; Ferruz, J & Ollero, A (2006); A cooperative

perception system for multiple UAVs: application to automatic detection of forest

fires, Journal of Field Robotics, vol.23, issue 3-4, pp.165-184, 2006

Muneer, T.; Gueymard, C & Kambezidis, H (2004); Solar Radiation and Daylight Models;

Butterworth-Heinemann; ISBN 0750659742, 9780750659741

Niranjan, S.; Gupta, G.; Sharma, N.; Mangal, M & Singh, V (2007); Initial Efforts toward

Mission-specific Imaging Surveys from Aerial Exploring Platforms, Proceedings of the Map World Forum, January 22 - 25, 2007, Hyderabad, India

Nonami, K (2007); Prospect and Recent Research & Development for Civil Use

Autonomous Unmanned Aircraft as UAV and MAV, Journal of System Design and Dynamics, Vol.1, No.2 Special Issue on New Trends of Motion and Vibration

Control pp.120-128, 2007

Nordberg, K.; Doherty, P.; Farneback, G.; Forssen, P.; Granlund, G.; Moe, A & Wiklund,

J.(2002); Vision for a UAV helicopter, Proceedings of IROS'02, Workshop on aerial robotics, October, 2002

Okrent, M.(2004); Civil UAV Activity within the framework of European Commission

Research, Proceedings of the AIAA 3rd "Unmanned Unlimited" Technical Conference,

Workshop and Exhibit, Chicago, Illinois, Sep 20-23, 2004

Puri, A (2004); A survey of Unmanned Aerial Vehicles (UAV) for Traffic Surveillance,

Technical Report 2004

Rathinam, S.; Kim, Z.; Soghikian, A & Sengupta, R (2005); Vision based following of locally

linear structures using an Unmanned Aerial Vehicle, 44th IEEE Conference on Decision and Control, 2005 and 2005 European Control Conference, pp.6085-6090, 12-

15 Dec 2005

Rogers, R.M (2007); Applied Mathematics in Integrated Navigation Systems – Third Edition;

American Institute of Aeronautics and Astronautics; ISBN – 1-56347-927-3; Reston,

VA, USA

Roskam, J (1979) ; Airplane Flight Dynamics and Automatic Flight Controls; Roskam Aviation

and Engineering Corp.; ISBN – 1-884-88517-9; Lawrence, Kans.:

Sarris, Z.(2001); Survey of UAV applications in Civil Markets, Proceedings of the 9th

Mediterranean Conference on Control and Automation, Dubrovnik, Croatia, June 27-29,

2001

Simon, D (2006); Optimal State Estimation; Wiley InterScience; ISBN – 0-471-70858-5;

Hoboken, NJ, USA

Sinha, A (2007); Linear Systems – Optimal and Robust Control; Taylor & Francis Group LLC;

ISBN – 0-8493-9217-9; Boca Raton, FL, USA

Skaloud, J.& Schaer, P.(2003); Towards a More Rigorous Boresight Calibration, Proceedings

of the ISPRS International Workshop on Theory, Technology and Realities of Inertial/GPS/Sensor Orientation, Commission 1, WG I/5, Castelldefels, Spain,

Semptember 9-12

Skogestad, S & Postlethwaite, I (2005); Multivariable Feedback Control Analysis and Design;

John Wiley & Sons; ISBN – 0-470-01168-8; Chichester, England

Stevens, B.L & Lewis, F.L (2003); Aircraft Control and Simulation – Second Edition; John

Wiley & Sons Inc.; ISBN – 0-471-37145-9; Hoboken, NJ, USA

Titterton, D.H.; & Weston, J.L.( 2004) Strapdown Inertial Navigation Technology – Second

Edition, American Institute of Aeronautics and Astronautics; ISBN – 1-56347-693-2; Reston, VA, USA

Valavanis, K.P., (2007) Advances in Unmanned Aerial Vehicles – State of the Art and the Road to

Autonomy; Springer, ISBN – 978-1-4020-6113-4, Dordrecht, The Netherlands

Van Schalkwijk, W A & Scrosati, B (2002); Advances in lithium-ion batteries; Springer; ISBN

0306473569, 9780306473562

Veth, M.; Anderson, R.; Webber, F & Nielsen, M (2008); Tightly-Coupled INS, GPS, and

Imaging Sensors for Precision Geolocation, Proceedings of the 2008 National Technical Meeting of the Institute of Navigation, January 28 - 30, 2008, San Diego, California

Westall, P.; Carnie, R.J.; O’Shea, P.; Hrabar, S & Walker, R.A (2007); Vision-based UAV

maritime search and rescue using point target detection, Proceedings of the 12th Australian International Aerospace Congress, 19-22 March 2007

Wilson, J.R.(2002); The Incredible Shrinking Sensor, Military & Aerospace Electronics

Magazine, March 2002

Wosylus, A (2008); High-end graphics conquers embedded computing applications, In

Boards & Solutions - The European Embedded Computing Magazine, Vol.2, pp.6-10,

April 2008

Zarchan, P & Musoff, H (2005); Fundamentals of Kalman Filtering: A Practical Approach –

Second Edition; American Institute of Aeronautics and Astronautics; ISBN – 47694-0; Reston, VA, USA

1-563-23

Tracking a Moving Target from a Moving Camera with Rotation-Compensated Imagery

Luiz G B Mirisola and Jorge Dias

Institute of Systems and Robotics - University of Coimbra

Portugal

1 Introduction

In our previous work [Mirisola and Dias, 2007b, Mirisola and Dias, 2008], orientation measurements from an Attitude Heading Reference System (AHRS) compensated the rotational degrees of freedom of the motion of the remotely controlled airship of Fig 1 Firstly, the images were reprojected in a geo-referenced virtual horizontal plane Pure translation models were then used to recover the camera trajectory from images of a horizontal planar area, and they were found to be especially suitable for the estimation of the height component of the trajectory In this paper, the pure translation model with best performance is used to recover the camera trajectory while it images a target independently moving in the ground plane The target trajectory is then recovered and tracked using only the observations made from a moving camera and the AHRS estimated orientation, including the camera and AHRS onboard the airship, as it is shown in Fig 2(b), and results

in a urban people surveillance context with known ground truth To compare our pure translation method with an image-only method, the camera trajectory is also recovered by the usual homography estimation and decomposition method, and the target is also tracked from the corresponding camera poses

GPS also can be utilized to recover the airship trajectory, but GPS position fixes are notoriously less accurate in the altitude than in the latitude and longitude axes, and this uncertainty is very significant for the very low altitude dataset used in this paper Uncertainty in the camera orientation estimate is the most important source of error in tracking of ground objects imaged by an airborne camera [Redding et al., 2006], and its projection in the 2D ground plane is usually anisotropic even if the original distribution is isotropic The Unscented Transform [Julier and Uhlmann, 1997], which has been used to localize static targets on the ground [Merino et al., 2005], is thus used to project the uncertainty on the camera orientation estimate to the 2D ground plane, taking into account its anisotropic projection

Kalman Filters are utilized to filter the recovered trajectories of both camera and the tracked target In the airship scenario, the visual odometry and GPS position fixes can be fused together by the Kalman Filter to recover the airship trajectory The target trajectory is represented, tracked, and filtered in 2D coordinates In this way the full geometry of the camera and target motion is considered and the filters involved may utilize covariances and constants set to the physical limits of the camera and target motion in actual metric units

and coordinate systems This should allow for more accurate tracking than when only pixel coordinates in the images are utilized

Figure 1 An unmanned airship and detailed images of the vision-AHRS system and the GPS receiver mounted onto the gondola

1.1 Experimental Platforms

The hardware used is shown in fig 1 The AHRS used are Xsens MTi [XSens Tech., 2007] for the airship experiment and a Xsens MTB-9 for the people tracking experiment Both AHRS models use a combination of 3-axes accelerometers, gyroscopes and magnetic sensors to output estimates of their own orientation in geo-referenced coordinates They output a rotation matrix WRAHRS|i which registers the AHRS sensor frame with the north-east-up axes The camera is a Point Gray Flea [Point Grey Inc., 2007], which captures images with resolution of 1024 × 768 pixels, at 5 fps The camera is calibrated and its images are corrected

for lens distortion [Bouguet, 2006], its intrinsic parameter matrix K is known, and f is its

focal length To establish pixel correspondences in the images the SURF interest point library is used [Bay et al., 2006]

1.2 Definitions of Reference Frames

The camera provide intensity images I(x, y)| i where x and y are pixel coordinates and i is a

time index Besides the projective camera frame associated with the real camera (CAM) and the coordinate system defined by the measurement axes of the AHRS, the following other reference frames are defined:

• World Frame {W }: A LLA (Latitude Longitude Altitude) frame, where the plane z=0 is

the ground plane It is origin is an arbitrary point

• Virtual Downwards Camera {D }| i: This is a projective camera frame, which has its origin in the centre of projection of the real camera, but its optical axis points down, in the direction of gravity, and its other axes (i.e., the image plane) are aligned with the north and east directions

(a) The virtual horizontal plane concept

(b) Target observations projected in the ground Figure 2 Tracking an independently moving target with observations from a moving

camera

1.3 Camera-AHRS Calibration and a Virtual Horizontal Plane

The camera and AHRS are fixed rigidly together and the constant rotation between both sensor frames AHRSRCAM is found by the Camera-Inertial Calibration Toolkit [Lobo and Dias, 2007]

The translation between both sensors frames is negligible and considered as zero The AHRS estimates of its own orientation are then used to estimate the camera orientation as

WRCAM|i = WRAHRS|i · AHRSRCAM The knowledge of the camera orientation allows the images

to be projected on entities defined on an absolute NED (North East Down) frame, such as a virtual horizontal plane (with normal parallel to gravity), at a distance f below the camera center, as shown in Fig 2(a) Projection rays from 3D points to the camera centre intersect this plane, projecting the 3D point into the plane This projection corresponds to the image

of a virtual camera such as defined in Sect 1.2 It is performed by the infinite homography [Hartley and Zisserman, 2000], which depends on the calculation of the rotation between the real and virtual camera frames: DRCAM|i = DRW · WRCAM|i where the rotation between the

{D }| i and {W } frames is, by definition, given by:

(1)

1.4 Recovering the Camera Trajectory with a Pure Translation Model

Suppose a sequence of aerial images of a horizontal ground patch, and that these images are reprojected on the virtual horizontal plane as presented in section 1.3 Corresponding pixels are detected between each image and the next one in the temporal sequence The virtual cameras have horizontal image planes parallel to the ground plane Then, each corresponding pixel is projected into the ground plane, generating a 3D point, as shown in figure 3(a) Two sets of 3D points are generated for two successive views, and these sets are directly registered in scene coordinates Indeed, as all points belong to the same ground plane, the registration is solved in 2D coordinates Figure 3(b) shows a diagram of this process

(a)

(b) Figure 3 Finding the translation between successive camera poses by 3D scene registration

Each corresponding pixel pair (x, x 0 ) is projected by equation (2) yielding a pair of 3D points

(X,X’), defined in the {D }| i frame:

where x = [x x , x y , 1]T , x’ = [x’ x , x’ y , 1]T, again in inhomogeneous form, h is the camera height

above the ground plane, t is defined as a four element homogenous vector t = [t x , t y , t z , t w]T

The t value which turns X’(t) = X is the translation which registers the {D }| i and {D }| i+1

frames, and which must be determined If there are n corresponding pixel pairs, this

projection yields two sets of 3D points, X = {X k |k = 1 n} and X ‘ = {X’ k |k = 1 n}

An initial, inhomogeneous, value for t0 is calculated by the Procrustes registration routine

[Gower and Dijksterhuis, 2004] It finds the 2D translation and scale factor which register the

two point sets taken as 2D points, yielding estimates the x and y components of t0 and of the

scale factor μ 0 The inputs for the Procrustes routine are the configurations X and X ‘(0)

From μ 0 and the current estimate of the camera height an initial estimate the vertical

component of t0 can be calculated, as μ0 = (h i −t z )/h i Outliers in the pixel correspondences are

removed by embedding the Procrustes routine in a RANSAC procedure Then t0 is used as

an initial estimate for an optimization routine which minimizes the registration error

between X and X ‘(t), estimating an updated and final value for t

The optimization variables are the four elements of t, with equation (2) used to update X ‘(t)

The function to minimize is:

(3)

The same process could be performed with an inhomogeneous, three element t But, as it is

the case with homography estimation, the over-parameterization improves the accuracy of the final estimate and sometimes even the speed of convergence In this case the extra dimension allows the length of the translation to change without changing its direction For datasets where the actual camera orientation is almost constant or the error in the orientation estimate is less significant, the algebraic Procrustes procedure obtains good results alone, with no optimization at all Indeed, if the assumptions of having both image and ground planes parallel and horizontal are really true, with outliers removed, and considering isotropic error in the corresponding pixel coordinates, then it can be proved that the Procrustes solution is the best solution in a least squares sense But the optimization step should improve robustness and resilience to errors, outliers and deviations from the model, and still exploit the available orientation estimate to recover the relative pose more accurately than an image-only method

More details and other pure translation models are shown in [Mirisola and Dias, 2007b, Mirisola and Dias, 2008]

1.5 Filtering the Camera Pose

The camera trajectory is recovered as a sequence of translation vectors t, considered as velocity measurements which are filtered by a Kalman Filter with a Wiener process acceleration model [Bar-Shalom et al., 2001] The filter state contains the camera position, velocity and acceleration The filter should reduce the influence of spurious measurements

and generate a smoother trajectory The process error considers a maximum acceleration increment of 0.35 m/s2, and the sequence of translation vectors is considered as a measurement of the airship velocity, adjusted by the sampling period of 0.2 s The measurement error is considered as a zero mean Gaussian variable with standard deviation

of 4 m/s in the horizontal axes and 1 m/s in the vertical axis The camera pose WXC(i) is taken from the filter state after the filtering

2 Tracking of Moving Targets

Once the camera pose is known, a moving target is selected on each reprojected image Problems such as image segmentation or object detection are out of the scope of this paper Nevertheless, to track its position on the plane, the target coordinates on the virtual image

must be projected on the reference {W } frame, considering the error in the camera position

and orientation Figure 4 summarizes this process which is detailed in this section

Figure 4 A block diagram of the tracking process

2.1 Target Pose Measurement: Projecting from Image to World Frame

The target coordinates in the image are projected into the ground plane by equation (2), and

then these coordinates are transformed into the {W } frame by the appropriate rotation - equation (1) - and translation (the origin of the {D } frame is WxC in the {W} frame)

The actual generation of reprojected images in the virtual horizontal plane does not by itself improve the measurement of the target position on the ground Interest point matching could be performed with the original images, and only the coordinates of the matched interest points need to be actually reprojected on the virtual horizontal plane in order to apply the pure translation method Nevertheless, interest point matching is faster or more robust if the rotation-compensated images are used [Mirisola and Dias, 2007a, Mikolajczyk

et al., 2005], and the reprojected images can be drawn on the ground plane forming a map as

the camera orientation estimate is supposed to have zero mean Gaussian error with standard variation of 5º

Therefore, given a sequence of camera poses with the respective images and an object detected on these images, this projection generates a sequence of 2D coordinates with anisotropic covariance ellipses for the target pose on the ground plane

2.2 Filtering of Target Pose

The target pose is tracked in the 2D reference frame, and filtered by a Kalman Filter similar

to the filter described in section 1.5, although the state position, velocity and acceleration are now 2D The process error considers a maximum acceleration increment of 1 m/s2, and the Unscented Transform supplies measurements of the target position with covariance matrices which are considered as the measurement error

The target observations projected in the ground plane have high frequency noise, due to errors in the camera position and orientation estimate, and in the target detection in each image This is clearly seen in the trajectories of Fig 11 where the ground truth trajectory is a sequence of straight lines These errors are accounted for by the Unscented Transform to estimate a covariance for the target observation, but nevertheless, the original target trajectory is filtered by a low pass filter before the input of the Kalman Filter Analyzing the spectrum of the trajectory of the walking person, most of the energy is concentrated below

1 Hz As the frequencies involved are too small, a low pass filter with too large attenuation

or too small cut frequency would filter out true signal features such as going from zero velocity to motion in the beginning of the movement, and introduce delays in the filtered signal after curves Therefore after empirical testing, the low pass filter parameters were set

to a cut frequency of 2 Hz and attenuation of −10 dB Thus the input of the Kalman Filter is a better conditioned signal, and the final trajectory is smoother

3 Results

3.1 Tracking of a Moving Target from Airship Observations

Firstly, an object of known dimensions in the ground was observed, and the height of the camera estimated from its image dimensions, eliminating the scale ambiguity inherent to relative pose recovery from images alone This was done a few seconds in the image sequence before the images shown Then the airship trajectory was recovered by the model

of Sect 1.4 Only the Procrustes procedure was necessary as the optimization did not improve the results

Figure 5(a) shows the recovered airship trajectories using the method of section 1.4 (red circles) and by the standard homography estimation and decomposition method (green crosses) The blue squares show the GPS measured trajectory In the ground the target (a moving car) trajectory derived from the airship trajectory recovered by our method is shown as blue stars

The trajectories recovered by the Procrustes method are shown again in figure 5(b) The images projected in the ground plane by using equation (2) to find the coordinates of their corners in the ground plane and drawing the image in the canvas accordingly One every three images is drawn

Figure 6 shows a 2D view of the target trajectory on the ground over the corresponding images for the pure translation (a) and image-only (b) methods The error in height

estimation for the image only method is apparent in figure 6(b) as an exaggeration in the

size of the last images The same low pass and Kalman filters were used with both methods

(a)

(b) Figure 5 A 3D view of the recovered trajectories: (a) Airship trajectories from GPS, pure translation and image-only method Target trajectory derived from pure translation airship trajectory (b) Trajectories recovered by the pure translation method, with registered images drawn on the ground plane

(a) Pure translation method

(b) Image-only method Figure 6 Tracking a car from the airship with the pure translation and the image only methods The green circles are the target trajectory with one standard deviation ellipses drawn in red

3.2 Tracking after fusing GPS and Visual Odometry

In this experiment, the car has been driven in a closed loop in the ground while the airship was flying above it To recover the airship trajectory, the translation recovered by the visual odometry was fused with GPS position fixes in a Kalman Filter with a constant acceleration model [Bar-Shalom et al., 2001] The usage of this model does not imply that the actual

acceleration of the vehicle or target is constant; it is just the approximation used by the filter The GPS outputs standard deviation values for its position fixes (shown as the red ellipses and red points in Fig 7), and the translation vectors from the visual odometry are interpreted as a velocity measurement between each pair of successive camera poses, with a manually set covariance smaller in the vertical axis than in the horizontal ones The GPS absolute position fixes keep the estimated airship position from diverging, while the visual odometry measurements improve the trajectory locally The fused airship trajectory is shown as green crosses in figure 7, while the target observations are shown as blue points in the ground plane The target could not be continuously observed, therefore the straight lines (for example the straight lines crossing the path) indicate where observations were missing and resumed at some other point of the path

Figure 7 The airship trajectory from GPS and from the fusion of GPS and visual odometry, with the target observations shown in the ground plane

Figure 8(a) shows the target trajectory drawn over a satellite image of the flight area The airship trajectory was taken directly from GPS Figure 8(b) shows the same target trajectory obtained when the airship trajectory is recovered by a Kalman Filter fusing both visual odometry and GPS In both figures, the squares show the coordinates of the target observations in the ground plane, the circles show the target trajectory filtered by its own Kalman Filter, and the crosses indicate that the target is “lost” The airship can not keep observing the target continuously, thus when there are not observations for an extended period of time the tracked trajectory diverges If the target position standard deviation becomes larger than 30 m than the target is declared “lost” and the filter is reinitialized at the next valid observation Fusing the visual odometry with GPS resulted in a smoother trajectory for the tracked target

3.3 Tracking People with a Moving Surveillance Camera

The method described in Sect 2 was applied to track a person moving on a planar yard, imaged by a highly placed camera which is moved by hand The camera trajectory was

recovered by the Procrustes method with the optimization described by equation (3) which improved the results (the AHRS was the less accurate MTB-9 model) The large squares in the floor provide a ground truth measure, as the person was asked to walk only on the lines between squares The ground truth trajectories of the camera and the target person are highlighted in Fig 9(a), and Fig 9(b) shows the recovered trajectories with the registered images in the top The camera height above the ground was around 8.6 m, and each floor square measures 1.2 m

Figure 8 The target trajectory over a satellite image of the flight area The car followed the dirty roads In (a), the airship trajectory was taken from GPS, in (b) a Kalman Filter

estimated the airship trajectory by fusing GPS and visual odometry

Figure 10 shows the target observations projected in the ground plane before (squares) and after (circles) applying a low pass filter to the data Figure 11(a) shows a closer view of the target trajectory to be compared with Fig 11(b) In the latter case, the camera trajectory was recovered by the homography model The red ellipses are 1 standard deviation ellipses for the covariance of the target position as estimated by the Kalman Filter In both figures, the large covariances in the bottom right of the image appear because the target was out of the camera field of view in a few frames, and therefore its estimated position covariance grew with the Kalman filter prediction stage When the target comes back in the camera field of view the tracking resumed The solid yellow lines are the known ground truth, marked directly over the floor square tiles in the image Comparing the shape of the tracked trajectories is more significant than just the absolute difference to the ground truth, as the image registration itself has errors The tracked trajectory after recovering the camera trajectory with the pure translation model appears more accurate than when the homography model is used The same low pass filter and Kalman Filter were used to filter the target observations in both cases generating the target trajectories shown

(a) (b) Figure 9 A photo with highlighted trajectories of camera and target person (a) A 3D view of the recovered trajectories, using the pure translation method to recover the camera trajectory (b)

4 Conclusions and Future Work

Our previous work on camera trajectory recovery with pure translation models was extended, with the same images being used to recover the moving camera trajectory and to track an independently moving target in the ground plane The better accuracy of the camera trajectory recovery, or of its height component, resulted in better tracking accuracy The filtering steps were performed in the actual metric coordinate frame instead of in pixel space, and the filter parameters could be related to the camera and target motion characteristics

With a low altitude UAV, GPS uncertainty is very significant, particularly as uncertainty in its altitude estimate is projected as uncertainty in the position of the tracked object, therefore recovering the trajectory from visual odometry can reduce the uncertainty of the camera pose, especially in the height component, and thus improve the tracking performance Visual Odometry can also be fused with GPS position fixes in the airship scenario, and the improvements in the recovered airship trajectory translate in a smoother recovered trajectory for the moving target in the ground As the GPS position fixes keep the system from diverging, the tracking can be performed over extended periods of time

In the urban surveillance context these methods could be applied to perform surveillance with a camera carried by a mobile robot, extending the coverage area of a network of static cameras The visual odometry could also be fused with other sensors such as wheel odometry or beacon-based localization systems

Figure 10 A low pass filter is applied to the observed target trajectory before the input of the Kalman Filter

Figure 11 The tracked trajectory of the target person In (a) the camera trajectory was

recovered by the pure translation method, in (b) by the image-only method

5 References

Bar-Shalom, Y., Li, X R., & Kirubarajan, T (2001) Estimation with Applications to Tracking

and Navigation John Willey & Sons, Inc

Bay, H., Tuytelaars, T., & van Gool, L (2006) SURF: Speeded Up Robust Features In the

Ninth European Conference on Computer Vision, Graz, Austria

Bouguet, J (2006) Camera Calibration Toolbox for Matlab

http://www.vision.caltech.edu/bouguetj/calib_doc/index.html

Gower, J C & Dijksterhuis, G B (2004) Procrustes Problems Oxford Statistical Science Series

Oxford University Press

Hartley, R & Zisserman, A (2000) Multiple View Geometry in Computer Vision Cambridge

University Press, Cambridge, UK

Julier, S J & Uhlmann, J K (1997) A new extension of the kalman filter to nonlinear

systems In International Symposium in Aerospace/Defense Sensing, Simul and Controls,

Orlando, FL, USA

Lobo, J & Dias, J (2007) Relative pose calibration between visual and inertial sensors

International Journal of Robotics Research, 26(6): 561-575

Merino, L., Caballero, F., de Dios, J., & Ollero, A (2005) Cooperative fire detection using

unmanned aerial vehicles In IEEE International Conference on Robotics and Automation, pages 1896-1901, Barcelona, Spain

Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schafialitzky, F., Kadir,

T., & van Gool, L (2005) A comparison of affine region detectors International Journal of Computer Vision, 65(7):43 - 72

Mirisola, L G B & Dias, J (2007a) Exploiting inertial sensing in mosaicing and visual

navigation In 6th IFAC Symposium on Intelligent Autonomous Vehicles, Toulouse,

France

Mirisola, L G B & Dias, J (2007b) Trajectory recovery and 3d mapping from rotation

compensated imagery for an airship In IEEE Int Conf on Robots and Systems (IROS07), San Diego, CA, USA

Mirisola, L G B & Dias, J (2008) Exploting inertial sensing in vision based navigation with

an airship Journal of Field Robotics (submitted for publication)

Point Grey Inc (2007) www.ptgrey.com

Redding, J., McLain, T., Beard, R., & Taylor, C (2006) Vision-based target localization from

a fixed-wing miniature air vehicle American Control Conference, Minneapolis, MN,

USA

XSens Tech (2007) www.xsens.com

24

An Open Architecture for the Integration

of UAV Civil Applications

E Pastor, C Barrado, P Royo, J Lopez and E Santamaria

Dept Computer Architecture, Technical University of Catalonia (UPC)

Spain

1 Introduction

The current Unmanned Aerial Vehicles (UAVs) technology offers feasible technical solutions for airframes, flight control, communications, and base stations In addition, the evolution of technology is miniaturizing most sensors used in airborne applications Hence, sensors like weather radars, SAR, multi spectrum line-scan devices, etc in addition to visual and thermal cameras are being used as payload on board UAVs As a result UAVs are slowly becoming efficient platforms that can be applied in scientific/commercial remote sensing applications (see Fig 1 for the most common subsystems in an UAV)

UAVs may offer interesting benefits in terms of cost, flexibility, endurance, etc Even remote sensing in dangerous situations due to extreme climatic conditions (wind, cold, heat) are now seen as possible because the human factor on board the airborne platform is no longer present On the other side, the complexity of developing a full UAV-system tailored for remote sensing is limiting its practical application Currently, only large organizations like NASA or NOAA have enough budget and infrastructure to develop such applications, and eventually may lease flight time to other organizations to conduct their experiments

Even though the rapid evolution of UAV technology on airframes, autopilots, communications and payload, the generalized development of remote sensing applications

is still limited by the absence of systems that support the development of the actual UAV sensing mission Remote sensing engineers face the development of specific systems to control their desired flight-profile, sensor activation/configuration along the flight, data storage and eventually its transmission to the ground control All these elements may delay and increase the risk and cost of the project If realistic remote sensing applications should

be developed, effective system support must be created to offer flexible and adaptable platforms for any application that is susceptible to use them

In order to successfully accomplish this challenge, developers need to pay special attention

to three different concepts: the flight-plan, the payload and the mission itself The actual flight-plan of the UAV should be easy to define and flexible enough to adapt to the necessities of the mission The payload of the UAV should be selected and controlled adequately And finally, the mission should manage the different parts of the UAV application with little human interaction but with large information feedback Many research projects are focused on only one particular application, with specific flight patterns

and fixed payload These systems present several limitations for their extension to new missions

This research introduces a flexible and reusable hardware/software architecture designed to facilitate the development of UAV-based remote sensing applications Over a set of embedded microprocessors (both in the UAV and the ground control station) we build a distributed embedded system connected by a local area network Applications are developed following a service/subscription based software architecture Each computation module may support multiple applications Each application can create and subscribe to available services Services can be discovered and consumed in a dynamic way like web services in the Internet domain Applications can interchange information transparently from network topology, application implementation and actual data payload

This flexibility is organized into a user-parameterizable UAV Service Abstraction Layer (USAL) The USAL defines a collection of standard services and their interrelations as a

basic starting point for further development by users Functionalities like enhanced plans, a mission control engine, data storage, communications management, etc are offered Additional services can be included according to requirements, but all existing services and inter-service communication infrastructure can be exploited and tailored to specific needs This approach reduces development times and risks, but at the same time gives the user higher levels of flexibility and permits the development of more ambitious applications

flight-Figure 1 Common elements in a civil UAV system

This chapter is organized as follows Section 2 generally describes the underlying service oriented technologies that will be applied to create the USAL Section 3 overviews the architecture of the USAL and describes the most relevant services that are included in the

USAL to facilitate the development of UAV applications Section 4 details the Virtual Autopilot System (VAS) that permits the USAL architecture to abstract from autopilot details

Section 5 describes the Flight Plan Manager service that together with its RNAV based dynamic flight-plans constitute the core of the navigation capabilities inside the USAL Finally, Section 6 concludes the chapter and outlines future research and development directions

2 System Overview

This section describes the architecture we propose for executing UAV civil missions: a distributed embedded system that will be on board the aircraft and that will operate as a payload/mission controller Over the different distributed elements of the system we will deploy software components, called services, which will implement the required functionalities These services cooperate for the accomplishment of the UAV mission They rely on a middleware (Lopez et al 2007) that manages and communicates the services The communication primitives provided by the middleware promote a publish/subscribe model for sending and receiving data, announcing events and executing commands among services

2.1 Distributed Embedded Architecture

The proposed system is built as a set of embedded microprocessors, connected by a Local Area Network (LAN), in a purely distributed and scalable architecture This approach is a simple scheme which offers a number of benefits in our application domain that motivates its selection

Development simplicity is the main advantage of this architecture Inspired in the Internet applications and protocols, the computational requirements can be organized as services that are offered to all possible clients connected to the network

Extreme flexibility is given by the high level of modularity of a LAN architecture We are free to select the actual type of processor to be used in each LAN module Different processors can be used according to functional requirements, and they can be scaled according to computational needs of the application We denominate node to a LAN module with processing capabilities

Node interconnection is an additional extra benefit in contrast with the complex interconnection schemes needed by end-to-end parallel buses While buses have to be carefully allocated to fit with the space and weight limitations in a mini/micro UAV, the addition of new nodes can be hot plugged to the LAN with little effort The system can use wake-on-LAN capabilities to switch on/off a node when required, at specific points of the mission development

2.2 Service Oriented Approach

Service Oriented Architecture (SOA) is getting common in several domains For example, Web

Services in the Internet world (W3C, 2004), and Universal Plug and Play (UPnP, 2008) in the home automation area The main goal of SOA is to achieve loose coupling among interacting components We name the distributed components services A service is a unit of work, implemented and offered by a service provider, to achieve desired final results for a service consumer Both provider and consumer are roles played by software agents on behalf of their owners

The benefits of this architecture are the increment of interoperability, flexibility and extensibility of the designed system and of their individual services In the implementation

of a system we want to be able to reuse existing services SOA facilitates the services reuse, while trying to minimize their dependencies by using loosely coupled services

Loose coupling among interacting services is achieved by employing two architectural constraints First, services shall define a small set of simple and ubiquitous interfaces,

available to all other participant services, with generic semantics encoded in them Second, each interface shall send, on request, descriptive messages explaining its operation and its capabilities These messages define the structure and semantics provided by the services The SOA constraints are inspired significantly by object oriented programming, which strongly suggests that you should bind data and its processing together

In a network centric architecture like SOA, when a service needs some external functionality, it asks the network for the required service If the system knows of another service which has this capability, its reference is provided to the requester Thus the former service can act as a client and consume that functionality using the common interface of the provider service The result of a service interface invocation is usually the change of state for the consumer but it can also result on the change of state of the provider or of both services The interface of a SOA service must be simple and clear enough to be easy to implement in different platforms, both hardware and software The development of services and specially their communication requires a running base software layer known as middleware

• Service management: The middleware is responsible of starting and stopping all the

services It also monitors their correct operation and notifies the rest of system about changes in service behaviour

• Resource management: The middleware also centralizes the management of the shared

resources of each computational node such as memory, processors, input/output devices, etc

• Name management: The services are addressed by name, and the middleware discovers

the real location in the network of the named service This feature abstracts the service programmer from knowing where the service resides

• Communication management: The services do not access the network directly All their

communication is carried by the middleware The middleware abstracts the network access, allowing the services to be deployed in different nodes

Fig 2 shows the proposed UAV distributed architecture Services, like the Video Camera or the Storage Module, are independent components executing on a same node located on the aircraft Also on board, there is another node where the Mission Control service executes Both nodes are boards plugged to the LAN of the aircraft The mission has also some services executing on ground The figure also shows two of them: the Ground Station service and a redundant Storage Module

Each node of the UAV distributed architecture executes also a copy of the Service Container software (see Fig 2) The set of all Service Containers compose the middleware which provides the four functionalities described above to the application services This includes acting as the communication bridge between the aircraft and the ground The middleware monitors the different communication links and chooses the best link to send information to ground or to air From the service point of view the middleware builds a global LAN network that connects the LAN on ground and the LAN on board

Figure 2 Overview of the architecture implementing the underlying middleware

2.4 Communication Primitives

For the specific characteristics of a UAV mission, which may have lots of services interacting many-to-many, we propose four communication primitives based in the Data Distribution Services paradigm It promotes a publish/subscribe model for sending and receiving data, events and commands among the services Services that are producing valuable data publish that information while other services may subscribe them The middleware takes care of delivering the information to all subscribers that declare an interest in that topic Many frameworks have been already developed using this paradigm, each one contributing with new primitives for such open communication scenario In our proposal we implement only a minimalistic distributed communication system in order to keep it simple and soft real-time compliant Next, we describe the proposed communication primitives, which have

been named as Variables, Events, Remote Invocations and File Transmissions

Variables are the transmission of structured, and generally short, information from a service

to one or more services of the distributed system A Variable may be sent at regular intervals

or each time a substantial change in its value occurs The relative expiry of the Variable information allows to send it in a best-effort way The system should be able to tolerate the lost of one or more of these data transmissions The Variable communication primitive follows the publication subscription paradigm

Events also follow the publication-subscription paradigm The main difference in front of

Variables is that Events guarantee the reception of the sent information to all the subscribed services The utility of Events is to inform of punctual and important facts to all the services that care about Some examples can be error alarms or warnings, indication of arrival at specific points of the mission, etc

Remote Invocation is an intuitive way to model one-to-one interactions between services

Some examples can be the activation and deactivation of actuators, or calling a service for