A study on radar signal processing and object segmentation for drone system applications

A Study on Radar Signal Processing and Object Segmentation for Drone System Applications NGUYEN, Huy Toan Department of Electronics and Computer Engineering Graduate School of Chonnam

TABLE OF CONTENTS

Contents i

LIST OF FIGURES iv

LIST OF TABLE vii

GLOSSARY viii

Abstract xi

Chapter 1 INTRODUCTION 13

1 Drone system overview 13

1.1 Drone system hardware configuration 14

1.2 Drone system architecture 15

2 Drone applications in this study 16

3 Objectives of the study 17

4 Contribution of the thesis 18

5 Outline 18

Chapter 2 IMPULSE RADAR SIGNAL PROCESSING 20

1 Motivations 20

2 The proposed radar system 20

2.1 Hardware configuration 20

2.2 Software algorithms 21

3 Experimental setup 25

4 Experimental results 26

4.1 Distance estimation result 26

4.2 Distance maintenance result 27

5 Conclusion 28

Chapter 3 FMCW RADAR SIGNAL PROCESSING 29

1 Motivation and Related Works 29

2 Data Collection Method 31

3 Methodology 34

3.1 Preprocessing Data 34

3.2 Background Modeling based on Robust PCA 35

3.3 Moving Objects Localization 39

4 Experimental setup 40

5 Experimental results 42

5.1 Performance across different approaches 42

5.2 Performance across different updating methods 48

5.3 Impact of the sliding window size 49

5.4 Impact of the number of iteration 50

6 Conclusion 51

Chapter 4 OBJECT SEGMENTATION BASED ON DEEP LEARNING 52

1 Motivation and Related Works 52

1.1 Motivation 52

1.2 Related works 54

2 Proposed method 59

2.1 Data preprocessing 60

2.2 The Proposed Network Architecture 61

2.2.1 Modified U-net network 64

2.2.2 High-level feature network 64

2.3 Training process 65

2.4 Data post processing 66

3 Experiment and results 67

3.1 Datasets 67

3.2 Experimental setup 68

3.3 Experimental results on CDF dataset 69

3.4 Experimental results on AigleRN dataset 71

3.5 Experimental results on cross dataset 75

4 Conclusion 77

Chapter 5 DRONE SYSTEM APPLICATIONS 79

1 Wind turbine inspection using drone system 79

1.1 Motivation and related works 79

1.2 Experimental setup and data record method 81

1.3 Experimental results 82

1.4 Conclusion 85

2 Plant growth stage recognition using drone system 86

2.1 Motivation and related works 86

2.2 Method 88

2.3 Experiments 90

2.4 Conclusion 93

Chapter 6 CONCLUSION AND FUTURE WORKS 94

1 Conclusion 94

2 Future works 95

References 96

Acknowledgments 105

(국문초록) 106

LIST OF FIGURES

Figure 1.1 The drone system applications (a) Monitoring applications; (b) Firefighting

application; (c) Rescue application, (d) Agriculture application 13

Figure 1.2 The prototype of drone system (a) Using Digital camera and IR-UWB radar, (b) Using RPi Camera and FMCW radar 14

Figure 1.3 The proposed system architecture 16

Figure 1.4 Drone system applications (a) Wind turbine inspection, (b) Plant growth stage recognition 17

Figure 2.1 Radar module hardware configuration 21

Figure 2.2 Radar module prototype 21

Figure 2.3 Distance measurement algorithm flow chart 22

Figure 2.4 Radar data normalization result 23

Figure 2.5 Shape of logarithm function 23

Figure 2.6 Smooth calibration function using Polynomial regression 24

Figure 2.7 Testing of IR-UWB radar sensor 26

Figure 2.8 Reference distance and computed output 26

Figure 2.9 Distance maintenance results 27

Figure 3.1 120 GHz Radar front end block diagram [19] 32

Figure 3.2 FMCW Radar sensor connection (a) Real connection, (b) Specific connection diagram 32

Figure 3.3 Raw data signal (a) Raw data frame, (b) Raw data matrix in the distance scale 33 Figure 3.4 Calibration experimental setup 33

Figure 3.5 Time-based sliding window 34

Figure 3.6 Block diagram for detecting moving objects 34

Figure 3.7 AMPD algorithm [26] 40

Figure 3.8 Experimental Scenarios (a) Indoor environment; (b) Outdoor environment 42

Figure 3.9 Original data with one moving object 42

Figure 3.10 Detection performance across different methods 43

Figure 3.11.Noise removed signals and target position for one moving object in Figure 3.9 (a) RPCA via IALM [15], (b) RPCA via GD [17], (c) Online RPCA [16], (d) Proposed method 45

Figure 3.12 Target detection results for multiple moving objects (a) Two moving objects, (b) Three moving objects, (c) Four moving objects, (d) Five moving objects (From top to bottom: Original data, RPCA via IALM [15], RPCA via GD [17]) 46

Figure 3.13 Target detection results for multiple moving objects (a) Two moving objects, (b) Three moving objects, (c) Four moving objects, (d) Five moving objects (From top to bottom: Original data, Online RPCA [16] and proposed method results) 47

Figure 3.14 Detection performance across different update methods 48

Figure 3.15 Impact of the sliding window size 50

Figure 3.16 Impact of the number of iteration 50

Figure 4.1 Overview of crack identification 54

Figure 4.2 Illustration of data pre-processing steps (a) Original image, (b) ground truth, (c) grey-scale image, (d) normalized image, (e) histogram equalization image, and (f) pre-processed image 62

Figure 4.3 The schematic architecture of the proposed network 63

Figure 4.4 Crack prediction results by our proposed method (From top to bottom: Original images, Ground truth, Probability map, Binary output) 67 Figure 4.5 Crack prediction results on CFD dataset (From top to bottom: Original image, ground truth, MFCD [46], CNN [56] and our results 70 Figure 4.6 Results on AigleRN dataset From left to right: Original images, Ground truth images, FFA, MPS, MFCD, CNN, the proposed method 73 Figure 4.7 Detection results on AigleRN dataset From top to bottom: Original images, Ground truth images, FFA, MPS, MFCD, CNN, and our results 74 Figure 4.8 Detection results on cross data generation (a), (b), (c), (d) Original images and ground truth of CFD dataset and AigleRN dataset, (e) Training / Testing: CFD / CFD, (f) Training / Testing: AigleRN / AigleRN, (g) Training / Testing: AigleRN / CFD, and (h) Training / Testing: CFD / AigleRN 77 Figure 5.1 Wind power energy in South Korea [72] 79 Figure 5.2 Proposed Network architecture 81 Figure 5.3 Wind turbine inspection using the drone system (a) Drone system working state, (b) The prototype of drone system 82 Figure 5.4 Illustration of predicting steps (a) Input image, (b) Network threshold output, (c) Contours detection, (d) Final abnormal appearance results 83 Figure 5.5 Real inspection flight on garlic fields 87 Figure 5.6 Scaling garlic size using ruler 89 Figure 5.7 Illustration of image processing to extract the garlic information (a) Garlic contours detection, (b) Final garlic size results 89 Figure 5.8 Example results of plant recognition 92

LIST OF TABLE

Table 2.1 Numerical results for distance maintenance algorithm 27

Table 3.1 Setup parameters 41

Table 3.2 Processing speed across different methods 44

Table 4 1 Comparison of different methods on the same data set (CFD dataset and AigleRN dataset) 58

Table 4.2 Comparison of major deep learning approaches for crack detection and segmentation 59

Table 4.3 Detection results with five pixels of tolerance margin on CFD dataset 71

Table 4.4 Detection results with two pixels of tolerance margin on CFD dataset 71

Table 4.5 Detection results with five pixels of tolerance margin on AigleRN dataset 75

Table 4.6 Detection results with two pixels of tolerance margin on AigleRN dataset 75

Table 4.7 Detection results on cross data generation with five pixels of tolerance margin 76

Table 4.8 Detection results on cross data generation with two pixels of tolerance margin 76

Table 5.1 Comparison between our results and the original U-net network 84

Table 5.2 Performance comparison 84

Table 5.3 Computational cost 85

Table 5.4 Pixel-wise performace on the test dataset 90

Table 5.5 Object-wise performace on the test dataset 91

GLOSSARY

AEE Average Euclidean Error

AMPD Automatic Multiscale-based Peak Detection

CFAR Constant False Alarm Rate

CFD Crack Forest Dataset

CLAHE Contrast Limited Adaptive Histogram Equalization

CNNs Convolutional Neural Networks

CPU Central Processing Unit

DCNN Deep Convolutional Neural Networks

FCN Fully Convolutional Network

FFT Fast Fourier Transform

FMCW Frequency-Modulated Continuous-Wave

GMM Gaussian Mixture Model

GPS Global Positioning System

GUI Graphical User Interface

IALM Inexact Augmented Lagrange Multipliers

IoT Internet of Things

IR-UWB Impulse Radio – Ultra Wideband

ISM Industry-Science-Medical

LBP Local Binary Pattern

LMS Local Maxima Scalogram

MCU Micro Controller Unit

MFCD Multiple-scale Fusion Crack Detection

MPS Minimal Path Selection

OR-PCA Online Robust Principal Component Analysis

PCA Principal Component Analysis

PGM-SVM Probabilistic Generative Model - Support Vector Machine

PI Proportional Integral

ReLU Rectified Linear Unit

RMSE Root Mean Square Error

RNN Recurrent Neural Network

RPCA Robust Principal Component Analysis

RPCA-GD RPCA via Gradient Descents

SGD Stochastic Gradient Descent

SNR Signal to Noise Ratio

SPI Serial Peripheral Interface

SSD Single Shot Detector

SVD Singular Value Decomposition

SVM Support Vector Machine

TDOA Time Difference Of Arrival

UART Universal Asynchronous Receiver / Transmitter

UAV Unmanned Aerial Vehicle

VCO Voltage-Controlled Oscillators

A Study on Radar Signal Processing and Object

Segmentation for Drone System Applications

NGUYEN, Huy Toan

Department of Electronics and Computer Engineering Graduate School of Chonnam National University (Supervised by Professor KIM, Jin Young)

Abstract

Drone system have been used in variety fields in last decades such as surveying, inspection, mapping, security, agriculture, mining, search & rescue In order to qualify as a reliable system, a drone must be adapted and integrated with sensors and software system However, processing acquired signals from drone system is still complex and difficult problems due to the higher of noise, uncertainty, and computation cost

This thesis investigates the sensor signal processing algorithms and object segmentation method for drone system applications We consider two kinds of radar sensor including Impulse Radio – Ultra Wideband (IR-UWB) radar sensor and Frequency-Modulated Continuous-Wave (FMCW) radar sensor for estimating the distance from the drone to obstacle

to avoid collisions during flight In addition, on drone, the equipped camera capture images and send them to ground station At the ground station, the object segmentation process is adopted to segment out the fruitful information from input images

A new impulse radar with hardware configuration and software algorithm is proposed in this study The impulse radar sensor hardware must be lightweight, low power consumption and easy to use We propose a real-time radar signal-processing algorithm based on logarithm compensation method and filters on original input data The propose impulse radar is able to

work at real-time speed with high accuracy

The thesis also proposes a new algorithm for FMCW radar signal processing based on Robust Principal Component Analysis (RPCA) for moving-targets detection The compensation and calibration are first applied, based on experiment, to the input signal Then, RPCA via Gradient Descents (RPCA-GD) is adopted to model the low-rank noisy background

A new update method for RPCA is proposed to decrease the processing time Objects moving

in the foreground are localized using an Automatic Multiscale-based Peak Detection (AMPD) method All processing steps are based on a sliding window approach, and the proposed scheme shows impressive results in both processing time and accuracy compared to other RPCA-based approaches when using real signals in various experimental scenarios

Moreover, in this study, we address the problem of object semantic segmentation as pavement crack detection and segmentation in pixel-level based on Deep Neural Network (DNN) using gray-scale images We propose a novel DNN architecture, which contains a modified U-net network and a high-level features network A further important contribution is the combination of these networks afforded through the fusion layer This combination is surprisingly able to boost the system performance We implement and thoroughly evaluate our proposed system on two open datasets: The Crack Forest Dataset (CFD) and the AigleRN dataset Experimental results show that our system outperforms eight state-of-the-art methods

on two open datasets

Finally, to confirm the effectiveness of ours propose algorithm on radar signal and image, our drone system is applied for inspection of the wind turbine at wind turbine farm and monitoring the growth stage of plant in agriculture field The experiments are conducted at YongGwang wind turbine farm, Gwangju Institute of Science and Technology and Chonnam National University The experimental results show that our system is safe, achieve high accuracy with real-time speed

Chapter 1 INTRODUCTION

1 Drone system overview

Over last decade, unmanned aerial vehicles (UAV) have been widely adopted and achieved many successes in military applications such as anti-aircraft target practice, intelligence gathering and then, more controversially, as weapons platforms [1] From technology point of view, a drone is an UAV, which can be considered as a flying robot Basically, the drone can be remotely controlled or autonomously controlled through software controlled flight plans in their embedded system working with various sensors Recently, drone

is also widely used for non-military applications such as search and rescue, disaster response, wildlife monitoring, firefighting, agriculture, and healthcare as shown in Figure 1.1 In this research, we study the radar signal processing and object segmentation using drone system The following sections briefly introduce our drone system hardware configuration and system architecture, and applications The objectives and main contributions of this thesis are also considered Finally, the outline of this thesis is provided in the last section of this chapter

Figure 1.1 The drone system applications (a) Monitoring applications; (b) Firefighting application; (c) Rescue application; (d) Agriculture application

1.1 Drone system hardware configuration

The drone system prototype used in this study is designed and assembled to effectively acquired image and avoid collision with other obstacles The prototype is developed based on Tarot T960 Foldable Hexacopter because it is a great platform to carry over 2kg payload and able to fly over 20 minutes with only one battery package For our monitoring task, the drone system is equipped with four essential components: a camera (Digital camera or Raspberry Pi camera), a radar sensor (IR-UWB radar or FMCW radar), an MCU flight controller, and a LTE Wifi module as shown in Figure 1.2

The radar sensor is performed measuring and providing distance data to MCU to maintain distance and avoid collision between drone and obstacles The radar sensor is combined with Raspberry Pi (RPi) for signal processing

The main controller of the drone system is Micro Controller Unit (MCU) working as the human brain All algorithms such as path planning, locating by GPS, controlling motors, avoiding obstacles, maintaining distance are performed by MCU

The base station includes a manual remote controller and a computer to monitor the flying and manually operate the UAV if necessary On the other hand, received images from the drone are processed in the base station computer to give fruitful information to users

1.2 Drone system architecture

The architecture of our drone system is depicted in Figure 1.3 On the drone, GPS sensor and compass sensor signals are sent to MCU to autonomous locate, navigate After that, these signals are transmitted to ground station via telemetry for flight monitor and manual remote if necessary The distance between the drone and other obstacles are measured by radar sensor based on distance measurement algorithm on RPi, this measured data is treated as the input of the Proportional-Integral (PI) controller The output of PI controller is the control signal to the propeller via electronic speed controllers (ESCs) The camera is used for capturing images and sending them to the ground station via the LTE system

At ground station, the telemetry is used to receive localization and navigation signals for mission planner monitoring The mission planner monitoring software is provided by the manufacturer Moreover, the captured images are received via LTE router A trained deep convolutional neural networks (CNNs) model with a non-overlapping sliding window is applied to extract the favorable information from input images

Figure 1.3 The proposed system architecture

2 Drone applications in this study

As aforementioned, the use of drones outside the military has grown enormously over the past decade The integration of drone and internet of things (IoT) has created numerous applications In this study, we investigate our proposed algorithms on wind turbine (WT) inspection application and plant growth state monitoring application as shown in Figure 1.4 Wind energy is increasing and becoming one of the critical renewable energy resources all over the world With these developments, WT maintenance is a crucial task with potential challenges and time consuming Due to that, we propose a drone system with equipped camera and radar sensor to detect precise sematic abnormal appearances on wind turbine surface

In agriculture applications, the monitoring of the crop growth stage in all steps starting from seedlings, planting, watering, disease controls, and harvesting is critical task The reliable information of crops enables the farmer to carry out timely interventions to increase global crop yields Agricultural researchers conventionally make the observation by manual; however, this traditional method is intensive, time-consuming, costly and difficult to expand to a large field In this study, a drone system based on convolutional neural network is proposed to perform the aforementioned goal

(a) (b) Figure 1.4 Drone system applications (a) Wind turbine inspection, (b) Plant growth stage recognition

3 Objectives of the study

In this study, we aim to develop and implement radar signal processing algorithms and the object segmentation algorithm for drone system There are two main requirements for radar system: the hardware must be small size and lightweight, the software algorithm must be reliable and worked with real-time speed The output of the radar system is the distance between the radar and obstacles We develop our algorithms on both two kind of radar contains IR-UWB radar and FMCW radar We test our radar system in practical environments We achieve very promising results and better performance in comparison with other methods For object segmentation task, we propose a new deep convolution neural network architecture We train and test our network on two open datasets namely CFD dataset and AigleRN dataset for segment cracks from the input image in pixel level The experimental results show that our method is out perform eight state-of-the-art methods for cracks semantic segmentation on the same dataset

Based on these above achievements of radar processing and image processing, we apply these algorithms on two main applications: Wind turbine inspection and recognition of plant’s growth stage

4 Contribution of the thesis

In this study, we propose to develop the distance estimation system using IR-UWB radar, moving object detection system using FMCW radar and object segmentation system based on Deep Convolutional Neural Network

This thesis makes the following contributions:

- We introduce the new impulse radar sensor with novel hardware configuration

- We investigate the new software algorithm to estimating distance between radar and obstacle based on filters and logarithm compensation method

- We propose a critical updating method to enhance the performance and processing speed on FMCW radar system based on RPCA via GD

- We conduct the new deep neural network structure for object segmentation problems

on crack dataset

- We evaluate the drone system on two main applications namely wind turbine inspection and plant growth stage using drone system to confirm the effectiveness of proposed algorithms on drone system

5 Outline

The remaining of the study is arranged below:

- Chapter 2: Proposes a new IR-UWB radar system in both hardware configuration and software algorithm

- Chapter 3: Introduces a novel FMCW radar signal processing algorithm based on RPCA We propose a critical update method to improve processing speed of RPCA on radar signal

- Chapter 4 Presents an objective segmentation method based on deep learning We introduce a new deep learning network structure for object segmentation problems on cracks dataset The experimental results show that our proposed network outperforms eight state-of-the-art algorithms on open dataset

- Chapter 5 Shows the drone system applications on wind turbine inspection and plant growth stage recognition problems

- Chapter 6: Draws conclusions and presents some future extension applications of our proposed drone system and algorithms

Chapter 2 IMPULSE RADAR SIGNAL PROCESSING

1 Motivations

Ultra-Wideband (UWB) technology has become a very popular topic in industrial and academia.UWB signals have an excellent spatial resolution and good penetration into many materials, which makes them very interesting for radar applications The main goal of our research is to design and implement an IR-UWB sensor system to avoid crashes cause by drone while flying Therefore, design a lightweight safety sensor for drone system is the critical requirement of radar sensor A crash-avoidance system that make human interaction with the drone safer is proposed Whenever a person came near it, the drone flies back away, avoiding contact with the human

Distance measurement is the core duty of radar signal processing IR-UWB can be used for positioning by utilizing the time difference of arrival (TDOA) of the RF signals to obtain the distance between the reference point and the target IR-UWB signals provide accurate position and location estimation In practical, IR-UWB is based on transmitting extremely short pulses and uses techniques that cause a spreading of the radio energy (over a wide frequency band) with a very low power spectral density The IR-UWB radar hardware is proposed to comfortably attach on vehicles and flying object such as drone with small size, lightweight, low power consumption

2 The proposed radar system

2.1 Hardware configuration

We attempt to introduce a new IR-UWB radar, which has not been explored in the market,

to detect obstacles The prototype of the IR-UWB radar system is small enough to be tested

by a drone and it is able to detect in front objects and return the precise distance in real time

We combine our radar with famous Raspberry Pi module for signal processing The receive signal from radar is the input signal of the Raspberry Pi module The processing algorithm is executed in RPi module The final distance result is the transfer to the UAVs hardware via

UART port The UART connection enables for various UAVs system The critical challenge

is to extract motion features across a wideband signal that has low Signal to Noise Ratio (SNR)

in common sense Our product is a friendly front-end easy usage device in both hardware and software design The radar module hardware configuration is illustrated in Figure 2.1

Figure 2.1 Radar module hardware configuration

The radar module includes one transmit antenna (TX) and one receive antenna (RX) The frequency bands for TX and RX antennas is 7.2 to 10.2 GHz The antenna signal is transferred

to the Central Processing Unit (CPU), using Raspberry Pi, via Serial Peripheral Interface (SPI) interface The system is able to work with the distance up to 15 meters The output of the system is transmitted to MCU by UART port on Raspberry Pi The prototype of our IR-UWB sensor system is depicted in Figure 2.2

Figure 2.2 Radar module prototype

2.2 Software algorithms

2.2.1 Distance measurement algorithm

Our idea approaches to predict the distance based on the difference between two successive recorded frames In general, a strong variation indicates for the moving target We propose a novel IR-UWB signal-processing algorithm based on various filters and logarithm compensation to accurately measure the distance between radar and obstacles The flow chart

of distance estimation algorithm is illustrated in Figure 2.3

Figure 2.3 Distance measurement algorithm flow chart

According to Figure 2.3, a list of different signal processing techniques is applied here The radar is set up to run across a long area to collect pure data in order to observe the attenuation of signal has the symmetric form First is the band pass filter where only a certain frequency range of raw signal is allowed to be passed through Even that, the output signal is still very weak due to low emission power To be convenient, we transform into a one-side wave by subtracting to the mean value and taking the absolute

Let assume the raw data at sample n is denoted as 𝑟(𝑛) then its corresponding normalization 𝑟(𝑛)̅̅̅̅̅̅ is defined as the following equation

𝑟(𝑛)

̅̅̅̅̅̅ = 𝑟(𝑛) −1

𝑁∑𝑁𝑛=1𝑟(𝑛) (2 1) where N is the total of samples at one specific time given by the stick value For example, one meter is assigned to 256 samples and in the case of our radar sensor that can detect up to

15 meters; the value of N is 3840 samples The normalization output is depicted in Figure 2.4

Figure 2.4 Radar data normalization result

After that, the logarithm regression model is taken to measure the shape of the logarithm function across the frame Mathematically, we try to derive the parameters 𝑎 and 𝑏 of the following equation

𝑟(𝑛)

̅̅̅̅̅̅ = 𝑎𝑙𝑜𝑔(𝑛) + 𝑏 (2 2) Regressing function and data can be visualized in the Figure 2.5

Figure 2.5 Shape of logarithm function

For each block condition, we compensate the data by subtracting to its mean value and then the compensation value represented by the variable Again the high pass and low pass filter is applied the second time to stabilize data To compensate that we propose the Logarithm compensation weight by empirical approach We sort the distance and pick the largest value

index to be our candidate We apply calibration method in compensation data to get the final distance result The calibration step is implement by experiment We get the step calibration function Finally, to increase the system accuracy, after calibration method, polynomial regression method is applied to smooth the calibration function as in Figure 2.6 These algorithms must be compatible with a use in real-time, which imposes operational limits of resource memory and in computing times for this reason that we limit this study to the 2th order polynomial

Figure 2.6 Smooth calibration function using Polynomial regression

Finally, the threshold α is adopted to identify the maximum value is from moving objects

or not When the object is identified, the distance between the radar to the object is extracted The output value is sent to other modules via the UART interface

2.2.2 Distance maintenance algorithm

The goal of this section is to develop an algorithm to ensure the safe distance between the drone and other obstacles to avoid the collision By using the open source code provided from the manufacturer, we modify the sour code to apply the PI controller to maintenance distance between drone and wind turbine The PI controller requires minimal resources, just the error signal, one memory cell, and a few mathematical operations lead to faster response

to MCU for controlling propellers

Basically, the PI controller is used for non-integrating processes The output value is fed

into the system as the manipulated input value We set the target value output called set point

(SP) The process value (PV) is able to deviate from the desired value The error between SP and PV is defined as an error by Equation 2.3.

𝑒(𝑡) = 𝑆𝑃 − 𝑃𝑉 (2 3) The PI controller is modeled by the following equation

𝑢(𝑡) = 𝑢𝑏𝑖𝑎𝑠+ 𝐾𝑐𝑒(𝑡) +𝐾𝑐

𝜏𝐼∫ 𝑒(𝑡)𝑑𝑡0𝑡 (2 4) When the controller is switched from manual to automatic mode in the first time, the

𝑢𝑏𝑖𝑎𝑠 term is set to the value of 𝑢(𝑡) There are two turning values of a PI controller are controller gain 𝐾𝑐 and the integral time constant 𝜏𝐼 The 𝜏𝐼 constant is in denominator, so the smaller values provide a larger weighting to the integral term Generally, the 𝜏𝐼 constant

is a positive value

3 Experimental setup

To validate the performance of our proposed methods including the distance estimation and distance maintenance algorithms, we first test our IR-UWB radar sensor processing system To do that, the system is set up in the football field with a computer for testing by manual remote control, and drone system flying The program is written in C++ programming language The experimental setting is illustrated in Figure 2.7 A metal reflector is used to evaluate the distance measurement performance The distance reference value, which is the real distance between the radar sensor and the reflector, is measured using a metric scale ruler The output of the measuring system is saved for evaluation To avoid noise, we set the maximum measuring range is 12 meters

Figure 2.7 Testing of IR-UWB radar sensor

4 Experimental results

4.1 Distance estimation result

The same experiment is executed over 20 times and taken the average for computing measurement error The results are shown in Figure 2.8, where the red line is the real value

and the measured value by our proposed method The coefficient of determination R2 = 0.98

that mean the estimated values and the reference values match well

Figure 2.8 Reference distance and computed output

4.2 Distance maintenance result

To confirm the capability of distance maintenance algorithm, we set the threshold distance between the drone system and the obstacle is 2 meters In the beginning, the drone is manually controlled flying closer to the obstacle The system automatically keeps the distance

to avoid the collision Even when we manually move the drone forward to objects, our algorithm is able to keep a safe distance between them The testing results are presented in Figure 2.9 Root Mean Square Error (RMSE) and Average Euclidean Error (AEE) metrics are selected to measure the performance of distance maintenance results The numerical results are showed in Table 2.1

Table 2.1 Numerical results for distance maintenance algorithm

Figure 2.9 Distance maintenance results

5 Conclusion

In this chapter, a radar sensor based on the new Ultra Wide-band technology is presented This radar will be equipped on the drone system in order to increase flight safety Consequently, the development of a small-size, micro-power, fine rage resolution and low probability of interference is presented The proposed IR-UWB radar show good precision in distance calculation In addition, we also examine the collision avoidance algorithm based on PI controller The PI algorithm is suitable for distance maintenance algorithm to avoid collision with other obstacles According to these achievements, the IR-UWB is an excellent candidate technology for drone system for many applications of obstacle detection and target identification in the short range

Chapter 3 FMCW RADAR SIGNAL PROCESSING

1 Motivation and Related Works

Moving-target is a common and critical task for security systems The most popular solution is to use a camera to gather video and extract useful information to determine the

presence of moving targets in video frames [2, 3] However, this method is sensitive to

variations in illumination and weather Conversely, radar systems can function independently

of these effects, producing highly accurate results In general, radar systems are divided into two categories: impulse radar and continuous wave radar In impulse radar system, only one common antenna for transmission and reception is used The pulsed waveform transmitter emits a sequence of finite duration pulse This pulsed sequence is separated by time When the transmitter is on, the receiver is off and conversely so that the target signals can be detected However, impulse radar requires comparative higher transmitting power On the other hand,

in continuous wave radar system, separate antennas are used for transmitting and receiving signals The transmitter is continually transmitting a signal without interrupting in all operating time The receiver continuously receives reflected signals from targets Continuous wave radar systems are generally low power consumption and suitable for short-range applications When estimating the distance and velocity of moving targets, Frequency Modulated

Continuous Wave (FMCW) radar is considered the choice [4] Range information can be

extracted via demodulation of the frequency signal, and the Doppler Effect of an FMCW radar system can be used to compute the velocity of the moving targets

However, moving target detection is a serious challenge, especially in complex environments The detection results are sensitive to noises or clutters resulting from small moving objects in radar scan areas or from multipath problems Moreover, strong reflection signals from large static objects are also a type of noise affecting moving target detection These issues may significantly degrade the radar’s capability to exact the range and velocity

of moving objects

To mitigate such effects, many algorithms have been carefully developed and implemented in both hardware and software First, a hardware configuration implementing a low-noise Colpitts voltage-controlled oscillators (VCO) with a transformer-based resonator and a 20 to 40GHz frequency double for use in 77–81GHz radar was presented in [5] M-T Dao et al proposed another method that increases the linearity of the sweep frequency based

on two wideband linear RF VCO to eliminate the phase noise of the signal [6] The linearity of the VCO tuning law was complemented by measuring the frequency of the divider signal [7] A delay-locked loop (DLL) was utilized as a frequency multiplier for the reference signal to improve both the phase noise performance of the phase-locked loop (PLL) and the linearity of the frequency sweep generated by the PLL [8] The aforementioned methods could improve the accuracy of the radar parameters, but they require a high cost for the hardware experiments Another way to improve the detection performance is based on a software algorithm to process the received data In [9], the authors proposed using a de-interleaved method in the time domain and a different number of Fast Fourier Transform samples to improve the range and velocity of the detection results Another method based on 2-D FFT with fast-ramp training was proposed in [10]

non-A technique based on a constant false alarm rate (CFnon-AR) algorithm [11] was introduced

to reduce the effect of noise Waveform design was also a good solution to improve the results

of the noise reduction and multiple target detection [12] Another method based on a machine learning technique to model the background and extract the moving foreground using Gaussian Mixture Model (GMM) with FMCW radar was presented in [13]

Various approaches have been explored to solve the moving object detection problems Among them, Robust Principal Component Analysis (RPCA) based low-rank matrix decomposition algorithms showed impressive results for background modeling [14] However, RPCA batch optimization processing are computationally complex and require a large memory footprint In [15], the Inexact Augmented Lagrange Multipliers (IALM) approach was defined and converged to solve the Robust PCA problem with a fast processing time Moreover, Feng

and Xu [16] recently proposed an Online-Robust PCA (OR-PCA) algorithm that processes one sample per time immediately based on stochastic approximations to reduce the processing time

A new method to solve the Robust PCA problem by using Gradient Descent (RPCA-GD) was presented in [17] showed that this method was effective not only in accuracy but also in reducing the computational cost

Inspired by these achievements, this chapter examines the ability of RPCA-GD in modelling the noisy background for a FMCW radar system based on the sliding window method To apply it in real-time for sequence data, we propose a new online updating approach for RPCA-GD The proposed algorithm shows remarkable results in terms of processing time and performance

2 Data Collection Method

Based on FMCW radar theory, the distance from other objects to the radar and the object’s relative velocity can be estimated from the FFT of the signal reflected from the moving objects

by applying the Doppler Effect The FMCW radar in this study is based on a 120GHz [18] 120GHz radar frontend includes 1 TX and 1 RX antenna as shown in Figure 3.1 The main application field of this radar frontend is in short range distance estimation, with the detection range about 10 meters The radar frontend is connected to the SiRad Easy Evaluation Kit from Silicon Radar Company [19, 20] for using in the Industry-Science-Medical (ISM) band 122GHz – 123 GHz

Figure 3.1 120 GHz Radar front end block diagram [19]

The system setup is depicted in Figure 3.2 to gather reflected signals from objects in experimental environments By using SiRad Easy Evaluation Kit, we are able to set several parameters, including the bandwidth, number of FFT points, and number of ramps The output

of this system is a block of data that includes the system information, status information, FFT and CFAR data, target information and error information The magnitude, phase, and other information can be extracted from the FFT output Moreover, the targets are detected by a CFAR operator For each measurement cycle, all data is immediately transferred to the PC via

For each received data block, the block data will be extracted to different data frames based on the start and stop bits In this paper, we only extract the original FFT information for further processing For each cycle, there is one FFT data frame, and this data frame will be stored as a column vector 𝑋𝑡∈ ℝ𝑁×1 with N elements After T cycles, the received data is saved as a rectangular matrix 𝑋 ∈ ℝ𝑁×𝑇, as shown in Figure 3.3

Figure 3.3 Raw data signal (a) Raw data frame, (b) Raw data matrix in the distance scale

All data is recorded and processed using the Matlab programming language for further comparison In order to get the exact distance for each FFT bin, we use the Corner Cube (RCS

= 1m2) and a tape measure to calibrate the input data The calibration process is presented in Figure 3.4

Figure 3.4 Calibration experimental setup

3 Methodology

This section presents the details of our scheme to detect moving foreground objects To apply it with real applications, we use a time-based sliding window approach to process the raw input data, as shown in Figure 3.5 We fix size of the window to M and process the window matrix 𝑌 ∈ ℝ𝑁×𝑀 We first accumulate M raw data vectors as the initialization matrix, and we execute the processing steps on the initial matrix After that, new data frames that come are added to the processing matrix, and the oldest data frame is discarded to process the new data matrix

Figure 3.5 Time-based sliding window

Our methodology consists of three main steps, as illustrated in Figure 3.6 The input data frame is calibrated and compensated based on experiments in pre-processing step After that, RPCA-GD is used to model the background We propose a new method that updates the initial background for RPCA when each new frame has been accumulated Finally, the positions of the moving objects are estimated using AMPD peak detection algorithm

Figure 3.6 Block diagram for detecting moving objects

3.1 Preprocessing Data

In theory, the radar equation showing the relationship between the received signal power

and the transmitted signal power is presented as following

𝑃𝑟

𝑃𝑡 =

𝐺𝑡𝐴𝑒𝜎

Where P r is the received signal power, P t is the transmitted signal power, G t is the gain

of the antenna, A e is the effective aperture of the receiving antenna, σ is the radar cross section,

and R is the range distance from the target to radar

The signal power received from one target will decrease when it moves farther away from the radar, which is called Spherical Spreading Loss [21] However, the above equation is applied to an isotropic radiator which does not exist in the real world To compensate for the actual radar signal, we need to estimate the relationship between the power and the range by the conducting the Corner Cube experiment described in Section 2 in this chapter Based on practical experiments, we select the compensation factor as 𝑙𝑜𝑔10(4𝜋𝑅2) The received signal power vector at each point has a weighted value 𝑙𝑜𝑔10(4𝜋𝑅2) Finally, the band pass filter is applied to reduce the effect of noise from the environment After this step, the processed signal includes only the signals reflected from static objects and moving targets

3.2 Background Modeling based on Robust PCA

There are various approaches to separate the background and moving foreground objects

As we previously mentioned, RPCA is selected to model the background based on its admirable performance The main purpose of having a RPCA method is to decompose the observation matrix 𝑋 ∈ ℝ𝑁×𝑇 into a matrix L with rank r and a sparse component matrix S

by using the following equation

where ‖𝐿‖∗ is the nuclear norm of matrix L, ‖𝑆‖1= ∑ |𝑆𝑖,𝑗 𝑖𝑗|, and 𝜆 is the positive weighting parameter

Various algorithms have been developed to solve the (3 3) problem As mentioned above, the proposed method is based on the Gradient Descent method, which was presented in [17] However, the RPCA algorithms are difficult to apply as online processing because they need

to handle all data whenever new data has been inserted As a result, when the size of the data increases, the computational cost increases, respectively Applying the sliding window approach to investigate the ability of RPCA on a radar signal was presented in [27] According

to that study, as the new frame was captured, RPCA was computed on the window matrix to model the background We realize that their approach is not effective for a time sequence signal as the radar signal because we have to re-compute the low rank components for the entire window matrix

The method proposed in this paper is based on the following observation: as the new frame is inserted, the low-rank component matrix changes a little compared to the previous calculation We re-use the low rank component matrix from the previous process and adapt the last column vector as the initialization matrix for optimization The proposed method is presented in Algorithm 1

First, we collect data for processing until t = M When t = M, we utilize the method in [17] to extract the low-rank component First, S init is generated based on the input window

matrix Y using a sorting-based sparse estimator algorithm called Phase 1 The sorting

algorithm is basically conducted as follows for any matrix 𝐴 ∈ ℝ𝑑1×𝑑2:

After generating S init Single Value Decomposition (SVD) method is applied on (𝑌 − 𝑆𝑖𝑛𝑖𝑡)

to obtain the matrix rank r

[H, Σ, K] = SVD(Y − S𝑖𝑛𝑖𝑡 ), (3.5) Let’s U𝑡← 𝐻Σ1/2, V𝑡← 𝐾Σ1/2 With a given U𝑡, 𝑉𝑡 an iterative method is implemented

to produce the sparse matrix S based on a sparse estimator on (𝑌 − U𝑡𝑉𝑡′) We consider the following optimization problem

𝑈∈𝒰,𝑉∈𝒱,𝑆∈𝒮𝛼ℒ(𝑈, 𝑉, 𝑆) +18‖𝑈𝑇𝑈 − 𝑉𝑇𝑉‖𝐹2, (3.6) The projected gradient descent is computed as in the following equations in Phase 2 to produce U𝑛𝑒𝑤, V𝑛𝑒𝑤

𝑈𝑛𝑒𝑤 = Π𝒰(𝑈𝑡− 𝜂∇𝑈ℒ(𝑈𝑡, 𝑉𝑡; 𝑆𝑡) −1

2𝜂𝑈𝑡(𝑈𝑡𝑇𝑈𝑡− 𝑉𝑡𝑇𝑉𝑡)), (3.7)

𝑉𝑛𝑒𝑤= Π𝑉(𝑉𝑡− 𝜂∇𝑉ℒ(𝑈𝑡, 𝑉𝑡; 𝑆𝑡) −1

2𝜂𝑉𝑡(𝑉𝑡𝑇𝑉𝑡− 𝑈𝑡𝑇𝑈𝑡)), (3.8) where 𝛾 and 𝜂 are the algorithmic tuning parameters The loss function ℒ(𝑈, 𝑉, 𝑆) is defined as follows

ℒ(𝑈, 𝑉, 𝑆): =1

2‖𝑈𝑉𝑇+ 𝑆 − 𝑌‖𝐹2, (3.9) Finally, the low-rank and sparse matrix is updated in Phase 3

Continuously, when the new frame has been captured, we propose Algorithm 2 to update and initialize U𝑡, 𝑉𝑡 from frame t = M+1 The oldest column has been removed based on a

sliding window approach We update U𝑡, 𝑉𝑡 by adding the previous low vector to the last column and then discarding the oldest vector of U𝑡 and 𝑉𝑡 The updated matrix U𝑡, 𝑉𝑡 is utilized as the initialization for the optimization based on the Gradient Descent method This critical initial step saves the processing time to optimize U𝑡, 𝑉𝑡 Other optimization processes are performed again as in Phase 2 and Phase 3 in Algorithm 1

Algorithm 1: Online RPCA via GD

Input: 𝑋 ∈ ℝ𝑁×𝑇, window size M, rank r, step 𝜂

Output: Moving Object Matrix 𝑆 ∈ ℝ𝑁×𝑇, Low-rank Matrix 𝐿 ∈ ℝ𝑁×𝑇

9 // Phase 2: Gradient Descent based iterations

21 Initial U𝑡, V𝑡 using Algorithm 2

22 // Phase2: Gradient Descent based iterations