It also needs memory space for storing one edge image and searching operation that extracts the exact start and end points of a line from this edge image and the line parameter.. This ed
Trang 1A Real-time Finite Line Detection System
Based on FPGA
Dongkyun Kim, Seung Hun Jin, Nguyen Tuong Thuy, Ki Hoon Kim, and Jae Wook Jeon School of Information and Communication Engineering, Sungkyunkwan University,
Suwon, Korea {bluster, coredev, ntthuy}@ece.skku.ac.kr, dkkh1126@gmail.com, jwjeon@yurim.skku.ac.kr
Abstract-Image processing and analysis are active research
topics An intelligent vehicle and a service robot require these
techniques In particular, there is a big demand for line detection
because it has a wide range of applications The line features in an
image are used for object identification, robot navigation, and
intelligent vehicle control To detect the lines, a Hough transform
is generally used The Hough transform has good detection results
and it is robust to noise, but it takes a long time to execute and it
requires a great deal of memory to store the parameter space
This paper proposes a dedicated line detection hardware system
To increase the processing speed, it has a parallel Hough
transform unit, and it partitions the parameter space to decrease
the memory requirements It can detect not only the line
parameters, but also the exact start and end points of each line,
and it sorts these lines by length It can display the detected line
on a monitor via the DVI interface This system is designed with
VHDL and implemented on an XC4VLX200 FPGA The device
usage is about 15% and the maximum clock frequency is 67MHz
It can detect up to 256 lines in one image frame and it can process
up to 149 frames per second The simulation and real
experimental results are given to verify the system performance.
I INTRODUCTION
Research continues in image analysis to identify the contents
and to understand an environment in an image In
understanding the contents of image, line features are
important information The line feature is widely used in
industrial applications like image analysis [1], intelligent
robots [2], intelligent vehicles [3], and 3D reconstruction [4]
The Hough transform [5] is a standard method to find the line
features in an image The Hough transform is robust to noise
and changes in the illumination level, but it requires a great
deal of memory and execution time It cannot satisfy the
real-time constraint, so it is difficult to apply in products that
require real-time performance
Many studies have suggested methods of solving the
problems associated with execution time and memory space
when detecting lines Some studies tried to improve the line
detection algorithm in a general PC environment and others
concentrated on the development of dedicated hardware to
detect lines The first group tried to improve the accuracy; the
other group tried to improve the speed Of course, the first
group also tried to improve the processing speed for line
detection, but the improvement was poor Kiryati et al [6]
suggested a probabilistic Hough transform to find the exact
start and end points of a line However, this complex approach
still requires substantial execution time and memory, so it cannot be applied to a real product that needs a rapid response Many dedicated hardware systems have been proposed to achieve real-time line detection Tagzout et al [7] suggested an incremental Hough transform to reduce the computation cost
It can replace multiplication by addition, but it needs more time to process and its accuracy is less than that of standard Hough transform Ming-Yang and Yi-Hsiang [8] presented a parallel Hough transform circuit It can process up to 166 frames per second of a 512x512 image, but it can find only line parameters Nagata and Maruyama [9] presented a line detection circuit to find the start and end points of a line However, it cannot identify multiple lines that have the same line parameters, so it detects only one line for each parameter and it is not accurate
We present a finite line detection system to increase the speed and detection accuracy and to decrease the memory requirement It has a parallel Hough transform unit, a partitioned parameter space, and a parameter to edge image mapping mechanism To detect a line feature, a series of image processing techniques are used First, it extracts an edge image from a raw image Second, it extracts the line parameters theta and rho, which are based on the origin point in an image Third,
it must find some parameters associated with features that are prominent and conspicuous Finally, it needs check up procedure to find the exact start and end point of each line feature It can detect multiple lines that have the same line parameters Moreover, it can sort the detected lines based on their lengths The system is designed using VHDL and implemented in an FPGA The system input consists of the image sequence from a progressive camera and the line detection results are displayed on a monitor The system has a line table that has aligned line information We validate this internal data using the ChipScope system Experiments are conducted in which the system analyzes various scenes and environments
This paper is organized as follow Ch.2 explains the two major problems in line detection Ch.3 shows the details of the OLQH GHWHFWLRQ V\VWHP¶V FRQILJXUDWLRQ DQG explains how it solves the line detection problem Ch.4 presents the results of the simulation and the real environmental experiment Ch.5 is the conclusion
]\\
{GplllGpGjGG
pGpGOpukpuGYWW_P
kjjSGkSGrGq GXZTX]SGYWW_
`^_TXT[Y[[TYX^XT_VW_VKY\UWWGⓒYWW_Gplll
Trang 2II THE PROBLEM OF LINE DETECTION
The line feature in an image is the oriented and connected
pixel groups that are located at the boundary of two different
intensity regions This line information plays an important role
in most applications that use of image analysis, but the line
detection is not easy because of two major problems One is
the time and space complexity of the Hough transform, and the
other is how to find the exact start and end points of a line In
this chapter, we explain these problems
The Hough transform has a high time and space complexity
[10], and these two problems are tightly related The time
complexity comes from the characteristic of requiring repeated
operations for a whole edge point Equation (1) is the line
equation (in polar form) that is used for the Hough transform
It needs two multiplications and one addition Moreover, this
equation is repeated for the whole theta resolution for each
edge point In general, a 9*$ UHVROXWLRQ LPDJH¶V YDOLG HGJH
point percentage is as high a 10%, and the rho and theta
resolutions are 800 and 315 In this case, the total number of
multiplications is 640*480*0.1(theta resolution)*2 and the
total number of additions is 640*480*0.1(theta resolution)
A more serious problem is the space complexity The
memory size for storing the parameter space to vote is decided
by rho, theta, and the image resolution The rho and theta
resolution affects the height and width of the parameter space
The image resolution affects the width of the accumulated cells
in the parameter space In a VGA resolution image, the
maximum line length is 1024, so the width of accumulated cell
size is 10bits The height and width of the parameter space is
315 and 800 respectively, so the total parameter size is
315*800*10 bits This is the optimal size for line detection of a
VGA resolution image The problem comes not only from
memory space, but also from the speed of memory operations
Indeed, even if the line equation can be processed faster, the
memory operation speed cannot be faster The memory
operations for (1) consist of one read and one write Equation
(1) calculates the rho from the given x, y, and theta and the
accumulation procedure is executed using the calculated rho
value Some cell values in the parameter space are read and
increased by one, so the next calculated value from (1) must
wait until this accumulating operation is finished This is the
bottleneck of the Hough transform, and the speed of a Hough
transform depends on the speed of the memory operations
We present a parallel Hough transform to reduce the
execution time and we partition the parameter space to reduce
the memory requirements The detailed description of the
Hough transform hardware architecture appears in Ch3
A Hough transform and peak detection can detect some line parameters that are associated with features that are prominent and conspicuous For these lines, it can determine the theta and rho values With this information, we can determine the length and orientation of a line based on its image origin and the number of occupied edge points, but this solution is suitable for various general applications Some applications need the exact start and end points location of line The line with a start and end point location is a finite line and a line with only line parameter is an infinite line Fig.1 explains the difference between the finite and infinite line cases In Fig.1(a), a yellow line is considered to be one long red line, but in Fig.1(b), the line is considered to be three short red lines In the infinite line case, it cannot distinguish multiple lines that have same parameters However, in a finite line, it can distinguish each line The finite line case is more accurate and useful An additional procedure is required to improve the interpretation from the infinite to the finite line case A mapping operation that used the line parameters and the edge image is needed This mapping operation requires a great deal of time and memory It also needs memory space for storing one edge image and searching operation that extracts the exact start and end points of a line from this edge image and the line parameter To solve this problem, Kiryati et al [6] suggested a probabilistic Hough transform algorithm It does not need an additional mapping procedure; it combines a Hough transform and edge-parameter mapping Unfortunately, this requires a random operation and a large parameter space, so it is not efficient to implement in hardware Nagata and Maruyama [9] presented a line detection circuit that can find the start and end points of a line However, this circuit cannot distinguish multiple lines that have the same line parameters, and it is overly influenced by edge points that are not part of a located line The result from detecting the start and end points of a line
is not accurate
We present a new finite line detection system It can identify multiple lines that have the same line parameters The system has new mechanism for line parameter to edge image mapping
By using this method, the system can detect the exact start and end points of a line The detailed explanation of this line identification is given in the next chapter
Fig.1 The difference of infinite and finite line case
]\]
Trang 3III HARDWARE ARCHITECTURE OF THE LINE DETECTION
SYSTEM
The hardware architecture of the line detection system is
implemented in an FPGA and the design of the system is
explained in this chapter The line detection system consists of
an FPGA, a memory IC, and a signal interface IC Fig.2 shows
the conceptual view of the line detection system This system
is divided into four units These parts are the ³&DPHUD ,QSXW
Unit´ the ³(GJH ([WUDFWLRQ Unit´ the ³+RXJK 7UDQVIRUP
Unit´ DQG the ³/LQH ,GHQWLILFDWLRQ Unit´ :H SUHVHQW HDFK
PRGXOH¶VLQSXWDQGRXWSXWVHTXHQFH and internal operation
The Camera Input Unit is organized as the ³/9'6
WUDQVPLWWHU´ WKH ³FORFN PDQDJHU´ and the ³coordinate
FRXQWHU´
The camera outputs a 4bit LVDS data signal and a 1bit
LVDS clock signal The LVDS transmitter converts the 4bit
LVDS to a 28bit TTL signal based on a 24.545MHz camera
clock In the 28bit TTL signal, the 24 bit signal represents the
red, green, and blue pixel data; each color pixel is allocated
8bits The others represent the vertical sync, the horizontal
sync, the data valid signal, and the reserved bit The color
information is not needed for line extraction A green pixel is
used only for edge extraction The control and clock signals are
communicated to every unit, so each unit synchronizes on
these control and clock signals
The clock manager increases the clock speed by factors of 2
and 4 using the DCM core in the FPGA, so this line extraction
system has three clock signals; 24.545MHz, 49MHz, and
98MHz The higher frequency clocks are used to increase the
processing speed of the Hough transform unit and the line
identification unit because the Hough transform unit and the
line identification unit can operate independently of the camera
operating speed The operation of the camera input unit and the
edge extraction unit depends on the camera operating speed, so
these units use only the original camera clock
The position counter tracks the position of the pixel that is
input from the camera at the moment The coordinate counter
uses the camera clock, the vertical sync, the horizontal sync, and the data valid signal to count the pixel coordinate position
If the data valid sLJQDO LV µ¶ then the horizontal position is increased by one at every rising edge of the clock, so the vertical position is increased by one at the horizontal sync The horizontal and vertical position is initialized to 1 at the vertical sync Therefore, the top-OHIW SL[HO¶V SRVLWLRQ LV DQG WKH bottom-ULJKW SL[HO¶V SRVLWLRQ LV 7KLV SL[HO coordinate data is transferred to the edge extraction unit
D Edge Extraction Part
The edge extraction unit is configured with three modules One is the ³edge calculator´, the second is the ³edge position estimator´, and the last is the ³edge position list´
The edge calculator extracts an edge image from the raw image using real-time window processing [11] The raw image pixel and its position are transferred from the camera input unit The edge calculator can store the local region of a raw image using the window buffer, but the edge calculator does not use a frame buffer By using the canny algorithm, the edge calculator determines whether or not the center pixel of the window buffer is an edge point If this pixel is an edge, then the edge calculator outputs a µ¶,IWKLVSL[HOLVQRWan edge point, the edge calculator outputs aµ¶, so the edge calculator converts a 640*480*8 bit raw image to a 640*480*1 bit edge image, which is binary image This edge image is transferred to the edge position estimator and the memory controller of the line identification unit
The edge position estimator makes an edge position list from the edge image The edge image is not suitable for calculating using line equation because the line equation needs the x and y coordinates, but the edge image is just binary information The edge position estimator checks the result of the edge calculator
If this pixel is determined to be an edge by the edge calculator, then WKHHGJHSRVLWLRQHVWLPDWRUVWRUHVWKLVSL[HO¶VSRVLWLRQ in the edge position list At the end of the edge image, the edge position estimator writes the ending data to the edge position list to indicate the end of the list In the ending data, all of the bits are µ¶, which is not valid a position, so the system can distinguish between the ending data and the valid positions
The edge position list stores the position information of every edge point in one edge image The edge position list actually maintains two similar lists to prevent conflicts during read and write operations It is impossible to perform read and write operations simultaneously One list is for reading; the other is for writing By using this double buffering method, the reduction of the frame processing rate is prevented Moreover, the Hough transform unit can operate independently of the camera clock, so the Hough transform can be processed at a higher speed by using a higher frequency clock
Unit
Line Identification Unit
clock manager
coordinate
counter
edge calculator
edge position estimator
edge position list
line equation calculator
voting mermory
peak detector
peak table
inverse line equation calculator
line identifier
line table
memory controller
clock&sync signals
Fig.2 The configuration of line detection system
]\^
Trang 4E Hough Transform Unit
The Hough transform unit is configured as four modules
One is the ³line equation calculator´ that calculates the rho
value using the line equation The second is the ³voting
memory´ that makes the partitioned parameter space using the
rho value The third is the ³peak detector´ that finds the peak
position in the partitioned parameter space The fourth is the
³peak table´ to store the theta and rho values of the detected
peak The last is the ³Hough transform controller´ that controls
these four modules The peak table transfers information to the
line identification unit to identify the exact line position Fig.3
presents the Hough transform unit The Hough transform unit
contains 15 line equation calculator and voting memory pairs,
so it can operate 15 times faster The line equation calculator
and voting memory operate in parallel to calculate 15 values of
theta simultaneously, so this operation must be repeated 21
times to calculate the 315 theta values
The line equation calculator operates according to (1) It has
a sine table, a cosine table, two multipliers, and one adder The
sine and cosine table are constructed using a read only memory
(ROM) These whole sub-modules are configured as pipelined
structure After defined pipeline latency, a final result
consisting of the rho value is produced at every clock cycle
The total pipeline latency that is involved in converting the (x,
y, theta) information into the rho value is 7 clock cycles
The voting memory is configured in one block memory of
the FPGA The voting memory has two operation modes One
is increase mode and the other is clear mode This voting
memory accumulates the input rho values in increase mode
First, it reads the value of a memory cell in the voting memory
that is indicated by the rho value, aQGWKLVPHPRU\FHOO¶VYDOXH
is increased and updated In clear mode, the contents of voting
memory are transfer to the peak detector, and then whole data
is cleared The address of the voting memory is sequentially
increased in clear mode
The peak detector detects the peak point in the voting
memory, which is partitioned in the parameter space The
partitioned parameter space size is 15*800 Fig.4 shows the
configuration of the peak detector The peak detector finds
local maxima in 15*15 parameter space, and then compares
this max value and a predefined threshold value
The peak table stores the line parameters of the detected peak point in the partitioned parameter space The contents of the peak table are transferred to the line identification unit to identify the exact line position
The line identification part identifies the exact line position and their peak line parameters in an edge image The line identification unit is configured with four modules The first is the ³inverse line equation calculator´ to find the position of an image domain using the line parameters The second is the
³line identifier´ which checks the existence of a line in an edge image with the position value of the inverse line equation calculator The third is the ³line table´ to store the identified OLQH¶VVWDUWDQGHQGSRVLWLRQ0RUHRYHUWKLVOLQH¶VLQIRUPDWLon
is sorted by line length The last is the ³memory controller´ which stores the edge image and transfers the edge pixels to the line identifier
The inverse line equation calculator operates using (2)
cos sin
sin cos
i y
else
i x
end if end for
T
T
T
d d
Using this equation, it is possible to know the line position
by using the line parameters which are rho and theta The index
of (2) is varies and it is a function of theta,IWKHWD¶VUHJLRQLV 0.79~2.35, the x coordinate is used as an index and (2) calculates the corresponding y position In this case, the range
line equation
calculator
theta, x, y
peak table
15 number voting memory15 number peak detector
Fig.3 The configuration of the Hough transform unit
Peak detector
delay buffer
voting memory 15to1 comparator
max in rows
15to1 comparator max in columns
1to1 comparator with treshold peak
Fig.4 The configuration of the peak detector
]\_
Trang 5of the index value is from 1 to 640 Alternatively, the y
coordinate can be used an index of (2), in which cased the
UDQJHRIWKHLQGH[ YDOXH¶VLVIURPWR The inverse line
equation produces the (x,y) position of the line parameters
This position data is transferred to the line identifier and
memory controller to find the exact line position The line
parameter comes from the peak table of the Hough transform
unit If the index reaches the maximum value (either 480 or
640), the inverse line equation calculator reads the next line
parameter in the peak table The inverse line equation
calculator repeats these operations until no more parameters
remain in the peak table
The line identifier identifies the line using the line
parameters and the edge image The line identifier receives the
position data of the line parameters from the inverse line
equation calculator, and it UHTXHVWV WKLV SRVLWLRQ¶V HGJH GDWD
from the memory controller The edge image in the memory
controller is a 640*480*8 bit image If the pixel is an edge,
then the value of the SL[HOLV³´otherwise the value
of the SL[HO LV ³´ 7KH OLQH LGHQWLILHU PDNHs a
connected list of line positions, as shown in Fig.5 In Fig.5, the
connection list represents the connectivity of the edges that are
based on the line parameters The connection list has 15 cells
If the edge is located by using these line parameters (as in
Fig.5(a)), then the connection list has an entry
³´ If no edge is located using the line
parameters (as in Fig.5(b)), then the connection list has an
entry ³´ 8VLQJ WKLV FRQQHFWLRQ OLVW LW FDQ
determine the start and end points of line If the connection list
is³;´, then this is the start point of the line
(as in Fig.5(c)) If the connection list is ³;´,
then this is the end point of the line (as in Fig.5(d)) The
position of the start and end points of the line is stored in the
line table
The line table stores the start and end positions of each identified line Moreover, the line table sorts the lines by length The line table is configured as two tables that have the same size, as shown in Fig.6 The position data is stored in the left list and this data is copied to the right table using a sorting operation The data of the left list is sorted according to its theta and rho A selection sorting is used to sort the line data of the line table It finds the maximum line length data, copies this data to the right list, and clears this data from the left list These operations are repeated until the left list is empty After sorting, the left list is empty, and the right list has line data that
is sorted by length
The memory controller stores the edge image and transfers WKHHGJHSL[HOLQWKLVHGJHLPDJHZLWKWKHLGHQWLILHU¶VUHTXHVW The memory controller controls two FIFO memories and two SRAMs The FIFO is the frame buffer to separate the operating speed of the line identification unit from the camera clock speed By using the FIFO memory, the line identification unit becomes independent of the camera clock, so the identification unit can operate faster than the camera clock speed The SRAM stores the edge image The SRAM read and write simultaneously, so the memory controller has two SRAMs and switches between them when performing read and write operation Fig.7 shows the configuration of the memory controller
IV EXPERIMENT
An experiment using the proposed system is conducted Fig.8 shows the real experimental system The real-time finite line extraction system is implemented on a FPGA and designed VHDL This system receives a 24bit and VGA camera image
at a rate of 60 fps The system can display processing result on
(a)
111111111111111
(b) 000000000000000
(c)
0000000X1111111
(d) 1111111X0000000
Fig.5 The connected list of line position
Start End Length
.
.
.
Start End Length
.
.
.
copy withselection
sort
Fig.6 The line table
memory controller
front FIFO
back FIFO SRAM odd
SRAM even
edge image
line image
line Identifier
Inverse line equation calculator
line position
Fig.7 The memory controller
]\`
Trang 6the monitor same at a rate of 60 fps.
Fig 9 shows the experimental result of this system These
images are a snapshot of operating DVI monitor Fig.9(a) is a
raw image Fig.9(b) is a edge image Fig.9(c) is a infinite line
image on the edge image Fig.9(d) is a infinite line image on
the raw image Fig.9(e) is a finite line image on the edge image
Fig.9(f) is a finite line image on the raw image Fig.10 is a line
table and a constructed image using the contents of line table
The system of the real-time finite line detection is designed
using VHDL and is implemented in a Virtex-4 XC4VLX200
FPGA, with 200,448 Logic Cells(about 20M system gates),
6,048 Kbit Block RAM, 960 user I/O pins [12] This system
takes an image from the camera and extracts an edge image
from the raw image The system transforms from the edge
image to a parameter space and finds peak line parameters
The system finds an exact start and end position of line using
the parameter to edge image mapping The system sorts line
information by length and store internal block memory
TABLE 1, shows the summary of the real-time finite line
detection system design The clock frequency is 24.54MHz In
this clock, our circuit can process at 60 fps The maximum
frequency is 61MHz Then it can process at 149 fps
V CONCLUSION
We present a real-time finite line detection system To
increase line detecting speed, the system uses the parallel
Hough transform and the partitioned parameter space To
identify finite line, the system uses the line parameter to edge
image mapping method This system can detect an exact start
and end position of line The detected line information is sorted
by length and stored in the line table This system can process
images with VGA resolution at speeds of up to 149 fps The
line information can be used in image analysis, 3D
reconstruction, and intelligent vehicle Our system can also be used in these fields It can help in the creation of low-cost, lower-power, and high-speed applications
REFERENCES [1] Nagy G. ³Twenty years of document image analysis in PAMI´ IEEE
Transactions on Pattern Analysis and Machine Intelligence, vol 22, pp
38-62, Jan 2000.
[2] Kahn P, Kitchen L, Riseman E.M, ³$IDVWOLQHILQGHUIRUYLVLRQ-guided robot naviJDWLRQ´IEEE Transactions on Pattern Analysis and Machine
Intelligence, vol 12, pp 1098-1102, Nov 1990.
[3] 4LQJ/1DQQLQJ=³6SULQJURERW $SURWRW\SHDXWRQRPRXVYHKLFOHDQG
LWVDOJRULWKPVIRUODQHGHWHFWLRQ´IEEE Transactions on Pattern Analysis
and Machine Intelligence, vol 5, pp 300-308, Dec 2004.
[4] C Baillard, C Schmid, A Zisserman, A Fitzgibbon, ³$XWRPDWLF OLQH PDWFKLQJ DQG G UHFRQVWUXFWLRQ RI EXLOGLQJV IURP PXOWLSOH YLHZV´
Proceeding of ISPRS Conference on Automatic Extraction of GIS Objects from Digital Imagery, IAPRS vol 32, Sep 1999.
[5] J Illingworth, J Kittler,³A survey of the Hough transform´Journal of
Computer Vision, Graphics, and Image Processing, vol 44, pp 87-116,
1988.
[6] N Kiryati, Y Eldar, A M Bruckstein, ³A probabilistic hough transform´Journal of Pattern Recognition, vol 24, pp 303-316, 1991 [7] S Tagzout, K Achour, U Djekoune, ³Hough transform algorithm for FPGA implementation´Journal of Signal Processing, vol 81, pp
1295-1301, Jun 2001.
[8] C Ming-Yang, L Yi-Hsiang, ³Desing and integration of parallel hough-transform chips for high-speed line detection´ Proceeding of the 2005
[9] N Nagata, T Maruyama, ³Real-time detection of line segments using the line hough transform´ Proceeding of the 2004 IEEE International
Conference on Field-Programmable Technology, pp 89-96, 2004.
[10] M G Albanesi, M Ferretti, D Rizzo, ³Benchimarking Hough transform architectures for real-time´ Journal of Real-Time Imaging, vol 6, pp 155-172, 2000.
[11] C Torres-Huitzil, M Arias-Estrada, ³FPGA-based configurable systolic architecture of window-based image processing´EURASIP Journal on
Applied Signal Processing, Vol 2005, Issue 1, January 2005.
[12] Xilinx Inc, ³Virtex-4 FPGA family data sheet´ available from www.xilinx.com.
Fig.8 The implemented system
TABLE I
T HE DESIGN SUMMARY
Used Available Utilization Slice Flip Flops 17,713 178,176 9%
4 Input LUTs 13.517 178,176 7%
Occupied Slices 13,793 89,088 15%
Equivalent Gate Count 16,452,974
Fig.9 The experimental result of the finite line detection system
(a)
(c)
(b)
(d)
]]W
... The configuration of line detection system< /small>]\^
Trang 4E Hough Transform... Fig.9(e) is a finite line image on the edge image
Fig.9(f) is a finite line image on the raw image Fig.10 is a line
table and a constructed image using the contents of line table
The... transform architectures for real- time? ? Journal of Real- Time Imaging, vol 6, pp 155-172, 2000.
[11] C Torres-Huitzil, M Arias-Estrada, ? ?FPGA -based configurable systolic architecture