This paper proposes a Histogram Based Exposure Time Selection (HBETS) method to automatically adjust proper exposure time of each lens for different scenes. It guarantees at least two valid reference values for HDR image processing. Adopting the proposed weighting function restrains random distributed noise caused by micro-lens and produces a high quality HDR image.
Trang 1A HIGH DYNAMIC RANGE IMAGING ALGORITHM AND ITS IMPLEMENTATION FOR A 4 BY 1 CAMERA ARRAY
Vu Hong Son*, Pham Ngoc Thang
Abstract: Camera specification becomes smaller and smaller accompanied with
great strides in technology and thinner product demands, which leads to some challenges and problems One of those problems is that micro lens captures less light than normal lens, which makes low quality noise-image Moreover, current image sensor cannot preserve whole dynamic range in real world High Dynamic Range (HDR) image with multi-exposure images overcomes the problems mentioned above Choosing good exposure time is a seldom-discussed but important issue in HDR imaging technology This paper proposes a Histogram Based Exposure Time Selection (HBETS) method to automatically adjust proper exposure time of each lens for different scenes It guarantees at least two valid reference values for HDR image processing Adopting the proposed weighting function restrains random distributed noise caused by micro-lens and produces a high quality HDR image An integrated tone mapping methodology, which keeps all details in bright and dark parts when compressing the HDR image to Low Dynamic Range (LDR) image for being displayed on monitors is also proposed We first align these images to the same plane, and then adopt the proposed methods The result image has extended the dynamic range, i.e comprehensive information is provided Eventually, we implement the entire system on Adlink MXC-6300 platform that can reach 10fps to demonstrate the feasibility of the proposed technology
Keywords: Auto-exposure; HDR image; Tone mapping
1 INTRODUCTION
In recent years, HDR imaging technology becomes more and more popular An HDR imaging system for micro camera array is composed of many stages such as auto-exposure control, HDR generation, and tone mapping HDR imaging requires multiple exposed photographs to reproduce higher quality and clearer images One kind of these methods is
using bracketing [1]–[5], which captures the different-exposure image sequence by
adjusting Exposure Value (EV) The other kind of methods is brute force, which photographs lots of different-exposure images with no pixels over-exposed and under-exposed Benjamin Guthier et al [6] exploited pre-established HDR radiance histogram to derive the exposure time, which satisfies the user-defined shape of LDR histogram O Gallo et al [7] also proposed an approach to estimate HDR histogram of the scene, and selected the appropriate exposure times for LDR images
HDR image processing technology can mainly be classified into two methods, i.e exposure fusion and recovering high dynamic range radiance maps Both approaches require multiple exposed photographs to reproduce higher quality and clearer images Image fusion technologies [8], [14]–[16] have been developed for several years, which include depth-of-field extension [10], image enhancement [11], and multi-resolution image fusion [12] Fusion of multi-exposure images [15] proposed by Goshtasby is a famous approach to reproduce high quality image, but it cannot handle the boundary of objects perfectly Exposure fusion technology proposed by Mertens et al [8] generates an ideal image by preserving the perfect portion of the multiple different exposure images Fusion process technique described in [8] inspired by Burt and Adelson [13] transforms the domain of image, and adopts multiple resolutions generated by pyramidal image decomposition Debevec et al [9] proposed a method, which uses differently exposed
Trang 2photographs to recover the camera response function and blends multiple exposed images into a single high dynamic range radiance map The final stage of the HDR imaging system is the tone mapping that is required to compress HDR image to LDR one.The tone mapping approaches can be classified into two categories, i.e local tone mapping and global tone mapping Fattal et al [17] proposed a local tone mapping method, called gradient domain HDR compression This method is based on the changes of luminance in high dynamic range image It uses diffident levels of attenuation to compress high dynamic range according to the magnitude of the gradient Reinhard et al [18] proposed a global tone mapping method, called linear mapping approach In this paper, we develop a high dynamic range imaging algorithm and its implementation for a 4 by 1 camera array with more implementation details and additional experimental results than our previous work [20]
The rest of this paper is organized as follows: Section 2 describes the proposed algorithm that combines a histogram based exposure time selection, new weighting function and integrated tone mapping Section 3 presents experimental results and performance analysis Finally, the conclusion is given in Section 4
2 PROPOSED ALGORITHM
In order to achieve a high quality and high dynamic range imaging system, we propose
a system that can deal with higher noises of the low dynamic range images captured by using micro camera arrays In addition, by using the proposed algorithm, all details in the extreme scene can be completely preserved The design flow of the overall system is shown in Figure 1
Figure 1 The design flow of the proposed HDR system
As shown in Figure 1, the proposed algorithm is composed of many stages for different purposes The upper part represents the initialization of system, and the others indicate multi-exposure high dynamic range imaging generation and tone mapping In the histogram based exposure time selection stage, appropriate images are chosen for generating high quality HDR images Then, the new weighting function is used in HDR generation stage Eventually, those pixel values of high dynamic range image over 255 must be compressed through the tone mapping stage for display The details of each stage
of the proposed work are presented in the following paragraphs
A Image Alignment
An image alignment consists of the mathematical relationships that map pixel coordinates from source images to target image, is used due to each camera in camera array has its own viewpoint A feature-based method is adopted to accomplish image alignment, which is described in the following The feature point, which has information
Trang 3about the position and its descriptor, is extracted from images We can recognize the similarity among these features in different images by the feature descriptors In order to calibrate images to the same coordinate system, the homography matrix, which is a three
by three coordinate transformation matrix, is adopted Only eight elements are needed in light of a two-dimensional image, as shown in (1) The relationship between the original coordinate and the objective coordinate is represented by (2) and (3)
1 1 h h
h h h
h h h
z' y' x'
32 31
23 22 21
13 12 11
y
x
(1)
1 '
32 31
13 12 11
y h x h
h y h x h
1 '
32 31
23 22 21
y h x h
h y h x h
y (2)
'
'
z
x
X ,
'
'
z
y
Y , Z 1 (3)
B Histogram Based Exposure Time Selection (HBETS)
We propose a method called Histogram Based Exposure Time Selection (HBETS) to choose suitable source images to generate the HDR images The flow of HBETS is shown
in Figure 2 and Figure 3 in [20] Figure 3 shows the generated HDR image by using the source images captured by the proposed HBETS method Comparing Figure 3 with Figure
2, we can see that the red box region in Figure 3 has higher performance than that in Figure 2 after adopting HBETS to guarantee two effective pixel values, one of which is a redundant pixel value to reckon as a remedy to suppress the noise effect, and construct a higher quality HDR image
Figure 2 The HDR result image
with distortion
Figure 3 The HDR resulting image
generated by using the source images chosen by the proposed HBETS
C HDR Generation for Image Continuity
The camera response function curve g(x) has intense slope near the maximum and minimum pixel values, so g(x) is considered to be less smooth and more inaccurate near these two sides To overcome this, Debevec et al [10] proposed the triangle weighting function that highlighted the importance in the middle of pixel values In the case of different exposure, short exposure images generally have larger noise than long exposure images The micro camera array composed of small lens receives less amount of light than common cameras The ISO value of micro camera should be increased for enhancement However, noise is also amplified After applying the process of the Debevec’s weighting function, the noise dominates the pixel value Hence, this resulting pixel value is not the realistic luminance In order to overcome the drawbacks mentioned above, we propose a new weighting function to enhance HDR image quality as shown in [20] This weighting
Trang 4function can suppress more noises than Debevec’s weighting function does in the result image as shown in Figure 4, where Figure 4(a) uses the weighting function proposed by Debevec, and Figure 4(b) uses the proposed weighting function
Figure 4 Comparison of adopting two different weighting functions (a) HDR image using
Debevec’s weighting function, (b) HDR image using the proposed weighting function
Moreover, we utilize Gaussian filter and Laplacian filter to denoise and enhance image for further improving the image quality The method of applying Gaussian filter is to obtain a smoother image through a convolution of image with a normal Gaussian distribution model A three by three Gaussian kernel is used to achieve this target as shown in Table I(a) Then, we adopt Laplacian filter to further enhance the image quality
by strengthening the region changing rapidly such as edges and making image clearer, as shown in Table I(b)
Table I Two Different Enhancement Kernels Adopted in The Proposed Algorithm
(a) Gaussian Kernel (b) Laplacian Kernel
(a) Gaussian kernel (b) Laplacian kernel
0.0751 0.1238 0.0751
0.1238 0.2042 0.1238
0.0751 0.1238 0.0751
In addition, we consider that the pixel having large value in short exposure than the one
in corresponding long exposure has a higher chance of noise by reason of noise characteristics Consequently, there is a correction on the problematic pixel value The average of eight pixels which are the neighborhood of the problematic pixel in short exposure image is calculated, and used to replace the problematic pixel As shown in Figure 5, the noise (i.e red dot in the Figure 5(a)) is eliminated by the proposed method of pixel correction
(a) Before denoising (b) After denoising
Figure 5 Denoised image by applying the proposed method
D Integrated Tone Mapping
There are two major kinds of tone mapping techniques, i.e global tome mapping and local tone mapping The global tone mapping technique such as photographic compression uses a fixed formula for each pixel in compressing HDR image into LDR image This approach is relatively fast, but it loses details in high luminance regions On the other
Trang 5hand, the local tone mapping technique such as gradient domain compression refers to nearby pixel values before compression As a result, all details can be retained, but it takes
a lot of computation time Since both kinds of tone mapping methods have pros and cons, this motivate us to propose a new tone mapping approach that can preserve details in bright regions accompanying with lower computation time
Figure 6(a) to Figure 6(d) show four input images captured respectively with exposure time 0.33 ms, 2.10 ms, 10.49ms, and 66.23 ms, selected by the proposed HBETS method Meanwhile, Figure 6(e) demonstrates photographic tone mapping, and Figure 6(f) to Figure 6(h) are images used in the proposed algorithm with the scaling parameters 0.8, 0.5, and 0.2, respectively Photographic tone mapping lost details in bright regions (e.g the shape of lamp and the word near the lamp) In the proposed tone mapping method, large scaling parameter leads to discontinuity and small scaling parameter causes the unclear details Hence, some corrections are put on (4)
Figure 6 (a) to (d) show four input images captured respectively with exposure time
0.33 ms, 2.10 ms, 10.49ms, and 66.23 ms, selected by the proposed HBETS method (e) demonstrates the result by using photographic tone mapping (f) to (h) are images used
in the proposed algorithm with the scaling parameters 0.8, 0.5, and 0.2, respectively (i)
is the final result of the proposed tone mapping
Our idea is to blend two lower exposure source images first, which preserves details and also adjusts the brightness for image continuity, and then use the same equation to gain the result image, as shown in (5)
y) (x, I
* y) (x,
* ) -(1 y) (x,
Iresult I photograph 'source (4)
I'source (x, y) ( 1 ) *I source1(x,y) *I source2(x,y) (5) where α is shown in (6), and β is also a Gaussian-like function as illustrated in (7)
2
255
255 ) , ( 4
exp
*
threshold
ic photograph
I
y x I
(6)
2 2
255
255 ) , ( 4
exp
v
(7)
Trang 6where u is a constant value which dominates the weighting to the two image’s pixel values, v is a scaling parameter and Isource2 is the second low exposure image’s luminance
By using the proposed tone mapping method as shown in (5), the result image, which retains details in the brightness regions and keeps color continuity, can be acquired as
shown in Figure 6(i) In our experiment, the setting of γ as 0.5, u as 0.1, and v as 0.4
obtains a better result Comparing Figure 6(i) with Figure 6(e) to Figure 6(h), the word and the texture of the lamp in the scene by using the proposed algorithm can be preserved comprehensively
3 EXPERIMENTAL RESULTS AND PERFORMANCE ANALYSIS
We have implemented the proposed 4-CAM HDR system on Adlink MXC-6300 platform that can reach VGA video @10 fps The camera array consists of four Logitech webcams is shown in Figure 7
Figure 7 Four Logitech webcams
to form a 4x1 camera array
Table II Computation Time Analysis of The
Proposed Algorithm
Functions
Execution time (ms) Before
optimization
After optimization
HDR generation 61.48 28.51
HBETS and data type conversions
23.03 14.62
Figure 8 Experimental results of the proposed HDR algorithm preserving more details in
dark regions (a) to (d) are input images, (e) is the result image of easyHDR, and (f) is the
result image of the proposed method
Table II shows a detailed computational complexity analysis of code optimization for the proposed algorithm From Table II, we can see that the proposed design achieves 1.79 times faster processing speed after code optimization Besides, Table II also shows that tone mapping and HDR generation are the two most computation intensive computations which occupy near 70% computation time of the proposed HDR algorithm To further
Trang 7illustrate the validity of the proposed algorithm, we show the high dynamic range imaging results processed by the proposed algorithm, and compare those to a well-known commercial software tool called easyHDR [19] in the following
(a)
(b)
(c)
(d)
Figure 9 Experimental results of the proposed HDR algorithm achieving more saturated
scene (a) to (d) are input images, (e) is the result image of easyHDR, and (f) is the result
image of the proposed method
The exposure times of input images are chosen by the proposed HBETS approach with
a four by 1 camera array, which means four source images are used to generate an HDR
image Figure 8 demonstrates the proposed HDR result image, where Figure 8(a) to Figure
8(d) are input images; Figure 8(e) is the result image of easyHDR; and Figure 8(f) is the result image of the proposed algorithm Experimental results shown in Figure 8 indicate that the proposed algorithm preserves more details in dark region than easyHDR The texture and the words inside the box are able to be viewed clearly, and vibrant scene is given in the result image by applying the proposed method Comparing easyHDR result image shown in Figure 9(e) and the result image adopting the proposed method shown in Figure 9(f), we can observe that using the proposed algorithm generates more saturated
image in HDR
4 CONCLUSION
The proposed HDR system has been implemented on a four-by-one micro-camera array
so that the four source images can be used to generate HDR image The proposed histogram-based adaptive exposure time selection, which conquers the problem of extreme environment that auto-exposure system cannot afford, not only enhances image contrast but also keeps the image details in light-regions, and it also reduces the noise effect HDR system in camera array records comprehensive details at extremely low-light scene, which could be applied on car event recorders, surveillance systems, HDR movies, smart phones, and etc This work solves severe conditions such as night vision, and provides better visibility of the video at night Moreover, for entertainment applications, movie filmed with this technique will produce realistic videos for human visual perception Higher quality in back-lighted scene can also be achieved by the proposed design The proposed
HDR system makes cameras achieving high dynamic range close to that of human eyes
REFERENCES
[1] Reinhard, E., Pouli, T., and Cunningham, D., “Image Statistics: From Data
Collection to Applications in Graphics,” Proc ACM SIGGRAPH, no 6, 2010
Trang 8[2] M A Robertson, S Borman, and R L Stevenson, “Estimation-theoretic approach
to dynamic range enhancement using multiple exposures,” J Electronic Imaging, pp
219–285, Apr 2003
[3] R.C Bilcu,A Burian, A Knuutila, M Vehvilainen, “High Dynamic Range Imaging
on Mobile Devices,” Proc IEEE International Conference on Electronics, Circuits
and Systems, pp 1312–1315, 2008
[4] M A Robertson, S Borman, and R L Stevenson, “Dynamic Range Improvement
through Multiple Exposures,” Proc International Conference Image Processing, pp
159–163, 1999
[5] W C Kao, C C Hsu, C C Kao, and S H Chen, “Adaptive Exposure Control and
Real-time Image Fusion for Surveillance Systems,” Proc IEEE International
Symposium on Circuits and Systems, pp 935–938, 2006
[6] B Guthier, K Ho, S Kopf, and W Effelsberg, “Determining exposure values from
HDR histograms for smartphone photography,” Proc International Conference on
Multimedia, pp 425–426, 2013
[7] Gallo O., Tico, M., Manduchi, R., Gelfand, N., and Pulli, K., “Metering for Exposure
Stacks,” Proc Computer Graphics Forum, pp 479–488, 2012
[8] T Mertens, J Kautz, and F V Reeth, “Exposure fusion,” Proc Pacific Graphics, pp
382–390, 2007
[9] P Debevec and J Malik, “Recovering High Dynamic Range Radiance Maps from
Photographs,” Proc ACM SIGGRAPH, pp 369–378, 1997
[10] E Eisemann and F Durand, “Flash photography enhancement via intrinsic
relighting,” Proc ACM SIGGRAPH, pp 673–678, 2004
[11] R Raskar, A Ilie and J Yu, “Image fusion for context enhancement and video
surrealism,” Proc International Symposium Non-Photorealistic Animation and
Rendering, pp 85–94, 2004
[12] T Stathaki, “Image Fusion: Algorithms and Applications,” Academic 2008
[13] P J Burt and E H, Adelson, “The Laplacian Pyramid as a Compact Image Code,”
IEEE Trans Commun., vol Com-31, no 4, pp 532–540, 1983
[14] M Tico, N Gelfand, and K Pulli, “Motion blur free exposure fusion,” Proc IEEE
International Conference on Image Processing, pp 26–29, 2010
[15] A A Goshtasby, “Fusion of multi-exposure images,” Image and Vision Computing,
vol 23, no 6, pp 611–618, Jun 2005
[16] Kong, J et al., “A Novel Fusion Approach of Multi-exposure Image,” Proc International Conference on “Computer as a Tool”, pp 163–169, 2007
[17] R Fattal, D Lischinski and M Werman, “Gradient domain high dynamic range
compression,” Proc ACM SIGGRAPH, pp 249–256, 2002
[18] E Reinhard, M Stark, P Shirley and J Ferwerda, “Photographic tone reproduction
for digital images,” Proc ACM SIGGRAPH, pp 267–276, 2002
[19] easyHDR, “High Dynamic Range photography made easy”,
http://www.easyhdr.com/
[20] P H Huang, Y H Maio, and J I Guo, “High Dynamic Range Imaging Technology
for Micro Camera Array,” Proc 2014 Asia-Pacific Signal and Information
Processing Association Annual Summit and Conference, pp.1–4, 2014
Trang 9TÓM TẮT
MỘT THUẬT TOÁN TẠO ẢNH DẢI ĐỘNG CAO VÀ SỰ THỰC HIỆN CỦA NÓ
CHO MỘT MẢNG CAMERA 4x1
Đặc điểm camera trở lên nhỏ hơn kèm theo những tiến bộ vượt bậc trong công nghệ và nhu cầu sản phẩm mỏng hơn dẫn tới một số thách thức và vấn đề Một trong những vấn đề đó là thấu kính nhỏ thu ánh sáng yếu hơn so với thấu kính thông thường, cái tạo ra ảnh - nhiễu chất lượng thấp Ngoài ra, cảm biến ảnh hiện tại không thể bảo toàn toàn bộ dải động ở thế giới thực Ảnh HDR với nhiều hình ảnh phơi sáng có khả năng khắc phục những vấn đề đã được đề cập ở trên Lựa chọn một thời gian phơi sáng tốt là một vấn đề ít được thảo luận nhưng lại là vấn
đề quan trọng ở kỹ thuật tạo ảnh HDR Bài báo này đề xuất một phương pháp lựa chọn thời gian phơi sáng căn cứ vào biểu đồ để tự động điều chỉnh thời gian phơi sáng phù hợp mỗi thấu kính cho các ngữ cảnh khác nhau Nó đảm bảo ít nhất hai giá trị tham chiếu hợp lệ cho xử lý ảnh HDR Thông qua hàm trọng số đã đề xuất để hạn chế nhiễu phân phối ngẫu nhiên được sinh ra bởi thấu kính nhỏ và tạo ra một ảnh HDR chất lượng cao Một phương pháp được tích hợp ánh xạ sắc để giữ tất cả các chi tiết ở các phần tối và sáng khi nén ảnh HDR cho ảnh dải động thấp để hiển thị trên các màn hình là cũng được đề xuất Đầu tiên chúng tôi sắp các ảnh này trên cùng một mặt phẳng, rồi sau đó thông qua các phương pháp đã đề xuất Ảnh kết quả đã được mở rộng dải động, tức là thông tin toàn diện là được cung cấp Cuối cùng chúng tôi thực hiện toàn bộ hệ thống trên nền Adlink MXC-6300, cái có thể đạt 10 khung hình trên giây để chứng minh cho tính khả thi của kỹ thuật đã đề xuất
Từ khóa: Phơi sáng tự động; Hình ảnh HDR; Ánh xạ sắc
Received date, 02 nd May, 2017 Revised manuscript, 10 th June, 2017 Published, 20 th July, 2017
Author affiliations:
Hung Yen University of Technology and Education
*Corresponding author: hongson.ute@gmail.com