Energy efficient algorithms and techniques for wireless mobile clients 3

Image and its Discrete HistogramDLS - Dynamic Backlight Luminance Scaling The principle of DLS is to save power by backlight dimming while restoring thebrightness of the image by appropr

CHAPTER 2 RELATED WORK

Power management in battery operated mobile devices has been an active area ofresearch There are several proposals that attempt to reduce the power consumption

of various components of the mobile devices In this chapter, we review a range

of techniques available in the literature, primarily focusing on display, network andprocessor components

Most commonly, the Liquid-Crystal (LC) cells forms the pixels of Liquid-CrystalDisplays (LCDs) These cells can react to the modulation of electricity fields andchange the polarization direction of light passing through them in response to anelectrical voltage Thin Film Transistor LCD (TFT LCD) has a sandwich-like struc-ture with LC filled between two glass plates as shown in Figure 2.1 TFT glass has asmany TFTs as the number of pixels displayed, while a Colour Filter glass has colourfilter which generates colour LCs move according to the difference in voltage betweenthe Colour Filter glass and the TFT glass These LCs are aligned electronically toform a pattern or image Based on this pattern the backlight passed or blocked to anouter layer (transmissive layer) to create the image

Figure 2.1 Structure of a Transmissive TFT LCD

Most of the LCD power is consumed by its Backlight There are a several works

to reduce the backlight power consumption Dynamically dimming the backlight isconsidered an effective method to save energy consumed by the mobile device dis-plays The resultant reduced brightness can be compensated by image enhancementtechniques such as, scaling up the pixel luminance Traditional LCD displays usedCold Cathode Fluorescent Lamps (CCFL) for the backlight while modern displaysuse Light Emitting Diode (LED) arrays instead Dynamic dimming techniques de-scribed below can be applied for both CCFL and LED based backlights However,

as LEDs consume less power than CCFL, LED based LCDs are power efficient thanCCFL based LCDs [44] A 3-in-1 RGB LED can obtain a wider colour gamut andbetter pre-mixed colours

Various techniques in the literature for conserving LCD backlight energy are cussed below

dis-Figure 2.2 Visibility of the Image in a Transmissive TFT in some EnvironmentLuminance Condition [2]

Backlight Auto-regulation

In Gatti et al [2], a technique called backlight auto-regulation was devised whichregulates the backlight based on ambient lighting levels The relation between re-quired backlight luminance and ambient lighting is shown in Figure 2.2 This tech-nique employs an on-board environment luminance sensor to determine the appropri-ate backlight needs based on an ambient light condition The authors adjusted theinput voltage of the backlight driver to modify the luminance Power saving ratiobecomes more significant, reaching up to 74% as the environment becomes darkerand the backlight luminance becomes lower

This is a very simple backlight dimming technique and it is already implemented

in almost all modern smartphones with some better calibrations However, this nique can be complemented with image enhancement technique to save more energy

tech-Figure 2.3 Image and its Discrete Histogram

DLS - Dynamic Backlight Luminance Scaling

The principle of DLS is to save power by backlight dimming while restoring thebrightness of the image by appropriate image compensation This scheme is generallyknown as Dynamic Backlight Luminance Scaling In Chang et al [33], a dynamicbacklight luminance scaling scheme is proposed Based on different scenarios, threecompensation strategies are discussed, that is, brightness compensation, image en-hancement, and context processing Brightness compensation results in image dis-tortion Hence, amount of increase (transformation) to pixel luminance is controlled

by a threshold TH (shown in Figure 2.3) which is, determined using distortion ratio(Di) Distortion ratio (Di) is given in Equation 2.1

Di =

P2 n −1 j=T HHj(Mi)

P2 n −1 j=0 Hj(Mi) (2.1)where, Mi represents the image or frame as a matrix of pixels, H is histogramfunction,Hj represents histogram value of a particular colour j

Brightness compensation allows a significant degree of backlight dimming whilekeeping the distortion ratio reasonable, as long as the image has a continuous his-togram (adjacent histogram values are close to each other) which is not severelyskewed to bright areas Brightness compensation is not efficient for images withdiscrete histogram For discrete histograms Image enhancement techniques such ashistogram stretching and histogram equalization are proposed The transformation

to the pixels are controlled by lower threshold TL and upper threshold TH as shown

in Figure 2.3 However, some minor colours may be merged into each other and arethus no longer distinguishable after histogram equalization This may result in smallparts of images which has similar colour as background to become indistinguishablefrom each other Hence, a context based processing is proposed

However, their calculation of the distortion does not consider that the clippedpixel values do not contribute equally to the quality distortion

CBCS - Concurrent Brightness and Contrast Scaling

In Cheng et al [34], a similar method, namely, Concurrent Brightness and trast Scaling (CBCS), is proposed CBCS aims at conserving power by reducing thebacklight illumination while retaining the image fidelity through preservation of theimage contrast The authors defined a contrast fidelity function (fc(x)) for measur-ing the image fidelity after backlight scaling The authors also added a logic whichcontrols the distribution of output voltages to an original voltage divider in imple-menting a scaling function of LCD’s transmissivity This consequently eliminated thepixel-by-pixel manipulation on the image A voltage divider is a specific hardware

Con-designed to produce fixed voltages required by a source driver for setting a certainlevel of LCD’s transmissivity.

Cheng et al [34] clearly modelled the observed luminance of a transmissiveobject as a product of the backlight luminance and transmissivity of TFT-LCD Themodified voltage divider was used to implement a programmable LCD reference driver(PLRD) which takes two input arguments, a lower bound and an upper bound, asguidance in modifying the voltage to control the transmissivity of an LCD panel.The relation between a luminance function (bt(x)) and those two bounds is shown inFigure 2.4, where x is pixel value, t is the transmissivity of a pixel value, and b is

a backlight factor The luminance function consists of three regions: the undershotregion [0,gl], the linear region [gl,gu], and the overshot region [gu,1] In other words,the lower bound gl and upper bound gu are the darkest and the brightest pixel valuesthat can be displayed without contrast distortion (overshooting or undershooting)after applying CBCS The contrast fidelity function is defined as the derivative ofbt(x) as shown in Equation 2.2

Figure 2.4 Luminance as a function of Backlight and Transmissivity

simultaneous scaling both brightness and contrast presents better image fidelity thanscaling just one of these attributes The goal is to find the optimal bounds where theoverall contrast fidelity reaches its maximum A large overall contrast fidelity (almost

or the same as 1, which is the original image contrast) is better The authors of CBCSmethod found the optimum bounds by using this in a number of experiments Fromtheir experiments, they claimed that CBCS can achieve a significant power saving ofmore than 50% with small contrast distortion for still images

However, CBCS cannot maximize the potential of a dynamic backlight scalingscheme in saving power due to its overestimation in measuring distortion [3] CBCSmaximizes only the number of preserved pixel values or minimizes the number ofsaturated pixels

HEBS - Histogram Equalization for Backlight Scaling

Iranli et al [3] proposed histogram equalization technique for backlight scalingwith a pre-defined distortion level HEBS tries to find an appropriate pixel trans-formation function for each displayed image They argued that the image distortionshould be considered as a complex function of visual perception, and it should be mea-

(a) Original (b) 50% Contrast

(c) 50% Brightness (d) 50% CBCS

Figure 2.5 Visual Effects of Adjusting Brightness (b), Contrast (c), and Both (d)when the Backlight is Dimmed to 50%

sured by combining the mathematical differences between pixel values (histograms)and the characteristic of a Human Visual System (HVS) HEBS works on an imagehistogram and a transformation function which transform an original histogram into

a new uniformly distributed one with a specified minimum dynamic range Dynamicrange is a ratio or range between the brightest and the darkest available pixel values

in an image A new histogram should be different minimally from the original one

of a backlight-scaled image After they get the histogram transformation function,they define a dynamic linear function to transform an original image to a desiredresultant one Practically, since it is difficult to measure a distortion degree, thistechnique needed a number of experiments to get a mapping table of dynamic ranges

to distortion ratios from some benchmark images An example of a mapping result

is shown in Figure 2.6 The results are then used to specify the minimum dynamicrange in a new uniform distribution histogram

HEBS method results in about 45% power saving with an effective distortionrate of 5% and 65% power saving for a 20% distortion rate This is significantlyhigher power savings compared to previously reported ambient independent backlightdimming approaches

HVS Based Dynamic Tone Mapping

All of the aforementioned techniques rely on the luminance values of pixels ofthe displayed image as their optimization variables They do not consider the HVSperceived quality Luminance value of a light source is not the same as its perceivedbrightness Figure 2.7 shows the relation between luminance and perceived bright-ness The slope of each curve represents the luminance contrast sensitivity of human

Figure 2.6 Luminance as a function of Backlight and Transmissivity [3]

eyes, that is, sensitivity of the HVS brightness perception to the changes in the nance As luminance adaptation level of human eyes decreases (each curve representsdifferent luminance adaptation level), the luminance contrast sensitivity decreases It

lumi-is also observable from the figure that, the HVS exhibits higher sensitivity to changes

in luminance in the darker regions of an image Iranli et al [36] considered theHVS characteristics and proposed a more efficient way of backlight scaling using tonemapping Their method is known as Dynamic Tone Mapping (DTM) method Tonemapping is a classic photographic task of mapping of the potentially high dynamicrange of real world luminance values to the low dynamic range of the photographicprint The success of photography has shown that it is possible to produce imageswith limited dynamic range that convey the appearance of realistic scenes This isfundamentally possible because the human eye is sensitive to relative, rather thanabsolute, luminance values Many researchers have worked on automatically using

Figure 2.7 Luminance vs Perceived Brightness

tone mapping to compress the high dynamic range of images to a lower dynamicrange [45–48] In addition, the benefits of different tone mapping operators havebeen compared using both subjective evaluations [49, 50] and SSIM based objectiveevaluations [37]

Iranli et al [36] define luminance scaling problem as a Converse Tone Mapping(CTM) problem and provide a possible solution to the problem The CTM problem

Converse Tone Mapping (CTM) Problem: Given an original image χorig and mum allowable image distortion Dmax, find the tone mapping operation ψ : [0, Lorig

maxi-max] →[0, LDT M

max ] such that, LDT M

max is minimized while,

D(χorig, χDT M) 6 Dmax, where, (2.3)

χDT M ≡ ψ(χorig)

The proposed method (Equation 2.3) is amenable to highly efficient hardwarerealisation because it does not need information about the histogram of the displayedimage However, to realise the new tone mapping methods or operators in smart-phones in an efficient way, significant changes are required at hardware level

Backlight Local Dimming for LED backlit LCDs

In the previous sections, we have presented several backlight dimming techniqueswhich dim all the light sources together with the same dimming factor Those ap-proaches belong to a kind of backlight dimming scheme called a global backlightdimming scheme There are backlight dimming techniques which employ a differentapproach, where they can dim each light source individually with different dimmingfactors These approaches belong to a local backlight dimming scheme Shrirai et

al [51] describe 0D, 1D and 2D adaptive dimming approach for LED backlit LCD plays 0D dimming refers to overall dimming of the backlight (same as the dimmingtechnique for CCFL backlit LCD displays, described above), 1D dimming refers to

dis-line dimming (dis-linear strip of backlight) and 2D refers to individual point light sourcedimming and this type of dimming is not possible in CCFL of backlight.

Dynamic Backlight Adaptation for Videos

Pasricha et al [32] claim that aggressive compensation in luminance scaling troduces noticeable artifacts in still images, but these are less discernible in videobecause several frames appear on the screen every second They propose a proxybased architecture for dynamic backlight scaling of videos for handheld devices Allcommunication between the handheld devices and the video server passes throughthe proxy server, which changes the video stream in real time The authors haveintroduced the concept of a Group of Scenes (GOS) which defines the granularity

in-at which backlight compensin-ation is performed Generally, in video streams, manyframes with similar average luminosity values are clustered together, providing amplescope to optimize for low power by uniformly compensating entire GOS entities andreducing the handheld’s backlight level GOS is defined as a group of contiguousframes in a video stream such that the variance of the average luminosity values ofeach frame belonging to the group is less than a threshold value

Pasricha et al provide three middleware adaptation policies that use the sation algorithm The first one is Simple Backlight Compensation (SBC), in whichthe backlight is reduced for GOSs which have average luminosity value greater thanthe threshold (τ ) The proxy sends a control signal with the GOS to the handheldstating the amount of backlight to be reduced The second approach is to reduce thebacklight to a prefix value and then compensate it by increasing the brightness ofthe GOPs This approach is called Constant Backlight with Video Luminosity Com-

compen-pensation (CBVLC) The final approach is Dual-Comcompen-pensation Approach (DCA),

in which the video stream and the backlight levels for different GOS entities are multaneously compensated The proxy dynamically compensates the GOS entities

si-in the video stream and begsi-ins streamsi-ing the video to the client, simultaneouslydirecting the client, through the control stream, to change its backlight level Theclient middleware sets the appropriate backlight intensity levels for the video play-back This approach provides more flexibility for aggressive optimizations and results

in far greater power savings In addition, the video frames are convolved with ahigh pass filter to minimise impact on picture detail (loss of contrast) after aggressiveluminosity compensation

However, Pasricha et al focus only on luminosity compensation, ignoring thecharacteristics of the HVS Moreover, applying linear changes to the pixels of eachframe is computationally intensive for real-time video

Quality Based Backlight Adaptation for Videos

Cheng et al [35], employ a technique to incorporate video quality into the light switching strategy and proposes a Quality Adaptive Backlight Scaling (QABS)scheme The backlight dimming affects the brightness of the video Therefore, QABSonly consider the luminance compensation such that the lost brightness can be re-stored

back-Reducing backlight and enhancing image to compensate the loss in brightness,results in quality distortion due to clipped pixels The authors use Mean Square Error(MSE) (Equation 2.4) to measure the quality distortion and control the backlightdimming based on MSE threshold (Qth)

Figure 2.8 Relation Between MSE and Backlight Level

The dynamic adaptation algorithm takes a threshold value Qth and performs anexhaustive search on the model shown in Figure 2.8 to find the optimal backlight level,Alf a Then, the backlight is set to Alf a and the loss of brightness is compensated byincreasing image illuminance To reduce the frequency of backlight switching, theypropose two supplementary schemes to smooth the backlight switch process suchthat the user perception of the video stream can be substantially improved First, alow-pass digital filter is proposed to eliminate any abrupt backlight switching that iscaused by the unexpected sharp luminance change Second, they propose to quantize

the number of backlight levels, that is, any backlight level between two quantizationvalues can be quantized to the closest level, by which we prevent the needless backlightswitching for small luminance fluctuations during one scene The evaluations showthat by applying the scheme, up to 40% power can be saved with negligible loss tovideo quality However, the metrics used for distortion and quality do not considerthe characteristics of HVS It is well known that MSE and PSNR (Peak Signal toNoise Ratio) are not the best measures to assess perceptual quality for most videosequences [52] [53] Widely adopted metrics such as, SSIM (Structural SimilarityIndex) [54] can be used provide better estimation and quality.

The power consumption of an OLED displays depends on the Lightness and Colour(Hue) of the pixels displayed Energy consumption of an OLED display can be com-puted as sum of energy consumption of all pixels Energy consumption of a pixel issimply sum of energy used to illuminate R (Red), G (Green) and B (blue) organicmaterials which forms the pixel [55, 56] We confirmed this with our own mesure-ments and presented the results in Section 4.1 Chapter 4 Hence transforming thecontent colours to their power efficient versions has become the primary mechanismfor conserving power in OLED devices There are several works on colour transfor-mation

Energy Aware Colour Set

Chuang et al [4] propose two colour mapping approaches that swaps currentcolours with an iso-lightness colour set within a certain restrictions, so that the overall

lightness of the final display is still the same with the original display The firstone is discrete optimisation approach, in which, from a set of M iso-lightness anddistinguishable colours, a subset of N colours (N ≤ M ) is chosen so that the sum ofthe energies associated with the chosen colours is the minimum of any subset of Ncolours To select the set of M adequate input colours the authors propose to adoptnamed (categorical) colours because they are sufficiently distinct and they exhibitproven perceptual benefits For example, Kawai et al [57] suggest that the time

it takes to distinguish multiple colours depends partly on their named colour region.The number of easily distinguishable colours is small For example, in his study

of categorical colours, Healey [58] finds that 7 distinct iso-lightness colours is themaximum number of colours that can be displayed at one time without lowering theresponse time and accuracy of target colour identification Based on Healey’s [58]study, Chuang et al have selected 6 quickly identifiable colours (green, blue, orange,purple, red, yellow) measured their power consumption (Figure 2.9 and Figure 2.10)

L∗ in these figures indicates lightness parameter as defined in CIELAB color spaceand E denotes normalised energy consumption Users can pick distinguishable iso-lightness colours with increasing energy cost by choosing colours from bottom to topfrom the figures

The second one is continuous optimisation approach, in which, the goal is tofind energy aware, distinguishable, iso-lightness colours with 3 input parameters: thenumber of colours N , the level of lightness L∗, and a minimum perceptual colourdistance d that must be enforced between every pair of colours.The rationale is thatthis minimum distance is a means of ensuring distinguishability of the colours L∗ is

Figure 2.9 Energy plot of the hues of 6 categorical colours (from left to right: blue,red, purple, orange, green, yellow) Evergy (E) vs Lightness (L∗) [4]

Figure 2.10 Categorical colours of varying lightness sorted by increasing energycost [4]

(a) Original Colourset (b)Discrete Optimization (c) Continuous Optimization

Figure 2.11 Tooth Dataset Coloured with Traditional Colours (Original) and

En-ergy Efficient Colours [4]

the lightness parameter as defined in CIELAB ( International Commission on

Illumi-nation recommended L∗, a∗, b∗) colour space A cost function is defined that takes

into account the energy of the colours and the distances between them in selecting

the colours

In both approaches, colour distinguishability and iso-lightness are ensured The

examples for 2D colour mapping and volume rendering achieves around 40% energy

saving However in Chuang et al.’s approaches, the original colours are significantly

changed (Figure 2.11) in the process and not suitable to be applied to photos and

videos, in which colour accuracy is important

Colour Transformation for GUIs

Dong et al [28] describe two approaches of reducing power consumption of

OLED-based displays on mobile phones through structured colour transformation of standard

background and foreground Graphical User Interface (GUI) objects, or unstructured

transformation of individual pixel colours of arbitrary GUI elements In unstructured

transformation, a power efficient colour for background is identified first, then colourswhich have high contrast with the selected background colour are be used as windowborders and foreground colours In unstructured transformation, Dong et al do aexhaustive search in the colour space to replace each colour in the GUI with a powerefficient colour To ensure readability, the approach ensures the colour differencebetween every two colours in the transformed GUI is no less than their counterparts

in the original

Dong et al propose two options for unstructured transformation First one ismono-colour mapping where the colours of the GUI are mapped to a single colourwith varying lightness The second one is rank-based colour mapping where colours ofthe GUI are mapped to different set of colours which are power efficient In their study,they observed that users preferred mono-colour than multi-colour GUI Their imple-mentation takes power efficient requirement and options (structured/unstructured;mono-colour/multi-colour) as input and maps to energy efficient colours However,the authors did not consider one of the key property (Colour Harmonicity) that deter-mines the visual quality of the GUI This may make the GUI visually unpleasant forthe human eyes In addition, the approach dramatically alters the original colour ofthe displayed objects (shown in Figure 2.12), again making it unsuitable for contentswhich are sensitive to colour fidelity changes, such as images and videos

Colour Transformation for Webpages

In their next paper, Dong et al [6] presented a mobile web browser which forms the colours of the webpages to reduce power consumption Their transforma-tion logic, has four options, namely Dark, Green, Arbitrary and Inversion Dark and

trans-Original Green MultiColour

Figure 2.12 Unstructured Transformed GUIs with Different Settings

Green options map the colours of the web page to darker version or shades of greencolour as green being the most energy efficient colour However, this makes the entireweb pages look dark and greenish Moreover, it won’t be interesting to view all webpages in same colour for the users This may bring dissatisfaction to the owners ordesigners of the web sites Arbitrary approach maps the colours to arbitrary coloursthat are energy efficient and Inversion approach simply inverts the colours

In all the four options, the brand identity (brand colour) of the page is lost makingthe page not recognisable by its colour scheme For example, their results for Enter-tainment and Sports Programming Network (ESPN) webpage (excluding Darkeningoption) shown in Figure 2.13 has totally different colours than the brand colours ofESPN, which are red, white and black The foreground images are simply darkenedwithout considering HVS characteristics Applying pixel-by-pixel colour transforma-tion during the rendering phase of the browser is also compute-intensive task Analternative processing, such as, processing the source code of the web page for colourtransformation will be computationally efficient

Original Green

Arbitrary Inversion

Figure 2.13 Colour Transformed ESPN Webpage

Original Colour Quantized

Figure 2.14 Colour Quantizing an Image to N = 32 Colours

Just Noticeable Deference Constrained Transformation Scheme

Hadizadeh et al [59] describe a colour mapping algorithm which maps the imagecolours to power efficient versions constrained by Just Noticeable Deference (JND)threshold The authors first colour quantize the entire image in to set of N distinctcolours (C1, C2 CN) as shown in Figure 2.14 The colour of all pixels in a quantum

i is same (Ci) Then, each colour Ci is replaced with another colour which is energyefficient and lies within the JND threshold

To obtain energy efficient colours, the authors propose the following algorithm.Let the colour of the given pixel C be (Y, Cb, Cr) in Y CbCr colour space Theyfirst compute J N DY which is the spatial luminance JND computed from the Ycomponent of the colour C Adding or subtracting upto J N DY amount of luminance

to the Colour C0s luminance component Y will have unnoticeable effect for humaneyes Hence, two new colours, C+ and C− are created, by adding and subtractingJNDy These two new colours can be considered visually indistinguishable from Cfor human eyes, since their chroma components are the same as those of C, andthe difference between their luminance components and the luminance component

of C does not exceed the JND threshold The three colours (C, C+, C ) are then