Adapting underwater physical link parameters using data driven algorithms

In the process of finding an optimal solution, several data driven algorithmssuch as Brute Force, First Hit, Random Selection and -Greedy were studied pri-marily and the effect of perfor

Adapting Underwater Physical Link Parameters

Using Data Driven Algorithms

D Melani Jayasuriya

BSc Eng (Hons), University of Moratuwa

A THESIS SUBMITTED FOR THE DEGREE OF MASTER OF ENGINEERING DEPARTMENT OF ELECTRICAL AND COMPUTER

ENGINEERING NATIONAL UNIVERSITY OF SINGAPORE

2010

I would like to thank Assistant Professor Mandar Chitre, for his supervisionand support during this research I am grateful to him for having been constantlysupportive and encouraging, and for his helpful suggestions and critical comments

It was tough at times to meet up with your expectations, but I should say it trulyhelped

First of all I should thank Professor Rohan Abeyratne, Director of MIT Alliance for Research and Technology (SMART) and Dr David Burke, thenProgram Manager of center for environmental sensing and modeling (censam)under SMART for introducing me to my supervisor I am truly grateful for theirhelp at the time

Singapore-I had the pleasure of working with people in the Acoustic Research Laboratory(ARL), Tropical Marine Science Institute (TMSI) who always treated me as part

of ARL family and always greeted with a warm smile I should especially mentionSatish Shankar for sharing his knowledge on the problem

I would like to thank Dr Paul Seekings and Kian Peen of Marine Mammal search Laboratory and Associate Professor Lonce Wyse of Interactive and DigitalMedia Institute for their support and encouragement

Re-I would also like to thank the wonderful people at Republic Polytechnic, fortheir understanding and support during this period I was able to attend all myevening lectures for two semesters, two days a week because of their kind support.Not many employers would have have done that, and I am deeply thankful to all

i

of them I am truly grateful for the support and understanding of Dr Wong HauShian for her continuous encouragement that has helped me in completing theresearch as well as the seminar requirements Dr Zhou Kainan deserves specialthanks for the many useful discussions we had and for her valuable advice and helpthroughout.I am grateful to have colleagues like Mr.Tan Kok Cheng and Ms.YapWoan Leng who helped me manage my work, specially during the stressful period

on top of all, for spending many hours day through night helping to simulate andanalyze the results as well as proof-reading and formatting the thesis I am sureyou now know more about this topic now than you ever wanted to Your endlessmoral support and words of wisdom did not ever let me give up no matter howmuch I wanted to I am truly blessed to have you all as my family Your love andsupport have brought me this far and I will never let you down

1.1 Background and Motivation 1

1.2 Problem Statement 2

1.3 Choice of Modem Parameters 3

1.4 Explore or Exploit? 4

1.5 Thesis Contributions 5

1.6 Thesis Organization 6

2 Background and Related Work 8 2.1 Multi Armed Bandit Problem 8

2.2 Application of Multi Armed Bandit to the Problem 9

2.3 Gittins Index Strategy 12

3 Problem Formulation 15 3.1 Definitions 16

3.2 Channel Representation 20

4 Preliminary Solution Strategies 23 4.1 Basic Strategies 23

4.1.1 Brute Force Strategy 24

4.1.2 First Hit Strategy 28

4.1.3 Random Selection Strategy 31

4.1.4 -Greedy Strategy 33

iii

Contents iv

4.2 Enhanced Strategies 37

4.2.1 Enhanced Brute Force Strategy 38

4.2.2 Enhanced First Hit Strategy 39

4.2.3 Enhanced -greedy Strategy 41

5 Ranked Exploration Strategy 43 5.1 Motivation 43

5.2 Weights Matrix 45

5.3 The Algorithm 45

5.4 Likelihood Assignment 48

5.4.1 Proportional Likelihood Assignment 48

5.4.2 Rank-based Likelihood Assignment 49

5.5 Adaptive Temperature Profile of Probability Distribution 50

5.5.1 Simulated Annealing 51

5.6 Selection using ‘Roulette Wheel Selection’ 54

6 Results and Discussion 56 6.1 Simulation formulation 56

6.2 Brute Force Strategy vs Enhancement 58

6.3 First Hit Strategy vs Enhancement 60

6.4 Random Selection Strategy 62

6.5 -Greedy Strategy 63

6.6 Enhanced -Greedy Strategy 65

6.7 Ranked Exploration Strategy 67

6.8 Discussion 69

7 Conclusion 75 7.1 Summary of Results 75

7.2 Future Work 77

A MATLAB codes for generating data sets 78 A.1 Generation of data sets 78

A.2 Sorting the data sets 80

This thesis addresses the question of transferring a file in minimum possibletime using an underwater acoustic link that can be tuned by changing physical linkparameters Assuming we have no prior knowledge about the average data rateresulting from any of the parameter choices, we have to decide between exploringfor new parameter values versus exploiting the best from the known parametervalues Hence our objective is to devise a strategy to balance this exploration andexploitation in order to transfer a file in minimum time

In the process of finding an optimal solution, several data driven algorithmssuch as Brute Force, First Hit, Random Selection and -Greedy were studied pri-marily and the effect of performance for varying search space, burst size and filesize for each algorithm were investigated When they did not produce promisingresults, we moved on to exploring our own strategies and enhancing the availablestrategies with the contextual information available Enhanced -Greedy is onesuch example

Learning from widely accepted theories of optimization, such as Simulated nealing and Rank-based assignments, the proposed Ranked Exploration Strategywas formulated It does not have a fixed  probability to explore, but rather it has

An-a distribution from which it decides whether to explore or to exploit And thisdistribution is not fixed either The more confident we become about the observa-tions made, the more biased the distribution becomes towards exploitation Thiswas also analyzed on its performance with respect to the various parameters

Summary viSimulations were performed on channel data matrices which effectively modelthe underwater acoustic environment Simulation results showed that the RankedExploration performed well while providing a computationally efficient solution.

List of Figures

3.1 PER vs BER for lj = 8000 bits and h = 2000 bits 21

3.2 Channel Model 22

4.1 Throughput of Brute Force Strategy - Analytical Results vs Sim-ulation Results 26

4.2 Throughput of Random Selection Strategy - Analytical Results vs Simulation Results 34

5.1 Flowchart of Ranked Exploration Algorithm 46

5.2 Sample Proportional Likelihood Assignment 49

5.3 Principle of Simulated Annealing 51

5.4 Temperature profiles for different values of λ 53

5.5 Comparison of throughput with search spaces with and without temperature profiling for Ranked Exploration Strategy 54

5.6 An instance of Roulette Wheel Selection for data in table 5.2 55

6.1 Variation of throughput with search space for Brute Force Strategy 59 6.2 Comparison of throughput with search space for Brute Force and Enhanced Brute Force 59

6.3 Variation of throughput with search space for First Hit Strategy 60

6.4 Comparison of throughput with search space for First Hit and En-hanced First Hit 61

6.5 Variation of throughput with search space for Random Selection Strategy 62

vii

List of Figures viii

6.6 Variation of throughput with search space for different  for Greedy Strategy 636.7 Variation of throughput with file sizes for different  for -GreedyStrategy 646.8 Variation of throughput with burst sizes for different  for -GreedyStrategy 656.9 Variation of throughput with search space for different  for En-hanced -Greedy Strategy 666.10 Variation of throughput with file sizes for different  for Enhanced

--Greedy Strategy 666.11 Variation of throughput with burst sizes for different  for Enhanced

-Greedy Strategy 676.12 Variation of throughput with search space for Ranked ExplorationStrategy 686.13 Variation of throughput with file sizes for Ranked Exploration Strat-egy 686.14 Variation of throughput with burst sizes for Ranked ExplorationStrategy 696.15 Comparative throughput for various strategies 706.16 Comparative throughput for various algorithms with varying searchspaces 716.17 Comparative throughput for various algorithms with varying file sizes 726.18 Comparative throughput for various algorithms with varying burstsizes 73

List of Tables

1.1 Average data rate for various link and coding schemes 4

5.1 A sample set of Selection Probabilities for Proportional LikelihoodAssignment 485.2 Selection Probabilities (according to a simple temperature profile)for a sample set of likelihoods and ranks 55

ix

ARL Acoustic Research Laboratory

BER Bit Error Rate

DPSK Differential Phase Shift Keying

FEC Forward Error Correction

MAB Multi Armed Bandit

OFDM Orthogonal Frequency Division MultiplexingPAPR Peak to Average Power Ratio

PER Packet Error Rate

RES Ranked Exploration Strategy

x

List of Notations

A All feasible combinations of codes and packet lengths

btx Number of packets in a transmitted burst

brx

u Number of successfully received packets in burst u

C All available code rates

cj Code rate of code-packet length combination j

di Uncoded Link Rate of link scheme i

Echij m × n channel PER matrix

fuxfer Amount of data transferred successfully by burst u

γj Packet efficiency factor for code-packet length combination j

L All available packet lengths

lj Packet length of code-packet length combination j

M Maximum length of a coded packet supported by the modem

m Number of link schemes

n Number of code-packet length combinations

TS Time taken to transfer the entire file corresponding to sequence S

tu Time taken to transfer fuxfer amount of data

tP Burst Penalty

tsw Time taken by the protocol to change parameter values

wi Transition Point

xi

Chapter 1

Introduction

Acoustics being the best medium available for underwater communication, is stillfaced with a lot of challenges compared to terrestrial communication [1] As Sozer

et al [2] pointed out, propagation speeds are considerably lower than radio wavepropagation; medium causes reverberation due to multi path propagation; andavailable bandwidth is relatively small Due to the multi-path propagation andambient noise, the effective data rates are lower and packet loss rate is usuallymuch greater Comprehensive reviews of underwater acoustic communications arepresented in [3],[4],[5]

As James Preisig [6] pointed out “There is no ‘typical’ underwater acoustic vironment, and no ‘typical’ underwater acoustic communications channel exists”

en-1

Chapter 1 Introduction 2The large variation in channel conditions among different locations such as dif-ferences between deep water and shallow water suggest that significantly differentcommunication parameters (modulation technique, frequency band, frame length,error correction methods, etc.) would be optimal for different locations [7] Hencethe design of reliable general purpose systems that work effectively across a broadspectrum of environments remains a challenge.

Fortunately, a vast majority of the components of a modern communicationsystem are implemented in software, affording us the ability to fine tune theirparameters during operation The objective for tuning the parameters could be

to optimize data rates, protect against errors, minimize power, and so on If thephysics of the channel is completely known, it is possible to determine the values

of these parameters for a given objective However in practice, it is quite difficult

to know the state of the channel completely The parameters usually interact witheach other, so tuning them in isolation is often not possible

We consider the problem of transferring a file in the minimum possible time using

an underwater acoustic link We assume that the channel remains static over thecourse of the file transfer The file will be transferred over multiple packet bursts

At every transmission, we may choose to tune modem parameter values Wehave to take note that we assume no knowledge about the physics of the channel.Any packet losses resulting from transmissions, can be measured using feedback

Chapter 1 Introduction 3channels Since we do not have any prior information about the average data rateresulting from any parameter choice, we need to choose between exploring newparameter values versus exploiting the parameter values that have yielded the bestresults so far The objective is to devise a strategy to balance this exploration andexploitation so that the file transfer time is minimized.

As an illustrative example of modem parameters that can be tuned, consider themodem implementation in [8] The Orthogonal Frequency Division Multiplexing(OFDM) based modem had three choices for modulation: 2-differential phaseshift keying (DPSK), 4-DPSK and 8-DPSK There were also two forward errorcorrection (FEC) encoders Each of the FEC encoders had 14, 13 and 12 rate codes,thus resulting in a total of 27 possible configurations, out of which 10 were testedand the corresponding bit error rates (BER) were recorded For a packet size of

8000 bits, the BER from the paper can be converted into packet error rates (PER)assuming that the errors are independent The average data rate is then simplythe product of the uncoded data rate, code rate and the PER Result are tabulated

Chapter 1 Introduction 4

Table 1.1: Average data rate for various link and coding schemes

Modulation Code Rate BER Avg Data Rate (bps)

We have two choices at every transmission:

1 Try out new link parameter values, thus exploring the parameter searchspace

2 Exploit the parameter values with the highest observed data rate

Exploration contributes to the knowledge base and improves channel knowledge.Improved channel knowledge enables us to choose the parameter set with thehighest average data rate and therefore minimize the time taken to transfer thefile Every exploration is associated with a cost: time is lost and the result of thetransmission may or may not be successful The state of the existing knowledge

Chapter 1 Introduction 5base and the size of the file remaining to be transferred are key factors in decidingbetween exploring and exploiting For example, if the size of the file is very large

or if we know nothing about the channel, it is probably better to spend sometime exploring the search space before settling on the best parameter values Onthe other hand; if we have a high confidence on the accuracy of our channelestimate (the estimate made from acquired channel knowledge), or if the size ofthe remaining file is very small, it is probably better to exploit the best knownparameter value to completion Thus an ideal decision policy would be one thatconsistently makes an appropriate choice in light of expected rewards, existingknowledge, and remaining file size

We present a data driven approach for tuning the physical layer parameters of acommunication link to optimize data rates, assuming the channel remains staticover the course of a file transfer Our approach does not need any knowledge ofthe physics of the channel Several data driven algorithms such as Brute Force,First Hit, Random Selection and -Greedy were studied primarily and the effect

on performance for varying search space, burst size and file size for each algorithmwere investigated Finally the performance of the proposed Ranked ExplorationStrategy was analyzed with respect to various parameters Simulations were per-formed on channel data matrices which effectively model the underwater acousticenvironment

ˆ Chapter 2 - Background and Related Work

Background information related to the research topic is presented in detail.Topics include algorithms already in place for various problems which aremulti armed bandit problems in nature

ˆ Chapter 3 - Channel Model

Describes how the channel model was developed in order to carry out lations and testing for the underwater acoustic environment

simu-ˆ Chapter 4 - Preliminary Solution Strategies

Preliminary strategies are introduced which proceed to discussions on theirproposed enhancements

ˆ Chapter 5 - Ranked Exploration Strategy

Our proposed approach, its methodology and performance is explained inthis chapter

ˆ Chapter 6 - Results and Discussion

The proposed algorithm and an analysis of its performance compared to therest of the existing strategies are discussed in this chapter

Chapter 1 Introduction 7

ˆ Chapter 7 - Conclusion

Discussion on the achieved results and analysis on the usability of the rithm A discussion of the future work is also provided

algo-Chapter 2

Background and Related Work

This chapter discusses the various strategies and existing theories which were lyzed in order to find a solution to the problem Literature reviews on topics such

ana-as multi-armed bandit problem showed how researchers have developed solutions

to similar problems in the past

Decisions in real life are often made in order to maximize some expected reward.But these decisions, or the actions they generate, do not just bring in more reward,they can also help discover new knowledge that could be used to improve futuredecisions Balancing reward maximization based on the knowledge acquired andattempting new actions to further increase knowledge, has always been a problem,which is also known as the exploitation vs exploration tradeoff in reinforcementlearning

8

Chapter 2 Background and Related Work 9The multi-armed bandit (MAB) problem, originally described by Robbins in

1952 [12], is an instance of this general problem A multi-armed bandit, also calledK-armed bandit, is similar to a traditional slot machine (one-armed bandit) but

in general has more than one lever Each lever of a k-armed slot machine provides

a reward drawn from a distribution associated to that specific lever Initially, thegambler has no knowledge about the probability distribution of each lever, butthrough repeated trials, he can focus on the most rewarding levers The objective

is to maximize the reward sum gained through iterative pulls of levers We assumethat the gambler has only a limited number of tries The problem of determiningthe best strategy for the gambler is called the multi-armed bandit problem [13],[12], [14]

Problem

In our problem for every packet burst transmission, we need to tune the modem

to values chosen from a pool of candidate options We group these options aslink schemes and coding schemes With link schemes, we refer to all tunableparameters of a communication system excluding FEC coding and packet size.And the coding schemes are FEC coding and packet size combinations At anygiven time, the user is to choose a link-coding scheme combination to transferthe file Hence, our problem can be considered as a multi-armed bandit problemwhere the link-coding schemes represent the arms of the bandit The horizon, at

Chapter 2 Background and Related Work 10any given time is the file size remaining to be transferred Thus finding the bestreward, maps into selection of the optimum parameter set to transfer the file inminimum time.

The objective for tuning the parameters could be to optimize data rates, tect against errors, minimize power, and so on If the physics of the channel iscompletely known, it is possible to determine the values of these parameters for

pro-a given objective However in prpro-actice, it is quite difficult to know the stpro-ate ofthe channel completely The parameters often interact with each other, so tuningthem in isolation is often not possible We often need to select from a large pool ofcandidate parameter values, but since it is not always obvious why a given set ofvalues perform better than the rest, the selection reduces to an arbitrary choice.This kind of exploration contributes to the knowledge base and improves channelknowledge Improved channel knowledge in turn will help us to choose the param-eter set with the highest average data rate and therefore minimize the time taken

to transfer the file The state of the existing knowledge base and the size of thefile remaining to be transferred are key factors in deciding between exploring andexploiting

Assume we have k link-coding schemes to choose from in order to transmit.The knowledge gained through iterative attempts on different schemes is given by

Xi(t) at any given time t

Xi = (XiN XiS)

Chapter 2 Background and Related Work 11

Xi = Current state of knowledge about scheme i

XiN = The number of trials made on scheme i

XiS = The number of successes achieved on scheme i

Control action Ui(t) for any given scheme is considered to be 1, when the sponding scheme is chosen at time t, and 0 otherwise, for i = 1, 2, , k U (t) is

corre-a 1 × k vector where Ui(t)  {0, 1}, and

kX

Chapter 2 Background and Related Work 12

A scheduling policy γ = (γ1, γ2, ) is a decision rule such that at each timeinstant t, the control action U (t) takes in values according to

Jγ = E

 kX

i=1

Ri(XiN(t), Ui(t)) | Z (0)



(2.3)

where Z (0) denotes the initial state

Many solution strategies have been proposed to Multi-armed bandit problems Aspointed out by Bellman in [15], dynamic programming is one such method, butits main drawback is the computational complexity Gittins Index, on the otherhand provided a solution which involved much less computations and therefore iswidely accepted as an optimal solution strategy for multi-armed bandit problems[16], [17], [13], [18]

Chapter 2 Background and Related Work 13For each bandit process one can compute a Gittins Index, which depends only

on that process, and then at each time the controller operates on the bandit processwith the highest index Thus, finding an optimal scheduling policy, which origi-nally requires the solution of a k-armed bandit problem, reduces to determiningthe indices for k single-armed bandit problems, thereby reducing the complexity

of the problem exponentially

For cases where the probability distribution of the arms of the slot machinesare independent from each other and only one arm may be attempted at a time,the problem is called multi-armed bandit and the Gittins index policy is proven

to provide an optimal solution [18]

There are many varieties of multi-armed bandit problems studied in literature.Some of them are infinite horizon problems where the user is given infinitely manyattempts [16] Some others have discussed trials with deadlines which address finitehorizon multi-armed bandit problems [18] Another school of literature [19],[20]talks about restless bandit problems, where the states of non-played arms canalso evolve over time, while some focused on rested bandit problems [21] Ourproblem is a rested multi-armed bandit problem because the distributions of thearms remain the same as time goes on It is of finite horizon since we only have afinite file size whereby the number of attempts given to transmit the file is limited.Thus we have limited our research to Gittins Index computation on finite horizon,rested multi-armed bandits

Although Gittins Index provided an optimal solution for multi-armed banditproblem, it has its drawbacks when applied to a channel model like ours The

Chapter 2 Background and Related Work 14channel model, which will be explained in detail in section 3.2, heavily relies onits structure Gittins Index fails to address the separation between FEC codingschemes and link schemes in the channel model, therefore resulting in an inefficientsearch.

We will be looking at more solution strategies in subsequent chapters of thisthesis

As discussed in the introduction, the problem is to transfer a finite sized file

in the minimum possible time using an underwater acoustic link We assume thatthe channel remains static over the course of the file transfer The file will betransferred over multiple packet bursts At every transmission, we may choose totune the modem parameter values Since we do not have any prior information

15

Chapter 3 Problem Formulation 16regarding the error rates for various link-coding scheme combinations, we will have

to rely on the packet loss measurements for each attempt Thus we could find thecorresponding average data rates, using Equation 3.1

Average data rate = Total file bits successfully received per second

= Link Rate × (1 − error rate) × Packet Efficiency factor

(3.1)

where

Packet Efficiency factor = Code Rate × Packet Length

Packet Length + Header Length

For subsequent file transfers, we need to choose between exploring new parametervalues versus exploiting the parameter values that have yielded the best results sofar The objective is to devise a strategy to balance this exploration and exploita-tion so that the file transfer time is minimized

1 Choice of link schemes, FEC codes, and payload sizes:

Let the size of the file to be transferred be F We define the n-tuple consisting

of all tunable parameters of a communication system excluding FEC codingand packet size as a link scheme Let the communication modem have mlink schemes available, each of them resulting in an uncoded link rate of di

Chapter 3 Problem Formulation 17bits per second, i ∈ Z+, i ≤ m Let c ∈ C denote all available code ratesand let l ∈ L denote all available packet lengths Any code can be combinedwith any other packet length subject to the constraint l/c ≤ M , where Mdenotes the maximum length of a coded packet supported by the modem.

We define a set A = C × L containing all feasible combinations of codes andpacket lengths The set A is indexed by j ∈ Z+, j ≤ n Let the packetheader size be h bits Let γj be the packet efficiency factor to account forthe effect of the code rate and the overhead of the packet header Usingequation 3.2, γj could be defined as:

chan-3 Sequences and bursts:

It is possible to change parameter values used for every single packet mission However in practice, the process of tuning often involves a hand-shaking based protocol The transmitter will need to initiate the process by

trans-Chapter 3 Problem Formulation 18specifying the parameter values and wait for an acknowledgement Let thetime taken by the protocol to change parameter values be tsw Due to thetime overhead tsw imposed by the protocol, for efficiency, we may wish totransfer the file in packet bursts [22] Let the time taken to wait for acknowl-edgement for each packet burst be tp Acknowledgements from larger burstsalso allow us to estimate the PER more accurately by measuring the fraction

of transmitted packets that are received successfully Let btx be the size of

a packet burst The file transfer will involve the transmission of a sequence

of bursts Let S = {(SI

u, SJ

u)∀u} denote such a sequence where i = SI

u isthe link scheme and j = SJ

u is the coding scheme selected at burst u Forbrevity, we define the notation i(u) to mean SuI and j(u) to mean SuJ Theamount of data transferred successfully fxfer

Chapter 3 Problem Formulation 19where ν ∈ {0, 1}; ν = 1, when the chosen link-coding scheme was switchedand ν = 0 if otherwise.

5 Key Performance Indicators:

Average throughput is defined as the number of payload bits per secondcorrectly received Average throughput, is a key measure of quality of service(QoS) for data transmission systems which we can use in order to assess theperformance of the various strategies

Throughput = F

Chapter 3 Problem Formulation 20

By substituting equations (3.5),(3.6),(3.7), in equation (3.8), TFS can be

utu

=

P

u(brxu × cj(u)× lj(u))P

Chapter 3 Problem Formulation 21

Figure 3.1: PER vs BER for lj = 8000 bits and h = 2000 bits

one coding scheme may have 0 < P < 1 This imposes a special structure to thechannel matrix We can then define a transition point wi, for link scheme i suchthat:

We note that for a given link scheme, the coding scheme used must be either

j = wi or j = wi+1 for maximum data rate For j < wi, P = 1, and no packet isreceived For j > wi, P = 0, but the data rate is sub-optimal The understandingand the terms introduced in this chapter on the channel model will be used in thesubsequent chapters

Chapter 3 Problem Formulation 22

Figure 3.2: Channel Model

Chapter 4

Preliminary Solution Strategies

The underlying characteristics of the channel model introduced in Chapter 3 wasused to analyze and compare the performance of different strategies This chapterstarts with introducing the basic strategies such as Brute Force, First Hit andRandom Selection, and then moves on to more advanced strategies such as -Greedy Then we have introduced how the knowledge about the underlying channelmodel has been used in order to enhance the performance of these strategies

Different approaches were studied in order to solve the problem We started withthe most simple and straightforward strategies such as Random Selection, BruteForce and First Hit After studying the behaviour of these, we then proceeded tomore sophisticated strategies such as -Greedy

23

Chapter 4 Preliminary Solution Strategies 24The brute force strategy systematically tries all parameter values, records theresulting data rate, and selects the one with the best recorded data rate to transferthe rest of the file This approach entails a comprehensive exploration of theparameter search space, and makes an educated choice based on the results ofthe exploration An alternative to brute force searching is the first hit strategy.The algorithm randomly selects parameter values till it finds one with an averagedata rate greater than a certain threshold, and uses it to transfer the rest ofthe file Yet another strategy is to simply randomly select parameter values atevery transmission The -greedy strategy consists of choosing random parametervalues with -frequency, and otherwise choosing parameter values with the highestestimated mean, the estimation being based on the rewards observed thus far[23],[24].

Although we assume a static channel, in reality the channel will only be static, i.e it stays static for a short period of time and then changes state Theresults of a brute force search may not be useful if the channel changes statebefore the search terminates The brute force or first hit strategies explore at thebeginning, and then exploit hence they are unable to track any changes in thechannel during the exploitation phase

In this strategy, all link-coding scheme combinations are systematically tried gorithm 4.1) At every transmission, a data burst of size btxpackets is transmitted

(Al-Chapter 4 Preliminary Solution Strategies 25The brute force average data rate matrix DBF is calculated as:

Di(u),j(u)BF = di(u)γj(u)b

rx u

Therefore file size F could be expressed using DBF

i(u),j(u) as follows

F =

i(u)≤m,j(u)≤nX

i(u)=1,j(u)=1

Di(u),j(u)BF × tu+ kDˆBFi(u)ˆj(u)× tu (4.2)

where ˆi, ˆj correspond to arg max DBF

i(u),j(u), k represent the number of attemptsmade on ˆi(u), ˆj(u) and tu is the time taken to transfer each burst u, as given inequation (3.5)

However, if mn is large and/or the file size is small, then the time taken toexplore the parameter space mn may be the primary contributor to the total filetransfer time, resulting in a low average data rate

The throughput (TBF) for brute force strategy can be analytically computed asfollows The u literal in i(u), j(u) etc in Equation (4.2) above have been droppedfor clarity of computation in Equation (4.3) although it still stands for the i, j

Chapter 4 Preliminary Solution Strategies 26chosen for burst u.

+ k ×

(1 − Eˆiˆchj) × btx× cˆj × lˆj



i≤n,j≤mX

i=1,j=1

(1 − Eijch) × btx× cj× lj

Figure 4.1: Throughput of Brute Force Strategy - Analytical Results vs

Sim-ulation Results for file size=500Mb, burst size=100

average throughput when simulated with different data sets, along with their spective analytical counterparts is shown in Figure 4.1

re-Chapter 4 Preliminary Solution Strategies 27

Algorithm 4.1 Brute Force Strategy

Require: Initial file size F , btx, h

Probability of success, Ps(i, j) := bbrxtx

Remaining file size, F := F − brx× cj × lj

Di,j := di× cj × Ps(i, j) × lj

l j +hTotal transmit time, T = T + tu

end if

end for

end for

while F > 0 do

transmit btx bursts using arg max Di,j parameters

measure successfully received bursts(brx)

transfer time, tu := btx×(lj +h)

d i

T := T + tu

end while

Chapter 4 Preliminary Solution Strategies 28

In this strategy, the algorithm randomly selects link-code scheme combinations till

it comes across one with a data rate greater than a threshold data rate or until itexhausts all mn link-code schemes, after which the best combination is used forthe rest of the file transfer (Algorithm 4.2) In this algorithm and the following,

we have assumed the existence of a function, rand(i, j) which returns a randominteger between the integers i and j, inclusive

Algorithm 4.2 First Hit Strategy

Require: Initial File size F , Threshold T hr, btx, h

i = rand(1, m)

j = rand(1, n)

transmit btx bursts using (i, j) parameters

measure successfully received bursts, brx

transfer time, tu := btx×(lj +h)

d i

Probability of success, Ps(i, j) := bbrxtx

Remaining file size, F := F − brx× cj × lj

Di,j := di× cj× Ps(i, j) × lj

l j +hwhile Di,j ≤ T hr AND F > 0 do

Total transmit time, T = T + tu

i = rand(1, m)

j = rand(1, n)

transmit btx bursts using (i, j) parameters

measure successfully received bursts, brx

ichosen = i, jchosen = j

while F > 0 do

transmit btx bursts using arg max Di,j parameters

measure successfully received bursts, brx

transfer time, tu := btx×(lj +h)

d i

T = T + tu

end while