A network centric approach to enhancing the interactivity of large scale distributed virtual environments

Keywords: Distributed Virtual Environments; Geographically distributed server architecture; Server placement; Client assignment; Interactivity enhancement 1.. Server placement problem A

Elsevier, and the attached copy is provided by Elsevier for the author’s benefit and for the benefit of the author’s institution, for non-commercial research and educational use including without limitation use in instruction at your institution, sending it to specific colleagues that you know, and providing a copy to your institution’s

administrator.

All other uses, reproduction and distribution, including without limitation commercial reprints, selling or licensing copies or access,

or posting on open internet sites, your personal or institution’s website or repository, are prohibited For exceptions, permission may be sought for such use through Elsevier’s permissions site at:

http://www.elsevier.com/locate/permissionusematerial

Author's personal copy

A network-centric approach to enhancing the interactivity of large-scale

distributed virtual environments

Duong Nguyen Binh Ta *, Suiping Zhou

School of Computer Engineering, Nanyang Technological University, Singapore 639798, Singapore

Received 25 December 2005; received in revised form 31 May 2006; accepted 31 May 2006

Available online 27 June 2006

Abstract

In Distributed Virtual Environments (DVEs), where many simultaneous human users interact with each other in a shared, 3D virtual world, enhancing interactivity is of crucial importance, since the success of a DVE greatly depends on users’ perceptions In general, enhancing interactivity for large-scale DVEs with thousands of geographically distributed users faces various challenges such as large Internet latencies, high resource demands and high potential of security threats In this paper, we propose a network-centric approach

to enhance the interactivity of DVEs by directly minimizing the network latencies in client–server communications We consider two key problems: the server placement problem and the client assignment problem We have formulated these two problems and proved that they are both NP-hard We then adapt and develop several degree-based server placement approaches and greedy client assignment algo-rithms Extensive simulation studies using realistic models have shown that our approach is very effective in enhancing the interactivity of large-scale DVEs

2006 Elsevier B.V All rights reserved

Keywords: Distributed Virtual Environments; Geographically distributed server architecture; Server placement; Client assignment; Interactivity

enhancement

1 Introduction

In recent years, advances in networking technologies,

computer graphic and CPU power have popularized the

deployments of Distributed Virtual Environments (DVEs)

Large-scale DVEs refer to a class of applications that

enable thousands of geographically distributed users1 to

simultaneously interact with each other in a shared,

com-puter-generated 3D virtual world, where each client is

rep-resented by an avatar[30] A client controls the behavior of

his/her avatar by various inputs, and the updates of an

ava-tar’s state need to be sent to other clients in the same zone

of the virtual world to support the interactions among

clients Prominent applications of DVEs include online games, military simulations, collaborative designs, virtual shopping mall, etc The latest study by the market research company IDC (http://www.idc.com) on the online gaming market of Asia/Pacific (excluding Japan) revealed that the subscription revenue is about US$1.09 billion in 2004, an increase at about 30% compared to 2003 Also according

to this study, this market will be more than doubled in 2009

Enhancing the interactivity of DVEs is of crucial

impor-tance, since the users may not feel that they are interacting

with other elements in the virtual world if the responses from the DVE are much slower than what the users experience in real-life However, in general, seeking a good balance between enhancing interactivity and other important issues in DVEs such as maintaining consistency, managing resource bottlenecks and securing the virtual world system under various technical problems such as large Internet latency, high resource demands and high

0140-3664/$ - see front matter  2006 Elsevier B.V All rights reserved.

doi:10.1016/j.comcom.2006.05.015

* Corresponding author Tel.: +6591186750; fax: +6567926559.

E-mail addresses: binhduong@pmail.ntu.edu.sg (D N B Ta),

1 In this paper, the terms ‘‘user’’, ‘‘participant’’ and ‘‘client’’ all refer to

DVE users and are used interchangeably.

www.elsevier.com/locate/comcom

Author's personal copy

potential of security threats is a very challenging task In

this paper, we have proposed a network-centric approach

to enhancing the interactivity of large-scale DVEs Our

methodology is different from the traditional

application-centric approaches, e.g., dead reckoning, which aims to

‘‘hide’’ the effect of network latencies by predicting the

state of remote users at the application level Since the

activities of human users in DVEs are highly unpredictable,

such approach may result in visual perception errors, or in

more serious cases, inconsistent DVE states[34]

Our network-centric approach is based on a

geographi-cally distributed server architecture (GDSA) [21,9], in

which multiple geographically distributed servers are

con-nected to each other Each client is concon-nected to one of

these servers, and clients interact with each other through

these servers This server-based architecture essentially

pro-vides good ways to implement consistency control,

resource management and security mechanisms In

addi-tion, in order to deal with large-scale DVEs with hundreds,

or even thousands of clients interacting simultaneously,

usually the virtual world is spatially partitioned into many

distinct zones, with each zone managed by only one server,

as in [3] A client only interacts with other clients in the

same zone,2and may move to other zones As a server only

needs to handle one or more zones instead of the entire

vir-tual world, the system is more scalable In this paper, we

refer to such a partitioning approach as the zone-based

approach.

In this paper, we consider two key problems with the

GDSA and the zone-based partitioning approach: the

serv-er placement problem and the client assignment problem.

Both problems aim to reduce the client–server

communica-tion delays, but the former achieves this goal by finding

good locations to place the servers, while the latter does

so by assigning clients to appropriate servers We have

for-mulated the two problems and proved that they are both

NP-hard To address these two problems, we have

devel-oped and adapted several server placement approaches

and some client assignment algorithms Extensive

simula-tion studies have shown that our approach is very effective

in enhancing the interactivity of large-scale DVEs

The rest of the paper is organized as follows Section2

introduces and addresses the server placement problem in

DVEs The client assignment problem and algorithms are

presented in Section 3 Section 4 discusses the relations

between server placement and client assignment

Simula-tion study is described in SecSimula-tion 5, and Section 6

con-cludes the paper

2 Server placement problem

A key question in the GDSA is where to place the

serv-ers to reduce the communication delay between the clients

and the servers This problem has been studied extensively

in the context of Web replica placement[10,27,24]

Howev-er, to our knowledge there is no existing work that assesses the suitability of Web replica placement approaches to the server placement problem in DVEs In this paper, we look

at the state-of-the-art results in the Web replica placement problem We then discuss the difference between the server placement problem in DVEs and the Web replica place-ment problem Finally, we suggest some possible

approach-es adapted from Web replica placement strategiapproach-es that we believe appropriate to the server placement problem in DVEs

2.1 Web replica placement problem

The explosive growth of the World Wide Web in the late 1990’s had demanded efficient content delivery architec-tures to offer better service to web users at lower costs

A Content Distribution Network (CDN) is a system of many (hundreds or thousands) well-connected servers deployed over the Internet to transparently deliver web contents to end users by allocating the replicas of contents

at the edges of the Internet, i.e., close to the users Hence,

in large CDNs like Akamai[1], the Web replica placement

is crucial to the system performance

Informally stated, the Web replica placement problem

concerns how to choose K web replicas among N potential sites (i.e., network locations) in the network, where K < N,

to minimize a given objective function, usually the clients’ access latencies, or the total bandwidth consumption After the replicas are in place, each web client connects to its closest replica to retrieve the contents needed Here, it is assumed that each client needs to access only one replica

to get all of its interested contents The Web replica place-ment is usually formulated as a minimum K-median prob-lem, which is a well-known NP-hard problem In[27], the authors proposed a greedy placement algorithm that is shown to be able to perform very well compared to a

‘‘super-optimal’’ algorithm3 that uses Lagrangian relaxation with subgradient optimization The authors showed in [27] that the median performance in terms of client–server communication delays of the greedy algorithm is within a factor of 1.1–1.5 of the optimal

2.2 Server placement problem in DVEs

Although the Web replica placement problem has been well studied in the context of CDNs with promising results, the appropriateness of the proposed approaches to the server placement problem in DVEs has not yet been stud-ied The main difference between the Web replica place-ment problem and the server placeplace-ment problem in DVEs lies in the ‘‘connection’’ of clients to servers In the

2 For simplicity, we say that a client is in a zone if its avatar is currently

residing in that zone.

3 The ‘‘super-optimal’’ solution may be better than the optimal, but may not be feasible.

Author's personal copy

first problem, it is relatively straightforward to let web

cli-ents connect to their closest replicas, hence the greedy

placement approach is feasible However, in DVEs, the

large virtual world is partitioned into multiple distinct

zones, and each zone is hosted by a separate server A client

cannot simply connect to its closest server since that server

may not manage its zone Hence, the greedy placement

approach is not appropriate for the server placement

prob-lem in DVEs, as it assumes that clients always connect to

their closest servers

So, for the server placement problem in DVEs, we need

an approach that can place the servers in appropriate

net-work locations to reduce clients’ communication latencies

without knowing in advance which server the clients will

connect to In[10,24], the authors have shown that by

plac-ing Web replicas at some key network locations, i.e., at the

network nodes (routers or Autonomous Systems (ASes))

with high node degrees, the clients’ communication

laten-cies can be significantly reduced, regardless of the clients’s

locations in the network We feel that such approaches are

applicable to the server placement problem in DVEs

How-ever, the mechanisms in[24,10]were only evaluated under

the assumption that clients would eventually connect to

their closest servers, although this assumption was not used

in deciding server placements Hence, in this paper, we

adapt the placement approaches in [24,10] to address the

server placement problem in DVEs, and evaluate the

appropriateness of these approaches in DVEs To our

knowledge, our work is the first to consider server

place-ments to enhance the interactivity of DVEs In the

follow-ing section, we suggest several possible server placement

approaches

2.3 Server placement approaches for DVEs

2.3.1 Random placement

The random placement serves as a basic approach for

comparison in the sense that any good server placement

approach should perform better than the random

placement In this approach, servers are randomly placed

in the network

Remark 2.1 The computational cost of the random placement approach is O (Mm), where M is the total number of routers, and m is the number of servers

2.3.2 Degree-based placement

The degree-based server placement approaches aim to place the servers at some key locations in the network These key locations are usually the network nodes (routers

or ASes) with high node degrees The degree-based approaches have been shown to perform comparably to the greedy server placement algorithm in[27]in the context

of the Web replica placement problem [24] However, whether they are appropriate for DVEs needs to be further investigated

2.3.3 Max-router

In the Max-router placement approach, we sort all the routers in the network in descending order of their node degrees using Quicksort, and place servers at the routers with the highest node degrees Intuitively, a node with

larg-er node degree is likely to be closlarg-er to othlarg-er nodes, hence it

is a good choice for server placement For example in

If a server is placed there, then the network distances (in number of hops4) from client c1and c2to the server are 2 and 1, respectively However, if the server is placed at

rout-er A, which has a node degree of 1, then the above client–

server distances become 3 and 2, respectively Thus, placing servers at nodes with large node degree may effectively reduce the client–server communication delay, which implies good interactivity

Fig 1 Max-router placement approach.

4 Indeed, network hop counts are good indications of round-trip network delays [23]

Author's personal copy

Remark 2.2 The computational cost of the Max-router

placement approach is O (M logM) + O (m), or

O (M logM), where M is the total number of routers in

the network topology, and m is the number of servers,

m > M

2.3.4 Max-AS/Max-router

The Max-AS/Max-router placement approach is similar

to the Max-router approach in the sense that they both aim

to place servers at nodes with large degree in the network

topology The main difference between the two approaches

is that Max-AS/Max-router uses a hierarchical placement

approach which is based on the structural nature of real

network topologies

First, Max-AS/Max-router approach selects the ASes

with the largest AS-level node degrees In each AS, we

place one server at the router with the largest node degree

By spreading servers over multiple ASes, this approach

may further reduce network latencies for geographically

distributed clients compared to the Max-router approach

To implement the Max-AS/Max-router approach, we

sort all the routers according to the node degrees of their

corresponding ASes (each router belongs to an AS) If

there are two routers with the same AS-level node degree,

we further consider their router-level node degrees We

then iterate over the sorted list of all routers to select one

best router in each AS to place the server

Remark 2.3 The computational cost of the

Max-AS/Max-router placement approach is O (M logM) + O (M), or

O (M logM), where M is the total number of routers

2.3.5 Other placement approaches

As comparative references, we also consider some

alter-native server placement approaches, namely Min-router,

Max-AS/router and AS/Max-router The

Min-router approach, which place the servers at the nodes with

the minimum node degrees, may serve as an ‘‘upper

bound’’ on how bad a server placement approach can

be The Max-AS/Min-router5 and Min-AS/Max-router6

may help to identify which (AS or router) is the major

factor in the hierarchical server placement approach

However, note that in practice, sometimes we have to

adopt these methods for server placement since typically

the routers with high node degrees may be very busy

due to the high volumes of network traffic, thus it may

be impossible to add any more service to them[24]

More-over, some economic and administrative constraints may

make it harder to place servers at the selected network

locations

3 Client assignment problem

In CDNs, after the web replica placement is done, assigning clients to servers is relatively straightforward, i.e., each client is connected to its closest server However,

in the context of DVEs with the zone-based partitioning approach, this simple client assignment approach is not appropriate, since a client may interact in a zone which is not hosted by the client’s closest server Hence, in this sec-tion, independent of any specific server placement, we

for-mulate a new problem, termed the client assignment

problem (CAP) This problem essentially concerns how to

assign clients to distributed servers to reduce the client– server communication delays

3.1 Definitions

Before formulating the CAP, we introduce the following notations and concepts:

•c i– A client in the DVE

•C = {c1, , c k} – The set that consists of all clients in the DVE

•z i– A zone in the DVE This is also used to denote a set that consists of all clients in a zone

•Z = {z1, , z n} – The set that consists of all zones in the DVE

•s i– A server in the DVE

•S = {s1, , s m} – The set that consists of all servers in the DVE

•Rsi – The resource consumption on a server s i This can

be measured by CPU usage, network bandwidth usage, etc Since the network bandwidth often represents the major operating cost in current server-based online games [18], in this paper, we assume that the server CPU is not a bottleneck, and measure the resource con-sumption by the network bandwidth usage

•Rzi – The total amount of resource, i.e., network band-width, used by all the clients that are interacting in zone

z i on the server which is hosting z i

•Csi – The resource capacity of a server s i

•dcisj – The round-trip network delay between a client c i

and a server s j

•D – The delay bound of a DVE The delay bound

indi-cates the required upper bound of the round-trip com-munication delay between a client and its server to guarantee the interactivity of the DVE For different types of DVEs, there are different delay bound require-ments For example, First Person Shooter (FPS) games typically require a delay bound of 250 ms[16], while car-racing games have much more stringent latency require-ment, at about 100 ms[25] It should be noted that the communication delay between a client and its server is different from the network delay between the client and its server The communication delay is the sum of the network delay and the processing delay at the server However, in this paper we assume that the server CPU is

5 This approach is similar to Max-AS/Max-router, but in each AS, the

router with minimum node degree is selected to place the server for that

AS.

6 This approach is similar to Max-AS/Max-router, but we select the

ASes with minimum node degrees, and in each selected AS we select the

router with maximum node degree to place the server.

Author's personal copy

not a bottleneck, thus the client–server communication

delay is determined by the client–server network delay

In this paper, the terms ‘‘network delay’’ and

‘‘commu-nication delay’’ are used interchangeably

For interactive applications like DVEs, the client–server

communication delay is the most important Quality of

Ser-vice (QoS) parameter that the system provides to clients

[16] In this paper, we say that a client is with QoS or

with-out QoS if the communication delay between the client and

its server is smaller or larger than the delay bound,

respectively

To illustrate the client assignment problem, we consider

the example in Fig 2 Assuming that dc 1 s 1 ¼ 50

ms; dc 1 s 4 ¼ 80 ms; dc 2 s 1 ¼ 150 ms; dc 2 s 4¼ 50 ms; dc 4 s 1¼

200 ms; dc 4 s 4 ¼ 50 ms and the delay bound of this DVE is

100 ms, then it would be better for the two clients c2and c4

which are interacting in the zone z1of the virtual world if z1

is hosted by server s4instead of s1 For client c1, her

expe-rience would not be damaged when zone z1is hosted by s4

since dc 1 s 4<100 ms From this example, it is obvious that

appropriate assignment of clients to servers7is crucial to

enhance the interactivity of DVEs

3.2 Formulation

With the geographically distributed server architecture

and the zone-based partitioning approach, the CAP

con-cerns how to assign each zone (with all of its clients) of

the virtual world to an appropriate server so that the total

number of clients with QoS in the system is maximized To

formulate this problem, we first propose the following

met-ric to measure the ‘‘cost’’ of assigning a zone z j to a server s i

Cij¼ jfck 2 zj: dc k s i >Dgj ð1Þ

where j Æ j denotes the cardinality of a set

C ij measures the number of clients in a zone z j that do

not satisfy the delay bound D, i.e., without QoS Therefore,

by minimizing the total cost of the assignment, the total

number of clients with QoS in the DVE would be maxi-mized The client assignment problem is formulated as follows

Definition 3.1 Client assignment problem (CAP) Let I = {1, , m} and J = {1, , n} be the set of indexes of servers and zones in the system, respectively For each i 2 I and j 2 J, given the cost Cijof assigning zone zjto server si as defined in Eq (1), find an assignment matrix

X = (xij), with xij= 1 if zone zj is assigned to server si or

xij= 0 otherwise, which minimizes the total cost CðX Þ ¼Xm

i¼1

Xn j¼1

Cijxij ð2Þ

subject to

Xn j¼1

Rz jxij 6Cs i; 8i 2 I; ð3Þ

Xm i¼1

xij¼ 1; 8j 2 J ; ð4Þ

xij2 f0; 1g; 8i 2 I; 8j 2 J : ð5Þ Remark 3.1 In Definition 3.1, constraint (3) ensures that the capacity of each server will not be exceeded Constraint

(4) means that each zone is assigned to only one server Theorem 3.1 The client assignment problem (CAP) as described in Definition 3.1is NP-hard

Proof We consider in our model the special case where the resource capacity Csi of each server is the same We further assume that the cost Cij when assigning a zone zj,j 2

J = {1, , n} to server si,i 2 I = {1, , m} has the special form Di  1, where D>1 is a constant, i.e assigning any zone

to server s1costs C1j= 1, to server s2costs C2j= D, to

serv-er s3costs C3j= D2, etc Then the optimization goal of CAP

in this specific case would be to assign all the zones to the smallest possible number M of servers, with server indexes range from 1 to M This is exactly the well-known Bin Packing problem[17], in which a bin corresponds to a

serv-er and an item corresponds to a zone Since the CAP gen-eralizes the Bin Packing problem, which is NP-hard, CAP

is also NP-hard h

3.3 Related work

To the best of our knowledge, there is no existing work that directly addresses the client assignment problem in DVEs as we have described Research on how to assign cli-ents to servers in DVEs is usually formulated as a load dis-tribution problem in a locally distributed server architecture, i.e., all the servers are placed in the same machine room [31,20] Such approaches may damage the interactivity of the DVE, since clients may be far away (in terms of network delays) from the servers

Fig 2 Geographically distributed server architecture.

7 Recall that the assignment of clients to servers is determined by the

assignment of zones to servers.

Author's personal copy

Recently, the interest in investigating server and

net-work architectures to reduce the effect of netnet-work latency

for DVEs has been increased In[22,29], the authors

intro-duced the concept of latency driven distribution (LDD) for

the distribution of a DVE over the networking

architec-ture, as opposed to the traditional resource driven

distribu-tion (RDD), i.e., load distribudistribu-tion Our client assignment

problem shares the idea of the LDD concept However,

the authors of [22,29] only investigate the provision of

immersive audio communication in DVEs within the

con-text of the LDD concept In their work, two clients that

belong to two adjacent DVE zones are able to hear each

other However, several important kinds of interactions

that may happen in a DVE such as shooting enemies,

manipulating objects, changing object ownership, etc.,

were not studied These interactions may raise complicated

consistency issues, if the clients in adjacent zones are

man-aged by different physical server machines

3.4 Assignment algorithms

Since the CAP is NP-hard, we seek some heuristic

solu-tions instead of the optimal one We propose three

algo-rithms for the CAP The first one randomly assigns the

zones, while the second one is a greedy heuristics to

mini-mize the number of clients without QoS in the system

The third algorithm is also a greedy heuristics similar to

the second one, except that it uses a different cost metric

3.4.1 Random assignment (Random)

In the random assignment algorithm, zones are assigned

to randomly selected servers with the only concern of not

to overload the servers In this algorithm, the following

procedure is repeated until all zones have been assigned:

first we select a random zone z j, and then a random server

s i with sufficient capacity is selected to take z j, i.e., the

tar-get server of all clients in z j is set to s i

Remark 3.2 The computational cost of the Random

algorithm is O (mn), where n is the number of zones and

m is the number of servers

3.4.2 Greedy assignment – algorithm 1 (Greedy-1)

Since the random assignment algorithm is oblivious to

client–server network delays when assigning zones to

serv-ers, the obtained performance in terms of the number of

clients with QoS may not be good, i.e., the cost of the

assignment as defined in Eq (2) may be high Hence, in

the Greedy-1 algorithm, we use a greedy heuristics to

min-imize the total number of clients without QoS in the system.

The pseudo-code is shown inFig 3 Let lij = C ijbe a

heuristic measure of the desirability of assigning zone z jto

server s i Thus, the smaller the cost C ij is, the higher the

desirability lij is The algorithm iteratively considers all

the unassigned zones and pick a zone z jwith the maximum

difference qjbetween the largest desirability lijjand the

sec-ond largest desirability l Then, z is assigned to a server s

with the highest value of lij and with sufficient resource capacity This procedure is adapted from the well-known approach used to solve the Generalized Assignment Prob-lem[28]

Let us illustrate the effectiveness of the Greedy-1 algo-rithm using an example Assuming that the DVE has two

zones z1 and z2, and two servers s1 and s2 Each server can only take one zone, due to its capacity constraint

Let us further assume that assigning z1 to s1 or s2 costs

C11= 10 or C21= 20, respectively Similarly, assigning z2

to s1or s2costs C12= 5 or C22= 10, respectively

The desirability difference q1 of zone z1 is equal to

l11 l21= C11 (C21) =  10  (20) = 10, while

q2 of z2 is equal to l12 l22= C12 (C22) =

5  (10) = 5 Hence, the Greedy-1 algorithm will select

z1to assign first As a result, z1is assigned to server s1, and

z2is assigned to server s2, since each server can only take one zone The total cost of this assignment is

10 + 10 = 20, i.e., in total there are 20 clients that are with-out QoS in the system

However, if we do not follow the procedure of

Greedy-1, then z2may be chosen to assign first In this case, z2will

be assigned to s1, thus z1has to be assigned to s2 The total cost in this case would be 20 + 5 = 25, which is larger than the cost obtained using Greedy-1 Thus, by sorting the zones according to their desirability differences q, Greedy-1 can effectively reduce the number of clients with-out QoS in the system

We analyze the computational cost of the Greedy-1

algorithm Let m, n and k denote the number of servers,

the number of zones and the total number of clients, respectively The Greedy-1 algorithm consists of four parts The first part (from line 2 to line 8) is to find all the values

of lij The cost of this part is O (mk) The second part (from

line 9 to line 12) is to find qj for all z j 2 Z In the line 10,

Quicksort is used Hence, the cost of this part is

O (n)[O (m logm) + O (1)], or O (mn logm).

The third part (line 13) is to sort the list of n values of q j

Using Quicksort, the cost is O (n logn) The final part (from line 14 to line 27) is to assign zones to servers There are m servers and n zones, hence the cost of this part is O (nm).

Hence, the overall computational cost of the Greedy-1

algorithm is O (mk) + O (mn logm) + O (n logn) + O (nm),

or O (max{O (mk), O (mn logm), O (n logn)}).

Remark 3.3 The complexity of the Greedy-1 algorithm is

O (max{O (mk), O (mn logm), O (n logn)}), where n is the number of zones, m is the number of servers and k is the number of clients in the system In practice, usually k  m and k  n, hence the overall complexity of Greedy-1 for most cases can be written as O (mk)

3.4.3 Greedy assignment – algorithm 2 (Greedy-2)

The only difference between Greedy-1 and Greedy-2 lies

in the cost metric they use With the cost metric in Eq.(1), Greedy-1 aims to minimize the total number of clients without QoS in the system directly For the Greedy-2

Author's personal copy

algorithm, we propose to use the following metric to

mea-sure the cost when assigning a zone z j to a server s i

C0ij¼

P

ck2z j

dcksi

jzjj ð6Þ

C0

ij measures the average delay from all client c k 2 z jto

server s i if they are all assigned to s i By minimizing this

cost function, it is expected that the selected server s i for

z j will be near (in terms of network delay) to the center

of mass of the client population of z i, thus the number of

clients with QoS may become large

3.4.4 Reference algorithm: lp_solve

For comparison, based on the Integer Programming

formulation of the CAP, we use the so-called

‘‘branch-and-bound’’ algorithm implemented in the free Mixed Integer Linear Programming (MILP) solver lp_solve

[4]to obtain the optimal solutions for the CAP Note that this approach is only applicable when the system size is small, otherwise the running time of lp_solve will become very long (on the order of several hours), which is clearly impractical for highly interactive applications like DVEs

3.5 Implementation considerations

In this section we discuss some practical considerations

on the implementation of the proposed assignment

algo-rithms The first consideration is the dynamic property of

DVEs During the course of interactions in the virtual world, clients may move from one zone to another, new cli-ents may join, existing clicli-ents may also leave the virtual

Fig 3 Greedy assignment, algorithm 1.

Author's personal copy

world An obtained client assignment may not be good

after some time Thus, the proposed algorithms need to

be executed from time to time to ensure good client

assignments

We should note that dynamic execution of the client

assignment algorithms may incur inter-server bandwidth

consumption due to the migrations of zones and clients

across servers Nevertheless, we believe this cost is much

smaller compared to the bandwidth consumption due to

the interactions of users in the DVE This suggests that

the bandwidth cost of re-executing the assignment

algo-rithms is not a big concern Indeed, a similar assumption

is used in the context of the Web replica placement

prob-lem[27] Moreover, in a recent study[19]of the very

pop-ular MMOG Lineage II developed by NCsoft, Korea, the

authors found that the average value of session durations,

i.e., the average time a player keeps on playing the game, is

about 3 h, which is rather long This may indicate that we

do not have to execute the assignment algorithms

frequently

Another issue is how to obtain the input data for the

assignment algorithms The input data includes the

cli-ent–server round-trip network delays and the resource

requirement of each client on a server The network delays

can be obtained using some scalable network measurement

tools such as King[14]or IDMaps[13] King uses existing

recursive DNS queries to accurately estimate the

round-trip network delays between arbitrary Internet end hosts,

while IDMaps relies on special end hosts (referred to as

the tracers) deployed at some strategic locations in the

Internet Both approaches are scalable and incur little

esti-mation overhead

In this paper, the server resource requirement of each

client is measured as the bandwidth requirement It is well

known that the bandwidth requirement in client–server

architectures increases quadratically with the total number

of clients that are interacting with each other[26] Thus, we

can estimate in advance the bandwidth requirement of each

client in a zone based on the number of clients in that zone,

as in[26]

4 Server placement versus client assignment

In Sections2 and3, we have discussed several methods

for interactivity enhancement for two different problems,

namely server placement and client assignment In this

sec-tion, we briefly discuss their relations

First, server placement and client assignment can

com-plement to each other in enhancing the interactivity of

DVEs For example, if all the servers are already placed

at some good network locations, then appropriate client

assignment algorithms can be used to further enhance the

interactivity of the DVE On the other hand, a good server

placement certainly helps to maximize the performance of

client assignment algorithms

Second, we believe that client assignment are more

flexible than server placement in the sense that the former

does not require any special infrastructure to be deployed

As we have already mentioned, placing servers at good net-work locations is not always achievable, due to several technical and economic constraints In such scenarios, effi-cient client assignments are highly desirable to enhance the interactivity of DVEs Hence, to some extent, we may say that client assignments are of greater importance than

serv-er placements, although it would be highly advantageous to combine the effectiveness of both methods

5 Performance evaluation

In this section, we compare various server placement approaches and client assignment algorithms We first study the impacts of server placement and client assign-ment separately Then, their combined impacts are investi-gated More specifically, while studying the impacts of server placement approaches, we use the basic client assign-ment mechanism, i.e., the Random assignassign-ment algorithm

On the other hand, while studying the impacts of client assignment algorithms, the Random server placement approach is used

5.1 Simulation models and parameters 5.1.1 Network models

Both synthetic topologies generated by the popular topology generator BRITE[2]and real Internet topologies are used in the simulations.Table 1lists the topologies used

in this paper

In our simulations, we used both synthetic topologies generated by the popular topology generator BRITE [2]

and real Internet topologies Table 1 lists the topologies used in this paper We use BRITE to generate both flat and hierarchical topologies based on the well-known Wax-man and Barabasi–Albert model (F-W, F-BA, H-BA/W) The Waxman model[33] considers all pairs of nodes, and then decides whether to add a link between any two nodes with a probability that depends on the distance between these two nodes and the longest distance between any two nodes in the network The Barabasi–Albert model[8]

generates network topologies that exhibit power-laws as observed in the seminal paper by Faloutsos et al [11] in 1999

To complement the synthetic topologies generated by BRITE, we use a real, flat Internet topology collected from NLANR[5] For diversity, we also collect a real AS-level

Table 1 Network topologies

Flat, Barabasi–Albert (F-BA) 3000 5997

Hierarchical, Barabasi–Albert/Waxman (H-BA/W) 3000 6197 Hierarchical, real/real (H-R/R) 3300 13442

Author's personal copy

topology (generated by processing the Border Gateway

Protocol (BGP) routing tables) with 110 nodes from [6],

and use the DFN (German Research Network) topology

[15]with 30 nodes as the router-level topology to construct

a realistic hierarchical topology (H-R/R) While these

topologies are not complete, they at least partially reflect

the ‘‘true’’ topology of the Internet, which may have great

impacts on the simulation results

For simplicity, we assume that the round-trip network

delay between any two nodes in the network topology is

proportional to the number of link-hops between them

This assumption is similar to the one used in most of the

previous work [27,24] In fact, a recent Internet

measure-ment [23] also showed that round-trip delay is

well-corre-lated with network hop counts Moreover, to obtain

more realistic simulations results, we use both

shortest-path routing and AS-level hierarchical routing [32]where

possible to calculate the network delays The AS-level

hier-archical routing is a more realistic routing strategy for our

simulations, since to some extent it may reflect the ‘‘true’’

routing practice in the current Internet

5.1.2 Workload models

Based on existing studies of real online game systems,

e.g., [12], we simulated different client distributions

(clus-tered, uniform, etc.) both in the physical and the virtual

world We obtained similar results for all tested network

topologies and client distributions For clarity, we only

present here the most representative results

Different DVE configurations are used for performance

evaluation A specific DVE configuration is determined by

the number of servers, the number of zones, the number of

clients, and the total resource capacity of the system We

use the notation number of servers-number of zones-number

of clients-capacity to denote a DVE configuration For

example, the notation 20s-400z-5000c-2500cp means that

the DVE has 20 servers, 400 zones, 5000 clients and

2500 Mpbs server bandwidth in total

5.1.3 Performance measure

The main performance measure used in the analysis is

the percentage of clients with QoS in the system, denoted

as pQoS Results presented here are obtained by averaging

the results of 50 simulation runs

5.1.4 Default settings

Unless otherwise stated, the following assumptions and default values are used in the simulations The clients are uniformly distributed in the physical world as well as in the virtual world To estimate bandwidth requirement

[26], the input sending frequency of each client (frame rate)

is set to 25 messages/s, and the size of each input or update

is 100 bytes, which are close to real settings[7] The maxi-mum round-trip delay between any two nodes in the net-work topology is set to 300 ms The interactivity

requirement, i.e., the DVE delay bound D, is set to

150 ms The default DVE configuration is 20s-400z-5000c-2500cp

5.2 Impacts of server placement

(CDF) of client–server round-trip communication delays using different server placement approaches and network topologies For flat topologies (Fig 4), we observe that the Max-router placement performs very well: in terms of

pQoS, the performance improvements are 157% (F-W

topology), 168% (F-BA topology) and 26% (F-R topology) compared to the Random placement, and 157%, 275% and 92% compared to the Min-router placement Moreover, from these figures, it is clear that Max-router not only has a high ratio of clients with QoS, but also provide better interactivity for clients that are without QoS

It should be noted that the performance improvement in F-R topology is not as high as that in F-W and F-BA topol-ogies However, this is not because the Max-router place-ment performs poorly in F-R topology As observed in

in F-R topology One possible explanation for this result is that the F-R topology, which is a real-world topology col-lected from [5], has good connectivity, i.e any node can reach any other node with a small number of hop counts Indeed, the average round-trip delay between any two nodes

in F-R is much smaller than those in F-W and F-BA For hierarchical topologies (Fig 5), we obtain similar results to the case of flat topologies, except that the Max-AS/Max-router placement performs the best in this case:

its performance improvements in terms of pQoS over the

Random placement are 38% (H-BA/W) and 48% (H-R/R)

0.9

1

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.9 1

0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

0.9 1

0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

150 175 200 225 250 275 300 150 175 200 225 250 275 300 150 175 200 225 250 275 300

delay (msec) delay (msec) delay (msec)

Random Max-router Min-router

Random Max-router Min-router

Random Max-router Min-router

Fig 4 Impacts of server placement, flat topologies (a) F-W topology (b) F-BA topology (c) F-R topology.