Market-Oriented Grid and Utility Computing pptx

PART IV RESOURCE ALLOCATION AND SCHEDULINGMECHANISMS 16 A Reciprocation-Based Economy for Multiple Services Nazareno Andrade, Francisco Brasileiro, Miranda Mowbray, and Walfredo Cirne 17

GRID AND UTILITY

COMPUTING

MARKET-ORIENTED GRID AND UTILITY COMPUTING

Published by John Wiley & Sons, Inc., Hoboken, New Jersey

Published simultaneously in Canada

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form

or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750 8400, fax (978)

750 4470, or on the web at www.copyright.com Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748 6011, fax (201) 748 6008, or online at http://www.wiley.com/go/permission.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose No warranty may be created or extended by sales representatives or written sales materials The advice and strategies contained herein may not be suitable for your situation You should consult with a professional where appropriate Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited

to special, incidental, consequential, or other damages.

For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762 2974, outside the United States at (317)

572 3993 or fax (317) 572 4002.

Wiley also publishes its books in a variety of electronic formats Some content that appears in print may not be available in electronic formats For more information about Wiley products, visit our web site

at www.wiley.com.

Library of Congress Cataloging-in-Publication Data:

Market oriented grid and utility computing / edited by Rajkumar Buyya, Kris Bubendorfer.

Rajkumar Buyya and Srikumar Venugopal

2 Markets, Mechanisms, Games, and Their Implications

Yibo Sun, Sameer Tilak, Ruppa K Thulasiram,

and Kenneth Chiu

3 Ownership and Decentralization Issues in Resource

Tiberiu Stef Praun

4 Utility Functions, Prices, and Negotiation 67

John Wilkes

5 Options and Commodity Markets for Computing Resources 89

Dan Cristian Marinescu, John Patrick Morrison, and Howard Jay Siegel

PART II BUSINESS MODELS

6 Grid Business Models, Evaluation, and Principles 123

Steve Taylor and Paul McKee

7 Grid Business Models for Brokers Executing

Dang Minh Quan and Jo€rn Altman

8 A Business-Rules-Based Model to Manage Virtual

Organizations in Collaborative Grid Environments 167

Pilar Herrero, Jose Luis Bosque, and Marıa S Perez

9 Accounting as a Requirement for Market-Oriented

Andrea Guarise and Rosario M Piro

PART III POLICIES AND AGREEMENTS

10 Service-Level Agreements (SLAs) in the

Bastian Koller, Eduardo Oliveros, and Alfonso Sanchez Macian

Paul McKee, Steve Taylor, Mike Surridge, and Richard Lowe

12 SLA-Based Resource Management and Allocation 261

Jordi Guitart, Mario Macıas, Omer Rana, Philipp Wieder,

Ramin Yahyapour, and Wolfgang Ziegler

13 Market-Based Resource Allocation for Differentiated Quality

H Howie Huang and Andrew S Grimshaw

14 Specification, Planning, and Execution of QoS-Aware

Ivona Brandic, Sabri Pllana, and Siegfried Benkner

Karim Djemame, James Padgett, Iain Gourlay, Kerstin Voss,

and Odej Kao

PART IV RESOURCE ALLOCATION AND SCHEDULING

MECHANISMS

16 A Reciprocation-Based Economy for Multiple Services

Nazareno Andrade, Francisco Brasileiro, Miranda Mowbray,

and Walfredo Cirne

17 The Nimrod/G Grid Resource Broker for Economics-Based

Rajkumar Buyya and David Abramson

18 Techniques for Providing Hard Quality-of-Service Guarantees

Pavan Balaji, Ponnuswamy Sadayappan, and Mohammad Islam

19 Deadline Budget-Based Scheduling of Workflows on Utility Grids 427

Jia Yu, Kotagiri Ramamohanarao, and Rajkumar Buyya

20 Game-Theoretic Scheduling of Grid Computations 451

Yu Kwong Kwok

21 Cooperative Game-Theory-Based Cost Optimization

Radu Prodan and Rubing Duan

Bj€orn Schnizler

23 Two Auction-Based Resource Allocation Environments:

Alvin AuYoung, Phil Buonadonna, Brent N Chun, Chaki Ng,

David C Parkes, Jeff Shneidman, Alex C Snoeren, and Amin Vahdat

Kris Bubendorfer, Ben Palmer, and Wayne Thomson

25 Using Secure Auctions to Build a Distributed Metascheduler

Kyle Chard and Kris Bubendorfer

26 The Gridbus Middleware for Market-Oriented Computing 589

Rajkumar Buyya, Srikumar Venugopal, Rajiv Ranjan, and Chee Shin Yeo

David Abramson Clayton School of Information Technology, Monash University,Clayton, Melbourne, VIC 3800, Australia [e-mail: david.abramson@infotech.monash.edu.au]

J€orn Altman Computer Networks and Distributed Systems, International versity, Campus 3, 76646 Bruchsal, Germany [e-mail: jorn.altmann@acm.org]Nazareno Andrade Departamento de Sistemas e Computac¸~ao, UniversidadeFederal de Campina Grande, Av Aprıgio Veloso, 882 Bodocongo´, Bloco CO58109-970, Campina Grande, PB, Brazil [e-mail: nazareno@dsc.ufcg.edu.br]Alvin AuYoung Department of Computer Science and Engineering, University ofCalifornia San Diego, 9500 Gilman Drive, MC 0404, La Jolla, CA 92093, USA[e-mail: alvina@cs.ucsd.edu]

Uni-Pavan Balaji Mathematics and Computer Science Division, Argonne NationalLaboratory, 9700 South Cass Avenue, Argonne, IL 60439, USA [e-mail: balaji

@mcs.anl.gov]

Siegfried Benkner Institute of Scientific Computing, University of Vienna,Nordbergstraße 15, 1090 Vienna, Austria [e-mail: sigi@par.univie.ac.at]Jose Luis Bosque Facultad de Informatica, Universidad Politecnica de Madrid,Campus de Montegancedo S/N 28.660 Boadilla del Monte, Madrid, Spain[e-mail: joseluis.bosque@unican.es]

Ivona Brandic Institute of Scientific Computing, University of Vienna,Nordbergstraße 15, 1090 Vienna, Austria [e-mail: brandic@par.univie.ac.at]Francisco Brasileiro Universidade Federal de Campina Grande, Departamento deSistemas e Computac¸~ao, Av Aprıgio Veloso, 882 Bodocongo´, Bloco CO 58109-

970, Campina Grande, PB, Brazil [e-mail: fubica@dsc.ufcg.edu.br]

Kris Bubendorfer School of Mathematics, Statistics, and Computer Science,Victoria University of Wellington, Kelburn Parade, Wellington, New Zealand[e-mail: kris@mcs.vuw.ac.nz]

Phil Buonadonna Intel Research Berkeley, 2150 Shattuck Avenue, Suite 1300,Berkeley, CA 94704, USA [e-mail: pbuonadonna@archrock.com]

Rajkumar Buyya Department of Computer Science and Software Engineering,The University of Melbourne, Parkville, Melbourne, VIC 3010, Australia [e-mail:raj@csse.unimelb.edu.au]

Kyle Chard School of Engineering and Computer Science, Victoria University

of Wellington, Kelburn Parade, Wellington, New Zealand [e-mail: Kyle.Chard@mcs.vuw.ac.nz]

Kenneth Chiu Department of Computer Science, State University of New York(SUNY) at Binghamton, Binghamton, NY 13902, USA [e-mail: kchiu@cs.binghamton edu]

Brent N Chun Intel Research Berkeley, 2150 Shattuck Avenue, Suite 1300,Berkeley, CA 94704, USA [e-mail: bnc@theether.org]

Walfredo Cirne Departamento de Sistemas e Computac¸~ao, Universidade Federal

de Campina Grande, Av Aprıgio Veloso, 882 Bodocongo´, Bloco CO 58109-970,Campina Grande, PB, Brazil [e-mail: walfredo@dsc.ufcg.edu.br]

Karim Djemame School of Computing, University of Leeds, Leeds LS2 9JT,

Uni-Andrea Guarise Istituto Nazionale di Fisica Nucleare Sezione di Torino, ViaPietro Giuria 1, 10125 Torino, Italy [e-mail: guarise@to.infn.it]

Jordi Guitart Barcelona Supercomputing Center, c/Jordi Girona 29, EdificiNexus II, 3aplanta, E08034 Barcelona, Spain [e-mail: jguitart@ac.upc.edu]

Pilar Herrero Facultad de Informatica, Universidad Politecnica de Madrid,Campus de Montegancedo S/N 28.660 Boadilla del Monte, Madrid, Spain[e-mail: pherrero@fi.upm.es]

H Howie Huang Department of Electrical and Computer Engineering, School ofEngineering and Applied Science, The George Washington University, 801 22ndStreet NW, Washington, DC 20052, USA [e-mail: howie@gwu.edu]

Mohammad Islam Department of Computer Science and Engineering, 595Dreese Lab, 2015 Neil Avenue, Ohio State University, Columbus, OH 43210,USA [e-mail: islammo@cse.ohio-state.edu]

Odej Kao Technische Universitat Berlin, Department of TelecommunicationSystems, Einsteinufer 17, 10587 Berlin, Germany [e-mail: Odej.Kao@tu-berlin.de]

Bastian Koller University of Stuttgart, High Performance Computing Center,Nobelstrasse 19, D-70569 Stuttgart, Germany [e-mail: koller@hlrs.de]

Yu-Kwong Kwok Department of Electrical and Computer Engineering, ColoradoState University, Fort Collins, CO 80523, USA [e-mail: Ricky.Kwok@colostate.edu]

Richard Lowe IT Innovation Centre, University of Southampton, 2 Venture Road,Southampton SO16 7NP, UK

Mario Macıas Barcelona Supercomputing Center, c/Jordi Girona 29, EdificiNexus II, 3aplanta, E08034, Barcelona, Spain [e-mail: mario.macias@bsc.es]Dan Cristian Marinescu School of Electrical Engineering and ComputerScience, University of Central Florida, 4000 Central Florida Boulevard, Orlando,

FL 32816, USA [e-mail: dcm@cs.ucf.edu]

Paul McKee Centre for Information and Security Systems Research, BT AdastralPark, British Telecom, Ipswich IP5 3RE, UK [e-mail: paul.mckee@bt.com]John Patrick Morrison Computer Science Department, University College, Cork,Ireland [e-mail: J.Morrison@cs.ucc.ie]

Miranda Mowbray Hewlett-Packard Laboratories Bristol, Filton Road,Stoke Gifford, Bristol BS34 8QZ, UK [e-mail: miranda.mowbray@hp.com]

Chaki Ng School of Engineering and Applied Science, Harvard University,Maxwell Dworkin 229, 33 Oxford Street, Cambridge, MA 02138, USA [e-mail:chaki@eecs.harvard.edu]

Eduardo Oliveros Telefo´nica Iþ D, Emilio Vargas, 6, Madrid 28043, Spain[e-mail: eod@tid.es]

James Padgett School of Computing, University of Leeds, Leeds LS2 9JT,

UK [e-mail: jamesp@comp.leeds.ac.uk]

Ben Palmer School of Engineering, and Computer Science, Victoria University

of Wellington, Kelburn Parade, Wellington, New Zealand [e-mail: Benjamin.Palmer@mcs.vuw.ac.nz]

David C Parkes School of Engineering and Applied Science, Harvard University,Maxwell Dworkin 229, 33 Oxford Street, Cambridge, MA 02138, USA [e-mail:parkes@eecs.harvard.edu]

Marıa S Perez Facultad de Informatica, Universidad Politecnica de Madrid,Campus de Montegancedo S/N 28.660 Boadilla del Monte, Madrid, Spain[e-mail: mperez@fi.upm.es]

Rosario M Piro Istituto Nazionale di Fisica Nucleare Sezione di Torino, ViaPietro Giuria 1, 10125 Torino, Italy [e-mail: piro@to.infn.it]

Sabri Pllana Institute of Scientific Computing, University of Vienna,Nordbergstraße 15, 1090 Vienna, Austria [e-mail: pllana@par.univie.ac.at]Radu Prodan Institut f€ur Informatik, Universit€at Innsbruck, Technikerstraße 21a,A-6020 Innsbruck, Austria [e-mail: radu@dps.uibk.ac.at]

Dang Minh Quan Computer Networks and Distributed Systems, InternationalUniversity, Campus 3, 76646 Bruchsal, Germany [e-mail: quandm@upb.de]Kotagiri Ramamohanarao Department of Computer Science and SoftwareEngineering, The University of Melbourne, Parkville, Melbourne, VIC 3010,Australia [e-mail: rao@csse.unimelb.edu.au]

Omer Rana School of Computer Science, Cardiff University, Queen’s Buildings,Newport Road, Cardiff CF24 3AA, UK [e-mail: o.f.rana@cs.cardiff.ac.uk]Rajiv Ranjan Department of Computer Science and Software Engineering, TheUniversity of Melbourne, Parkville, Melbourne, VIC 3010, Australia [e-mail:rranjan@csse.unimelb.edu.au]

Ponnuswamy Sadayappan Department of Computer Science and Engineering,

595 Dreese Lab, 2015 Neil Avenue, Ohio State University, Columbus, OH 43210,USA [e-mail: saday@cse.ohio-state.edu]

Alfonso Sanchez-Macian IT Innovation Centre, University of Southampton, 2 ture Road, Southampton SO16 7NP, UK [e-mail: asm@it-innovation.soton.ac.uk]

Ven-Bj€orn Schnizler Institute of Information Systems and Management (IISM), versit€at Karlsruhe (TH), Englerstraße 14, D-76131 Karlsruhe, Germany [e-mail:Schnizler@iism.uni-karlsruhe.de]

Uni-Jeff Shneidman School of Engineering and Applied Science, Harvard University,Maxwell Dworkin 229, 33 Oxford Street, Cambridge, MA 02138, USA [e-mail:jeffsh@eecs.harvard.edu]

Howard Jay Siegel Department of Electrical and Computer Engineering, ColoradoState University, Fort Collins, CO 80523, USA [e-mail: HJ@ColoState.edu]

Alex C Snoeren Department of Computer Science and Engineering, University ofCalifornia, San Diego, 9500 Gilman Drive, MC 0404, La Jolla, CA 92093, USA[e-mail: snoeren@cs.ucsd.edu]

Tiberiu Stef-Praun Computation Institute, University of Chicago, 5640 S EllisAve, Chicago, IL 60637, USA [e-mail: tiberius@ci.uchicago.edu]

Yibo Sun San Diego Supercomputer Center, University of California, San Diego,

10100 Hopkins Drive, MC 0505, La Jolla, CA 92093, USA [e-mail: sunyibo@gmail.com]

Mike Surridge IT Innovation Centre, University of Southampton, 2 Venture Road,Southampton, SO16 7NP, UK

Steve Taylor IT Innovation Centre, University of Southampton, 2 Venture Road,Southampton, SO16 7NP, UK [e-mail: sjt@it-innovation.soton.ac.uk]

Wayne Thomson School of Engineering and Computer Science, Victoria versity of Wellington, Kelburn Parade, Wellington, New Zealand [e-mail: Wayne.Thomson@mcs.vuw.ac.nz]

Uni-Ruppa K Thulasiram Department of Computer Science, University of Manitoba,Winnipeg, MB R3T 2N2, Canada [e-mail: tulsi@cs.umanitoba.ca]

Sameer Tilak San Diego Supercomputer Center, University of California,San Diego, 10100 Hopkins Drive, MC 0505, La Jolla, CA 92093, USA [e-mail:sameer@sdsc.edu]

Amin Vahdat Department of Computer Science and Engineering, University ofCalifornia, San Diego, 9500 Gilman Drive, MC 0404, La Jolla, CA 92093, USA[e-mail: vahdat@cs.ucsd.edu]

Srikumar Venugopal Department of Computer Science and Software ing, The University of Melbourne, Parkville, Melbourne, VIC 3010, Australia[e-mail: srikumar@csse.unimelb.edu.au]

Engineer-Kerstin Voss Paderborn Center for Parallel Computing, University ofPaderborn, Furstenallee 11, 33102 Paderborn, Germany [e-mail: kerstinv@upb.de]

Philipp Wieder Dortmund University of Technology, ITMC, Campus South, GB V,Room 101, August-Schmidt-Str 12, 44227 Dortmund, Germany [e-mail:philipp.wieder@udo.edu]

John Wilkes HP Laboratories, 1501 Page Mill Rd., MS 1139, Palo Alto, CA 94304,USA [e-mail: john.wilkes@hp.com]

Ramin Yahyapour Dortmund University of Technology, ITMC, Campus South,

GB V, Room 101, August-Schmidt-Str 12, 44227 Dortmund, Germany [e-mail:ramin.yahyapour@udo.edu]

Chee Shin Yeo Department of Computer Science and Software Engineering, TheUniversity of Melbourne, Parkville, Melbourne, VIC 3010, Australia [e-mail:csyeo@csse.unimelb.edu.au]

Jia Yu Department of Computer Science and Software Engineering, The versity of Melbourne, Parkville, Melbourne, VIC 3010, Australia [e-mail:yujia mail@yahoo.com]

Uni-Wolfgang Ziegler Fraunhofer-Institut f€ur Algorithmen, und WissenschaftlichesRechnen SCAI, Schloss Birlinghoven, 53754 Sankt Augustin, Germany [e-mail:Wolfgang.Ziegler@scai.fraunhofer.de]

The growing popularity of the Internet and Web, and the availability of powerfulcomputers and high-speed networks as low-cost commodity components, arechanging the manner in which computing, communication, and business arecarried out These technological advances enable the coupling of a wide variety

of geographically distributed resources, such as supercomputers, storage systems,databases, sensors, scientific instruments, and software services, to establish thenext-generation paradigm for distributed computing, called Grid computing Theinterest in creating Grids for sharing resources from multiple, “autonomous”organizations is growing due to the potential for solving large-scale problemsthat cannot be typically tackled using resources in a single organization There areseveral initiatives and projects all over the world that are actively exploring thedesign and development of Grid technologies, service, and applications, and theinfrastructure to support them

The Grid is analogous in concept to the power (electricity) Grid It aims to coupledistributed resources, and offers consistent and inexpensive access to them irrespec-tive of their physical location Thus, Grid computing is enabling the delivery ofcomputing as the fifth utility to users after water, gas, electricity, and telephone Such amodel of computing is popularly called “utility computing” in the business worldwhere service providers maintain and supply information technology (IT) services toconsumers, and receive payment in return

As the Grid matures, a vision of a truly global Grid infrastructure has started toemerge In this global Grid, computational resources are acquired on demand andprovided with an agreed-on quality of service Participation is open to all, and resourcesmay be used or potentially provided by institutions, companies, or the general public.Such a global Grid will motivate the establishment of new marketplaces for trading

application, computation, bandwidth, and storage services These marketplaces willhelp enhance the value of utility delivered to the end users.

Ultimately, Grid computing requires a paradigm shift, whereby resources aretraded, negotiated, allocated, provisioned, and monitored according to users’quality of service requirements The market-oriented Grid will underpin theevolution of the Grid from a collection of computational islands into a globalcomputational environment capable of delivering different levels of services, withefficient handling of risks and costs, depending on the preferences of the user At thesame time, it will provide economic “incentives” for service providers for sharingresources and encourage the delivery of services that meet users’ quality-of-serviceexpectations

Such a market-oriented Grid will create value for all participants Resourceproviders can generate revenue, allowing long-term investments in their resources,and thus, outsourcing of peak loads can be automated Users can better express theirpreferences, trade costs against performance, access to a larger pool of resources, andnegotiate service-level agreements (SLAs) to enhance the observed stability of theirapplications

The purpose of this book, entitled Market-Oriented Grid and Utility Computing,

is to capture the state of the art in both market-oriented Grid and utility computingresearch, and to identify potential research directions and technologies that willfacilitate creation of a global commercial Grid system We expect the book to serve as

a reference for large audiences such as systems architects, practitioners, developers,new researchers, and graduate-level students This book will also come with anassociated Website (hosted at http://www.gridbus.org/gridmarket/) containing poin-ters to advanced online resources and teaching material

ORGANIZATION OF THE BOOK

This book contains chapters authored by several leading experts in the field of oriented Grid and utility computing The book is presented in a coordinated andintegrated manner starting with the fundamentals, and followed by the technologiesthat implement them

market-The contents of the book are organized into four parts:

I Foundations

II Business models

III Policies and agreements

IV Resource allocation and scheduling mechanisms

Part I presents fundamental concepts of market-oriented computing, introducesvarious market mechanisms along with their implications on global Grids followed bythe issues and challenges in allocating resources in a decentralized computingenvironment It also presents utility functions capturing goals of users and service

providers, and various types of markets, such as options and commodity markets, forcomputing resources.

Part II covers business models for services providers and brokers supportingdifferent types of distributed applications It also presents business-rules-basedmodels for managing virtual organizations, and for accounting operations andservices in Grid computing environments

Part III introduces policies, agreements, and specifications for the negotiation andestablishment of contracts between providers and consumers It also covers differentapproaches for resource allocation, based on SLAs and management of risksassociated with SLA violations

Part IV presents market-oriented resource allocation mechanisms and varioustechnologies supporting different market models It covers economic models, such ascommodity models, reciprocation, auctions, and game theory, and middlewaretechnologies such as Nimrod-G and Gridbus for market-oriented Grid computingand utility-oriented resource allocation

We thank our family members Smrithi, Soumya, and Radha Buyya; and Andrea,Gretchen, and Katja Bubendorfer for their love and understanding during thepreparation of the book

Finally, we would like to thank the staff at Wiley, particularly Paul Petralia (SeniorEditor), Anastasia Wasko, Michael Christian (Editorial Assistants), and Sanchari Sil(at Thomson Digital) They were wonderful to work with!

AAA authentication, authorization, and accounting

AC activity class; alternating current

aet average execution time

ANC average normalization cost

ANT average normalized time

API application program interface

ASP application service provider

AT agreement template

BES basic execution service

BFS breadth-first search

BIC Bayesian incentive compatibility

BLO business-level objective

BPEL business process execution language

B2B business-to-business (business business)

B2C business-to-consumer (business consumer)

CAM collaborative/cooperative awareness management/model

CAP combinatorial allocation problem; catallactic access pointCCS command control subsystem

CD cost distribution

CDE CAM data extension

cdf cumulative distribution/density function

CORA coallocative oversubscribing resource allocation

COVITE collaborative virtual team(s)

CP control partition

CRH collaborative resolution history

CSCW computer-supported cooperative work

CSF community scheduler framework; critical success factor

DAC directed acyclic graph

DGAS distributed Grid accounting system

DHT distributed hash table

DNS domain name service

DRIVE distributed resource infrastructure for a virtual economy

DSIC dominant strategies incentive compatibility

DSS database search service

EDL encourage/discourage, linear

EDN encourage/discourage, nonlinear

eet expected execution time

EGEE Enabling Grids for E-sciencE (proprietary name)

EMH efficient market hypothesis

EN enterprise network

EPIC ex post incentive compatibility

EPIR ex post individual rationality

EPR endpoint reference

FCFS first come first serve

FLVM first-level virtual machine (SLVM, TLVM¼ second-,

third-level VM)FQAN fully qualified attribute name

FSK frequency shift key(ing)

GA genetic algorithm

GAF general auction framework

GASA Grid accounting services architecture

GASS global access to secondary storage

GC global constraint; greedy cost

GCC Grid compute commodity

GFA Grid-Federation agent

GIIS Grid Information Indexing Service

GIS Grid Information Service

GMD Grid Market Directory

GMM Grid market manager

GMP Grid marketplace

GOC Grid operations center

GP guest partition

GQWS GMD query Web service

GRAAP Grid Resource Allocation Agreement Protocol

GRACE Grid architecture for computational economy

GrADS Grid Application Development Software

GRAM Grid/Globus resource allocation management

GRASP greedy randomized adaptive search procedure

GRB Grid resource broker

GSC Grid service customer

GSI Globus security infrastructure

GSP Grid service provider

GTS Grid trade server/Trading Service

GUI graphical user interface

HEFT heterogeneous earliest finish time

HPC high-performance computing

HPCC high-performance computing center

IaaS infrastructure as a service

IBV integrated bid vector

ICE IntercontinentalExchange (proprietary)

ICNIP iterated contract-net interaction protocol

IETF Internet Engineering Task Force

IP integer programming

IPR intellectual property right

IRB Intel Research Berkeley

iRODS Internet Rule Oriented Data System (proprietary)

iSCSI Internet small-computer systems interface

ISM industrial scientific medical (RF band)

ISP Internet service provider

IT information technology

JSDL job submission description language

KPI key performance indicator

(L)APW (linearized) augmented plane wave

LEAD linked environments and atmospheric discovery

LHC Large Hadron Collider

LRM local resource manager

MACE multiattribute combinatorial exchange

MAP MACE allocation problem

MAUT multiattribute utility theory

MCDM multicriteria decisionmaking

MCT minimum completion time

MDFA metadata flow analyzer

MDS message delivery service; monitoring and discovery service;

metacomputing directory serviceMDV metadata variable

MET minimum execution time

MFSS maxillofacial surgery simulation

MGS master Grid service

MRT modified real time

MSB modified slack-based

MTTF mean time to failure

MTTR mean time to respond

NAA numerical aerodynamic application

NGS National Grid Service (UK)

OASIS Organization for Advancement of Structural Information

Standards

OC opportunity cost

OCEAN Open Competition Exchange and Arbitration Network

OGSA open Grid service(s) architecture

OGSI open Grid service(s) infrastructure

OLB optimistic load balancing

OSG Open Science Grid

P&L profit and loss

pdf probability distribution/density function

PLC PlanetLab Center

PoF probability of failure

PSA parameter sweep application

PSHA probabilistic seismic hazard analysis

P2P peer-to-peer (P3¼ parallel P2P)

PV payload variable

QoE quality of experience

QoPS QoS for parallel (job) scheduling (VQoPS¼ value-aware QoPS;

DVQoPS¼ dynamic VQoPS)QoS quality of service

QoWL QoS-aware (Grid) workflow language

QWE QoS-aware (Grid) workflow engine

RI reputation index

RLQ resource lookup query

rms root mean square

RMS resource management system

ROI return on investment

RPC remote procedure call

RSS resource selection service

RUQ resource update query

RUS resource usage service

SaaS software as a service

SAGE storage accounting for Grid environments

SAM SLA action manager

SCEC Southern California Earthquake Center

SCS social choice function

SD supplier database; Storage@desk

SDT service description term

SGVA secure generalized Vickrey auction (vSGVA¼verifiable SGVA)SGT strain green tensor (seismic index used by SCEC)

SLA service-level agreement

SLI service-level indicator

SLO service-level objective

SMI special model of interaction

SOA service-oriented architecture

SOAP Simple Object Access Protocol (proprietary)

SOI service-oriented infrastructure

SP service provider/profile

SPEC Standard Performance Evaluation Corporation

SPMD single program multiple data

SRM storage resource management

SSL Secure Sockets Layer (proprietary)

SVM standard virtual machine

TD time distribution

TF tolerance factor

TFE task-farming engine

UCITA Uniform Computer Information Transaction Act

UETA Uniform Electronic Transaction Act

VGE Vienna Grid Environment

VLDB very large database

VO virtual organization

VOMS virtual organization membership service

VPOT verifiable proxy oblivious transfer

WAP wireless access point

WfMC Workflow Management Coalition

WORM write once read many

WS workstation; Web services

WSAS Web Services Agreement Specification

WSC Web services consumer

WSLA Web Service Level Agreement

WSP Web services provider

WSRF Web services resource framework

XML eXtensible Modeling Language

XPML eXtended Parametric Modeling Language

PART I

FOUNDATIONS

an analogous infrastructure, called the computational power Grid [4], for wide-areaparallel and distributed computing [6] The motivation for computational Grids wasinitially driven by large-scale, resource (computational and data)-intensive scientificapplications that require more resources than a single computer [PC, workstation(WS), supercomputer, or cluster] could provide in a single administrative domain.

A Grid enables the sharing, selection, and aggregation of a wide variety of graphically distributed resources, including supercomputers, storage systems, datasources, and specialized devices owned by different organizations for solving large-scale resource-intensive problems in science, engineering, and commerce Because ofits potential to make impact on the twenty-first century as much as the electric power

geo-Market Oriented Grid and Utility Computing Edited by Rajkumar Buyya and Kris Bubendorfer Copyright  2010 John Wiley & Sons, Inc.

Grid did on twentieth century, Grid computing has been hailed as the next revolutionafter the Internet and the Web.

These developments foreshadow the realization of the vision of Leonard rock, one of the chief scientists of the original Advanced Research Projects AgencyNetwork (ARPANET) project that seeded the Internet, who said in 1969 [3]: “As ofnow, computer networks are still in their infancy, but as they grow up and becomesophisticated, we will probably see the spread of ‘computer utilities,’ which, likepresent electric and telephone utilities, will service individual homes and officesacross the country.”

Klein-Utility computing is envisioned to be the next generation of information ogy (IT) evolution that depicts how computing needs of users can be fulfilled in thefuture IT industry [13] Its analogy is derived from the real world, where serviceproviders maintain and supply utility services, such as electrical power, gas, and water

technol-to consumers Consumers in turn pay service providers according technol-to their usage.Therefore, the underlying design of utility computing is based on a service provision-ing model, where users (consumers) pay providers for using computing power onlywhen they need to

resources and enables the formation and management of virtual organizations (VOs).

It also supports application and services composition, workflow expression, scheduling,and execution management and service-level agreement (SLA)-based allocation ofresources

As there are a large number of projects around the world working on developingGrids for different purposes at different scales, several definitions of Grid abound TheGlobus project (Argonne National Laboratory, USA) defines Grid as “an infrastruc-ture that enables the integrated, collaborative use of high-end computers, networks,databases, and scientific instruments owned and managed by multiple organizations.”Another utility notion-based Grid definition put forward by the Gridbus project(University of Melbourne, Australia) is “Grid is a type of parallel and distributedsystem that enables the sharing, selection, and aggregation of geographicallydistributed ‘autonomous’ resources dynamically at runtime depending on theiravailability, capability, performance, cost, and users’ Quality of Service (QoS)requirements.”

The development of the Grid infrastructure, both hardware and software, hasbecome the focus of a large community of researchers and developers in bothacademia and industry The major problems being addressed by Grid developmentsare the social problems involved in collaborative research:

. Improving distributed management while retaining full control over locallymanaged resources

. Improving the availability of data and identifying problems and solutions to dataaccess patterns

. Providing researchers with a uniform user-friendly environment that enablesaccess to a wider range of physically distributed facilities improving productivity

A high-level view of activities involved within a seamless and scalable Gridenvironment is shown in Figure 1.2 Grid resources are registered within one or more

Figure 1.2 A worldwide Grid computing environment

Grid information services The end users submit their application requirements to theGrid resource broker, which then discovers suitable resources by querying theinformation services, schedules the application jobs for execution on these resources,and then monitors their processing until they are completed A more complex scenariowould involve more requirements and therefore, Grid environments involve servicessuch as security, information, directory, resource allocation, application development,execution management, resource aggregation, and scheduling Software tools andservices providing these capabilities to link computing capability and data sources

in order to support distributed analysis and collaboration are collectively known asGrid middleware

In order to provide users with a seamless computing environment, the Gridmiddleware systems need to solve several challenges originating from the inherentfeatures of the Grid [8] One of the main challenges is the heterogeneity that resultsfrom the vast range of technologies, both hardware and software, encompassed bythe Grid Another challenge involves the handling of Grid resources that are spreadacross political and geographic boundaries and are under the administrative control

of different organizations It follows that the availability and performance of Gridresources are unpredictable as requests from within an administrative domain maygain higher priority over requests from outside Thus, the dynamic nature of Gridenvironment poses yet another challenge

To tackle these challenges, a Grid architecture has been proposed based on thecreation of virtual organizations (VOs) [9] by different physical (real-world) orga-nizations coming together to share resources and collaborating in order to achieve acommon goal A VO defines the resources available for the participants and the rulesfor accessing and using the resources Within a VO, participants belonging to memberorganizations are allocated resource shares according to the urgency and priority of

a request as determined by the objectives of the VO Another complementary Gridarchitecture [10] is based on economic principles in which resource providers(owners) compete to provide the best service to resource consumers (users) whoselect appropriate resources according to their specific requirements, the price ofthe resources, and their quality-of-service (QoS) expectations from the providers.Two examples of QoS terms are the deadline by which the resource needs to beavailable and the maximum price (budget) that can be paid by the user for the service.QoS terms are enforced via service-level agreements (SLAs) between the providersand the consumers, the violation of which results in penalties

1.3 GRID COMPONENTS

In a worldwide Grid environment, capabilities that the infrastructure needs to supportinclude

. Remote storage and/or replication of datasets

. Publication of datasets using global logical name and attributes in the catalog. Security access authorization and uniform authentication

. Uniform access to remote resources (data and computational resources). Publication of services and access cost

. Composition of distributed applications using diverse software componentsincluding legacy programs

. Discovery of suitable datasets by their global logical names or attributes. Discovery of suitable computational resources

. Mapping and scheduling of jobs (aggregation of distributed services)

. Submission, monitoring, and steering of job execution

. Movement of code/data between user desktop computers and distributed Gridresources

. Enforcement of QoS requirements

. Metering and accounting of resource usage

These capabilities in Grid computing environments play a significant role in enabling

a variety of scientific, engineering, and business applications Various Grid nents providing these capabilities are arranged into layers Each layer builds on theservices offered by the lower layer in addition to interacting and cooperating withcomponents at the same level (e.g., resource broker invoking secure process manage-ment services provided by core middleware) Figure 1.3 shows four layers of thehardware and software stack within a typical Grid architecture: fabric, core middle-ware, user-level middleware, and applications/portals layers Adaptive managementcapabilities are supported by implementing principles of market-oriented resourcemanagement mechanisms in different horizontal layers

compo-The Grid fabric layer consists of distributed resources such as computers, works, storage devices, and scientific instruments The computational resources

net-Grid resources:

Desktops, servers, clusters, networks, applications, storage, devices + resource manager + monitor

Security Services:

Authentication, Single sign on, secure communication

Job submission, info services, storage access, trading, Accounting, License management Resource management and scheduling

Grid programming environment and tools:

Languages, API, libraries, compilers, parallelization tools

represent multiple architectures such as clusters, supercomputers, servers, andordinary PCs that run a variety of operating systems (such as UNIX variants orWindows) Scientific instruments such as telescopes and sensor networks providereal-time data that can be transmitted directly to computational sites or are stored in

a database

The core Grid middleware offers services such as remote process management,coallocation of resources, storage access, information registration and discovery,security, and aspects of QoS such as resource reservation and trading These servicesabstract the complexity and heterogeneity of the fabric level by providing a consistentmethod for accessing distributed resources

The user-level Grid middleware utilizes the interfaces provided by the low-levelmiddleware to provide higher-level abstractions and services These include applica-tion development environments, programming tools, and resource brokers formanaging resources and scheduling application tasks for execution on globalresources

Grid applications and portals are typically developed using Grid-enabled gramming environments and interfaces and are deployed on Grids using brokeringand scheduling services provided by user-level middleware An example application,such as parameter simulation of a grand-challenge problem, would require computa-tional power and access to remote datasets, and may need to interact with scientificinstruments Grid portals offer Web-enabled application services, where users cansubmit their jobs to remote resources and collect results from them through the Web.The design aims and benefits of Grids are analogous to those of utility computing,thus highlighting the potential and suitability of Grids to be used as utility computingenvironments The current trend of implementing Grids based on open standardservice-based architectures to improve interoperability is a step toward supportingutility computing Even though most existing Grid applications are scientific researchand collaboration projects, the number of applications in business and industry-related projects is gradually increasing It is thus envisioned that the realization

pro-of utility computing through Grids will follow a course similar to that pro-of the WorldWide Web, which was first initiated as a scientific project but was later widely adopted

by businesses and industries

1.4 GRID INITIATIVES AROUND THE WORLD

Given the possibilities of Grid computing, it is no surprise that there is keen interest

in this technology around the world (globally) Currently, Grid projects that havebeen initiated on the global scale can be broadly classified into two categories [8]:(1) Grid infrastructure development, which involves setting up hardware, software,and administrative mechanisms to enable application scientists to make use of thesefacilities for their research; and (2) Grid middleware research, which investigatesthe development of software and policy mechanisms that assist in realizing the fullpotential of Grid computing Many of these projects are motivated by large-scalescientific projects that will involve the production and analysis of data at an

unprecedented scale One frequently cited such large-scale scientific project is theLarge Hadron Collider (LHC) experiments [11] at the European Organisation forNuclear Research (CERN), which began data production in 2008 The volume of datagenerated by these experiments is in the petabyte (PB) range, for distribution tophysicists around the world for analysis As the Grid has been mandated as the ITinfrastructure for handling the massive workloads of LHC experiments, all thecollaborating nations are setting up Grid infrastructure in one form or another.

In the following sections, we will describe some of the major Grid infrastructureand middleware projects around the world

1.4.1 United States of Amercia (USA)

Production Grid testbeds for various application domains have been deployed overphysical (hardware) Grid infrastructure such as the National Science Foundation(NSF)-funded TeraGrid [17] in the United States, which provides over 40 tera-floating-point operations per second (Tflops) of computing power at eight sites aroundthe country with 2 PB of available storage interconnected by a network operating

at a speed of 10 30 gigabits per second (Gbps) The BioInformatics ResearchNetwork (BIRN) is another testbed for the purpose of furthering biomedical science

by sharing data stored in different repositories around USA The NEESGrid enablesscientists in the earthquake engineering community to carry out experiments indistributed locations and analyse data through a uniform interface

Of the Grid middleware efforts in the United States, the Globus toolkit fromthe Globus Alliance led by Argonne National Laboratory is the most widely known.Other notable efforts are the Condor project (University of Wisconsin, Madison) forhigh-throughput computing mechanisms, and “i Rule Oriented Data Systems”(iRODS) [58] from the San Diego Supercomputing Center (SDSC) for Grid datamanagement In addition, several commercial organizations such as IBM, SunMicrosystems, Hewlett-Packard (HP), and Oracle are actively involved in thedevelopment of enterprise and global utility Grid technologies

1.4.2 Europe

Two pioneering Grid efforts in Europe, started in early 2001, were the UnitedKingdom (UK)’s e-Science program [5] and the European Union (EU)-funded DataGrid project [11] The latter was succeeded by the EGEE (Enabling Grids forE-sciencE) project, which aims to create a Grid infrastructure available to scientistsand to develop robust middleware for application deployment CERN manages theLHC Computing Grid (LCG) project, which has created a production Grid infra-structure for researchers involved in the experiments using the LHC

Other notable EU-funded projects include GridLab [53], providing a Gridapplication development toolkit; Cactus framework, for scientific programming;GridSphere, for creating a Web portal environment for Grid users; P-Grade, providing

a visual environment for application development; Triana, for workflow formulation;and OGSA-DAI, for integration of relational databases in Grid environments

1.4.3 Asia–Pacific

Several countries in the Asia Pacific region have started national Grid programssimilar to those initiated in the United States and Europe In addition, countries such asAustralia, China, Japan, South Korea, and Singapore are active participants inworldwide Grid projects such as the LCG Some of the notable Grid programs arethe National Research Grid Initiative (NAREGI) in Japan, China National Grid inChina, KGrid in South Korea, and Garuda National Grid in India

Prominent Grid middleware projects include the Ninf project (Tokyo Institute

of Technology) for building a Grid-based remote procedure call (RPC) system [29],the Grid Datafarm (Gfarm) project (AIST, Japan) for providing a petascale datastorage processing system, the Nimrod/G project (Monash University, Australia)for parametric computations on Grid resources [28], and the Gridbus project(University of Melbourne, Australia) for market-oriented Grid and utilitycomputing [54]

1.4.4 Standardization Efforts

Given the large amount of middleware development happening in this area ofresearch, standardization is important to ensure interoperability between differentproducts and implementations Grid standardization efforts led by the Open GridForum (OGF) [12] have produced standards for almost all aspects of Gridtechnology Work at the OGF has produced the open Grid service infrastructure(OGSI) specification and its successor, the Web services resource framework(WSRF), which have paved the way for integration of Web services within Gridarchitecture This is important as Web services allow Grid developers to takeadvantage of standard message formats and mechanisms such as HTTP and XMLfor communicating between heterogeneous components and architectures Otherstandardization bodies such as World Wide Web Consortium (W3C), Organizationfor Advancement of Structured Information Standards (OASIS), and InternetEngineering Task Force (IETF) also produce standards relevant to aspects of GridComputing

1.5 MARKET-ORIENTED GRID RESOURCE MANAGEMENT

Resource management and scheduling in Grid environments is a complex taking The geographic distribution of resources owned by different organizationswith different usage policies, cost models, and varying load and availability patterns isproblematic The producers (resource owners) and consumers (resource users) havedifferent goals, objectives, strategies, and requirements Classical Grids are motivated

under-by an assumption that coordinated access to diverse and geographically distributedresources is valuable However, this paradigm needs mechanisms that allow not onlysuch coordinated access but also sustainable, scalable models and policies thatpromote utility-oriented sharing of Grid resources

To address these resource management challenges, several groups of researchershave proposed a distributed computational economy-based1framework [10,14,50,56],for resource allocation and to regulate supply and demand of the availableresources This economy-based framework offers an incentive to resource owners forcontributing and sharing resources, and motivates resource users to think abouttradeoffs between the processing time (e.g., deadline) and computational cost (e.g.,budget), depending on their QoS requirements It can be observed that, even inelectricity Grids, bid-based electricity trading over the Internet has been adopted todevelop competitive forces in the electricity marketplace [20].

Resource management and scheduling systems for Grid computing need to manageresources and application execution depending on resource consumers’ and owners’requirements, and they need to continuously adapt to changes in the availability ofresources This requirement introduces a number of challenging issues that need to beaddressed, namely: site autonomy, heterogeneous substrate, policy extensibility, resourceallocation or coallocation, online control, resource trading, and QoS-based scheduling.1.5.1 Assessing Wants and Needs

In an economy-based Grid computing environment, resource management systemsneed to provide mechanisms and tools that allow resource consumers (end users) andproviders (resource owners) to express their requirements and facilitate the realization

of their goals Resource consumers need

. A utility model to determine how consumers demand resources and theirpreference parameters

. A broker that supports resource discovery and strategies for application duling on distributed resources dynamically at runtime depending on theiravailability, capability, and cost along with user-defined QoS requirementsResource providers need

sche-. Tools and mechanisms that support price specification and generation schemes

to increase system utilization

. Protocols that support service publication, trading, and accounting

For the market to be competitive and healthy, coordination mechanisms are required

to help reach equilibrium price the market price at which the supply of a serviceequals the quantity demanded

1.5.2 Computational Economy and Its Benefits

Like all systems involving goals, resources, and actions, computations can be viewed

in economic terms With the proliferation of networks, high-end computing systems

1 The terms “economic/economy based” and “market based” are synonymous and interchangeable.

architecture has moved from centralized toward decentralized models of control andaction; the use of economy-driven market mechanisms would be a natural extension

of this development The ability of trade and price mechanisms to combine localdecisions by diverse entities into globally effective characteristics reflect their valuefor organizing computations in large systems such as Internet-scale computationalGrids

The need for an economy-driven resource management and scheduling systemcomes from the answers to the following questions:

. What constitutes the Grid, and who owns its resources?

. What motivates resource owners to contribute their resources to the Grid?. Is it possible to have access to all resources in the Grid by contributing ourresource?

. If not, how do we have access to all Grid resources?

. If we have access to resources through collaboration, are we allowed to use themfor any other purposes?

. Do resource owners charge the same or a different price for different users?. Is access cost the same for peak and off-peak hours?

. How can resource owners maximize their profits?

. How can users solve their problems within a minimum cost?

. How can a user get high priority over others?

. If the user relaxes the deadline by which results are required, can solution cost bereduced?

Several individuals or organizations that have contributed resources to the Gridhave been motivated largely by the public good, prizes, fun, fame, or collaborativeadvantage This is clearly evident from the construction of private Grids (but onvolunteer resources) or research testbeds such as SETI@Home [18], Condorpool [38], and TeraGrid [17] The chances of gaining access to such computationaltestbeds for solving commercial problems are low Furthermore, contributingresources to a testbed does not guarantee access to all the other resources in thattestbed

Commercial companies such as Entropia, ProcessTree, Popular Power, UnitedDevices, and DataSynapse are exploiting idle central processing unit (CPU) cyclesfrom desktop machines to build a commercial computational Grid infrastructurebased on peer-to-peer (P2P) networks [19] These companies are able to developlarge-scale infrastructure for Internet computing and use it for their own financialgain by charging for access to CPU cycles for their customers, without offering fiscalincentive to all resource contributors However, in the long run, this model does notsupport the creation of a maintainable and sustainable infrastructure, as the resourcecontributors have no incentive for their continued contribution Therefore, a Grideconomy seems a better model for managing and handling requirements of both Grid

providers and consumers The benefits of economy-based resource managementinclude the following:

. It helps in building a large-scale Grid as it offers incentive for resource owners tocontribute their (idle) resources for others to use and profit from

. It helps in regulating the supply and demand for resources

. It offers an economic incentive for users to back off when solving low-priorityproblems and thus encourages the solution of time-critical problems first.. It removes the need for a central coordinator (during negotiation)

. It offers uniform treatment of all resources; that is, it allows trading of thing including computational power, memory, storage, network bandwidth/latency [22], data, and devices or instruments

every-. It allows users to express their requirements and objectives

. It helps in developing scheduling policies that are user-centric rather thansystem-centric

. It offers an efficient mechanism for allocation and management of resources.. It helps in building a highly scalable system as the decisionmaking process isdistributed across all users and resource owners

. It supports a simple and effective basis for offering differentiated services fordifferent applications at different times

. Finally, it places the power in the hands of both resource owners and users,enabling them to make their own decisions to maximize the utility and profit

1.6 REQUIREMENTS FOR ECONOMY-BASED GRID SYSTEMS

To deliver value to users greater than that possible with traditional systems, based resource management systems need to provide mechanisms and tools that allowresource consumers (end users) and providers (resource owners) to express theirrequirements and facilitate the realization of their goals In other words, they need(1) the means to express their requirements, valuations, and objectives (valueexpression); (2) scheduling policies to translate them into resource allocations (valuetranslation); and (3) mechanisms to enforce selection and allocation of differentialservices, and dynamic adaptation to changes in their availability at runtime (valueenforcement) Similar requirements are raised [2] for market-based systems in asingle-administrative-domain environment such as clusters, and they are limited tocooperative economic models since they aim for social welfare Grids need to usecompetitive economic models as different resource providers and resource consumershave different goals, objectives, strategies, and requirements that vary with time.Essentially, resource consumers need a utility model to allow them to specifyresource requirements and constraints For example, the Nimrod/G broker allowsusers to specify the deadline and budget constraints along with optimization

parameters such as optimizing for time (value expression) They need brokers thatprovide strategies for choosing appropriate resources (value translation) and dyna-mically adapt to changes in resource availability at runtime to meet user requirements(value enforcement) The resource owners need mechanisms for price generationschemes to increase system utilization and protocols that help them offer competitiveservices (value expression) For the market to be competitive and healthy, coordina-tion mechanisms are required that help the market reach an equilibrium price theprice at which the supply of a service equals the quantity demanded Grid resourceshave their schedulers (e.g., OS or queuing system) that allocate resources (valuetranslation) A number of research systems have explored QoS-based resource (e.g.,CPU time and network bandwidth [22,23]) allocation in operating systems andqueuing systems, but the inclusion of QoS into mainstream systems has been slow-paced (e.g., the Internet mostly uses the best effort allocation policy [24], but this ischanging with IPv6 [25]) Some research systems support resource reservation inadvance (e.g., reserving a slot from time t1to t2using the Globus Architecture forReservation and Allocation (GARA) [21] and binding a job to it) and allocateresources during reserved time (value enforcement).

1.7 MARKET-ORIENTED GRID ARCHITECTURE

A reference service-oriented architecture for market-oriented Grids is shown inFigure 1.4 The key players in a market-oriented Grid are the Grid user, Grid resourcebroker, Grid middleware services, and Grid service providers (GSPs) The Grid userwants to make use of Grids to complete their applications Refactoring existingapplications is thus essential to ensure that these applications are Grid-enabled torun on Grids [26] The Grid user also needs to express the service requirements to befulfilled by GSPs Varying QoS parameters, such as a deadline for the application to becompleted and budget to be paid on completion, are defined by different Grid users,

Figure 1.4 A reference service oriented architecture for utility Grids [1]