A typical batch analysis session would involve the following operations: sign on to the Grid,specification of requested datasets, generation and review of an execution plan this includes
Trang 1an Interactive Grid Analysis Environment
Service ArchitectureJulian Bunn ( julian@cacr.caltech.edu ), Rick Cavanaugh ( Cavanaug@phys.ufl.edu ),
Iosif Legrand ( iosif.legrand@cern.ch ), Harvey Newman ( newman@hep.caltech.edu ),
Suresh Singh ( suresh@cacr.caltech.edu) , Conrad Steenberg ( conrad@hep.caltech.edu )
Michael Thomas ( thomas@hep.caltech.edu ), Frank van Lingen( fvlingen@caltech.edu )
1 Introduction
This document is a proposal that describes an Interactive Grid Analysis Service Architecture , with
a focus on High Energy Physics applications, including a set of work packages and milestones toimplement this architecture
The ideas outlined in this architecture document are motivated by a lack of distributedenvironments dedicated to interactive Grid analysis, and the RTAG activity ARDA (ArchitecturalRoad map towards Distributed Analysis) [1] within LHC The foundation of the architecturedescribed here will be based on web services and peer to peer systems. Web Services have beenidentified as being suitable for Grid applications [2]. Peer to peer technologies are well suited forenvironments composed of dynamic resource providers [3].
The goal of this document is to identify services that can be used within the GAE. Besides reuse
of existing services (e.g. components), this document describes characteristics of an interactiveGrid Analysis Environment (GAE) and identifies new”Grid services and functionality that areneeded to enable distributed interactive analysis
One of the big differences between an GAE and production or batch analysis oriented Grids is thatbehavior of the users is much more unpredictable. This unpredictable behavior of a collection ofusers is too complex for humans to steer usage of resources “manually”. Instead applications areneeded to act on behalf of humans in an autonomous fashion to steer usage of resources and tooptimize resource usage. Furthermore this documents identifies several components that can beused for policy management within the Grid. Policy management prevents that certain usersallocate more resources than they are entitled too.
While this architecture is being developed to solve specific problems related to CMS data analysis[4], many of the services described in this architecture can be used in other Grid systems. Inaddition, while some of the services have already been developed, or are being developed, theyhave never been assembled into such an interactive system before
Trang 2A Grid as used within High Energy Physics consists of a heterogeneous collection of resourcesand components. A secondary goal within this proposal is to define platform and language neutralAPIs in order to allow smooth operation between the heterogeneous Grid resources andcomponents and minimize dependencies between components. The secondary goal is closelyrelated to work done in the PPDG CS 11 work group [5] and is also stated in the HEPCAL IIdocument [6].
Section 2 discusses several characteristics of interactive analysis. Requirements and use casesare discussed in Section 3. Section 4 describes GAE and the different components it comprised of.Based on the description of section 4 several scenarios are discussed in section 5 that showinteraction between different GAE components in typical analysis scenarios. Section 6 and Section
7 map the identified components of GAE to use cases identified for interactive analysis in HEPCAL[7] and HEPCALII [6] and to existing components and applications that can be used within thedevelopment of GAE. Finally Section 8 maps the components described in the architecture section
to work packages and milestones
The content of this proposal focuses on interactive analysis. However, components developed for
an interactive analysis environment could also be used in a batch or production environment andvice versa (see section 2 and section 4).
The research problems, service components, and requirements addressed by this proposal are notonly important for interactive analysis; batch analysis will benefit as well. The work outlined in thisproposal will pave the way for a high energy physics data grid in which production, batch andinteractive behavior coexist.
2 Interactivity in a Grid Environment
Part of the motivation for this proposal is the difference between batch and interactive analysis andthe focus on batch analysis within Grid software development for High Energy Physics. Thissection serves to explain the characteristics of interactive analysis and how it differs from batchanalysis. Additional information on interactive and batch analysis can be found in HEPCALII [6]. 2.1 Interactive versus Batch Analysis
The structure of a batch analysis environment is well known within high energy physics [8] . Alarge number of computing jobs are split up into a number of processing steps arranged in adirected acyclic graph and are executed in parallel on a computing farm. A batch analysis session
is fairly static in that the Directed Acyclic Graph (DAG) structure of the processing steps beingexecuted is known in advance. The only interaction between the user and the batch job is theability to see the progress of the computing job and the ability to restart processing steps that may
Trang 3A typical batch analysis session would involve the following operations: sign on to the Grid,specification of requested datasets, generation and review of an execution plan (this includes theresource allocation for executing the plan), plan submission, plan monitoring and steering, andcollection of analysis results
A typical interactive analysis session is quite similar to a batch analysis session. The maindifference is that the execution plan for interactive analysis is an iterative loop that builds on theresults of previous commands. The structure of the job graph is not known in advance as it is withbatch processing. The user will submit a command to be executed, analyze the results, thensubmit new commands that build upon the results of the previous commands.
Both interactive and batch analysis make use of “processing instructions”. For batch analysis,these processing instructions are comprised of shared libraries, interpreted scripts, and commandparameters that are all known in advance, or can be automatically or programmatically determinedfrom the results of an earlier processing job. In an interactive analysis session, however, theprocessing instructions are not known in advance. The end user will analyze the results of eachprocessing step and change the parameters, libraries, and scripts for succeeding processing steps.Human intervention, as opposed to computer automation, determines the details of eachprocessing step. As humans tend to be more impatient, computers and processing instructions in
an interactive session will have a much shorter execution time than processing instructions in batchjobs. Low latency will be required between the submission of a processing instruction and thereceipt of its results
Both interactive analysis and batch analysis can be seen as two extremes of the analysiscontinuum: It is possible to have an analysis session where both batch and interactive typeanalysis is done. Furthermore it is difficult to assign a “batch” label to any particular analysis. Ifthere is enough low latency and computing power then the response time of a batch analysis could
be short enough such that is becomes part of a larger interactive analysis
The next subsections describe several issues, important for interactive analysis some of whichcould also be used in a batch environment. However none of the current Grid batch environmenthave addressed these issues such that it could be used within an interactive environment
2.2 Execution state
Execution state is the entire set of information needed to recreate the current analysis session.This includes logical dataset names, processing scripts, shared libraries, processing steps alreadytaken, processing parameters, data provenance, and any temporary results (both logical andphysical). It is assumed that the size of the execution state will be small enough to be easily
Trang 4
Once an interactive session is finished, all the execution steps and their order are known by theexecution state. As such the “recorded” interactive session can be “replayed” as a batch sessionwith minimum intervention from a user
Batch and interactive analysis both involve the allocation of execution nodes on which the actualprocessing occurs. In batch analysis, there is little advantage to reusing the same execution nodemultiple times as with batch analysis the execution state and order is known in advance.Interactive analysis, however, benefits from the reuse of an execution node since much of the state
of the interactive session resides on the execution node
The state for an interactive session can not be completely stored on the execution node. Localscheduling policies and resource failures can make an execution node unavailable for the rest of
an interactive session. As such: The interactive client must be able to transfer the state of the
interactive session to another execution node with no required involvement from the user.
Transition from one execution node to another (due to node failure) should be transparent to theuser. The only reliable source of the current interactive execution state must be stored in the clientapplication (or for limited resource clients such as hand held devices, the execution state should bestored on the grid service entry host). The state stored on an individual execution node must betreated as a cache of this canonical state that is stored in the client application. This executionnode state cache can be leveraged to provide lower execution latencies by reusing the sameexecution node as much as possible during an interactive session. The state for an interactivesession is sent with every interactive command so that execution node failures can be handledtransparently to the client application (more on this below)
Figure 2.1 execution stateFigure 2.1 execution state shows how state is managed during an interactive session as described
Trang 5Scenario 1: State Information:
The Grid scheduler locates a suitable Job Processing and Execution Service (JPES) serviceinstance to handle the job. The JPES then sends the command to an execution node. Theexecution node executes the command and returns the new state back to the client (one possibility
is to store the execution state in a CVS repository which would not only track the state but also thestate history) The execution node retains a local cache of the application state for futureprocessing. This is shown as step 1 in Figure 2.1
The client application then sends the second command to a Grid scheduler. The Grid schedulerattempts to use the previous execution service to handle the command. When the executionservice receives the command, it attempts to use the previous execution node to handle thecommand. If the node is still available, then it is used to handle the command. This is also shown
by step 1 in Figure 2.1
If the previous execution node is not available (it was rescheduled for a higher priority task, or itexperienced some hardware failure) then the execution service returns a failure code back to theScheduler, which then attempts to locate another suitable execution service to handle thecommand (execution service 2). A new execution node will be allocated (step 2 in the figure). Thenew execution node uses the state that was sent along with the command to recreate theenvironment of the previous execution node and begins to process new commands. This is shown
by step 2.1 in Figure 2.1
If the execution service becomes unavailable, the Scheduler sends the command to the anotheravailable execution service instance (step 3 in the figure). As before, the state is sent along withthe command so that the new execution node can recreate the environment of the previousexecution node
By sending all interactive commands through a Grid Scheduler, the client application never needs
to know about execution node failures, or about execution service failures (except for monitoringpurposes). State information (job traceability) is also described in HEPCALII [6]
2.3 Steering and Intelligent Agents
In order to satisfy users' lust for responsiveness in an interactive system, constant status andmonitoring feedback must be provided. In any Grid system it must be possible for the user toredirect slow running jobs to faster execution nodes, and perform other redirections based onperceived optimizations. By giving users this control certain patterns of behavior will start toappear and will give valuable information on how to tune the entire system for optimum
Trang 6As certain optimization strategies are discovered, it will be possible to write intelligent agents thatcan perform these optimizations on behalf of the user. These agents will be able to reschedulejobs and reprioritize requests based on the currently available resources. Intelligent agents canalso be used to monitor resource usage profiles and match them to the profiles desired by thegreater Grid community
In addition to policies on who can use what resource, there also needs to an accountingmechanism that keeps track of who is using what resources and a history on resource usage. Oneway of adjusting user priorities is by setting local policies. Managers of the scarce resources canset local policies that give preference to certain users and groups. This is analogous to UNIXprocess priorities, except that the users are given priorities instead of processes
A more sophisticated way to manage priorities is through the use of a Grid Economy. Users andgroups in the Grid are given some number of Grid Dollars (Higgies). These dollars can be spent toprocure more resources for more important Grid tasks. Similarly, users and groups would “sell”their own resources to earn more Higgies to spend on their own Grid tasks. Large groups (such as
a research institutes) could distribute Higgies to individual researchers in the form of a Grid Salary,giving a higher salary to researchers who they believe will use their Higgies more wisely. Withinthis paper we think it is important to recognize the concept of a Grid Economy, however it will not
be addressed in the first phases of the GAE
3 Requirements
Use cases and requirements for an GAE have been described in detail in HEPCAL [7], HEPCAL II[6] , and the PPDG CS 11 requirements document [9]. Some of the requirements focus on specificcomponents that will be used within the GAE. Section 6 associates GAE components with usecases that are relevant for interactive analysis. While requirements and use cases have beenfinalized in HEPCAL and PPDG CS 11, during the development of GAE, requirements and usecases can be subject to change and improvements as developers of the system gain more
Trang 7One GAE requirement has been described in section 2.2: the interactive client must be able totransfer the state of the interactive session to another execution environment (node or farm) with
no required involvement from the user
Requirements in [7], [6], and [9], focused on physics analysis, mainly from a single user point ofview. This section lists several requirements that the overall GAE system needs to satisfy:
As stated in [7] section 2.5: physics analysis is less predictable than other physics processes such
as reconstruction and simulation Furthermore, physicists from all over the world will accessresources and share resources. These resources change/move/disappear in time. Additional to therequirements stated in [7], [6], and [9] for interactivity in a Grid are the following requirements:
Requirement 1, Fault tolerance: Data loss and loss of time because of failures of components
should be minimized. A corollary of this is that the web services must operate asynchronously sothat the application can continue to run in the face of a slow or unresponsive service
Requirement 2, Adaptable: Access to data will be unpredictable, however patterns could be
discovered that can be exploited to optimize the overall performance of the system
Requirement 3, Scalable: The system is able to support an everincreasing number of resources
(CPU, network bandwidth, storage space) without any significant loss in responsiveness. Thesystem is also able to support increasing numbers of users. As more users enter the system, thesystem will respond predictably and will not become unusable for everyone
Requirement 4, Robust: The system should not crash if one site crashes (no chain reaction).
Requirement 5, Transparent: It should be possible for Grid clients to get feedback on the
estimated state of grid resources. This would allow clients to make better decisions on whatresources to use within the Grid
Requirement 6, Traceability: If the performance of Grid resources drop, there should be diagnostic
tools available to analyze these anomalies
Requirement 7, Secure: Security within the GAE will be certificate based: Users get a
certificate/key pair from a certificate authority. This certificate allows a user to access service withinthe Grid using a proxy that acts on behalf of the user for authentication. Depending on the securitysettings and policies (e.g. VO management settings) of different Grid sites and individual services
of that site, a user can access these services offered by a particular site. Clarens [10] is one of theapplications that support certificate base security and VO management
Trang 8“stand alone” components: There will be no specific dependency between any two components.For example, in order to use replica manager “X” I will need to deploy metadata catalog “Y” withinthe architecture. Stand alone components and the use of web services allow to “plug in” differentcomponents (e.g. different metadata catalogs) that export the same interface as used within theGAE.
Requirement 9, Policies: It should not be possible for users or groups to allocate more resources
on the Grid than they are allowed to. There should be a mechanism in the Grid that allowsorganizations to specify policies and that monitors the use of resources by users and groups based
on these policies. Policies will describe a measure of “fair use of Grid resources” for all users andgroups involved (provided all uses agree on this policy). It should be noted that deciding on a policycan not be solved by any piece of software but is subject to discussion between groups an users ofthat policy
4 Architecture
The first sub section describes the high level view of the architecture, while the second sub sectionidentifies the different services that are needed for interactive analysis within such a high levelarchitecture.
4.1 Peer Groups
The LHC experiments computing models were initially based on a hierarchical organization of Tier
0, multiple Tier 1's and multiple Tier 2's, etc . Most of the data will “flow” from the Tier 0, to Tier1's and fromTier 1's to Tier 2's. Furthermore, Tier 0 and Tier 1 are powerful in terms of cpu power,data storage, and bandwidth. This hierarchical model has been described in “Data Intensive Gridsfor highenergy physics” [11] . Data is organized such that the institutes (represented by tiers) will
be (in most cases) physically close to the data they want to analyze
The hierarchical model described in [11] is a good starting point, but there will be a certaindynamics within the LHC computing/data model. Although it is possible to make predictions onhow to organize data, the patterns of the end users for analysis will be unpredictable. Depending
on how “hot” or “cold” data is, attention will shift to different data sets. Furthermore multiplegeographically dispersed users might be interested in data that is geographically dispersed on thedifferent Tier's. This geographically dispersed user group for particular data can lead to datareplication in order to prevent “hot spots” in data access
Trang 9Figure 4.1 shows a modification of the hierarchical model: hierarchical peer to peer model. In whichthe different tier x centers act as peers. The thickness of the arrows represent the data movementbetween peers while the size of the peer represents its power in terms of cpu, and storage. Thegreen part shows the number of resources that is idle, yellow: resources being used, red:resources that are offline. Associated with every peer are data. Red colored data represents “hot”data that is accessed often. Blue colored data represents “cold” data that is accessed lessfrequent.
The hierarchical model is the basis on which data will be distributed, however the unpredictablebehavior of physics analysis as a whole will lead to data and jobs being moved around betweendifferent peers These data movements and job movements outside the hierarchical modelalthough relative small compared to the data movement within the hierarchical model, will besubstantial. When users submit jobs to a peer, middleware will discover what resources areavailable on other peers to execute this job request. Although a large number of job requests willfollow the hierarchical model, other job requests will not. The more powerful peers (super peers)will receive more job requests, and data requests and will host a wider variety of services. Withinthe figure 4.1 T0 and T1's and 1 T2 act as super peers
Trang 10The services developed within GAE should be robust and scalable enough to be deployed in the(hierarchical) peer to peer model
4.2 Services
Based on discussion on interactive analysis (section 2) , use cases mentioned in HEPCAL and,requirements it is possible to identify a set of core services for GAE
Within this section the word service will be used to describe the different components within thearchitecture, because these components will be implemented as a web service within GAE. Whilethey have been selected due to their necessity for an interactive Grid environment, many of theseservices could be used in other Grid environments. In order to satisfy fault tolerance requirementsall services must be able to operate asynchronously
The following services (components) have been identified for GAE:
Signon and Virtual Organization: These services provide authentication, authorization, and
access control for data and computing resources. Virtual Organization management services arealso provided to assist in maintaining and modifying the membership of virtual organizations.Virtual organization service relates to requirement 7 and 9 on security and policy
Data Catalog / Virtual Data: These services provide data lookup capabilities They allow
datasets to be looked up based on Physics Expressions or other meta data. These services returnlists of Logical Filenames. The various HEP experiments will have their own implementations ofthis service to handle the experimentspecific metadata and dataset naming
Lookup: Lookup services are the main entry point to access the Grid. Locations of the various
services are never known in advance; they must be located using a lookup service. The lookupservices allow dynamic lookup of the Grid services, eliminating the need to know the servicelocations in advance. Decentralized peer to peer style technologies are key to the operation of thelookup services to keep their service listings accurate and up to date.
Three possible architectures for lookup service implementations are described below Eacharchitecture builds upon the previous. This makes it is possible to implement the simplest solution
Trang 11The first architecture a centralized lookup service. All clients would contact the centralized service
to locate existing service implementations. This would not fit with the decentralized approach withinGAE
A second architecture is similar to the first, except that the lookup service information is replicatedacross many lookup service instances. When a new service instance is registered on one lookupservice, the lookup service will propagate the registration to all other known lookup services. Newlookup service hosts are added to the system just as any service instance is added. Lookupservice hosts will have to ping other service hosts periodically to ensure that their list of services iskept up to date in case of a failure in the propagation of a service registration. An advantage to thisarchitecture is that service lookups should be very fast, as the lookup happens locally. A drawback
is that as the number of available service instances grows, so does the amount of data replicated
on each lookup service host
A third architecture involves distributed queries and distributed service location storage. When anew service instance is registered with a lookup service, the registration stays local; it is notpropagated to other lookup service hosts. When a lookup request is received by a lookup service, itcan no longer respond authoritatively to the request. Instead, it has to propagate the request on toother lookup services, collect responses, and return them to the requester. This architecture ismore fully distributed than the first two architectures. An advantage is that the local serviceinformation storage remains small. But the disadvantage is that the response time for a lookuprequest increases as multiple lookup services are queried
It has yet to be determined whether the second or third architectures would be more ideal.Measurements on the responsiveness of the system with respect to number of services andnumber of service hosts need to be taken before a decision can be made. Lookup services create
a more transparent Grid environment as users and applications are able to locate and identifydifferent services within a dynamic distributed environment as mentioned in requirement 5
Processing Wrapper: The details of using many of the services together should be hidden from
the application. When submitting a processing job request, for example, the application should nothave to contact the Meta data Catalog, Job Scheduler, and the Job/Process Execution servicesdirectly. To simplify these steps, a Processing Wrapper service will be used. This service collectsall of the inputs that would normally be given to the Meta Data Catalog, Job Scheduler, and JPESservices. It then contacts these other services on behalf of the application
Scheduler: This service resolves an abstract job plan into a concrete job execution plan for Grid
applications. It also submits the concrete plan to the grid for execution. The abstract job plan is abreakdown of the Individual actions and the order in which they will occur. The process execution
Trang 12plan for an interactive job may be as simple as “allocate n interactive execution nodes”. The
abstract and concrete plans used by the scheduler must be usable by any scheduler on the systemfor execution; they are not specific to the scheduler that produced them. This reduces the severity
of a failure of a scheduling server
Replica Management: These services provide the capability to copy and move data around the
Grid. One particular use would be to replicate remote data to a local staging location for moreefficient processing. The Replica Management service makes use of lower level Replica Location,Meta data, and Replica Optimization services to control the movement of data around the Grid.Replica management will prevent bottlenecks in data access on the grid and lowers the chance oflosing valuable data As such replica management adds to Grid robustness and scalability(requirement 4 and 5)
Replica Selection: This service is used to locate the optimum replica to use for processing. It is
used primarily by the Scheduling service in conjunction with the Execution Location service todetermine the optimum locations for staging and processing data.
Steering/Optimization: The client application obtains a handle to a steering service once the
concrete job plan is submitted to the scheduler for execution. The steering service provides a set
of APIs for sending commands to interactive execution nodes and tuning concrete job processes.These APIs will allow jobs to be transferred to different execution nodes and allow data to bereplicated between hosts. The steering service acts as a proxy between the client and thescheduler. The steering service also provides a message subscription API (such as Java MessageService [12] ) to allow the client to subscribe to specific monitoring and job update notices. Thesteering service is responsible for routing the appropriate messages from the various monitorsback to the client. Intelligent agents can also use the steering service to make decisions on behalf
of the user, or the greater Grid community. If a failure in the Steering service occurs then the clientapplication can submit the concrete job plan to another Steering service instance. The newsteering service will contact any active job tasks (as indicated by the concrete job plan) andresubscribe the client to the monitoring information
Trang 13Steering and optimization allow job requests to adapt to the dynamic environment of the Grid.Adaptability was discussed in requirement 2
Monitor/Workflow: These services provide information on the current state of the job plan. A
publish/subscribe messaging API (such as JMS [12] ) is used to register listeners to monitoringdata. The Steering service uses Monitors on the execution nodes to send job status feedback tothe user. A monitoring service should not only keep track of the state of the job, but also about thestate of the resources on which jobs are submitted (Grid weather). This information can then beused by steering services to take decisions on moving jobs around (self organization). Monitoringenables diagnosing and analyzing Grid resources and provides transparency (requirement 5 andrequirement 6)
Quota/Accounting:
The system needs to be able to satisfy all jobs by allocating storage, computer and networkresources in a fashion that satisfies both global and local constraints. Global constraints includecommunitywide policies governing how resources should be prioritized and allocated Localconstraints include sitespecific control as to when or how much external users are allowed uselocal resources. For example, individual resource providers (as well as a Virtual Organization as awhole) may assign usage quotas to intended users and job monitoring mechanisms may be used
to tally accounting information for the user. As such, the architecture requires services for storingquota and accounting information so that other services (such as a scheduling service or a steeringservice) may mine that information for enforcement strategies. Due to the distributed nature of theproblem, enforcement of global constraints would necessarily be "soft" and on a besteffort basis,whilst enforcement of local constraints would be enforced by the local site itself and thus be "hard."Accounting enables a “fair” access to resources as specified by policies (requirement 9)
Estimators: Estimators let the application know, in advance, how much resource (time, space,
etc.) a particular action might take, and provide a “progress meter” to the application. An estimatorservice can use past performance as a predictive measure of future performance and canextrapolate from the current action's execution status. Most services will use estimators to give
Trang 14feedback to users, applications, and agents. Estimators are not necessarily a service untothemselves; they may be part of the service API of the other services. For example, the ReplicaManagement Service may contain an estimator as part of its API for moving data around the Grid.Estimators relate to requirement 5.
Job/Process Execution: Executes a set of jobs as part of a concrete job plan. The service
provides job progress and estimate updates to the steering service. The Steering service collectsthe information from all of the JPES involved in the user's session and sends the collection ofupdates back to the user
Data Collection: Provides a way for the application to obtain the final result from the execution of
the job/process execution service, and combines the data that has been generated on multiplelocations. The result will be either a logical dataset name for large result sets or the actual resultdata for small result sets (as requested by the client). The result should be a Logical Dataset whichcan then be mapped to a physical dataset using the Replica Management service. The datacollection service also collects intermediate results from the individual job/process plan steps
Figure 4.1. GAE services
Trang 154.3 Self organization
Self organization means that components withing GAE make decisions based on monitorinformation without intervention of humans Within several components of GAE autonomousbehavior can be inserted to create a “self organizing” Grid:
• Steering/optimization: Monitor the Grid weather and decide if a job can be moved to othercomputing nodes based on policies for this job
• Scheduling: Keep track of history of scheduling decisions and how “good” these decisions were
• Replica manager: Keep track of access patterns of data and base replica strategy on it
5 Scenarios
Based on the use cases in [7], [6], [9] , requirements and components described above it ispossible to formulate several scenarios that show the interaction of the GAE components. It is notthe intention to give an exhaustive list of scenarios, but rather a few, that identify core componentinteractions . One scenario related to state information within GAE has been described in section2.2.
5.1 Scenario 2: A simple interactive session from a single user point of view
This scenario is based on a description of the analysis activity described in section 3 of theHEPCALII document [6]:
1) Application contacts Virtual Organization Service to authenticate. An authentication token isreturned and passed to other Grid services to determine authorization of service use at a locallevel
2) Application uses a Lookup Service to locate a suitable Process Wrapper service on the Grid.3) Application sends an abstract job plan to the Process Wrapper. This abstract plan can be asabstract or concrete as necessary. If the user knows in advance where data is stored or whichexecution nodes must be used, then that information can be included in the plan. Or the plancan leave these as abstract entries to be resolved by the scheduler. The Process Wrapper acts
as an intermediary between the scheduler, the execution service, and the metadata service,eliminating the need for the application to have to communicate with these services directly.4) Process Wrapper uses a lookup service to locate Scheduling and Metadata services on the
Trang 165) A process wrapper sends the abstract plan to the scheduler. The scheduler further resolves theabstract plan into a concrete plan by locating logical file replica and available execution sites.6) The scheduler contacts the accounting service to make an estimate of how many resources thisuser/group has in use and contacts estimator services to get an estimate on how manyresources and for what period this job will use
7) The scheduler sends the concrete job plan back to the user for approval. This gives the user achance to verify the plan and view the estimate of resource usages
8) The scheduler sends a copy of the job plan to a steering service, and sends a handle to thesteering service back to the user
9) The scheduler initiates any data movement on the grid and starts submitting jobs to the localexecution services. Data processing begins.
10)As processing is happening, monitoring services return status information back to theapplication (via a steering service) so that the user gets real time feedback on the state of theprocessing. Steering services are used to modify the plan, resubmit processing tasks, and finetune the processing. The Job/Process Execution Service returns partial data results to a DataCollection Service. New interactive tasks are submitted iteratively
11)When processing is completed, the Data Collection Service returns the final Logical DatasetFilename to the application including a description of a state description of the job that can beused for further processing
12)User reviews the results and submits a new set of processing instructions for furtherprocessing
Within GAE there will be hundreds of users that would perform the above scenario several timesfor an interactive session in a decentralized unpredicted manner
Figure 5.1 shows the set of interacting web services that comprise GAE as described above,combined with a typical execution flow within the system. The numbers in figure 5.1 correspondwith the numbers of the scenario 2 of section 5. The colors in figure 5.1 highlight the differentclasses as shown in figure 4.1. Within figure 5.1 service can reside on different grid service hosts.The user is not aware how these service are distributed. Figure 5.1 is a snapshot in time asservices can disappear (failure, obsolete), or being duplicated at other grid service hosts,dynamically in time The grid service hosts act as peers and super peers within the gridenvironment