Implementing Remote Procedure Calls

Implementing Remote Procedure Calls

Implementing Remote Procedure Calls

ANDREW D BIRRELL and BRUCE JAY NELSON

Xerox Palo Alto Research Center

Remote procedure calls (RPC) appear to be a useful paradig m for providing communication across a network between programs written in a high-level language This paper describes a package providing

a remote procedure call facility, the options that face the designer of such a package, and the decisions

~we made We describe the overall structure of our RPC mechanism, our facilities for binding RPC clients, the transport level communication protocol, and some performance measurements We include descriptioro~ of some optimizations used to achieve high performance and to minimize the load on server machines that have many clients

CR Categories and Subject Descriptors: C.2.2 [ C o m p u t e r - C o m m u n i c a t i o n N e t w o r k s ] : Network

Protocols protocol architecture; C.2.4 [Computer-Communication Networks]: Distributed Sys-

tems-distributed applications, network operating systems; D.4.4 [ O p e r a t i n g Systems]: Communi- cations Management message sending, network communication; D.4.7[Operatiug Systems]: Or- ganization and Design distributed systems

General Terms: Design, Experimentation, Performance, Security

Additional Keywords and Phrases: Remote procedure calls, transport layer protocols, distributed naming and binding, inter-process communication, performance of communication protocols

1 INTRODUCTION

1.1 Background

The idea of remote procedure calls (hereinafter called RPC) is quite simple It is

based on the observation that procedure calls are a well-known and well- understood mechanism for transfer of control and data within a program running

on a single computer Therefore, it is proposed that this same mechanism be extended to provide for transfer of control and data across a communication network When a remote procedure is invoked, the calling environment is suspended, the parameters are passed across the network to the environment where the procedure is to execute (which we will refer to as the callee), and the

desired procedure is executed there When the procedure finishes and produces its results, the results are passed backed to the calling environment, where execution resumes as if returning from a simple single-machine call While the calling environment is suspended, other processes on that machine may (possibly)

Authors' address: Xerox Palo Alto Research Center, 3333 Coyote Hill Road, Palo Alto, CA 94304 Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery To copy otherwise, or to republish, requires a fee and/or specific permission

© 1984 ACM 0734-2071/84/0200-0039 $00.75

ACM Transactions on Computer Systems, Vol 2, No 1, February 1984, Pages 39-59

40 • A D Birrell and B J Nelson

still execute (depending on the details of the parallelism of that environment and the RPC implementation)

There are many attractive aspects to this idea One is clean and simple semantics: these should make it easier to build distributed computations, and to get them right Another is efficiency: procedure calls seem simple enough for the communication to be quite rapid A third is generality: in singie-machine com- putations, procedures are often the most important mechanism for communica- tion between parts of the algorithm

The idea of R P C has been around for many years It has been discussed in the public literature many times since at least as far back as 1976 [15] Nelson's doctoral dissertation [13] is an extensive examination of the design possibilities for an RPC system and has references to much of the previous work on RPC However, full-scale implementations of RPC have been rarer than paper designs

Notable recent efforts include Courier in the Xerox NS family of protocols [4],

and current work at M I T [10]

This paper results from the construction of an RPC facility for the Cedar

project We felt, because of earlier work (particularly Nelson's thesis and asso- ciated experiments), that we understood the choices the designer of an R P C facility must make Our task was to make the choices in light of our particular aims and environment In practice, we found that several areas were inadequately understood, and we produced a system whose design has several novel aspects Major issues facing the designer of an R P C facility include: the precise semantics

of a call in the presence of machine and communication failures; the semantics

of address-containing arguments in the (possible) absence of a shared address space; integration of remote calls into existing (or future) programming systems; binding (how a caller determines the location and identity of the callee); suitable protocols for transfer of data and control between caller and callee; and how to provide data integrity and security (if desired) in an open communication network In building our R P C package we addressed each of these issues, but it not possible to describe all of them in suitable depth in a single paper This paper includes a discussion of the issues and our major decisions about them, and describes the overall structure of our solution We also describe in some detail our binding mechanism and our transport level communication protocol We plan to produce subsequent papers describing our facilities for encryption-based

security, and providing more information about the manufacture of the stub

modules (which are responsible for the interpretation of arguments and results

of RPC calls) and our experiences with practical use of this facility

1.2 Environment

The remote-procedure-call package we have built was developed primarily for use within the Cedar programming environment, communicating across the Xerox research internetwork In building such a package, some characteristics of the environment inevitably have an impact on the design, so the environment is summarized here

Cedar [6] is a large project concerned with developing a programming environ- ment that is powerful and convenient for the building of experimental programs and systems There is an emphasis on uniform, highly interactive user interfaces, and ease of construction and debugging of programs Cedar is designed to be used

Implementing Remote Procedure Calls • 41

on single-user workstations, although it is also used for the construction of servers (shared computers providing common services, accessible through the communication network)

Most of the computers used for Cedar are Dorados [8] The Dorado is a very

powerful machine (e.g., a simple Algol-style call and return takes less than 10 microseconds) It is equipped with a 24-bit virtual address space (of 16-bit words) and an 80-megabyte disk Think of a Dorado as having the power of an IBM 370/168 processor, dedicated to a single user

Communication between these computers is typically by means of a 3-megabit- per-second Ethernet [11] {Some computers are on a 10-megabit-per-second Ethernet [7].) Most of the computers running Cedar are on the same Ethernet, but some are on different Ethernets elsewhere in our research internetwork The internetwork consists of a large number of 3-megabyte and 10-megabyte Ether- nets (presently about 160) connected by leased telephone and satellite links (at data rates of between 4800 and 56000 bps) We envisage that our RPC commu- nication will follow the pattern we have experienced with other protocols: most communication is on the local Ethernet (so the much lower data rates of the internet links are not an inconvenience to our users), and the Ethernets are not overloaded (we very rarely see offered loads above 40 percent of the capacity of

an Ethernet, and 10 percent is typical)

The P U P family of protocols [3] provides uniform access to any computer on this internetwork Previous P U P protocols include simple unreliable (but high- probability) datagram service, and reliable flow-controlled byte streams Between two computers on the same Ethernet, the lower level raw Ethernet packet format

is available

Essentially all programming is in high-level languages The dominant language

is Mesa [12] (as modified for the purposes of Cedar), although Smalltalk and InterLisp are also used There is no assembly language for Dorados

1.3 Aims

The primary purpose of our RPC project was to make distributed computation easy Previously, it was observed within our research community that the con- struction of communicating programs was a difficult task, undertaken only by members of a select group of communication experts Even researchers with substantial systems experience found it difficult to acquire the specialized exper- tise required to build distributed systems with existing tools This seemed undesirable We have available to us a very large, very powerful communication network, numerous powerful computers, and an environment that makes building programs relatively easy The existing communication mechanisms appeared to

be a major factor constraining further development of distributed computing Our hope is that by providing communication with almost as much ease as local procedure calls, people will be encouraged to build and experiment with distrib- uted applications RPC will, we hope, remove unnecessary difficulties, leaving only the fundamental difficulties of building distributed systems: timing, inde- pendent failure of components, and the coexistence of independent execution environments

We had two secondary aims that we hoped would support our purpose We wanted to make R P C communication highly efficient (within, say, a factor of

42 • A.D Birrell and B J Nelson

five beyond the necessary transmission times of the network) This seems important, lest communication become so expensive that application designers strenuously avoid it The applications that might otherwise get developed would

be distorted by their desire to avoid communicating Additionally, we felt that it was important to make the semantics of the RPC package as powerful as possible, without loss of simplicity or efficiency Otherwise, the gains of a single unified communication paradigm would be lost by requiring application programmers to build extra mechanisms on top of the R P C package An important issue in design

is resolving the tension between powerful semantics and efficiency

Our final major aim was to provide secure communication with RPC None of the previously implemented protocols had any provision for protecting the data

in transit on our networks This was true even to the extent that passwords were transmitted as clear-text Our belief was that research on the protocols and mechanisms for secure communication across an open network had reached a stage where it was reasonable and desirable for us to include this protection in our package In addition, very few (if any) distributed systems had previously provided secure end-to-end communication, and it had never been applied to RPC, so the design might provide useful research insights

1.4 Fundamental Decisions

It is not an immediate consequence of our aims that we should use procedure calls as the paradigm for expressing control and data transfers For example, message passing might be a plausible alternative It is our belief that a choice between these alternatives would not make a major difference in the problems faced by this design, nor in the solutions adopted The problems of reliable and efficient transmission of a message and of its possible reply are quite similar to the problems encountered for remote procedure calls The problems of passing arguments and results, and of network security, are essentialy unchanged The overriding consideration that made us choose procedure calls was that they were the major control and data transfer mechanism imbedded in our major language, Mesa

One might also consider using a more parallel paradigm for our communication, such as some form of remote fork Since our language already includes a construct for forking parallel computations, we could have chosen this as the point at which

to add communication semantics Again, this would not have changed the major design problems significantly

We discarded the possibility of emulating some form of shared address space among the computers Previous work has shown that with sufficient care mod- erate efficiency can be achieved in doing this [14] We do not know whether an approach employing shared addresses is feasible, but two potentially major diffÉculties spring to mind: first, whether the representation of remote addresses can be integrated into our programming languages (and possibly the underlying machine architecture) without undue upheaval; second, whether acceptable effi- ciency can be achieved For example, a host in the P U P internet is represented

by a 16-bit address, so a naive implementation of a shared address space would extend the width of language addresses by 16-bits On the other hand, it is possible that careful use of the address-mapping mechanisms of our virtual memory hardware could allow shared address space without changing the address

Implementing Remote Procedure Calls • 43

width Even on our 10 megabit Ethernets, the m i n i m u m average round trip time for a packet exchange is 120 microseconds [7], so the most likely way to approach this would be to use some form of paging system In summary, a shared address space between participants in R P C might be feasible, but since we were not willing to undertake that research our subsequent design assumes the absence of shared addresses Our intuition is that with our hardware the cost of a shared address space would exceed the additional benefits

A principle that we used several times in making design choices is that the semantics of remote procedure calls should be as close as possible to those of local (single-machine) procedure calls This principle seems attractive as a way

of ensuring that the R P C facility is easy to use, particularly for programmers familiar with single-machine use of our languages and packages Violation of this principle seemed likely to lead us into the complexities that have m a d e previous communication packages and protocols difficult to use This principle has occa- sionally caused us to deviate from designs that would seem attractive to those more experienced in distributed computing For example, we chose to have no time-out mechanism limiting the duration of a remote call (in the absence of machine or communication failures), whereas most communication packages consider this a worthwhile feature Our argument is that local procedure calls have no time-out mechanism, and our languages include mechanisms to abort an activity as part of the parallel processing mechanism Designing a n e w time-out arrangement just for R P C would needlessly complicate the programmer's world Similarly, we chose the building semantics described below (based closely on the existing Cedar mechanisms) in preference to the ones presented in Nelson's thesis [13]

1.5 Structure

T h e program structure we use for R P C is similar to that proposed in Nelson's thesis It is based on the concept of stubs W h e n making a remote call, five pieces

of program are involved: the user, the user-stub, the R P C communications

package (known as RPCRuntime), the server-stub, and the server Their relatidn-

ship is shown in Figure 1 T h e user, the user-stub, and one instance of R P C R u n - time execute in the caller machine; the server, the server-stub and another instance of R P C R u n t i m e execute in the callee machine W h e n the user wishes

to m a k e a remote call, it actually makes a perfectly normal local call which invokes a corresponding procedure in the user-stub T h e user-stub is responsible for placing a specification of the target procedure and the arguments into one or more packets and asking the R P C R u n t i m e to transmit these reliably to the callee machine O n receipt of these packets, the R P C R u n t i m e in the callee machine passes them to the server-stub T h e server-stub unpacks them and again makes

a perfectly normal local call, which invokes the appropriate procedure in the server Meanwhile, the calling process in the caller machine is suspended awaiting

a result packet W h e n the call in the server completes, it returns to the server- stub and the results are passed back to the suspended process in the caller machine There they are unpacked and the user-stub returns them to the user

R P C R u n t i m e is responsible for retransmissions, acknowledgments, packet rout- ing, and encryption Apart from the effects of multimachine binding and of machine or communication failures, the call happens just as if the user had

44 • A D Birrell and B J Nelson

User User-stub RPCRuntime

transmit

Fig 1 The components of the system, and their interactions for a simple call

invoked the procedure in the server directly Indeed, if the user and server code were brought into a single machine and bound directly together without the stubs, the program would still work

R P C R u n t i m e is a standard part of the Cedar system The user and server are written as part of the distributed application But the user-stub and server-stub are automatically generated, by a program called Lupine This generation is specified by use of Mesa interface modules These are the basis of the Mesa (and Cedar) separate compilation and binding mechanism [9] An interface module is mainly a list of procedure names, together with the types of their arguments and results This is sufficient information for the caller and callee to independently perform compile-time type checking and to generate appropriate calling se- quences A program module that implements procedures in an interface is said to

export that interface A program module calling procedures from an interface is said to import that interface When writing a distributed application, a program- mer first writes an interface module Then he can write the user code that imports that interface and the server code that exports the interface He also presents the interface to Lupine, which generates the user-stub, (that exports the interface) and the server-stub {that imports the interface) When binding the programs on the 'caller machine, the user is bound to the user-stub On the callee machine, the server-stub is bound to the server

Thus, the programmer does not need to build detailed communication-related code After designing the interface, he need only write the user and server code Lupine is responsible for generating the code for packing and unpacking argu- ments and results (and other details of parameter/result semantics), and for dispatching to the correct procedure for an incoming call in the server-stub RPCRuntime is responsible for packet-level communications The programmer must avoid specifying arguments or results that are incompatible with the lack

of shared address space (Lupine checks this avoidance.) The programmer must also take steps to invoke the intermachine binding described in Section 2, and to handle reported machine or communication failures

2 BINDING

There are two aspects to binding which we consider in turn First, how does a client of the binding mechanism specify what he wants to be bound to? Second, ACM Transactions on Computer Systems, Vol 2, No 1, February 1984

Implementing Remote Procedure Calls • 45

how does a caller determine the machine address of the callee and specify to the callee the procedure to be invoked? The first is primarily a question of naming

and the second a question of location

2.1 Naming

The binding operation offered by our RPC package is to bind an importer of an interface to an exporter of an interface After binding, calls made by the importer invoke procedures implemented by the (remote) exporter There are two parts to the name of an interface: the type and the instance The type is intended to specify, at some level of abstraction, which interface the caller expects the callee

to implement The instance is intended to specify which particular implementor

of an abstract interface is desired For example, the type of an interface might correspond to the abstraction of "mail server," and the instance would correspond

to some particular mail server selected from many A reasonable default for the type of an interface might be a name derived from the name of the Mesa interface module Fundamentally, the semantics of an interface name are not dictated by the RPC package they are an agreement between the exporter and the importer, not fully enforceable by the RPC package However, the means by which an exporter uses the interface name to locate an exporter are dictated by the RPC package, and these we now describe

2.2 Locating an Appropriate Exporter

We use the Grapevine distributed database [1] for our RPC binding The major attraction of using Grapevine is t h a t it is widely and reliably available Grapevine

is distributed across multiple servers strategically located in our internet topology, and is configured to maintain at least three copies of each database entry Since the Grapevine servers themselves are highly reliable and the data is replicated,

it is extremely rare for us to be unable to look up a database entry There are alternatives to using such a database, but we find them unsatisfactory For example, we could include in our application programs the network addresses of the machine with which they wish to communicate: this would bind to a particular machine much too early for most applications Alternatively, we could use some form of broadcast protocol to locate the desired machine: this would sometimes

be acceptable, but as a general mechanism would cause too much interference with innocent bystanders, and would not be convenient for binding to machines not on the same local network

Grapevine's database consists of a set of entries, each keyed by a character string known as a Grapevine RName There are two varieties of entries: individ-

ua/s and groups Grapevine keeps several items of information for each database entry, but the RPC package is concerned with only two: for each individual there

is a connect-site, which is a network address, and for each group there is a

member-list, which is a list of RNames The RPC package maintains two entries

in the Grapevine database for each interface name: one for each type and one for each instance; so the type and instance are both Grapevine RNames The database entry for the instance is a Grapevine individual whose connect-site is a network address, specifically, the network address of the machine on which t h a t instance was last exported The database entry for the type is a Grapevine group whose members are the Grapevine RNames of the instances of t h a t type which

46 • A.D Birrell and B J Nelson

have been exported For example, if the remote interface with type FileAccess.Alpine and instance Ebbets.Alpine has been exported by a server running at network address 3 # 2 2 # , and the remote interface with type FileAccess.Alpine and instance Luther.Alpine has been exported by a server running at network address 3 # 2 7 6 # , then the members of the Grapevine group FileAccess.Alpine would include Ebbets.Alpine and Luther.Alpine The Grapevine individual Ebbets Alpine would have 3 # 2 2 # as its connect-site and Luther.Alpine would have 3 # 2 7 6 #

When an exporter wishes to make his interface available to remote clients, the server code calls the server-stub which in turn calls a procedure, Exportlnterface,

in the RPCRuntime Exportlnterface is given the interface name (type and instance) together with a procedure (known as the dispatcher) implemented in the server-stub which will handle incoming calls for the interface Exportlnterface calls Grapevine and ensures that the instance is one of the members of the Grapevine group which is the type, and that the connect-site of (the Grapevine individual which is) the instance is the network address of the exporting machine This may involve updating the database As an optimization, the database is not updated if it already contains the correct information this is usually true: typically an interface of this name has previously been exported, and typically from the same network address For example, to export the interface with type FileAccess.Alpine and instance Ebbets.Alpine from network address 3 # 2 2 # , the

R P C R u n t i m e would ensure that Ebbets.Alpine in the Grapevine database has connect-site 3 # 2 2 # and that Ebbets.Alpine is a member of FileAccess.Alpine The R P C R u n t i m e then records information about this export in a table main- tained on the exporting machine For each currently exported interface, this table contains the interface name, the dispatcher procedure from the server-stub, and

a 32-bit value that serves as a permanently unique (machine-relative) identifier

of the export This table is~implemented as an array indexed by a small integer The identifier is guaranteed to be permanently unique by the use of successive values of a 32-bit counter; on start-up this counter is initialized to a one-second real time clock, and the counter is constrained subsequently to be less than the current value of that clock This constrains the rate of calls on Exportlnterface

in a single machine tO an average rate of less than one per second, averaged over the time since the exporting machine was restarted The burst rate of such calls can exceed one per second (see Figure 2)

When an importer wishes to bind to an exporter, the user code calls its user- stub which in turn calls a procedure, Importlnterface, in the RPCRuntime, giving

it the desired interface type and instance The R P C R u n t i m e determines the network address of the exporter (if there is one) by asking Grapevine for the network address which is the connect-site of the interface instance The RPCRuntime then makes a remote procedure call to the R P C R u n t i m e p a c k a g e

on that machine asking for the binding information associated with this interface type and instance If the specified machine is not currently exporting that interface this fact is returned to the importing machine and the binding fails If the specified machine is currently exporting that interface, then the table of current exports maintained by its R P C R u n t i m e yields the corresponding unique identifier; the identifier and the table index are returned to the importing machine

ACM Transactions on Computer Systems, Vol 2, No 1, February 1984

Implementing Remote Procedure Calls • 47

User User-stub RPCRuntime

Table lookup

Fig 2 The sequence of events in binding and a subsequent call The callee machine exports the remote interface with type A and instance B The caller machine then imports that interface We then show the caller initiating a call to procedure F, which is the third procedure of that interface The return is not shown

and the binding succeeds The exporter network address, identifier, and table index are remembered by the user-stub for use in remote calls

Subsequently, when that user-stub is making a call on the imported remote interface, the call packet it manufactures contains the unique identifier and table index of the desired interface, and the entry point number of the desired procedure relative to the interface When the RPCRuntime on the callee machine receives

a new call packet it uses the index to look up its table of current exports (efficiently), verifies that the unique identifier in the packet matches that in the table, and passes the call packet to the dispatcher procedure specified in the table

There are several variants of this binding scheme available to our clients If the importer calling Importlnterface specifies only the interface type but no instance, the RPCRuntime obtains from Grapevine the members of the Grape- vine group named by the type The RPCRuntime then obtains the network address for each of those Grapevine individuals, and tries the addresses in turn

to find some instance that will accept the binding request: this is done efficiently,

ACM Transactions on Computer Systems, Vol 2, No 1, February 1984

48 • A D Birrell and B J Nelson

and in an order which tends to locate the closest (most responsive) running exporter This allows an importer to become bound to the closest running instance

of a replicated service, where the importer does not care which instance Of course, an importer is free to enumerate the instances himself, by enumerating the members of the group named by the type

The instance may be a network address constant instead of a Grapevine name This would allow the importer to bind to the exporter without any interaction with Grapevine, at the cost of including an explicit address in the application programs

2.3 Discussion

There are some important effects of this scheme Notice t h a t importing an interface has no effect on the data structures in the exporting machine; this is advantageous when building servers t h a t may have hundreds of users, and avoids problems regarding what the server should do about this information in relation

to subsequent importer crashes Also, use of the unique identifier scheme means

t h a t bindings are implicitly broken if the exporter crashes and restarts (since the currency of the identifier is checked on each call) We believe t h a t this implicit unbinding is the correct semantics: otherwise a user will not be notified of a crash happening between calls Finally, note t h a t this scheme allows calls to be made only on procedures t h a t have been explicitly exported through the RPC mechanism An alternate, slightly more efficient scheme would be to issue importers with the exporter's internal representation of the server-stub dis- patcher procedure; this we considered undesirable since it would allow unchecked access to almost any procedure in the server machine and, therefore, would make

it impossible to enforce any protection or security schemes

The access controls t h a t restrict updates to the Grapevine database have the effect of restricting the set of users who will be able to export particular interface names These are the desired semantics: it should not be possible, for example, for a random user to claim t h a t his workstation is a mail server and to thereby

be able to intercept my message traffic In the case of a replicated service, this access control effect is critical A client of a replicated service may not know a priori the names of the instances of the service If the client wishes to use two- way authentication to get the assurance t h a t the service is genuine, and if we wish to avoid using a single password for identifying every instance of the service, then the client must be able to securely obtain the list of names of the instances

of the service We can achieve this security by employing a secure protocol when the client interacts with Grapevine as the interface is being imported Thus Grapevine's access controls provide the client's assurance t h a t an instance of the service is genuine (authorized)

We have allowed several choices for binding time The most flexible is where the importer specifies only the type of the interface and not its instance: here the decision about the interface instance is made dynamically Next (and most common) is where the interface instance is an RName, delaying the choice of a particular exporting machine Most restrictive is the facility to specify a network address as an instance, thus binding it to a particular machine at compile time

We also provide facilities allowing an importer to dynamically instantiate inter- faces and to import them A detailed description of how this is done would be