Choosing a Hot-Site Vendor

SECTION VI BUSINESS CONTINUITY MANAGER’S TOOL KIT

Chapter 29 Choosing a Hot-Site Vendor

Philip Jan Rothstein

ANYORGANIZATION THATHAS DECIDED TO BACKUPCRITICAL IT OPER- ATIONSMUSTDECIDEHOWITWILLDOSO. Two common methods of operations backup are hot sites and cold sites. Hot sites are fully equipped, ready-to-run computer centers designed to be activated when a subscriber declares that a disaster has occurred. Cold sites are ready-to-use facilities without computers or communications equipment in place. Other options include using company-owned facilities. Although using a hot site can be an expensive option, an organization whose survival depends on its computer processing capability may find that it is ultimately the most cost- effective choice. This chapter provides some guidelines on choosing a hot- site vendor.

TYPES OF VENDORS

There are basically four categories of hot-site vendors. The first consists of organizations that are dedicated to disaster recovery. This includes the three companies that constitute the majority of the hot-site market, Com- disco Disaster Recovery Services, Inc., Sungard Recovery Services, Inc., and IBM Business Recovery Services. Each of these companies offers multiple locations and classes of service.

The second category of hot-site offerings comprises hardware vendors (other than IBM), including Hewlett-Packard, which provide recovery services primarily for their customers. Hewlett-Packard has alliances with other organizations to support multiplatform recovery capabilities.

The third category consists of regional, local, or specialty hot-site vendors. Companies that depend on multivendor computing environments or complex communications networks may find the hot-site services of many of these vendors to be too limited.

The fourth category is composed of mobile recovery sites. Mobile recovery sites provide the convenience of having a self-contained backup

0-8493-0907-7/00/$0.00+$.50

AU0907/frame/ch29 Page 303 Monday, July 31, 2000 4:54 PM

BUSINESS CONTINUITY MANAGER’S TOOL KIT

computer facility trucked or airlifted to or near the subscriber’s premises.

The time delay for transportation may or may not be acceptable. CSC Prov- ident Mobile Recovery Systems, Sungard Recovery Services, XL/Data- comp, Inc., and Comdisco Disaster Recovery Services are among the firms that offer relocatable recovery sites. Some of the minicomputer equipment vendors also have truck- or van-mounted emergency systems.

INDUSTRY OVERVIEW

Since this chapter was last updated, the business continuity hot-site industry has continued an accelerating trend toward consolidation. Three major vendors constitute the vast majority of the industry segment: Com- disco Disaster Recovery Services, IBM Business Recovery Services, and Sun- gard Recovery Services. The remaining independent hot-site vendors con- sist of either computer hardware vendor captives (e.g., Hewlett-Packard) or

“boutique” vendors focusing on either specific niches (trading floor recovery, mailing recovery, specialized computer environments) or specific geo- graphic regions. Many of these captives or boutiques have formed alliances with one or more of the three major providers.

The second, accelerating trend since this chapter was last updated is the broadening of hot-site services to increasingly address mid-frame, client/server, and distributed computing environments, complex networks, and work-area recovery. As a result, the ability to handle complex, techno- logically sophisticated hot-site recovery capabilities at all may constrain vendor choices.

FACTORS FOR CHOOSING A HOT-SITE VENDOR

Before beginning the hot-site vendor selection process, the disaster recovery planner should ensure that all hot-site vendors to be evaluated provide equipment that is compatible with that of the organization. This will eliminate the time-wasting process of reviewing inappropriate candidates.

Capacity and Growth

Once the list of compatible vendors has been compiled, the ability of the hot-site vendor to handle the projected processing workload, data storage, and data communications volumes as well as physical space requirements should be considered. The business continuity planner not only should consider the organization’s current needs but should project those needs at least through the duration of the prophot-siteosed hot-site contract.

Flexibility on the part of the hot-site vendor in meeting a certain degree of unanticipated capacity requirements or in adapting to functional changes over time is essential.

AU0907/frame/ch29 Page 304 Monday, July 31, 2000 4:54 PM

Choosing a Hot-Site Vendor

Ideally, the hot-site vendor should guarantee the available capacity and space through the duration of the contract and provide reasonable assur- ance (preferably in the written contract) of future capacity that would support contract renewal.

In addition to computer space and processor storage capacity, noncom- puter space needs should also be evaluated. Sufficient space should be available for storing magnetic tapes and other media, printer paper, and custom forms and operational documentation. Consideration should be given to work areas for the staff who will be operating the computer system and to office space for support staff who will work in proximity to the computer area. In the event of a lengthy stay at the hot site, additional office, storage, or other space may be needed.

Recovery Experience

The key test for any hot-site vendor is its performance during an actual disaster. The ratio of actual disaster recoveries to the total client base may be low — or even zero. Given that actual recoveries are relatively rare, it should be reasonable to expect that a hot-site vendor has never mishan- dled an actual recovery. The odds are that any vendor who fails in a client recovery would quickly be out of business. Therefore, contrary to expecta- tions that emphasize vendor experience in recovering clients during disasters, vendor stability, technology and other factors tend to be weighted more significantly.

The extent of vendor participation in the development of the business continuity plan for clients who have successfully recovered can indicate the hot-site vendor’s commitment to its clients. Interviewing clients who have experienced disasters can help uncover weaknesses.

Comdisco Disaster Recovery Services, IBM Business Recovery Services, and Sungard Recovery Services each have substantial experience supporting client recoveries.

Testing Capabilities

Without regular testing of the business continuity plan, the ability to recover at a hot site is shaky at best. Most hot-site vendors provide a testing allowance; some even insist contractually on a minimum level of subscriber testing. Most also participate proactively in client testing. The availability and convenience of testing and vendor support of the testing process can significantly affect the cost as well as effectiveness of business continuity; if it is inconvenient or costly to test, testing may not be per- formed adequately; as a result, in an actual disaster the recovery effort could be useless.

AU0907/frame/ch29 Page 305 Monday, July 31, 2000 4:54 PM

BUSINESS CONTINUITY MANAGER’S TOOL KIT

The frequency and duration of testing varies from vendor to vendor.

Twenty-four to 72 hours of test time annually is typical. Additional test time can usually be arranged at extra cost. Any additional testing needs should be specified in the initial contract. The time to negotiate testing allowances is before signing a contract. Once an agreement is executed, additional test time may only be available at extra cost.

Availability of the hot site for testing should also be evaluated. Depend- ing on the number of hot-site clients and their testing needs, the lead time to schedule a test could be significant. Some hot-site testing schedules are busiest on weekends, others on weekdays. It is a good idea to check into test schedules as early as possible.

Of course, a declared disaster preempts any testing at a hot site. Even test time scheduled well in advance may be unexpectedly cancelled or interrupted; provision for such an event may be appropriate in the vendor agreement as well as in test plans.

The hot-site vendor should be included in the process of initiating and orchestrating a test. On completion of the test, the vendor should be able to offer specific feedback and recommendations.

In some cases, remote testing may be necessary — or even more appropriate than on-site testing. Some vendors offer remote testing (and some- times remote recovery) as an option, which could be particularly valuable if the hot site is far away.

Geography

The business continuity planner must consider location of the hot site in light of both recovery and testing. Costs for communications, transportation, and lodging are likely to be higher if the recovery site is farther away. On the other hand, a more distant site is less likely to be affected by a regional disaster (e.g., flood, hurricane, toxic contamination, or communications or power failure). In general, a hot site should be at least 25 miles away.

The business continuity planner should consider the impact of the hot site’s location on personnel. In addition to affecting costs, a remote hot site requires that employees make an abrupt transition to a distant location for recovery; the disruption in their lives could hinder the recovery effort.

In the event the contracted hot site becomes unavailable because of a declared disaster by another client or for any other reason, an alternate hot site may be offered either at the same location or at a different location.

It is important to make provisions in the recovery plan for the possibility of a different hot-site location. Issues related to logistics and possibly system configuration must be addressed, particularly if travel is involved.

AU0907/frame/ch29 Page 306 Monday, July 31, 2000 4:54 PM

Choosing a Hot-Site Vendor

Most of the same consideration used in situating a data center apply to the location of a hot site. A hot site should not be a in a high-risk location that is subject to external disruption. Access to transportation, including airports, interstate highways, and rail lines, is important to expediting business continuity and even more important in the case of a regional disaster. Such predictable occurrences as rush-hour traffic or the effects of inclement weather should not seriously impair access to the hot site.

Cost

Hot site costs fall into several categories. Some hot-site vendors offer testing and initial setup at no cost. Most charge a monthly or annual subscription fee to the organization for the ability to use their facilities and services when needed.

Subscription Fees. Subscription fees may range from a few hundred dollars to tens of thousands of dollars a month (for very large, complex main- frame environments).

Declaration Fees. Most hot-site vendors charge a one-time declaration, or activation, fee. This is designed in part to ensure that clients take the decision to declare a disaster seriously and in part to cover the vendor’s immediate costs in supporting the client’s recovery. These fees can range from a few thousand dollars to tens of thousands and are paid only when a disaster is declared.

Recovery Use. Once the hot-site vendor has received a declaration of a disaster, the clock usually starts ticking. In addition to hourly or daily use fees, there may be other time- or resource-related fees during the period of the actual recovery as well as during the subsequent operation at the hot- site facility.

Some hot-site vendors limit the amount of time a hot site may be occupied after a disaster is declared, so that the hot site can be returned to

“ready” status for other clients. This may mean that it becomes necessary to relocate from the hot site in a matter of weeks after the disaster, possibly to a cold site within the same facility, or elsewhere. The length of time the hot site may be used should be defined in the contract, along with clear responsibilities for the additional work and expense should a relocation become necessary.

The return to normal operations following a disaster may involve additional costs and effort. This should be considered in recovery planning, testing, and budgeting.

External Costs. Expenses incurred on the client’s behalf by the hot-site vendor are usually billable and should be identified explicitly, both for testing

AU0907/frame/ch29 Page 307 Monday, July 31, 2000 4:54 PM

BUSINESS CONTINUITY MANAGER’S TOOL KIT

and in the event of a disaster declaration. These will likely include both one- time and duration-dependent costs, such as telecommunications connec- tions or extra equipment rental.

Storage and Maintenance Costs. The vendor may charge a fee for locker or storage space used to store documentation, materials, supplies, and so forth if beyond the basic allotment. If dedicated equipment or network connec- tions are necessary, it is likely the hot-site vendor will charge a fee covering space occupied, equipment maintenance and monitoring, electricity, etc.

Testing Costs. Testing at the hot site should be done at least once a year and, in many cases, as many as four to eight times a year. The hot-site vendor provides a testing allowance addressing time and resources. Addi- tional costs may be incurred for testing beyond the basic agreement.

Before an organization enters into a contract with a hot-site vendor, testing costs should be projected and reflected in the agreement.

Cost Increases. Some hot-site contracts have annual or other periodic cost increases. In addition, the costs for future increases to function or capacity should be negotiated and committed up front to the furthest degree practical. Hot-site contracts typically cover multiyear periods, because the upfront investment in establishing a working hot-site vendor- subscriber relationship may be substantial relative to ongoing costs.

Software Costs. Even if business continuity planning software is bundled with the hot-site subscription, there may be additional costs for options, enhancements, and annual software maintenance.

Other Costs. Specialized resources, vendor personnel time, and such direct expenses as office furniture, supplies, food, and external vendor fees, may be chargeable to the subscriber during testing as well as during a recovery. These costs should be identified in advance whenever practical. Costs associated with evaluating hot-site vendors and negotiating an agreement may include travel to vendor sites and legal costs.

Insurance. Insurance may cover some of the costs associated with recovery to a hot site. The hot-site vendor may be expected to provide the insurer with detailed documentation in order to obtain compensation from insurance coverage.

Technical Environment

As stated previously, the vendor’s technical environment must be compatible with the client’s configuration and environment. The vendor’s technical staff should be well versed in the specifics of the client’s environment.

AU0907/frame/ch29 Page 308 Monday, July 31, 2000 4:54 PM

Choosing a Hot-Site Vendor

In some cases, the vendor may provide operating system software, communications drivers, or other offerings that may or may not be under the subscriber’s control or may be shared with other subscribers. Technical and technological compatibility must be an integral aspect of the contract.

Increasingly, client technical environments are becoming more complex and volatile. Information technology trends such as client/server, Internet connectivity, remote computing, and mobile computing have made the process of hot-site recovery planning and execution more difficult. The effective hot-site vendor will demonstrate flexibility as well as the ability to adapt and improvise as needed to cope with evolving client needs.

In addition, clients operating multiple computing platforms may find that hot-site vendors may house the recovery platforms in more than one location, or may operate a particular platform in only one location. The vendor’s ability to deliver computing services through dedicated or public communications networks to the client’s site will be essential to a success- ful recovery.

Recovery Center Facilities

The hot site should be operated as a going business, with appropriate attention to maintenance, testing, security, cleaning, and staffing. It should offer a professional environment appropriate to a facility that will house the subscriber’s vital operations in a crisis.

Physical security and access control are especially important at a hot- site facility. Multiple clients may be testing or recovering concurrently within a facility housing multiple hot sites. Some hot-site facilities may share hardware (such as logically partitioned processors) and communications facilities. The client should understand the level of physical as well as logical security provided by the hot-site vendor and the mechanisms used to prevent security breaches and to detect violations.

The client may also wish to employ multiple levels of security clearance among its own personnel. For example, certain people may be permitted to enter the computer area or tape library, and others restricted to office work areas. Many hot-site vendors offer card-key access control systems that may be programmed to allow this.

The infrastructure of the hot-site facility is also important. Some (but not all) hot-site vendors provide alternate electric power sources, with uninterruptible power supply systems and backup generators. Redundant environmental systems are also common.

Some clients find it useful to store key documentation, including their business continuity plan and operating procedures, at the hot site. They

AU0907/frame/ch29 Page 309 Monday, July 31, 2000 4:54 PM

BUSINESS CONTINUITY MANAGER’S TOOL KIT

may also store backups of key systems data to expedite the startup process. Many hot-site vendors provide limited storage or locker space for this purpose as well as limited space for such on-hand supplies as special printer forms. Arrangements can be made to ensure that the client can access this documentation or data.

Alternate Facilities

A hot site may be only part of a recovery solution for a client. Several hot-site vendors offer alternate facilities for computer recovery. These may include mobile, fully equipped, self-sufficiency, trailer-mounted computer rooms; “quickship” capabilities, with computer and support equipment rapidly delivered to the client location (or alternate recovery location);

even self-contained electrical power generation and stand-alone communications facilities such as satellite or microwave links.

These alternate facilities may be integrated with the basic hot-site agreement or may be contracted separately. The client’s recovery timeframe, economic constraints, technical environment and location should be weighted carefully when considering an alternate facility or hot site.

Communications Capabilities

In the past few years, communications capabilities have become one of the most critical aspects of disaster recovery. A backup computer configuration in a remote conditioned space is of little value if the computer can- not communicate effectively with the subscriber’s business.

More than line access is involved. The vendor should play an active role in designing, implementing, and managing the backup network. Expertise and facilities should be in place to operate, troubleshoot, and reconfigure the network. Custom generation of network control software to meet subscriber needs should be possible. Spare capacity, redundancy, and diag- nostic capabilities should be in place. Modems, multiplexers, patch panels, cables, dial backup units, and other components should be inventoried, tested, and ready to use when needed. The network capabilities should also be compatible with the subscribers’ requirements.

It may prove necessary to install client communications equipment or circuits to the hot site in advance. The hot-site vendor may charge fees for the space occupied by client equipment, as well as for management, testing, and coordination.

Voice and other communications needs should be considered as well as data. Beyond basic PBX or Centrex service, the client may need to redirect incoming telephone calls and faxes to the hot site. Availability of auto- mated call distribution systems, voicemail, and other specialized equipment should be considered if it is in use at the client’s existing site.

AU0907/frame/ch29 Page 310 Monday, July 31, 2000 4:54 PM

Risk and the Need for Business Continuity Planning

The Four Phases of Risk Realization