Syngress building SANs with brocade fabric switches kho tài liệu training

Chris Beauchamp AuthorJosh Judd Author Benjamin Kuo Contributor SANs Brocade Building Fa b r i c S w i t c h e s How to Design, Implement, and Maintain Storage Area Networks SANs with Br

Chris Beauchamp Author

Josh Judd Author

Benjamin Kuo Contributor

SANs

Brocade

Building

Fa b r i c S w i t c h e s

How to Design, Implement, and Maintain Storage Area

Networks (SANs) with Brocade Fabric Switches

• Step-by-step instructions for establishing your SAN requirements—such as high

availability, performance, and cost savings—and translating those requirements

into an effective SAN design

• Detailed examples to guide you through the process of installing and

trouble-shooting your Brocade SAN

• Practical discussions about SAN components and popular SAN configurations

such as storage consolidation, disaster tolerance, and LAN-free backup

with

s o l u t i o n s @ s y n g r e s s c o m

With more than 1,500,000 copies of our MCSE, MCSD, CompTIA, and Ciscostudy guides in print, we continue to look for ways we can better serve theinformation needs of our readers One way we do that is by listening

Readers like yourself have been telling us they want an Internet-based vice that would extend and enhance the value of our books Based onreader feedback and our own strategic plan, we have created a Web sitethat we hope will exceed your expectations

ser-Solutions@syngress.com is an interactive treasure trove of useful

infor-mation focusing on our book topics and related technologies The siteoffers the following features:

■ One-year warranty against content obsolescence due to vendorproduct upgrades You can access online updates for any affectedchapters

■ “Ask the Author”™ customer query forms that enable you to postquestions to our authors and editors

■ Exclusive monthly mailings in which our experts provide answers toreader queries and clear explanations of complex material

■ Regularly updated links to sites specially selected by our editors forreaders desiring additional reliable information on key topics

Best of all, the book you’re now holding is your key to this amazing site

Just go to www.syngress.com/solutions, and keep this book handy when

you register to verify your purchase

Thank you for giving us the opportunity to serve your needs And be sure

to let us know if there’s anything else we can do to help you get the maximum value from your investment We’re listening

www.syngress.com/solutions

1 YEAR UPGRADE

B U Y E R P R O T E C T I O N P L A N

Chris Beauchamp Author

Josh Judd Author

Benjamin Kuo Contributor

SANs Brocade

Building

Fa b r i c S w i t c h e s

with

Syngress Publishing, Inc., the author(s), and any person or firm involved in the writing, editing, or

production (collectively “Makers”) of this book (“the Work”) do not guarantee or warrant the results to

be obtained from the Work.

There is no guarantee of any kind, expressed or implied, regarding the Work or its contents.The Work is sold AS IS and WITHOUT WARRANTY.You may have other legal rights, which vary from state to state.

In no event will Makers be liable to you for damages, including any loss of profits, lost savings, or other incidental or consequential damages arising out from the Work or its contents Because some states do not allow the exclusion or limitation of liability for consequential or incidental damages, the above limitation may not apply to you.

You should always use reasonable case, including backup and other appropriate precautions, when working with computers, networks, data, and files.

Syngress Media®, Syngress®, and “Career Advancement Through Skill Enhancement®,”are registered marks of Syngress Media, Inc “Ask the Author™,”“Ask the Author UPDATE™,”“Mission Critical™,”

trade-“Hack Proofing™,” and “The Only Way to Stop a Hacker is to Think Like One™” are trademarks of Syngress Publishing, Inc Brands and product names mentioned in this book are trademarks or service marks

of their respective companies.“Brocade®,” “SilkWorm®,” and the Brocade logo are registerd trademarks of Brocade Communications Systems, Inc., in the United States and/or any other countries.

Building SANs with Brocade Fabric Switches

Copyright © 2001 by Syngress Publishing, Inc All rights reserved Printed in the United States of

America Except as permitted under the Copyright Act of 1976, no part of this publication may be duced or distributed in any form or by any means, or stored in a database or retrieval system, without the prior written permission of the publisher, with the exception that the program listings may be entered, stored, and executed in a computer system, but they may not be reproduced for publication.

repro-Printed in the United States of America

1 2 3 4 5 6 7 8 9 0

ISBN: 1-928994-30-X

Technical Editors: Chris Beauchamp, Freelance Editorial Manager: Maribeth Corona-Evans Josh Judd, Benjamin Kuo Cover Designer: Michael Kavish

Acquisitions Editor: Catherine B Nolan Page Layout and Art by: Shannon Tozier

Developmental Editor: Kate Glennon Indexer: Jennifer Coker

Copy Editor: Beth A Roberts

Syngress Acknowledgments

v

We would like to acknowledge the following people for their kindness and support

in making this book possible

Greg Reyes, Jack Cuthbert, Doug Wesolek, Maggie Conroy, Julie Chiu, Elaine Tite,Jeff Seltzer, and Chris Mingrone at Brocade, for championing the idea of a BrocadeSANs book Also special thanks to Viet Dao, John Gareri, Mark Murphy, Jay Rafati,Ron Totah, Ezio Valdevit, John Bae, James Carpignano, Steve Daheb, Derek Granath,Jay Kidd, Omy Shani, James Bleess, Owen Higginson, Leo Kappeler, Chris M

Nguyen, Mark Peluso, and Henry Robinson for their help in making this book areality

Ralph Troupe of Callisma for his invaluable insight and guidance Ralph’s expertise inSAN architecture and design solutions for next-generation storage networking

implementations helped define our vision for this book

Richard Kristof and Duncan Anderson of Global Knowledge, for their generousaccess to the IT industry’s best courses, instructors, and training facilities

Karen Cross, Lance Tilford, Meaghan Cunningham, Kim Wylie, Harry Kirchner,Kevin Votel, Kent Anderson, and Frida Yara of Publishers Group West for sharingtheir incredible marketing experience and expertise

Mary Ging, Caroline Hird, Simon Beale, Caroline Wheeler,Victoria Fuller, JonathanBunkell, and Klaus Beran of Harcourt International for making certain that ourvision remains worldwide in scope

Anneke Baeten and Annabel Dent of Harcourt Australia for all their help

David Buckland,Wendi Wong, Daniel Loh, Marie Chieng, Lucy Chong, Leslie Lim,Audrey Gan, and Joseph Chan of Transquest Publishers for the enthusiasm withwhich they receive our books

Kwon Sung June at Acorn Publishing for his support

Ethan Atkin at Cranbury International for his help in expanding the Syngress program

This book was designed and written to provide information about storage area working architectures Every effort has been made to make this book as completeand accurate as possible However, the information in this book is provided to you

net-“AS IS,” without warranty of any kind, including, without limitation, any implied

warranty of merchantability or fitness for a particular purpose

The authors and Brocade Communications Systems, Inc., shall have no liability orresponsibility to any person or entity with respect to any loss, cost, liability, or dam-ages arising from the information contained in this book or the computer programsthat accompany it, and specifically disclaim any implied

vi

Brocade Acknowledgments

This book truly represents a complete Brocade team effort.We would like toacknowledge several people in particular.Without their help, dedication, and knowl-edge, this book would not have been possible.The thorough technical review by VietDao, John Gareri, Mark Murphy, Jay Rafati, Ron Totah, and Ezio Valdevit shaped ourmanuscripts into a book that Brocade can be proud of John Bae, James Carpignano,Steve Daheb, Derek Granath, Jay Kidd, and Omy Shani provided several timely con-tributions to the content.We also incorporated material written by others withinBrocade: James Bleess, Owen Higginson, Leo Kappeler, Chris M Nguyen, MarkPeluso, and Henry Robinson.We would also like to thank Maggie Conroy and DougWesolek for their guidance and help with the publication process

—Josh Judd and Chris Beauchamp

Authors

Chris Beauchampis a Senior SAN Architect for BrocadeCommunications Systems, Inc Chris moved to Brocade in 1998 as aSystems Engineer supporting several strategic customers with the applica-tion and qualification of SilkWorm fabric switches Chris now focuses onSAN design and architecture, with an emphasis on scalability and trou-bleshooting His specialties include Sun servers, storage performance anal-ysis and capacity planning, Fibre Channel trace analysis, scripting invarious languages, and SAN administration Chris holds a Master ofScience in Computer Engineering from Villanova University and aBachelor of Science in Computer Science degree from West ChesterUniversity Chris’s background includes positions as a SystemsAdministrator at General Electric and a Systems Engineer at SunMicrosystems Chris currently resides outside of San Jose in the SantaCruz Mountains with his wife Sarah and daughter Meagan

Josh Juddis a Senior SAN Architect with Brocade CommunicationsSystems, Inc In addition to writing technical literature, he providessenior-level strategic support for major OEMs and end-users of Brocadestorage network products worldwide.When he first went to work forBrocade, he was the company’s Senior IT Specialist, responsible for escala-tions in every area of the company’s network, server, and desktop infras-tructure Josh’s career as an IT consultant has given him a diverse skill set,which includes senior-level expertise in several UNIX variants,Windows9x/NT/2k administration, RAID configuration and optimization, storagevirtualization and clustering software (such as that produced by VERITASSoftware), and network engineering with many vendors, including Cisco,Foundry, Lucent, and 3com Before joining Brocade four years ago, Joshworked at IBM Global Services, LSI Logic, and Taos Mountain

Consulting He lives in San Jose, California

Alex Neefusis the Lead Interoperability Test Engineer at LampreyNetworks, Inc Lamprey Networks offers certification testing services andtest tool development to the Fibre Channel industry Alex has worked ondeveloping testing tools for the SANmark program hosted by the FCIA.This program certifies Fibre Channel devices for conformance and inter-operability Alex has also co-authored and written a number of confor-mance test suites for both the FCIA and the University of New

Hampshire Interoperability Lab Alex’s background also includes workingfor the UNH Interoperability Lab in the Fibre Channel Consortium forover a year and a half At the lab his primary work is in developing proce-dures and tools for testing Fibre Channel products, and working withmembers of the industry to solve interoperability problems with devices

on the market Alex resides in Durham, New Hampshire

Contributor

xi

Reducing Network Congestion from Backup 13

Designing Around the Application 16Assessing Speed, Bandwidth, and

Learn When to

Deploy a SAN

Things to consider when

deciding whether a SAN is

the right solution:

Chapter 2 Fibre Channel Basics 29

Introduction 30

Fibre Channel Protocol Basics 38

Chapter 3 SAN Components

Introduction 60Overview of Fibre Channel Equipment 61

Master Fabric Services

Fabric services provide

information to nodes in a

switched fabric topology.

Services can be distributed

across all switches,

creating the appearance

GBICs and Connectors 61Hubs 63Switches 63Storage 64

Copper Versus Optical: Selecting Your Media 65

Single-Mode Optical Cabling 68

The HSSDC Copper Connector 70

High-Density Fiber-Optic Connectors 72Comparing GBICs to Fixed Media 73

Pros and Cons of Using GBICs 74

Serialized Versus Nonserialized 74Common Problems with GBICs 75

LIP Service: Fibre Channel LIPs,

Getting Out of the Loop: Migrating

Using Switches and Fibre Channel Fabrics 80

Understand Fibre Channel Equipment

Any single-mode or multimode laser can damage your eyes if it is trans- mitted at 1300 nm.

The 1300 nm length is not in the visible spectrum, so you will not see a laser being trans- mitted like in 850 nm fiber A 1300 nm laser is dangerous, because it can cause severe retina

wave-damage.

Features of Fibre Channel Switches 82Zoning 83

IP over Fibre Channel Broadcasting 88

Loop Operation: Making Your Switch

Telnet 91SNMP 91

Application-Based Management 94

Connecting Your Servers with Host

Connecting Hosts to the Fabric 95

Speeds 97Ports 98

Fabric-Capable Versus Loop Adapters 98

Default LUN Access Permissions 100Upper-Level Protocol Access Permissions 100Dynamic Versus Static Discovery 101Configuration Management Software 101

Remote Boot across the SAN 103

Connecting Legacy Devices into Your SAN 106

Types of SCSI Ports,Termination 108Selective LUN Presentation 108

Bridging and Routing to IP Networks

Fibre Channel across IP Networks 110

IP over Fibre Channel to Gigabit Ethernet 110

Individual Disk Drives and JBODs 111

Selective LUN Presentation 113LUN Export across Multiple Ports 113

Summary 115

Chapter 4 Overview of Brocade

Brocade WEB TOOLS is a software utility that enables you to manage and monitor your fabric through a Web browser interface and Java plug-in.

Using WEB TOOLS, you can view all switches in the SAN from a single interface from any workstation in your enterprise—even at a remote location.

SilkWorm 6400 Integrated Fabric 130SilkWorm 12000 Core Fabric Switch 131Understanding the Brocade Fabric OS 132

Fibre Channel Services for Reconfiguration 133

Facilities for End-to-End SAN Management 135Brocade Command Line Interface 135Using Optional Brocade Features 135

Understanding Loop Support, QuickLoop,

Future Capabilities in the Brocade Intelligent

More Robust Hardware-EnforcedZoning 142Enhanced End-to-End Performance

Analysis 143

Summary 144

Introduction 150Looking at the Overall Lifecycle of a SAN 151

Maintenance 155

What Overall Business Problem Are

What Are the Business Requirements

What Is Known about the Nodes thatWill Attach to the SAN? 160Which SAN-Enabled Applications

Which Components of the Solution

Future Performance Characteristics? 172How Much Downtime Is Acceptable to Production Components During

How Much Downtime Is Acceptable for Routine Maintenance? How Much Downtime Is Acceptable for Upgradesand Architectural Changes? 174When Do You Need Each Piece

of the Solution to Be Complete? 175

Master the seven phases of the SAN design lifecycle:

1 Data Collection

2 Data Analysis

3 Architecture Development

4 Prototype and Test

5 Transition

6 Release to Production

7 Maintenance

Summary List of Questions 176Conduct a Physical Assessment 176

Processing What You Have Collected 177Establishing Port Requirements 182

The Return On Investment Proposition 188The Rest of the Process and the

Summary 191

Chapter 6 SAN Applications

Introduction 196Configuring a High-Availability Cluster 196Typical HA Application or Database Server 198

Using a SAN for Storage Consolidation 203

Shared Storage Using a Web Farm 206Storage Partitioning Using Switch Zoning 208Switch Zoning Configuration for

Storage Partitioning Using Storage LUNMasking 210Storage Partitioning Using HBA LUN

Masking 210

SAN-Based Third-Party Copy Data Movers 215Making Your Enterprise Disaster Tolerant 216Data Replication and Remote Backup 218

A:Most major databases

now support fabric

Q:I would like to have my

Exchange Mail Server

highly available What

should I do?

A:Brocade has developed

HA solutions for the

Exchange Server that

can be used in setting

up your desired SAN

configuration.

Metropolitan Area Network Solutions 219Summary 222

Chapter 7 Developing a SAN Architecture 227

Introduction 228Identifying Fabric Topologies and SAN

Working with the Core/Edge Topology 246

Determining Levels of Availability 256

Using Any-to-Any Connectivity 268Evaluating Performance Considerations 269When Is Over-Subscription Bad? 270Considerations Outside the Fabric 270Summary 272

Introduction 278The Troubleshooting Approach:The SAN

Develop a SAN Architecture

fabric-related issue impacts

many devices, while a

device-related issue affects

only a few devices.

A Typical Scenario: “I Cannot See My Disks” 279Where to Start and What Data to Gather 283Take a Snapshot: Describe the

Problem and Gather Information 284

What to Look for in a Malfunctioning Fabric 317

The nsShow and nsAllShow Commands 320Now that You Suspect a SAN Issue:

Timeout of Edge Devices During

Port Configuration Conflict or

Troubleshooting Devices that Cannot Be Seen 327What to Look for in the Fabric 329Are the Host and Storage Visible via

switchShow on Their Respective Switches? 329

Do the Devices Show Up in the NameServer? 332

Edge Device Not in the Name Server 334

Troubleshooting Marginal Links 335Marginal Point-to-Point/Fabric Device Links 335

Nx_Port (Host/Storage) Behavior with

a Marginal Port in the Loop 338

How the Switch Can Help: Fabric Watch

Overview of SilkWorm Port Error Statistics 341

Summary 344

Chapter 9 SAN Implementation,

In-Band or Out-of-Band Management? 356

What Fabric OS Version Should I Use? 361Licenses 366Automating Switch Administration Activities 367

on Your Switch

core1:admin> licenseShow

SRzy9Sz9zeTS0zAG:

Web license bbSz9eQb9zccT0AQ:

Zoning license RdzdSRcSyzSe0eTn:

QuickLoop license cSczRScd9RdTd0SY:

Fabric license

Hard Zoning and Soft Zoning

Scripting Zoning Operations 379

The Configuration Log: Key Information

to Gather and Maintain about Your SAN 391Backing Up and Restoring a Switch

Configuration 393

Expanding a Fabric: Merging Fabrics, Adding

a Switch, or Replacing a Switch 395

Issues Applicable to Both Hot and

Performing a Hot Fabric Upgrade 399Performing a Cold Fabric Upgrade 400How to Automate firmwareDownload 400Replacing or Adding an Edge Device in

Summary 403

Appendix Building SANs with

Why Write a Book about SANs?

During the last few years, Storage Area Networks (SANs) have fundamentally changedthe way organizations design, build, and manage their enterprise networks As a supe-rior alternative to direct-attached storage models, SANs have enabled a wide range ofnew configurations and applications In turn, those applications have generated avariety of benefits for the organizations that have implemented them.These advan-tages include superior scalability, simplified storage management, optimized resourcesharing, higher availability, and tremendous cost savings to name just a few

As a primary facilitator of the networked storage model, BrocadeCommunications Systems actively seeks out new opportunities to raise industryawareness about the value of SANs One of our primary goals at Brocade is to helpeducate all kinds of organizations about the advantages a networked storage environ-ment can offer

Based on feedback from our customers and business partners, we realized thatthere was no self-contained, effective guidebook for implementing Fibre ChannelSANs.To help fill that void, we have joined with Syngress Publishing to bring you

Building SANs with Brocade Fabric Switches This book details the design, installation,

configuration, and troubleshooting of Brocade-based SANs —basically everythingyou need to know before beginning your own SAN implementation

Who Should Read This Book?

Building SANs with Brocade Fabric Switches is written for anyone who plans to design,

build, and manage SANs using Brocade switches and software In particular, this bookprovides a “how to” reference that describes what you can do today, given the tech-nologies currently available

xxv

Foreword

By necessity, the focus is Brocade-centric and features the theory of operationbehind Brocade SilkWorm switches and Fabric OS However, this book is notintended to be a comprehensive guide for every configuration and scenario possible.After all, with the rapid expansion of the SAN marketplace, there will undoubtedly

be other technologies available in the not-so-distant future

What Does the Book Contain?

In addition to providing an overview of current technology, tools, products, anddesign topologies, this book should serve as a guideline for actual SAN implementa-tion For instance, the book begins with a detailed analysis of technology require-ments and the benefits of implementing a SAN Next, you can learn about FibreChannel concepts and definitions as well as the full range of SAN components

We then introduce you to the Brocade SilkWorm series of Fibre Channelswitches, including guidelines for integrating these switches into your existing ITenvironment.The book concludes with examples of design processes, popular SANapplications, and detailed troubleshooting and maintenance tips In addition, eachchapter features a high-level summary and FAQs for anyone who needs a quickoverview of the SAN basics

Our goal is to make this book a valuable tool for implementing your own SAN infrastructure and teach how a well-designed SAN can deliver a

competitive advantage for your organization.We welcome your feedback on ourefforts If you have any comments or suggestions about this book, please let us know

at www.syngress.com/solutions

—Kumar Malavalli Vice President,Technology Brocade Communications Systems

www.syngress.com

Introduction

to SANs

Solutions in this chapter:

■ Benefits of Building a SAN

; Summary

; Solutions Fast Track

; Frequently Asked Questions

Chapter 1

1

In the early 1980s, direct-attach disk storage through interconnects such as SmallComputer Systems Interface (SCSI) was the standard way to connect to data.Thisworked well for the amount of data typically handled at the time, and became the standard way to connect high-speed, high-performance storage to computersystems

However, as computer systems increased in speed and data storage needsincreased, the parallel bus architecture of SCSI soon started hitting performanceand distance limits In response to these needs, Fibre Channel was developed toprovide gigabit-speed serial networking capabilities for storage Fibre Channelincludes support for the SCSI protocols for storage, the Internet Protocol (IP) fornetworking, and the Virtual Interface (VI) protocol for clustering, which aremapped onto a network architecture.The Fibre Channel standard combines longdistances of up to 10 km, simplified serial cabling over multiple media types,gigabit speeds, and the ability to simultaneously use more than a single protocolover the same wire.These features won adoption for Fibre Channel throughoutthe 1990s as a replacement for parallel SCSI, and Fibre Channel is now used formost high-capacity, high-end direct storage devices

With the advent and market acceptance of Fibre Channel as a point-to-pointreplacement for parallel bus SCSI, a new technique has emerged that combinespure storage usage with networking—the Fibre Channel Storage Area Network(SAN) A SAN is a network of storage and system components, all communi-cating on a Fibre Channel network, that can be used to consolidate and sharestorage, provide high-performance links to data devices, add redundant links tostorage systems, speed up data backup, and support high-availability clustered systems

This chapter provides an overview of what a SAN is, what types of problemsare best solved with a SAN, and some steps to make a SAN deployment suc-cessful After reading this chapter, you should be able to determine if you shoulddeploy a SAN, identify the types of applications best solved by SAN technology,and be ready to create a deployment plan for your SAN

Overview of SANs

Throughout the 1980s, the standard way of connecting hosts to storage deviceswas point-to-point, direct-attach storage through interfaces such as IntegratedDrive Electronics (IDE) and parallel SCSI (Figure 1.1) Parallel SCSI offered

relatively fast (5 or 10 Mbit/sec) access to SCSI-enabled disks, and several diskscould be connected at once to the same computer through the same interface.

This worked well for the time, with fairly reliable, fast-speed connectionsallowing administrators to connect internal and external storage through justsimple ribbon cabling or multiconductor external cables However, as storagesubsystems became larger and larger and computers faster and faster, a newproblem emerged—external storage (which at one time was just a simple diskdrive on the desk next to a machine) started to get bigger.Tape libraries,Redundant Array of Inexpensive Disks (RAID) arrays, and other SCSI devicesbegan to require more and more space—requiring the parallel SCSI connection

to be stretched farther and farther away from the host Input/Output (I/O) ratesalso increased, pushing on the physics of keeping signal integrity in a large bundle

of wires (32- and 64-bit data bus widths) Simple parallel SCSI variants weredevised to enable longer distances and to address the signal integrity issues

However, they all eventually ran up against the difficulties of high-speed signalsacross the parallel SCSI bus architecture

The solution to all of this was slow in coming, but eventually the storageindustry settled on using a serial protocol with high-speed transceivers—offeringgood noise immunity, ease of cabling, and plentiful bandwidth Different specifi-cations (Serial Storage Architecture [SSA] and Fibre Channel as well as moreadvanced parallel SCSI technologies) competed for adoption, and companiesbegan experimenting with different serial communications media New high-speed circuits made serial transfers (using a simple pair of wires to transmit bitsserially, in order, rather than a large number of wires to transfer several bytes or

Figure 1.1Parallel SCSI Bus Connection

words of data at a time) the most practical solution to the signal problems.Thehigh speed of the circuits enabled the data rates for Fibre Channel to offer up to

100 Mbit/sec transfers, versus the slower 10 to 20 Mbit/sec parallel limitations.When Fibre Channel was first applied to the area of storage connections, theprimary reason for the technology was for the extended distances and simplifiedcabling that the technology offered.This extension of direct-attach operationbasically replaced the old parallel SCSI attachments with a high-speed serial line(Figure 1.2).The new Fibre Channel connections offered a much faster interfaceand simplified cabling (four copper wire connections through DB-9 connectors,

as well as optical cabling), and could be used to distribute storage as far as 10 kmaway from a host computer, or 30 km away with optical extenders

The connections to disks at this time began using the Fibre ChannelArbitrated Loop (FC-AL) protocol, which enabled disks to negotiate their

addresses and traffic on a loop topology with a host (Figure 1.3) Because of thecombined ability to easily cable and distribute storage, users were now able toadd separate racks of equipment to attach to hosts A new component, the FibreChannel hub, began to be used to make it easier to plug in devices.The hub, apurely electrical piece of equipment that simply connected pieces of a FibreChannel loop together, made it possible to dynamically add and remove storagefrom the network without requiring a complete reconfiguration As these com-ponents began to be used in increasingly more complex environments, manufac-turers began to add “intelligence” to these Fibre Channel hubs, enabling them toindependently deal with such issues as failures in the network and noise in thenetwork from loops being added and removed An alternative to the hub came inthe form of the Fibre Channel switch, which, unlike a hub, was not just con-necting pieces of a loop, but instead offered the packet-switching ability of traditional switches

Figure 1.2Using Fibre Channel to Extend Distances from Storage

Host

Fibre Channel Link

Up to 10 km

Storage Array

Because there was now a Fibre Channel network available, other hosts (notstorage) were added to take advantage of the same network.With the addition

of SAN-aware software, it was suddenly possible to share storage between twodifferent devices on the network Storage sharing was the first realization of themodern SAN, with companies in the multimedia and video production areaspaving the way by using the Fibre Channel network to share enormous data files between workstations, distribute jobs for rendering, and make fully digitalproduction possible (Figure 1.4)

The next big step in Fibre Channel evolution came with the increased bility and manageability of a Fibre Channel switched fabric Early implementations

relia-of FC-AL were sometimes difficult to manage, unstable, and prone to ability problems between components Because the FC-AL protocol was quitecomplex, what sometimes would happen would be an inability for anything tocommunicate and stay operational on a loop.The solution to this was a move to a

interoper-switched fabric architecture, which not only enhanced the manageability and reliability

of the connection, but provided switched, high-speed connections between allnodes of a network instead of a shared loop As a result, each port on a switch nowprovides a full 1 Gbit/sec of available bandwidth rather than just a portion of thetotal 1 Gbit/sec of bandwidth shared between all the devices connected to theloop Fabrics now make up the majority of Fibre Channel installations A typicalFibre Channel switched fabric installation (Figure 1.5) has multiple hosts andstorage units all connected into the same Fibre Channel network cloud throughone or more Fibre Channel switches

Figure 1.3Arbitrated Loop Disk Configuration Attached to a Single Host

Fibre Channel Arbitrated Loop

Disk

Disk

Disk Disk

Host

Figure 1.4Multiple Host Arbitrated Loop for Storage Sharing

Fibre Channel Arbitrated Loop

Disk Disk

RAID

Disk Host

Host Host

Figure 1.5Switched Fabric, Multiple Host, and Storage Unit Configuration

Today, the modern SAN looks much like any other modern computer work Network infrastructures such as switches, hubs, bridges, and routers helptransport frame-level information across the network Network interface cardsinterface computer systems to the same network (called HBAs in the SAN world,

net-as they replaced SCSI Host Bus Adapters) Figure 1.6 shows an example of howthese components could be used in conjunction with Fibre Channel switches

Figure 1.6Typical Deployed SAN Configuration with Multiple Hosts, Storage, and Tape Devices

Fibre Channel Cloud

JBOD

RAID Array

Legacy Parallel SCSI Storage

Storage Array Host

Channel-to-Tape Array

Fibre SCSI Router

Channel-to-HBA Database Server

Fibre DWDM Bridge

Channel-to-Remote SAN

Fibre Channel Switch

Fibre Channel Switch

Fibre Channel Switch

Fibre Channel Switch

Fibre Channel Hub

ISL (Inter-Switch Link)

Taming the Storage Monster

The advent of SANs has been driven by today’s insatiable appetite for storage.The Internet, e-mail, multimedia, and the increasing digital nature of society have resulted in an ever-increasing demand for ways to store, retrieve, and back

up that data

For example, e-mail has been on a staggering growth path in the last fewyears, as more and more people have gone online and businesses have made e-mail a critical part of their communications infrastructure According to the

Year-End 2000 Mailbox Report, there are over 891 million e-mail mailboxes now

in existence Corporate mail usage grew 34 percent in 2000, bringing with it ahuge increase in the need for data storage to save all of that e-mail Multimediaattachments, the movement of business processes to e-mail, and just the sheervolume of e-mail have made the storage and backup of e-mail one of the mostpressing requirements of IT departments

Rather than relying on just the equipment vendor, an effective way to learn and become an expert on the technology is to track the industry and attend conferences, meetings, and tutorial sessions about the subject Additional resources for learning more about SAN technology are the industry organizations devoted to this area The Storage Networking Industry Association (SNIA) offers white papers and educational resources, holds technical tutorial sessions and Storage Networking World conferences, and supports both the users and vendors involved in the storage networking field More information can be found at www.snia.org The Fibre Channel Industry Association (FCIA) provides resources for users and vendors, conducts the SANmark suite of Fibre Channel interoperability tests, and holds conferences and meetings to help promote Fibre Channel technology Their site can be found at www.fibrechannel.org.

Resources for SAN Information

The Internet has also affected the need for storage, with increasing numbers

of Web servers and storage required to support those Web servers As information

is increasingly digitized and published on the Web, there is an insatiable appetitefor storage to contain that information Music and full-motion video, even withcompression, take an immense amount of disk space, and the movement of stu-dios and companies to run a “full digital” shop has resulted in an enormousdemand for storage capacity Databases, which used to be considered big if theywere gigabytes in size, are now well beyond a terabyte—with companies talkingabout eventually having to manage petabytes of database storage

In addition, with caching servers,Web load balancing, and Web farms built todistribute the processing load for Web traffic, the data being presented on Websites has to be duplicated 10, 20, and even 100 times to serve those distributedhosts with information.With the increased connectivity of the Internet, informa-tion and content are being generated and distributed faster than ever before inhistory—so much, in fact, that the University of California at Berkeley recentlyreleased a study that claims that more data will be created in the next two yearsthan was produced in the history of mankind

All of this data has to go somewhere, and it has exceeded the space availableand beyond what can practically be managed on local, direct-attached storage tohosts Because local storage is relatively fixed and difficult to expand, and becauseits local nature is difficult to manage, organizations have started to look for abetter way to manage this data.The solution has come in the form of very largestorage arrays, capable of storing terabytes and terabytes of data, and farms ofinexpensive Just A Bunch Of Disks (JBOD) All of this needs to be connected,and the logical way to connect high-speed, block-oriented traffic is through aFibre Channel SAN Increased manageability, the ability to centrally managestorage, and consolidation of storage space have made the SAN a necessity in anygrowing enterprise

Data growth is increasing at such a rapid pace that IBM recently reportedthat storage sales now exceed server sales at a 70:30 ratio.The requirements tostore data are increasing at a greater rate than the requirement for CPU cycles,and the entire industry is changing as a result.This gain has meant that data isnow managed separately from the machines that consume that data, makingSANs an ideal choice to break the dependency of hosts from the storage, andincreasing the manageability and usability of a corporation’s investment in datastorage

Implementing a SAN is an ideal technique for taming the storage ment monster that has resulted from the growth of the Internet and increasedconnectivity of our electronic age.

require-Benefits of Building a SAN

A number of practical, real-world uses for SANs have emerged in recent years.Knowledgeable administrators have figured out the types of problems that SANtechnology best solves SANs are typically used for the most business-critical,technically challenging problems a company faces Critical, high-availability sys-tems used for e-mail, database, and file servers have been the first to switch toSANs A need to consolidate storage and centrally manage volumes has resulted

in a trend toward using SANs for storage consolidation.With the increase in datagrowth, backups have also become a problem, with companies looking to accel-erate backup cycles Protocols such as IP available on Fibre Channel also makeSANs attractive for some general networking applications, and VI clustering sup-port allows installations to leverage their SAN infrastructure for VI-enabled clus-tering applications Finally, the distance capabilities of Fibre Channel and bridges

to Metropolitan Area Networks (MANs) and even Wide Area Networks (WANs)have enabled a new level of disaster tolerance for storage resources

Ensuring High Availability

As the Internet and digital data have grown exponentially,Web caching niques,Web load balancers and distributed server clusters, and other techniqueshave been used to handle the demands of serving up Web requests for staticpages Images, files, and Web pages that do not change often can be copied across

tech-a btech-ank of hosts, tech-all of which ctech-an service tech-a request from tech-a user

However, these techniques cannot be applied in many critical applications.For example, an e-mail server requires one single, consistent image for e-mailstorage Back-end databases of e-commerce applications require combining live,real-time inventory data with live pricing data to service requests correctly None

of these can be cached across a Web server due to the real-time, non-cacheablenature of the information.This dependency on a consistent, single image of datacannot be solved by just replicating data or sharing across a cluster.The result is anew, critical point of failure in the e-mail server or database Especially with thegrowth in data, more and more vital data is being trusted to those single points of

failure, raising the stakes and potential losses if those services go down.The cern over these critical points of failure has resulted in a renewed focus on highlyavailable (fault tolerant) solutions, particularly in the storage area In combinationwith failover software packages such as Microsoft Cluster Server or VERITASCluster Server, high-availability hardware and software has come to the forefront

con-in ensurcon-ing the performance and availability of these critical systems

For example, one area where the use of SANs is ideal has been the use ofhigh-availability solutions for managing and running very large MicrosoftExchange databases.With the immense increase in data stored in Exchangeservers all over the world, there has been an increase in the amount of back-endstorage required for serving those Exchange installations Because of the nondis-tributed nature of Exchange mail databases, there has been a concentration ofdata tied to single hosts and storage units—a single point of failure that couldcripple many businesses.The natural solution has been to use application clus-tering techniques combined with a robust, fully redundant high-availability SAN

to support those clusters and share redundant storage between hosts

High-availability systems are now regularly used for ensuring fault-tolerantaccess to storage A focus on eliminating single points of failure has stimulateddemand for fault-tolerant equipment configurations, specific fault-tolerant net-work equipment, and techniques for ensuring high availability SAN technology isideal for these types of solutions It allows host-to-host connectivity for heartbeat,equipment status, and network communications, as well as for sharing criticalstorage between alternate and backup servers

The availability of SAN connections has solved one of the big problems withhigh-availability, clustered installations: access to the same data across a network

In combination with high-availability features in storage arrays and other ment, the SAN allows for multiple redundant paths to be made from multipleredundant hosts, dramatically increasing the reliability of critical systems In addi-tion, with flexible SAN interconnections, the large amount of data that needs to

equip-be accessed can more easily equip-be managed separately, rather than equip-being captive to apotential failure in a host

Consolidating Storage

As data needs have increased, it has become increasingly difficult to manage thehundreds of hosts and local disks attached to those hosts In order to manage thisgrowth, administrators have begun to centralize their storage resources Largestorage arrays and pooled storage are much more efficient and infinitely moremanageable than local storage

As storage needs have increased, the model of attaching local storage to hostshas broken down Administrators figured out that, even though a company as awhole might own enough storage for all of its needs, that storage was not neces-sarily in the right place For example, a Web server might be running out of space,with no more space available on local disks and not enough SCSI connections toadd more external storage, while the database server next to it has gigabytes free.

In the old model of local storage, there was no way to take advantage of that fact.You ended up purchasing much more storage than you needed, because you had avery low rate of utilization—yet you never had enough capacity.You also ended

up purchasing more servers than you needed, because you did not need moreCPU cycles, but rather, more storage slots

With the advent of the Fibre Channel network, the ability of both clients andstorage to coexist and share storage has spawned a new crop of solutions that takeadvantage of that sharing Sharing of storage, which previously was limited to ver-tical markets such as video editing and multimedia, has become a general techniqueused anywhere that storage is more easily managed in a pool, such as in InternetService Provider (ISP) and Application Service Provider (ASP) installations Indeed,most corporate IT environments can take advantage of this technique

Through software such as VERITAS Volume Manager,Tivoli SANergy, andDataCore SANsymphony, users are now able to allocate and share storage amongmultiple hosts

By using the SAN infrastructure, large centralized pools of disks can bedivided between hosts, and new volumes allocated as needed from the generalpool.This results in a huge increase in efficiency in use of storage, eliminating thepools of expensive, local, unusable storage Instead, one large, easily managed vir-tual storage pool can be centrally administered, and storage costs and administra-tion centralized and consolidated

Sharing is accomplished through this high-level software, which discovers andmanages all of the storage on the network Drivers and software in the host managewhich machines do and do not get access to a specific part of a storage device Ingeneral, a central administrator is able to allocate arbitrary pieces of storage to specific hosts, all while the network and all hosts are running in real time

A typical example of this is an ISP with a large number of user Web page accounts Extensive pools of clustered and independent Web servers help toease the traffic load and provide redundancy on the Internet, while being tied into

home-a single- or duhome-al-redundhome-ant SAN Storhome-age home-allochome-ation home-and shhome-aring softwhome-are is run onall of these hosts, and the different Web homepage accounts are allocated to dif-ferent Web servers.When a failure on a host or storage device occurs, either an

automated process or manual intervention will re-allocate those user volumes toanother Web server, or fail over to another storage device, resulting in uninter-rupted service and no dependency of specific users on a local disk In some cases,multiple Web servers can access the same, read-only data on the SAN, providing ahigh-bandwidth pipe and eliminating the need for expensive, redundant copies ofthe same data.

Reducing Network Congestion from Backup

A typical problem any administrator faces is that of data backup Because of thehuge growth in data, even on local disks, and the increasing criticality of the datastored on networks, backup has become very important Software packages such

as VERITAS NetBackup, Legato NetWorker, and other packages have long relied

on agents that transport data over IP connections to a central backup host.Theresult has been a noticeable slowdown due to the vast amount of data beingtransported across these IP packets over Ethernet connections—and not just late

at night.The backup window for many enterprises has extended from overnight

to include hours of peak system operation, simply because there is too much data

to fit into the more traditional and convenient backup windows

Anecdotal stories from system administrators illustrate how entire corporatenetworks have become swamped with daily backups over IP, slowing not only e-mail, but critical file servers, print servers, and Web access Some shops havegone as far as to install separate, high-speed Ethernet networks in an attempt tooffload this problem

SANs lend themselves to several techniques that directly help the backupproblem One of these techniques is the use of IP over Fibre Channel to offloadthe network congestion on the Ethernet network IP, when transported overFibre Channel, is identical in form and function to IP over Ethernet and othernetworks.Taking advantage of the fact that there are already Fibre Channel connections into a network for access to shared data, administrators have installed

IP drivers into their servers and entirely offloaded the backup function onto theFibre Channel network.This frees the corporate Ethernet from the immense job

of transporting IP backup traffic, and takes advantage of the increased bandwidthefficiency that is characteristic of Fibre Channel Due to the connection-orientedprotocols built on Fibre Channel, IP traffic impacts the Fibre Channel networkless and helps administrators gain better usage out of their networks In addition,

an increasing number of applications can perform shared backups over a SANusing the backup devices’ native SCSI protocol, which greatly increases the efficiency of the backup process