Student Guide - Sun StorageTek™ 6000 Modular Product Line Installation and Configuration Training

● Each 6998 controller has four 4Gb/s Fibre Channel host I/O ports eight per dual controller storage system supporting direct host or SAN attachments.. The Sun StorageTek 6540 controller

Student Guide

Sun StorageTek™ 6000 Modular

Product Line Installation and

Configuration Training

August 13, 2007 12:07 pmCopyright 2007 Sun Microsystems, Inc., 901 San Antonio Road, Palo Alto, California 94303, U.S.A All rights reserved

Third-party software, including font technology, is copyrighted and licensed from Sun suppliers

Sun, Sun Microsystems, the Sun logo, Sun StorEdge, Sun StorageTek, Solaris, and Java are trademarks or registered trademarks of Sun Microsystems, Inc in the U.S and other countries

All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc in the United States and other countries Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc UNIX is a registered trademark in the U.S and other countries, exclusively licensed through X/Open Company, Ltd

Export Laws Products, Services, and technical data delivered by Sun may be subject to U.S export controls or the trade laws of other countries You will comply with all such laws and obtain all licenses to export, re-export, or import as may be required after delivery to You You will not export or re-export to entities on the most current U.S export exclusions lists or to any country subject to U.S embargo

or terrorist controls as specified in the U.S export laws You will not use or provide Products, Services, or technical data for nuclear, missile,

or chemical biological weaponry end uses.

DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS, AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE

OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

THIS MANUAL IS DESIGNED TO SUPPORT AN INSTRUCTOR-LED TRAINING (ILT) COURSE AND IS INTENDED TO BE USED FOR REFERENCE PURPOSES IN CONJUNCTION WITH THE ILT COURSE THE MANUAL IS NOT A STANDALONE TRAINING TOOL USE OF THE MANUAL FOR SELF-STUDY WITHOUT CLASS ATTENDANCE IS NOT RECOMMENDED.

Export Control Classification Number (ECCN): March 24, 2006

Copyright 2007 Sun Microsystems Inc., 901 San Antonio Road, Palo Alto, California 94303, Etats-Unis Tous droits réservés.

Ce produit ou document est protégé par un copyright et distribué avec des licences qui en restreignent l’utilisation, la copie, la distribution,

et la décompilation Aucune partie de ce produit ou document ne peut être reproduite sous aucune forme, par quelque moyen que ce soit, sans l’autorisation préalable et écrite de Sun et de ses bailleurs de licence, s’il y en a.

Le logiciel détenu par des tiers, et qui comprend la technologie relative aux polices de caractères, est protégé par un copyright et licencié par des fournisseurs de Sun.

Sun, Sun Microsystems, le logo Sun, Sun StorEdge, Sun StorageTek, Solaris, et Java sont des marques de fabrique ou des marques déposées

de Sun Microsystems, Inc aux Etats-Unis et dans d’autres pays

Toutes les marques SPARC, utilisées sous licence, sont des marques déposées ou enregistrées de SPARC International, Inc aux Etats-Unis

et dans d’autres pays Les produits portant les marques SPARC sont basés sur une architecture développée par Sun Microsystems, Inc UNIX est une marque enregistree aux Etats-Unis et dans d’autres pays et licenciée exclusivement par X/Open Company Ltd.

Législation en matière dexportations Les Produits, Services et données techniques livrés par Sun peuvent être soumis aux contrôles américains sur les exportations, ou à la législation commerciale dautres pays Nous nous conformerons à lensemble de ces textes et nous obtiendrons toutes licences dexportation, de ré-exportation ou dimportation susceptibles dêtre requises après livraison à Vous Vous nexporterez, ni ne ré-exporterez en aucun cas à des entités figurant sur les listes américaines dinterdiction dexportation les plus courantes,

ni vers un quelconque pays soumis à embargo par les Etats-Unis, ou à des contrôles anti-terroristes, comme prévu par la législation américaine en matière dexportations Vous nutiliserez, ni ne fournirez les Produits, Services ou données techniques pour aucune utilisation finale liée aux armes nucléaires, chimiques ou biologiques ou aux missiles.

LA DOCUMENTATION EST FOURNIE “EN L’ETAT” ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES, DANS LA MESURE AUTORISEE PAR LA LOI APPLICABLE, Y COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A L’APTITUDE A UNE UTILISATION PARTICULIERE OU A L’ABSENCE DE CONTREFAÇON.

CE MANUEL DE RÉFÉRENCE DOIT ÊTRE UTILISÉ DANS LE CADRE D’UN COURS DE FORMATION DIRIGÉ PAR UN INSTRUCTEUR (ILT) IL NE S’AGIT PAS D’UN OUTIL DE FORMATION INDÉPENDANT NOUS VOUS DÉCONSEILLONS DE L’UTILISER DANS LE CADRE D’UNE AUTO-FORMATION.

About This Course 1-vii

Course Goals 1-vii

Sun StorageTek™ 6540 Product Overview 1-1

Objectives 1-1Sun StorageTek 6540 Product Overview 1-2Compare the Sun StorEdge™ 6140 and the Sun StorageTek

6540 Arrays 1-4Hardware Overview 1-6Hardware Components of the Sun StorageTek 6540 1-6Controller Tray 1-7

6540 Controller Enclosure FRU details 1-8

6540 Controller Canister highlights 1-17Knowledge Check - 6540 Controller 1-31

Sun StorageTek™ 6140 Product Overview 2-37

Objectives 2-37Sun StorageTek 6140 Product Overview 2-38Compare the Sun StorEdge™ 6130 and the Sun StorageTek

6140 Arrays 2-39Hardware Components of the Sun StorageTek 6140 2-41Storage Management Software 2-42Hardware Overview 2-43Controller Tray 2-43Back View of Controller Module 2-50Controller Architecture 2-63Knowledge Check - 6140 2-64

CSMII Expansion Tray - Back View 3-82Architecture Overview 3-87Switched Bunch of Disks (SBOD) Architecture 3-88Knowledge Check 3-89

Sun StorageTek 6540 - 6140 Hardware Installation 4-93

Objectives 4-93Overview of the Installation Process 4-94Cabling Procedures 4-95Cable Types 4-95Recommended Cabling Practices 4-97Cabling for Redundancy – Top-Down

Bottom-Up 4-98Cabling for Performance 4-99Hot-adding an expansion enclosure 4-101Cabling Summary 4-105Recommended Cabling Practices 4-106Drive Cabling for Redundancy – Top-Down or

Bottom-Up 4-106Considerations for Drive Channel Speed 4-113Proper Power Procedures 4-114Turning On the Power 4-114Turning Off the Power 4-116Set the Controller IP Addresses 4-117Configuring Dynamic IP Addressing 4-117Configuring Static IP Addressing 4-118Serial Port Service Interface 4-119Serial Port Recovery Interface Procedure 4-119Use the Hardware Compatibility Matrix to Verify SAN

Components 4-122Attach the Host Interface Cables 4-122Host Cabling for Redundancy 4-122Connecting Data Hosts Directly 4-123Connecting Data Hosts through an external FC switch 4-124

Sun StorageTek 6x40 - Common Array Manager 5-131

Objectives 5-131What is the Sun StorageTek Common Array Manager? 5-132The CAM Interface 5-134SMI-S Overview 5-134Software Components 5-136Firmware and NVSRAM files 5-138CAM Management Method 5-139Out-of-Band Management Method 5-139Sun StorageTek Common Array Manager Installation 5-141Sun StorageTek Common Array Manager Navigation 5-142

Additional Navigation Aids 5-145Initial Common Array Manager Configuration 5-147Configure IP Addressing 5-147Accessing the Managment Software 5-150Naming an Array 5-150Configuring The Array Password 5-151Setting the System Time 5-151Adding Additional Users 5-151Setting Tray IDs 5-152

Array Configuration Using Sun StorageTek Common Array

Manager 6-153

Objectives 6-153Configuration Components of the Common Array Manager 6-154Creating a Volume With Common Array Manager 6-156Storage Profiles 6-156Storage Pools 6-160Volumes 6-160Virtual Disks 6-165Administration functions and parameters 6-166Knowledge Check 6-171

Storage Domains 7-173

Objectives 7-173What are Storage Domains? 7-174Storage Domains Benefits (pre-sales) 7-175Storage Domains Benefits (technical) 7-176Storage Domains Terminology 7-177Steps for creating a Storage Domain 7-181How Storage Domains Works 7-183What the Host Sees 7-184What the Storage System Sees 7-185Summary of Creating Storage Domains 7-188Knowledge Check 7-189

Monitoring Performance and Dynamic Features 8-193

Objectives 8-193Storage system parameters that can improve Performance 8-198

Snapshot Benefits (technical) 9-220How does Snapshot work? 9-221Examples of how Snapshot works 9-222Standard Read – No Snapshot 9-222Snapshot is Created 9-223Read From Snapshot (1st Case) 9-224Write to Base 9-225Re-Write to Base 9-226Read From Snapshot (2nd Case) 9-227Write to Snapshot 9-227Write to Base (1st Case) 9-229Write to Base (2nd Case) 9-230Disabling and Recreating 9-230Snapshot Considerations 9-231Snapshot OS support 9-232Managing Snapshots 9-234Creating a Snapshot 9-234Creating a Snapshot 9-237

Integrated Data Services – Volume Copy 10-239

Objectives 10-239Volume Copy Overview 10-240Volume Copy Terminology 10-240Volume Copy – Benefits (pre-sales) 10-242Volume Copy- Benefits (technical) 10-243How Volume Copy Works 10-244Factors Affecting Volume Copy 10-245Volume Copy States 10-245Volume Copy – Read/Write Restrictions 10-247Creating a Volume Copy 10-248Functions that can be performed on a Copy Pair 10-248Recopying a Volume 10-249Stopping a Volume Copy 10-249Removing Copy Pairs 10-250Changing Copy Priority 10-251Volume Permissions 10-251Volume Copy Compatibility with Other Data Services 10-252Volume Copy OS Support 10-255Configuring a Volume Copy 10-257Configuring a Volume Copy with Common Array Manager 10-257

Integrated Data Services – Remote Replication 11-263

Objectives 11-263Remote Replication Overview 11-264Remote Replication Terminology 11-265

Suspend and Resume 11-275Role Reversal 11-275How Remote Replication Works 11-276What Happens When an Error Occurs? 11-277Configuring Remote Replication 11-278Configuring the Hardware for Data Replication 11-278Configuring Data Replication with CAM 11-280Examples of Remote Replication Configurations 11-290Knowledge Check - Snapshot, Volume Copy, Remote Replication 11-293

Problem Determination 12-297

Objectives 12-297Problem Determination 12-298What tools are available? 12-299Sun StorageTek™ Common Array Manager CLI (SSCS) 12-319

About This Course

Course Goals

Upon completion of this course, you should be able to:

● Describe the features, functions and terminology of the Sun StorageTek 6540 array

● Describe the customer benefits and requirements to migrate to or use the Sun StorageTek 6540 array

● Describe the architecture of the Sun StorageTek 6540 array

● Install the Sun StorageTek 6540 array hardware

● Install the management software (Common Array Manager)

● Configure the 6540 array using CAM

● Attach production hosts to the Sun StorageTek 6540 array

● Configure and use Snapshots on the Sun StorageTek 6540 array

● Configure and use Volume Copies on the Sun StorageTek 6540 array

● Configure and use Replication Set on the Sun StorageTek 6540 array

● Diagnose problems using available tools

Course Goals

Sun StorageTek™ 6540 Product Overview

Objectives

Upon completion of this module, you should be able to:

● Describe the Sun StorageTek 6540 key features

● Identify the hardware components of the 6540 controller enclosure

● Describe the functionality of the 6540

● Interpret LEDs for proper parts replacement

Sun StorageTek 6540 Product Overview

Sun StorageTek 6540 Product Overview

Today's open systems environments create unique challenges for storage systems Round-the-clock processing requires the highest availability and online administration Varying applications result in a range of

performance requirements: from transaction-heavy (I/O per second) to throughput-intensive (Mbyte per second) Unpredictable capacity growth demands efficient scalability Finally, the sheer volume of storage in today's enterprise requires centralized administration and simple storage management

Sun StorageTek provides storage systems that are designed specifically to address the needs of the open systems environment: the Sun StorageTek

6140 and 6540 Both storage systems are high-performance, class, full 4-gigabit per second (Gbps) Fibre Channel/SATA II solution that combine outstanding performance with the highest reliability, availability, flexibility and manageability

enterprise-This course focuses on the Sun StorageTek 6540 storage system

The Sun StorageTek 6540 storage system provides the performance demanded by high performance computing (HPC) environments that store and utilize vast amounts of data for high-bandwidth programs and complex application processing The Sun StorageTek 6540 has the

powerful 6998 controller architecture and 4 Gb/s interfaces which are ideally-suited for bandwidth-oriented applications such as sophisticated data-intensive research, visualization, 3-D computer modeling, rich media, seismic processing, data mining and large-scale simulation

The 6998 controller used in the 6540 storage system is the most sophisticated and highest-performing controller to date from SUN StorageTek for the 6000 mid-range disk product line Its sixth-generation XBB architecture boasts our fastest cache memory, 4 Gbps Fibre Channel host and drive interfaces, high-speed busses, and multiple processing elements to optimize resource utilization

The 6998 controller's high-speed XOR engine generates RAID parity with

no performance penalty, enabling this compute-intensive task to be handled efficiently and effortlessly A separate 2.4 Ghz Xeon processor focuses on data movement control, allowing setup and control

instructions to be processed and dispatched independent of data

Two 6998 controllers are integrated into a controller enclosure and

combined with one or more drive enclosures to create a fully featured

6540 storage system These dual controller systems are fully-redundant and support up to eight 4, 2 or 1 Gbps Fibre Channel host connections and

224 Fibre Channel or SATA disk drives

The 6540 storage system has eight 4 Gbps FC-AL host or FC-SW SAN connections and eight 4 Gbps FC-AL drive expansion connections

Extensive compatibility and ability to auto-negotiate 4, 2, or 1 Gbps FC host connectivity speeds results in minimal or no impact on existing storage network, protecting customers’ infrastructure investment

The SUN StorageTek 6140 and 6540 storage systems run similar firmware This unique implementation creates a lower total cost of ownership and higher return on investment by enabling seamless data and model

migration, common features and functionality, centralized management, a consistent interface and reduced training and support costs Additionally, the 6140 storage system can be upgraded to a high-performance 6540 HPC storage system And in each instance, all configuration and user data remains intact on the drives

The Sun StorageTek 6540 storage system is modular and rack mountable, and scalable from a single controller tray (CRM=Controller RAID

Module) plus one expansion tray (CEM=Controller Expansion Module) to

a maximum of 13 additional expansion trays

Summary of he features offered by the Sun StorageTek 6540 storage system:

● The SUN StorageTek 6540 has two 6998 controllers

● Each 6998 controller has four 4Gb/s Fibre Channel host I/O ports (eight per dual controller storage system) supporting direct host or SAN attachments The eight 4 Gb/s Fibre Channel host ports

support 4, 2 and 1 Gb/s connectivity

● Each 6998 controllers has the powerful 2.4 Ghz Intel Xeon processor Each controller also has a dedicated next generation ASIC to perform the RAID parity calculation thereby off loading the processor of this function

Sun StorageTek 6540 Product Overview

● 4GB, 8GB and 16GB cache options are available (2GB / 4GB / 8GB per controller respectively)

● 4 drive loops per controller that can support either 2Gb/s or 4Gb/s drive enclosures

● All components are hot-swappable

Table 1-1 Comparison Chart: 6140 and 6540 Differences

Sun StorageTek

6140 Lite

Sun StorageTek 6140

Sun StorageTek 6540

ControllerCPU Processor 667 Mhz

Xscale w/

XOR

667 Mhz Xscale w/

XOR

2.4 Ghz Xeondedicated XORHost Ports 1/2/4 Gb/s

ctlrEthernet Ports 2 per ctlr 2 per ctlr 2 per ctlr

Disk Types 2/4 Gb/s: FC,

SATA II

2/4 Gb/s: FC,SATA II

2/4 Gb/s: FC, SATA

2/4 Gb/s: FC, SATAII

ConfigurationMaximum Hosts 512

(256 redundant)

512(256 redundant)

512

Performance TargetsBurst I/O rate -

120,10044,0009,000

575,00085,00022,000Sustained

6140 Lite

Sun StorageTek 6140

Sun StorageTek 6540

Hardware Overview

Hardware Overview

Hardware Components of the Sun StorageTek 6540

The Sun StorageTek 6540 storage system is comprised of two main trays: the 6540 controller tray and a minimum of one expansion tray The expansion tray is also known as the Common Storage Module 2 (CSM2 or CSMII)

Figure 1-1 Sun StorageTek 6540 storage system

This section describes the main components of the Sun StorageTek 6540 controller tray (CRM) The CSMII is covered in another module

Figure 1-2 Components of the 6540 controller enclosure

Controller Tray

Figure 1-2 shows a block diagram for the Sun StorageTek 6540 The blocks represent placement of controllers, power-fan canisters and removable mid-plane canister

Figure 1-3 Block diagram for the Sun StorageTek 6540

The Sun StorageTek 6540 controller enclosure has five main canisters:

● two Power-Fan canisters

● one Interconnect canister (removable mid-plane)

● two controller canisters

There are also two battery FRU’s (Field Replaceable Units) within the Interconnect-Battery canister, bringing the total number of FRU’s for the

6540 controller enclosure to seven

The enclosure does not have a mid-plane but instead has been designed such that all the canisters interconnect with one another

Caution – Service Advisor procedures should be followed when

removing a FRU because there is interdependency between the FRU’s

The two controllers are located in the rear of the enclosure.

All canisters are hot swappable as long as interdependencies between the FRU’s are taken into consideration

• Power-Fan canister (x2) – Power supply – Fans

– Battery chargers (x2) – Thermal sensor

• Interconnect-Battery canister – Mid-plane

– Battery packs (x2) – Audible alarm – Front bezel LEDs

• Controller canisters in the back (x2) – Base controller board

– Manufacturing configurable host interface card

2 Battery Chargers Thermal Sensor

Right Power/Fan Canister

Connects to controller A

Power Supply Fans

2 Battery Chargers Thermal Sensor

Interconnect Canister Connects to both controllers

2 battery packs Audible alarm LED’s

Power-Fan Canister

Figure 1-5 6540 Power Fan Canister and Battery Canister LEDs

The main purpose of the Power-Fan canister is as the name suggests - to provide power and cooling to the storage system

Each Power-Fan canister contains:

● a power supply - provides power to the controllers by converting incoming AC voltage to the appropriate DC voltages In addition to the AC-to-DC power supply, a DC-to-DC power supply will be supported when it becomes available (there is a DC connector on the controller canister but it is not currently functional)

● two system cooling fans - the fans are powered by the power supply

in both Power-Fan canisters If either power supply fails, the fans will continue to operate

● two battery chargers - the battery chargers perform battery tests when the 6540 enclosure is first power on, and every 25 hours thereafter If needed, the batteries will be recharged at that time The batteries are located in the Interconnect-Battery canister

● thermal sensor - prevents power supplies from overheating Under normal operating conditions, with an ambient air temperature of 5°C

Hardware Overview

● fan failure

● defective circuitry in the power supply

● blocked air vent

● failure in other devices installed in the cabinet

If the internal temperature rises above 70°C (158°F) one or both power supplies will automatically shut down, and the storage management software will report the exception Critical event notifications will also be issued if event monitoring is enabled and event notification is configured

In the figure above, note the black connector when looking at the back of the canister - this connector connects to one of the controllers The Power-Fan canister on the right has the connector at the top and therefore connects to controller A The Power-Fan canister on the left is upside down and has the connector on the bottom and therefore connects to controller B

Figure 1-6 6540 Power Fan canister LEDs

Information about the condition of the power supplies, fans and battery charger is conveyed by indicator lights on the front of each Power-Fan canister You must remove the front cover of the 6540 enclosure to see the indicator lights

Power-Fan canister LEDs

Typically there is a one-to-one relationship between the Needs

Attention/Service Action Required (SAR) and Ok to Remove/Service Action Allowed (SAA) LEDs But there are exceptions An example is if both Power-Fan canisters have a fault, one due to a power fault, and the other due to a fan fault The Power-Fan canister with the power fault should be removed and replaced first If the Power-Fan canister with the fan fault is removed, the system would be left with no power In this case, the Power-Fan canister with the fan fault would have the SAR LED ON, but the SAA LED OFF

Interconnect-Battery Canister

The purpose of the Interconnect-Battery canister is to serve as a midplane for pass thru of controller status lines, power distribution lines, and drive channels Additionally it contains the batteries to hold data in cache in the event of loss of power, summary indicators for the entire storage system, and the audible alarm

The Interconnect-Battery canister contains:

● a removable mid-plane - provides cross-coupled signal connection between the controller canisters The control output from each controller canister is connected to the control input of the alternate controller canister

● two battery packs - provide backup power to the controller cache memory Each battery pack is sealed and contains two clusters of lithium ion batteries Each battery pack is connected to both

controllers - one cluster to controller A, the other to controller B The battery pack voltage ranges from 9 to 13 V When two battery packs are present, the 6540 storage system data cache will be backed up for

3 days

Hardware Overview

● front bezel LED’s - the LED’s that are displayed through the front cover are located on the Interconnect-Battery canister

Figure 1-7 Inter-connect battery canister LEDs

Information about the condition of the interconnect-battery canister is conveyed by indicator lights on the front of the Interconnect-Battery canister

The Power, Service Action Required, and Locate lights are general indicators for the entire command enclosure, not specifically for the Interconnect-Battery canister The Service Action Required light turns on

if a fault condition is detected in any component in the controller enclosure The Power, Service Action Required, and Locate lights shine through the front cover

The Service Action Allowed LED is for the Interconnect-Battery canister itself.

Caution – Never remove the Interconnect-Battery canister unless directed

to do so by Customer Support Removing the Interconnect-Battery

canister after either a controller or a Power-Fan canister has already been removed could result in loss of data access

In the unlikely event an Interconnect-Battery canister must be replaced (i.e due to a bent pin, or as a last resort to resolve a problem) then the storage management software will provide details on the procedure Data access is limited to only one controller (Controller A) when the

Battery canister is removed Removal of the Battery canister will automatically suspend controller B, and all I/O will

Interconnect-be performed by controller A It is recommended that you prepare for the removal of the Interconnect-Battery canister instead of just pulling it out Preparation involves:

● Placing controller B offline so that host failover software can detect the offline controller and re-route all I/O to controller A

● Turning ON the Service Action Allowed LED using the storage management software

● Removing and replacing the Interconnect-Battery canister

● Turning OFF the Service Action Allowed LED using the storage management software

● Placing controller B on-line and re-balancing the volumes

Hardware Overview

Interconnect-Battery Canister - Battery pack.

Figure 1-8 6540 Interconnect battery canister showing a single battery

pack

The above figure shows the Interconnect-Battery canister with the access cover removed For clarity, the picture shows only one battery pack, there would normally be two The battery pack is mounted to a sheet metal bracket You can see the flange at the end of the bracket closest to the access - grasp the flange to remove the battery pack When replacing the battery pack, the battery pack must be pushed firmly into the

interconnect-battery canister to ensure it completely engages with the connectors at the back of the Interconnect-Battery canister

Power Distribution and Battery System

Figure 1-9 6540 as seen from the top, showing the power distribution

The 6540 enclosure does not have a midplane (sometimes also referred to

as a backplane) that can be found in all pre-sixth generation SUN

StorageTek 6140 and 6130 products

This diagram shows how the canisters are interconnected, and also gives

an overview of how the power distribution and battery system work

The power from the left Power-Fan canister is distributed via controller B, and power from the right Power-Fan canister is distributed via controller

A Both controllers must be in place in order to provide redundant power

Charger

Charger

Charger

Cache Memory Voltage Regulator

A

Battery Packs

Hardware Overview

Figure 1-10 Which component should be replaced first - Right

Power/Fan Canister or the Left Power/Fan Canister?

Service Advisor procedures must be followed carefully if both the power supply connected to controller B fails (the left Power-Fan canister), and controller A fails Removing controller A before replacing the failed Power-Fan canister will cause controller B to lose power, resulting in loss

of data access This occurs because power distribution from each Fan canister is through the controller physically connected to that Power-Fan canister

Power-Figure 1-11 Which component should be replaced first - Controller A or

the Left Power/Fan Canister?

The battery system spans all the canisters:

● The two battery packs are in the Interconnect-Battery canister Half

of each battery pack is dedicated to each controller

● The two charging circuits in each of the Power/Fan Canisters - one charger for one battery cluster in each of the battery packs

● Voltage regulator in each controller to ensure the lithium ion

batteries are not over charged

6540 Controller Canister highlights

Processors

● 6091-0901 controller model number (also referred to as 6998)

● Next generation hardware XOR engine

● Two 4 Gbps FC loop switches per controller

● Total of 8 drive loops per system

● Run at 2 Gbps and 4 Gbps

● Auto-detect drive side speed

● Can support both 2 Gbps and 4 Gbps drive enclosures behind the same controller on different drive channels

Dual 10/100 Ethernet for out-of-band management

● One for customer out-of-band management

● One for service diagnostics, serviceability

Hardware Overview

6540 Controller Canister

Figure 1-12 6540 Controller Canister LEDs in Front

The 6540 command enclosure has two 6998 controllers Both controllers are identical The controllers install from the rear of the command enclosure The top controller is controller A and the bottom controller is controller B All connections to the hosts and the drives in the storage controller are through the controller canisters

The host side connections support fibre-optic connections The drive side connections support either copper or fibre-optic connections

The 6998 controller inside the controller canister is comprised of two circuit boards:

● The base controller board - contains the 2.4 GHz processor, the DIMM slots for cache memory, and four Emulex SOC 422 ’loop switch’ chips for the four drive channels Each loop switch combines two loops together for one drive channel, and also provides an external connection for each loop

● The host interface card - plugs into the base controller board and provides the four 4 Gbps host side connections In the future, there will be several variations of the host interface card, thus allowing the customer to order a host interface card that has the number, type and speed of host connections that meets his needs

6540 Controller Canister Connections

Figure 1-13 6540 Controller canister connections

6540 Controller Canister LED indicators

Figure 1-14 6540 Controller canister LED indicators

Each 6540 controller canister provides the following connections and LED indicators which are described in detail in the following sections:

● Four 4 Gbps Host Interface Ports

● Four 4 Gbps Disk expansion ports

● Dual 10/100 Base-T Ethernet Ports With EEPROM

Hardware Overview

Figure 1-15 Controller Tray LEDs and Indicators

6540 4 Gbps Host Interface Ports

Figure 1-16 6540 Host and Drive interface ports

The 6540 storage system has eight 4 Gbps FC-AL host or FC-SW SAN connections

● The host side connections perform link speed negotiation on each host channel port (also referred to as auto-negotiation) for 4, 2, or 1 Gbps FC host connectivity speeds resulting in minimal or no impact

on the existing storage network Link speed negotiation for a given host channel is limited to link speeds supported by the Small Form-factor Pluggable (SFP) transceiver on that channel The controllers will enter into auto-negotiation at these points in time:

● Detection of a link-up event after a previous link-down event

If the auto-negotiation process fails, the controllers will consider the link

to be down until negotiation is again attempted at one of these points in time For a 4-Gb controller, the supported link speeds are 1, 2, and 4Gbps

Auto-Negotiation

The Fibre Channel host interface performs link speed negotiation on each host channel Fibre Channel port This process, referred to as auto-

negotiation, means that it will interact with the host or switch to

determine the fastest compatible speed between the controller and the other device The fastest compatible speed will become the operating speed of the link If the device on the other end of the link is a fixed speed device or is not capable of negotiating, the controller will automatically detect the operating speed of the other device and set its link speed accordingly

6540 4 Gbps Disk Expansion Ports

Figure 1-17 Disk expansion ports

Each 6540 controller canister has two dual-ported drive channels Each

6540 controller has two drive channels, each channel consists of two drive

Hardware Overview

The connections for Channel 1 and Channel 2 are on Controller A The connections for Channel 3 and Channel 4 are on Controller B When attaching drive enclosures, it is important that the drive enclosure is cabled to a drive channel on each controller to ensure redundancy

The drive channels can operate at 2 Gbps or 4 Gbps The drive channels perform link speed detection (which is different than link speed

negotiation) - the controller will automatically match the link speed of the attached drive enclosures Drive channels can operate at different link speeds, but both ports of a single channel must run at the same speed

Two LEDs indicate the speed of the channel of the disk drive ports, as shown in the figure below

Figure 1-18 Disk Expansion Ports

The behavior of the LEDs is as follows:

● When both LEDs are OFF, there is no FC connection or the link is down

● With the first LED in the OFF position and the right LED in the ON position, the port is at 2 Gbps

● When both LEDs are in the ON position, the port is at 4 Gbps

Fibre Channel Port By-Pass Indicator

The fibre channel port by-pass indicator has two settings: on and off Figure 1-19 shows the indicator

Figure 1-19 Port By-Pass Indicator

When in the OFF position, no SFP is installed or port is enabled In the

ON position, no valid device is detected and the channel or port is internally bypassed (AMBER)

2 4

P1 Ch 2 (Ctrl B)Ch 2 (Ctr l A) P2

6540 Drive Channels and Loop Switches

Figure 1-20 6540 Drive Channels and Loop Switches

Each drive port is capable of delivering 400 MB/s of bandwidth; however, both ports of a loop switch (one channel) will run at the same link speed - either both ports will run at 4 Gbps or 2 Gbps

Each controller has two dual-ported 4 Gbps FC Chips Each FC Chip is attached to a Loop Switch chip on both Controller A and Controller B, therefore both controllers are connected to all four Drive Channels Both the FC Chips and the Loop Switch chips support concurrent full link speed on both ports of each chip

Each Loop Switch chip represents a Drive Channel

Each Drive Channel can support a maximum of 126 devices (drives, IOM’s and controllers) The 6540 subsystem supports a maximum of 224 disks

Each Drive Channel has two independent drive loops, represented by the two ports per Drive Channel

Hardware Overview

Figure 1-21 Each drive channel has 2 ports

Host and drive side cabling will be covered after a hardware overview of the CSMII drive enclosure

Dual 10/100 Base-T Ethernet Ports With EEPROM

Figure 1-20 illustrates the ethernet status LEDs

Figure 1-22 Ethernet Status Lights

The 6540 has two RJ-45 ports per controller canister Ethernet port 1 must

be for Management Host while port 2 reserved for future use Do not use this port for management of the trays

Default IP addresses (default subnet is 255.255.255.0):

ControllerA interface 0: 192.168.128.101ControllerA interface 1: 192.168.129.101ControllerB interface 0: 192.168.128.102ControllerB interface 1: 192.168.129.102

Serial Port Connector

To access the serial port, use a RS232 DB9 null modem serial cable This port is used to access the Service Serial Interface used for viewing or setting a static IP address for the controllers This interface can also clear the storage system password

Figure 1-21shows the RS232 DB9 null modem cable for serial port access

Figure 1-23 RS232 null modem cable

Seven-Segment Display

Light Color Normal Status

Ethernet Link Speed Green LED Off = 10 Base-T

On = 100 Base-TEthernet Link Activity Green LED Off = No link established

On = Link establishedBlinking = Activity

Hardware Overview

The numeric display consists of two seven-segment LEDs that provide information about enclosure identification and diagnostics When the controller enclosure is operating normally, the numeric display shows the tray identification (tray ID) of the controller enclosure

The controller enclosure tray ID is intentionally set from 80-99 by the controller firmware and automatically adjusts during power-on to avoid conflicts with existing drive tray IDs There is no physical ID selector on the controller enclosure but you can, however, set the controller enclosure tray ID through the storage management software The controller tray ID should not be changed to an ID below 80 as it will not work properly

Each digit of the numeric display has a decimal point, and is rotated 180 degrees relative to the other digit With this orientation, the display looks the same regardless of controller orientation The numeric display as shown in Figure 1-24 shows the tray identification (Tray ID) or a diagnostic error code

The heartbeat is the small decimal on the lower right hand corner of the 1st digit - when the heartbeat is blinking the number displayed is the Tray

ID The diagnostic light is the small decimal in the upper left hand corner

of the 2nd digit - when the diagnostic light is blinking the number displayed is a diagnostic code

The tray ID is an attribute of the 6540 command enclosure; both controllers display the same tray ID It is possible, however, that one controller will display the tray ID, while the other controller displays a diagnostic code

● Power on behavior - The Diagnostic Light, the Heartbeat Light, and all 7 segments of both digits will be on if a power-on or reset occurs The tray ID display may be used to temporarily display diagnostic codes after each power cycle or reset The Diagnostic Light will remain on until the tray ID is displayed After diagnostics are completed, the current tray ID will be displayed

● Diagnostic behavior - Diagnostic codes in the form of Lx or Hx, where x is a hexadecimal digit, are used to indicate state information

In general, these codes are displayed only when the canister is in a non-operational state The canister may be non-operational due to a configuration problem (such as mismatched IOM and/or controller types), or it may be non-operational due to hardware faults If the controller/IOM is non-operational due to system configuration, the

controller/IOM Fault Light will be off If the controller/IOM is operational due to a hardware fault, the controller/IOM Fault Light will be on

non-Value Description

Boot FW is booting up

FF Boot Diagnostic executing

88 This controller/IOM is being held in reset by the

other controller/IOM

AA ESM-A application is booting

bb ESM-B application is booting

L0 Mismatched IOM types

L2 Persistent memory errors

L3 Persistent hardware errors

L9 Over temperature

H0 SOC (Fibre Channel Interface) Failure

H1 SFP Speed mismatch (2 Gb SFP installed when

operating at 4 Gb)H2 Invalid/incomplete configuration

H3 Maximum reboot attempts exceeded

H4 Cannot communicate with the other IOM

H5 Mid-plane harness failure

H6 Firmware failure

H7 Current enclosure Fibre Channel rate different than

rate switchH8 SFP(s) present in currently unsupported slot (2A or

2B)

Hardware Overview

Controller Service Indicators

Figure 1-25 shows controller service indicators

Figure 1-25 Controller Service Indicators

Service Action Allowed LED

Figure 1-26 shows the Service Action Allowed LED

Figure 1-26 Service Action Allowed LED

● Normal Status is OFF

● Problem Status is ON - OK to remove canister A service action can be performed on the designated component with no

adverse consequences (BLUE)

Each drive, power-fan, and controller/IOM canister has a Service Action Allowed light The Service Action Allowed light lets you know when you can remove a component safely

Caution – Potential loss of data access – Never remove a drive, power-fan,

or controller or IOM canister unless the Service Action Allowed light is turned on

● If a drive, power-fan, or controller/IOM canister fails and must

be replaced, the Service Action Required (Fault) light on that canister turns on to indicate that service action is required The Service Action Allowed light will also turn on if it is safe to remove the canister If there are data availability dependencies

or other conditions that dictate that a canister should not be removed, the Service Action Allowed light will remain off

● The Service Action Allowed light automatically turns on or off

as conditions change In most cases, the Service Action Allowed light turns on when the Service Action Required (Fault) light is