Student Guide - IES-421 Sun Fire™ 15K Server Maintenance and Administration

6-60Task 1: Configuring Sun StorEdge Traffic Manager Software .... About This Course Course Goals Upon completion of this course, you should be able to: ● Remove and install most Sun Fir

Sun Microsystems, Inc.

UBRM05-104

500 Eldorado Blvd Broomfield, CO 80021

U.S.A.

Sun Fire™ 15K Server Maintenance and Administration

IES-421

Copyright 2001 Sun Microsystems, Inc., 901 San Antonio Road, Palo Alto, California 94303, U.S.A All rights reserved.

This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation No part of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any.

Third-party software, including font technology, is copyrighted and licensed from Sun suppliers.

Sun, Sun Microsystems, the Sun logo, JumpStart, OpenBoot, Solaris, Solstice DiskSuite, Sun Enterprise, SunFastEthernet, Sun Fire, Sun Management Center, SunOS, Sun Quad FastEthernet, SunSolve, SunSolve Online, SunSpectrum, Sun StorEdge, SunVTS, and Ultra are trademarks or registered trademarks of Sun Microsystems, Inc in the U.S and other countries.

All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc in the U.S and other countries Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc.

UNIX is a registered trademark in the U.S and other countries, exclusively licensed through X/Open Company, Ltd.

U.S Government approval might be required when exporting the product.

RESTRICTED RIGHTS: Use, duplication, or disclosure by the U.S Government is subject to restrictions of FAR 52.227-14(g)(2)(6/87) and FAR 52.227-19(6/87), or DFAR 252.227-7015 (b)(6/95) and DFAR 227.7202-3(a).

DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS, AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

THIS MANUAL IS DESIGNED TO SUPPORT AN INSTRUCTOR-LED TRAINING (ILT) COURSE AND IS INTENDED TO BE USED FOR REFERENCE PURPOSES IN CONJUNCTION WITH THE ILT COURSE THE MANUAL IS NOT A STANDALONE TRAINING TOOL USE OF THE MANUAL FOR SELF-STUDY WITHOUT CLASS ATTENDANCE IS NOT RECOMMENDED.

Sun Proprietary: Internal Use Only

Copyright 2001 Sun Microsystems Inc., 901 San Antonio Road, Palo Alto, California 94303, Etats-Unis Tous droits réservés.

Ce produit ou document est protégé par un copyright et distribué avec des licences qui en restreignent l’utilisation, la copie, la distribution,

et la décompilation Aucune partie de ce produit ou document ne peut être reproduite sous aucune forme, par quelque moyen que ce soit, sans l’autorisation préalable et écrite de Sun et de ses bailleurs de licence, s’il y en a.

Le logiciel détenu par des tiers, et qui comprend la technologie relative aux polices de caractères, est protégé par un copyright et licencié par des fournisseurs de Sun.

Sun, Sun Microsystems, le logo Sun, Java, JumpStart, OpenBoot, Solaris, Solaris Operating Environment, Solaris Web Start, Solstice DiskSuite, Sun Cluster, Sun Enterprise, Sun Enterprise 10000, Sun FastEthernet, Sun Fire, Sun Fireplane interconnect, Sun Management Center, SunOS, Sun Quad FastEthernet, SunSolve, SunSolve Online, Sun Spectrum, Sun StorEdge A5000, Sun StorEdge T3, SunVTS, Ultra

5, et Ultra 10 sont des marques de fabrique ou des marques déposées de Sun Microsystems, Inc aux Etats-Unis et dans d’autres pays Toutes les marques SPARC sont utilisées sous licence sont des marques de fabrique ou des marques déposées de SPARC International, Inc aux Etats-Unis et dans d’autres pays.

Les produits portant les marques SPARC sont basés sur une architecture développée par Sun Microsystems, Inc.

L’accord du gouvernement américain est requis avant l’exportation du produit.

LA DOCUMENTATION EST FOURNIE “EN L’ETAT” ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES, DANS LA MESURE AUTORISEE PAR LA LOI APPLICABLE, Y COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A L’APTITUDE A UNE UTILISATION PARTICULIERE OU A L’ABSENCE DE CONTREFAÇON.

CE MANUEL DE RÉFÉRENCE DOIT ÊTRE UTILISÉ DANS LE CADRE D’UN COURS

DE FORMATION DIRIGÉ PAR UN INSTRUCTEUR (ILT) IL NE S’AGIT PAS D’UN OUTIL DE FORMATION INDÉPENDANT NOUS VOUS DÉCONSEILLONS DE L’UTILISER DANS LE CADRE D’UNE AUTO-FORMATION.

Table of Contents

About This Course xxiii

Course Goals xxiii

Course Map xxiv

Topics Not Covered xxv

How to Use Course Materials xxvi

Conventions xxvii

Icons xxvii

Typographical Conventions xxviii

Introducing the Sun Fire™ 15K Server 1-1

Objectives 1-1 Relevance 1-2 Additional Resources 1-2 Introducing the Sun Fire™ 15K Server 1-3 Generation Scaling 1-4 The Sun Fire Systems 1-6 System Enclosures 1-6 Slot 0 Boards 1-7 Slot 1 Boards 1-7 System Controller Boards 1-7 System Configurations 1-8 Cabinets 1-9 Sun Fire 15K I/O Support 1-11 Interdomain Networking 1-11 Sun Fire 15K Server Peripherals 1-12 Advanced RAS Features 1-13 Exercise: Training Lab Guided Tour 1-14 Preparation 1-14 Task 1-14 Exercise Summary 1-15 Check Your Progress 1-16

Sun Fire™ 15K Server Architecture 2-1

Objectives 2-1Relevance 2-2Additional Resources 2-2Architecture Overview 2-3The Sun Fire 15K Bus Architecture 2-4Address Bus 2-4Data Bus 2-4Interconnect Levels 2-5The Address Interconnect 2-6Data Interconnect 2-7Sun™ Fireplane Interconnect 2-8Memory Coherency 2-10Snoopy Coherency 2-10Scalable Shared Memory 2-11System Components 2-12Board Sets 2-13System Board Set 2-14Control Board Set 2-16System Boards 2-18System Board Operational Overview 2-19CPU and Memory Configuration 2-20Memory Configuration Rules 2-22System Board LEDs 2-24 Sun Fire 15K I/O Support 2-25hsPCI I/O Board Operational Overview 2-26hsPCI I/O Assembly Slot Assignments 2-27hsPCI I/O Assembly Status LEDs 2-28MaxCPU Board 2-30MaxCPU Board Components and Status LEDs 2-31Sun Fire Link Board 2-33Sun Fire Link Board Components and Status LEDs 2-34Sun StorEdge™ S1 2-36StorEdge S1 Switch and Indicators 2-37StorEdge S1 Daisy-Chain Configuration 2-39Expander Board 2-40The System Controller 2-42System Controller Components 2-43The System Controller Faceplate 2-44The System Controller LEDs and Controls 2-45System Control Board LEDs 2-45System Controller Peripheral Board LEDs 2-48System Controller Physical Locations 2-51Centerplanes 2-52Sun Fireplane Interconnect 2-52Power Centerplane 2-54

Fan Backplane 2-55Carrier Plates 2-56Carrier Plate Label 2-58Electrical Specifications 2-59Processor Cabinet Power System 2-59Specifications 2-60Input Power Requirements 2-61Power Distribution 2-62

AC Power Supplies 2-63Power Supply LEDs 2-63Installing Power Cables 2-65System Power On and Power Off 2-67Powering On the Sun Fire 15K Server 2-67Powering Off the Sun Fire 15K Server 2-68Processor Cabinet Cooling System 2-69Fan Trays 2-70FrameManager 2-72FrameManager Front Panel 2-73Physical Locations 2-75Exercise: Sun Fire 15K Server Component Identification 2-77Preparation 2-77Tasks 2-77Exercise: Sun Fire 15K FRU Removal and Installation 2-80Preparation 2-80Tasks 2-82Exercise: Powering the Platform On and Off 2-85Preparation 2-85Tasks 2-85Exercise Summary 2-86Exercise: Sun Fire 15K Server Component Identification 2-87Task Solutions 2-87Exercise: Sun Fire 15K Server FRU

Removal and Installation 2-90Task Solutions 2-90Exercise: Powering the Platform On and Off 2-93Task Solutions 2-93Check Your Progress 2-94

Sun Fire™ 15K System Controller Architecture 3-1

Objectives 3-1Relevance 3-2Additional Resources 3-2System Controller Architecture Overview 3-3Introducing the System Controller 3-4System Controller Hardware Complex 3-5

Control Board 3-9System Controller Peripheral (SCPER) Board 3-11Centerplane Support Board 3-12System Controller-to-Platform Interfaces 3-13Management Networks (MANs) 3-14Console Bus 3-21

I2C Bus 3-22System Clock 3-23System Controller Software 3-25SMS Capabilities 3-25SMS Startup 3-27SMS Daemons 3-30Message Logging Daemon 3-31SMS Startup Daemon 3-33Hardware Access Daemon 3-36Management Network Daemon 3-38FRU Access Daemon 3-41Failover Management Daemon 3-42Platform Configuration Database Daemon 3-43Task Management Daemon 3-44Domain Status Monitoring Daemon 3-45Environmental Status Monitoring Daemon 3-46OpenBoot PROM Support Daemon 3-47Key Management Daemon 3-48Domain X Server 3-50Domain Configuration Administration Server 3-51System Controller Environment Variables 3-53SMS File System Structure 3-54SMS Installation 3-57Manually Installing the

System Management Services Software 3-57Configuring the System Controller 3-59Changing the Configuration

During the Initial Startup 3-59Configuring the Management Network 3-60Thesmsconfig Script 3-61Using thesmsconfig Script 3-64Configuring the Name Services 3-68Logging In To The System Controller 3-69Using thetelnet Command 3-69Using the Serial Port 3-69Exercise: Configuring the System Management Services 3-71Preparation 3-71Task 1: Installing and Configuring SMS Software 3-71Task 2: Logging In To the Actual System Controller 3-71Exercise Summary 3-74

Exercise: Configuring the System Management Services 3-75Task 1 Solutions 3-75Task 2 Solutions 3-75Check Your Progress 3-78

Configuring the Sun Fire™ 15K Platform 4-1

Objectives 4-1Relevance 4-2Additional Resources 4-2Architecture Overview 4-3Sun Fire 15K Administrative Privileges 4-4Managing Administration Groups 4-5Adding Administrators 4-8Group Privileges 4-11Platform Administrator Group 4-11Platform Operator Group 4-13Platform Service Group 4-14Domain Administrator Group 4-15Domain Configuration Group 4-16Superuser Privileges 4-17ACL 4-18Platform Administration Tasks 4-19Administrative Commands 4-20Displaying Platform Configuration 4-23Thesmsversion Command 4-24Theshowplatform Command 4-25Theshowboards Command 4-30Theshowcomponent Command 4-32Monitoring Platform Environmentals and Status 4-34Theshowenvironment Command 4-34Theshowlogs Command 4-39Setting Up the Platform 4-41Thesetupplatform Command 4-41Thesetdate Command 4-43Thesetdefaults Command 4-45Powering Platform Components On and Off 4-46Thepoweron Command 4-46Thepoweroff Command 4-48Updating the Firmware 4-50Theflashupdate Command 4-51Examples of theflashupdate Command 4-52Backing Up and Restoring the SMS Environment 4-53Thesmsbackup Command 4-53Thesmsrestore Command 4-54Re-initializing the System Controller 4-55

Exercise: Platform Monitoring and Configuration 4-56Preparation 4-56Task 1: Configuring Administration Groups 4-56Task 2: Monitoring the Platform 4-57Task 3: Configuring Domain ACLs 4-58Task 4: Updating Flash PROM Images 4-58Task 5: Backing Up and

Restoring the SMS Environment 4-59Exercise Summary 4-60Exercise: Platform Monitoring Configuration 4-61Task 1 Solutions 4-61Task 2 Solutions 4-62Task 3 Solutions 4-63Task 4 Solutions 4-63Task 5 Solutions 4-64Check Your Progress 4-65

Configuring Sun Fire™ 15K Domains 5-1

Objectives 5-1Relevance 5-2References 5-2Architecture Overview 5-3Sun Fire 15K Domains 5-4Static and Dynamic Domains 5-4Domain Configuration 5-5DCUs 5-5Domain Configuration Requirements 5-6Configuring Static Domains 5-7Theaddtag Command 5-8Thedeletetag Command 5-9Theaddboard Command 5-10Thedeleteboard Command 5-13Themoveboard Command 5-16Thesetobpparams Command 5-19Domain Configuration Example 5-21The Virtual Keyswitch 5-24Thesetkeyswitch Command 5-24Displaying the Virtual Keyswitch Setting in Domain 5-27Accessing the Domain Console 5-28Theconsole Command 5-28Displaying the Domain Devices 5-31Theshowdevices Command 5-31Resetting the Domain 5-34Thereset Command 5-34The OpenBoot PROM Device Tree 5-36OBP Device Tree 5-37

Sun Fire™ Physical Device Mapping 5-40Device Mapping Algorithm 5-40Decoding CPU and Memory Locations 5-42Decoding I/O Card Locations 5-45The Agent ID 5-45IOC PCI Buses 5-47Slot Number 5-48The Sun Fire 15K Solaris OE 5-50Configuring the Solaris OE in a New Domain 5-51Setting Up the OBP for the Default Boot Device 5-51Configuring the Domain’s MAN Interface 5-52Setting the Domain’s Time-of-Day Clock 5-53Configuring NTP in the Domain 5-55Finishing the Installation 5-55System Controller Domain Log Files 5-56Displaying the System Configuration 5-57Thendd man_get_hostinfo Command 5-58Theprtconf Command 5-59Theformat Command 5-60Theprtdiag Command 5-61Theeeprom Command 5-64Exercise: Creating a Sun Fire 15K Server Domain 5-65Preparation 5-65Task 1: Configuring a Domain 5-68Task 2: Analyzing the OpenBoot PROM Device Tree 5-70Task 3: Booting the Solaris OE For the First Time 5-72Task 4: Assessing the Domain Using

Standard Solaris OE Utilities 5-74Exercise Summary 5-75Exercise Solutions: Creating a Sun Fire 15K

Server Domain 5-76Task 1 Solutions 5-76Task 2 Solutions 5-78Task 3 Solutions 5-80Task 4: Assessing the Domain Using

Standard Solaris OE Utilities 5-82Check Your Progress 5-83

Multipathing I/O Management 6-1

Objectives 6-1Relevance 6-2Additional Resources 6-2Multipathing I/O 6-3Sun StorEdge™ Traffic Manager 6-3Internet Protocol Multipathing 6-3

Sun StorEdge Traffic Manager Features 6-4Product Comparison 6-5Supported Software and Hardware 6-6Sun StorEdge Traffic Manager Terms 6-7Overview 6-9Sun StorEdge Traffic Manager Device Tree 6-11Sun StorEdge Traffic Manager Device Paths 6-12Sun StorEdge Traffic Manager Path Properties 6-14Path States 6-14SCSI Device Attributes 6-15Configuring Sun StorEdge Traffic Manager Software 6-16Domain Configuration Verification 6-16System Patch Installation 6-18Downloading System Patches 6-18Post-Installation Setup and Verification 6-21Configuring Sun StorEdge Traffic Manager 6-22Upgrading the Solaris OE 6-23Automatic Path Discovery 6-24Sun StorEdge Traffic Manager Management 6-25Theluxadm Command 6-25Theformat Command 6-31Automatic Failover 6-32IPMP 6-34IPMP Features 6-34IPMP Components 6-35IPMP Hardware and Software Requirements 6-37Hardware Requirements 6-37Software Requirements 6-37IPMP Group Requirements 6-38How IPMP Works 6-39Network Path Failure Detection 6-39Network Path Failover 6-40Network Path Failback 6-40Multipathing Configuration File 6-41Starting thein.mpathd Daemon 6-42Theifconfig Command Status Flags and Subcommands 6-43Newifconfig Command Status Flags 6-43Newifconfig Command Subcommands 6-44Configuring IPMP 6-45Creating an IP Multipathing Group 6-45Adding and Deleting Physical Network Interfaces 6-46Configuring Test Interfaces 6-46Configuring Standby Interfaces 6-47Configuring Logical Interfaces 6-47IPMP Implementation 6-48Configuring A Single Active Path With Standby 6-48

Configuring Multiple Active Paths Without Standby 6-52Automating IPMP Configuration in IPv4 6-56Configuring IPMP in IPv6 6-57Manually Configuring IPMP in IPv6 6-57Automating IPMP Configuration in IPv6 6-58

DR Interaction With IPMP 6-59Exercise: Installing Sun StorEdge

Traffic Manager Software 6-60Preparation 6-60Task 1: Configuring Sun StorEdge

Traffic Manager Software 6-61Task 2: Confirming Sun StorEdge

Traffic Manager Software Operation 6-61Exercise: Configuring and Managing IPMP 6-63Preparation 6-63Task 1: Exploring IPMP 6-64Task 2: Configuring IPMP 6-65Exercise Summary 6-66Exercise: Installing Sun StorEdge

Traffic Manager Software 6-67Task 1 Solutions 6-67Task 2 Solutions 6-67Exercise: Configuring and Managing IPMP 6-69Task 1 Solutions 6-69Task 2 Solutions 6-70Check Your Progress 6-72

Dynamic Reconfiguration 7-1

Objectives 7-1Relevance 7-2Additional Resources 7-2Introducing Dynamic Reconfiguration 7-3Benefits of Dynamic Reconfiguration 7-3Dynamic Reconfiguration Operational Locations 7-4Sun Fire 15K System Controller 7-4Sun Management Center™ 3.0 7-5Domain 7-5Dynamic Reconfiguration Concepts 7-6Preparing the Domain for Dynamic Reconfiguration

Operations 7-7Dynamic Reconfiguration Attach 7-8Attach States 7-8Dynamic Reconfiguration Detach 7-9Detach States 7-9I/O Board Considerations 7-10

Non-Multipath Configuration 7-11Detach-Safe and Suspend-Safe Devices 7-12Mirror Disk Configuration 7-13Non-Vital I/O 7-13Hot-Pluggable Hardware 7-14System Memory Considerations 7-15Detaching Permanent Memory 7-15Swap Space 7-16Performing Dynamic Reconfiguration

From the System Controller 7-17SMS Commands 7-17Theaddboard Command 7-18Thedeleteboard Command 7-19Themoveboard Command 7-21Thercfgadm Utility 7-22Viewing Dynamic Reconfiguration Status 7-24Standard View 7-24Detailed View 7-27Dynamic Reconfiguration Procedures 7-29Replacing a System Board 7-29Replacing an I/O Board 7-32Moving Physical Resources Between Domains 7-34Exercise: Performing Dynamic Reconfiguration 7-36Preparation 7-36Task 1: Replacing a System Board 7-36Task 2: Replacing an I/O Board 7-39Task 3: Moving Physical Resources

Between Domains 7-41Task 3a: Moving Physical Resources

Between Domains 7-42Exercise Summary 7-43Exercise: Performing Dynamic Reconfiguration 7-44Task 1 Solutions 7-44Task 2 Solutions 7-46Check Your Progress 7-49

Sun Management Center™ 3.0 on the Sun Fire™ 15K Server 8-1

Objectives 8-1Relevance 8-2Additional Resources 8-2Introducing Sun Management Center 8-3Sun MC Capabilities 8-3How Sun MC Works on Sun Fire 15K Servers 8-4Sun MC, Sun Fire 15K Server Agents 8-5Installing Sun MC Software on a Sun Fire 15K Server 8-6

Required Software Versions 8-7Network Port Configuration 8-8Groups and Security 8-9Sun MC Groups 8-10Sun Fire Administrative Groups 8-11Sun Fire 15K Topology Composites 8-12Discovering the Sun Fire 15K Server 8-15Detailed View 8-16Managing the Platform From Sun MC 8-17Config Reader Module 8-18Sun Fire 15K Platform Details Window 8-19Views From the Hardware Tab 8-25Physical View of the Sun Fire 15K Platform 8-28Logical View of the Sun Fire 15K Platform 8-30Logical View of a Sun Fire 15K Domain 8-32Platform/Domain State Management (PDSM) 8-34Selecting PDSM 8-34Running SMS Commands From PDSM 8-38Platform Administration 8-39Domain Administration 8-40Powering On and Powering Off a Board 8-42Dynamic Reconfiguration 8-43Exercise: Sun MC Review 8-46Preparation 8-46Tasks 8-46Exercise: Sun Fire 15K Monitoring Tasks 8-49Preparation 8-49Task 1: Monitoring Domains 8-50Task 2: Monitoring the Platform 8-52Task 3: Monitoring the System Controller 8-53Exercise: Sun Fire 15K Administrative Tasks 8-54Preparation 8-54Task 1: Performing Platform Administrative Tasks 8-54Exercise Summary 8-56Exercise: Sun MC Review 8-57Task Solutions 8-57Exercise: Sun Fire 15K Monitoring Tasks 8-61Task 1 Solutions 8-61Task 2 Solutions 8-63Task 3 Solutions 8-64Exercise: Sun Fire 15K Administrative Tasks 8-65Task 1 Solutions 8-65Check Your Progress 8-67

Troubleshooting Sun Fire™ 15K Servers 9-1

Relevance 9-2References 9-2Diagnostic Programs 9-3Power-On Self-Test 9-3SunVTS™ 9-3SunSolve OnlineSM 9-4Sun™ Explorer 9-4The POST Process 9-5Domain Configuration 9-5Probing 9-6Built-In Self-Test 9-7The CPU/Memory LPOST 9-8I/O LPOST 9-9Sun Fire 15K LPOST 9-9POST Logs 9-10Thehpost Test 9-11File Locations of the.postrc File 9-11Theblacklist andredlist Files 9-12Theblacklist File 9-12Automatic System Reconfiguration (ASR) 9-15Identifying an ASR Event 9-16Isolating Centerplane Problems 9-17Thesetbus Command 9-17Centerplane Fault Isolation Example 9-18System Controller Failover 9-21How System Controller Failover Works 9-22The Main SC Role During Startup 9-22The Spare SC Role During Startup 9-22File Propagation 9-23Failover Triggering Faults 9-23Failover Disabling Faults 9-24Detailed Failover Scenarios 9-25Fault on Main SC Detected by Main SC 9-25Fault On Main SC Detected By Spare SC 9-26I2 Network Fault 9-27Fault On Main SC and I2 Network Also Down 9-27System Controller Failover Commands 9-28Theshowfailover Command 9-28Theshowcmdsync Command 9-30Thesetfailover Command 9-31Forcing System Controller Failover 9-32System Failures Requiring Specific Handling 9-35Reboot Request 9-36System Panic 9-37Considerations 9-37Watchdog, Redmode, and XIR Resets 9-38

Forcing a Domain Reset 9-39Thereset Command 9-39Thesetkeyswitch Command 9-41Recovering From a Hung Domain 9-42Cannot Log In To the Domain 9-42Hard-Hung Domain 9-42Hung System Controller Console 9-43Heartbeat Failure (Hung Host) 9-44Manual Intervention With a Hung Host 9-44Domainstops and Recordstops 9-45Creating a Hardware State Dump File 9-46Theredx Utility 9-47Modes of Operation 9-47Configuring theredx Utility 9-48Theredx Utility Run Control files (.redxrc) 9-48Configuring the Offlineredx Standalone Script 9-49Verifying the redx Utility Configuration in Offline ModeFrom the System Controller 9-49Running theredx Utility in Offline Modes 9-50Running theredx Utility From a Workstation 9-50Running theredx Utility From a System Controller 9-51Loading Domain Stop and Record Stop Logs 9-53Loading the Dumpfile 9-53Viewing a Loaded Dumpfile 9-55Thewfail operation 9-55Sun™ Explorer Data Collector 9-58Installing and Running Explorer 9-58Viewing an Explorer Capture 9-59Sun Fire 15K Unique Files 9-60Technical Information for Escalation 9-61General Information Needed 9-61Problem-Specific Information Needed 9-61Exercise: Troubleshooting the Sun Fire 15K Server 9-63Preparation 9-63Task 1: Running POST Diagnostics 9-63Task 2: Blacklisting Management 9-64Task 3: Assess a Domain Configuration 9-65Task 4: Assess adstop andrstop dump File

Using theredx Utility 9-67Exercise: Troubleshooting Practice 9-69Preparation 9-69Tasks 9-69Worksheet 1 9-70Worksheet 2 9-71Worksheet 3 9-72

Exercise: Troubleshooting the Sun Fire 15K Server 9-74Task 1 Solutions 9-74Task 2 Solutions 9-74Task 3 Solutions 9-76Task 4 Solutions 9-77Check Your Progress 9-79

Sun Fire™ 15K Server Group Privileges A-1

Objectives A-1Additional Resources A-2Sun Fire 15K Administrator Privileges A-3

JumpStart™ – Automatic Solaris™ OE Installation B-1

Objectives B-1Additional Resources B-2Introducing JumpStart™ B-3JumpStart Usage B-3JumpStart Components B-4Using theadd_install_client Script B-5Setting Up Boot Services B-7JumpStart Client Boot Sequence B-8Boot Operation Support Files B-10Adding a Bootable Image B-14Adding Install Clients B-15Setting Up Client Identification B-16Using thesysidcfg File to Identify a Client B-16Setting Up an Installation Server B-19Theadd_to_install_server Script B-20Themodify_install_server Script B-20Setting Up the Configuration Server B-21Setting Up a Configuration Server Directory B-22Creating therules File B-23Creating theclass Files B-27Keywords and Arguments B-27Testing the Configuration With thepfinstall Command B-30Syntax B-30Examples of thepfinstall Command B-31Using theinstall Scripts B-34Running theadd_install_client Script B-34Adding a Client Using a Solaris CD-ROM Image on the LocalDisk B-35The/etc/bootparams File Content B-36The/etc/dfs/dfstab File Content B-36Initiating a JumpStart Installation B-38Booting Installation Clients B-38JumpStart Capabilities and Limitations B-39Worksheet for Configuring for JumpStart Installation B-42

Flash Install C-1

Objectives C-1Additional Resources C-2Introducing Flash Install C-3Creating the Master C-4Creating the Flash Archive C-4Deploying the Flash Archive to the Clones C-4Flash Install Requirements C-5Hardware Requirements C-5Software Requirements C-5Limitations C-7Flash Commands C-8Theflarcreate Command C-8Theflar Command C-8Using the JumpStart Method C-11JumpStart Keywords C-12Performing a Flash Installation C-13Solaris OE Installation C-14Creating the Flash Archive C-16Deploying the Flash Archive to the Clone C-17Flash Administration Tasks C-26Clone Post Installation Tasks C-26Customizing the Master C-27Creating a Custom Archive C-29Creating an Archive to Tape C-29Troubleshooting Tips C-31Logs C-33Common Errors C-33

Sun Fire 15K Sever Field-Replaceable Units D-1

Introduction D-2Access Doors and Side Panels D-3Top Cap Assembly D-4Kick Plate Assembly D-5System Board Assemblies and Carrier Plates D-6Fan Tray Cooling Components D-17Power Supplies and Air Filters D-18XOPTIONS D-20

Sun Enterprise™ 10000 Server-to-Sun Fire™ 15K

Server Dictionary E-1

Daemons E-2Commands E-3Configuration Files E-5Miscellaneous Names E-6

Interdomain Networking F-1

Objectives F-1References F-2Interdomain Networks F-3IDN Operation F-4SRD Failover F-6IDN Links F-7Building an IDN F-8Domain Requirements F-8Identifying the Domains F-8IDN User Roles F-9Labelling the Domains F-10Thelabeldomain Command F-10Creating the IDN F-11Linking the Client Domains F-12Specified SRD Environment F-12Automatic SRD Assignment F-13Activating the Interfaces F-15Domain Preparation F-15Checking IDN Status F-16Thexcidn Driver F-17Thexcidn.conf File F-17Changing thexcidn.conf File F-18Theidnd Daemon F-19IDN Communities F-20Link Connection Rules With Communities F-20

A Simple Configuration F-21More Complex Examples F-22Changing the Community Set for a Domain F-24Changing the Community Set for an IDN Link F-24IDN Configuration Rules F-25Dynamic Reconfiguration and IDN F-25Managing an IDN F-26Adding an Alternate SRD F-26Removing an SRD F-27Changing the SRD Managing an IDN Link F-27Changing the State of an IDN F-28Deleting an IDN F-29Unlinking Client Domains F-30

IP Interface Support F-32IDN and thesnoop Command F-32IDN Performance Data F-32Configuring an Example IDN F-33Issue F-33Configuration F-33Solution F-33

Thexcidnhelp Command F-35Check Your Progress F-36

Glossary and Acronyms Glossary-1 Index Index - 1

About This Course

Course Goals

Upon completion of this course, you should be able to:

● Remove and install most Sun Fire™ 15K field replaceable units(FRUs)

● Describe the Sun Fire 15K server physical architecture

● Configure the Sun Fire 15K server platform and domains

● Perform platform administration activities on the Sun Fire 15Kserver

● Use available Solaris™ Operating Environment and platform specifictools to troubleshoot the Sun Fire 15K

Course Map

Course Map

The following course map enables you to see what you haveaccomplished and where you are going in reference to the course goals

Sun Fire Architecture

Sun Fire 15K Platform Configuration

Introducing theSun Fire 15K Server

Sun Fire 15K

Sun Fire 15K System

Controller Architecture, Configuring the

Sun fire 15K Platform

Configuring the

Multipathing I/O

Management

DynamicReconfiguration

Troubleshooting

Sun Fire 15K Server Administration and Troubleshooting

Sun ManagementCenter 3.0Sun Fire 15K Servers on

Sun Fire 15K Servers

Sun Fire 15K DomainsMaintenance

Configuration Installation andServer Architecture

and Configuration

Topics Not Covered

Topics Not Covered

This course does not cover the following topics Many of these topics arecovered in other courses offered by Sun Educational Services:

● Solaris administration – Covered in SA-118: Fundamentals of

Solaris™ 8 Operating Environment for System Administrators

● Storage administration – Covered in ES-255: Sun™ Hardware RAID

and T3 Storage System Administration

● Sun Fire 3800, 48x0, 4810 and 6800 server administration – Covered

in ES-420: Sun Fire™ Workgroup/Enterprise Server Administration

● This course does not cover advanced architecture theory

Refer to the education.central/ITT web site for specific information andregistration

How to Use Course Materials

How to Use Course Materials

To enable you to succeed in this course, these course materials use alearning module that is composed of the following components:

● Goals – You should be able to accomplish the goals after finishing

this course and meeting all of its objectives

● Objectives – You should be able to accomplish the objectives after

completing a portion of instructional content Objectives supportgoals and can support other higher-level objectives

● Lecture – The instructor will present information specific to the

objective of the module This information will help you learn theknowledge and skills necessary to succeed with the activities

● Activities – The activities take on various forms, such as an exercise,

self-check, discussion, and demonstration Activities are used tofacilitate mastery of an objective

● Visual aids – The instructor might use several visual aids to convey

a concept, such as a process, in a visual form Visual aids commonlycontain graphics, animation, and video

Additional resources – Indicates other references that provide additional

information on the topics described in the module

1

2

3

Demonstration – Indicates a demonstration of the current topic is

recommended at this time

Note – Indicates additional information that can help students but is not

crucial to their understanding of the concept being described Studentsshould be able to understand the concept or complete the task withoutthis information Examples of notational information include keywordshortcuts and minor system adjustments

Caution – Indicates that there is a risk of personal injury from a

nonelectrical hazard, or risk of irreversible damage to data, software, orthe operating system A caution indicates that the possibility of a hazard(as opposed to certainty) might happen, depending on the action of theuser

Caution – Indicates that either personal injury or irreversible damage of

data, software, or the operating system will occur if the user performs thisaction A warning does not indicate potential events; if the action isperformed, catastrophic events will occur

Caution – You display the heat caution icon in the same manner as the

note and electrical caution icon Use this icon where the risk of injury due

to heat or hot surfaces will result

Typographical Conventions

Courieris used for the names of commands, files, directories,programming code, and on-screen computer output; for example:

Use ls -alto list all files

system% You have mail

Courieris also used to indicate programming constructs, such as classnames, methods, and keywords; for example:

The getServletInfomethod is used to get author information.The java.awt.Dialogclass contains Dialogconstructor

Courier boldis used for characters and numbers that you type; forexample:

To list the files in this directory, type:

To delete a file, use thermfilenamecommand

Courier italic boldis used to represent variables whose values are to

be entered by the student as part of an activity; for example:

Type chmod a+rwx filenameto grant read, write, and executerights for filename to world, group, and users

Palatino italic is used for book titles, new words or terms, or words that

you want to emphasize; for example:

Read Chapter 6 in the User’s Guide.

These are called class options.

Module 1

Introducing the Sun Fire™ 15K Server

Objectives

Upon completion of this module, you should be able to:

● List the functional goals of the Sun Fire™ 15K server

● Compare the Sun Fire 15K to other Sun Fire™ servers and theirpredecessors

● Describe the key features of the Sun Fire 15K server

● Describe some of the advanced reliability, availability, andserviceability (RAS) features of the Sun Fire 15K server

Relevance

Present the following questions to stimulate the students and get them thinking about the issues and topics presented in this module While they are not expected to know the answers to these questions, the answers should be of interest to them and inspire them to learn the material presented in this module.

?

!

Discussion – The information found in this module is relevant to your job

because it introduces you to the Sun Fire 15K server The followingquestions are relevant to understanding what this module is all about:

● Which Sun Fire server models are available?

● What are the key features of the Sun Fire 15K server?

● How does the Sun Fire 15K server differ from other members of theSun Fire family?

Additional Resources

Additional resources – The following references provide additional

details on the topics discussed in this module:

Part Number 805-7373

Introducing the Sun Fire™ 15K Server

Introducing the Sun Fire™ 15K Server

The Sun Fire 15K server (Figure 1-1) succeeds the Sun Enterprise™ 10000high-end server and has the largest platform in the Sun Fire server family.The Sun Fire server family succeeds the Sun Enterprise™ 3000–10000group of servers

Figure 1-1 Sun Fire 15K Server With I/O Expansion Rack

The Sun Fire 15K server uses the UltraSPARC™ III (or later) centralprocessing units (CPUs) for compute power, industry standard hot-swappable peripheral component interconnect (PCI) cards forInput/Output (I/O), and Sun Fire™ link boards for high-bandwidthclustering It runs the Solaris™ 8 (or later) Operating Environment (OE)and is binary-compatible with earlier versions of Sun server families

Generation Scaling

Generation Scaling

System performance, bandwidth, and memory capacity are scaled up bythree to four times over the previous generation The system interconnecthas headroom for faster processors The Sun Enterprise™ 10000 dynamicsystem domain and RAS capabilities are included throughout the Sun Firefamily The largest family member, the Sun Fire 15K server, has greaterdomains and RAS capabilities Figure 1-2 illustrates the server

generations

Figure 1-2 Sun Enterprise Server Generations

UltraSPARC™ I/II processor UPA interconnect

UltraSPARC™ III processor Sun Fireplane interconnect

Sun Enterprise 10000 server The axis tick marks show 25 percent, 50percent, and 75 percent of the Sun Fire 15K server value Memory capacityand bandwidth in the Sun Fire 15K server are scaled by more than theinitial processor performance improvements, leaving headroom for futurecentral processing unit (CPU) clock improvements.

Figure 1-3 Scaling Between the Sun Enterprise 10000 and the

Sun Fire 15K Servers

Note – The maximum number of CPUs supported by the Sun Fire 15K

server can be increased from 72 to 106 by replacing I/O boards withMaxCPU boards Each MaxCPU board supports two additional CPUs

(Gflops)

Processors

PCI buses

System bisection bandwidth (GB/s)

Memory capacity (GB)

E10000

400-MHz processors 128-MB DIMMs (the largest possible)

0% 25% 50% 75% 100%

✖

Axis tick marks:

The Sun Fire Systems

The Sun Fire Systems

The Sun Fire family covers a range of 8 to 106 (with MaxCPU boards)processors The five Sun Fire family system boxes share many hardware

components, including the UltraSPARC™ processors, the dual in-line

memory modules (DIMMs), and the CPU/memory boards They useindustry standard peripheral component interface (PCI) cards for massstorage and networking interfaces

System Enclosures

The Sun Fire family enclosures are shown in Figure 1-4 They are sized toaccommodate from 2 to 18 Slot 0 boards, and from 2 to 18 Slot 1 boards.The Slot 0 boards (CPU/memory boards) and the Slot 1 boards (hot-swapPCI assembly [hsPCI], MaxCPU, and Sun Fire boards) fit in separate types

of slots, so a system can be simultaneously configured with the maximumnumber of both types of boards

Figure 1-4 Sun Fire Family System Enclosures

PCI I/O board

CPU/Mem boards (9 more

in back)

CPU/Mem boards

(4 I/O in back)

I/O CPU/Mem

boards

CPU/Mem boards

I/O

P P M M

P P M M

I I

The Sun Fire Systems

Slot 0 Boards

Slot 0 is populated by the system (CPU/memory) boards Each systemboard holds four processors Each processor has an associated memorysubsystem of eight DIMMs, so memory bandwidth and capacity are bothscaled up as processors are added The memory capacity of the board is

32 Gbytes using 1 Gbyte DIMMs The maximum memory bandwidth on aboard is 9.6 Gbyte/sec The CPU/memory board has a 4.8 Gbyte/sec dataconnection to the rest of the system

Slot 1 Boards

The Sun Fire hot-swap PCI (hsPCI) assembly has two I/O controllers.Each controller provides one 66 Mhz PCI bus, and one 33 Mhz PCI bus,for a total of two of each speed on the hsPCI board A Sun Fire hsPCIboard has a 2.4 Gbyte/sec connection to the rest of the system

There are four types of Sun Fire PCI assemblies, all of which share acommon architecture The Sun Fire 15K server I/O board has four hsPCIslots

Note – The Sun Fire™ 48x0 through Sun Fire™ 6800 servers use two types

of I/O assemblies; one provides eight PCI slots and another provides fourhot-pluggable compact PCI (cPCI) slots The Sun Fire™ 3800 server’s I/Oassembly has six hot-pluggable cPCI slots

Two other Slot 1 boards are provided The MaxCPU board adds two moreCPUs (but no memory), and the Sun Fire™ link board adds three FiberOptic inter-system links and two hsPCI slots

System Controller Boards

The system controller board is the heart of the Sun Fire family availabilityand serviceability technology It configures the system, coordinates theboot process, sets up the dynamic system domains, monitors the systemenvironmental sensors, and handles error detection, diagnosis, and

recovery You can configure two system controller boards into the system

to provide redundancy and automatic failover in the event that one of thesystem controller boards fails The Sun Fire 3800 through Sun Fire 6800

Memory capacity

(with 1-Gbyte DIMMs)

64 Gbytes 96 Gbytes 192 Gbytes 576 Gbytes

Centerplane Active Passive Active

PCI assembly types Hot-swap

cPCI

PCI and hot-swapcPCI

Hot-swapPCIPCI slots per assembly 6 8 per PCI, 4 per cPCI 4

Maximum total PCI slots 12 16 32 72

Power requirements 100–120 or 220–240 VAC 220–240 VAC

System controller boards 2

Redundant cooling Yes

Enclosure

Rack-mount

Desk-sideor

rack-mount

mount

Rack-Sun Fire6800cabinet

Sun Fire15000cabinetRoom in enclosure

for peripherals?

* Additional processors are provided through the MaxCPU board The MaxCPU boardinstalls in place of an I/O board and provides two additional CPUs (but no additionalmemory) The total of 106 processors assumes the installation of 17 MaxCPU boards

Cabinets

The system can consist of two or more air-cooled cabinets; a systemcabinet and one or more customer selected I/O expansion racks Thesystem cabinet (Figure 1-5 and Figure 1-6 on page 1-10) includes theprocessors, memory, and peripheral device controllers, such as PCI cards

Figure 1-5 System Cabinet – Front View

4 fan trays (7 fans each) Top cap and extension

Slot 0 (top) CPU boards

4 fan trays (7 fans each)

Circuit breakers 4K watt AC

to 48 VDC power supplies (3 per side)

75.5 in (1917.7 mm) high 33.3 in (848.0 mm) wide 56.25 in (1428.75 mm) deep

Slot 1(bottom) boards

Remote power control to I/O expansion

(hsPCI, link, MaxCPU)

Air inlet (DVD ROM, tape drive,

Figure 1-6 System Cabinet – Back View

4 fan trays (7 fans each) Top cap and extension

Slot 0 (top) CPU boards

4 fan trays (7 fans each)

Circuit breakers 4K watt AC

to 48 VDC power supplies (3 per side)

75.5 in (1917.7 mm) high 33.3 in (848.0 mm) wide 56.25 in (1428.75 mm) deep

Slot 1(bottom) boards

Remote power control to I/O expansion

(hsPCI, link, MaxCPU)

Air inlet (DVD ROM, tape drive,

Sun Fire 15K I/O Support

Sun Fire 15K I/O Support

The Sun Fire 15K server supports standard PCI cards Four PCI cards aresupported by each of the 18 possible Sun Fire 15K PCI assemblies ThePCI boards are installed in special hsPCI cassettes that enable the removaland replacement of the cards while the Solaris OE is running, through theDynamic Reconfiguration interface

Each hsPCI cassette can be powered on and off and individually removed

An entire Slot 1 board can be removed and replaced though DynamicReconfiguration Any Slot 1 board can be configured into any domain

Note – The Sun Fire 15K server does not support cPCI.

Interdomain Networking

Interdomain Networking (IDN), a method of simulatingTransmission Control Protocol/Internet Protocol (TCP/IP) networkinterfaces that transfer data between domains using the Sun Fire 15Kcenterplane, is supported on the Sun Fire 15K server IDN is more reliable

in the Sun Fire 15K server than in the Enterprise 10000 server Forexample, the domain is completely isolated, so a hardware failure in onedomain does not take down the other domains in the network

IDN is implemented using a Shared Resource Domain (SRD), a domainexclusively dedicated to managing the IDN traffic and data transfer Youcan configure a second SRD, providing transparent IDN failover

capability if the active SRD fails

IDN is available with Solaris™ 8 OE update 7 and higher

Sun Fire 15K I/O Support

Sun Fire 15K Server Peripherals

The Sun Fire 15000 cabinet does not have room for peripherals In allcases, more peripheral devices can be configured in additional peripheralexpansion racks The following I/O controllers are supported:

● Sun Quad FastEthernet™ PCI adapter

● Sun™ Gigabit Ethernet PCI adapter 2.0 (GEM)

● Sun™ GigaSwift Ethernet UTP PCI adapter (Cassini)

● Dual Channel Differential UltraSCSI PCI Host adapter (UD2S)

● SunATM™ PCI Adapter 4.0 (155 Mbps with multimode fiberinterface)

● SunATM™ PCI Adapter 4.0 (155 Mbps with CAT 5UTP interface)

● SunATM ™PCI Adapter 4.0 (622 Mbps with multimode fiberinterface)

● Dual Fibre Channel PCI Network adapter (Crystal+)

● Single Fibre Channel PCI Network adapter (Amber)

● Single-ended SCSI and Fast Ethernet Combination adapter (FreshChoice)

● Sun™ Dual Fast Ethernet and Dual Ultra-SCSI PCI adapter(Cauldron)

The following external disk arrays are supported:

● Sun StorEdge™ 99XX

● Sun StorEdge™ T3 (bootable)

● Sun StorEdge™ A5200

● Sun StorEdge™ S1 (bootable)

● Sun StorEdge™ D1000The following external backup storage devices are supported:

● Sun StorEdge™ L9

● Sun StorEdge™ L180

● StorEdge™ L700