Student Guide - IES-443 Advanced Sun Fire™ Mid-Range Troubleshooting

1-17Figure 1-11 Sun Fireplane Switch Board Slot Assignmentsfor the Sun Fire 6800 Server .... Enterprise Services, Revision A Figure 5-25 System Data Order and Bit Slicing in Double Pump

Trang 1

Sun Microsystems, Inc.

UBRM05-104

500 Eldorado Blvd Broomfield, CO 80021

U.S.A Revision A

Advanced Sun Fire™ Mid-Range

Troubleshooting

IES-443

Trang 2

This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation No part of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any.

Third-party software, including font technology, is copyrighted and licensed from Sun suppliers.

Sun, Sun Microsystems, the Sun Logo, Sun Enterprise, Sun StorEdge, Sun Fire, Netra, Sun Enterprise and Solaris are trademarks or registered trademarks of Sun Microsystems, Inc in the U.S and other countries.

UNIX is a registered trademark in the U.S and other countries, exclusively licensed through X/Open Company, Ltd.

U.S Government approval might be required when exporting the product.

RESTRICTED RIGHTS: Use, duplication, or disclosure by the U.S Government is subject to restrictions of FAR 52.227-14(g)(2)(6/87) and FAR 52.227-19(6/87), or DFAR 252.227-7015 (b)(6/95) and DFAR 227.7202-3(a).

DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS, AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

THIS MANUAL IS DESIGNED TO SUPPORT AN INSTRUCTOR-LED TRAINING (ILT) COURSE AND IS INTENDED TO BE USED FOR REFERENCE PURPOSES IN CONJUNCTION WITH THE ILT COURSE THE MANUAL IS NOT A STANDALONE TRAINING TOOL USE OF THE MANUAL FOR SELF-STUDY WITHOUT CLASS ATTENDANCE IS NOT RECOMMENDED.

ECCN Date – August 2002

Trang 3

Recycle

Ce produit ou document est protégé par un copyright et distribué avec des licences qui en restreignent l’utilisation, la copie, la distribution,

et la décompilation Aucune partie de ce produit ou document ne peut être reproduite sous aucune forme, par quelque moyen que ce soit, sans l’autorisation préalable et écrite de Sun et de ses bailleurs de licence, s’il y en a.

Le logiciel détenu par des tiers, et qui comprend la technologie relative aux polices de caractères, est protégé par un copyright et licencié par des fournisseurs de Sun.

Sun, Sun Microsystems, the Sun Logo, Sun Enterprise, Sun StorEdge, Sun Fire, Netra, Sun Enterprise et Solaris sont des marques de fabrique

ou des marques déposées de Sun Microsystems, Inc aux Etats-Unis et dans d’autres pays.

UNIX est une marques déposée aux Etats-Unis et dans d’autres pays et licenciée exclusivement par X/Open Company, Ltd.

L’interfaces d’utilisation graphique OPEN LOOK et Sun™ a été développée par Sun Microsystems, Inc pour ses utilisateurs et licenciés Sun reconnaît les efforts de pionniers de Xerox pour larecherche et le développement du concept des interfaces d’utilisation visuelle ou graphique pour l’industrie de l’informatique Sun détient une licence non exclusive de Xerox sur l’interface d’utilisation graphique Xerox, cette licence couvrant également les licenciés de Sun qui mettent en place l’interface d’utilisation graphique OPEN LOOK et qui en outre

se conforment aux licences écrites de Sun.

L’accord du gouvernement américain est requis avant l’exportation du produit.

LA DOCUMENTATION EST FOURNIE “EN L’ETAT” ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES, DANS LA MESURE AUTORISEE PAR LA LOI APPLICABLE, Y COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A L’APTITUDE A UNE UTILISATION PARTICULIERE OU A L’ABSENCE DE CONTREFAÇON.

CE MANUEL DE RÉFÉRENCE DOIT ÊTRE UTILISÉ DANS LE CADRE D’UN COURS DE FORMATION DIRIGÉ PAR UN INSTRUCTEUR (ILT) IL NE S’AGIT PAS D’UN OUTIL DE FORMATION INDÉPENDANT NOUS VOUS DÉCONSEILLONS DE L’UTILISER DANS LE CADRE D’UNE AUTO-FORMATION.

Trang 5

Table of Contents

About This Course Preface-xxiii

Course Goals Preface-xxiiiCourse Map Preface-xxivTopics Not Covered Preface-xxvHow Prepared Are You? Preface-xxvIntroduction Preface-xxviHow to Use Course Materials Preface-xxviConventions Preface-xxviIcons Preface-xxviTypographical Conventions Preface-xxviii

Reviewing the Sun Fire Servers 1-1

Objectives 1-1Relevance 1-2Additional Resources 1-2Sun Fire Server Models 1-3Server Naming 1-3Sun Fire Features 1-5Interconnect Capabilities 1-7Peak System Bandwidth 1-8System Board Physical Locations 1-9Sun Fire 6800 Server 1-9Sun Fire 4810 and 4800 Servers 1-10Sun Fire 3800 Server 1-10Sun Fire I/O Assemblies 1-11I/O Assembly Locations 1-12Sun Fire 6800 Server 1-12Sun Fire 48x0 Server 1-13Sun Fire 3800 1-14Compact PCI I/O 1-15Sun Fire 4800/4810/6800 Four-Slot cPCI Board 1-15Sun Fire 3800 Six-Slot cPCI Board 1-17Sun Fireplane Switch Boards 1-18Sun Fire 6800 Server 1-18

Trang 6

Sun Fire 4800, 4810, and 3800 Servers 1-18Sun Fireplane Switch Board Physical Locations 1-19Agent IDs 1-21CPU Locations and Agent IDs 1-21Memory Controller Mapping 1-22IOC AID 1-23Locating I/O Devices in Sun Fire 6800/4810/

4800 Systems 1-24Four Slot cPCI I/O Assembly 1-27Sun Fire 3800 I/O Device Location Mapping 1-29Six Slot cPCI I/O Assembly 1-30Sun Fireplane Interconnect 1-31How the Sun Fireplane Interconnect Works 1-32Troubleshooting Tools 1-33

Service Mode 1-33System Logging 1-35Explorer 1-37Parity 1-38Parity Checking 1-39Problems with Parity 1-39ECC 1-40Check Your Progress 1-41

Power Management and the Frame Manager 2-1

Objectives 2-1Relevance 2-2Additional Resources 2-2Power 2-3The RTU and RTS 2-4Redundant Power 2-6The AC/DC Power Supplies 2-8Housekeeping Voltage 2-9Power Distribution 2-10Sun Fire 3800 2-11Sun Fire 48x0 2-12Sun Fire 6800 2-14DC/DC Component Power Supplies 2-16Board Voltage Requirements 2-17The Frame Manager 2-18Exercise: Managing Power 2-20Objective 2-20

Trang 7

Check Your Progress 2-36

Domains and Segments 3-1

Objectives 3-1Relevance 3-2Additional Resources 3-2Virtual Servers 3-3Domains 3-4Segments 3-6Domains and Segments in the Sun Fire 6800 3-8Impact of Multiple Segments and Domains 3-11Power Grid Segmentation 3-12Power Grids on Sun Fire 48x0/3800 3-12Power Grids on Sun Fire 6800 3-13Domain Recovery 3-15Sun Fire 3800, 48x0, and 6800 Board

Configurations 3-15Segment and Domain Summary 3-19Exercise: Identifying Domains and Segments 3-20Objective 3-20Preparation 3-20Task 1 3-20Task 2 3-22Exercise Summary 3-31Check Your Progress 3-32

The System and I/O Boards 4-1

Objectives 4-1Relevance 4-2Additional Resources 4-2System Board 4-3System Board Interconnects 4-4CPU Placement Rules 4-6Sun Fire I/O 4-7I/O Assembly Configurations 4-7The PCI I/O Board 4-9PCI Card Support 4-10PCI Board Slot Configuration 4-10The cPCI Board Assembly 4-12cPCI Card Slot Configuration 4-14cPCI Operation 4-14The 3800 cPCI Board 4-15

3800 cPCI Card Slot Configuration 4-17I/O Board Power 4-18PCI Card Power 4-18Fireplane Bus Interface 4-19Failure on a PCI Board 4-20

Trang 8

Exercise: Explaining System and I/O Board Operation 4-21Objective 4-21Preparation 4-21Task 1 4-21Task 2 4-23Task 3 4-25Exercise Summary 4-28Check Your Progress 4-29

The Sun Fireplane Interconnect Bus 5-1

Objectives 5-1Relevance 5-2Additional Resources 5-2Fireplane Bus Introduction 5-3Fireplane Bus Design 5-4The Fireplane Switch Overview 5-5Address Interconnect Overview 5-8Data Interconnect Overview 5-9L1 Data Flow 5-11System Board Port Configuration 5-11I/O Board Port Configuration 5-12The ASIC Tree 5-13Circuitry on L1 System Boards 5-16Safari Ports 5-17Console Bus 5-17Fireplane Bus Flow Control Circuitry 5-18The Fireplane Buses in the Sun Fire 3800, 4800, and

4810 5-19Address Interconnect 5-19SDC Management 5-23Data Interconnect 5-27The Fireplane Buses in the Sun Fire 6800 5-32Address Interconnect 5-32SDC Management Interconnect 5-34Data Interconnect 5-36Fireplane Bus Data Paths 5-38Quadword Data Line Structure 5-39Data Pathing 5-40System Board to L2 Data Path 5-40I/O Board to L2 Data Path 5-41Sun Fire 3800, 4800, and 4810 Data Path 5-42

Trang 9

Objective 5-51Preparation 5-51Task 1 5-51Task 2 5-53Exercise Summary 5-58Check Your Progress 5-59

Parity and ECC Detection and Recovery 6-1

Objectives 6-1Relevance 6-2Additional Resources 6-2Error Detection 6-3Parity 6-4Parity Detection in the Data Path 6-5Parity Detection in the Address Path 6-8Parity Detection in the Control Path 6-9Error Correction Code (ECC) 6-11ECC Error Types 6-11ECC Creation 6-13End-to-End ECC Protection 6-14ECC Syndromes 6-16The ECC Syndrome Table 6-16Signaling ECC Syndrome Codes 6-18ECC Error Identification 6-20Locating a DX ECC Error 6-20The DX ECC Status Register 6-22CPU-Caused Interconnect ECC Error Example 6-24ECC Errors from Memory 6-26Data Bit Identification 6-28ECC Error Reporting 6-32Asynchronous Fault Status Register 6-33The AFSR Table 6-33AFT Labels 6-36Asynchronous Fault Address Register 6-37Physical Address Space 6-38AFAR Addressing 6-39AFSR Overwrite Policy 6-43AFSR/AFAR Overwrite Policy 6-43ECC and MTAG Syndrome Fields Overwrite

Policy 6-44Exercise: Identifying and Diagnosing Parity

and ECC Errors 6-45Preparation 6-45Task 1 6-45Task 2 6-47Task 3 6-49

Trang 10

Exercise Summary 6-55Check Your Progress 6-56

Caching and Interconnect Operations 7-1

Objectives 7-1Relevance 7-2Additional Resources 7-2Introduction 7-3The UltraSPARC III Caches 7-4The Instruction Cache 7-4The Data Cache 7-4The External Cache 7-4The Write Cache 7-5The Prefetch Cache 7-6Cache Snooping 7-7Snoopy Coherency 7-8Fireplane Address Bus Snooping Operation 7-10Snoop Response Signals 7-10Cache Data State Tags 7-13CTags 7-13MOESI State Transitions 7-14The DTags 7-15Fireplane Bus Transactions 7-16Interconnect Signal Groups 7-16Address Interconnect Operation 7-18Address Transaction (ATrans) 7-18Sun Fire Transaction Request Codes 7-19Requests for Data 7-20The Data Interconnect 7-22Data Transaction (DTrans) 7-22Request Flow 7-23Address Read /Write Transaction 7-23Data Transaction 7-25Read-to-Share Cache Example 7-29Arbitration on AR and SDC ASICS 7-31Address Interconnect Arbitration 7-31Data Bus Arbitration and Operation 7-33Direction of ECC Error Reporting 7-35The SDC ECC Error Register 7-39General Notes on L1 DX ECC Errors & SC

Messages 7-41

Trang 11

Exercise Summary 7-51Check Your Progress 7-52

Memory Interleaving 8-1

Objectives 8-1Relevance 8-2Additional Resources 8-2Memory 8-3SIMMs 8-3DIMMs 8-3Logical and Physical Memory Banks 8-4Interleaving 8-11Memory Interleave Configuration 8-12Interleave Scope 8-14Interleave Rules 8-14Interleave Mode 8-15Configuring Interleave 8-16Checking the Interleave Configuration 8-16Exercise: Explaining and Using Memory 8-20Objective 8-20Preparation 8-20Task 8-21Exercise Summary 8-34Check Your Progress 8-35

Hardware Control Buses 9-1

Objectives 9-1Relevance 9-2Additional Resources 9-2Sun Fire Hardware Control Buses 9-3The Console Bus 9-4The Console Bus Hub 9-5The BootBus Controller 9-7System/Board Reset/Error Status and Control 9-8SBBC Control Paths 9-9System Controller SBBC Control Paths 9-9System Board SBBC Control Paths 9-11I/O Board SBBC Control Paths 9-12The SBBC Error Register 9-14The PROM Bus 9-17JTAG 9-19The I2C Bus 9-20The Global I2C Bus 9-21Local I2C Buses 9-28The ID Board and the I2C Bus 9-31Error Signaling 9-34ASIC Error Reporting Policy 9-34

Trang 12

The EChip 9-35System Board Error Path 9-36I/O Board Error Path 9-37Fireplane Switch Board Error Path 9-38

SC Board Error Path 9-39Reading an Error Report 9-41Error Pause 9-42Clocking in the Sun Fire Servers 9-44Sun Fire System Clocking Requirements 9-45Sun Fire System Clocking Users 9-47Local Clock Distribution 9-48Initial Clock Source Selection 9-49Clock Failover 9-50

SC Hot Swap Clock Management 9-50Exercise: Interfacing Hardware Control 9-51Objective 9-51Preparation 9-51Task 1 9-51Task 2 9-53Exercise Summary 9-58Check Your Progress 9-59

Workshop 10-1

Objectives 10-1Relevance 10-2Additional Resources 10-2Classes of Errors 10-3Analyzing an Error Report 10-7Analysis 10-8Helpful Hints 10-11Exercise: Troubleshooting the Sun Fire Mid-Range

Servers 10-13Preparation 10-13Task 10-13Exercise Summary 10-26

Memory Architecture A-1

SRAM and DRAM A-2Memory Chip Architecture A-4Refresh A-5Static RAM A-6

Trang 13

Tag Addressing A-11Building the ECC Syndrome Table A-12ECC Calculation Example A-12

The UltraSPARC III CPU B-1

Introduction B-2The UltraSPARC Processor B-3Processor Architecture B-4Superscalar Execution B-5Pipelining B-6Pipeline Stages B-8The Instruction-Fetch Stages B-8Instruction Issue B-9Execution B-9Trap and Done B-10Summary B-10Processor Subunits B-11UltraSPARC III Functional Overview B-12Instruction Issue Unit (IIU) B-13Integer Execution Unit (IEU) B-15Floating Point and Graphics Unit (FGU) B-17Instruction Latency B-17Data Cache Unit (DCU) B-18The Prefetch Cache B-19L1 Data Cache SAM Addressing B-20External Cache Unit (ECU) B-21Memory Control Unit (MCU) B-22DIMM Sizes B-23System Interface Unit (SIU) B-24CPU Error Detection and Correction B-25

Caching C-1

Cache Characteristics C-2Cache Terminology C-3Virtual Address Cache C-3Physical Address Cache C-4Harvard Caches C-5Cache Hit Rate C-6Example C-6Effects of CPU Cache Misses C-7Cache Thrashing C-8Measuring CPU and Caching Statistics C-10The cpustat Utility C-10

The cputrack Utility C-11

Glossary/Acronyms Glossary-1Index Index-1

Trang 15

Assembly Locations 1-12Figure 1-7 Sun Fire 4800 Server I/O Assembly Locations 1-13Figure 1-8 Sun Fire 3800 cPCI I/O Assembly Locations 1-14Figure 1-9 Four-Slot cPCI I/O Board Logical Block Diagram 1-15Figure 1-10 Six-Slot cPCI I/O Board Logical Block Diagram 1-17Figure 1-11 Sun Fireplane Switch Board Slot Assignments

for the Sun Fire 6800 Server 1-19Figure 1-12 Sun Fireplane Switch Board Slot Assignments

for the Sun Fire 48x0 Server 1-20Figure 1-13 CPU Mapping Example 1-22Figure 1-14 Memory Controller Mapping Example 1-22Figure 1-15 Example I/O Device Path for Sun Fire

6800/4810/4800 Systems 1-24Figure 1-16 I/O Assembly Physical Slot Designations 1-26Figure 1-17 Sun Fire 6800/48x0 Servers Four-Slot

cPCI Physical Slot Designations 1-28Figure 1-18 Example I/O Device Path for

Sun Fire 3800 Systems 1-29Figure 1-19 Sun Fire 3800 System Six-Slot

cPCI Physical Slot Designations 1-30Figure 1-20 Sun Fireplane Interconnect Operational View 1-31Figure 2-1 RTU and RTS Power Connections 2-5Figure 2-2 RTS and RTU Units 2-6Figure 2-3 Sun Fire 3800 Logical Power Distribution 2-11Figure 2-4 Sun Fire 3800 Power Distribution 2-12Figure 2-5 Sun Fire 48x0 Logical Power Distribution 2-12Figure 2-6 Sun Fire 48x0 Power Distribution 2-13

Trang 16

Figure 2-7 Sun Fire 6800 Logical Power Distribution 2-14Figure 2-8 Sun Fire 6800 Power Distribution 2-15Figure 2-9 Frame Manager Cable Diagram 2-19Figure 3-1 Bus Clocking with Two Domains 3-5Figure 3-2 Segments in a Sun Fire 3800, 4800, or

4810 Server 3-7Figure 3-3 Two Domains in a Sun Fire 6800 3-8Figure 3-4 Two Segments in a Sun Fire 6800 3-9Figure 3-5 Domains and Segments in a Sun Fire 6800 3-10Figure 4-1 System Board Major Components 4-4Figure 4-2 System Board Logical Components 4-5Figure 4-3 PCI I/O Board Assembly 4-9Figure 4-4 cPCI I/O Board 4-13Figure 4-5 cPCI Board and TI HPC-3130 HPC-PCI Chip 4-15Figure 4-6 3800 cPCI I/O Board 4-16Figure 4-7 PCI Bus-Fireplane Bus Relationship 4-19Figure 4-8 I/O Board SBBC Interfaces 4-20Figure 5-1 Bus Hierarchy Levels 5-4Figure 5-2 Fireplane Switches in the Sun Fire Mid-Range

Platforms 5-6Figure 5-3 Fireplane Switch Board Layout 5-7Figure 5-4 Address Interconnect Levels 5-8Figure 5-5 Data Interconnect Levels 5-10Figure 5-6 System Board Data Flow 5-11Figure 5-7 I/O Board Data Flow 5-12Figure 5-8 ASIC Tree 5-13Figure 5-9 System Board ASIC Tree 5-14Figure 5-10 System Board ASIC Tree (Continued) 5-15Figure 5-11 Major System Board Interconnect Pathways 5-16Figure 5-12 Address Repeater 5-20Figure 5-13 Sun Fire 3800, 4800, and 4810 AR Level 1 and 2

Configurations 5-21Figure 5-14 SDC Interconnect 5-24Figure 5-15 Dual CPU Data Switch ASIC 5-27Figure 5-16 DX Level 2 Configuration for the Sun Fire 3800, 4800,and 4810 5-29Figure 5-17 AR Level 2 Configuration for the Sun Fire 6800 5-32Figure 5-18 SDC Level 2 Configuration for the Sun Fire 6800 5-34Figure 5-19 Sun Fire 6800 Data Interconnect 5-36Figure 5-20 Fireplane Data Path Bandwidth 5-38Figure 5-21 Data Line Quadword Structure 5-39

Trang 17

Figure 5-25 System Data Order and Bit Slicing in

Double Pump Mode Fireplane Switch (Part B) 5-45Figure 6-1 Parity Checking in Data Path 6-5Figure 6-2 Safari Internal Ports on the I/O Board DX Chip 6-6Figure 6-3 Level 1 Data Repeater Internal Parity Detectors

and Regenerators 6-7Figure 6-4 Parity Protection for Address Interconnects 6-8Figure 6-5 ECC Error Detection and Reporting Path 6-13Figure 6-6 End-to-End ECC Protection

(Including Intermediate ECC Detection) 6-14Figure 6-7 Data Path ECC Locations 6-20Figure 6-8 DX Incoming/Outgoing Data Paths 6-23Figure 6-9 Uncorrectable ECC Error From a Bad CPU 6-24Figure 6-10 Uncorrectable ECC Error From a Memory

Module 6-26Figure 6-11 Physical Address Spaces 6-38Figure 7-1 UltraSparc III Processor Layout 7-5Figure 7-2 Caches and Data Currency 7-7Figure 7-3 Address Bus Transaction Broadcast 7-10Figure 7-4 Cache Coherency State Transitions 7-14Figure 7-5 ATransID Bit Layout 7-19Figure 7-6 The Address Request Transaction 7-23Figure 7-7 Data Transaction Read Response 7-25Figure 7-8 Target Transaction 7-26Figure 7-9 Write Transaction 7-28Figure 7-10 Read-to-Share from Memory on

Another Board 7-29Figure 7-11 SDC Arbiter 7-33Figure 7-12 Inbound and Outbound Transfers on the

DX Switch 7-35Figure 8-1 Memory Subsystem 8-9Figure 9-1 Console Bus Structure 9-4Figure 9-2 Console Bus Hub 9-5Figure 9-3 SC SBBC Block Diagram 9-9Figure 9-4 System Board SBBC Connection Block Diagram 9-11Figure 9-5 I/O Board SBBC Block Diagram 9-13Figure 9-6 SBBC, I/O PROM and I2C Buses 9-18Figure 9-7 SC I2C Bus Multiplexing 9-21Figure 9-8 CPU Board Global I2C Bus 9-27Figure 9-9 ID Board 9-32Figure 9-10 System Board Error Signal Paths 9-36Figure 9-11 I/O Board Error Signal Paths 9-37Figure 9-12 Fireplane Switch Board Error Signal Paths 9-38Figure 9-13 SC Board Error Signal Paths 9-39Figure 9-14 Error Reporting Hierarchy 9-41Figure 9-15 Error Pause Signal Distribution 9-42

Trang 18

Figure 9-16 Error Reporting Paths 9-43Figure 9-17 Basic Clock Distribution 9-44Figure 9-18 Local Clock Distribution (System Board) 9-48Figure A-1 SRAM and DRAM A-2Figure A-2 Single DRAM Bit Cell Structure A-4Figure A-3 DRAM Data and Control Flow A-5Figure A-4 SRAM Single Bit Cell Design (Six Transistor) A-7Figure A-5 DRAM Data Access Mechanism A-8Figure B-1 UltraSPARC III Physical Layout B-4Figure B-2 The UltraSPARC III Pipeline B-7Figure B-3 UltraSPARC-III Functional Units B-12Figure B-4 Instruction Fetch Logic Flow B-13Figure B-5 Integer Execute Unit B-15Figure B-6 Data Cache Unit B-19Figure B-7 Prefetch Cache Data Flow B-19Figure B-8 Data Cache SAM Addressing B-20Figure B-9 Memory Subsystem Interconnect B-22Figure C-1 Performance Loss to Cache Misses C-7Figure C-2 Example of Cache Thrashing C-9

Trang 19

Specifications 1-7Table 1-5 Sun Fire ASIC List 1-16Table 1-6 CPU Numbering 1-21Table 1-7 IOC AID Numbering 1-23Table 1-8 Device Path to I/O Card Slot Location Mapping 1-25Table 1-9 Device Path to I/O Card Slot Location Mapping 1-27Table 1-10 3800 I/O Assembly AIDs 1-29Table 1-11 Physical Slot Numbers for Sun Fire

3800 Systems 1-30Table 1-12 Service Mode Command Summary 1-33Table 1-13 OpenBoot Promerror-reset-recovery

Variable 1-34Table 1-14 Even and Odd Parity 1-37Table 2-1 AC/DC Power Supply Ratings 2-8Table 2-2 System Centerplane Configurations 2-10Table 2-3 Dual Power Grid Configuration for

Sun Fire 6800 2-14Table 2-4 Component Power Supply Ratings 2-16Table 2-5 Board Voltage Requirements 2-17Table 3-1 Relative Bus Performance of Segments

and Domains 3-11Table 3-2 Single Segment Configuration for

Sun Fire 48x0/3800 3-12Table 3-3 Dual Segment Configuration for

Sun Fire 48x0/3800 3-12Table 3-4 Sun Fire 48x0/3800 MAC/HostIDs 3-13Table 3-5 Single Segment Configuration for 6800 3-13Table 3-6 Dual Segment Configuration for Sun Fire 6800 3-13Table 3-7 Sun Fire 6800 MAC/HostIDs 3-14

Trang 20

Table 3-8 Sun Fireplane Switch Board Names 3-15Table 3-9 Configuration Status for Sun Fire 3800

and 48x0 Platforms 3-15Table 3-10 Configuration Status for

the Sun Fire 6800 Platform 3-17Table 4-1 I/O Assembly Types by Model 4-7Table 4-2 PCI Board Slot Characteristics 4-10Table 4-3 cPCI Board Slot Characteristics 4-14Table 4-4 3800 cPCI Board Slot Characteristic 4-17Table 4-5 I/O Board Power Requirements 4-18Table 5-1 Address Repeater Port Interconnects 5-22Table 5-2 SDC Port Interconnects 5-25Table 5-3 System and Repeater Board DX Internal

and External Connections 5-30Table 5-4 Sun Fire 6800 System Board DX Port

Connections 5-39Table 5-5 Sun Fire 6800 Connections 5-40Table 5-6 DCDS Memory Interface (M) and CPU

Interface (C) Bit Assignment 5-45Table 5-7 I/O Board IOC and DX Bit Assignments

with Fireplane Bus Meaning 5-47Table 6-1 Sun Fire ECC Syndrome Table 6-17Table 6-2 DX ECC Register 6-22Table 6-3 DX Register Decoded 6-23Table 6-4 DRAM IDs 6-28Table 6-5 Reference Designator Position of the

DRAM Chip 6-30Table 6-6 Asynchronous Fault Status Register 6-33Table 6-7 Asynchronous Fault Address Register 6-37Table 6-8 Physical Address Mappings 6-39Table 6-9 AFAR Address Decoded 6-41Table 7-1 Data Origins of Snoop Response Signals 7-12Table 7-2 Transaction Command Field Values 7-19Table 7-3 Address Arbitration Round-Robin Priorities 7-32Table 7-4 L2 Data Arbitration Determination 7-34Table 7-5 SDC ECC Register 7-38Table 7-6 SDC Register Decoded 7-40Table 8-1 Sun Fire DIMM Capacities (discounting ECC) 8-3Table 8-2 Memory Bank and DIMM Locations 8-5Table 9-1 The SBBC CPU Board Error Register 9-15Table 9-2 Global I2C Bus Assignments 9-21

Trang 21

Table 10-1 Errors and Probable FRUs 10-3Table 13-1 Cache Characteristics C-2

Trang 23

The Advanced Sun Fire™ Mid-Range Troubleshooting course goes beyond

field replacement unit (FRU) maintenance by focusing oninterrelationships between application-specific integrated circuits (ASICs)and the resulting error outputs

The strategy provided by the preface is to introduce students to the course before they introduce themselves

to you and one another By familiarizing them with the content of the course first, their introductions will have more meaning in relation to the course prerequisites and objectives.

Upon completion of this course, you should be able to:

● List and identify the models, interconnect, and key features of theSun Fire server product line

● Explain the differences in power management among the Sun Fireserver models

● Identify misconfigured domains and segments and build an actionplan to rectify the situation

● Identify circuitry on an L1 board using error reports

● Describe how the Sun Fire family uses the Fireplane switch boardsdifferently

● Given error checking and correction (ECC) reports, identify anddiagnose faulty components

● Discuss and interpret different types of failure information

Trang 24

Domains and Segments

Sun™ Fireplane Interconnect Bus

Parity and ECC Detection and Recovery

Caching and Interconnect Operations

Memory Interleaving

Hardware

and I/O Boards

The System

Trang 25

Topics Not Covered

This course does not cover the following topics Many of these topics arecovered in other courses offered by Sun Educational Services:

● FRU Maintenance – Covered in IES-SM30: Sun Fire Field Maintenance

● Storage Area Networks – Covered in ES-475: Design and

Administration of Storage Area Networks

● Solaris Administration – Covered in SA-289: Solaris™ 8 System

Administration II

Refer to the Sun Educational Services catalog for specific information andregistration

How Prepared Are You?

To be sure you are prepared to take this course, can you answer yes to thefollowing questions?

● Have you attended the IES-SM30 course?

● Can you identify and replace FRU components of the Sun Fireproduct line?

● Can you implement domains on the Sun Fire product line?

● Can you administer the Solaris Operating Environment?

● Do you have six months’ minimum field/on-the-job experience withthe Sun Fire product line?

Trang 26

Introduction

Now that you have been introduced to the course, introduce yourself toeach other and the instructor

How to Use Course Materials

To enable you to succeed in this course, these course materials employ alearning model that is composed of the following components:

● Goals – You should be able to accomplish the goals after finishingthis course and meeting all of its objectives

● Objectives – You should be able to accomplish the objectives aftercompleting a portion of the instructional content Objectives supportgoals and can support other higher-level objectives

● Lecture – The instructor will present information specific to theobjective of the module This information will help you acquire theknowledge and skills necessary to succeed with the activities

● Activities – The activities take on various forms, such as an exercise,self-check, discussion, and demonstration Activities are used tofacilitate mastery of an objective

● Visual aids – The instructor might use several visual aids to convey aconcept, such as a process, in a visual form Visual aids commonlycontain graphics, animation, and video

Conventions

The following icons and typographical conventions are used in this course

to represent various training elements and alternative learning resources

Icons

Trang 27

?

!

Discussion – Indicates a small-group or class discussion on the current

topic is recommended at this time

Power user – Indicates additional supportive topics, ideas, or other

optional information

Note – Indicates additional information that can help but is not crucial to

understanding of the concept being described Examples of notationalinformation include keyword shortcuts and minor system adjustments

Caution – Indicates that there is a risk of personal injury from a

nonelectrical hazard, or risk of irreversible damage to data, software, orthe operating system A caution indicates that the possibility of a hazard(as opposed to certainty) might happen, depending on the action of theuser

Caution – Indicates that either personal injury or irreversible damage of

data, software, or the operating system will occur if the user performs thisaction A warning does not indicate potential events; if the action isperformed, catastrophic events will occur

Caution – Indicates the risk of injury due to heat or hot surfaces will

result

Trang 28

Typographical Conventions

Courieris used for the names of command, files, and directories, as well

as on-screen computer output For example:

Use ls -alto list all files

system% You have mail

Courier boldis used for characters and numbers that you type Forexample:

system% su

Password:

Courier italicis used for variables and command-line place-holdersthat are replaced with a real name or value For example:

To delete a file, type the rm filename command

Courier italic boldis used to represent variables whose values are to

be entered by the student as part of an activity; for example:

Type chmod a+rwx filenameto grant read, write, and executerights for filename to world, group, and users

Palatino italics is used for book titles, new words or terms, or words that

are emphasized For example:

Read Chapter 6 in User’s Guide.

You must be root to do this.

Trang 29

Notes to the Instructor

You should refer the instructor to the appropriate README file, SETUP file, or both files for additional

information.

The README file should contain information specific to the content of the course.

The SETUP file should contain specific setup instructions about how to set up this course It should also contain any special instructions for setting up the HTML overheads The sample templatesetup.txt

contains a sample of the type of information that could go in the SETUP file.

Both the README and SETUP files are text files.

This section should be tagged with paragraph tags containing the prefix, IG; for example, the heading is IGHead1, and the paragraph is IGParagraph IG stands for Instructor Guide, and this text should appear only

in the Instructor Guide You must also make this section conditional, using the following steps:

1 Highlight the text you want to change to conditional text, including the paragraph marker, if appropriate.

2 Select Conditional Text from the Special menu.

3 In the Conditional Text dialog box in the Not in: list, select IGCondition, and move it to the In: list.

4 Click Apply.

Do not change the IG-prefixed paragraph tags to be the color red They should turn red when you apply the IGCondition conditional property to them.

Trang 31

Upon completion of this module, you should be able to:

● List the models and key features of the Sun Fire server product line

● Review the board layouts of each Sun Fire server model

● List the interconnect speeds of the Sun Fire servers

● Identify the Sun Fire board resources by location and function

● Define parity and Error Correction Code (ECC)

Trang 32

Relevance

Present the following questions to stimulate the students and get them thinking about the issues and topics presented in this module While they are not expected to know the answers to these questions, the answers should be of interest to them and inspire them to learn the material presented in this module.

?

!

Discussion – The information found in this module is relevant to your job

because it introduces you to the Sun Fire server products The followingquestions are relevant to understanding what this module is all about:

● What Sun Fire server models are available?

● How is each Sun Fire server model used?

● What are the key features of each Sun Fire server model?

Additional Resources

Additional resources – The following references provide additional

details on the topics discussed in this module:

● Sun Microsystems, Inc., Sun Fire System Overview Manual, Part

Number 805-7362

● Sun Microsystems, Inc., Sun Fire Systems Platform Administration

Manual, Part Number 805-7373

● Sun Microsystems, Inc., Sun Fire 6800 Installation Guide, Part Number

● Sun Microsystems, Inc., Sun Fire Midrange Systems Hardware Reference

Manual, Part Number 805-7363

● Sun Microsystems, Inc., Sun Enterprise Serengeti-12i, 12, 8, and

Trang 33

Sun Fire Server Models

There are four models of Sun Fire servers as shown in Figure 1-1:

● Sun Fire 3800 server

Figure 1-1 Sun Fire Server Models

Server Naming

Engineering specifications show that the members of the Sun Fire familyhave been renamed several times The various names are included inTable 1-1 to avoid confusion when using the reference documentation

Sun Fire 3800 Sun Fire 4800 Sun Fire 4810 Sun Fire 6800

Table 1-1 Sun Fire Family Names

Release Name Sun Fire 3800 Sun Fire 4800 Sun Fire 4810 Sun Fire 6800

Internal Name Serengeti-8 Serengeti-12 Serengeti-12i Serengeti-24

SPServiceProvider

MDMid-RangeDeskside

MEMid-RangeEnterprise

DSDatacenterServer

Trang 34

There are also component names that are different from the engineeringspecifications in the announced product Some of these names are listed inTable 1-2

The primary difference between UltraSPARC III and UltraSPARC III+ is an improvement in the way in which the UltraSPARC III+ accesses cache.

Table 1-2 Sun Fire Family Component Names Sun Fire Name Internal Name

Sun FireplaneInterconnect Bus

Safari

Trang 35

Sun Fire Features

Table 1-3 provides a comparison of the features of each of the Sun Fireservers

Table 1-3 Sun Fire Family Maximum Configurations

Feature Sun Fire

3800

Sun Fire 4800

Sun Fire 4810

Sun Fire 6800

Number ofDIMMs

Memory capacity(with 1 GB*

DIMMs)

Fireplane switchboards

Hot-swapCompactPCI

PCI and hot-swap CompactPCI

PCI slots perassembly

Max I/Oassemblies

Powerrequirements

100-120 or220-240VAC

220-240 VAC

Trang 36

Sun Fire Features

* 1 GB DIMMS only available with 900 MHz CPUs

Systemcontrollers

2 boards

Redundantcooling

Yes

Redundant powersupplies

Yes; N+1

Redundant ACinput

Yes

Rack

Table 1-3 Sun Fire Family Maximum Configurations (Continued)

Feature Sun Fire

3800

Sun Fire 4800

Sun Fire 4810

Sun Fire 6800

Trang 37

Interconnect Capabilities

Table 1-4 summarizes the best case of the Fireplane interconnect capacitiesfor each of the Sun Fire servers

Table 1-4 Sun Fire Family System Interconnect Specifications

Feature Sun Fire 3800 Sun Fire

4800 /4810 Sun Fire 6800

Address interconnect 9.6 GB/secSystem board internal

bisection bandwidth

4.8 GB/sec

System boarddata bandwidth

4.8 GB/sec

I/O boarddata bandwidth

2.4 GB/sec

Inter-board datainterconnect

4 x 4 crossbar 5 x 5 crossbar 10 x 10 crossbar

Trang 38

Peak System Bandwidth

The peak bandwidth of a system bus can be calculated as the lesser of thesystem’s address-limited bandwidth and data-limited bandwidth

An address transaction is required to initiate each data movement Themaximum for each address request can be 64 bytes To calculate theaddress-limited bandwidth (the maximum amount of data that can berequested), use:

150 MHz system clock * 64-byte cache line = 9.6 Gbyte/sec

Since the peak bandwidth is the smaller of either the address or datalimited bandwidth, the peak bandwidth is 9.6 Gbyte/sec, imposed by theaddress bus

Trang 39

System Board Physical Locations

System board slot assignments for each server are shown in the followingfigures

Sun Fire 6800 Server

Figure 1-2 shows the location of the Sun Fire 6800 server system boards

Figure 1-2 Sun Fire 6800 System Board Slot Assignments

SB4 SB2 SB0

SB1 SB3 SB5

Trang 40

Sun Fire 4810 and 4800 Servers

Figure 1-3 shows the location of each system board in the Sun Fire 4810and Sun Fire 4800 servers

Figure 1-3 Sun Fire 48x0 System Board Slot Assignments

Sun Fire 3800 Server

Figure 1-4 shows the system board locations in the Sun Fire 3800 server

SB0 SB2 SB4

Sun Fire 4800 Server Sun Fire 4810 Server

SB0 SB2 SB4

SB1

SB0

Định dạng
Số trang	560
Dung lượng	3,99 MB