Student Guide - IES-421 Sun Fire™ High-End Server Maintenance and Administration - Slide

Reviewing the System Controllers• Provides an interface to the customer’s network • Provides a programmable system and processor clock • Sets up the server and coordinates the boot proce

Sun Fire™ High-End Server Maintenance and Administration

IES-421

Sun, Sun Microsystems, the Sun logo, OpenBoot, Solaris, Sun StorEdge, SunVTS, Sun Fire, SunSolve, and SunSolve Online are trademarks or registered trademarks of Sun Microsystems, Inc in the U.S and other countries.

All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc in the U.S and other countries Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc.

UNIX is a registered trademark in the U.S and other countries, exclusively licensed through X/Open Company, Ltd.

U.S Government approval might be required when exporting the product.

RESTRICTED RIGHTS: Use, duplication, or disclosure by the U.S Government is subject to restrictions of FAR 52.227-14(g)(2)(6/87) and FAR 52.227-19(6/87), or DFAR 252.227-7015 (b)(6/95) and DFAR 227.7202-3(a).

DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS, AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY

OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

Sun, Sun Microsystems, le logo Sun, OpenBoot, Solaris, Sun StorEdge, SunVTS, Sun Fire, SunSolve, et SunSolve Online sont des marques de fabrique ou des marques déposées de Sun Microsystems, Inc aux Etats-Unis et dans d’autres pays.

Toutes les marques SPARC sont utilisées sous licence sont des marques de fabrique ou des marques déposées de SPARC International, Inc aux Etats-Unis et dans d’autres pays Les produits portant les marques SPARC sont basés sur une architecture développée par Sun Microsystems, Inc.

UNIX est une marques déposée aux Etats-Unis et dans d’autres pays et licenciée exclusivement par X/Open Company, Ltd.

L’accord du gouvernement américain est requis avant l’exportation du produit.

LA DOCUMENTATION EST FOURNIE “EN L’ETAT” ET TOUTES AUTRES CONDITIONS, DECLARATIONS ET GARANTIES EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES, DANS LA MESURE AUTORISEE PAR LA LOI APPLICABLE, Y COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE, A L’APTITUDE A UNE UTILISATION PARTICULIERE OU A L’ABSENCE DE CONTREFAÇON.

Preface

About This Course

Course Goals

Upon completion of this course, you should be able to:

• Remove and install most Sun Fire™ high-end server(HES) field-replaceable units (FRUs)

• Configure the Sun Fire HES server platform and

Course Goals

• Use available Solaris™ Operating System (Solaris OS)and platform-specific tools to troubleshoot the Sun FireHES

• Identify and administer Capacity-on-Demand (COD)systems

Course Map

Sun Fire Server Review and Installation

Sun Fire HES Configuration

Sun Fire HES

Configuring the Sun Fire HES

Configuring

Exploring Sun Fire

HES Exploring Dynamic

Reconfiguration

Troubleshooting

Sun Fire HES Administration and Troubleshooting

Sun Fire HES

Sun Fire HES

Managing the Sun Fire Networks and

HES System Controller

Capacity-on-Demand

Platform Product and

Version 2.0

Installing Sun Fire High-End Servers Architecture Review

and Assessment

Topics Not Covered

• Solaris OS administration – Covered in SA-118:

Fundamentals of Solaris™ 8 Operating Environment for System Administrators

• Storage administration – Covered in ES-255: Sun™

Hardware RAID and T3 Storage System Administration

• Sun Fire 3800, Sun Fire 4800, Sun Fire 4810, and SunFire 6800 server administration – Covered in ES-420:

Sun Fire™ Workgroup/Enterprise Server Administration

This course does not cover advanced architecture theory

Additional resources

URL ResourcesDiscussion

Power user

Note

?

!

Caution

Caution – Electrical

Typographical Conventions

directories, programming code, programmingconstructs, and on-screen computer output

that you type, and for each line of programming codethat is referenced in a textual description

command-line placeholders that are replaced with areal name or value

whose values are to be entered by the student as part of

an activity

Typographical Conventions

• Palatino italic is used for book titles, new words or

terms, or words that are emphasized

Module 1

Sun Fire™ High-End Server Product and Architecture Review and Assessment

Describe the Sun Fire HES products and architecture

Reviewing Sun Fire HES

This course is designed to provide you with the informationto:

• Remove and replace FRUs on the high-end servers

• Install and configure the System Management Services(SMS) software

• Manage the platform with the SMS software

• Create and manage domains on the HES

• Perform dynamic reconfiguration

• Perform basic troubleshooting using both manual andautomatic diagnosis techniques

Sun Fire HES

Product Review

• Cabinet with up to nine board sets on each side

• System controller (SC) board set on each side

• Products

• Sun Fire 15K/12K server

• Sun Fire E25K/E20K server

• Domain configuration units (DCUs)

• Slot 0 boards and Slot 1 boards

• Expander (can be split between domains)

Reviewing the System Controllers

• Provides an interface to the customer’s network

• Provides a programmable system and processor clock

• Sets up the server and coordinates the boot process

• Monitors environmental sensors

• Indicates the status and control of power supplies

• Analyzes errors and takes corrective action

• Provides the server console functionality

• Provides a fully redundant system controller board

• Sets up the server domains

• Provides centralized time-of-day

• Provides centralized reset logic

Sun Fire 15K Server System Interconnect

Unique 3-level bus architecture

Mem CPU

Mem CPU Mem CPU

Mem CPU

Dual 3X3

Dual 3X3

PCI card PCI card PCI card PCI card

PCI ctl

PCI ctl 3X3

CPU

Mem

Mem Mem

Mem

Dual 3X3

Dual 3X3

PCI ctl

PCI ctl

1.2

1.2

43 GB/s centerplane

System Control boardset

Understanding Memory

Coherency

• Snooping within an expander board

• Scalable shared memory (SSM)

coherency across board sets

SSM Agent

P M

P M Snoopy coherence domain

SSM Agent

M

P M

P M

P M Snoopy coherence domain

SSM Agent

M

P M

P M

P M Snoopy coherence domain

Snoopy coherence domain

SSM Agent

P M

P M

P M

P M

IO IO

Snoopy coherence domain

SSM Agent

P M

P M

P M

P M

IO IO

Snoopy coherence domain

SSM Agent

P M

P M

P M

P M

Interconnect Levels

C/M Level 0 Level 1 Level 2 Level 3

C/M C/M C/M

I/O I/O

I/O I/O

Address Interconnect Levels

Address repeater Address repeater

Link controller

To 17 other board sets

Slot-0 board type:

4 CPUs/4 memory units

Slot-1 board type:

2 PCI controllers

Snoopy coherency Point-to-point coherency

Data Interconnect Levels

Dual proc data switch

Dual proc data switch

Data switch Data switch Proc

Board set

System data interface

Level-2 data switch

Centerplane data switch (18x18) (43 GB/s bisection bandwidth)

Slot-0 board type:

4 CPUs/4 memory units

Slot-1 board types:

Point-to-point coherency

PCI card

PCI card

Optics card

Optics card

PCI controller

Link controller

0.2 GB/s

0.4 GB/s 2.4 GB/s 2.4 GB/s2.4 GB/s 2.4 GB/s

2.4 GB/s 2.4 GB/s2.4 GB/s 2.4 GB/s

Module 2

Installing Sun Fire High-End Servers

• Describe the Sun Fire HES Enterprise Installation

Services (EIS) methodology

• Locate relevant installation information on the EIS

CD-ROM

• Identify FRUs by location and function

• Identify power requirements

• Power on and power off the Sun Fire HES

• How does the Sun Fire HES use EIS methodology tosupport a good installation?

• What components make up the Sun Fire HES models?

• What role does each component play in Sun Fire HESoperation?

• Where are the components located?

• What status indicators are associated with each

component?

Architecture Overview Hardware Layer

Applications

Solarisª Operating System

OpenBootª PROM

System Controller Environment

Platform Hardware

Understanding Enterprise Installation

Services Methodology

The EIS methodology consists of:

• A defined set of deliverables

• Documentation sets (site planning, system preparation,and so forth)

• EISdoc tool (documentation generation)

• Best practices (cable labeling and grounding)

• EIS companion CD-ROM

• Technical install checklists

Understanding the EISdoc Tool

The EISdoc tool lets you create, archive, search, retrieve, edit,and print documents using web-based templates and forms.The pre-defined templates include document

custom-tailoring, such as fill-in forms and check boxes,

supported within a web browser or an ASCII text format

Goals of the EISdoc Tool

• A supportable tool for documentation

• An easy-to-use tool whose operation is self-evident andbased on well-known web idioms

• A flexible tool letting you create new types of

documents by supporting new templates withoutmodifying the existing tool

• Support for archiving documents on a central server

• Support for search of document archives on a centralserver

• Support for check-out and check-in work flow of

documents in the archive on the central server

Goals of the EISdoc Tool

• Automatic creation of high-quality,

professional-looking, predictable documents thatconform to legal specifications

• Support for localization (editing and printing of

documents in many languages)

EISdoc Resources

As of the writing of this module the EISdoc tool is at

version 3.2 The download site is listed below The tool hassupported versions that can be installed on a Microsoft

Windows-based personal computer (PC) or on Solaris OS,

and is automatically configured to retrieve document

templates, and retrieve an archive documents on a

worldwide-accessible central server (on SWAN only)

Using the EIS Installation and Patch

• The installation is faster

• The installation is of a higher quality due to the

extensive check lists that are followed

• The installation is consistent

Using the EIS Installation and Patch

CD-ROM

Sun Fire HES EIS Installation Checklist

One of the main features of the EIS-CD is the EIS installationchecklist for Sun Fire HES This checklist provides referencesand verification to guarantee a consistent and functional

installation of the Sun Fire hardware, including customer

peripherals

Using the EIS Installation and Patch

CD-ROM

Other EIS-CD Tools

Some of these tools include:

• Scripts to mirror the system controller’s boot disk usingSolaris Volume Manager software

• Network Time Protocol (NTP) configurations for SCsand domains

• Instructions on how to set the device alias when theboot device is a Sun StorEdge™ T3 array

• A script to generate /etc/ethers for the media accesscontrol (MAC) address of each domain

Using the EIS Installation and Patch

CD-ROM

• An overnight burn-in script that runs the power-on

self-test (POST) for long periods of time and save thelog files

• A script that sets up numerous “standards” on Sun

servers

• The latest version of Explorer

Major System Components

System cabinet

Fan backplane

Address and data ASICs

Address and data ASICs

Slot 0 boards:

4 CPUs and E-caches

32 DIMMs Slot 1 boards:

4 PCi cards in hot-swap cassettes Expander board

System board sets

(9 each)

System board sets (9 each)

Control board set

Control board set

Sun Fireplane

Fan trays (4 each)

Bulk power supplies

Sun Fire HES Board Sets

System board set

Expander board System carrier plate and expander

Logic centerplane

Power centerplane

Centerplane support board

CPU memory

Sun Fire™ Link

hsPCI

Control board set

System control carrier plane

Exploring System Boards

System Board Set

The Sun Fire HES has three types of the Slot 1 board types.They are:

• The hot-swap peripheral component interface (hsPCI)board

• The Sun Fire link board

• The MaxCPU board

Sun Fire HES System Board

Exploring System Boards

• The central processing unit (CPUs)

on the board are UltraSPARC III

Cu 900-MHz or 1.05-GHzmicroprocessors

• Each CPU has a built-in memorycontroller with two physical

memory banks

• UltraSPARC IV system board

• Two “cores” per CPU module

• Physically shared level-2 cache

Exploring System Boards

CPU and Memory Configuration

• Each physical memory bank consists of four DIMM

slots, each CPU has direct access to eight DIMM slots,

or two physical memory banks

• Each physical bank has two logical banks

• The physical path of a memory component is expressed

Exploring System Boards

System Board DIMM Slot Assignments

DIMM

Slot Processor

Physical Bank DIMM

Logical Bank, Front

Logical Bank, Back

Exploring System Boards

System Board DIMM Slot Assignments

DIMM

Slot Processor

Physical Bank DIMM

Logical Bank, Front

Logical Bank, Back

Exploring System Boards

System Board DIMM Slot Assignments

DIMM

Slot Processor

Physical Bank DIMM

Logical Bank, Front

Logical Bank, Back

Exploring System Boards

System Board DIMM Slot Assignments

DIMM

Slot Processor

Physical Bank DIMM

Logical Bank, Front

Logical Bank, Back

Exploring System Boards

Memory Configuration Rules

• The system board supports 256-Mbyte, 512-Mbyte, and1-Gbyte DIMM sizes (2-Gbyte for UltraSPARC IV)

• Each CPU on the system board can be configured with:

• No memory installed

• One bank (Bank 0) installed

• Two banks (Bank 0 and Bank 1) installed

• A populated DIMM bank must have a correspondingCPU installed

• When memory is installed, all DIMMs in a physical

bank must be present and the same size

CPU and Memory Locations

DIMM 1, Bank 0 DIMM 1, Bank 1 DIMM 0, Bank 0 DIMM 0, Bank 1

DIMM 2, Bank 1 DIMM 2, Bank 0 DIMM 3, Bank 1 DIMM 3, Bank 0

Dual CPU data switch

System Board LEDs

Removal OK LED (amber)

Fault LED (amber)

Activated LED (green)

Exploring Sun Fire HES I/O Support

• The Sun Fire server only supports standard peripheralcomponent interface (PCI) cards in its hsPCI interfaces

• Sun’s SBus is not supported on Sun Fire HES

• Input/output (I/O) boards host four PCI cards in

hsPCI I/O Assembly Slot Assignments

Cassette 5V (blue)

C5V0 (slot 1)

C3V0 (slot 0)

Cassette 3.3V (yellow) Front view

C3V1 (slot 2) C5V1 (slot 3)

hsPCI I/O Assembly Status LEDs

Cassette 5V (blue)

Cassette 3.3V (yellow)

Front view

Removal indicator (amber)

Activation indicator (green) Service indicator (amber)

Removal indicator (amber)

Activation indicator (green) Service indicator (amber)

Removal indicator (amber)

Activation indicator (green) Service indicator (amber) Removal indicator (amber)

Activation indicator (green) Service indicator (amber)

MaxCPU Board LED Indicators

Activation indicator (green) Service indicator (amber) Removal indicator (amber)

Sun Fire Link Board LED Indicators

Cassette 5V Cassette 3.3V PAROLI*

PAROLI*

Insert/eject lever

*Single PAROLI shown (dual PAROLI optional)

Activation indicator (green) Service indicator (amber) Removal indicator (amber)

Removal indicator (amber) Service indicator (amber) Activation indicator (green)

Activation indicator (green) Service indicator (amber) Removal indicator (amber) Activation indicator (green) Service indicator (amber) Removal indicator (amber) Activation indicator (green) Service indicator (amber) Removal indicator (amber)

System Expander Board

System expander board

CDC SRAM DIMM

J2 LED connector

Exploring the System Controller

• An interface to the customer’s administrative network

or other user network

• A console bus interface to every board—JTAG access toall boards through the console bus

• 100BASE-TX Ethernet interface

• TTYA/TTYB serial interfaces