Perform the following steps:
1. Boot to the Solaris OS.
2. Observe the boot messages.
3. Log in as root. Use thepsanddfcommands to view the processes that are running and what file systems are mounted.
4. Identify the domain’s network interfaces. Which is the domain console interface?
domain# ifconfig -a
______________________________________________
5. Run the nddcommand that shows information about the domain’s connection to the system controller.
Write the command used: ____________________________
Based on the command output, note the following information:
I1 domain IP address = ______
I1 domain IP netmask = ______
I1 domain IP netnum = ______
I1 domain Ethernet address = _______
I1 SC IP address = ______
I1 SC Ethernet address = ______
I/O boards in this domain = ________
I/O board containing the golden IOSRAM = _____
6. If not already configured, configure the domain’s MAN interface in the Solaris OS based on the information noted in the previous step.
7. Display the domain’s TOD and compare it to the TOD kept on the system controller.
Does the domain TOD, match the system controller’s TOD value for the domain? Why or why not?
______________________________________________
8. Change to the /devicesdirectory. Find devices that correspond to the OpenBoot devices noted in Steps 2, 3, and 4 of Task 2.
Exercise Summary
?
!
Discussion – Take a few minutes to discuss what experiences, issues, or discoveries you had during the lab exercise.
Manage the discussion based on the time allowed for this module, which was provided in the “About This Course” module. If you do not have time to spend on discussion, highlight just the key concepts students should have learned from the lab exercise.
● Experiences
Ask students what their overall experiences with this exercise have been. Go over any trouble spots or especially confusing areas at this time.
● Interpretations
Ask students to interpret what they observed during any aspect of this exercise.
● Conclusions
Have students articulate any conclusions they reached as a result of this exercise experience.
● Applications
Explore with students how they might apply what they learned in this exercise to situations at their workplace.
Exercise Solutions
Following are task solutions for this exercise.
Task 1 – Configuring a Domain
Perform the following steps:
1. Log in to the system controller as your assigned user.
2. Create a tag name for your assigned domain using the tag name given to you by the instructor at the beginning of this exercise.
Write the command you used.addtag -d domain_ID tag_name What commands can you use to verify the alias?showplatform What effect does the tag name have on the Solaris OS running in the domain?
None.
3. Verify that the ACL for your domain matches your current domain configuration.
Write the commands you used.showplatform -d domain_ID showboards
What impact does the ACL have on the domain administrator?
The ACL restricts the domain administrator to managing only the boards stated in the ACL.
4. Remove all the current domain resources.
Write the command you used.
deleteboard -c unassign SB# IO#
5. Add the system resources, obtained earlier from your instructor, to your assigned domain.
Write the command you used.
addboard -c assign SB# IO#
Were you able to successfully add the assigned system resources to your domain?
No.
6. Set the virtual keyswitch tostandby. Describe what happens.
From theoffposition, thestandbyposition powers on all boards assigned to the domain (if not already powered on).
7. Set the virtual keyswitch tooff. Describe what happens.
From the standbyposition, the offposition puts all boards into low-power mode.
8. Write the domain’s virtual NVRAM parameters controlled by the domain administrator using SMS software.
auto-boot?=false diag-switch?=true fcode-debug?=false use-nvramrc?=true security-mode=none
9. Set the virtual keyswitch toon. Describe what happens.
From the offorstandbyposition, theonposition powers on all boards assigned to the domain (if not already powered on). Then the domain is brought up.
10. Use the appropriate command to display the devices attached to the boards in your assigned domain.
Write the command you used. showdevices What is the speed of the CPUs?
The answer varies.
Record the ID of each CPU in your domain.
The answer varies.
Which board contains the permanent kernel memory?
The answer varies.
What is the total physical memory in your domain?
The answer varies.
Record the I/O board configured in your domain.
The answer varies.
What type of network controller is currently installed in your assigned domain?
The answer varies.
What is the network controller’s current Ethernet and IP addresses?
Task 2 – Analyzing the OpenBoot PROM Device Tree
Perform the following steps:
1. Verify that your domain is currently in OpenBoot mode.
2. Display the port IDs of the CPUs on the system board currently configured in your domain.
Write the port IDs, bus offsets, slots, and board types.
The following example shows SB3:
a. /SUNW,UltraSPARC III@60, 0 b. /SUNW,UltraSPARC III@61, 0 c. /SUNW,UltraSPARC III@62, 0 d. /SUNW,UltraSPARC III@63, 0
Decode the information, listed previously, to determine the following:
Item a:
System/expander board number3 CPU number (decimal) 96
Item b:
System/expander board number3 CPU number (decimal) 97
Item c:
System/expander board number3 CPU number (decimal) 98
Item d:
System/expander board number3 CPU number (decimal) 99
3. Display the port IDs, bus offsets, slots, and board types of the I/O board currently configured in your domain.
Write the full physical path, including the port IDs, bus offsets, slots, and board types.
The following example shows IO3:
a. /pci@7c,600000/pci@1/controller_name b. /pci@7c,700000/pci@1/controller_name c. /pci@7d,600000/pci@1/controller_name d. /pci@7d,700000/pci@1/controller_name
Decode the information, listed previously, to determine the following:
Item a:
I/O expander board number3 PCI IOC number 0
PCI busA
I/O cassette slot number0 (C3V0) Item b:
I/O/expander board number 3 PCI IOC number 0
PCI busB
I/O cassette slot number1 (C5V0) Item c:
I/O board/expander board number 3 PCI IOC number 1
PCI bus0
I/O cassette slot number2 (C3V1) Item d:
I/O board/expander board number 3 PCI IOC number 1
PCI busB
I/O cassette slot number3 (C5V1)
4. Display the default boot device.
Record the full path of the default boot device.
The answer varies.
Decode the default boot device path to determine the following:
I/O board number:The answer varies.
PCI IOC number:The answer varies.
PCI bus:The answer varies.
I/O cassette slot number:The answer varies.
Task 3 – Booting the Solaris OS for the First Time
Perform the following steps:
1. Boot to the Solaris OS.
2. Observe the boot messages.
3. Log in as root. Use thepsanddfcommands to view the processes that are running and what file systems are mounted.
4. Identify the domain’s network interfaces. Which is the domain console interface?
domain# ifconfig -a dman0: flags=...
qfe0: flags=...
lo0: flags=...
5. Run thenddcommand that shows information about the domain’s connection to the system controller.
Write the command used: ndd -get /dev/dman man_get_hostinfo
Based on the command output, note the following information:
I1 domain IP address =The answer varies.
I1 domain IP netmask = The answer varies.
I1 domain IP netnum = The answer varies.
I1 domain Ethernet address =The answer varies.
I1 SC IP address = The answer varies.
I1 SC Ethernet address = The answer varies.
I/O boards in this domain =The answer varies.
I/O board containing the golden IOSRAM = The answer varies.
6. If not already configured, configure the domain’s MAN interface in the Solaris OS based on the information noted in the previous step.
7. Display the domain’s TOD, and compare it to the TOD kept on the system controller.
Does the domain TOD match the system controller’s TOD value for the domain? Why or why not?
The domains get their time of day from the system controller at boot time.
They then manage their TOD clock without reference to the system controller.
8. Change to the/devicesdirectory. Find devices that correspond to the OpenBoot devices noted in Steps 2, 3, and 4 of Task 2.
9. Open the /etc/path_to_instfile. Find the path-to-instance entries that correspond to the OpenBoot devices noted in Steps 2, 3, and 4 of Task 2 and the device files found in Step 8.
Exploring Dynamic Reconfiguration
Objectives
Upon completion of this module, you should be able to:
● Describe the benefits of dynamic reconfiguration
● Discuss the restrictions and concerns that can occur with DR
● Analyze domain configuration status using DR
● Identify the steps required for removing system boards and I/O boards using DR
● Identify the steps required for installing system boards and I/O boards using DR
● Move system boards and I/O boards between domains using DR
Relevance
Present the following questions to inspire the students and get them thinking about the issues and topics presented in this module. While they are not expected to know the answers to these questions, the answers should be of interest to them and inspire them to learn the material presented in this module.
?
!
Discussion – In this module, you learn how to perform concurrent maintenance activities using DR. The following questions are relevant to understanding what this module is all about:
● What is DR?
● What factors must be considered before performing DR?
● Which commands perform DR?
● What are the steps involved in performing various DR tasks?
Additional Resources
Additional resources – The following references provide additional details on the topics described in this module:
● Sun Microsystems, Inc. System Management Services 1.4 Reference Manual, part number 817-3057-10.
● Sun Microsystems, Inc. System Management Services 1.4 Administrator Guide, part number 817-3056-10.
● The rcfgadmman page.
● The rcfgadm_sbdman page.
Introducing Dynamic Reconfiguration
DR performs resource management by altering the configuration of a running system through adding or removing system boards and I/O controllers with minimal or no interruption of domain operations or a system reboot.
DR is also useful in mission-critical environments if a system board fails and must be replaced or if new system boards must be added to the system for additional performance and capacity. DR capability is built into the Solaris OS and is a critical part of the concurrent maintenance strategy.
Benefits of DR
DR increases system availability and flexibility by providing
hot-swappable system board and I/O controller capability in the server hardware. In addition to hot-swappable system boards and I/O
controllers, DR includes:
● Dynamically assigns system boards and I/O boards to domains
● Dynamically unassigns system (Slot 0) boards and I/O (Slot 1) boards from domains
● Dynamically moves boards between domains
● Tests initialization on system components
● Displays board status
● Hot-swaps, adds, or removes individual PCI cards from a domain Note –Only an entire I/O board (slot one board) can be unassigned from the domain or moved to a different domain.
At the time of this writing, DR has the following limitations:
● System boards (CPU/memory) and hsPCI I/O controllers are supported.
● Slot 1 boards (I/O boards and MaxCPU boards) are supported with SMS 1.3.
● hsPCI controllers support Dynamic Reconfiguration activation with SMS 1.1.
DR Operational Locations
DR operations can be performed from three locations:
● From the Sun Fire HES system controller
● From the local domain
● From Sun Management Center 3.x software
Sun Fire HES System Controller
You can execute DR operations from the Sun Fire HES SC by using the SMS commandsaddboard,moveboard, anddeleteboard, or by using the rcfgadmDR utility to accomplish the following tasks:
● Board addition – Dynamically adds a system board to a domain or a new board to the system as a whole.
● Board deletion – Removes a system board from the domain or the system as a whole.
● Board replacement – Replaces a system board with another board, or removes and replaces the system board, adds or removes
components, such as processors, memory, I/O cards, or I/O devices.
Note –A Sun Fire HES does not have a concept of creating a new domain, but you can use DR to free up boards from other domains to assign them to a domain that has never had any boards.
Managing Individual PCI Cards
From the system controller, only the rcfgadmcommand can be used to remove and add individual PCI cards. They cannot be unassigned from the domain or moved individually to different domains.
DR on individual PCI cards supports hot removal, replacement, and addition of I/O cards without rebooting the system.
Sun Management Center 3.x Software
If you prefer to use a GUI for DR, use the Sun Management Center 3.x software on the system controller.
The optional Sun Management Center 3.xsoftware provides the following features:
● Domain management
● A GUI to thercfgadmDR command
● A CLI
Domain
You can also use thecfgadmcommand directly from the domain. The cfgadmcommand was first introduced for the previous Sun Enterprise server line of platforms.
Performing thecfgadmcommand as root from the domain is functionally identical to performing the equivalent rcfgadmcommand from the system controller. When running thecfgadmcommand from the domain, you have the same privileges as if you were running the equivalent rcfgadmcommand as the domain administrator on the system controller.
DR Concepts
DR lets you perform maintenance on an active system by disconnecting and then reconnecting system circuit boards without bringing the system down. In addition to maintenance, DR also provides enhanced
performance by adding boards (thus more processors, memory, and I/O) at periods when resource requirements are high.
Note – Figure 6-1 is just an example of how DR can work. This example does not necessarily reflect Sun Microsystems best practices for domain configurations.
To illustrate reconfiguration of system resources, consider the Sun Fire server configuration depicted in Figure 6-1. Domain A contains System Boards 0 and 2 and I/O Board 1. Domain B contains System Boards 1 and 3 and I/O Board 2.
Figure 6-1 Domains A and B Before Reconfiguration
System Board 0 System Board 1 System Board 2 System Board 3 • • •
I/O 0 I/O 1 I/O 2 I/O 3 • • •
Domain A Domain B
System Board 16 System Board 17
I/O 16 I/O 17
The following system configuration, shown in Figure 6-2, is the result of performing a DR operation, where SB1 is detached from Domain B and attached to Domain A.
Figure 6-2 Domains A and B After Reconfiguration
Notice that only the way in which the boards are connected has changed, but not the physical layout of the boards within the cabinet.
Preparing the Domain for DR Operations
The system must be properly configured so system boards containing critical system resources can be removed. You must plan ahead to ensure that the system is properly configured.
This means that critical system disks must be configured with
multipathing, disk mirroring, or both. If therootor/usr partition is on a disk attached to a controller on the board to be removed, the board cannot be detached unless there is a hardware alternate path to the disk or the disk is mirrored using controllers on other I/O boards.
System board 0 System Board 1 System Board 2 System Board 3 • • • System Board 16 System Board 17
I/O 0 I/O 1 I/O 2 I/O 3 • • • I/O 16 I/O 17
Domain A Domain B
The same applies to network controllers. The board that hosts the primary interface that connects to the domain cannot be detached unless multiple network paths exist to the same network using another board in the same domain.
DR attach Operation
The DR attachoperation adds the specified system board to a domain running the Solaris OS. When the board is added and the board’s resources and interfaces are configured to the operating system, the operating system uses them without any difficulty. The steps for an attach operation are:
1. Assign the board.
2. Connect the board by running POST and entering OpenBoot PROM.
3. Configure the board with the Solaris OS.
DR attach States
When performing a DRattachoperation, two levels of state change are supported:
● Connect – When connect is specified, the DR software performs hardware-specific operations to put the receptacle in the connected state, which allows an occupant to operate normally through the receptacle.
If theconnectoperation is complete, but theconfigureoperation has not yet been performed, the complete in question (board or PCI card) is recognized by the OpenBoot PROM but not yet by the Solaris OS.
● Configure – When configure is specified, the DR software performs hardware-specific operations that allow an occupant’s hardware resources for use by the Solaris OS software. Occupants that are configured are part of the system configuration and are available for manipulation by the Solaris OS software device manipulation
maintenance commands, such aspsradm,mount, and ifconfig. Once theconfigureoperation is complete, the component in question is recognized by the Solaris OS.
If you are attaching I/O devices (an I/O board or card) you might still have to run the devfsadmcommand manually to create new device files.
Note –When attaching boards to a domain, the existing controllers retain their original numbering. New disk controllers on a newly attached board are assigned the next available lowest number.
DR detach Operation
A DR detachoperation removes the specified system board from a domain, which must be already running the operating environment.
While attaching a board is a straightforward operation, detaching a board can be complex. When removing a board from a running domain, the operating environment must stop using all of the board’s resources. These resources fall into three general categories:
● I/O to the interface controllers, both disk and network, connected to the board
● Data from the board’s memory, including the kernel if detaching the board where the kernel resides
● Work from the processors, including interrupts from device drivers Each resource has inherent issues that determine whether DR can be performed. For example, the issues associated with dynamically reconfiguring a system board with nonpageable memory are different from a system board with pageable memory or an I/O board removal.
These types of issues are described throughout this module. The steps for an detach operation are:
1. Unconfigure the board from the Solaris OS.
2. Disconnect the board down to an OpenBoot PROM state.
3. Unassign the board.
DR detach States
When performing a DR detachoperation, two levels of state change are supported:
● Unconfigure– When unconfigure is specified, the DR software performs hardware-specific operations that logically remove an occupants hardware resources from the system. The occupant must currently be configured, and its hardware resources must not be in use by the Solaris OS.
● Disconnect – The change the receptacle state to disconnected. If the occupant state is configured, the disconnect function first attempts to unconfigure the occupant. The disconnect function powers the board
I/O Board Considerations
When a new I/O board is installed into an operational domain, the new I/O board is automatically tested. I/O board testing requires dedicated CPU and memory resources. In an operational domain, a CPU and memory resources are suspended from application use by the Solaris OS to provide available resources for testing (runninghpost) the new I/O board.
When performing DR detachoperations on I/O boards, you must consider the following:
● Multipath I/O configuration
● Non-multipath configuration
● Detach-safe and suspend-safe devices
● Mirrored disk configuration
● Non-vital I/O
● Hot-pluggable hardware
Multipathed I/O Configuration
In a multipath I/O configuration, multiple physical paths (channels) to the same media are mapped to the same logical device. The logical device then accesses the media. The benefit of this configuration is that if one of the physical paths is disabled, I/O access continues on the remaining physical path with little interruption. The Solaris OS provides two forms of multipathed I/O:
● Sun StorEdge Traffic Manager or Volume Manger DMP software
● IPMP
Both Sun StorEdge Traffic Manager software and IPMP are key
components of DR operations on I/O components. If the domain’s vital system resources are multipath configured, then a DR operation can be performed on one of the physical paths without interrupting access to the device.
If you are planning to perform DR operations on I/O components, you must first determine whether multipathing is currently implemented.