Preface xvii Preface No single troubleshooting resource can anticipate every possible glitch that can be encountered in internetworks. But any significant contribution that can be made toward preventing connectivity blockages is a step in the right direction. We hope that this publication contributes to the body of knowledge that makes networks more manageable. Audience Internetworking Troubleshooting Handbook is intended for network administrators who are responsible for troubleshooting internetworks that implement Cisco products and Cisco-supported protocols. Administrators should have hands-on experience in configuring, administering, and troubleshooting a network, should know how to configure routers, switches and bridges, and should be familiar with the protocols and media that their hardware has been configured to support. Awareness of the basic topology of their network is also essential. Document Organization The Internetworking Troubleshooting Handbook providesthe information necessary to troubleshoot many problems commonly encountered in internetworks using Cisco hardware and software products. This publication consists of the following six parts: • The chapters in Part 1, “Introduction to Troubleshooting,” provide an introduction to troubleshooting techniques and an overview of common troubleshooting tools. • The chapters in Part 2, “Hardware, Booting, and Media Problems,” provide information for troubleshooting hardware problems, LAN media problems, and booting (system initialization) problems. • The chapters in Part 3, “Troubleshooting Desktop and Entreprise Routing Protocols,” provide information on troubleshooting common connectivity and performance problems in TCP/IP, Novell IPX, AppleTalk, IBM, and other widely-implemented network environments. • The chapters in Part 4, “Troubleshooting Serial Lines and WAN Connections,” provide information on troubleshooting problems that commonly occur on serial lines and WAN links such as ISDN, Frame Relay, and X.25. • The chapters in Part 5, “Troubleshooting Bridging and Switching Environments,” provide information on troubleshooting problems commonly encountered in ATM switching, LAN switching, and bridging environments. Using This Publication xviii Book Title • The chapters in Part 6, “Troubleshooting Other Internetwork Problems,” provide information on troubleshooting CiscoWorks installations, and on troubleshooting security implementations, including TACACS troubleshooting and password recovery. • Appendixes provide supplemental troubleshooting information, including information on creating core dumps, memory maps for different Cisco routers, technical support information, and a list of references and recommended reading. In addition, at the end of the book are several perforated troubleshooting worksheets to assist you in gathering information when problems occur. Using This Publication This publication is designed to provide users with the information needed to troubleshoot common problems encountered in Cisco-based internetworks. Most chapters focus on describing symptoms, identifying their causes, and suggesting specific actions to resolve the problem. Some material describes preventative measures or tips for identifying problems by interpreting command output. Document Conventions Our software and hardware documentation uses the following conventions: • The symbol ^ represents the key labeled Control. For example, ^D means hold down the Control key while you press the D key. • A string is defined as a nonquoted set of characters. For example, when setting up a community string for SNMP to “public,” do not use quotes around the string, or the string will include the quotation marks. Command descriptions use these conventions: • Examples that contain system prompts denote interactive sessions, indicating that the user enters commands at the prompt. The system prompt indicates the current command mode. Forexample, the prompt router(config)# indicates global configuration mode. • Commands and keywords are in boldface font. • Arguments for which you supply values are in italic font. • Elements in square brackets ([ ]) are optional. • Alternative but required keywords are grouped in braces ({ }) and separated by vertical bars (|). Examples use these conventions: • Terminal sessions and information the system displays are in screen font. • Information you enter is in boldface screen font. • Nonprinting characters, such as passwords, are in angle brackets (< >). • Default responses to system prompts are in square brackets ([ ]). • Exclamation points (!) at the beginning of a line indicate a comment line. • When part of the command output has been omitted (to conserve space), the deleted output is indicated with italicized brackets and ellipsis ( [ .] ) Note This is a special paragraph that means reader take note. It usually refers to helpful suggestions, the writer’s assumptions, or reference to materials not contained in this manual. CHAPTER Troubleshooting Overview 1-3 1 Troubleshooting Overview Internetworks come in a variety of topologies and levels of complexity—from single-protocol, point-to-point links connecting cross-town campuses, to highly meshed, large-scale wide-area networks (WANs) traversing multiple time zones and international boundaries. The industry trend is toward increasingly complex environments, involving multiple media types, multiple protocols, and often interconnection to “unknown” networks. Unknown networks may be defined as a transit network belonging to a Internet service provider (ISP) or a telco that interconnects your private networks. In these unknown networks, you do not have control of such factors as delay, media types, or vendor hardware. More complex network environments mean that the potential for connectivity and performance problems in internetworks is high, and the source of problems is often elusive.The keys to maintaining a problem-free network environment, as well as maintaining the ability to isolate and fix a network fault quickly, are documentation, planning, and communication. This requires a framework of procedures and personnel to be in place long before any network changes take place. The goal of this book is to help you isolate and resolve the most common connectivity and performance problems in your network environment. Symptoms, Problems, and Solutions Failures in internetworks are characterized by certain symptoms. These symptoms might be general (such as clients being unable to access specific servers) or more specific (routes not in routing table). Each symptom can be traced to one or more problems or causes by using specific troubleshooting tools and techniques. Once identified, each problem can be remedied by implementing a solution consisting of a series of actions. This book describes how to define symptoms, identify problems, and implement solutions in generic environments. You should always apply the specific context in which you are troubleshooting to determine how to detect symptoms and diagnose problems for your specific environment. General Problem-Solving Model When you’re troubleshooting a network environment, a systematic approach works best. Define the specific symptoms, identify all potential problems that could be causing the symptoms, and then systematically eliminate each potential problem (from most likely to least likely) until the symptoms disappear. Figure 1-1 illustrates the process flow for the general problem-solving model. This process flow is not a rigid outline for troubleshooting an internetwork; it is a foundation from which you can build a problem-solving process to suit your particular environment. General Problem-Solving Model Book Title 1-4 Figure 1-1 General Problem-Solving Model The following steps detail the problem-solving process outlined in Figure 1-1: Step 1 When analyzing a network problem, make a clear problem statement. You should define the problem in terms of a set of symptoms and potential causes. To properly analyze the problem, identify the general symptoms and then ascertain what kinds of problems (causes) could result in these symptoms. For example, hosts might not be responding to service requests from clients (a symptom). Possible causes might include a misconfigured host, bad interface cards, or missing router configuration commands. Step 2 Gather the facts you need to help isolate possible causes. Ask questions of affected users, network administrators, managers, and other key people. Collect information from sources such as network management systems, protocol analyzer traces, output from router diagnostic commands, or software release notes. Step 3 Consider possible problems based on the facts you gathered. Using the facts you gathered, you can eliminate some of the potential problems from your list. Depending on the data, you might, for example, be able to eliminate hardware as a problem, so that you can focus on software problems. At every opportunity, try to narrow the number of potential problems so that you can create an efficient plan of action. Step 4 Create an action plan based on the remaining potential problems. Begin with the most likely problem and devise a plan in which only one variable is manipulated. Changing onlyone variable ata time allows you to reproduce a given solutionto a specific problem. If you alter more than one variable simultaneously, you might solve the problem, but identifying the specific change that eliminated the symptom becomes far more difficult and will not help you solve the same problem if it occurs in the future. Step 5 Implement the action plan, performing each step carefully while testing to see whether the symptom disappears. (If symptoms persist…) (If symptoms stop…) Define problem Problem resolved; terminate process Gather facts Consider possibilities based on facts Create action plan Implement action plan Observe results Repeat process Troubleshooting Overview 1-5 Preparing for Network Failure Step 6 Whenever you change a variable, be sure to gather results. Generally, you should use the same method of gathering facts that you used in Step 2 (that is, working with the key people affected in conjunction with utilizing your diagnostic tools). Step 7 Analyze the results to determine whether the problem has been resolved. If it has, then the process is complete. Step 8 If the problem has not been resolved, you must create an action plan based on the next most likely problem in your list. Return to Step 4, change one variable at a time, and reiterate the process until the problem is solved. Note If you exhaust all the common causes and actions (either those outlined in this book or ones that you have identified for your environment), you should contact your Cisco technical support representative. Preparing for Network Failure It is always easier to recover from a network failure if you are prepared ahead of time. Possibly the most important requirement in any network environment is to have current and accurate information about that network available to the network support personnel at all times. Only with complete information can intelligent decisions be made about network change, and only with complete information can troubleshooting be done as quickly and easily as possible. During the process of troubleshooting the network that it is most critical to ensure that this documentation is kept up-to-date. To determine whether you are prepared for a network failure, answer the following questions: • Do you have an accurate physical and logical map of your internetwork? Does your organization or department have an up-to-date internetwork map that outlines the physical location of all the devices on the network and how they are connected, as well as a logical map of network addresses, network numbers, subnetworks, and so forth? • Do you have a list of all network protocols implemented in your network? For each of the protocols implemented, do you have a list of the network numbers, subnetworks, zones, areas, and so on that are associated with them? • Do you know which protocols are being routed? For each routed protocol, do you have correct, up-to-date router configuration? • Do you know which protocols are being bridged? Are there any filters configured in any bridges, and do you have a copy of these configurations? • Do you know all the points of contact to external networks, including any connections to the Internet? For each external network connection, do you know what routing protocol is being used? • Do you have an established baseline for your network? Has your organization documented normal network behavior and performance at different times of the day so that you can compare the current problems with a baseline? If you can answer yes to all questions, you will be able to recover from a failure more quickly and more easily than if you are not prepared. Preparing for Network Failure Book Title 1-6 CHAPTER Troubleshooting Tools 2-7 2 Troubleshooting Tools This chapter presents information about the wide variety of tools available to assist you in troubleshooting your internetwork, including information on using router diagnostic commands, using Cisco network management tools, and third-party troubleshooting tools. Using Router Diagnostic Commands Cisco routers provide numerous integrated commands to assist you in monitoring and troubleshooting your internetwork. The following sections describe the basic use of these commands: • The show commands help monitor installation behavior and normal network behavior, as well as isolate problem areas. • The debug commands assist in the isolation of protocol and configuration problems. • The ping commands help determine connectivity between devices on your network. • The trace commands provide a method of determining the route by which packets reach their destination from one device to another. Using show Commands The show commands are powerful monitoring and troubleshooting tools. You can use the show commands to perform a variety of functions: • Monitor router behavior during initial installation • Monitor normal network operation • Isolate problem interfaces, nodes, media, or applications • Determine when a network is congested • Determine the status of servers, clients, or other neighbors Following are some of the most commonly used show commands: • show interfaces—Use the show interfaces exec command to display statistics for all interfaces configured on the router or access server. The resulting output varies, depending on the network for which an interface has been configured. Some of the more frequently used show interfaces commands include the following: — show interfaces ethernet — show interfaces tokenring Using Router Diagnostic Commands Book Title 2-8 — show interfaces fddi — show interfaces atm — show interfaces serial — show controllers—This command displays statistics for interface card controllers. For example, the show controllers mci command provides the following fields: MCI 0, controller type 1.1, microcode version 1.8 128 Kbytes of main memory, 4 Kbytes cache memory 22 system TX buffers, largest buffer size 1520 Restarts: 0 line down, 0 hung output, 0 controller error Interface 0 is Ethernet0, station address 0000.0c00.d4a6 15 total RX buffers, 11 buffer TX queue limit, buffer size 1520 Transmitter delay is 0 microseconds Interface 1 is Serial0, electrical interface is V.35 DTE 15 total RX buffers, 11 buffer TX queue limit, buffer size 1520 Transmitter delay is 0 microseconds High speed synchronous serial interface Interface 2 is Ethernet1, station address aa00.0400.3be4 15 total RX buffers, 11 buffer TX queue limit, buffer size 1520 Transmitter delay is 0 microseconds Interface 3 is Serial1, electrical interface is V.35 DCE 15 total RX buffers, 11 buffer TX queue limit, buffer size 1520 Transmitter delay is 0 microseconds High speed synchronous serial interface Some of the most frequently used show controllers commands include the following: — show controllers token — show controllers FDDI — show controllers LEX — show controllers ethernet — show controllers E1 — show controllers MCI — show controllers cxbus — show controllers t1 — show running-config— Displays the router configuration currently running — show startup-config—Displays the router configuration stored in nonvolatile RAM (NVRAM) — show flash—Group of commands that display the layout and contents of flash memory — show buffers—Displays statistics for the buffer pools on the router — show memory—Shows statistics about the router’s memory, including free pool statistics — show processes—Displays information about the active processes on the router — show stacks—Displays information about the stack utilization of processes and interrupt routines, as well as the reason for the last system reboot — show version—Displays the configuration of the system hardware, the software version, the names and sources of configuration files, and the boot images There are hundreds of other show commands available. For details on using and interpreting the output of specific show commands, refer to the Cisco Internetwork Operating System (IOS) command references. Troubleshooting Tools 2-9 Using debug Commands Using debug Commands The debug privileged exec commands can provide a wealth of information about the traffic being seen (or not seen) on an interface, error messages generated by nodes on the network, protocol-specific diagnostic packets, and other useful troubleshooting data. To access and list the privileged exec commands, complete the following tasks: Step 1 Enter the privileged exec mode: Command: Router> enable Password: XXXXXX Router# Step 2 List privileged exec commands: Router# debug ? Note Exercise care when using debug commands. Many debug commands are processor intensive and can cause serious network problems (such as degraded performance or loss of connectivity) if they are enabled on an already heavily loaded router. When you finish using a debug command, remember to disable it with its specific no debug command (or use the no debug all command to turn off all debugging). Use debug commands to isolate problems, not to monitor normal network operation. Because the high processor overhead of debugcommands can disrupt router operation, you should use them only when you are looking for specific types of traffic or problems and have narrowed your problems to a likely subset of causes. Output formats vary with each debug command. Some generate a single line of output per packet, and others generate multiple lines of output per packet. Some generate large amounts of output, and others generate only occasional output. Some generate lines of text, and others generate information in field format. To minimize the negative impact of using debug commands, follow this procedure: Step 1 Use theno logging console global configuration command on your router. This command disables all logging to the console terminal. Step 2 Telnet to a router port and enter the enable exec command. The enable exec command will place the router in the privileged exec mode. After entering the enable password, you will receive a prompt that will consist of the router name with a # symbol. Step 3 Use the terminal monitor command to copy debug command output and system error messages to your current terminal display. By redirecting output to your current terminal display, you can view debug command output remotely, without being connected through the console port. If you use debug commands at the console port, character-by-character processor interrupts are generated, maximizing the processor load already caused by using debug. If you intend to keep the output of the debug command, spool the output to a file. The procedure for setting up such a debug output file is described in the Debug Command Reference. This book refers to specific debug commands that are useful when troubleshooting specific problems. Complete details regarding the function and output of debug commands are provided in the Debug Command Reference. Using Router Diagnostic Commands Book Title 2-10 In manysituations, using third-party diagnostictools can be more useful and lessintrusive than using debug commands. For more information, see the section “Third-Party Troubleshooting Tools” later in this chapter. Using the ping Command To check host reachability and network connectivity, use the ping exec (user) or privileged exec command. After you log in to the router or access server, you are automatically in user exec command mode. The exec commands available at the user level are a subset of those available at the privileged level. In general, the user exec commands allow you to connect to remote devices,change terminal settings on a temporary basis, perform basic tests, and list system information. The ping command can be used to confirm basic network connectivity on AppleTalk, ISO Conectionless Network Service (CLNS), IP, Novell, Apollo, VINES, DECnet, or XNS networks. For IP, the ping command sends Internet Control Message Protocol (ICMP) Echo messages. ICMP is the Internet protocol that reports errors and provides information relevant to IP packet addressing. If a station receives an ICMP Echo message, it sends an ICMP Echo Reply message back to the source. The extended command mode of the ping command permits you to specify the supported IP header options. This allows the router to perform a more extensive range of test options. To enter ping extended command mode, enter yes at the extended commands prompt of the ping command. It is a good idea to use the ping command when the network is functioning properly to see how the command works under normal conditions and so you have something to compare against when troubleshooting. For detailed information on using the ping and extended ping commands, refer to the Cisco IOS Configuration Fundamentals Command Reference. Using the trace Command The trace user exec command discovers the routes that a router’s packets follow when traveling to their destinations. The trace privileged exec command permits the supported IP header options to be specified, allowing the router to perform a more extensive range of test options. The trace command works by using the error message generated by routers when a datagram exceeds its time-to-live (TTL) value. First, probe datagrams are sent with a TTL value of 1. This causes the first router to discard the probe datagrams and send back “time exceeded”error messages. The trace command then sends several probes and displays the round-trip time for each. After every third probe, the TTL is increased by one. Each outgoing packet can result in one of two error messages. A “time exceeded” error message indicates that an intermediate router has seen and discarded the probe. A “port unreachable” error message indicates that the destination node has received the probe and discarded it because it could not deliver the packet to an application. If the timer goes off before a response comes in, trace prints an asterisk (*). The trace command terminates when the destination responds, when the maximum TTL is exceeded, or when the user interrupts the trace with the escape sequence. As with ping, it is a good idea to use the trace command when the network is functioning properly to see how the command works under normal conditions and so you have something to compare against when troubleshooting. For detailed information on using the trace and extended trace commands, refer to the Cisco IOS Configuration Fundamentals Command Reference. [...]... additional troubleshooting information This chapter begins with the following sections on hardware problems: • Cisco 7500 Series Startup—Describes hardware and boot process troubleshooting for Cisco 7500 series routers • Cisco 7000 Series Startup—Describes hardware and boot process troubleshooting for Cisco 7000 series routers • Cisco 4000 and Cisco 3000 Series Startup—Describes hardware and boot process troubleshooting. .. Startup—Describes hardware and boot process troubleshooting for Cisco 2500 series routers • Cisco 2000 Series Startup—Describes hardware and boot process troubleshooting for Cisco 2000 series routers • Catalyst 5000 Series Startup—Describes hardware and boot process troubleshooting for Catalyst 5000 series LAN switches • Catalyst 3000 Series Startup—Describes hardware and boot process troubleshooting for Catalyst... switches • Catalyst 2900 Series Startup—Describes hardware and boot process troubleshooting for Catalyst 2900 series LAN switches • Catalyst 1600 Token Ring Switch Startup—Describes hardware and boot process troubleshooting for Catalyst 1600 Token Ring LAN switches • LightStream 2020 Startup—Describes hardware and boot process troubleshooting for LightStream 2020 ATM switches • Testing and Verifying... analyzer to the suspect network is less intrusive and is more likely to yield useful information without interrupting the operation of the router.The following are some typical third-party troubleshooting tools used for troubleshooting internetworks: • Volt-Ohm meters, digital multimeters, and cable testers are useful in testing the physical connectivity of your cable plant • Time domain reflectors (TDRs)... the contents of frames Monitors are useful for baselining, in which the activity on a network is sampled over a period of time to establish a normal performance profile, or baseline Troubleshooting Tools 2-13 Third-Party Troubleshooting Tools Monitors collect information such as packet sizes, the number of packets, error packets, overall usage of a connection, the number of hosts and their MAC addresses,... combined with information about the network configuration and operation, to diagnose and solve, or offer potential solutions to, network problems C H A P TER 3 Troubleshooting Hardware and Booting Problems This chapter provides procedures for troubleshooting hardware and booting problems Although it provides specific procedures for some Cisco products, always refer to your hardware installation and maintenance... Management Tools Using Cisco Network Management Tools Cisco offers several network management products that provide design, monitoring, and troubleshooting tools to help you manage your internetwork The following three internetwork management tools are useful for troubleshooting internetwork problems: • CiscoWorks internetwork management software, a set of Simple Network Management Protocol (SNMP)–based... management platforms and build on industry-standard platforms to provide applications for monitoring device status, maintaining configurations, and troubleshooting problems Following are some of the applications included in the CiscoWorks product that are useful for troubleshooting your internetwork: • Device Monitor—Allows the network manager to specify which network devices to monitor for information about... Using Cisco IOS embedded RMON agents and SwitchProbe standalone probes, managers can view enterprise-wide network traffic from the link, network, transport, or application layers The Troubleshooting Tools 2-11 Third-Party Troubleshooting Tools TrafficDirector multilayer traffic summary provides a quick, high-level assessment of network loading and protocol distributions Network managers then “zoom in”... problem using the procedures outlined in this chapter, collect the following information for your technical support representative: • ROM images (Use the show version exec command.) Troubleshooting Hardware and Booting Problems 3-19 Troubleshooting Hardware • Programmable ROM labels (This information is printed on the physical chip, and an example is shown in Figure 3-1.) Figure 3-1 An Example of a Boot ROM . more manageable. Audience Internetworking Troubleshooting Handbook is intended for network administrators who are responsible for troubleshooting internetworks. their network is also essential. Document Organization The Internetworking Troubleshooting Handbook providesthe information necessary to troubleshoot many