cisco bluesnet troubleshooting WLANs with centralized controllers

AP Join Troubleshooting First, the AP must hunt for the IP addresses of possible WLCs to join WLCs, to find out which ones are alive Then the AP picks the best WLC and tries to join it

Troubleshooting Wireless LANs

Troubleshooting 101

Problem Definition

will make life easier for the TAC engineer and you

Good: client with an X card running Y driver version has issues authenticating to a WLAN on WLC version Z with WPA2-AES using MS-CHAPv2 due to no EAP-Identity-Request from AP

Bad: client can’t connect to network

What Is Needed:

(multichannel), Wireshark, etc

Grep/editor tools

Basic Concepts

Steps to Building an 802.11 Connection

1 Listen for Beacons

y

CAPWAPP Changes (5.2 and Above)

5246 for control

5247 for data

Radio Resource Management Refresher

Selects channels for the radios to use Responds to interference

Reduces radio power, to ensure that each radio hears exactly three others at or above the tx-power-thresh value Can adjust for missing radio, increasing power

power, to compensate

Getting It Right

Build Out Your Infrastructure

Wired Network Requirements—

LAG Can’t Reassemble Fragments

from Multiple Ports

No IP fragments < 32 bytes (CSCsh96186)

All fragments of any IP datagram must arrive on the same port

src-dst-port will lose

balance packets into different LAG ports

4404 Subsystem

Link Aggregation Bundles

4404 Subsystem

WiSM

4402

Link Aggregation Bundle

4404

Link Aggregation Bundle

Wired Network Requirements—WLC to

WLC Mobility Tunnel (EoIP Path)

EoIP path (as of 4.2.61.0)

Encapsulation

ip tc p a

dju st-m

ss 1 300

Radio Resource Management—autoRF

–60 to –80 dBm—lower values for denser installations)

Thresh

–68

Thresh

–73

Radio Resource Management—Detune CHD

Shrink coverage threshold (e.g., to 6 dB) Boost min clients (e.g., to 5)

Networks”, Document ID 71113, cisco.com

I Can’t Hear this Client too Well—

Better Boost

Or use config catcher, available on same download folder

of AP RF info

http://supportwiki.cisco.com/wiki/index.php/

WLC_Config_Analyzer

WLC Config Checker—Warnings

RF Analysis

RF Analysis

Best Practices—RF

Site survey, in case of doubts, site survey, after

installation, site survey

Same device types Same coverage band Same intended service

and never use with voice

Best Practices—Network

Check multicast routing!

Avoid using application multicast address

Best Practices—Network

For fastest failover, AP ports should be configured for:

spanningtree portfast and switchport mode access for local mode

spanningtree portfast and switchport mode access or

spanningtree portfast trunk and switchport mode trunk for H-REAP mode

Best Practices—Mobility

group name

Best to use symmetric mobility (only option as of 5.2)

Use multicast mobility group if 5.x

Best Practices—Security

And debugging! NTP is the easiest way to automatically synchronize debugs on multiple controllers, sniffer captures, etc.

Best Practices—Administration

XML configuration in 4.2+, however, practice

caution anyway)

At least primary, preferably secondary and tertiary as well

No reason to have to hunt for the AP across your network when you need to check something

AP Troubleshooting

Address assignment Time set at WLC SSC issues

Regulatory domain issues Fragmentation

What has changed

AP Join Troubleshooting

First, the AP must hunt for the IP addresses of possible WLCs to join

WLCs, to find out which ones are alive

Then the AP picks the best WLC and tries to join it

Wireless LAN Controller (WLC)”, Document ID 70333,

cisco.com

WLAN Controller Selection Algorithm

secondary, and/or tertiary controller, the AP will attempt to join these first (this is resolved

in the controller name field in the LWAPP/

CAPWAPP discovery response)

as a master controller

greatest excess AP capacityThe AP Selects the Controller to Join

Using the Following Criteria

Troubleshooting Lightweight APs

(check the DHCP server leases for the AP’s

MAC address)

If the AP’s address is statically set, ensure it is

correctly configured

If pings are successful, ensure the AP has at least one

Check the Basics First

Successful LWAPP AP Join

(WLC_CLI) >debug mac addr 00:0b:85:54:ce:00

(WLC_CLI) >debug lwapp events enable

Received LWAPP DISCOVERY REQUEST from AP 00:0b:85:54:ce:00

LWAPP Join-Request MTU path from AP 00:0b:85:54:ce:00 is 1500,

remote debug mode is 0

Successfully transmission of LWAPP Join-Reply to AP 00:0b:85:54:ce:00

Register LWAPP event for AP 00:0b:85:54:ce:00 slot 0

Received LWAPP CONFIGURE REQUEST from AP 00:0b:85:54:ce:00

Failed LWAPP AP Authentication

(WLC_CLI)> debug mac addr 00:12:80:ad:7a:9c

(WLC_CLI)> debug lwapp events enable

[TIME]: Received LWAPP DISCOVERY REQUEST from AP

[TIME]: LWAPP Join-Request does not include valid certificate in

CERTIFICATE_PAYLOAD from AP 00:12:80:ad:7a:9c.

Set the WLC’s Time

(WLC_CLI) >show time

(WLC_CLI) >config time manual <MM/DD/YY> <HH:MM:SS>

(WLC_CLI) >config time ntp server <Index> <IP Address>

(WLC_CLI) >config time ntp interval <3600 - 604800 sec>

The Number One Reason APs Fail to Join Is

Inaccurate Controller Time

Taking Care of SSCs

(this date is like your favorite beer’s born-on date)

AP authentication

an AP Has Created an SSC, in the

AP’s CLI, Do a #show crypto

ca certificates Any Output Indicates that Your AP’s Got Its Certificates

If Controller Time Is Correct, but SSC APs Fail to Join,

Make Sure:

What if the AP’s SSC Hash Is Missing?

O=Cisco Systems

O=Cisco Systems

<SNIP> Self-Signed Certificate Check </SNIP>

98936bf9c90b30bf3c6bb9a0d7b23668887af49a

* <SNIP> DEBU CTRLR </SNIP>

The Fix:

(WLC_CLI) >config auth-list add ssc 00:12:80:ad:7a:9c 98936bf9c90b30bf3c6bb9a0d7b23668887af49a

In the WLC GUI, Go to:

Security | AAA
AP Policies

Does Regulatory Domain Matter? Yes!

(WLC_CLI) > debug mac addr 00:12:80:ad:7a:9c

(WLC_CLI) > debug lwapp events enable

[TIME]: * spamVerifyRegDomain:6202 AP 00:12:80:ad:7a:9c

80211bg Regulatory Domain (-A) does not match with country (JP)

reg domain -JP for slot 0

[TIME]: DEBU CTRLR spamVerifyRegDomain:6167 spamVerifyRegDomain RegDomain set for slot 1 code 0 regstring -A regDfromCb -J

[TIME]: * spamVerifyRegDomain:6202 AP 00:12:80:ad:7a:9c 80211a Regulatory Domain (-A) does not match with country (JP) reg

domain -JP for slot 1

[TIME]: DEBU CTRLR spamVerifyRegDomain:6210 spamVerifyRegDomain AP RegDomain check for the country JP failed

[TIME]: * spamProcessConfigRequest:1730 AP 00:12:80:ad:7a:9c:

Regulatory Domain check Completely FAILED The AP will not be allowed to join.

AP Join Problem—Path MTU/Firewall

1 AP sends discover packet (small)—gets through

2 WLC sends discover response (small) gets through

3 AP sends join packet (small)—gets thru

4 WLC sends join response (BIG)—first fragment gets through, but…

4’ The second fragment is blocked by the firewall; remedied by

X

1 4

4’

When Does AP Fail Over?

What Happens Then?

APs will failover to other WLCs if the control plane is

interrupted after either:

A missed heartbeat to WLC (sent every 30 seconds)

A non-ACKd control packet Then:

The AP will send five successive heartbeats (each a second apart)

If no reply is received, the AP/WLC path is assumed down and the AP will attempt to join another controller

Get Your APs to Join

allow all APs to join

Make sure all WLCs run the same software version Make sure all WLCs are set to the correct time

Make sure all WLCs have all upgraded APs’ SSC hashes Check for potential MTU Issues

a Lightweight Access Point Not Joining a Wireless

LAN Controller”

states to migrate APs

Differences Between LWAPP

Path-MTU Discovery Not Supported

Has a Robust P-MTU Discovery Mechanism, Can also Detect Dynamic

MTU Changes

Control Channel

Encryption between

AP and WLC

Yes (Using AES) Yes (Using DTLS)

Data Channel Encryption

CAPWAP Troubleshooting—

What Is Different

Everything is encrypted except the discovery message

Sniffer traces now are less helpful Debugs still very important

Join priority

If enabled, APs with higher priority can force APs with lower priority to disconnect from WLC

CAPWAP Troubleshooting differences

dumps/decode logs

(Cisco Controller) >debug capwap ?

events Configures debug of CAPWAP events and state

errors Configures debug of CAPWAP errors

detail Configures debug of CAPWAP detail

info Configures debug of CAPWAP info

packet Configures debug of CAPWAP packet

payload Configures debug of CAPWAP payloads

hexdump Configures debug of CAPWAP payloads

CAPWAP Join

(Cisco Controller) > debug capwap events enable

*Jan 09 05:02:07.952: 00:17:df:a8:bf:00 Discovery Request from

192.168.100.103:41824

*Jan 09 05:02:07.952: 00:17:df:a8:bf:00 Join Priority Processing status = 0, Incoming

Ap's Priority 1, MaxLrads = 6, joined Aps =0

*Jan 09 05:02:07.952: 00:17:df:a8:bf:00 Discovery Response sent to

192.168.100.103:41824

*Jan 09 05:02:18.949: DTLS connection not found, creating new connection for

192:168:100:103 (41824) 192:168:100:4 (5246)

*Jan 09 05:02:19.881: DTLS connection established

*Jan 09 05:02:19.881: DTLS Session established server (192.168.100.4:5246), client

CAPWAP Failure

*Jan 09 07:44:45.781: 00:17:df:a8:bf:00 Discovery Request from

192.168.100.104:41825

*Jan 09 07:44:45.781: 00:17:df:a8:bf:00 Join Priority Processing status = 0, Incoming

Ap's Priority 1, MaxLrads = 6, joined Aps =0

*Jan 09 07:44:45.781: 00:17:df:a8:bf:00 Discovery Response sent to

192.168.100.104:41825

*Jan 09 07:44:55.779: DTLS connection not found, creating new connection for

192:168:100:104 (41825) 192:168:100:4 (5246)

*Jan 09 07:44:56.710: DTLS connection established

*Jan 09 07:44:56.710: DTLS Session established server (192.168.100.4:5246), client

(192.168.100.104:41825)

*Jan 09 07:44:56.710: Starting wait join timer for DTLS connection 0xc332dbc!, AP:

192.168.100.104:41825

*Jan 09 07:44:56.713: 00:17:df:a8:bf:00 Join Request from 192.168.100.104:41825

*Jan 09 07:44:56.714: 00:17:df:a8:bf:00 In AAA state 'Idle' for AP 00:17:df:a8:bf:00

*Jan 09 07:44:56.714: 00:17:df:a8:bf:00 State machine handler: Failed to process msg type = 3 state = 0 from 192.168.100.104:41825

*Jan 09 07:44:56.714: Failed to process CAPWAP packet from 192.168.100.104:41825

*Jan 09 07:44:56.714: Failed to process packet from 192.168.100.104:41825

*Jan 09 07:44:56.715: Disconnecting DTLS session 0xc332dbc for AP 00:17:df:a8:bf:00

Mobility

Mobility—Intra-Controller

Mobility—Layer 3

Layer 3 roaming (a.k.a anchor/foreign)

New WLC does not have an interface on the subnet the client is on New WLC will tell the old WLC to forward all client traffic to the

Mobility—Messaging Flow

When a client connects to a WLC for the first time,

the following happens:

New WLC sends MOBILE_ANNOUNCE to all controllers in the mobility group when client connects (note: if possible, configure multicast mobility to lower CPU load and handoff times)

Old WLC sends HANDOFF_REQUEST, telling the new WLC

I have an entry for this client, here is the client status

Mobility—Messaging Flow—WLAN Anchor

attempting to establish the mobility tunnel

updates client state to mobile

If the client is already anchored to a different WLC, the new foreign will send an AnchorXfer to tell the anchor to transfer the entry from the old foreign

replied by HandoffEndAck by the anchor, indicating the client entry has been deleted

Troubleshooting Clients

Client card problems

Connectivity Issues

Typical problem: client(s) can not connect to the network

policy manager state and status

Connectivity Issues

Remember

client states!

DHCP_REQ: address negotiation issue

Probing: Config/IE Dot1x_req: pending

or failed auth

Dissecting Connectivity Problems

Remember to Reduce Scope

Wireless Sniff Wireless Sniff

Tools—Wireless Sniff

Wireless packet capture is essential for 802.11 support

Wireless Sniff—Some Tips

One separate packet capture per wireless channel of interest—a capture

that scans across multiple channels is only useful for device discovery

Exception: 3-ch airpcap adapters from CACE

Always perform unfiltered captures unless you know there are no RF

issues (need to see beacons, acks)

Configure analyzer to cut a new file every 20–30 MB

Configure analyzer not to display updated packet list during capture

(reduce CPU load and minimize drops)

When troubleshooting a roaming client, will need multiple analyzers

moving in concert with the client under test (put everything on a cart

and get some good exercise)

To capture multiple channels at once, the CACE USB 11a/g adapter is a

good option—put a bunch of them into the same USB hub and get it all

on the same PC

an 11

DHCP

Spectrum Analysis

SpEx Tips

installed, enabled, but configured not to associate

to a WLAN

Spectrum expert cannot identify 802.11 devices (MAC address, etc.) without an 802.11 adapter’s aid

NTP sync your SpEx host!

Always use external antenna

If searching for Interferers, good idea to turn off your

wireless network

Kill your phones!

LWAPP (IOS) AP Debugs

To Collect Debugs from LWAPP Cisco IOS APs, You Can:

AP Debugs—Tips

By default, radio debugs (debug dot11 dot11radiox)

appear only on the console To see radio debugs in your

telnet/ssh/WLC CLI session, use the command

no debug dot11 dot11radio x printf, where x is 0 or 1

Useful radio debugs:

debug dot11 dot11radiox trace print: mgmt keys beacon rcv xmt (beacon, rcv, xmt apt to be extremely verbose!)

Useful LWAPP join debugs:

debug dhcp debug ip udp debug lwapp client {config, error, event, packet}

Tools—Wired Sniff

Wired Capture

(watch out for packets in the wrong VLANs) You may need to

(115200 bps) session

debug client MACaddress

See Logs and Reports section of the ACS User Guide

System configuration -> service control -> full for a

deep dive, but beware memory/disk exhaustion—use

under medical supervision

DHCP Logs

Problems at Probing Stage

Is it configured for the SSID of interest?

Is it (WLC) configured for the SSID of interest?

Do you have RF coverage from this AP? (can you see beacons from it?)

Does it like the IEs that the AP is sending out? Try different crypto settings; disable Aironet extensions; try different

basic rates; etc.

1 Client probes for the SSID

2 Client authenticates/associates in 802.11 to an AP

3 EAP takes place

Association

Authentication Authentication

Association

1 Client probes for the SSID

EAP blah blah blah blah EAP SUCCESS

Pass Keys All Around the Place

802.11 Capture of 802.1X

MS-PEAP (dWEP)

Wireshark Capture of MS-PEAP (WPA2)

Successful 802.1X Client Authentication

<SNIP> Series of 802.1X EAP Requests/Responses </SNIP>

Type 25)

00:13:ce:57:2b:84

debug dot1x events

Failed 802.1X Client Authentication

(WLC_CLI) > debug mac addr 00:13:ce:57:2b:84

(WLC_CLI) > debug dot1x events enable

[TIME]: * dot1x_auth_txReqId:2827 Sending EAP-Request/Identity to mobile 00:13:ce:57:2b:84

<SNIP> Series of 802.1X EAP Requests/Responses </SNIP>

[TIME]: * dot1x_process_aaa:898 Processing Access-Challenge for mobile 00:13:ce:57:2b:84

[TIME]: * dot1x_bauthsm_txReq:465 Sending EAP Request from AAA to mobile 00:13:ce:57:2b:84 (EAP Id 14)

debug dot1x events—Username/Password Failure