• Site router blocked much of activity• Internal sensor detected initial SYN activity to many hosts Examining tcpdump output for a given hour, a scan appeared of high-numbered port 3277
Trang 1Real World Analysis
All material Copyright Novak, 2000, 2001 All rights reserved
Trang 2Real World Examples
• Introduction to tcpdump
• Writing tcpdump Filters
• Examination of Datagram Fields
• Beginning Analysis
• Real World Examples
• Step by Step Analysis
This page intentionally left blank
Trang 3The final challenge in intrusion detection was to synthesize confirmed anomalies to further
understand the exploit or trends In this section, we demonstrate, using examples, how we delve into the analysis process and come up with a more complete explanation of the activity
Trang 4You’ve Had a Compromise!!!!
• Notification from parent CERT organization of
compromised host
• The notifier only knew:
• IP numbers of the hostile host and
compromised host
• Day and approximate time of incident
• What tools have we learned about that might help
us validate and investigate this incident?
For starters, we need to examine tcpdump output for the time of the reported event to determine out what happened
From there, we can use the new scripts we created if more detail is required
So, tcpdump records were dumped for the site:
tcpdump –r tcpdumpfile ‘hostile-IP and compromised-IP’
where hostile-IP and compromised-IP were the actual IP’s
Trang 55
What a “Normal” TCP Connection
Session Looks Like
• Before looking at output from the tcpdump output from the break-in,
let’s examine typical output from a telnet session
boulder.myplace.com.38060 > aspen.myplace.com.telnet: S 3774957990:
3774957990(0) win 8760 <mss 1460> (DF)
aspen.myplace.com.telnet > boulder.myplace.com.38060: S 2009600000:
2009600000(0) ack 3774957991 win 1024 <mss 1460>
boulder.myplace.com.38060 > aspen.myplace.com.telnet: ack 1 win 8760 (DF)
aspen.myplace.com.telnet > boulder.myplace.com.38060: F 4289:4289(0) ack 92
win 1024
boulder.myplace.com.38060 > aspen.myplace.com.telnet: ack 4290 win 8760 (DF)
boulder.myplace.com.38060 > aspen.myplace.com.telnet: F 92:92(0) ack 4290
win 8760(DF)
aspen.myplace.com.telnet > boulder.myplace.com.38060: ack 93 win 1024
boulder.myplace.com.38060 > aspen.myplace.com.telnet: P 1:28(27) ack 1 win 8760 (DF)
aspen.myplace.com.telnet > boulder.myplace.com.38060: P 1:14(13) ack 1 win 1024aspen.myplace.com.telnet > boulder.myplace.com.38060: P 14:23(9) ack 28 win 1024
A TCP session where connectivity has been established and data has been exchanged has the pattern
of the above records
Namely:
• The connection is established by:
• The client (boulder) sending a SYN to the server (aspen) to the desired port
• The server (aspen) sending a SYN/ACK back to the client (boulder)
acknowledging the request
• The client (boulder) sending an ACK to the server (aspen) to complete the three-way handshake
• Data is exchanged:
• The PUSH flag is set for data transmission
• Actual data bytes are sent (this can be seen before the ACK on records with the PUSH set, you see values in parentheses, i.e ( 27), (13), (9))
• The connection is stopped:
• Gracefully with the two-way FIN exchange
• Aborted with a RESET
Note: timestamps have been removed from the tcpdump records to make the records more readable
Trang 6dns.myplace.com.22 > whatsup.net.25118: R 0:0(0) ack 2035824357 win 0 (DF)
dns.myplace.com is the alleged compromised host and whatsup.net is the hostile IP
What’s wrong with the break-in theory? Well, no connections were established, no data exchanged, and
no connections were stopped after being established
This looks like a typical scan from whatsup.net Occasionally, you’ll see dns.myplace.com respond with a RESET/ACK meaning that the port that is being scanned is inactive Other than that, there is no evidence of any two-way conversation
If no RESET/ACK is seen that means that the traffic never actually made it to the inactive port to be rejected In other words it was blocked by an exterior router The sensor for this site resides outside the filtering router
This, of course, assumes that there is no backdoor entry to the network/host that the sensor does not see
It turns out that this incident was a communication problem among the CERT organizations One reported it as a scan and somehow several phone calls later, it got reported as a break-in Because we had the tcpdump output, we were confident that a break-in did not occur
Had we not had tcpdump or some other software that had an audit trail of activity, we would not have
Trang 7• Site router blocked much of activity
• Internal sensor detected initial SYN activity to
many hosts
Examining tcpdump output for a given hour, a scan appeared of high-numbered port 32775 The sensor that picked this up was inside the packet-filtering device signaling that some traffic got through to internal hosts At this point, all we could see was that there were initial SYN connections attempted We would need to further investigate to discover if there were any responses to these connections
Trang 8What we see above is a site that has a filtering router that blocks some traffic and allows in others Specifically, they allow in some high-numbered ports for certain subnets to accommodate users or system administrators who need to use outside services such as active FTP that might try to open up these ports The sensor collecting the traffic is inside the packet filtering router so anything we see has gotten through the packet filtering device A scan from hostile host 1.1.1.1 was received directed
to destination port 32775
Trang 9We do know that all subnets and hosts that had access rules that allowed access to ports greater than
1023 were scanned That is why it is suspected that the entire class B network was probed Either that or the scanner did some prior reconnaissance and knew that there were some subnets that allowed traffic through to high-numbered ports
TCP port 32775 doesn’t have a well-known service associated with it, but it is in the range of ports considered to be Remote Procedure Call (RPC) services This range is 32770 – 34000 both TCP and UDP So, we begin to surmise that the prober is looking for some sort of RPC service being offered – either one that we know is vulnerable or an entirely new exploit that has yet to be reported
Trang 10Conventional inetd Services
telnet server x
listens on port 23
telnet server q
listens on port 23
telnet server z
listens on port 23
A moment of digression to explain the difference in the way conventional internet services started by the Unix inetd daemon are identified versus the way RPC services must be identified A server can start any given internet service on any port, but convention dictates that if we want to offer a
universal service, we must listen on a standard well-known port For telnet, this is port 23 Almost all servers offering the telnet service will listen on port 23 so the clients will know where to find the telnet port
Trang 11Runs ToolTalk
Listens on port 32773
Runs ToolTalk
Listens on port 32774
Trang 12Portmapper
• Registers RPC services run by a host
• Can query portmapper using command:
• rpcinfo –p hostname/IP
• Uses TCP port 111 to query
• Prudent to block this port on packet-filtering device
• For each RPC service running, query returns:
is listening After gleaning that information, the attack usually takes place
The problem with this approach is that most sites except those that are totally security unaware will block TCP port 111 which is the port at which the portmapper listens for queries If this is done, the only other recourse for the hacker is to scan the ports in the RPC range to see if hosts are listening
Trang 1313
Sample rpcinfo Output
program vers proto port
Since the port at which an RPC service listens may vary, we need a universal indicator to identify a given RPC service for all hosts running that service This is known as the program number In the above output, we see that the RPC service sadmind used for Solaris remote administration has a program number of 100232 It listens on port 32775 on this host Any sadmind service running on any host will have a program number of 100232 This is the number we must key on to identify services
How do you determine associated RPC services and program numbers? Most hosts have a file /etc/rpc that lists the possible RPC services (much like /etc/services for the inetd services) Yet, this
is often not complete for all services that might be run
Trang 14it is most likely some RPC server Additional scanning can be done to determine exactly what RPC service is running on a responding port.
Since the scan made it through the packet filtering device, the concern then became – did any host respond to the 32775 probe If none responded, we could just dismiss this as our good fortune and a relatively harmless scan But if any answered, we have more investigation to do
Unfortunately, many hosts responded with a SYN/ACK indicating that they listened with some RPC service on port 32775 We don’t know exactly what service each host listened for at this point, we only know that something was listening The next step after seeing this was to look at the hosts that responded and see if they pushed (most often the P flag will be set) data from the probed host to the scanner or vice versa Fortunately, this did not occur This indicates that the scanner most likely was doing reconnaissance only at this phase Exchanged data would elevate the level of concern because this means the scanned host either divulged some information or suffered some indignity at the hands of the prober
Trang 1515
What RPC Services Were Running on Port 32775?
rpcinfo -p hostname | grep "tcp 32775"
+ rpcinfo -p host1.net | grep "tcp 32775"
For each of the responding hosts, we want to determine the exact RPC service running on port
32775 This can be done by issuing the command rpcinfo –p hostname and then examining the
output for the string “tcp 32775” The scan looked at TCP port 32775 It is also possible to have an RPC service running on udp 32775 so that is why we make the distinction As we see above, we have three hosts running a service that has no description, but has the universal program number of
100083 This is what we must use to search for the service running on this host The responding hosts ran many different services – all of those had to be identified for known exploits
A best guess is that the scanner is looking for a service that has been successfully exploited recently
At the time of this writing (7-2000), the most popular exploited RPC service has been ToolTalk See SANS write-up entitled “Consensus List of Top Ten Internet Threats” found at
www.sans.org/topten.htm RPC services (specifically citing ToolTalk) are the third listed threat
Trang 16• Very dangerous, very widespread
The CERT advisory can be found at:
http://www.cert.org/advisories/CA-98.11.ToolTalk.html
Here is an excerpt from this advisory:
“The ToolTalk service allows independently developed applications to communicate with each other
by exchanging ToolTalk messages Using ToolTalk, applications can create open protocols which allow different programs to be interchanged, and new programs to be plugged into the system with minimal reconfiguration
The ToolTalk database server (rpc.ttdbserverd) is an ONC RPC service which manages objects needed for the operation of the ToolTalk service ToolTalk-enabled processes communicate with each other using RPC calls to this program, which runs on each ToolTalk-enabled host This
program is a standard component of the ToolTalk system, which ships as a standard component of many commercial Unix operating systems The ToolTalk database server runs as root
Due to an implementation fault in rpc.ttdbserverd, it is possible for a malicious remote client to formulate an RPC message that will cause the server to overflow an automatic variable on the stack
By overwriting activation records stored on the stack, it is possible to force a transfer of control into arbitrary instructions provided by the attacker in the RPC message, and thus gain total control of the server process.”
Trang 17We cannot be absolutely sure that the scanner was trolling for reconnaissance about hosts that might
be running ToolTalk But, we see enough evidence on this network that if the scanner returns to do some damage, it is very possible that we have unpatched versions of ToolTalk that might be
susceptible to this exploit There could be other RPC services running on other hosts that have exploits associated with them such as sadmind
The immediate reaction was to block this port Granted, this is a stop-gap solution for this particular event The hacker can return and probe other RPC ports and probably find other ports that listen and serve up ToolTalk A more all-encompassing solution would be to block inbound all the RPC ports
32770 – 34000 The conservative approach until more investigation is done is simply to block this specific port
Trang 1919
Port 31789 Data
• Curious only 1 byte of UDP data sent
• Why not send more, or
• Why send any at all?
• What is this curious one byte?
• Examine records in hex and using tcpshow
• What scripts can we use to check out this one
Trang 20Examining Port 31789
Records in Hex
• Execute following command:
tcpdump –r tcpdumpfile –x ‘dst port 31789’
12:13:10.570000 1.2.3.4.31790 > 192.168.21.63.31789: udp 1
4500 001d a932 0000 7311 10cf 0102 0304 c0a4 153f 7c2e 7c2d 0009 38b1 413b 0000
0001 0000 0000 0000 0356 4752 0341 12:13:10.570000 1.2.3.4.31790 > 192.168.21.90.31789: udp 1
4500 001d c432 0000 7311 f5b3 0102 0304 c0a4 155a 7c2e 7c2d 0009 3896 41cc 8000
Trang 21UDP Header UDP Data
This is the first record dumped in hex You’ll notice 18 bytes of UDP data
The IP header encompasses the first 20 bytes of data The UDP header follows with 8 bytes of data The UDP data follows these headers One data byte consists of two hexadecimal characters such as the 41
Trang 22Examining Port 31789 Records With tcpshow
• Generate tcpshow output:
tcpdump –r tcpdumpfile –enx ‘dst port 31789’ | tcpshow
-nolink
In an attempt to discover anything more about the traffic bound for port 31789, we use tcpshow to interpret the output for us We can dump the tcpdump data using tcpshow to interpret the output In this case, we search for the pattern of destination port 31789
Trang 23TTL: 115 Encapsulated Protocol: UDP Header Checksum: 0x10CF Source IP Address: 1.2.3.4 Destination IP Address: 192.169.21.63
The above tcpshow interpretation of the IP header shows nothing unusual Note the 29 byte datagram length
Source Port: 31790 (<unknown>)
Destination Port: 31789 (<unknown>)
Datagram Length: 9 bytes (Header=8, Data=1)
Trang 24UDP Data
A -
Packet 2
Look at the UDP datagram length derived from the UDP header length field It assumes that since that UDP header says it is 9 bytes long (8 bytes for header and 1 for data) that there is only one byte
of data to interpret - so it decrypts the first byte to A
You’ll also see a “Packet 2” Chances are that this is the additional data bytes that the tcpshow command doesn’t know how to interpret
Sometimes, as you can see, the hex output is more accurate because it reveals the actual bytes tcpshow doesn’t always interpret unconventionally crafted traffic correctly
What this mutant packet shows is that the UDP header was probably crafted Again, the reason is unknown; it is worthwhile noting that this probably is not for benevolent purposes It could be an attempted denial of service against a particular operating system that will not be able to handle this packet Or, it can be an attempt to send a covert message – one that is not to be discovered by anyone other than the intended recipient
Trang 25IP datagram total length UDP length
The IP datagram total length agrees with the crafted UDP header
because it is 29 bytes (20 IP header + 8 UDP header + 1 UDP data)
The UDP length has been crafted to be 9 bytes (8 UDP header + 1
Trang 26Examining the Ethernet Frame
• Dump Ethernet header to see length:
• The following command was run:
tcpdump -r tcpdumpfile -n -e ‘dst port 31789’
12:13:10.570000 0:60:5c:53:96:a8 0:e0:4f:65:98:0 0800 60 : 1.2.3.4.31790
> 192.168.21.63.31789: udp 1
4500 001d a932 0000 7311 10cf 0102 0304 c0a4 153f 7c2e 7c2d 0009 38b1 413b 0000
0001 0000 0000 0000 0356 4752 0341
Ethernet frame length
As a final check into the “wholesomeness” of the IP datagram, the Ethernet header was dumped to determine what it believed the length to be
The Ethernet frame length is 60 (decimal) bytes long This reflects the true length of the Ethernet frame (not including the Ethernet trailer of 4 bytes)
14 Ethernet header bytes + 20 IP header bytes + 8 UDP header bytes + 18 UDP data bytes = 60 bytes
We can conclude from this that the Ethernet length was not unnaturally altered
Trang 2727
Time Exceeded in-transit
router1 > 192.168.234.0: icmp: time exceeded in-transit [tos 0xc0]
router1 > 192.168.126.0: icmp: time exceeded in-transit [tos 0xc0]
router1 > 192.168.17.0: icmp: time exceeded in-transit [tos 0xc0]
router1 > 192.168.113.0: icmp: time exceeded in-transit [tos 0xc0]
router1 > 192.168.100.0: icmp: time exceeded in-transit [tos 0xc0]
router1 > 172.16.118.0: icmp: time exceeded in-transit [tos 0xc0]
router2 > 172.16.125.0: icmp: time exceeded in-transit
router2 > 172.16.145.0: icmp: time exceeded in-transit
router2 > 192.168.42.0: icmp: time exceeded in-transit
router2 > 172.16.223.0: icmp: time exceeded in-transit
router2 > 172.16.182.0: icmp: time exceeded in-transit
router2 > 172.16.125.0: icmp: time exceeded in-transit
router3 > 192.168.101.0: icmp: time exceeded in-transit [tos 0xc0]
router3 > 192.168.206.0: icmp: time exceeded in-transit [tos 0xc0]
router3 > 192.168.160.0: icmp: time exceeded in-transit [tos 0xc0]
router3 > 192.168.152.0: icmp: time exceeded in-transit [tos 0xc0] A
Just before the change of the millennium, sites everywhere began to see a flood of ICMP time exceeded in-transit messages Theories abounded, the first one was that we were being attacked by the routers sending the ICMP in-transit messages Then, after examination, it seemed more
reasonable that the routers delivering these messages were just routers that were in the paths when the TTL dropped to 0
Then, there were theories that this was perhaps the effects of routers being shut down to protect against the Y2K threat When that theory was investigated, this didn’t seem the appropriate behavior for routers that were turned off and this continued for days Router convergence shouldn’t take days
Another clue that this wasn’t your normal garden variety ICMP activity is that all the destination addresses the routers were informing of an ICMP error are of the address x.x.x.0 These appeared to
be spoofed since the sites receiving these messages sent no such traffic The mystery has yet to be solved!
Trang 28In the green, you see the block set aside for the embedded protocol information Specifically, what should be included in here is the IP header of the datagram that caused the error, plus 8 bytes of the embedded protocol in that datagram This can be used for forensics or diagnostics, if need be So adding all the bytes up, remember that this ICMP message is embedded in its own IP header which has minimally 20 bytes, then we have 8 bytes for this ICMP header and zero-filled bytes, followed by 20 bytes of the embedded IP header and 8 bytes of its embedded protocol This means we need to collect
56 bytes of the datagram
Now, recall we automatically collect the link-layer header, that is Ethernet, then we are getting 14 bytes for the Ethernet frame header + 56 required bytes So we need to collect 70 bytes of snaplen to see all the data The default snaplen used by tcpdump may only be 68 bytes so we may miss the final 2 bytes
of the embedded protocol
Trang 2929
ICMP Embedded Message
15:10:26.100000 1.1.1.1 > 1.2.3.0: icmp: ip reassembly time
An ICMP reassembly time exceeded message means that a destination host didn’t receive all the expected fragments in a certain time This will prevent a destination host from waiting indefinitely for data that may never be received
Unfortunately, tcpshow does not attempt to decrypt the partial embedded datagram in the ICMP message
Trang 30Current IP Header and Protocol
15:10:26.100000 1.1.1.1 > 1.2.3.0: icmp: ip reassembly time
Embedded protocol = ICMP
ICMP type = 11 ICMP code = 1
Now, let’s break down what we find in the current IP header and embedded protocol We could use tcpshow to do this, but this will be in preparation for decrypting the embedded datagram that tcpshow does not attempt to interpret
In the IP header we see the first field of interest is the protocol field which indicates that the embedded protocol is ICMP which is what we would expect Looking at the ICMP message header
in red, we break that down into bytes and fields We see that the first byte of the ICMP message is the ICMP type which is 11 ICMP type 11’s are known as time exceeded messages A code of 0 specifically means that this is a reassembly error
Trang 31Embedded UDP header
Embedded protocol = UDP
UDP source port = 13236
UDP dest port = 8548
More frag =1
Dest IP = 1.1.1.1 Source IP = 1.2.3.0
Now, let’s dip down into the embedded datagram found after the ICMP data We see this in green
To interpret all the fields, you need to acquire a diagram of the IP header and embedded protocol format which is UDP Looking at a few pertinent fields, we see that the more fragments flag is set meaning that more fragments are expected after this This makes sense given the error message was due to a fragmentation error Next, we discover in the protocol field a hexadecimal 11 which is a decimal 17 which translates to UDP So the embedded protocol in red is expected to be UDP The original sending source IP was 1.2.3.4 that attempted to connect to 1.1.1.1
When we analyze the UDP header, we see the source and destination ports These are not known ports so we can’t make much of a guess about what the original source host attempted to do And finally, we are supposed to get 8 bytes of the embedded UDP protocol, but because our snaplen for data is only 54, we are deficient 2 bytes
Trang 32Netbus Scan
• Massive scan to destination port 12345
• Typically associated with Windows netbus trojan
• Targeted network had some open access to high-number
ports behind the packet-filtering device
In the next incident, we examine a scan to destination port 12345, that is typically associated with the netbus trojan that affects Windows hosts This particular scan was launched against a Class B subnet
so it set off all kinds of alarms The network that was scanned had some high-numbered port access open through the packet-filtering devices