ARP Poisoning in a Production Environment

Here is a sanatized email the I sent last night that has a very interesting problem that we ran into at work. Never ran into ARP Poisoning before...
-------------------------

Sent: Thursday, April 16, 2009 12:21 AM
To: XXXXX XXXXX
Subject: ARP Poisoning

Ok, so here is 12 hours of work boiled down into a couple sentences of “what happened” …

Basic Topography: 10.70.0.1(hop1) > 10.50.0.1(2) > 10.50.0.9(3) > Internet(4) > Destination(5)

MAC Address for 10.50.0.1 = XX:XX:XX:XX:b4:23

Scenario: We isolated the issue (mainly by completely replacing 10.70.0.1 (ISA 2006) & 10.50.0.1 (Core Router) to no avail) so that we knew that traffic was going from 10.70.0.1 out to the internet, hitting the destination and generating response traffic. This response traffic made it to the firewall but died before reaching back to the 10.70.0.1 ISA server. This leaves 10.50.0.1 and 10.50.0.9 as the only possible culprits for the missing traffic. After replacing 10.50.0.1 we discovered that the traffic still exhibited the same behavior and we realized that the chances of 2 Routers both being bad was really remote so we focused on the firewall. Taking a packet trace with the network up and another when the network was dead we found a very subtle difference in the packets. While the network was operating normally the packets were flowing from the firewall to the core router using the level 2 routing address’ of:

Src: XX:XX:XX:XX:ea:b0 - dst: XX:XX:XX:XX:b4:23
When the network was broken the level 2 flow of inbound packets was like:
Src: XX:XX:XX:XX:ea:b0 - dst: 00:0c:29:a9:d2:25

So what we have at this point is ARP Poisoning where another machine on the 10.50.x.x is impersonating 10.50.0.1 which is the Core Router; the result of this is that all traffic coming inbound from the internet (hop3 > hop2) was getting redirected to the mystery machine (hop3 > hop_blackhole) with the mac of 00:0c:29:a9:d2:25. Going from switch to switch we tracked the mac to MachineNameX. After unplugging the machine from the network and clearing the ARP cache on the firewall traffic immediately started working normally and the network is happy. Check out the cool wireshark screenshot attached… now you know what ARP poisoning looks like.

Bottom line: Wireshark is awesome, the 10.50.x.x switches have a command “show bridge address-table” which shows you the mac address’ that are associated with each port on the switch, 2 heads and 4 eyes are better than 1 head and 2 eyes… and sleep is overrated.. :)