I've been troubleshooting some very strange behaviors on our network lately. I suspect some (all?) of them have to do with our fairly old Cisco Catalyst 6500s with Sup2's and Sup1a's in our data center, as well as the dinosaur Catalyst 2948 access switches in our closets. There are times when our monitoring system throws alerts saying it can't ping certain devices. But minutes later, things return to normal. (Don't you just love intermittent problems?) One tool that any good network engineer will consider when dealing with such a problem is a packet capture product such as the ever-popular Wireshark.
When I fired up Wireshark on my desktop computer, I had to filter through the muck to see what was going on. By "muck" I'm referring to the traffic I don't care about, such as the traffic my box is generating, as well as broadcast and multicast. I slowly added more and more exceptions to my capture filter (see below) to narrow the scope of my capture.
not host [my IP address] and not host [directed broadcast for my subnet] and not broadcast and not host 188.8.131.52 and not host 184.108.40.206 and not host 220.127.116.11 and not host 18.104.22.168 and not host 22.214.171.124 and not ether proto 0x0806 [for CDP] and not ether host 01:00:0c:cc:cc:cc [for HSRP] and not host 126.96.36.199 and not host 188.8.131.52 and not host 184.108.40.206 and not host 220.127.116.11 and not host 18.104.22.168 and not stp and not host 22.214.171.124 and not host 126.96.36.199
Once I filtered out enough to see more clearly, I noticed a TON of syslog (UDP 514) traffic destined for another host on my subnet. After scratching my head and consulting with co-workers, I started looking at the mac-address tables (or CAM tables). My upstream switch didn't have a CAM table entry for the mac address of the syslog server. Neither did it's upstream switch. In fact, the Cat 6500 directly connect to the syslog server didn't have a CAM table entry for it.
Checking the timeouts for the CAM table on one of the CatOS switches gave us this:
CatOS-Switch> (enable) sh cam agingtime
VLAN 1 aging time = 300 sec
VLAN 2 aging time = 300 sec
VLAN 9 aging time = 300 sec
VLAN 17 aging time = 300 sec
VLAN 18 aging time = 300 sec
VLAN 20 aging time = 300 sec
VLAN 21 aging time = 300 sec
VLAN 25 aging time = 300 sec