Snowmageddon vs. The Corporate Network

A major winter storm can make for some very interesting statistics. Let's look at the primary firewall for Company XYZ, also used for remote access VPN.  We've got a failover pair of Cisco ASA5510s licensed for 100 simultaneous AnyConnect WebVPN connections as well as 750 IPSEC VPN connections. Our "road warriors" are set up with the IPSEC VPN on their laptops, but folks who work from home using their own personal computers usually come in using the AnyConnect WebVPN (SSL-based).

You can see from the IPSEC VPN Connections chart below that we apparently have about 80-100 "road warriors" that just keep their home computers connected all the time (based on the lowest number of connections each day).  Over the last week we've peaked around 160-180 except for today, which has taken us up close to 200. One of the reasons for this is because of the next chart.

The WebVPN Connections chart below shows on most days we have up to 30 connections at our peak times. Since the sky opened up and dumped snow on us overnight, you can see that we've more than maxed out our connection limit for WebVPN.  For days like this, our WebVPN page has a message that says something like "If there is inclement weather today and you are having problems connecting, there may be too many other people trying to connect at the same time.  You may connect using a different method, by downloading an alternate VPN client using the appropriate link below." Then there are links for 3 .zip files: Windows XP/2000, Windows Vista/Win7, and Macintosh.  Each zip file contains the Cisco IPSEC VPN client EXE as well as two PCF files that provide limited-access profiles for the IPSEC VPN.  

Unfortunately, there doesn't seem to be any nice error message that says "no more connections available" to indicate a user is running into a connection limit. Is there some way to do that I don't know about?

The chart that got all this analysis started this morning also generated an e-mail telling my team the ASA VPN appliance was running high on CPU.  (Well, the chart didn't generate the e-mail--the network monitoring system did.)  Take a look at the following Average CPU Load and you'll see we're running about 80% today vs. a typical day at or below 60%.

The next chart shows the bandwidth impact all this VPN traffic has on our DS3 circuit. The green line shows uplink to the Internet and is peaking close to the 45Mbps mark today. I wonder how many of those users are RDP'd to their desktops and the screensaver has kicked in, causing high bandwidth utilization. *sigh*

In case you're wondering, all these graphs were pulled from Solarwinds Orion Network Performance Monitor (NPM). In particular, the first two charts showing connection numbers utilize Orion's Universal Device Poller (UnDP) funtionality. There wasn't any built-in way I could find to measure what I wanted, so I found ideas on Thwack.com (Solarwinds' user community site) to use SNMP polling via UnDP to get those numbers. 

So who's winning the battle...Snowmaggedon or The Corporate Network?  You decide!  Let me know on Twitter (@swackhap) or in the comments below.