Thursday, March 4, 2010

The Way Of The Dinosaur


It's been a
long time. I can't remember how long, and I'm too lazy/busy to look it up. But somewhere around two (yep, count 'em, TWO!) years ago we had a major problem at work. One of our Cisco Catalyst 6509 core Ethernet switch had major problems. Turns out we had some bent pins on the backplane in slot 2. In laymen's terms, the place where you plug the brains into the switch was broke. We still had one "brain" (a.k.a. supervisor module) but the redundant one couldn't be used. The only solution to get our redundancy back? Replace the whole chassis.

Replacing an entire switch chassis is NOT a small job. There were literally hundreds of servers connected to this switch in the data center. So we set out on a very. long. journey. We got a replacement chassis from Cisco and sloooooooowly began moving one server network connection at a time from the old switch to the new switch.

Fast forward to today. Thanks to a big push in the last few days by some coworkers and me, we currently have only 7 more connections on this switch. And if things go according to plan, they'll all be changed to the new switch by Saturday afternoon. (Yeah, I have to go to work on Saturday. And it's supposed to be nice weather, too! Bummer...)

Some might not see the significance of this accomplishment, but those of us that have worked on it over these many months are psyched! We've scheduled a ceremonial power-off ceremony for Monday afternoon. Two of us will switch off the dual redundant power supplies, and everyone present will have the opportunity to disconnect one of the many ancient RJ-21 Ethernet cable connections. It will be stupendous when this switch makes itself extinct, and we can go on with our other more exciting, less mundane, projects.

1 comment:

  1. We had this issue at work today; we had a vs-s720-10g-3c come in (retails around £5,000 refurb)put in our test chassis and it was totally dead. no LED no console, zero. So put in chassis 2 again nothing, presumed card was DOA. We were struggling for answers as our supplier is one of the best there is. Looking at the amphenol connector on the card noticed very slight disconnect at far left near power connector. We then peered inside and realised both of the chassis had damaged pins . Supervisor has gone for repair, waiting for verdict on fixability and we corrected the pins in the chassis with a just a screw driver (plus torch). Its easy to do you just have to be patient and use a gradual technique. Focus on the worst pin and just give it a tiny prod with the screw-driver and note the amount of movement. Keep correcting the worst pin until they are very closely aligned to the position. When you insert the next card in the slot it will correct everything 100%. This is why you should never use above a certain level of force when inserting a line-card, as you are in effect pushing against a long row of delicate pins.

    ReplyDelete