Friday, April 29, 2011

Dead Servers, Dead Hard Drives, Dead Video Matrices....

Geez....I'm surround by DEATH at work!
Well, defunct hardware anyway.....
18 months ago, shortly before I was "released" as a contractor from the site I was assigned to, we came up with a very comprehensive shut-down procedure, and a set of minimum environmental control standards to be maintained after the equipment was powered down. Some of it happened, and some of it didn't. We knew the server/communications room was apt to get too hot, and we even had a portable air conditioner set up, with its exhaust directed into the air return for the ship's HVAC system. It kept the room at a nice 65 degrees, a welcome change from the times it got over one-hundred degrees in there when something went wacky with the ship's HVAC system.
WELL....we never finished shutting everything down, as each round of layoffs (that's really what they were) left fewer and fewer qualified people to work the procedures. It turns out that some of the servers were left operating because nobody was around to properly power them off, BUT because the air conditioning and HVAC were just soooo easy to turn off, that's what happened.
And none of the more obscure video equipment used to route the video sources, and run the Big Screens, got shut down.
And these things run hot.
Last week we started the process of bringing things back up, and some things just didn't turn back on, either at all, or properly. Bringing the servers up and examining their logs showed that whoever had shut the down basically just "pulled the plug".
I was able to get the systems we needed up and running so that we could do some testing, but I've got two application servers that have wonked out hard disks in them, and a couple of the others are throwing errors.
My suggestion will be to image the hard drives in the known good ones, replace ALL the hard drives in the 8 "mission critical" machines, and then re-image them with the good image.
You do NOT want an applications server to go down at T minus 10 minutes!
The video gear mostly has fried power supplies. I pulled a supply out of one of our good shelf-spares, and used it to confirm that was the problem with all the "failed" equipment. The supplies are being used at about 85% of their capacity (850mA from a 1 Amp supply), which I consider borderline for long-term reliability. Eventually we'll be upgrading all the "Big Screen" stuff, but for now we'll just lay in a good supply of the little power modules inside the boxes, and live with it.
The company that originally made these boxes wants $450 to "repair" them, and since the supplies can be had for about $20 shipped, it's a good way to save the reconstituted enterprise some money.
And we haven't even started on the RF systems and Weather Radar yet. The next four months should be a lot of fun.....

On the home front, I've been trimming trees all weekend, and *finally* got the lights on my little tower trailer wired up so I can haul it down to the DMV and get it licensed. Field Day is coming up the last weekend in June, and I need to get plates on it so I can tow it down to the Field Day site.

4 comments:

  1. There is a pretty good chance that the Damsel and I will be there that weekend. Maybe we can repeat our visit from last year and check out the new stub tower trailer. No guarantee, but it would be nice to hook up again.

    ReplyDelete
  2. You guys are always welcome to come by.
    It's not a "stub" tower anymore. I extended it out to 9 feet!

    ReplyDelete
  3. In other words, you're busier than a one legged man in an ass kicking contest :-)

    ReplyDelete
  4. Yeah, and the new owners are very "cost conscious".

    ReplyDelete

Keep it civil, please....