The upgrade to CentOS 4.3 went well tonight, 133 packages at 95MB and I have didn’t have to reconfigure a thing. This is the only Linux distro I have ever used that made the upgrading process so easy. Some people complain that CentOS is not cutting edge, but who really wants cutting edge on a server. No for me stability and longevity on a server is what being an admin has taught me are high priority issues.
Walking down the hall of the office I see one of the guys that works on the production floor standing in the hall. Thing is he is not dressed in his normal blue jeans t-shirt dirty environment work clothes, he is dressed like he is going to Sunday church. I stop to ask him why he is all dressed up and he tells me he took a job as a sales person at another company. I make nice chit chat and congratulate him and say good luck shaking his hand then I make a dash to the HR department.
HR has a tendency to not tell IS when someone is coming or leaving the company so we are always ticked when we get a call that they are trying to log someone in to a computer and want to know the username/password but have forgotten to ever tell us this person is coming. What is worse is when we find out that someone has been gone for a week or two and HR still has not told us to disable this person’s account. The guy who I saw in the hall is someone you would call a disgruntled employee. He has been caught twice now running a home business from the computer at work during his shift, some less then stellar emails (I am being nice here) and some other thing that while we were never able to directly connect him to the event, it bore all the markings of his MO. If anyone needs his access turned off it is this guy. So I talk to the manager of HR and she says she is fully aware that he was going to try out another line of work on his two week vacation, but since he has not turned in his resignation, to leave his account on. My heart sinks. Not that I dislike this guy, to me he is a very likeable character, but I don’t trust a disgruntled employee with his full account. The only thing holding him back in the past was the chance that he might get fired. Now that he has gotten another job, do your damage because what is the worst the company can do, you already have another job.
I guess all I can do now is be vigilant and hope he turns in his letter of resignation soon.
**UPDATE**
At a 9:43AM EST he emailed someone else in the company that he had officially resigned. HR still has not contacted IS to disable the account, but is it now disabled.
Got a page at 5am this morning that a server went down in our remote office. Then a little while later get a call from that office. I have the person go to the room that has our rack of servers. It is one of the Liebert with the built in ECM cooling units, one on top one on bottom. Apparently the one on top let loose and water came cascading down upon the servers. The email server was not so lucky and was fried. I hop in the car and start down to the remote office some 2 hours away, stop by the main office to pick up a little DL-140 that we don’t use to setup as a new temporary mail server.
Once I get to the remote office I go to the rack and there is water every where. The top server is still running even though there is water all over the top of the case. The second server down is dead. I pull that server out on the rails and the water is just pouring out of the server like a water fall. I take it into the design lab and start taking it apart and drying parts hoping that once it is dry I could start the thing backup and running. After a few hours of delicately dissecting and drying then finally putting it back together, I fire it up. Nothing. All hope is lost.
Fine, so I take the DL-140 and start getting it installed with Windows 2003 and Exchange 2003. It is about time for the repair guy to show up and start working on the AC unit. I don’t want the servers in the rack if the top AC unit is full of water and ready to spill on the rest of the servers. So I tell people that I am taking down the server and all the boxes go down. Unrack them and take them to the design lab where I will setup a make shift server room. The ML-530 weights 160lbs and makes for a long difficult trip. I get everything backup and running. Moving the tape library I apparently tip the device too much and knocks tapes out of the holding bins in the way of the robotic arm. I don’t have the key with me to open up the front of the library to move the tapes out of the way. So I unscrew the drives from the back of the library and remove the tape drive and shove my hand in to move the tapes back into place. I can’t reach all the tapes but hope that it is good enough so that I can start the restore from last night’s back up and get the new mail server on the 140 up and running. No luck, the tape library is out of commission. At this point the only option I have is to pull the hard drives out of the water logged DL-380 and take them back to GR where I can mount them in another DL-380 and hopefully pull the email off the server and get it on the new server.
Looking at the once water logged DL-380, for what ever reason I decide I am going to try the power on it one more time. Put power to it and hit the on button. It fires up! I am absolutely giddy with excitement. I call my boss and tell him “It’s alive!”
“What’s alive” slight pause then “The mail server!?”
At this point we are both celebrating and dancing for joy.
Now at this point we don’t trust this server and are going to junk it. Breaks my heart to toss out a $20,000 server but with all the water corrosion marks on the motherboard and the fact the hard drive were wet, this box is no longer ready for mission critical assignments.
The repair guy told us not to put anything in the rack for about 3 days and to make sure the water problem does not come back. I went to Menards and spent $11.32 for two 3/8 ID hoses and a 4 gallon bucket to make sure the drain plugs are far enough away from the server that this does not happen again. In hindsight if we would have done this to begin with $11 dollars would have saved $20,000. Ouch.
Now I am finishing moving the remote office mailboxes over to the main office mail server in a different storage group. The email will just have to be picked up over the WAN now. Once that is done I am heading home. It has been a long day and I stink. I am wearing the same clothes I had on yesterday because they were the easiest to grab and in my haste this morning to get out the door they were worn. Now with a good layer of sweat and two days use, I did not have time to shower this morning, my hair is nasty and I am rank to the point that I think I stink badly.
Bought two Infrant Ready NAS 1000S devices yesterday. Since our preferred vendor can’t get these devices I bought them on my credit card. Nothing line a $5000 purchase I need to expense. Accounting is going to love me.
Went with these devices because of some of the features it offers and the price could not be beat for a low end NAS device. The OS is kept on a compact flash card so you don’t have to worry about losing the OS to a hard drive failure and if you really screw it up and you can always get a new CF card from the manufacturer. Hot swappable SATA drive and hardware RAID. You can attach a USB hard drive to it and the NAS device just mounts it as another volume you can access through the NAS. It has snapshot level backups for the data stored on the device to keep backups really small and quick and makes file recovery a breeze. Allows for quotas, security, and you can remove two drives from the device without destroying your RAID array.
What we are going to use it for is all the software that we install in the environment is kept on a couple of small really ancient Linux servers. All this software takes up some 68 GB of space and will be moved to this NAS device. They also fully support Veritas B2D functionality and so we will use this as a near disk backup solution for quick backups of data.
Once I get them in place I will let you know how they work out.