The Mixed Blessing of System Failure

ok, so here’s the scoop. I was planning on switching unicron over to CentOS from Ubuntu, but during the prep process, I must have bumped a sata cable on my raid array- the result was me rebooting, rebuilding the drive and shaking it off as a fluke.

A day later, that drive and another failed, which is bad news on a 4 disk raid5 array- usually it means everything is lost.

By some minor miracle, I was able to restore the array, but in the process installed cent into one of the two 1-gig swap partitions (formatted to ext3). From there I was able to make sure the data was backed up.

I had already created the 20 gig partition I planned to install CentOS to (it was unavailable during the raid failure, hence I didn’t use it), so I figured, “what the hell, the server is down I might as well rebuild it now.”

It’s a pain in the ass, but in the end it forced my hand to rebuild the server. The hardest part is remembering all the functionality I’ve set up over the last 2 years. Here’s what I’ve done so far since Friday:

• reinstall
• fix funky ext3 partition (/share drive)
• install screen, irssi, sudo which, lsof, lsusb
• install bind, dhcp
• configure bind, dhcp
• activate bind, dhcp
• shutdown linksys dhcp
• modify iptables to let traffic through
• install apache, mod_ssl
• reconfigured apache, based on work config
• imported mysql DBs (including wiki)
• reinstalled wiki
• setup ssl certs
• set up unrealircd
• ssh keys working
• configure sshd
• set up ldap
• implement ldap into system accounts
• get ldapadmin working
• setup snmpd

I’ve done quite a bit more, but this is just what I remembered to document so far. What’s left?

• get subversion working
• set up myphpadmin
• set up /share share
• set up backups for LE websites
• set up nessus
• set up nagios
• set up nagiosgraph
• recreate ldap accts
• set up netflow/ntop

I’m sure this list will grow over the next few days.