This article was originally written on July 19th, 2010, but never published. Documentation is another topic where there appears to be disagreement in the sysadmin world. When to document, what to document, who do document for, and where to store that documentation always seem to be subjects of contention. Everyone likes documentation, but no one has the time to document, and the rules for documentation often feel arbitrary. I’d like to open this up for discussion and figure out some baselines.

This concept is something I’ve carried around with me for my last 3 jobs, and since I’m writing it up for my current employer, I figured I should document it here as well. I’ve mainly worked in Linux/Windows environments, so you may sense a bit of bias away from older systems. It’s not intentional, just a result of my experience. Thanks to Mick for introducing me to this schema. The purpose of this documentation is to provide a clean-cut and straight-forward convention for naming servers.

We recently ran across a problem in production that we could not replicate in lower environments. Since this is not only a high use application, but an exceptionally “chatty” app, searching the logs was an excersise in futility (*one* of yesterday’s production logs was 6,975,291 lines long, with multiple logfiles per app, multiple apps and multiple servers). So how do you find a needle in the haystack? Get a smaller haystack.

My employer is currently looking for a sysadmin. If you’re interested, contact me for details. SR SYSTEMS ENGINEER ROLE IN FARMINGTON HILLS, MI Summary: We are looking for someone who will administer web hosting Linux systems infrastructure, including server hardware, operating system, enabling software, and application software/data for Internet-facing application systems. Direct other departments’ work on dependent systems such as network, firewall, load balancer, and external storage systems. Provide consultative expertise for our businesses to provide technical guidance, standards, knowledge and understanding of business and technology processes, and integration of technologies to deliver Internet-facing learning products and services.

New job, new laptop. Many utilities here are windows only, so it requires a bit of… effort… to get myself up and running efficiently. The solution to the windows problem is VirtualBox. I had set this up on my last laptop with little effort, but this time around required a bit more effort. Hopefully the instructions below will help others get up and running quickly. Disclaimer– your laptop may catch on fire and explode (or worse) if you attempt this… or something.

As a system administrator, monitoring is a key job responsibility, yet arguments seem to arise on how to implement it (usually with people who won’t be paged at 3am). Before writing this, I looked around for an article on the goals and philosophy of system monitoring, but found very little that really applied to this topic. Hopefully this will help set some expectations for admins, managers and stakeholders on what you should monitor, and why it should be monitored.

So, I’m home sick again. Fourth time this year I’ve been sick. Cold, flu, cold, Bronchitis. Awesome. Will I get to rest today? No, of course not. My server (Unicron) has been up and running for 2 years now- I got the parts right after Ian was born. I set up a nice software raid array at the time that’s served me well. I’d never set up a raid array like this before, so I wasn’t really sure how to monitor it.

So I’m testing a nifty new wordpress plugin… check this out: I wonder if it’ll work? update: no, no it will not.

So I’ve been pretty quiet since I hit 100k words- what’s been going on? Round of layoffs at work Friend diagnosed with cancer Another round of layoffs at work. Jackie became a pampered chef consultant Finances have been wiped out from christmas and getting her PC stuff off the ground. 10% paycut at work Guitar lessons are now done because no one can afford them. Have been reading Manuscript Makeover for ways to improve my book Decided to do an initial cleanup of the first draft of my script, then rewrite the outline before starting draft #2 started yet another opensource project- this time it’s a collection of Nagios Plugins.

anyone know of any good jabber clients for the blackberry? I’ve tried a couple with little luck, and most of them cost more than I can afford for this test. Features required Must run on BlackBerry 8703e v4.1.0x Connection server can be configured differently than jid address (i.e. you@morgajel.net for jid, jabber.morgajel.net for connection server.) This rules out Mobber as far as I can tell Requires SSL/TLS Non-strict cert checking Let me know if you have any suggestions.

My internet connection. SO here’s the scoop 5 days until cutover: I call AT&T, tell them I’m moving and need to transfer my Static IP DSL service on the 30th(Monday). Tech says no problem it’s all set. I am pleasantly surprised at how little of a hassle it was and that it was way smoother than any other interaction I’ve had with them. Saturday, 2 days until cutover: We’re planning on doing the actual moving Sunday morning and plan to spend Saturday packing and planning.

Had an amusing conversation with a developer at work that had the feeling of an abbot and costello bit. I’m leaving his name out of this to protect him, but he’s read this site before and will know instantly that it’s him. The conversation revolves around our new continuous integration system, and how the terminology has changed. BTW, QA=Quality Assurance, UAT= User Acceptance Testing (staging) (10:50:50 AM) morgajel: ok, so after talking to mick, it looks like my suspicions were correct

Can’t locate object method “seek” via package “File::Temp” at /usr/lib/perl5/site_perl/5.8.8/MIME/Parser.pm line 816. The underlying problem is perl was updated and overwrote the “correct” version of File::Temp that you probably installed when setting up RT and forgot about. To fix this issue <br></br>cpan install File::Temp<br></br>/etc/init.d/httpd restart<br></br> MAKE SURE TO RESTART APACHE! I didn’t, and it cost me probably 2 hours of screwing around with it. I’m posting this because http://www.nabble.com/RT-3.6.5-and-Sendmail-error-and-looks-like-perl-error-td15989015.html Didn’t really mention what the final working solution was.

For those of you who can’t get those three to work together, make sure you specify both TLS_CACERT tls_cacertfile- I didn’t and it caused me grief.

I recently wrote a nice little script in ruby for ldapifying new ubuntu servers- all the server needs is a ssh key set up for root, the rest is cake… jmorg@util3:~/base_configs# ./ldapify -h Usage: ldapify --install hostname [$options] ldapify --check hostname ldapify --uninstall hostname --install hostname hostname to ldapify(foo.pub.local) --uninstall hostname removes ldapification from hostname -a, --access_group access_group access_group that has access to hostname --no_group_dn No access_group limitations- use with caution.

Whoops. This happened after work on Tuesday. Jackie was at home. I was on a small service drive that T’d into a 4.5 lane road that was fairly busy. looked left, saw a car with it’s right turn signal on in the right lane. looked right, say a red light. looked left, saw the car with the blinker slowing down as if to turn down the road I was on. looked right, saw the red light turn green, meaning I had about 10 seconds to react.

Doing another annual recap this year like I did last year. 2006 was a pretty rough year- move from DC Metro back to Grand Rapids, got a crappy job, found out jackie was pregnant, got laid off from crappy job, spent 3 months unemployed, and at the end of November was offered an awesome new job in Troy, MI (the other side of the state). Other than Ian, and the new job, the best good news is we finally have a decent couch- two of them, actually.

Ok, trying out Kubuntu on my new work laptop and I’m liking it quite a bit- the only problems I’ve had so far are with Hibernate (which I think is self- inflicted) and wireless stuff. I’ve figured out the wireless stuff and wanted to mention it for the people out there having the same troubles as I did. First up, a little info on my setup: Model: IBM T42 (note, not the T42p, which is awesome, the crappier model)

So after 3 months of unemployment, I was offered a job over in Troy, so I’ll be moving about 3 hours away from Grand Rapids. The position is for a linux sys admin. Unlike the last place, this one seems like a winner from the get go- much more lively and has a (for lack of a better term) DC vitality, compared to companies in Grand Rapids which sort of feel like they’re recycling coffee grounds and requesting employees bring in their own toilet paper.

So I just got hit with a new low… Looks like I’m inelegible for unemployment. Apparently I haven’t been in michigan long enough to warrant helping me out. when I look back on my life, this will be the moment when I said “this is as bad as it got.” I’m hoping it doesn’t get much worse at least. The depression that I’m sitting in right how is the soul crushing type where you lose the will to do anything.

Today I find myself a little of both. I was called into the confrence room today and told that the company I was contracted to requested that today be my last day. Yes, I am officially unemployed. Fortunately I was laid off, not fired- turns out business wasn’t as good as they were hoping and they can’t afford someone to try and fix their broken infrastructure. Since they fired a developer a few weeks ago, and the lead developer put in his 1 week notice today, I can’t say it’s suprising- if they’re getting rid of me after losing two other people, they must be hemmoraging money pretty badly.

so the devs are having a hell of a time with the new dev server- it’s constantly locking up on certain threads for 30-45 seconds. apache mainly, but occasionally grep and other things. It appears to be completely random. I’ve been pulling my hair out trying to find the problem, and I think I got it nailed down thanks to a post on the gentoo forums (http://forums.gentoo.org/viewtopic-t-189180-highlight-writethru.html) that said to change some settings in the raid card’s firmware… I tested this on the prod machine since it’s not in use yet(they’re both dell 2850’s with the lsi megaraid cards) and benchmarked before and after with bonnie++… here are the results(- is original, + is new):

So we just got these shiny new Netgear GSM7224 layer two 24 port managed switches, and I went about setting up MRTG. mrtg and snmp are one of my weak areas- I’m not too good at networking stuff to begin with, and mrtg has always seemed just out of my reach. Well, between rewiring half of the serverroom and threatening mrtg with a stick, I got it all working! now I can finally monitor network traffic and figure out which one of these network cards is a chatty cathy.

I have a habit of moving stuff to my website’s root directory and sending temp links to people. I’ve recently went about cleaning it out (which I do about once a year) and found some interesting bits… Here’s a job posting I wrote for a position we were hiring for at a previous employer: We're currently looking for a Mid-Upper level Perl Web Application Developer. If FrontPage or Dreamweaver is on your resume, please do not reply.

So I learned something new today. Apparently our UPS is rated to 3K, and we just added 3 new machines to it, putting it at 4 out of 5 bars for load. These three new machines are running gentoo on dual-xeon processors. I made the mistake of attempting to recompile subversion on all three machines at the same time. Suddenly, all the servers stopped responding. When we went into the serverroom, all the machines in the dell rack were off- the UPS had tripped.

ran into a problem with mod_rewrite and IE- a real one this time. IE was choking on files that were being downloaded via mod_rewrite. same script worked without rewrite, died with it. Did some research and with a lot of help from noodl of #apache figured out it was because mod_rewrite adds a “Vary: Host” line to the header, which apparently IE chokes on. loaded the header module in apache and added “Header unset Vary” after my rewrite and all is good- cvs, pdf, zip all download properly.

So apparently my problem yesterday with aw_foo.jmorgan.example.com didn’t work because _ is not a valid dns character. Every-freakin-other browser allows it to work except IE. Figures- the one time microsoft didn’t embrace and extend and did the minimum amount of work, it bit me in the ass.

ok, here’s my latest bout of IE stupidity. The new system I’m setting up will have several developers working on several projects. We’d like to be able to use subversion to manage the projects without dealing with pathname stupidity, hence all new projects should have / as their base since they will later become full fledge sites. So, how do we do that? I had the simple idea of mapping http://foo.

I don’t know if BSD can smell the gnu in my blood or what, but it is seriously putting up a fight. As I mentioned in the previous post’s followups, I got kde working finally- however every time I pop open Konqueror and hit slashdot, the entire machine locks solid. Not just konqueror, not just X; the whole machine. Now, I know slashdot has a troll meme about BSD dying, but this is ridiculous.

So, I’m compiling a list of what’s missing from my BSD install from the get go. tab-complete – stupid default shell is csh, which means no tab complete. Come on guys, jump on up to 1999. alt key – This is probably a keymap issue, but the alt and delete keys do not work. Alt acts like it does nothing, and delete behaves like a tilde. This means no alt tab.

Holy crap, welcome to 1980. The FreeBSD install is going to take a little bit longer than I thought. I booted off the Install CD and the first thing I noticed was the lack of color. Not shiny pretty GUI color, but angry fruit salad color. As pages of white text on black blackground whizzed past my screen, nothing stuck out as important; I noticed no dividers between sections. This is a very small, trivial thing, but it is nice.

Well, the new job is gonna have a bucket full of FreeBSD servers. It’s been a while since I tinkered with FBSD, so it’ll be a challenge. To prepare for that challenge, I’ve decided to convert Draccus (my workstation) to BSD. The following is a list of hurdles I’ll need to surpass before I start in April: get draccus up and running get KDE up and running get Gimp working get my Wacom Graphire 3 tablet working get my Logitech Quickcam working get my m-audio keyboard working get rosegarden recording again burn a cd-rw burn a dvd+rw I’m presuming I’ll have no problems with my Nvidia 6600 GT, Audigy 4 soundcard, or other common hardware.

For the few people who haven’t heard yet, Jackie and I will be returning to Grand Rapids. I was offered a position at a local GR company as a Network Administrator. The company seems really nice from what I saw last friday. I can’t describe how much I really hate the DC area. It’s just too crowded for me. Among the many benefits of this move, Jackie and I will:

So yesterday a coworker came up to me and asked if I wanted to go to FOSE- I have a vague idea it was a government tech convention, but that’s about it. Since I have a Gov’t ID badge, I was able to get in for free. So Cris(the boss), David and Terrance went into DC in search of the washington DC convention center. It should be noted that this is not the MCI Center, Metro Center, or the Verizon Center- it’s a completely generic convention center with a completely generic name.

Got an interesting call today. ———— guy on phone: Hi, I’m with thje intrusion detection team- we had a wolverine alert that your machine set off, I was wondering if you could enable your machine so I could remote in and check it out and make sure you’re not infected. me: Which machine was it? guy on phone: *reads off a machine number* me: Ah. ok, you won’t be able to- that’s my laptop, and it’s not running windows, it’s running linux.

Yeah you, the jackasses who decided that the middle of a crowded elevator lobby was a good place to stop and chat. While I understand it’s of dire importantance you tell each other about what your coworkers said to annoy your other coworkers, could you please NOT STAND IN FRONT OF THE ELEVATOR CALL BUTTONS while doing it? I know there’s not much space there, with it being lunch rush and 50 other people crammed into this small space, but we all have to make sacrifices.

So i’m getting slapped around a bit by karma right now. I recently moved our DNS services to two new servers. everything appeared to be fine… I got cocky. Now it turns out they’re not behaving like proper slaves for certain domains. fore xample, we run foo.com, bar.com and bar.net as the primary DNS. We ALSO act as slaves for baz.org… except we’re not anymore. For some reason, we’re not getting updates from the primary DNS for baz.

I don’t even know where to begin. This morning has sucked and it’s no ones fault but my own. It started with the DMA crap- the one machine where it would really help is the fileserver. I needed to simply change the chipset driver, recompile the kernel, and reboot. This should be simple for even a semi-competent linux users. However, I am a moron. Each mistake I make, I’m gonna put a little * next to it.

Hrm… I think I’ll try something new this year. As some of you know, 12/31 is my birthday, and I’m 26 now. That makes it twice as good to look back on my life and reflect. So what’s changed? Quite a bit. I always had a goal growing up, that when I was 25, it would be the one of the best years of my life. Looking back, I think that it very well could be.

So I woke up at 2:30am with severe back pain- the stabbing kind that makes it so you can’t breathe unless you sit/stand up. When THEY WHO SHALL NOT BE NAMED hired me, they brought me down from Michigan and paid to have our stuff moved. Long story short, their movers of choice were incompetent, and it ended up taking a month to get my bed. Since we were dead broke, I couldn’t afford a spare, and the cats kept popping the air mattress.

So in order to keep this job, I have to be cleared for a position of public trust. This means I have to document Every Place I’ve lived in the last 7 years Every Place I’ve workd in the last 7 years A Contact for each one of the above 3 people who know me well. So if I start asking weird questions, and you get calls from the Feds, don’t freak out- I’m not going to jail- it’s just clearance.

Yesterday I turned in my resignation. October 21st will be my last day. I was offered a job at another place, a big Gov’t contractor in the US as the Lead Linux Systems Administrator. The pay is better and the work sounds more fun. Just wanted to share that with you. Perhaps later down the road I’ll release my reasons for actually leaving- you know, the dirty secrets that disgusted me to the point of find another job.

So, Today we did the the rest of the training, which was just a big Q&A. Whether or not this is the right choice of software, I’ve given up caring- I’m not in a position to make the decision, and the decision will be made for political, not technical reasons which I will have no effect on. So I’ve quit agonizing about it. You could say, I suppose, [meme] I don’t care.

Alright, yesterday was a long day. training started at 8:30am, went till 5:30pm. What did I learn? That’s a damn good question. I learned how to use a web interface for extremely complicated tracking program . My biggest complaint is this training cannot be abstracted to be useful on other projects; it’s basically a 2 day training course on an obscure application. What I liked: clean interface, worked in both Firefox and IE.

So I’m being “sent” to training today and tomorrow. We’ll see how it goes. The training is for some business modelling software, or something. I’m still not sure what it does or how we’re going to use it. Meh, I’ll find out sometime today.

Alright, so it’s been a month and a half since Pete left SPX. What have I been up to? GLAD YOU ASKED. Update- upon some sage advice from the boat guy and a phool, I’d like to express how cool all the stuff I’m doing is.