Archive for the ‘Work’ Category

home sick

Tuesday, March 24th, 2009

So, I’m home sick again. Fourth time this year I’ve been sick. Cold, flu, cold, Bronchitis. Awesome. Will I get to rest today? No, of course not.

My server (Unicron) has been up and running for 2 years now- I got the parts right after Ian was born. I set up a nice software raid array at the time that’s served me well. I’d never set up a raid array like this before, so I wasn’t really sure how to monitor it. The raid array has been running fine for 2 years, so I just sorta let it slide.

Over this past weekend, I did some work resizing the lvm partitions and had to poke around with the raid stuff. I found not one, but two ways to monitor it one was to set up a monitor with the mdadm tools and have it email me if there was a problem, and the other lead me to create a simple nagios monitor. I set both up sunday night.

Flash forward to this morning- jackie wakes me up, asks me if I’m going to work (I’d come home sick the day before and was still out of it.) My intention was to wake up long enough to IM my manager and supervisor and let them know I was gonna be sick. I do so, then minimize the im stuff. Staring me in the face was the following:
raidfailure

Wait, wait, wait- my script must be crappy, there’s no way the raid array choked right after setting up the monitoring. I sorta go into denial and check my email:

This is an automatically generated mail message from mdadm
running on unicron

A Fail event had been detected on md device /dev/md2.

It could be related to component device /dev/sdc2.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid5 sda3[0] sdd2[3] sdc2[4](F) sdb3[1]
1461633600 blocks level 5, 64k chunk, algorithm 2 [4/3] [UU_U]

md3 : active raid1 sdc1[0] sdd1[1]
979840 blocks [2/2] [UU]

md1 : active raid1 sda2[0] sdb2[1]
979840 blocks [2/2] [UU]

md0 : active raid1 sda1[0] sdb1[1]
192640 blocks [2/2] [UU]

unused devices:

Frak.

So here I am, praying it was a hiccup while I reboot and rebuild my raid array. it looks like sdc2 went nutty about 45 minutes after I went to bed. I restarted the server and sdc2 reappeared, and I’m rebuilding now to see what happens,

md2 : active raid5 sdc2[4] sda3[0] sdd2[3] sdb3[1]
1461633600 blocks level 5, 64k chunk, algorithm 2 [4/3] [UU_U]
[=================>...] recovery = 85.0% (414436248/487211200) finish=26.8min speed=45182K/sec

One thing is for sure, I need to get a auxiliary drive in case this one goes kaput for real. I said I’d buy one in june… 2007. suppose I better get one that, huh?

My apologies of this was nonsensical, I’m really tired.

new plugin test.

Thursday, March 12th, 2009

So I’m testing a nifty new wordpress plugin… check this out:

I wonder if it’ll work?

update: no, no it will not.

What’s up?

Thursday, January 29th, 2009

So I’ve been pretty quiet since I hit 100k words- what’s been going on?

  • Round of layoffs at work
  • Friend diagnosed with cancer
  • Another round of layoffs at work.
  • Jackie became a pampered chef consultant
  • Finances have been wiped out from christmas and getting her PC stuff off the ground.
  • 10% paycut at work
  • Guitar lessons are now done because no one can afford them.
  • Have been reading Manuscript Makeover for ways to improve my book
  • Decided to do an initial cleanup of the first draft of my script, then rewrite the outline before starting draft #2
  • started yet another opensource project- this time it’s a collection of Nagios Plugins.

So I’ve been pretty busy. I’ve finished the cleanup of the first two chapters of book 1; hopefully I’ll finish the rest shortly, but it’s very slow going. We’ll see where things head in the next few months- I expect more crappiness.

Free Jabber / XMPP clients for a Blackberry?

Wednesday, August 6th, 2008

anyone know of any good jabber clients for the blackberry? I’ve tried a couple with little luck, and most of them cost more than I can afford for this test. Features required

  • Must run on BlackBerry 8703e v4.1.0x
  • Connection server can be configured differently than jid address (i.e. you@morgajel.net for jid, jabber.morgajel.net for connection server.) This rules out Mobber as far as I can tell
  • Requires SSL/TLS
  • Non-strict cert checking

Let me know if you have any suggestions.

What’s blue and white and still not working?

Thursday, July 10th, 2008

My internet connection.

SO here’s the scoop

5 days until cutover:
I call AT&T, tell them I’m moving and need to transfer my Static IP DSL service on the 30th(Monday). Tech says no problem it’s all set. I am pleasantly surprised at how little of a hassle it was and that it was way smoother than any other interaction I’ve had with them.

Saturday, 2 days until cutover:
We’re planning on doing the actual moving Sunday morning and plan to spend Saturday packing and planning. However at 3am Saturday morning, the internet connection drops, leaving me unable to contact many of the people who may be able to help us move. It sucks, but ok, we can work around it. I still have enough people to get by with and have ways to contact most of them. Since we asked to be connected on Monday, maybe they had to disconnect the old line the day before in order to get their stuff in place. Maybe they cut it on Saturday rather than Sunday because nobody works on Sunday. I get that, I can understand it. While annoying, it’s still better than my previous interactions with them.

Sunday,move day, day before cutover:
We move on Sunday and realize that we never actually checked to see if the house had any phone cables in it. It didn’t. Fortunately my father-in-law knows a bit about phone installation and was able to help me wire up a stub for the AT&T guy to connect to.

Monday, 1 day after move:
AT&T shows up, runs cable, says service will be enabled withing X hours. yippie.

Tuesday, 2 days after move:
Connection is there, but my ip address has changed. “crap,” I think, “now I gotta update dns entries for our sites.” But I can understand this, perhaps my old static ip was tied to the network near my old apartment and didn’t reach this area. I can buy that. So I change my DNS entries… and they don’t work. I look again and I apparently mistyped the IP because the new DNS entry doesn’t match the external IP on the router. So I change it again. and 20 minutes later the external IP has changed again.

They had me on a fricking dynamic IP. For the non techies out there, large ISPs only have a limited number of ip addresses, and more often than not don’t have one for every customer. Since few customers stay online 100% of the time, they can take addresses away from people not using them and redistribute them as needed. This is called a Dynamic IP account. For people who run servers from their homes, keeping the same IP is important, so when your computer goes to connect to morgajel.com, it needs to be able to find the right IP address. That’s why I pay extra for AT&T to guarantee me the same IP address. That is why I am pissed. While there are ways to get around this (dyndns) but they’re a pain in the ass an not an option for me since I run an IRC server as well.

So I tinker around, thinking maybe *I* did something wrong- maybe my router was reset and it cleared the static info. I dig around with Jackie’s help and find the original documentation and try to set up the networking listed manually. No dice. Then I remember that yes, they did manage the info via the PPOE settings, and that just required a user name and password, which is what I was originally using. I switch it back and get yet another dynamic IP. I should point out that my static IP range was 75.x.x.x, while the dynamic stayed in the 66.x.x.x range- this made it easy to keep track of what was going on.

So I call them up and surprise surprise, they screwed up. See, they don’t really transfer accounts so much as shut off the old one and create a new one. The tech didn’t bother to notice I had a static account and replaced it with a dynamic account. I’m livid at this point, and tell them that it needs to be switched back. “Ok, I’ll put in the order. It’ll be ready in 10 days.” Now, this should NOT take 10 days from a technical point of view, this is all red tape causing the delay. But WTF can I do, so I say hell with it and go along with it.

At some point my father-in-law comes back over to help with the baby gate and notes that the technician illegally ran the line through the neighbor’s yard. While I’m half tempted to yell at them to fix it, I just wanna get a connection up and running again so I can actually write about the house.

Saturday, 5 days after cutover (timeline gets a little fuzzy here)
Connection is still flaky, but generally working. I call to check on the status of the static IP order, and find out it was never placed. They’ll get right on that.

Sunday, 6 days after cutover
Connection goes down at 7:37am. Completely. It does not come back. Jackie calls tech support this time. Flames, brimstone cries of the undead ensue. Eventually I take the phone and find out there’s still no mention of a static order of any sort for our account. Guess what? They can’t do anything about it because “orders” isn’t open on weekends. They agree to send out a tech to look at the line since they can’t see the modem from their end. He should be out between 8am and noon on monday

Monday, 7 days after cutover:
Connection begins working again around 7am- I think to myself “great, maybe they just took it down to switch over to the static IP- finally I can get my stuff up.” Nope, still a dynamic IP address. I call AT&T to get the static IP address set up and let them know the connection is up. They say hold off until the technician confirms it’s not an issue. ok. I’ll call back later. I spend my time waiting for the technician looking for any other ISPs in the area on dslreports.com

Technician comes out, nice guy, doesn’t see anything wrong, says he’s seen this behavior before when switching from dynamic to static, but the business won’t fess up to it. Whatever. At least the wiring was good, presuming that both the installer and the inspecting tech were both competent. While he was tooling around, I found out that Cyberonic, my ISP from DC, covers this area (they didn’t in grand rapids or rochester hills). They resell business class Covad lines to residential customers. I contemplate switching over to them, but figure it would be too much effort since I’ve gotten this far. I’m not even sure they’d have a decent plan in this area.

So he leaves and I call AT&T back and get the static all set up. She also said the static IP would be in place tomorrow. Just as we’re finishing she informs me that since I don’t have a contract, my payment will go up to $70 a month from $55. “WTF, this isn’t my screwup- you guys said you could transfer service, then you pooch it, then you want to charge me for it??”

“Oh, no,” she says, “When we transfer service, we don’t transfer contracts. If you want the original rate, you’ll have to sign up for another year of service.”

This is where Jesse snaps.

“You know what? Fine, make it the month to month price, because it’ll take me about 3 weeks to get covad in here.” She was a bit shocked by that statement, and the conversation ended awkwardly. I think she was supposed to ask if I was please with my experience but she knew the answer.

I then spent 10 minutes looking through DSL reports for ISPs in the area and narrowing down their plans- turns out that Cyberonic offers the same plan I had in DC for $60. Lets compare the plans side by side:

AT&T Cyberonic
Download speed 3meg 6meg
Upload speed 386k 768k
IP address 5 static 5 static
Stability False True
Cost $55/mo $60/mo

I call up cyberonic, phone is picked up on the 3rd ring. I tell the technician that I’m interested in their plan, I get signed up, cc infos taken, etc. The entire call lasted 22 minutes and 28 seconds. I was never transferred once, my call was never dropped, the technician never once said “I don’t know,” and they were going to do a hotswap on the line and cancel the AT&T DSL for us since we obviously can’t have 2 DSL services on the same line. The transfer should take place in the next 7-14 business days.

I’d like to point out that AT&T still hasn’t got their act together as of this morning (Thursday), and dropped my connection while I was beginning a deployment for work. That was real awesome btw. Thankfully my neighbor is allowing us to use his wireless connection until we get it straightened out. If the issues aren’t resolved by switching to cyberonic, I’ll have the neighbor report the cable crossing his yard and they’ll have to come out and redo it (this is my backup plan).

The good news is we’ve moved our blogs to gopedro.net. I’m still in the process of converting them, but expect to be done by next Monday. The only site that will still point to my static IP is morgajel.com, for my streaming music, IRC server, etc. We’ve also decided to move all of our pictures to flickr, so expect to see broken images for a while.

I really want to thank gopedro.net for in all of this. I highly recommend them for any domain name purchases or hosting. They’ve been handling our domain names for years now, and their service is outstanding. I’d also like to thank our new neighbor Bobby for being one hell of a cool guy.

I’ll keep you updated on how things go. Hopefully I’ll start writing about the house soon.

*UPDATE 2008-07-14*
Cybronic called and told me they’d be sending a technician out tomorrow to verify the lines. Hopefully I should have a working connection soon.

*UPDATE 2008-07-16*
My bad, it was wednesday. Connection is up now and I’m back online with a static IP!

What’s on QA…

Friday, May 16th, 2008

Had an amusing conversation with a developer at work that had the feeling of an abbot and costello bit. I’m leaving his name out of this to protect him, but he’s read this site before and will know instantly that it’s him. The conversation revolves around our new continuous integration system, and how the terminology has changed.

BTW, QA=Quality Assurance, UAT= User Acceptance Testing (staging)

(10:50:50 AM) morgajel: ok, so after talking to mick, it looks like my suspicions were correct
(10:50:54 AM) morgajel: there is no QA
(10:51:04 AM) morgajel: the name QA is misleading.
(10:51:16 AM) DeveloperX: hmmm
(10:51:25 AM) morgajel: trunk is for development, release is for qa, uat and prod
(10:51:26 AM) DeveloperX: what do u mean
(10:52:47 AM) DeveloperX: i have a release branch…
(10:52:50 AM) morgajel: think about it- anything going to QA is being marked as a release.
(10:53:12 AM) morgajel: so so release gets deployed to QA, then if it passes it goes on up the chain to production
(10:53:52 AM) DeveloperX: this is how the system currentlyis
(10:54:23 AM) DeveloperX: it seems to make snese to me? assueming nothing makes it ti UAT unlees it goes through QA
(10:54:36 AM) morgajel: that’s the way it should be.
(10:54:49 AM) DeveloperX: i think thats how it is
(10:54:57 AM) morgajel: the difference between this and the old system is, rather than labelling it projectX[QA], they’re now calling projectX_release_
(10:55:07 AM) DeveloperX: no? i only have a trunk and a release
(10:55:11 AM) morgajel: that’s the main difference for you
(10:55:20 AM) DeveloperX: which means the only thing goingto UAT will be from the release
(10:55:51 AM) morgajel: correct- the only thing going to qa or uat or prod will be from the release.
(10:56:12 AM) DeveloperX: so why can’t we still call it QA?
(10:56:42 AM) morgajel: because it’s *more* that QA- QA was a bad name for it from a organizational perspective
(10:57:32 AM) DeveloperX: is his only going to affect me, or is this the new approach for everyone?
(10:57:40 AM) morgajel: for everyone
(10:58:27 AM) DeveloperX: so your saying there is no pointin multiple srevers?
(10:58:33 AM) morgajel: no
(10:59:04 AM) morgajel: I’m saying whatever we put on UAT should be the EXACT same thing as QA
(10:59:13 AM) DeveloperX: so we will still push from trunkto release to UAT?
(10:59:20 AM) morgajel: and whatever we put in prod should be the EXACT same thing as both of them.
(10:59:24 AM) DeveloperX: it is
(10:59:28 AM) DeveloperX: currenlty
(10:59:29 AM) morgajel: well, from your side, here’s what happens
(10:59:37 AM) DeveloperX: at least for anything i;ve worked on
(10:59:44 AM) morgajel: you guys develop in trunk, and when you’re ready to release, you mark it as release.
(10:59:53 AM) DeveloperX: k
(10:59:55 AM) morgajel: then the QA people will check out release to the QA servers
(11:00:04 AM) DeveloperX: k
(11:00:10 AM) morgajel: once they give it a go, we will check it out to UAT,
(11:00:22 AM) DeveloperX: k
(11:00:24 AM) morgajel: once that’s good, it will be checked out from release toprod,
(11:00:54 AM) DeveloperX: put the code that is working it’s way up is exacly the same in each instance
(11:01:12 AM) morgajel: correct- as it should be.
(11:01:18 AM) DeveloperX: it is jsut being push multiple times, to make sure everything will work when we finally hit orid
(11:01:20 AM) DeveloperX: prod
(11:01:27 AM) morgajel: correct
(11:02:06 AM) morgajel:
now, I’m not sure what the exact mechanism will be for pushing to uat
and prod- we may check it out directly from svn, but more likely than
not, we won’t.
(11:02:33 AM) DeveloperX: ya, i would guess not
(11:02:37 AM) morgajel: the important thing is we can say “release 11234234 of projectX” and can match it up with a specific commit in subversion.
(11:03:50 AM) DeveloperX: i gotcha there…we used to havesomethiing like that, but it completely maintianed by me
(11:04:16 AM) DeveloperX: i have a new release branch for every deployment i make to projectX
(11:04:34 AM) DeveloperX: so I code in theory go back to 3deployments ago
(11:04:46 AM) DeveloperX: but you guys should have access to this…not just me
(11:05:35 AM) DeveloperX: make sense?
(11:08:09 AM) morgajel: you’re close
(11:08:23 AM) morgajel: I’m presuming you’re talking about http://buildserver/svn/projectX/branches/, correct?
(11:08:32 AM) morgajel: each of those listed is a different release branch, correct?
(11:09:01 AM) morgajel: what they’re doing is instead of creating a folder for each release, they’re creating a single folder
(11:09:17 AM) morgajel: and making note of the commit version when they commit acode to it
(11:10:14 AM) morgajel: hence when subversion says “committed version #1123223″,you’d call it “release 1123223″ in your notes
(11:10:30 AM) morgajel: I think that’s what brandon is trying to aim for, but we’ll have to have him verify.
(11:11:00 AM) DeveloperX: ok, i think i gocha

Request Tracker 3.6.5 broken after updating Cent OS

Monday, April 7th, 2008

Can’t locate object method “seek” via package “File::Temp” at /usr/lib/perl5/site_perl/5.8.8/MIME/Parser.pm line 816.

The underlying problem is perl was updated and overwrote the “correct” version of File::Temp that you probably installed when setting up RT and forgot about. To fix this issue


cpan install File::Temp
/etc/init.d/httpd restart

MAKE SURE TO RESTART APACHE! I didn’t, and it cost me probably 2 hours of screwing around with it.

I’m posting this because
http://www.nabble.com/RT-3.6.5-and-Sendmail-error-and-looks-like-perl-error-td15989015.html
Didn’t really mention what the final working solution was.

LDAP+ Sudo +TLS fix

Tuesday, October 9th, 2007

For those of you who can’t get those three to work together, make sure you specify both TLS_CACERT tls_cacertfile- I didn’t and it caused me grief.

ldapifying an ubuntu server

Monday, March 19th, 2007

I recently wrote a nice little script in ruby for ldapifying new ubuntu servers- all the server needs is a ssh key set up for root, the rest is cake…

jmorg@util3:~/base_configs# ./ldapify -h
Usage: ldapify --install hostname [$options]
       ldapify --check hostname
       ldapify --uninstall hostname
        --install hostname           hostname to ldapify(foo.pub.local)
        --uninstall hostname         removes ldapification from hostname
    -a, --access_group access_group  access_group that has access to hostname
        --no_group_dn                No access_group limitations- use with caution.
    -c, --clusters x,y,z             clusters in which hostname belongs
        --aliases x,y,z              other aliased hostnames for the host
        --ignore-home                don't mount/unmount home
    -v, --verbose                    enable verbosity
    -q, --quiet                      silence all unneeded messages
    -h, --help                       Show this message

jmorg@util3:~/base_configs# ./ldapify --install log1.pub.local -a devboxes
backing up sources.list...
updating sources.list...
updating package list...
adding nfs entry to /etc/fstab...
Complete.
Mounting home, please wait...
complete.
installing debconf-utils...
patching debconf selections...
installing libnss-ldap ...
symlinking ldap.conf...
copy ssl cert and ldap.conf...
complete.
No Access group was given, using admin_only by default.
backup nsswitch.conf and pam.d files...
complete.
install nsswitch.conf and pam.d files...
complete.
ldap requires the manager password:
please verify the manager password:
store manpass...
installing sudo-ldap...

jmorg@util3:~/base_configs# ./ldapify --uninstall log1.pub.local
restore nsswitch.conf and pam.d files...
complete.
remove ssl cert and ldap.conf...
complete.
removing nfs entry...
complete.
unmounting home...
complete.
removing debconf-utils, libnss-ldap and libpam-ldap ...
removing ldap.conf symlinks...
removing sudo-ldap, restoring sudo...
retore sources.list...
updating package list...
ldap requires the manager password:
please verify the manager password:
jmorg@util3:~/base_configs#

So what all does it do?

  • Sets up ldap authentication of user accounts
  • mounts the nfs-based home directory
  • Sets up ldap-based sudo rules
  • Creates a host entry in the ldap server
  • Adds an entry in the ldap server for the distro’s cluster and ldapified hosts cluster
  • Can completely revert back to the original state

This script takes about 2:45 to run (mostly due to the 120 seconds of waiting for the /home dir to mount), and saves roughly half an hours worth of work. It’s not very stable (pre-ldapified boxes cause it to freak out when trying to re-install/remove) , but it will be a lot of help as we move towards ubuntu as our standard distro.

I wrecked the car.

Sunday, February 4th, 2007

Whoops. This happened after work on Tuesday. Jackie was at home.

I was on a small service drive that T’d into a 4.5 lane road that was fairly busy. looked left, saw a car with it’s right turn signal on in the right lane. looked right, say a red light. looked left, saw the car with the blinker slowing down as if to turn down the road I was on. looked right, saw the red light turn green, meaning I had about 10 seconds to react. I had been at this intersection for about 3 cycles of that light, and saw this as my oppertunity. as I pulled out, the lady with the turn signal showed me that she actually was meaning to go straight. She sheared off the front of the car from the front of the front tire foreward- bumper, lights, hood, etc. radiator and engine were intact.

Cops gave me a ticket for failing to yield, but I’m gonna try and fight it since she had on her turn signal and failed to add that to her report. The witness on the scene saw the turn signal, but said it was because she was changing from left to right lanes and then just left it on.

So the car has $10k worth of damage because the frame is bent. They’re probably going to total it. I got away with just a sprained wrist.

Oh well, at least we’ll get a new car out of it (probably).