Open Source

28

Whew, it’s been a busy year. Back in 2005 I wrote a list of things I’d like to do eventually that I’d be able to do if I had the time and resources that winning the lotto would provide. I’ve made a positive step towards realizing a couple of those.

Family

The big change this year was the addition of Ian. We didn’t get off too well in the beginning, but we’re doing pretty good now. He’s just now to the point where he’s taking his first steps. He’s a smart kid, very inquisitive, but stubborn. Within a few years he’ll be learning how to read and write- I’m guessing it won’t be long until I’m teaching him how to program. He’s got an eye for any type of electronics, but goes nutty for my blackberry or laptop. Yeah, he’s a geek in training.

Jackie’s been doing fairly well, all things considered. She’s been staying home with Ian rather than going back to work, and it’s driving her a little stir crazy. To help ease the cabin fever, she got a laptop for christmas- Ian wasn’t letting her stay in the office, so now she has a way to keep in contact with people from the living room.

Weddings

This was the year of the wedding… Brad and Erin, Jordan and Beth, Matt and Carrie, William and Charleen, Jackie’s friend Joslyn, Jackie’s sister Lori, and Paul and Kristen. I only made it to 4 of the weddings, but I was standing up in two of them. The ones we went to were all in Grand Rapids, so we had the joy of driving back and forth 155 miles each way, with the added bonus of dropping Ian off somewhere along the way

Work

Work started off pretty rough. I have a habit of reflecting the attitudes of those around me, and my supervisor at the beginning of the year was pretty negative. That coupled with Ian’s birth and all the associated expenses turned me into a pretty miserable person. Fortunately the supervisor left and was replaced with a cautious optimist. Things got better- so much so that I just passed the 1 year mark at my current place and still find it interesting and entertaining.

Technologies

I’ve played with many new technologies this year- Jboss, Netscaler and CentOS being the three foremost. I’ve dug back into ruby with a couple of new projects, as well as LDAP and a plethora of new utilities. I even picked up a bit of python. This trend doesn’t look like it’s gonna let up, so the job should be exciting and interesting for some time to come.

Projects

This year saw me move away from music and go back towards software (although I have been playing harmonica in the car at stoplights on the way to work). I joined the Luma team as head cheerleader and started the QT4 branch. After becoming frustrated with the python, qt3->qt4 transition and the codebase in general, I started an unofficial sister project, Ruma. I still sit in the Luma channel and help when I can, but I’m gonna sit back and wait until we’re ready for QT4- hopefully we’ll scrap the entire codebase and start fresh.

Right now I have two other developers “working” for me on Ruma, Lars and Hiro. They’ve put a lot of effort into bringing this bad boy to life, and I’m thankful I’ve had their help. For those of you not aware, Luma and Ruma are LDAP administration tools. As a System Administrator, I use LDAP a lot, so having specialized tools can be helpful.

Another project I started this year was competing in National Novel Writing Month. I finished the rough draft for my first book, Sinblade, a week early. It’s only 150 pages or so, but not a bad start all things considered. Jackie’s (sorta) working on revising it, and once she’s done I’ll open it up for others to start giving me feedback. I’m excited and nervous about it. If it takes off and people like it, I have 6 or so more books that I’m interested in writing.

The money situation is still pretty tight, but it’s getting better. Hopefully next year will be as much of an improvement over this year as this was over last year.

Introduction to Subversion

I was planning on simply republishing my previous svn article, but realized that it sucked compared to what I know now.

Prerequisites

I’ll presume you have the following things.
– a Linux machine
– subversion already installed

Terminology to Know

There are a few terms that get mangled if you’re coming from other types of source control. This is just to clear things up.
Repository: the central storage place on the subversion server where your data is kept.
Checked-out copy: Unlike VSS, saying something is checked out does not imply that the file is locked. Also referred to as a local copy; but bear in mind that it doesn’t contain *all* of the data of the actual Repository.
commit: save the changes you made locally to the repository.

Mental Hurdles

All of the files stored in a subversion repository are stored in a meta-filesystem. Much like an ISO image sitting on your desktop is not simply a folder full of files, a subversion repository is not directly accessible when you open it. Instead, you’ll see the guts of the repository- DB files, hooks, locks, etc. Don’t go digging through there to manually change your files- it’ll break things.

Another important one is the meta-filesystem. The “inside” of your repository is just a big filesystem. Much like /bin and /home exist on a linux machine by convention, there are certain base-level directories you should create for convention’s sake. The first thing you should do with a new repository is create three new base level directories: /tags, /trunk and /branches. Your main development will take place in /trunk. Don’t worry about the other two at the moment.

When I refer to the root of the repository, I’ll often refer to it as / or root. This *IS NOT* your server’s root directory or the physical location of the repository- it’s the part of the meta-filesystem where /trunk, /tags and /branches reside.

Step 1: Creating the Repository

Creating a repository is fairly simple. Anyone can create a repository where ever they have write access. All they must do is run

$ svnadmin create ~/project1/

This should create an empty repository. This will demonstrate what I’m talking about when I say that your repository is a meta-filesystem:

morgajel@unicron ~ $ svn ls file:////home/morgajel/project1
morgajel@unicron ~ $ ls /home/morgajel/project1
conf dav db format hooks locks README.txt
morgajel@unicron ~ $ svn mkdir file:////home/morgajel/project1/trunk -m 'creating trunk'

Committed revision 1.
morgajel@unicron ~ $ svn ls file:////home/morgajel/project1
trunk/
morgajel@unicron ~ $ ls /home/morgajel/project1
conf dav db format hooks locks README.txt

Here you can see the repository was empty, then we created a directory called trunk (using a commit message to describe the change we made), then showing that the directory was in fact created. Do the same thing for /tags and /branches.

We now have a working repository!

Checking out the Repository

Creating a repository is fine, but using it would be much more… useful. Next you should check out a copy of your project. Under normal conditions you’ll only be checking out the /trunk, but you mileage may vary in different situations. Since this is on our local machine, we can use the file:// protocol. Other protocols exist, like http:// (webdav), svn:// (svnserve), and svn+ssh:// (svn over ssh), but you don’t have to worry about them right now.


morgajel@unicron ~ $ svn co file:////home/morgajel/project1/trunk my_project1
Checked out revision 3.

This should check out the empty /trunk directory to a local folder called my_project1. The only thing in this directory is a hidden .svn directory which holds the guts of your local copy repository info. It’s similar in function to the CVS directory in CVS. Unfortunately a nearly-empty directory isn’t much use, so let’s add some content.

Adding Content

So let’s add some content.

morgajel@unicron ~ $ cd my_project1/
morgajel@unicron ~/my_project1 $ mkdir -p lib bin share/docs
morgajel@unicron ~/my_project1 $ touch configure Makefile share/docs/README lib/foo.pm bin/widget.pl
morgajel@unicron ~/my_project1 $ svn status
? configure
? share
? lib
? bin
? Makefile

Here I created a bunch of directories and created some empty files. When I ran svn status, svn told me that there were 5 things it wasn’t versioning. Let’s add them.

morgajel@unicron ~/my_project1 $ svn add configure share lib bin Makefile
A configure
A share
A share/docs
A share/docs/README
A lib
A lib/foo.pm
A bin
A bin/widget.pl
A Makefile
morgajel@unicron ~/my_project1 $ svn status
A configure
A share
A share/docs
A share/docs/README
A lib
A lib/foo.pm
A bin
A bin/widget.pl
A Makefile

As you can see, it recursed down into share, bin and lib and added all the goodies inside of each directory. You can also see svn status shows these as well. Keep in mind they’re just slated to be added to the repository- they’re not added yet. Let’s go ahead and commit them.


morgajel@unicron ~/my_project1 $ svn commit -m "a bunch of empty files and directories"
Adding Makefile
Adding bin
Adding bin/widget.pl
Adding configure
Adding lib
Adding lib/foo.pm
Adding share
Adding share/docs
Adding share/docs/README
Transmitting file data .....
Committed revision 4.

Modifying Data

So suppose you’d like to modify these files, you you decide to move the README to the root of your local copy (~/my_project1/):


morgajel@unicron ~/my_project1 $ svn mv share/docs/README README
A README
D share/docs/README
morgajel@unicron ~/my_project1 $ svn stat
D share/docs/README
M bin/widget.pl
A + README

Notice that I used svn mv to move files rather than regular old mv- That’s to make sure svn is aware of the move and keeps the file history associated with the new file. You can also see bin/widget.pl now include some new info as well, and displays an M[odified] next to it. The + next to README shows that it copied the history over from it’s previous position. So what happens if we move a file without svn mv?

morgajel@unicron ~/my_project1 $ mv configure config
morgajel@unicron ~/my_project1 $ svn status
? config
! configure
D share/docs/README
M bin/widget.pl
A + README
morgajel@unicron ~/my_project1 $ mv config configure

You can see that svn panics(!) that configure has gone missing, and sees this new file called config that it’s currently not revisioning. It doesn’t know that they’re the same file.

Commit Messages

You’ve seen me use the -m flag a couple of times now- I’m using it to keep things flowing. If you don’t use it, you’re prompted in your favorite $EDITOR to create a commit statement, which includes the list of modified files. Using the -m flag is useful if you’re scripting commits (I use this when dumping and committing a nightly config file from our load-balancer).

Most of the time however, you’ll use your Editor. Make sure to keep your commit messages sweet and to the point- other’s will see them.


morgajel@unicron ~/my_project1 $ svn commit
[vim shows up, I enter the following text]
Small changes to demonstrate movements
- moved the README
- added shebang to widget.pl
[save and exit vim]
"svn-commit.tmp" 8L, 190C written
Adding README
Sending bin/widget.pl
Deleting share/docs/README
Transmitting file data .
Committed revision 5.
morgajel@unicron ~/my_project1 $ svn update
At revision 5.
morgajel@unicron ~/my_project1 $ svn log
------------------------------------------------------------------------
r5 | morgajel | 2007-11-20 15:55:21 -0500 (Tue, 20 Nov 2007) | 4 lines

Small changes to demonstrate movements
- moved the README
- added shebang to widget.pl

------------------------------------------------------------------------
r4 | morgajel | 2007-11-20 14:17:04 -0500 (Tue, 20 Nov 2007) | 1 line

a bunch of empty files and directories
------------------------------------------------------------------------
r1 | morgajel | 2007-11-20 13:35:28 -0500 (Tue, 20 Nov 2007) | 1 line

creating trunk
------------------------------------------------------------------------

You’ll notice that revisions 2 and 3 aren’t listed- if you’ll remember correctly, they were used to commit the /tags and /branches directories. svn log only reports changes that affect the current target (in this case, ~/my_project1 which is a local copy of /trunk).

There’s a couple more tips and tricks I could go on about- if there’s any interest in this post maybe I’ll write some more about more advanced topics.

LDAP+ Sudo +TLS fix

For those of you who can’t get those three to work together, make sure you specify both TLS_CACERT tls_cacertfile- I didn’t and it caused me grief.

Intro to Vim Tip #5 (Recording)

Search and replace is a great feature in most text editors, but what happens when you want to do more? Vim has a solution- recording macros. Suppose you have the following output from some ancient program that needs to be tweaked:


X1222 22323 2A22 3303 0000 3334esss test 123
X2222 22353 2A22 3303 0001 3334esss tacd 456
X3222 22383 2A22 3303 0010 3334esss fals 789
X4222 22393 2A22 3303 0011 3334esss true 012

It is doesn’t really matter what it is, this example is somewhat contrived. Suppose you needed to make the following changes for each line that starts with X:
* change the ID from X_222 to Y_223
* reverse the 4th and 5th fields
* copy the second character from the beginning and insert it before the last character of the line

If it were only 4 lines, you could handle this yourself, but it would be very tedious. Suppose rather than 4 lines, you had 400- it’d be much easier to automate it. The best way to take care of it would be with a macro:

[esc]qa
/^X[enter] i[delete]Y[right][right][right][delete]3
[esc]wwwdww[left]p
0[right]d[ctrl+v]y$[shift+p]
q

That right there is a MESS, but gets the job done- it’s not something you want to repeat for fear of a typo. Notice that the first characters you typed were qa: ‘q’ starts recording, and ‘a’ is the slot we’re using to store the macro. From here we record how *we* would make the changes, making sure to keep our keystrokes to a minimum. When we’re done, we stop the macro by pressing ‘q’ again.

To run our macro on the next line, press ‘@a’ to run the newly created ‘a’ macro- it should find the next line that starts with an X (notice the /^X in our first command) and run those commands to massage our text.

Remember how we were talking about 400 lines like this? Even at 2 characters each, that’s 800 characters to type which is still annoying. Here’s where the magic comes in- you can record macros of macros:

qb
@a@a@a@a@a@a@a@a@a@a
q

Now each time you run @b, you’ll run the a macro 10 times. A more efficient way to handle this would be to use

@a
398@@

the first one was done manually to record the macro, the second to play the macro, and the third to say run it 398 more times.

And there you go- a quick tour of recording macros. I’m sure there’s much more than what I’ve shown, but that’s enough to keep you busy.

Epson Perfection v350 on Ubuntu Feisty Fawn

Ok, I’ve done it twice now, I think I’ve figured it out.
Get the source files from Epson’s linux website; I don’t recall the url but these are the files you need (or newer, no idea what the future holds):

iscan-2.3.0-1.c2.i386.rpm
iscan-plugin-gt-f700-2.0.0-0.c2.i386.rpm  

Use alien to convert them to debs (normally bad, but acceptable this time around).

Install both newly created deb packages with dpkg:
dpkg -i iscan_2.3.0-2_i386.deb iscan-plugin-gt-f700_2.0.0-1_i386.deb
Install sane, sane-utils:

apt-get install sane sane-utils

Edit /etc/sane.d/dll.conf, replacing epson with epkowa (yes, I’m serious).

add the following to /etc/udev/rules.d/45-libsane.rules:

# Epson Perfection v350
SYSFS{idVendor}=="04b8", SYSFS{idProduct}=="012f", MODE="664", GROUP="scanner"

And I think that’s about it. Make sure you’re in the scanner group, and run:

scanimage -L

That should simply list your scanner- if it doesn’t let me know- I might have missed something. From this point on you should be able to access the scanner with either sane or iscan.

The one downside to this driver is it doesn’t do the full 48000 PSI, only 24000- this means a scan of a guitar pick is *only* 16 inches across, rather than 32 inches 🙂 Unfortuantely it’s not the preferred resolution for negatives.

Spell checking only *.txt files in Vim

So vim 7 has native spell checking- which is great if you remember to turn it on. Enabling it in your .vimrc is fine if all you view is written text files, but annoying when it’s running while working on code.

That’s why I created this little snippet of sunshine:

add this to ~/.vimrc:

au BufNewFile,BufRead *.txt call ConfigureTxtFile()

func! ConfigureTxtFile()
    setlocal spell spelllang=en_us
    set wrapmargin=80
    set textwidth=80
endfunction

I also use it to set my text width and wrap margins. You can have it match other types of files as well and create new functions- I plan on using this to set .py files to use space-replaced tabs. As I go forward I might have it do something a bit more fancy.

Dissappointing state of LDAP support in Ruby

So I’ve been working on a new project called Ruma(more on it later), and I’m beginning to get frustrated with the LDAP support. So far I’ve found
* ruby/ldap, ruby-ldap, ldap-ruby by ian macdonald, last release 8/2006
* net-ldap, Net::LDAP by Francis Cianfrocca, last release 8/2006
* ruby-activeldap, ActiveLDAP-ruby by Will Drewry, last release 5/2007

ldap-ruby appears to be the frontrunner, but hasn’t been updated in a year- the forums appear to think it’s been abandoned. Net::LDAP appears to be a work in progress that isn’t quite complete. ActiveLDAP-ruby appears to be mainly for rails, so I don’t know how useful it is.

When you search for ruby ldap, you can any of those name combinations, and it’s impossible to figure out which is alive, which is stable, and which is useful. At this point I don’t think any of the projects meets all three criteria at the moment.

Supporting Standards vs. Supporting all Testcases

So I’ve been working on a side app that connects to remote servers. While writing validation code for inputs, I came across an interesting dilemma. I’d like to validate an address input to be valid- be it a hostname, domainname, or ip address. Unfortunately it’s not as simple as it sounds.

According to the domain name RFC, section 3.5 domain names must start with a letter, end with a letter or number, and can have letters, numbers or hyphens in between. When you add in periods to separate it out it’s messy, but workable.

But not everyone follows the RFC- take fifth/third bank for example- I’m not sure how they got away with it, but it doesn’t really follow this (historically I think 3com is the one to blame).

So that leaves me with two choices: follow the standard, or allow for the outliers. It’s doubtful anyone will be using my app to manage 53.com’s ldap server, and less likely that 53.com IS an ldap server. It is however possible that some one out there has either named or inherited a server named 3app6ldp02.foo.internal.

So do I support the guy? Should I care? should I even bother validating the address?

What I Dislike About Python

Since I began working on the Luma project, I’ve been playing a lot with Python, a language that I’ve been around for years but never bothered to learn. Since Luma is written in Python and I’m not on the team, I figured it was time to jump in feet first. Coming from a perl/php/ruby/java background, it wasn’t a big leap to make.

However, the more I read and write, the less I like it. Some are minor complaints, while some are a bit larger. I’m gonna list them here as I think of them, from trivial to more important

* indentation – This is really an annoyance more than anything. If you’re not aware, python is whitespace sensitive- rather than using curly braces, it uses indentation to show scope and what belongs where. Oh, and you better make sure to replace your tabs with spaces, otherwise you’ll ned up with bizarre errors that are a pain in the ass to track down.

* debugging tools – This one is a bit more aggravating- I may not have found the equivalents to Data::Dumper or -w in perl, or .inspect in ruby, but that doesn’t mean they don’t exist. I’m sure I’ll find them in time. And for those of you who suggest using the python interpreter for debugging, that doesn’t really work that well when examining a GUI app.
* self – it seems really redundant that inside a class method, the first parameter you pass is self- in other languages I’ve used, self is so prevalent inside a class that it’s implicitly expected, not explicitly stated.
* non-object old-style objects – When is an object not an object? When it’s an old style python object. Apparently the Python object setup is sorta broken- new-style objects are subclassed from an actual object class, while old-style aren’t. People complain that Perl’s OO was tacked on, but this seems far more kludgy.
* super – related to the above point- new-style objects can use super, which requires being told which parent class it should be looking at (since python supports multiple inheritance). The problem is old-style classes do not, so if you’re subclassing a module you didn’t write which itself is subclassed, you may have to dig to figure out if you can use super or not. Old-style classes use a different method- you call ParentClass.method() and pass it self as the first parameter. I think this probably ties into the kludgy way self is handled. For more on super, check out Python’s Super being Harmful (which goes way over my head).

* a line fails but it doesn’t say why, it just stops– I’ve run into this problem while trying to debug code- take the following code for example:

print "in workerthreadadd- will it add? 1"
searchResult = self.ldapServerObject.add_s(self.dn, self.modlist)
print "in workerthreadadd- will it add? 2"

The first line will print- the second one won’t. I don’t know what’s going on in that ldapServerObject- I didn’t write it, but there’s not stackdump or errors spewing out- it just stops. Now I have to go through and debug the object I’m calling to figure out why it’s dying. (This isn’t the actual code that gave me this problem. I can’t remember what was doing it previously.)

That said I’m still gonna keep at it- I didn’t learn perl overnight, nor ruby. I suspect python will take a while to grow on me. When I learned php it was great compared to java because it didn’t need to be compiled; when I learned perl it handled text and gui interfaces better than php; when I learned ruby it’s OO abilities were far greater than perl’s. I just have to figure out what it’s good at that ruby and perl aren’t.

update:
It looks like the old vs. new style objects will be gone in python 3.0- they’re planning on switching to new-style only. I think it’ll be a positive step overall, but after talking to the guys in #python, python 3.0 is much like Perl 6- it should be here sometime within the next decade, but I probably shouldn’t even narrow it down that much 🙂

BusyBusy

So I just realized that I’ve become uber busy as of late. My current side projects look like this:
– Warhammer Army builder for ChrisP written in Ruby (on Rails)
– reviving radio.morgajel.com and rebuilding my stream (RoR, again)
– SecretWord module and Logging for Ziggy (Perl + Po::Co::IRC)
– ImportLDIF plugin for Luma

This isn’t including the miniature painting, work, or Ian. The weird part is it feels nice to be busy- but only when it’s busy on my terms 🙂

Go to Top