Linux

Unicron

So I got the new hardware parts, so I figured I might as well document the process. I’ve decided on the name Unicron because it’s the biggest PC I’ve ever owned. I’ve been waiting over a year to really rebuild my servers- since I ran out of ide connectors in Pablo, I knew something had to change.

The new case is MASSIVE- it’s sitting next to jackie’s machine and is a good 6″ longer and 3″ higher. processor went in easy, as did the ram. motherboard went in fine, and the harddrive rails (with rubber footies to prevent vibration noise ) made it a painless job of inserting drives and wiring them up.

The first problem I noticed right away- the Ars Technica review said it came with a power supply, but apparently I didn’t buy it from the same vendor, and hence ran into roadblock #1. I also noticed that, rather carelessly, I didn’t verify there was an onboard video card. Having only PCIe and PCI slots, none of my AGP cards would fit. After a bit of digging, I found an old 8mb Trident card from the late 90’s and popped it in. it won’t run a gui installer, but I don’t need it 🙂

The next real problem was that Ubuntu 6.10 doesn’t recognize sata hardware raid- it’s seeing all 4 drives individually. This is probably going to be the showstopper. I’m pretty sure that once installed, it’ll handle them fine, but it leads me to a chicken and egg scenario. I have a couple of possibilities:

1) I need to figure out which Linux raid driver this chipset uses and load it into the kernel, then re-detect the drives. This seems like the most obvious fix, but the real question is “why didn’t it already do this?” My fear is there is no Linux driver for the hardware raid.

2) throw an IDE drive in, install to it, then run the installer from the fully loaded Ubuntu install rather than the trimmed down install disk. the one flaw with this plan is there is only one IDE chain, meaning the cdrom and IDE HD have to play nicely during the install. So far they haven’t- booting of the install CD gives me errors about hdb not being ready, and then booted every so slowly to the first set of prompts. I had to head to work before I even got to the partitioning section.

I’ll hopefully get a chance to try more this weekend. Updates to follow.

UPDATE: the board is an ASUS M2N-E, so if you know how to get raid5 working in ubuntu, let me know.

UPDATE 20070331: you ever notice how when you look at something for months and it never changes, you forget the details? I noticed (after getting a full night’s sleep) that there WAS a wait to set up software raid in Ubuntu’s installer, I was just blind. So after talking to shaldannon and the trilug guys, I decided to go with the following partition scheme:

200meg (md0), raid 1 (sda1, sdb1)
1gig (md1), raid 1 (sda2, sdb2)
1.5Tb (md2), raid 5 (sda3, sdb3, sdc2, sdc2)
1gig (md3), raid 1 (sdc1, sdd1)

Once that was done, the installation was a breeze. Ubuntu is now installed and I’m slowly but surely re-implementing everything.

Useful Utility: diff

Diff is a handy little command used to compare two text files- useful if trying to determine what’s changed in different versions of files, used by subversion to show what files have been changed, and can even create patch files for updating sourcecode. So what are some of the more useful flags?

* -i lets us ignore any capitalization changes
* -b lets us ignore any spacing changes
* -B ignore blank lines
* -w just ignore all white spaces
* -q just say if the files are different
* -y side by side comparison
* -r recursively compare directories
* -d find a smaller set of changes
* -u unified format

I often use the unified format(-u) simply because I find the +/- more intuitive than >/< . The whitespace and capitalization ignoring is great if you change the indentation of a file or fix a comment's capitalization, but don't want to make a big deal of it. Another great use is comparing directories- for example, before upgrading apache, make a backup copy of /etc/apache, run the upgrade, then run diff -rq /etc/apache.bak /etc/apache to see a list of files that were modified. Once you get that list, you can use diff to compare the two versions on a more granular level. Not a great example, but I have been in situations were I needed to compare two directories to see what had changed. So what are some of the more unique uses? You can use the -s flag to confirm two files are the same or exclude files from a recursive compare with -x pattern. You can also use stdin for one of the comparisons with cat foo|grep badstuff | diff - bar or cat foo|grep badstuff | diff bar -. You can even create a patch file with diff -Naur file.old file.new >file.patch.

If you have any other uses for diff, leave them in the comments below.

Useful Utility: route

Route is one of these hate-inspiring, jaw droppingly obtuse programs that you always get the syntax wrong on. The purpose is simple enough- show and/or change the routing table. The most common uses are:

  1. route – shows the current entries
  2. route add – adds a new entry
  3. route del – removes an entry
  4. route flush– removes all entries

Checking out your Routes
The simplest use of route is to simply run route at the command line:

morgajel@p-nut ~ $ /sbin/route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.0.0     *               255.255.255.0   U     0      0        0 eth0
loopback        *               255.0.0.0       U     0      0        0 lo
default         192.168.0.1     0.0.0.0         UG    0      0        0 eth0

You’ll see 3 routes total in this example (which is very simple)- The first route points all 192.168.0.x traffic to the network card(eth0), the second points all loopback traffic (127.x.x.x) back to the local device (lo), and anything that doesn’t fit into either of those categories goes to the network card (eth0). You may wonder why that first route is in there if the default would just catch it anyways- you see, this allows multiple network cards to point traffic to different gateways or routers on the same network.
Take the next example:

morgajel@p-nut ~ $ /sbin/route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.100.0.0     10.100.0.1              255.255.0.0   U     0      0        0 eth1
192.168.7.0     192.168.7.1               255.255.0.0   U     0      0        0 eth0
loopback        *               255.0.0.0       U     0      0        0 lo
default         192.168.0.1     0.0.0.0         UG    0      0        0 eth0

all of your normal traffic would proceed to 192.168.0.1, but any traffic to 192.168.7.x would go to a different gateway(192.168.7.1) and all traffic to 10.100.x.x would go to a secondary network card(eth1) and be sent to yet another gateway (10.100.0.0). This would be useful if you were using a load balancing device like a Netscalar or F5, or if you had an internal network and a secondary DMZ’d network or something.

Adding and Removing Routes
if you’re manually setting up routes for a static IP, you’ll generally do something like

route add default gw 192.168.0.1

or remove it with

route del default
route del -net 192.168.0.0 netmask 255.255.0.0 eth1

Deleting routes is always the pain- the default route you can simply remove like the first example, but anything else needs the netmask specified and the -net flag used. If you want to see some more examples of routes, try

route -C

This will show you… uh, I guess the dynamic routes that have been recently used and cached. If you’re unsure about how to remove or add a route, you can run “route add” or “route del” without any addition parameters to see more options- I think the most I’ve used is something like

route add -net 10.100.32.0 netmask 255.255.248.0 gw 10.100.32.1 eth0

Final Thoughts
One thing that I learned while writing this is that there is an additional parameter called “reject” which you can append to the end of an add line to basically form a crude firewall to that route ( note- it’s not really a firewall, but it will reject packets). And of course, to get rid of any line you’ve added, you can change an add to a del and it’ll probably remove it (you may need to remove some add-specific parameters like reinstate or mss).

The manpage is pretty straightforward and has some decent examples- Looking at it, I’m not sure why I’ve had such problems with route in the past. Overall it seems pretty simple as long as you get the del syntax correct. Overall I think writing this helped me more than it’ll probably help you.

New OS: Kubuntu 6.10

Ok, trying out Kubuntu on my new work laptop and I’m liking it quite a bit- the only problems I’ve had so far are with Hibernate (which I think is self- inflicted) and wireless stuff. I’ve figured out the wireless stuff and wanted to mention it for the people out there having the same troubles as I did. First up, a little info on my setup:

Model: IBM T42 (note, not the T42p, which is awesome, the crappier model)
OS: Kubuntu 6.10
Wireless NIC: Intel Corporation PRO/Wireless LAN 2100 3B Mini PCI Adapter (rev 04)
Wireless Driver: ipw2100 1.3

Wireless

There are a lot of choices for network setup, and since I use WPA2 with PSK, I’m going to document that.

in /etc/network/interfaces:
auto eth1
iface eth1 inet dhcp
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf.local

in /etc/wpa_supplicant/wpa_supplicant.conf.local
ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=0
eapol_version=1
ap_scan=1
fast_reauth=1
#For WPA-PSK network
network={
ssid="myssid"
key_mgmt=WPA-PSK
proto=WPA2
pairwise=TKIP
group=TKIP
psk="my big secure passphrase"
}

/etc/modprobe.d/ipw2100.modprobe
options ipw2100 disable=0

and that was it- I spent time tinkering with all sorts of things, but I think this was all there was. If I get brave and re-do it next weekend, I’ll make sure to fully document the process.

Hibernate

While trying to get wireless working, I noticed that hibernation stopped working- I’m not sure what I did, but I’m pretty sure it’s my fault- it WAS working originally.

UPDATE
DOH, it helps if the swap partition is mounted so hibernation has someplace to store the data! it’s all working good now.

Useful Utility: tee

I have two requirements for a program being on this list: the first one is it has to be a utility- something scriptable or usable on the command line. The second is it also needs to have multiple arcane flags that I can write about, or just be so unknown that it’ll bring it to the attention of people that have never heard of it. Tee falls into the “never heard of it” group. It may only have two flags, but it’s useful nonetheless.

Tee splits a STDIN stream in two, sending one stream out output to a file, and the other to STDOUT. The data is identical; this just allows you to take a snapshot of what the data is like at a certain point a long piped command. For example, lets suppose you had a nice pretty command like this:

ls -lrt |awk '{print $6" "$7" "$8" "$3"  "$9}'|grep morgajel|tail -n 3

This provides you with the date, owner and filename of the 3 newest files in the current directory. Suppose you wanted a list for all the users as well as just morgajel’s latest three

ls -lrt |awk '{print $6" "$7" "$8" "$3"  "$9}'|tee all.txt|grep morgajel|tail -n 3

By adding the “tee all.txt” a copy of the stream at that point is diverted into a text file. you can view that later. The example is a bit fake, but you’ll eventually run into something needing this functionality. When you do, you can use tee. The only two real flags of interest are -a which appends data to the file rather than recreating it, and -i to ignore interrupts, but that could be really bad if you need to ctrl-c out of something…

either way, enjoy.

Ruby on Rails

So I’ve had this on again, off again thing with ruby for a while now. Since I first started playing with ruby it got pretty big with rails, and I completely missed that boat. Well, now I’m playing with rails and it’s fairly interesting once you get it up and running. I picked up the O’Reilly book Ruby On Rails and have been walking through it’s Photo project. I’ve went so far as to even throw it in a subversion repository in case I pooch something.

One thing I really like so far is the Scaffolding system- once you create an object model (say, a Photo) and have it generate the table to store it in, it can auto-generate the web interface to allow you to create/edit/delete entries without having to muck with it. The coolest part is if you make any DB changes, the interface is automatically updated. That was something I’ve always hated- updating interface code to reflect DB changes.
At DP I wrote a minor system to do something like that in ASP, but it was still pretty crappy (and dangerous). Dendrite had created a system at SPX called Skel which was hideous (his words), but did sorta the same thing. The main project I worked on there (before it was cancelled) was a replacement for Skel that was very similar to this, but for perl. Now I really wish I woulda kept up with ruby because had I known of rails, I could have essentially ported it to perl and saved a lot of time.

I’m hoping that the rest of rails is as cool as this scaffolding code. It’s taking me a while (5 hours broken up) to wrap my head around it, but it’s finally starting to make sense. I’m looking forward to finding out more about it.

Udev + wacom on gentoo: dynamic links

I have a wacom drawing pad, and one thing that’s always bothered me was the whole way linux handled usb items- depending on the order they were plugged in, they would be given different names- sometimes my keyboard would be “/dev/input/event1”, sometimes my wacom or mouse.

Udev was made to get around these issues, but I’ve been to distracted to give it much thought. lately I’ve been getting the following error messages with my ruby projects:

X Error: BadDevice, invalid or uninitialized input device 165
  Major opcode:  147
  Minor opcode:  3
  Resource id:  0x0
Failed to open device
X Error: BadDevice, invalid or uninitialized input device 165
  Major opcode:  147
  Minor opcode:  3
  Resource id:  0x0
Failed to open device

and figure something screwy with my usb is the culprit. So with my sights set on an acceptable scapegoat, I delved into udev. I found a great walkthrough for writing udev rules that gave me exactly what I needed. 20 minutes later, I had the following gem:

BUS=="usb", DRIVER="wacom", SYMLINK+="wacom"

That’s it. Now my wacom will always be /dev/wacom, and I can configure X to always use it. not sure if this will fix my error, but I’ve been meaning to do this for a while, so I’m glad I finally got around to it.

Pissed off vs. Pissed on

Today I find myself a little of both.

I was called into the confrence room today and told that the company I was contracted to requested that today be my last day.

Yes, I am officially unemployed.

Fortunately I was laid off, not fired- turns out business wasn’t as good as they were hoping and they can’t afford someone to try and fix their broken infrastructure. Since they fired a developer a few weeks ago, and the lead developer put in his 1 week notice today, I can’t say it’s suprising- if they’re getting rid of me after losing two other people, they must be hemmoraging money pretty badly.

The worst part is, not only did I see this coming, but I worked my ass off there knowing it was coming. I was hoping they’d at least have the decency to give me some severance pay, but apparently since I’m just a contractor they don’t feel the need. They did however give me a nice 1 paragraph recommendation letter- I guess that makes up for laying off a guy with a kid on the way who they shouldn’t have hired if they knew they wouldn’t be able to afford him.

I’m pretty pissed off right now because I feel like I made a mistake coming back to michigan- I won’t be able to find work around here since I’m pretty specialized (michigan is Microsoft territory, and at this point I’m pretty specialized in linux) and the economy is crap to boot.

The worst part about all of this is jackie and I went so far into debt to move back and we still haven’t dug out. Now jackie’s pregnant and my hopes for employment are slim to none, and we couldn’t even afford to move somewhere else if we wanted to because our credit cards are maxed out from the last move.

I feel pretty bad for the other sysadmin because they’re gonna crap back on him again and he’s just gonna take it. The Devs are screwed because I did a lot to make their development environment more professional- seperate user accounts, testbeds, standardized development/beta/production servers, version control, etc- most of that will go down the shitter because it will all be neglected if it’s even kept at all. the other sys admin will probably either format them or ignore them.

It’s been a hell of a day, and I need to take a shower still after my walk- then I’m going to bed. If you know of anyone looking to hire a Linux Sys Admin or a perl/php/ruby developer (or some combination of either), send them my way.

stupid dell raid card..

so the devs are having a hell of a time with the new dev server- it’s constantly locking up on certain threads for 30-45 seconds. apache mainly, but occasionally grep and other things. It appears to be completely random. I’ve been pulling my hair out trying to find the problem, and I think I got it nailed down thanks to a post on the gentoo forums (http://forums.gentoo.org/viewtopic-t-189180-highlight-writethru.html) that said to change some settings in the raid card’s firmware… I tested this on the prod machine since it’s not in use yet(they’re both dell 2850’s with the lsi megaraid cards) and benchmarked before and after with bonnie++… here are the results(- is original, + is new):

 Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-
 Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
 Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
-prod             7G   331  98 27139  10 17554   4  1156  99 65858   7 371.8   5
-Latency               104ms   41538ms     804ms   19824us   46554us     631ms
+prod             7G   335  98 54977  23 27923   7  1075  99 61596   6 342.9   4
+Latency             76755us     331ms     590ms   12104us   24805us     414ms

Notice the second column under Latency? 41538ms vs 331ms – a 40 second latency when using the dell default settings…

I think this is where my lag issue is coming from… here’s hoping.
I’ll report back when I make the change on dell and see if the devs are still having lag problems.

Update-
so I switched the write policy to ‘write-thru’ instead of write back and changed the readpolicy to normal rather than adaptive… changed the benchbark, but not the problematic behavior- well, actually now, ALL of the devs are locking up at the same time…

spring cleaning

I have a habit of moving stuff to my website’s root directory and sending temp links to people. I’ve recently went about cleaning it out (which I do about once a year) and found some interesting bits… Here’s a job posting I wrote for a position we were hiring for at a previous employer:

We're currently looking for a Mid-Upper level Perl Web Application
Developer.  If FrontPage or Dreamweaver is on your resume, please do not
reply. Degree In Computer Science a plus.


Needs:
        - over 2 years experience with Perl.
        - Strong Web Development background
        - Strong Postgresql experience
        - Strong Linux Experience
        - Ability to learn new Technologies
Wants:
        - Flash experience
        - Java experience
        - Debian experience
        - Strong understanding of related web technologies (CSS, javascript,etc)

nice to have:
        - php experience
        - mysql experience
        - XML experience
        - Subversion experience

Please include a list of any open source projects you have contributed to.
Go to Top