Open Source

Unfinished Drafts: Useful Utility: tar

This is another article that sat in the drafts folder for far too long- Last edited Feb 21st, 2006.

 

I fear writing about tar, and that is why I’m determined to finish it in this sitting, so it won’t fester and scare me off of this series. Why am I scared of writing about tar? Well, this is their flags list verbatim from the man page:

       [  --atime-preserve  ] [ -b, --blocking-factor N ] [ -B, --read-full-records ] [ --backup BACKUP-TYPE ] [ --block-com-
       press ] [ -C, --directory DIR ] [ --check-links ] [ --checkpoint ] [ -f, --file [HOSTNAME:]F ] [ -F,  --info-script  F
       --new-volume-script F ] [ --force-local   ] [ --format FORMAT ] [ -g, --listed-incremental F ] [ -G, --incremental ] [
       --group GROUP ] [ -h, --dereference ] [ --help ] [ -i, --ignore-zeros ] [ --ignore-case ] [ --ignore-failed-read  ]  [
       --index-file  FILE  ]  [ -j, --bzip2 ] [ -k, --keep-old-files ] [ -K, --starting-file F ] [ --keep-newer-files ] [ -l,
       --one-file-system ] [ -L, --tape-length N ] [ -m, --touch, --modification-time ] [ -M, --multi-volume ] [ --mode  PER-
       MISSIONS  ]  [  -N,  --after-date DATE, --newer DATE ] [ --newer-mtime DATE ] [ --no-anchored ] [ --no-ignore-case ] [
       --no-recursion ] [ --no-same-permissions ] [  --no-wildcards  ]  [  --no-wildcards-match-slash  ]  [  --null      ]  [
       --numeric-owner  ]  [  -o,  --old-archive, --portability, --no-same-owner ] [ -O, --to-stdout ] [ --occurrence NUM ] [
       --overwrite ] [ --overwrite-dir ] [ --owner USER ] [ -p, --same-permissions, --preserve-permissions ]  [  -P,  --abso-
       lute-names  ] [ --pax-option KEYWORD-LIST ] [ --posix ] [ --preserve ] [ -R, --block-number ] [ --record-size SIZE ] [
       --recursion ] [ --recursive-unlink ] [ --remove-files ] [ --rmt-command CMD ] [ --rsh-command  CMD  ]  [  -s,  --same-
       order, --preserve-order ] [ -S, --sparse ] [ --same-owner ] [ --show-defaults ] [ --show-omitted-dirs ] [ --strip-com-
       ponents NUMBER, --strip-path NUMBER (1) ] [ --suffix SUFFIX ] [ -T, --files-from F ] [ --totals   ]  [  -U,  --unlink-
       first ] [ --use-compress-program PROG ] [ --utc ] [ -v, --verbose ] [ -V, --label NAME ] [ --version  ] [ --volno-file
       F ] [ -w, --interactive, --confirmation ] [ -W, --verify ] [ --wildcards ] [  --wildcards-match-slash  ]  [  --exclude
       PATTERN  ]  [  -X,  --exclude-from  FILE  ]  [  -Z,  --compress,  --uncompress  ] [ -z, --gzip, --gunzip, --ungzip ] [
       -[0-7][lmh] ]

So it’s a bit overwhelming. The good news is there are two common uses for tar- creating tarballs and opening tarballs. This will be a majority of your interaction with it. You get all sorts of fun options with tar, such as using different compression libraries, but it’s still pretty straight forward.

Simple Archive

Tar produces tarballs, which in its simplest form is a bunch of data files run together into a larger file. in the following instance, -c means create, and -f means “create the following as a file called foo.tar”

tar -cf foo.tar bar/

This takes the bar directory and throws it all into a single file called foo.tar. Apart from some binary mojo to mark the separators between files, it’s almost as it all of the files were pasted end-to-end inside another file. if foo.tar is copied to another machine or place, you could untar the file with the following command:

tar -xf foo.tar

Again you see the -f flag, but the -c flag has been replaced by the extract flag, -x. This will create a directory called bar/ which will contain the contents identical to the original.

Compressing Archives

You also have the option of compressing tarballs in the process of creating them. There are three types of compression built into the version of tar I’m using: -Z, which uses the compress utility (ancient?); -z which uses gzip (old standard); and -j, which uses b2zip, which is good for compressing binaries (appears to be the new standard).

When creating a tarball that is compressed, it’s generally expected that you label it as such by appending the type to the filename, for example:

tar -cZf foo1.tar.Z bar1/
tar -czf foo2.tar.gz bar2/
tar -cjf foo3.tar.bz2 bar3/

Unless you have a specific reason, you’ll probably want to use bz2. You’ll probably never deal with a tar.Z file, but if you do, you’ll know how to deal with it. To uncompress these puppies, switch out the -c flag for the -x flag like we did in the previous example.

tar -xZf foo1.tar.Z
tar -xzf foo2.tar.gz
tar -xjf foo3.tar.bz2

One last option you may want to look at is -v. It will show you files as they’re being processed, and can be good for troubleshooting.

Unfinished Drafts: Useful Utility: cat

This article was originally written back on Feb 21st, 2006. While never completed, I thought it was worth sharing.

Cat is a very simple utility- so simple I debated added it to this list. There are however three really useful flags. I’ll try to write as much as I can about it so you don’t feel ripped off by this article. hrm… did that last sentence sound like filler? I swear it wasn’t meant to- that’s completely on accident.

So what is cat? Cat is a utility for printing the contents of a file or files to the screen. for example:

morgajel@FCOH1W-8TJRW31 ~/docs $ cat path.txt
paths


database admin

system admin

network admin management
morgajel@FCOH1W-8TJRW31 ~/docs $ 

You can also specify several files as well if you want to train them all together and pipe them to another utility.

morgajel@FCOH1W-8TJRW31 ~/docs $ cat foo.log bar.log baz.log |grep "Invalid user"> invalid_users.txt

Cat Flags

So there are three useful flags for cat. the first one is -n, which adds a linenumber to the output, like so:

morgajel@FCOH1W-8TJRW31 ~/docs $ cat path.txt -n
     1  paths
     2
     3
     4  database admin
     5
     6  system admin
     7
     8  network admin management
morgajel@FCOH1W-8TJRW31 ~/docs $

This can be useful when debugging source files. The next option is somewhat related; the -b option adds a line number, but only to non-blank lines. if you’re wanting to figure out for some reason what the 5th item is, not including blank lines, this is the way to go. Here’s an example of what it would look like:

morgajel@FCOH1W-8TJRW31 ~/docs $ cat path.txt -b
     1  paths


     2  database admin

     3  system admin

     4  network admin management
morgajel@FCOH1W-8TJRW31 ~/docs $

Notice how it only counted to 4? There were only 4 text lines. The final option that may or may not be of use is the -s flag, which smushes (that’s a technical term) blank lines together- it leaves single blank lines alone, but if there’s more than one blank line next to each other, it removes all except one. using our file above, watch what happens between “paths” and “database admin” in our example:

morgajel@FCOH1W-8TJRW31 ~/docs $ cat path.txt -s
paths

database admin

system admin

network admin management
morgajel@FCOH1W-8TJRW31 ~/docs $

Notice how there is only one blank line? That’s what -s does. if you’ve ever had a file where you’ve systematically removed text but not newlines and end up with a 500 line file with 20 lines of text, this can be useful for making it readable on a single page.

Well, that’s all I can really say about cat. If you have anything else to add, do so in the comments.

SOLVED: Halp, “screen” is broken on CentOS on my Linode instance!

So while migrating some content to a Linode instance, I attempted to fire up screen ran face first into a brick wall. For context, this was on an up-to-date CentOS 5.5 instance running screen-4.0.3-3.el5. Steps to reproduce:

  • logged in as my normal user
  • typed in “screen” and hit enter
  • console went blank for 5 seconds
  • previous content returned, along with “[screen is terminating]” and my normal command prompt.

Screen didn’t work for any regular users, however it did work for root.

I reinstalled the package, compared it to another centOS server, compared configs checked strace, etc. I could find no reason for this behavior.

Eventually I noticed I was using ttyp0 rather than pts/0 for my TTY; It was the only difference I could see between the two CentOS servers. Since I’d never seen ttyp (that I recall), it struck me as weird. I wasn’t sure if this was related to the fact that the broken instance was a xen guest or what.

It turns out, when Linode builds their CentOS image, they cut the following line out of /etc/fstab:

/dev/ptmx /dev/pts devpts defaults 0 0

That prevents pts from existing, causing you to fall back to the older ttyp, which is apparently incompatible with screen.

once I got that puppy mounted, everything was good.

Humble Bundle #2

So this year I managed to snag the Humble Bundle package;  If you’re not familiar with it, it’s a pack of Indie Games bundled together. There are five good reasons why you should consider getting it:

  • Name your own price ($5000 or $0.01 – your call)
  • You choose who the money goes to (charities and/or developers)
  • If you pay more than the average donation, you get Humble Bundle #1 from last year.
  • All games work on Windows, Mac and Linux
  • No DRM

Our Christmas funds were already drained, but Jackie and I agreed that we could spare $15.  I’m going through and trying the games now and thought I’d write up my notes on them:

Osmos

You’re a little bubble that propels itself by ejecting some of it’s mass in a given direction; the more mass you eject, the faster you go. As you float around, you see other blobs- some smaller, some larger. When one blob touches another, the smaller of the two is absorbed by the larger, bur as your mass grows, velocity slows. It becomes a balancing act of launching yourself to get blobs slightly smaller than yourself; if you eject too much, when you hit your target, you’ll be smaller and will be absorbed by your target (and thus dying).

There are two different types of play; “become the biggest” “absorb the ___”. Become the Biggest is played in a square or round sandbox, or it may be orbiting a massive blob; each has it’s own challenges.  Absorb the ___ involves trying to absorb moving targets- other AI blobs or repulsers (runs away from you). There may be more modes of gameplay, but I haven’t gotten to them yet.

I’ve only dug into it a bit, but it’s very addictive from what I’ve played. The only issue I’ve had is it occasionally freezes;  I have to alt-tab out of it and kill it when it happens.  It’s only happened twice, so it’s not that bad. I am grateful that the devs decided to package it as a .deb file so installing it is as simple as doing a sudo dpkg -i. Overall it’s an excellent game and easily worth $15 by itself. I could also see it ending up on the Wii due to it’s simplicity and controls.

Braid

Braid is one of those horrible horrible games like Tetris or Columns where the gameplay sinks into your brain and your start envisioning it everywhere you go.  The crux of the game is that your character can rewind time;  while it appears to be a side-scrolling platformer, you can’t actually die; so it’s closer to a puzzle game despite the running and jumping.

Each level you open up unlocks a new “feature” items that sparkle green ignore the reverse flow of time; even though you run backwards, the badguys keep moving, or the key stays where you left it. purple sparkles cause a shadow version of you to re-perform the actions you just performed; if you need to get in a gate but the purple lever is too far away and the gate closes before you get there, you can run to the gate, run to the lever, then pull it; rewind until you’re back to the gate and watch the shadow you run over and pull the lever. while he’s doing that, you can run through the gate.

I haven’t gotten very far, but the storyline is very interesting; we all wish we could rewind time and undo our mistakes, but eventually you will become a prisoner of other people’s expectations. Very fun, and again, easily worth $15. The controls are a little funky on my laptop, but this would be a great xbox arcade game. They just have a bin file which creates a dir on your system; it woulda been nicer if it had a .deb file so I could track it.

World of Goo

I’ve heard a lot about this game, but this is the first time I’ve seen it. It was actually part of Humble Bundle #1, but since I paid over the average ($7.53 at the time I purchased it), I got HB#1 as well. It’s a basic physics/building game; think tinkertoys made out of jello. The goal is to get these little goo balls into a tube; usually this is done by building some type of bridge or tower that the gooballs can climb. when you place a gooball near an existing structure, it forms a goo strut; if it only connects to one, it’ll flop like a joint. if you place it next to two, it’ll form a triangle which is relatively sturdy.

There are many different types of goo; some that drip like snot, some that can only be placed on a structure once, some that can’t form structures, etc. To top it off, if you build an awkward tower out of the gooball, the struts will bend and give way, causing the tower to collapse.

It’s a fun game, broken into many levels and chapters, but not really as enthralling as Braid. I could see this on the Wii for $15 or so by itself.

Revenge of the Titans

Revenge of the Titans is a simple tower defense game based around an alien invasion; the graphics are stylized  but simple, and the gameplay is nothing special. I’ve played probably close to 50 or so tower defense games;  It’s a little more fast paced than most and seems to just be one solid, continual wave.  It’s a well made and decent game, but I’m sorta burned out on the genre.

Samorost2

Samorost2 is a Myst-like puzzle game. Unlike the others so far, this one was flash based. The story revolves around a guy living on a small planet with his dog. When aliens stop to steal his pears and kidnap his dog, he follows them back to their base to rescue him. Each phase of gameplay involves a static screen with minor animations; you identify something on the screen that’s interactive, like a flower or ladder, and when you click on it, it triggers an animation. The gameplay is based on trying different combinations of events and timings. I was able to beat it in around an hour and a half, which makes it the shortest of the games so far. The artwork is great, but in the end it’s “just a flash game.”

Still worth a play for the artwork alone.

Lugaru HD

Lugaru appears to be a single-player  3rd person RPG, similar to WoW. The graphics are a bit rough, but it looks interesting nonetheless. Unfortunately, the controls aren’t great on a laptop, but if I had a mouse I’d probably be more interested in it. Walking through the tutorial, I got a feel for the combat, which seems crazy and fun. I’ll have to spend more time on it later.

So thats just a quick sample of what’s in the pack- I haven’t had time to play the other games. This pack is easily worth $15 several times over, and I sorta feel like I.  If you haven’t heard of the Humble Bundle, now is your chance to check it out. Hurry though, it’s not available after Christmas day.

Raw WinXP Virtualbox Partitions on a Thinkpad

New job, new laptop. Many utilities here are windows only, so it requires a bit of… effort… to get myself up and running efficiently. The solution to the windows problem is VirtualBox. I had set this up on my last laptop with little effort, but this time around required a bit more effort. Hopefully the instructions below will help others get up and running quickly.

Disclaimer– your laptop may catch on fire and explode (or worse) if you attempt this… or something.

We’ll be presuming that you’ve already resized your windows partition and have both a working Windows and Linux partition.

In Windows

Log into XP, grab MergeIDE.zip from Virtualbox’s site, extract and run it. It should be a quick flash and be done. (Note: I am not 100% sure this step is needed)

Create a new hardware profile and name it virtualbox. Make sure to set it as a choice during boot. Try rebooting into native windows once to ensure that it does offer you profile options.

In Linux

You’ll need the following packages installed (May differ for non-ubuntu systems):
mbr, virtualbox-ose, virtualbox-ose-qt

Create a stand-alone mbr file to use for booting (yes, you need the force flag):

install-mbr ~/.VirtualBox/WindowsXP.mbr --force

We’re presuming that your windows partition is /dev/sda1. In the below command, we are defining

  • a vmdk file (WindowsXP.vmdk)
  • which raw disk to read (/dev/sda)
  • which partition (1)
  • the new MBR file we just created

VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/WindowsXP.vmdk -rawdisk /dev/sda -partitions 1 -mbr ~/.VirtualBox/WindowsXP.mbr -relative -register

Note that you’ll need read/write access to that drive as your user, so you may want to figure out a cleaner/securer way to implement this, rather than adding your user to the disk group (which is very dumb and insecure). I would, but it’s working and I have more important things to do at the moment.

Another issue- apparently thinkpads report the drive heads and cylinders oddly (T410 for me and T60p in article), so we have to add some vmdk settings before virtualbox creates them incorrectly. Open ~/.VirtualBox/WindowsXP.vmdk and add the following at the bottom:

ddb.geometry.biosCylinders="1024"
ddb.geometry.biosHeads="240"
ddb.geometry.biosSectors="63"

The biosHeads appears to be the magic value- it seems to work if it’s set to 240, but the default is 255 (which fails).

Once you add those, start up virtualbox and check the virtual media manager, your new vmdk should be listed there. Once it’s confirmed, create a new virtual machine. Rather than creating a disk, select your vmdk as an existing disk.

After you finish, go the the VM settings->system and make sure the motherboard tab as io-apic  enabled (I also had PAE/NX enabled under processor and VT-x enabled under Acceleration).

Start the VM

There are several errors that could pop up. I’m sure there are plenty more that I stumbled across, but these were the two big ones:

  • a disk read error occurred, press ctrl+alt+del to restart – Caused by incorrect biosHeads- check and make sure it’s set to 240 (this was the fix for me, results may vary).
  • Complaint about kvm/vmx – Virtualbox does not like kvm. Uninstall qemu-kvm.

If things go well, it should flicker mbr in the corner, then go to the hardware profile selection screen. Select the virtualbox profile, and continue, then log in.

What follows is a half-hour of installing generic drivers and dealing with hardware specific auto start apps complaining that they won’t work on this installation. Windows will warn that the new drivers are not blessed, so be forewarned.

Once completed, at the top of the VM windows select Devices-> Install Guest Additions. This will download and mount an ISO, and windows will pop open a folder with the addition executables. Select the one best for you and run the installer. It will prompt you for video and mouse drivers (and trust me, you want them).

The final step is to shut down the windows VM, then reboot into the native windows partition to make sure it still works.  I did receive a few blue-screens before logging in at the beginning, but they appeared random and haven’t happened since.

And that’s all there is to it- simple, eh? Your windows partition should now run in native mode and vm mode.

Useful Utility: less

Less is more.

That’s the common joke about less- It provides the same functionality as the older utility, more; but oh, how much more than more!

Less allows you to easily scan backwards as well as forwards- something more is not too good at (though it is possible). Less also allows you to navigate with arrow keys, page up and page down, home and end.

Less provides a quick way to view files as well- Many editors (like vim) need to read an entire file first, and often create a temporary copy (for editing) of the file. This isn’t a problem until you try to browse a 56 gig log file of a rabid JBoss instance- if your root directory isn’t large enough, chaos ensues. Less circumvents this by only loading a page at a time. Yes- searches do take longer, but it beats crashing a system.

Useful Commands:

  • R: Repaints the screen, rereading the file from the disk. useful if the file is constantly changing (like a logfile)
  • / and ?: Just like in vim, used for searching forwards and backwards
  • g and G: Jumps to the top or bottom of the file, just like vim
  • :n and :p: Jump to the next/previous file (if you pass less more than one file)
  • q: quits. slightly important
  • ! [cmd]: runs a given command outside of less- useful if you can’t remember the output of another command and need a quick reminder (ifconfig, tail, route, ls, etc)

Useful Flags:

  • -N: Shows line numbers
  • -s: squeeze adjacent blank lines into one
  • -R: shows raw control characters, a.k.a. “turns color on!”

As usual, there are dozens of more options, flags, variables and behavior- you’d be surprised what less can do for you.

Free Jabber / XMPP clients for a Blackberry?

anyone know of any good jabber clients for the blackberry? I’ve tried a couple with little luck, and most of them cost more than I can afford for this test. Features required

  • Must run on BlackBerry 8703e v4.1.0x
  • Connection server can be configured differently than jid address (i.e. you@morgajel.net for jid, jabber.morgajel.net for connection server.) This rules out Mobber as far as I can tell
  • Requires SSL/TLS
  • Non-strict cert checking

Let me know if you have any suggestions.

Intro to Vim Tip #6 (Multiple viewports)

One feature of vim I don’t use enough is the ability to split the screen and view multiple files at once. I use this feature all the time when I use vimdiff, but not really any other time. I thought I’d take a moment to lay out some uses of it (thank Linux.com for the reference):

From Command Mode:

  • :sp splits the screen horizontally
  • :vsp splits the screen vertically
  • Ctrl-w Ctrl-w moves between viewports
  • Ctrl-w [right arrow] moves active viewport 1 to the right
  • Ctrl-w [left arrow] moves active viewport 1 to the left
  • Ctrl-w 3[left arrow] moves active viewport 3 to the left
  • Ctrl-w q will close the active window.

and remember that you can open a file with 😮 filename in the newly created viewport

Shutup Shutup Shutup!

die you wretched pc speaker beep that is so loud when I use the find bar in firefox or tab complete in the cli and nothing is found!

  rmmod pcspkr
  echo blacklist pcspkr >> /etc/modprobe.d/blacklist

aaah, glorious silence.

Request Tracker 3.6.5 broken after updating Cent OS

Can’t locate object method “seek” via package “File::Temp” at /usr/lib/perl5/site_perl/5.8.8/MIME/Parser.pm line 816.

The underlying problem is perl was updated and overwrote the “correct” version of File::Temp that you probably installed when setting up RT and forgot about. To fix this issue


cpan install File::Temp
/etc/init.d/httpd restart

MAKE SURE TO RESTART APACHE! I didn’t, and it cost me probably 2 hours of screwing around with it.

I’m posting this because
http://www.nabble.com/RT-3.6.5-and-Sendmail-error-and-looks-like-perl-error-td15989015.html
Didn’t really mention what the final working solution was.

Go to Top