Morgajel.net Welcome Guest | Login or Register

I was planning on simply republishing my previous svn article, but realized that it sucked compared to what I know now.

Prerequisites

I’ll presume you have the following things.
- a Linux machine
- subversion already installed

Terminology to Know

There are a few terms that get mangled if you’re coming from other types of source control. This is just to clear things up.
- Repository: the central storage place on the subversion server where your data is kept.
- Checked-out copy: Unlike VSS, saying something is checked out does not imply that the file is locked. Also referred to as a local copy; but bear in mind that it doesn’t contain *all* of the data of the actual Repository.
- commit: save the changes you made locally to the repository.

Mental Hurdles

All of the files stored in a subversion repository are stored in a meta-filesystem. Much like an ISO image sitting on your desktop is not simply a folder full of files, a subversion repository is not directly accessible when you open it. Instead, you’ll see the guts of the repository- DB files, hooks, locks, etc. Don’t go digging through there to manually change your files- it’ll break things.

Another important one is the meta-filesystem. The “inside” of your repository is just a big filesystem. Much like /bin and /home exist on a linux machine by convention, there are certain base-level directories you should create for convention’s sake. The first thing you should do with a new repository is create three new base level directories: /tags, /trunk and /branches. Your main development will take place in /trunk. Don’t worry about the other two at the moment.

When I refer to the root of the repository, I’ll often refer to it as / or root. This *IS NOT* your server’s root directory or the physical location of the repository- it’s the part of the meta-filesystem where /trunk, /tags and /branches reside.

Step 1: Creating the Repository

Creating a repository is fairly simple. Anyone can create a repository where ever they have write access. All they must do is run

$ svnadmin create ~/project1/


This should create an empty repository. This will demonstrate what I’m talking about when I say that your repository is a meta-filesystem:

morgajel@unicron ~ $ svn ls file:////home/morgajel/project1
morgajel@unicron ~ $ ls /home/morgajel/project1
conf dav db format hooks locks README.txt
morgajel@unicron ~ $ svn mkdir file:////home/morgajel/project1/trunk -m 'creating trunk'

Committed revision 1.
morgajel@unicron ~ $ svn ls file:////home/morgajel/project1
trunk/
morgajel@unicron ~ $ ls /home/morgajel/project1
conf dav db format hooks locks README.txt


Here you can see the repository was empty, then we created a directory called trunk (using a commit message to describe the change we made), then showing that the directory was in fact created. Do the same thing for /tags and /branches.

We now have a working repository!

Checking out the Repository

Creating a repository is fine, but using it would be much more… useful. Next you should check out a copy of your project. Under normal conditions you’ll only be checking out the /trunk, but you mileage may vary in different situations. Since this is on our local machine, we can use the file:// protocol. Other protocols exist, like http:// (webdav), svn:// (svnserve), and svn+ssh:// (svn over ssh), but you don’t have to worry about them right now.


morgajel@unicron ~ $ svn co file:////home/morgajel/project1/trunk my_project1
Checked out revision 3.


This should check out the empty /trunk directory to a local folder called my_project1. The only thing in this directory is a hidden .svn directory which holds the guts of your local copy repository info. It’s similar in function to the CVS directory in CVS. Unfortunately a nearly-empty directory isn’t much use, so let’s add some content.

Adding Content

So let’s add some content.

morgajel@unicron ~ $ cd my_project1/
morgajel@unicron ~/my_project1 $ mkdir -p lib bin share/docs
morgajel@unicron ~/my_project1 $ touch configure Makefile share/docs/README lib/foo.pm bin/widget.pl
morgajel@unicron ~/my_project1 $ svn status
? configure
? share
? lib
? bin
? Makefile


Here I created a bunch of directories and created some empty files. When I ran svn status, svn told me that there were 5 things it wasn’t versioning. Let’s add them.

morgajel@unicron ~/my_project1 $ svn add configure share lib bin Makefile
A configure
A share
A share/docs
A share/docs/README
A lib
A lib/foo.pm
A bin
A bin/widget.pl
A Makefile
morgajel@unicron ~/my_project1 $ svn status
A configure
A share
A share/docs
A share/docs/README
A lib
A lib/foo.pm
A bin
A bin/widget.pl
A Makefile

As you can see, it recursed down into share, bin and lib and added all the goodies inside of each directory. You can also see svn status shows these as well. Keep in mind they’re just slated to be added to the repository- they’re not added yet. Let’s go ahead and commit them.


morgajel@unicron ~/my_project1 $ svn commit -m "a bunch of empty files and directories"
Adding Makefile
Adding bin
Adding bin/widget.pl
Adding configure
Adding lib
Adding lib/foo.pm
Adding share
Adding share/docs
Adding share/docs/README
Transmitting file data .....
Committed revision 4.


Modifying Data

So suppose you’d like to modify these files, you you decide to move the README to the root of your local copy (~/my_project1/):


morgajel@unicron ~/my_project1 $ svn mv share/docs/README README
A README
D share/docs/README
morgajel@unicron ~/my_project1 $ svn stat
D share/docs/README
M bin/widget.pl
A + README


Notice that I used svn mv to move files rather than regular old mv- That’s to make sure svn is aware of the move and keeps the file history associated with the new file. You can also see bin/widget.pl now include some new info as well, and displays an M[odified] next to it. The + next to README shows that it copied the history over from it’s previous position. So what happens if we move a file without svn mv?

morgajel@unicron ~/my_project1 $ mv configure config
morgajel@unicron ~/my_project1 $ svn status
? config
! configure
D share/docs/README
M bin/widget.pl
A + README
morgajel@unicron ~/my_project1 $ mv config configure

You can see that svn panics(!) that configure has gone missing, and sees this new file called config that it’s currently not revisioning. It doesn’t know that they’re the same file.

Commit Messages

You’ve seen me use the -m flag a couple of times now- I’m using it to keep things flowing. If you don’t use it, you’re prompted in your favorite $EDITOR to create a commit statement, which includes the list of modified files. Using the -m flag is useful if you’re scripting commits (I use this when dumping and committing a nightly config file from our load-balancer).

Most of the time however, you’ll use your Editor. Make sure to keep your commit messages sweet and to the point- other’s will see them.


morgajel@unicron ~/my_project1 $ svn commit
[vim shows up, I enter the following text]
Small changes to demonstrate movements
- moved the README
- added shebang to widget.pl
[save and exit vim]
"svn-commit.tmp" 8L, 190C written
Adding README
Sending bin/widget.pl
Deleting share/docs/README
Transmitting file data .
Committed revision 5.
morgajel@unicron ~/my_project1 $ svn update
At revision 5.
morgajel@unicron ~/my_project1 $ svn log
------------------------------------------------------------------------
r5 | morgajel | 2007-11-20 15:55:21 -0500 (Tue, 20 Nov 2007) | 4 lines

creating trunk
------------------------------------------------------------------------


You’ll notice that revisions 2 and 3 aren’t listed- if you’ll remember correctly, they were used to commit the /tags and /branches directories. svn log only reports changes that affect the current target (in this case, ~/my_project1 which is a local copy of /trunk).

There’s a couple more tips and tricks I could go on about- if there’s any interest in this post maybe I’ll write some more about more advanced topics.

Search and replace is a great feature in most text editors, but what happens when you want to do more? Vim has a solution- recording macros. Suppose you have the following output from some ancient program that needs to be tweaked:


X1222 22323 2A22 3303 0000 3334esss test 123
X2222 22353 2A22 3303 0001 3334esss tacd 456
X3222 22383 2A22 3303 0010 3334esss fals 789
X4222 22393 2A22 3303 0011 3334esss true 012


It is doesn’t really matter what it is, this example is somewhat contrived. Suppose you needed to make the following changes for each line that starts with X:
* change the ID from X_222 to Y_223
* reverse the 4th and 5th fields
* copy the second character from the beginning and insert it before the last character of the line

If it were only 4 lines, you could handle this yourself, but it would be very tedious. Suppose rather than 4 lines, you had 400- it’d be much easier to automate it. The best way to take care of it would be with a macro:

[esc]qa
/^X[enter] i[delete]Y[right][right][right][delete]3
[esc]wwwdww[left]p
0[right]d[ctrl+v]y$[shift+p]
q

That right there is a MESS, but gets the job done- it’s not something you want to repeat for fear of a typo. Notice that the first characters you typed were qa: ‘q’ starts recording, and ‘a’ is the slot we’re using to store the macro. From here we record how *we* would make the changes, making sure to keep our keystrokes to a minimum. When we’re done, we stop the macro by pressing ‘q’ again.

To run our macro on the next line, press ‘@a’ to run the newly created ‘a’ macro- it should find the next line that starts with an X (notice the /^X in our first command) and run those commands to massage our text.

Remember how we were talking about 400 lines like this? Even at 2 characters each, that’s 800 characters to type which is still annoying. Here’s where the magic comes in- you can record macros of macros:

qb
@a@a@a@a@a@a@a@a@a@a
q

Now each time you run @b, you’ll run the a macro 10 times. A more efficient way to handle this would be to use

@a
398@@

the first one was done manually to record the macro, the second to play the macro, and the third to say run it 398 more times.

And there you go- a quick tour of recording macros. I’m sure there’s much more than what I’ve shown, but that’s enough to keep you busy.

I’ve been using and promoting open source for about 7 years now- it started back in 2000 when I began writing php and playing with apache. In 2003 I made the full switch to Linux. Since then I’ve produced several small little projects and put them under the GPL in hopes that it would help someone else. I’ve also written many useful utility articles and how-tos for my site to help spread what I’ve learned.

This past week has been a major milestone for me as I’ve joined the Luma team. For those of you not familiar, Luma is an LDAP browser and administration tool written in Python and QT. My capacity is limited at the moment, since I don’t really know python and am only remotely familiar with QT development- I have, however, been using luma for 2 years now for various ldap based projects.

I’ve already cleaned up and helped organize the bugtracker, and am currently working on an LDIF Importer plugin to feel my way around Luma’s API and python in general. I finally feel like I am giving back to the community.

If you have access to an LDAP server (be it AD, OpenLDAP, Domino, Novell’s, etc), give Luma a try and give us some feedback. We’re looking for beta-testers and python developers- Heck, if you wanted to just comment the code, that’d be great as well. Feel free to stop by #luma on freenode- we’re always minimally paying attention and can probably answer any questions you may have.

Diff is a handy little command used to compare two text files- useful if trying to determine what’s changed in different versions of files, used by subversion to show what files have been changed, and can even create patch files for updating sourcecode. So what are some of the more useful flags?

* -i lets us ignore any capitalization changes
* -b lets us ignore any spacing changes
* -B ignore blank lines
* -w just ignore all white spaces
* -q just say if the files are different
* -y side by side comparison
* -r recursively compare directories
* -d find a smaller set of changes
* -u unified format

I often use the unified format(-u) simply because I find the +/- more intuitive than >/< . The whitespace and capitalization ignoring is great if you change the indentation of a file or fix a comment's capitalization, but don't want to make a big deal of it. Another great use is comparing directories- for example, before upgrading apache, make a backup copy of /etc/apache, run the upgrade, then run diff -rq /etc/apache.bak /etc/apache to see a list of files that were modified. Once you get that list, you can use diff to compare the two versions on a more granular level. Not a great example, but I have been in situations were I needed to compare two directories to see what had changed.

So what are some of the more unique uses? You can use the -s flag to confirm two files are the same or exclude files from a recursive compare with -x pattern. You can also use stdin for one of the comparisons with cat foo|grep badstuff | diff - bar or cat foo|grep badstuff | diff bar -. You can even create a patch file with diff -Naur file.old file.new >file.patch.

If you have any other uses for diff, leave them in the comments below.

Route is one of these hate-inspiring, jaw droppingly obtuse programs that you always get the syntax wrong on. The purpose is simple enough- show and/or change the routing table. The most common uses are:

1. route - shows the current entries
2. route add - adds a new entry
3. route del - removes an entry
4. route flush- removes all entries

Checking out your Routes
The simplest use of route is to simply run route at the command line:

morgajel@p-nut ~ $ /sbin/route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.0.0     *               255.255.255.0   U     0      0        0 eth0
loopback        *               255.0.0.0       U     0      0        0 lo
default         192.168.0.1     0.0.0.0         UG    0      0        0 eth0

You’ll see 3 routes total in this example (which is very simple)- The first route points all 192.168.0.x traffic to the network card(eth0), the second points all loopback traffic (127.x.x.x) back to the local device (lo), and anything that doesn’t fit into either of those categories goes to the network card (eth0). You may wonder why that first route is in there if the default would just catch it anyways- you see, this allows multiple network cards to point traffic to different gateways or routers on the same network.
Take the next example:

morgajel@p-nut ~ $ /sbin/route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.100.0.0     10.100.0.1              255.255.0.0   U     0      0        0 eth1
192.168.7.0     192.168.7.1               255.255.0.0   U     0      0        0 eth0
loopback        *               255.0.0.0       U     0      0        0 lo
default         192.168.0.1     0.0.0.0         UG    0      0        0 eth0

all of your normal traffic would proceed to 192.168.0.1, but any traffic to 192.168.7.x would go to a different gateway(192.168.7.1) and all traffic to 10.100.x.x would go to a secondary network card(eth1) and be sent to yet another gateway (10.100.0.0). This would be useful if you were using a load balancing device like a Netscalar or F5, or if you had an internal network and a secondary DMZ’d network or something.

Adding and Removing Routes
if you’re manually setting up routes for a static IP, you’ll generally do something like

route add default gw 192.168.0.1

or remove it with

route del default
route del -net 192.168.0.0 netmask 255.255.0.0 eth1

Deleting routes is always the pain- the default route you can simply remove like the first example, but anything else needs the netmask specified and the -net flag used. If you want to see some more examples of routes, try

route -C

This will show you… uh, I guess the dynamic routes that have been recently used and cached. If you’re unsure about how to remove or add a route, you can run “route add” or “route del” without any addition parameters to see more options- I think the most I’ve used is something like

route add -net 10.100.32.0 netmask 255.255.248.0 gw 10.100.32.1 eth0

Final Thoughts
One thing that I learned while writing this is that there is an additional parameter called “reject” which you can append to the end of an add line to basically form a crude firewall to that route ( note- it’s not really a firewall, but it will reject packets). And of course, to get rid of any line you’ve added, you can change an add to a del and it’ll probably remove it (you may need to remove some add-specific parameters like reinstate or mss).

The manpage is pretty straightforward and has some decent examples- Looking at it, I’m not sure why I’ve had such problems with route in the past. Overall it seems pretty simple as long as you get the del syntax correct. Overall I think writing this helped me more than it’ll probably help you.

I recently wrote a nice little script in ruby for ldapifying new ubuntu servers- all the server needs is a ssh key set up for root, the rest is cake…

jmorg@util3:~/base_configs# ./ldapify -h
Usage: ldapify --install hostname [$options]
       ldapify --check hostname
       ldapify --uninstall hostname
        --install hostname           hostname to ldapify(foo.pub.local)
        --uninstall hostname         removes ldapification from hostname
    -a, --access_group access_group  access_group that has access to hostname
        --no_group_dn                No access_group limitations- use with caution.
    -c, --clusters x,y,z             clusters in which hostname belongs
        --aliases x,y,z              other aliased hostnames for the host
        --ignore-home                don't mount/unmount home
    -v, --verbose                    enable verbosity
    -q, --quiet                      silence all unneeded messages
    -h, --help                       Show this message

jmorg@util3:~/base_configs# ./ldapify --install log1.pub.local -a devboxes
backing up sources.list...
updating sources.list...
updating package list...
adding nfs entry to /etc/fstab...
Complete.
Mounting home, please wait...
complete.
installing debconf-utils...
patching debconf selections...
installing libnss-ldap ...
symlinking ldap.conf...
copy ssl cert and ldap.conf...
complete.
No Access group was given, using admin_only by default.
backup nsswitch.conf and pam.d files...
complete.
install nsswitch.conf and pam.d files...
complete.
ldap requires the manager password:
please verify the manager password:
store manpass...
installing sudo-ldap...

jmorg@util3:~/base_configs# ./ldapify --uninstall log1.pub.local
restore nsswitch.conf and pam.d files...
complete.
remove ssl cert and ldap.conf...
complete.
removing nfs entry...
complete.
unmounting home...
complete.
removing debconf-utils, libnss-ldap and libpam-ldap ...
removing ldap.conf symlinks...
removing sudo-ldap, restoring sudo...
retore sources.list...
updating package list...
ldap requires the manager password:
please verify the manager password:
jmorg@util3:~/base_configs#

So what all does it do?

• Sets up ldap authentication of user accounts
• mounts the nfs-based home directory
• Sets up ldap-based sudo rules
• Creates a host entry in the ldap server
• Adds an entry in the ldap server for the distro’s cluster and ldapified hosts cluster
• Can completely revert back to the original state

This script takes about 2:45 to run (mostly due to the 120 seconds of waiting for the /home dir to mount), and saves roughly half an hours worth of work. It’s not very stable (pre-ldapified boxes cause it to freak out when trying to re-install/remove) , but it will be a lot of help as we move towards ubuntu as our standard distro.

I have two requirements for a program being on this list: the first one is it has to be a utility- something scriptable or usable on the command line. The second is it also needs to have multiple arcane flags that I can write about, or just be so unknown that it’ll bring it to the attention of people that have never heard of it. Tee falls into the “never heard of it” group. It may only have two flags, but it’s useful nonetheless.

Tee splits a STDIN stream in two, sending one stream out output to a file, and the other to STDOUT. The data is identical; this just allows you to take a snapshot of what the data is like at a certain point a long piped command. For example, lets suppose you had a nice pretty command like this:

ls -lrt |awk '{print $6" "$7" "$8" "$3"  "$9}'|grep morgajel|tail -n 3

This provides you with the date, owner and filename of the 3 newest files in the current directory. Suppose you wanted a list for all the users as well as just morgajel’s latest three

ls -lrt |awk '{print $6" "$7" "$8" "$3"  "$9}'|tee all.txt|grep morgajel|tail -n 3

By adding the “tee all.txt” a copy of the stream at that point is diverted into a text file. you can view that later. The example is a bit fake, but you’ll eventually run into something needing this functionality. When you do, you can use tee. The only two real flags of interest are -a which appends data to the file rather than recreating it, and -i to ignore interrupts, but that could be really bad if you need to ctrl-c out of something…

either way, enjoy.

If you need to paste into vim from somewhere else, and your code has tabs or spaces in it, you’ll notice that vim may add extra tabs. see, vim doesn’t see it as a paste event, it sees it as “you typing really fast”- and one thing vim does will is auto-indent. The problem is when you paste, you don’t want auto-indentation because your code is already indented.

to temporarily turn off auto-indenting, try this from insert mode:

[esc]:set paste[enter]

go back to insert mode and you should be able to paste without the extra tabs.

Another well used mode is Visual Mode, which turns your cursor into a hilighter.
open a textfile with several lines of text ad move the cursor to the middle

switch from command mode to visual mode:

v

You’ll notice as you move the cursor around, you highlight different sections from the point you started to the point you left. you can press [esc] to return to command mode.

hilight a few lines of text from command mode:

[shift]v[upkey]

my adding the modifier [shift] when pressing V, you switch to ‘visual line mode’. This allows you to copy paragraphs easily.

hilight a block of text from command mode:

[ctrl]v[upkey][upkey][rightkey]

Now what good is visual mode? we” for deleting or replacing, of course! This is great when you have the following block of text:

           [option min="1"  max="10"   ][/option]
            [option min="11" max="19"   multiplier="10000" ]cp[/option]
            [option min="20" max="38"   multiplier="1000"  ]sp[/option]
            [option min="39" max="95"   multiplier="100"   ]gp[/option]
            [option min="96" max="100"  multiplier="10"    ]pp[/option]

and you’ve decided you no longer need min and max.
- move the cursor in command mode over the ‘m’ in ‘min’ on the first line
- press [ctrl]v[downkey][downkey][downkey][downkey]
- press the right arrowkey until the closing quote on “100″ is covered
- press ‘d’
all your text will be gone. But suppose you didn’t want the text to be gone, you wanted to replace it with something else?
- press ‘u’ and the text will reappear
- re-highlight it and press ‘c’ (text should disapepar)
- type your replacement string (use=”false”) and press [esc]
within a moment or two, use=”false” should jump up across all five lines

Visual modes are also useful for narrowing the scope of a search and replace, but I’ll get to that when I cover search and replacing.

Deleting in vim can be done several ways- in insert mode, the delete key and backspace key perform as you’d expect them to, but what if you want more?

delete the character to the left of the cursor:

[esc]d[left arrow]

delete the character to the right of the cursor:

[esc]d[right arrow]

deleting the current line from insert mode:

[esc]dd

deleting the current line and the one below from insert mode:

[esc]d[downkey]

deleting the current line and the one above from insert mode:

[esc]d[upkey]

deleting current character and 4 to the right:

d5[rightkey]

deleting current line and 2 below:

d2[downkey]

You’ll notice from those last two examples that deleting characters to the left and right include the current character in the count, but deleting lines above and below do not. Weird.