Utility

To qualify for this section, the utility has to be command line based, useful, and either not well known, or have a bunch of little-used options. Cut, tee and ls are good examples, while rm is not because it’s too simple.

I’ll try to review at least one of these a week.

Useful Utility: tee

I have two requirements for a program being on this list: the first one is it has to be a utility- something scriptable or usable on the command line. The second is it also needs to have multiple arcane flags that I can write about, or just be so unknown that it’ll bring it to the attention of people that have never heard of it. Tee falls into the “never heard of it” group. It may only have two flags, but it’s useful nonetheless.

Tee splits a STDIN stream in two, sending one stream out output to a file, and the other to STDOUT. The data is identical; this just allows you to take a snapshot of what the data is like at a certain point a long piped command. For example, lets suppose you had a nice pretty command like this:

ls -lrt |awk '{print $6" "$7" "$8" "$3"  "$9}'|grep morgajel|tail -n 3

This provides you with the date, owner and filename of the 3 newest files in the current directory. Suppose you wanted a list for all the users as well as just morgajel’s latest three

ls -lrt |awk '{print $6" "$7" "$8" "$3"  "$9}'|tee all.txt|grep morgajel|tail -n 3

By adding the “tee all.txt” a copy of the stream at that point is diverted into a text file. you can view that later. The example is a bit fake, but you’ll eventually run into something needing this functionality. When you do, you can use tee. The only two real flags of interest are -a which appends data to the file rather than recreating it, and -i to ignore interrupts, but that could be really bad if you need to ctrl-c out of something…

either way, enjoy.

Intro to Vim Tip #4 (Pasting)

If you need to paste into vim from somewhere else, and your code has tabs or spaces in it, you’ll notice that vim may add extra tabs. see, vim doesn’t see it as a paste event, it sees it as “you typing really fast”- and one thing vim does will is auto-indent. The problem is when you paste, you don’t want auto-indentation because your code is already indented.

to temporarily turn off auto-indenting, try this from insert mode:

[esc]:set paste[enter]

go back to insert mode and you should be able to paste without the extra tabs.

Intro to Vim Tip #3 (Visual Mode)

Another well used mode is Visual Mode, which turns your cursor into a hilighter.
open a textfile with several lines of text ad move the cursor to the middle

switch from command mode to visual mode:

v

You’ll notice as you move the cursor around, you highlight different sections from the point you started to the point you left. you can press [esc] to return to command mode.

hilight a few lines of text from command mode:

[shift]v[upkey]

my adding the modifier [shift] when pressing V, you switch to ‘visual line mode’. This allows you to copy paragraphs easily.

hilight a block of text from command mode:

[ctrl]v[upkey][upkey][rightkey]

Now what good is visual mode? we” for deleting or replacing, of course! This is great when you have the following block of text:

           [option min="1"  max="10"   ][/option]
            [option min="11" max="19"   multiplier="10000" ]cp[/option]
            [option min="20" max="38"   multiplier="1000"  ]sp[/option]
            [option min="39" max="95"   multiplier="100"   ]gp[/option]
            [option min="96" max="100"  multiplier="10"    ]pp[/option]

and you’ve decided you no longer need min and max.
– move the cursor in command mode over the ‘m’ in ‘min’ on the first line
– press [ctrl]v[downkey][downkey][downkey][downkey]
– press the right arrowkey until the closing quote on “100” is covered
– press ‘d’
all your text will be gone. But suppose you didn’t want the text to be gone, you wanted to replace it with something else?
– press ‘u’ and the text will reappear
– re-highlight it and press ‘c’ (text should disapepar)
– type your replacement string (use=”false”) and press [esc]
within a moment or two, use=”false” should jump up across all five lines

Visual modes are also useful for narrowing the scope of a search and replace, but I’ll get to that when I cover search and replacing.

Intro to Vim Tip #2 (deleting)

Deleting in vim can be done several ways- in insert mode, the delete key and backspace key perform as you’d expect them to, but what if you want more?

delete the character to the left of the cursor:

[esc]d[left arrow]

delete the character to the right of the cursor:

[esc]d[right arrow]

deleting the current line from insert mode:

[esc]dd

deleting the current line and the one below from insert mode:

[esc]d[downkey]

deleting the current line and the one above from insert mode:

[esc]d[upkey]

deleting current character and 4 to the right:

d5[rightkey]

deleting current line and 2 below:

d2[downkey]

You’ll notice from those last two examples that deleting characters to the left and right include the current character in the count, but deleting lines above and below do not. Weird.

Intro to Vim Tip #1

Vim is a great tool, but using is can be a pita in the beginning- hence, we go through the basics. There are several command modes, but we’ll only discuss a few at first: Command Mode and Insert Mode.

Command mode is used to perform actions like saving, searching, etc. Insert mode is used to insert and delete text. You’ll be switching between them a lot.

Open a file from the cli:

  $ vim foo.php

change to insert mode from Command mode:

  press 'i'

change to command mode from insert mode:

  press [esc]

save from insert mode:

  press [esc]:w[enter]

save and quit from insert mode:

  press [esc]:wq[enter]

quit and do NOT save from insert mode:

  press [esc]:q![enter]

An there you have it.

recursing vimrc

I use vim a lot. a *LOT*. One thing that really annoys me is page width. When I’m writing code, I like to have a width set to 78 characters. But in some instances, say when I’m working on a book, I like the width set to 90 characters since it’s easier to read. This got me thinking… if I had vim check the current directory for a config, I could have custom configs for different directories. so in ~/.vimrc, I added

if filereadable("./.vimrc")
        source ./.vimrc
endif

and this works great, with one exception… what if i’m sitting in my home directory?

morgajel@p-nut ~ $ vim .vimrc
Error detected while processing /home/morgajel/.vimrc:
E169: Command too recursive
Hit ENTER or type command to continue

it goes into a recursive loop… so now my question is, how do I prevent that? I’m presuming that “changing the if statement to also check and make sure the current directory isn’t home” is the way to go, but I’m not seeing much documentation on how to go about doing it, or at least I’m not looking for the correct thing. Any suggestions?

Useful Utility: sed

Sed is a powerful utility for going regexes on the fly. Regular Expressions (regex) are beyond the scope of this artcle, but I’ll try to write one later. As I go, I’ll explain the regexes I use, but you really should learn about them because they’re handy as hell in many different utilities.

First up, we’ll use a simple example of a regular expression. Suppose for some reason, you want a list of the Input Device names used by Xorg, and plan on piping it into another script later on. We can look at the Xorg log and find a lot of data, but locating the Input Devices is like searching for a needle in a haystack.
Our first step is to cat the file, then grep for the phrase “Input Device”. This provides us with

morgajel@FCOH1W-8TJRW31 ~/docs $ cat /var/log/Xorg.0.log \ 
| grep "Input Device"
(**) |-->Input Device "Mouse1"
(**) |-->Input Device "Keyboard1"

Which tells us what we need to know, but your script is more picky about the format. This is where sed comes in to play. we can tell sed to throw away everything before and including Input Device

morgajel@FCOH1W-8TJRW31 ~/docs $ cat /var/log/Xorg.0.log \
| grep "Input Device"|sed -e "s/.*Input Device //"
"Mouse1"
"Keyboard1"

Lets break this down. Sed can accept regular expressions one of two ways, either through the -e flag, as shown above or through a file using the -f flag. The advantage of using the -e flag is it’s quick and simple, whereas the advantage of using the -f flag is you can build compound regular expressions executing again one at a time, in order.

Now, what’s the regex doing? it’s saying switch(s) the first parameter(/.*Input Device /) with the second parameter (//, or nothing) the first time you see it in every line. the “.*” means “match every character” before “Input Device”. Note that we’re replacing “every character before Input Device” AND “Input Device” and the space after “Input Device”, and replacing it with nothing.

The good news is we’ve gotten it down to the base information we need- the bad news is it’s still surrounded by double quotes, which your script doesn’t want. How can we fix this? by using another sed to remove the quotes!

morgajel@FCOH1W-8TJRW31 ~/docs $ cat /var/log/Xorg.0.log \
| grep "Input Device" |   sed -e "s/.*Input Device //" |   sed -e "s/\"//g"
Mouse1
Keyboard1

Lets bread this one down. There are two important things to learn here- ” is a special character and needs to be escaped since we’re writing this on the command line, so it has a “\” in front of it so the regex isn’t closed prematurely. one of the drawbacks of using the -e flag is properly escaping characters not only for regex, but for bash as well. The next thing to note is the trailing “g” at the end of the regex. the g stands for global, and says to replace all instances of the match with the replacement term (which is again, nothing).

Alright, that was the simple example, ready for the more difficult ones? Too bad, I’m tired of explaining regex. Instead I’ll focus back on what sed can do. If this is a command we’re going to be running often, it might benefit us to move the regexes to a file- I’ll call mine “myscript”, because I’m original like that:

morgajel@FCOH1W-8TJRW31 ~ $ cat myscript
s/.*Input Device //
s/"//g

Note that the double quote is not escaped. Our command to use this file goes like this:

cat /var/log/Xorg.0.log|grep "Input Device"|sed -f myscript

Much cleaner, if you ask me- but then again, you need to create a file to do so, so it’s a trade off.

Here’s an ugly ugly regex, but it has some useful things in it. Suppose we wanted a list of shells and the users who used those shells, in alphabetical order. One way to do that (although there are much better tools for this sorta thing,) is with sed.

 cat /etc/passwd|sed -e "s/^\([^:]*\).*:\([^:]*\)$/\2\t\t\1/"|sort
[edited for length and security]
/bin/false              squid
/bin/false              sshd
/bin/false              uucp
/bin/false              vpopmail
/bin/false              xfs
/bin/sync               sync
/sbin/halt              halt
/sbin/shutdown          shutdown

There are some more important things in this one as well- this is a good lesson in building expressions.

  • “^” tells sed that it must start at the beginning of the line.
  • “\(” and “\)” are bash-escaped parenthesis that tells sed “save this bit for later
  • “[” to “]” represents a range- normally, you would see it in the context of “[a-z]” or “[13579]” matching a single lowercase a-z and/or a single odd digit, respectively.
  • “^:” the carrot has a different function when it a range- it negates it, basically saying “any character but what follows”, in this case, a colon( : )
  • “^\([^:]*\)” put together means “match zero or more characters starting at the beginning of the line until you reach a colon- oh, and save that glob for later.”
  • “$” is similar to the first use of “^”- it matches the end of the line.
  • “\2” represents a glob that we set aside with the parenthesis- notice they start counting in order: “\1” picks up the first glob (username) and \2 picks up the second glob (shell name). Notice that most of the line was ignored- “.*:” hiding in the center of the regex matches everything else, but since it wasn’t surrounded by parenthesis, it doesn’t get saved.
  • “\t” is last, but not least- it places a tab character between \2 and \1, spacing them out a bit. I put 2 in there to make it easier to read.

Argh, I said I wasn’t going to explain any more regex! Oh well, screw it. I’d also suggest checking out the info page for sed as well as the man page- running “info sed” will also give you a lot of information on sed’s regex.

Useful Utility: whereis

Whereis is an older utility- it’s functionality shows us of a time when a program not only had a man page, but also stored the source on the machine in question. That’s becoming more rare as programs like firefox come into play- firefox, for example, has no man page and doesn’t install the source (due to size issues).

Whereis locates the binary, man page and source of a given command on the current machine.

 $ whereis ls
ls: /bin/ls /usr/bin/ls /usr/X11R6/bin/ls /usr/bin/X11/ls 
/usr/share/man/man1/ls.1.gz /usr/share/man/man1p/ls.1p.gz

 $ ls -l /bin/ls /usr/bin/ls \
/usr/X11R6/bin/ls /usr/bin/X11/ls /usr/share/man/man1/ls.1.gz  \
/usr/share/man/man1p/ls.1p.gz
-rwxr-xr-x  1 root root 70708 Jan 24 15:53 /bin/ls
lrwxrwxrwx  1 root root     7 Jan 24 15:53 /usr/X11R6/bin/ls -> /bin/ls
lrwxrwxrwx  1 root root     7 Jan 24 15:53 /usr/bin/X11/ls -> /bin/ls
lrwxrwxrwx  1 root root     7 Jan 24 15:53 /usr/bin/ls -> /bin/ls
-rw-r--r--  1 root root  6403 Mar  6 10:52 /usr/share/man/man1/ls.1.gz
-rw-r--r--  1 root root  8070 Mar  6 10:52 /usr/share/man/man1p/ls.1p.gz

The flags are a little boring- you can limit your searches to binaries (-b), source (-s) or manuals (-m).

Something I found interesting is the capitalized versions of those flags states to only check the directories immediately listed after, but only when the last directory is followed by a -f flag.

$ whereis -m -M /usr/share/man/man1/ -f  ls
ls: /usr/share/man/man1//ls.1.gz

I’ve never seen anything like this before- a space-delimited option with a flag terminator. I’m sure other examples exist, I just can’t think of any. Normally you’d just use commas as seperators and follow it with a whitespace to terminate. The man page also states that due to an specific function it uses(chdir), make sure you use full, not relative paths.

Honestly I was hoping whereis would have a little more meat to it. This is probably one of the more useless programs I’ve written about.

Useful Utility: chown

Since I covered chmod last week, I figured I should touch upon chown this week. chown is infinitely less complex than chmod because you don’t have to worry about actual permissions. chown is mainly used by root, but I suppose it could be used by others as well, although it will happen much less often.

chown can change the owner and group of a file or files.

Standard usage goes something like this

 $ chown morgajel:svngroup samplefile

You can check to make sure changes took by using

 $ ls -l samplefile

Much like chmod, you can use -c to show changes, -v for verbose and -f for quiet. There’s also the -R recursive flag, which works in a similar fashion, but it has several related flags that can be used in conjunction.

One interesting feature of chown is how it handles symbolic links when working recursively- the default behavior is not to traverse and just ignore them. If you use the -H in conjunction with -R, it will traverse the command line argument if it’s a symbolic link. In other words

 $ ls -l mywww
  lrwxrwxrwx  1 morgajel users 9 Mar  9 10:08 mywww -> /var/www
  $ chown bob   -R -H ~/mywww

will recursively change the files in /var/www, but not symbolic links located IN that location. This behavior differs from -L flag used in conjunction with -R, which will traverse ALL symbolic links encountered.

This can be useful to know if you need to change ownership of an entire directory structure and it’s full of symbolic links to other places.

One interesting flag I found is the conditional –from=user:group flag. It is appended to any other chown command and will only change the file if it meets the condition of having a particular user and/or group. This little tidbit could save you a couple lines of shell scripting down the road- I could see it being useful on rare occasions.

The last flag of interest is the –reference=file flag, where you can reference a file and use it’s user and group to set the target file without explictly stating it. Not incredibly useful, but interesting nonetheless.

Sort of a lame article, I know, but the midi stuff has been keeping me busy.

Useful Utility: chmod

Chmod is a utility used for changing permissions. It is fairly well known, and doesn’t have a lot of obscure flags, which makes it an odd choice for this series. I’m including it because it seems like the most logical way to touch on linux file permissions, which can be the bane of new linux user. Let me cover permissions first, then we’ll move on to chmod.

Simple Permissions

(I’m only touching on “simple” permissions because they’re difficult enough to grasp without throwing in super user and stickey bits, or attributes like immutable.)

Every file has three types of simple permissions: Read, Write and Execute. Each file also has three types of users: the (u)ser, (g)roup, and (o)thers. Each of these groups has each of these permissions, resulting in a total of 9 simple permissions (there’s also a 10th one that determines what type of file you’re dealing with at the front of the line). When you type “ls -l filename” at the command line, you can see these permissions:

morgajel@FCOH1W-8TJRW31 ~/docs $ ls -l foo.pl
-rwxr-xr--  1 morgajel users 984 Feb 27 13:01 foo.pl

The file permissions are the first 10 characters.

  file type     user     group     others
       -             rwx        r-x          r--

Permissions will always be listed in this order- user, group,others. They’ll also be in the order or read,write,execute for each of those groups. A permission is active when the corresponding character is shown, and inactive when a dash appears. The above permissions show this is a regular file, and the user (morgajel) has read, write and execute permissions. The group that it belongs to (users) have read and execute permissions. Everyone else (others) have only read permissions.

Permissions for directories are a bit different than regular files- they control who has access to view and write to their contents. This is a bit out of the scope of what I intended to write (and I’m a tiny bit fuzy on the edges myself), so for the sake of simplicity, I won’t touch on them (for now). I encourage you to explain it if you have a firm grasp yourself and can provide sources.

So all your files have these wonderful permissions, but how do we change them? Well, there are two ways- symbolic and numeric. The simplest is with symbolic, but as you continue you’ll learn to use both.

Symbolic Permissions

Suppose in the file above, we want to block other users from reading the file. to do this, we need to remove the read permission from (o)thers on foo.pl

morgajel@FCOH1W-8TJRW31 ~/docs $ chmod o-r foo.pl
morgajel@FCOH1W-8TJRW31 ~/docs $ ls -l foo.pl
-rwxr-x---  1 morgajel users 984 Feb 27 13:01 foo.pl

we use the symbols o-r to signify “subtract read from others.” Suppose we also want to remove read and execute permissions from the group that this file is in?

morgajel@FCOH1W-8TJRW31 ~/docs $ chmod g-rx foo.pl
morgajel@FCOH1W-8TJRW31 ~/docs $ ls -l foo.pl
-rwx------  1 morgajel users 984 Feb 27 13:01 foo.pl

See, we can stack them up like that. We can stack user groups as well: go+r, guo+rwx, and gu-rx are all acceptable. It’s worth noting that you can also use the equals sign (example: oga=rx) to explicitly set permissions. On top of this, rather than writing “ugo”, you can use “a”, which stands for “all”.

morgajel@FCOH1W-8TJRW31 ~/docs $ ls -l foo.pl
----------  1 morgajel users 984 Feb 27 13:01 foo.pl
morgajel@FCOH1W-8TJRW31 ~/docs $ chmod a+rx foo.pl
morgajel@FCOH1W-8TJRW31 ~/docs $ ls -l foo.pl
-r-xr-xr-x  1 morgajel users 984 Feb 27 13:01 foo.pl

The problem you run in to during all of this is setting complex permissions. Suppose you want to change from -r-xr-xr-x to -rw-r-x-r– ? I don’t know of a way to do it symbolically (although there might be and I just don’t know it).

Numeric Permissions

Think about those 9 original permissions again:

-rwxr-xr--  1 morgajel users 984 Feb 27 13:01 foo.pl

Ignore the file type in the beginning of that line, you have 3 groups of 3.

 rwx      r-x      r--

Now, the fact that they use the letters r, w and x are just visual cues- Instead of letters, imagine them being placeholders in contrast to nothing- permissions are either on, or off.

 ###      #_#      #__

Now we’re going to make a bit of a logical jump. What we end up with are 3 three-digit binary numbers; all the digists are either On (#) or Off (_). The highest you can count in binary with three digits is 7(if you start at 0). The bad news is you/re about to learn to count to 7 in binary. The good news is it’s very easy.

0   ___     000
1   __#     001
2   _#_     010
3   _##     011
4   #__     100
5   #_#     101
6   ##_     110
7   ###     111

Now lets look at those permissions again and convert them from symbols to placeholders to binary to regular digits:

 rwx      r-x      r--
 ###      #_#      #__
 111      101      100
  7        5        4

Now lets try setting a file with now permissions to have the permissions above.

morgajel@FCOH1W-8TJRW31 ~/docs $ ls -l foo.pl
----------  1 morgajel users 984 Feb 27 13:01 foo.pl
morgajel@FCOH1W-8TJRW31 ~/docs $ chmod 754 foo.pl
morgajel@FCOH1W-8TJRW31 ~/docs $ ls -l foo.pl
-rwxr-xr--  1 morgajel users 984 Feb 27 13:01 foo.pl

The most common numbers you’ll use are the following:

7 most permissions you can give: read, write and execute
7 read and write, used on personal documents
5 read an execute, but no write access- often given to scripts so other users can run them
4 read only, used on system documents
0 nothing: you get no access

Using Chmod

We’ve seen the basis of how chmod works, but what fun things can we do with it? Well, there’s the -R flag which changes permissions recursively. The -c flag lists the changes that are actually made for each file:

morgajel@FCOH1W-8TJRW31 ~/docs $ chmod 750 -c foo.pl
mode of `foo.pl' changed to 0750 (rwxr-x---)

The -v flag lists ALL changes that are attempted for each file. The -f flag hides all mesages, which can also be done with the more obvious –silent or –quiet.

It should be noted that Users can only set the permissions they have access to; for example, Bob can change the permissions on files that Bob owns, but not on files that Joe owns. Root can of course change any permissions and even write to unwritable files.

An interesting note is that a file set to 007 will allow others to read, write and execute, but not the user or group that it belongs to- in other words, permissions don’t stack.

The 10th Permission

Remember that leading – on our permissions? I referred to it as the file type. Well, here’s a more complete explination. There are several types of files:

  • – = regular file
  • d = directory
  • l = symbolic link
  • s = socket
  • p = named pipe
  • c = character (unbuffered) device file special
  • b = block (buffered) device file special

Regular files and directories are the most common types you’ll run into, followed by symbolic. These are all beyond the scope of this document (yet again), but I will mention this- Symbolic link files will always have their permissions set to 777 (rwxrwxrwx).

Advanced Permissions

Remember how I said the r, w and x in “rwxr-xr–” were just placeholders, and didn’t hold any meaning? well, that was partly true. there are 3 special “optional” flags that can be used. The first one is the Stickey bit, often showing up as a T in the (o)ther’s execute space. This tells the kernel to keep a copy of this executable in swap space so it’ll run without incurring the delay of loading it from the harddrive the next time around. It’s rare to see it, but when you do, it’ll look like this:

morgajel@FCOH1W-8TJRW31 ~/docs $ ls -l foo.pl
---------T  1 morgajel users 984 Feb 27 13:01 foo.pl

Next up are the setuserid (SETUID) and setgroupid (SETGID) bits. There are actually two- one for groups and one for users. When the group SETGID bit is set, anyone who runs this program will execute it as if they’re a member of whatever group the file belongs to. If the user SETUID bit is set, the program will run as if it’s being run by the user it belongs to.

morgajel@FCOH1W-8TJRW31 ~/docs $ ls -l myprogram*
---x--Sr-x  1 morgajel webdevs 984 Feb 27 13:01 myprogram1
---S--xr-x  1 morgajel webdevs 984 Feb 27 13:01 myprogram2

For example, if bob was to execute the myprogram1 file, it would execute with webdevs’ group permissions, perhaps granting it access to files bob otherwise might not be able to get to. If he were to run myprogram2, it would execute as if morgajel was running it. These bits can be useful with applications like cdrecord, where the program needs low level access to hardware. They do pose a security risk however- an insecure program can result in a malicious user getting permission to files they otherwise wouldn’t be able to.

Be wary when using the SETUID and SETGID bits, and use them sparingly.

I think that covers a good majority of what you need to know to use chmod. Later on I’ll go into umask, chown and file attributes. If you have anything to add, feel free to include it below.

Go to Top