SHELLdorado Newsletter 1/2005 - April 30th, 2005

The "SHELLdorado Newsletter" covers UNIX shell script related
topics. To subscribe to this newsletter, leave your e-mail
address at the SHELLdorado home page:

View previous issues at the following location:

"Heiner's SHELLdorado" is a place for UNIX shell script
programmers providing

     Many shell script examples, shell scripting tips & tricks,
     a large collection of shell-related links & more...


 o  Shell Tip: How to read a file line-by-line
 o  Shell Tip: Print a line from a file given its line number
 o  Shell Tip: How to convert upper-case file names to lower-case
 o  Shell Tip: Speeding up scripts using "xargs"
 o  Shell Tip: How to avoid "Argument list too long" errors

>> Shell Tip: How to read a file line-by-line

    Assume you have a large text file, and want to process it
    line-by line. How could you do it?

    	for line in `cat $file`	# WRONG
	    echo "$line"
	done no solution, because the variable "line" will in turn
    contain each (whitespace-delimited) *word* of the file, not
    each line. The "while" command is a better candidate for
    this job:

    	while read line
	    echo "$line"
	done < "$file"

    Note that the "read" command automatically processes its
    input: it removes leading whitespace from each line, and
    concatenates a line ending with "\" with the one following.
    The following commands suppress this behaviour:

	OIFS=$IFS; IFS=		# Change input field separator
	while read -r line
	    echo "$line"
	done < "$file"
	IFS=$OIFS		# Restore old value

    There still is one disadvantage to this loop: it's slow. If
    the processing consists of string manipulations, consider
    replacing the loop completely e.g. with an AWK script.
     	"read -r" is available with ksh, ksh93, bash, zsh,
	POSIX, but not with older Bourne Shells (sh).

>> Shell Tip: Print a line from a file given its line number

    Regular expressions can be very powerful, and there are many
    tools (like "egrep") allowing to use them on any file.  But
    what if we simply want to get the 5th line of a file? No
    elaborate regular expression required:

    	sed -n "${lineno}p"

    prints the 5th line without involving ^.[]*$ or other
    meta-characters resembling the noise of a defective serial
    interface. "sed -n" means: do not automatically print each
    line. "5p" indicates: print line 5. We have to use
    "${lineno}p" here instead of "$linenop", because otherwise
    the shell would try to expand the variable "$linenop", not
    knowing that "p" is an "sed" command.

    This could be improved upon. Assume the input file is
    "/usr/dict/words",  and consists of 25143 lines. The "sed"
    command above would not only dutifully print line 5, but
    also continue to read the following 25138 lines, doing what
    it was told to do: ignore them. The following command makes
    "sed" stop reading after line 5:

	sed -n "${lineno}{p;q;}"

    So you think you have a better solution for this problem?
    Prove it: send me your suggestion
    (, closing date: 2005-05-31),
    and I'll measure the speed of all contributions on a Linux
    and a Solaris system. The fastest (or most elegant) solution
    using only POSIX shell commands will be published in the
    next SHELLdorado Newsletter.

>> Shell Tip: How to convert upper-case file names to lower-case

    Admit it: you sometimes copy files from an  operating system
    with a name ending in *indows. A frequent annoyance are file
    names IN ALL UPPER CASE.

    The following command renames them to contain only lower
    case characters:

    	for file in *
	    lcase=`echo "$file" | tr '[A-Z]' '[a-z]'`

	    # Does the target file exist already? Do not
	    # overwrite it:
	    [ -f "$lcase" ] && continue

	    # Are old and new name different?
	    [ x"$file" = x"$lcase" ] && continue # no change

	    mv "$file" "$lcase"

    The KornShell (and ksh93) has the useful "typeset -l"
    option, which will automatically convert the contents of a
    variable to lower case:

    	$ typeset -l lcase=ABCDE
	$ echo "$lcase"

    Changing the above loop to use "typeset -l" is left as an
    exercise for the reader.

>> Shell Tip: Speeding up scripts using "xargs"

    The essential part of writing fast scripts is avoiding
    external processes.

    	for file in *.txt
	    gzip "$file"

    is much slower than just

    	gzip *.txt

    because the former code may need many "gzip" processes for a
    task the latter command accomplishes with only one external
    process.  But how could we build a command line like the one
    above when the input files come from a file, or even
    standard input? A naive approach could be

    	gzip `cat textfiles.list archivefiles.list`

    but this command can easily run into an "Argument list too
    long" error, and doesn't work with file names containing
    embedded whitespace characters. A better solution is using

    	cat textfiles.list archivefiles.list | xargs gzip

    The "xargs" command reads its input line by line, and build
    a command line by appending each line to its arguments
    (here: "gzip"). Therefore the input


    would result in "xargs" executing  the command
    	gzip a.txt b.txt c.txt
    "xargs" also takes care that the resulting command line does
    not get too long, and therefore avoids "Argument list too
    long" errors.

>> Shell Tip: How to avoid "Argument list too long" errors

    Oh no, there it is again: the system's spool directory is
    almost full (4018 files); old files need to be removed, and
    all useful commands only print the dreaded "Argument list
    too long":

    	$ cd /var/spool/data
	$ ls *
	ls: Argument list too long
	$ rm *
	rm: Argument list too long

    So what exactly in the character '*' is too long? Well, the
    current shell does the useful work of converting '*' to a
    (large) list of files matching that pattern. This is not the
    problem. Afterwards, it tries to execute the command (e.g.
    "/bin/ls") with the file list using the system call
    execve(2) (or a similar one). This system call has a
    limitation for the maximum number of bytes that can be used
    for arguments and environment variables(*), and fails.
    It's important to note that the limitation is on the side of
    the the system call, not the shell's internal lists.

    To work around this problem, we'll use shell-internal
    functions, or ways to limit the number of files directly
    specified as arguments to a command.


     o  Don't specify arguments, to get the (hopefully) useful

    	$ ls

     o  Use shell-internal functionality ("echo" and "for" are
        shell-internal commands):

	$ echo *
	file1 file2 [...]

	$ for file in *; do rm "$file"; done	# be careful!

     o  Use "xargs"

	$ ls | xargs rm		# careful!

	$ find . -type f -size +100000 -print | xargs ...

     o  Limit the number of arguments for a command:

     	$ ls [a-l]*
	$ ls [m-z]*

    Using this techniques should help getting around the

    (*) Parameter ARG_MAX, often 128K (Linux) or 1 or 2 MB

If you want to comment on this newsletter, have suggestions for
new topics to be covered in one of the next issues, or even want
to submit an article of your own, send an e-mail to

To unsubscribe, send a mail with the body "unsubscribe" to