SHELLdorado Newsletter 1/2002 - February 13th, 2002

================================================================
The "SHELLdorado Newsletter" covers UNIX shell script related
topics. To subscribe to this newsletter, leave your e-mail
address at the SHELLdorado home page:

	http://www.shelldorado.com/

View previous issues at the following location:

	http://www.shelldorado.com/newsletter/

"Heiner's SHELLdorado" is a place for UNIX shell script
programmers providing

     Many shell script examples,
     shell scripting tips & tricks + more...
================================================================

This issue focuses on Korn Shell 93

Contents

 o  Editorial
 o  How to use associative arrays
 o  How to get string lengths and sub-strings
 o  How to create easy counting loops with the new "for" syntax
 o  How to access variables by name using "typeset -n"
 o  How to use the built-in printf command
 o  Q&A: Where can I find more articles about ksh93?
 o  Q&A: How can I write my own POP3 e-mail client using ksh?

-----------------------------------------------------------------
>> Editorial
-----------------------------------------------------------------

    Much has changed since Stephen "Steve" R. Bourne wrote his
    command line interpreter the "Bourne Shell", or just "sh".
    David Korn rewrote it from scratch, adding new features on
    his way. The result, KornShell 88 (named after the year it
    was published) is available on almost any UNIX system today.

    This issue of the SHELLdorado Newsletter focuses on the
    latest version of the KornShell: KornShell 93 (or "ksh93").
    Many people already know and value the shell as the base of
    the "dtksh" shell, which comes as part of the "Common
    Desktop Environment" CDE for many UNIX systems, e.g. Solaris
    or AIX (search for /usr/dt/bin/dtksh).

    ksh93 is not only a new version of the old KornShell with
    many new features particularly for shell script programmers:
    it could become the new standard shell for all new UNIX
    systems, including Linux.

    The source code is now available, together with pre-compiled
    binaries for many UNIX systems, including MacOS, Linux,
    FreeBSD, HP-UX, AIX, Linux, Solaris, SCO UnixWare and
    even Windows (for use with U/WIN emulation):

	http://www.research.att.com/sw/download/

    If you are not interested in ksh93, you may still find the
    small POP3 e-mail client written in "conventional" KornShell
    (ksh88) interesting.

    If you have comments or suggestions for this newsletter, or
    even want to write an article by your own, please write me
    an e-mail.
    
    Heiner Steven, Editor
    <heiner.steven@shelldorado.com>


-----------------------------------------------------------------
>> How to use associative arrays
-----------------------------------------------------------------

    An associative array is an array indexed by a string. This
    lets us easily create lookup tables, e.g. a table containing
    the full names of all users given their login id:

	typeset -A name			# "name" is an array

	# Input consists of fields separated by a colon (':')
	OIFS=$IFS; IFS=:
	while read login pw uid gid fullname ignore
	do
	     name[$login]=$fullname
	done < /etc/passwd
	IFS=$OIFS

	# $login sequentially contains all indices from name[]
	for login in ${!name[@]}
	do
	    print "$login: ${name[$login]}"
	done

    This prints e.g.

	root: root
	nobody: nobody
	lp: Printing daemon
	heiner: Heiner Steven
	[...]

    Within the script, print ${name[heiner]} could be used to
    print the full name of the user with the login id "heiner".


-----------------------------------------------------------------
>> How to get string lengths and sub-strings
-----------------------------------------------------------------

    Some frequent string manipulations get easier with ksh93.
    The length of each string can be printed in the following
    way:

	a="Heiner's SHELLdorado"
    	$ print ${#a}		# string length
	20

	$ print ${a:9}		# string starting with position 9
	SHELLdorado

	$ print ${a:9:5}	# starting with position 9; 5 characters
	SHELL

    Even "sed" like substitutions are available.  The following
    command replaces all "U" characters in the variable "a" with
    an "X" in the output:

    	$ print ${a//U/X}

    These command make string manipulation not just easier,
    but also much faster.


-----------------------------------------------------------------
>> How to create easy counting loops with the new "for" syntax
-----------------------------------------------------------------

    The old Bourne shell forced us to write code like the
    following for simple tasks as counting from 1 to 10:

    	i=1
	while [ $i -le 10 ]
	do
	    echo $i
	    i=`expr $i + 1`
	done
    
    Compare this to the new ksh93 syntax:

    	integer i
	for (( i=1; i<=10; i++ ))
	do
	    echo $i
	done


-----------------------------------------------------------------
>> How to access variables by name using "typeset -n"
-----------------------------------------------------------------

    Sometimes it's useful to specify the name of a variable as
    an argument to a function, e.g.

    	getstring firstname "First Name:"

    The function "getstring" should read a string from the user,
    and return the result in our variable "firstname". This used
    to be solved with "eval", but ksh93 has an easier solution:

    	function getstring { # varname promptstring
	    typeset -n vname=$1
	    typeset prompt=${2-"?"}

	    print -u2 "$prompt\c"
	    read vname
	}

    Since "vname" contains the name of a variable (not its
    value), all manipulations on "vname" change the value of the
    variable specified as an argument to the function.

    Without "typeset -n" we would have written the function
    similar to the following example:

    	getstring () { # varname promptstring
	    vname=$1
	    prompt=${2-"?"}

	    echo -n "$prompt" >&2
	    read answer
	    eval "$vname=\"$answer\""
	}


-----------------------------------------------------------------
>> How to use the built-in printf command
-----------------------------------------------------------------

    Why should we use yet another version of the "printf"
    command? Many systems already have a version in /usr/bin.

    Well, one advantage of printf being a built-in is, that the
    behaviour of the function is system-independent, and ksh93
    programmers can rely on its existence.

    But this built-in version of printf is special in some other
    ways, too:

     o	If there are more arguments than formats in the format
        string, the format string is reused. This can be used
	e.g. to replace "cut" in some places:

	    cut -c1,10 < /etc/passwd

	to list the first 10 characters of each line in
	/etc/passwd can be rewritten in the following way:

	    printf ".10s\n" /etc/passwd
	    
     o	It can convert regular expressions to shell patterns,
     	and vice versa:

	    $ printf "%P\n" "a(b|c)x"
	    *a@(b|c)x*

	    $ printf "%R\n" "*a@(b|c)x*"
	    a(b|c)x

     o	"printf" can help quoting:

	    $ printf "%q\n" "Heiner's SHELLdorado"
	    $'Heiner\'s SHELLdorado'

    ...and printf even knows something about HTML and XML, and
    how to expand special characters in a way suitable for HTML
    text:

    	$ printf "%H\n" "<Tips & Tricks>"
	<Tips & Tricks>

    Well, that does not look too readable, but we are not the
    interpreters targeted ;-) A web browser knows how to convert
    these "character entities" to a readable representation.


-----------------------------------------------------------------
>> Q&A: Where can I find more articles about ksh93?
-----------------------------------------------------------------

    The KornShell 93 is available free of charge from AT&T
    Labs-Research:

    	http://www.research.att.com/sw/download/

    A concise list of the new features of ksh93 is listed
    in the following article:

	http://www.cs.princeton.edu/~jlk/ksh93.html

    The author of the shell, David Korn, maintains a web site
    with many more articles:

    	http://www.kornshell.com/

    A description of the new features of ksh93 from the Linux
    Journal:

        http://www.linuxjournal.com/article.php?sid=1273

    A manual page

        http://www.cs.princeton.edu/~jlk/kornshell/doc/man93.html

    Frequently Asked Questions (FAQ):

    	http://www.kornshell.com/doc/faq.html

    Many more links to shell scripts and shell scripting related
    articles are available at the SHELLdorado Links section:

    	http://www.shelldorado.com/links/


-----------------------------------------------------------------
>> Q&A: How can I write my own POP3 e-mail client using ksh?
-----------------------------------------------------------------

    Sometimes these viruses (like "SirCam") can really be
    annoying. They make completely strangers send you 3 MB
    holiday pictures, or some 5 MB audio file you did not ask
    for. Most POP3 e-mail clients (like Netscape) will download
    the files from the server to your local file system before
    allowing the relieving click on the "delete" button.

    In these cases the following script is useful: it can delete
    files directly on the POP3 e-mail server, before they reach
    the local hard disk. It can also list the e-mails, and
    display them one by one.

    The script will certainly not replace your e-mail
    application of choice, but it may be useful for the purpose
    stated above, or just as an example of KornShell
    co-processes.

    Note that the script does NOT require ksh93, it should run
    with any KornShell dialect.

	#! /usr/bin/ksh
	# popc.ksh - example of a POP3 e-mail client written in KornShell
	# Heiner Steven, heiner@shelldorado.com
	#
	# This example implements the "LIST" command to list all messages,
	# "RETR" to retrieve a message text by number, and "DELE" to delete
	# a message by number.
	#
	# Refer to RFC 1939 (http://www.ietf.org/rfc/rfc1939.txt) for a full
	# description of the POP3 protocol.
	#
	# Needs the non-standard program "netcat" (aka "nc") to establish a
	# TCP connection. You could use "socket(1)" instead, if installed.

	host=localhost			# Name or address of POP3 server
	port=110			# POP3 port (standard is 110)
	user=${USER:-$LOGNAME}
	pass=dontsay

	typeset -u pop_status		# Global last status { "+OK" | "-ERR" }
	typeset pop_msg			# Global last server status message

	function Fatal { print -u2 "$@"; exit 1; }

	# sendline - send one line to co-process

	function sendline { print -p -- "$@"; }

	# getack - get positive or negative acknowledgement from server
	#
	# The server will answer with either "+OK ..." or "-ERR ...". We
	# evaluate the first word read from the server, and set the
	# return value accordingly

	function getack {
	    read -p -r pop_status pop_msg; print -u2 "DEBUG: $pop_status $pop_msg"
	    [[ $pop_status == '+OK' ]] && return 0
	    print -u2 -- "$pop_status $pop_msg"
	    return 1
	}

	# getlist - get multi-line response from server
	#
	# Multi-line responses are terminated by a line consisting only of
	#  a period '.'

	function getlist {
	    typeset line

	    while read -p -r line
	    do
		line=${line%+(?)}		# Remove trailing CR (ASCII 13)
		[[ $line == '.' ]] && break	# End of list
		print -- "<$line>"
	    done
	}

	function ListCmd {
	    sendline "LIST"
	    getack && getlist
	}

	function DeleteCmd {
	    integer msgno

	    read msgno?"DELETE message Number: "
	    sendline "DELE $msgno" && getack
	}

	function RetrieveCmd {
	    integer msgno

	    read msgno?"Retrieve message Number: "
	    sendline "RETR $msgno"
	    getack && getlist
	}

	# Startup the server as a co-process. We will use "print -p"
	# and "read -p" to write and read data.

	netcat "$host" "$port" |&	popdpid=$!

	# Check if the server is ready
	getack || Fatal "cannot start POP3 server $host:$port"

	# Authentication

	set -e		# First error terminates script
	sendline "USER $user" && getack
	sendline "PASS $pass" && getack
	set +e

	print -u2 "Authentication successful"
	PS3="POP3 command (RETURN prints menu): "
	select choice in List Retrieve Delete Quit
	do
	    case "$choice" in
	    (List)	ListCmd ;;
	    (Retrieve)	RetrieveCmd ;;
	    (Delete)	DeleteCmd ;;
	    (Quit)	break;;
	    esac
	done

	# Shut down co-process
	exec 3<&p 3>&p		# redirect co-process fd to fd 3
	exec 3<&- 3>&-		# close fd 3
	kill $popdpid >/dev/null 2>&1	# if the above does not work...

	exit 0

    [Download location for "netcat":
ftp://ftp.uni-stuttgart.de/pub/org/uni-s/rus/security/unix/hobbit/nc110.tgz

    Check your local system first; many systems (e.g. Linux)
    already have this program installed.
    
    The script was tested with pdksh v5.2.14 99/07/13.2
    (Linux) and ksh93 Version M 1993-12-28 m (Linux).]


----------------------------------------------------------------

The examples were tested using Linux 2.4 and KornShell 93
Version M 1993-12-28 m. Use "print ${.sh.version}" to find your
ksh93 revision number.

----------------------------------------------------------------
If you want to comment on the newsletter, have suggestions for
new topics to be covered in one of the next issues, or even want
to submit an article of your own, send an e-mail to

	mailto:heiner.steven@shelldorado.com

================================================================
To unsubscribe send a mail with the body "unsubscribe" to
newsletter@shelldorado.com
================================================================