Introducing (UNIX) Shell Pipes

Shell Pipes are very useful data i/o mechanisms which greatly enchance the power of the shell in terms of employing multiple utilities to work together. They reflect the modular problem solving model of Unix. Complex tasks are solved using different combinations of tiny programs or utilities. These programs are 'connected' by pipes which facilitate exchange of data among them. I will try and explain the three most commonly used pipes implemented in shells such as BASH, CSH etc.

The ">" Pipe

The ">" pipe redirects the ouput (the stdout, to be specific), to a file . The general syntax of usage is

	$ program arg1 arg2 ... > file_name

It must be noted that file_name, regardless of wether it already exists or not, is written to, like a brand new file. So if it already exists, it will be overwritten with the output of the program, or if it doesn't exist, the file is created and the output is written to it. The below example, stores the output of the 'ls -l' command (which lists the contents of the working directory along with their attributes), to the file contents.txt.

	$ ls -l > contents.txt

The ">>" pipe, is similar to the ">" pipe, in that it also redirects the output to a file. The only difference being that, if the file already exists, the output is appended instead of being written over the existing contents. In the below example, the first 'grep' ed output is redirected to the file lines.txt. If it already exists, it will be overwritten. However, in the second step, the output is appended to the same file. So lines.txt contains all the lines, of prog.c, which contain the word "long" followed by those which contain the word "char".

	$ grep "long" prog.c > lines.txt
$ grep "char" prog.c >> lines.txt

The "<" Pipe

The "<" pipe takes data from the specified file and channels it to the input of the program (the stdin, to be specific). The general syntax of usage would be

	$ program arg1 arg2 ... < file_name

It must be noted that, file_name, must exist (obviously!). In the below example, the contents of the file prog.c are given to grep as input using the "<" pipe.

	$ grep "long" < prog.c

(which produces the same result as $ grep "long" prog.c )

The "|" Pipe

The "|" Pipe sends the output (stdout) of one program to the input (stdin) of another program. In some of the common shells such as sh or bash, two programs are 'connected' by a pipe by specifying a '|' between the program names in the following manner.

	$ program-1 args | program-2 args

In the above example, the output (stdout) of program-1 is sent to program-2's input (stdin). For example if you want to find the number of lines in a file which contain a particular word, you can utilise the "|" pipe with the grep and wc commands to do it. Heres how it can be done ..

	$ grep "bash" test.txt | wc -l

which will print the number of lines of file which contain the word 'bash' in the file test.txt. Basically whats happening here is that, the pipe sends the grep'ed output, which is all the lines of test.txt which contain the word "bash", to the input of 'wc -l', which counts and prints the number of lines of text, input. The above operation can be performed using intermediate files as well...

	$ grep "bash" test.txt > the_lines.txt
	$ wc -l < the_lines.txt

In the above example, the grep output is sent to the file 'the_lines.txt'. And after the execution of that command, 'wc -l' is called with the 'the_lines.txt' as the (redirected) input.