Streams, pipes and lists

Bash has powerful tools for working with other programs and their outputs. Using streams we can send the output of a program into another program or file and thereby write logs or whatever we want.

Pipes give us opportunity to create conveyors and control the execution of commands.

It is paramount we understand how to use this powerful and sophisticated tool.

Streams

Bash receives input and sends output as sequences or streams of characters. These streams may be redirected into files or one into another.

There are three descriptors:

Redirection makes it possible to control where the output of a command goes to, and where the input of a command comes from. For redirecting streams these operators are used:

Here are few examples of using redirections:

# output of ls will be written to list.txt
ls -l > list.txt

# append output to list.txt
ls -a >> list.txt

# all errors will be written to errors.txt
grep da * 2> errors.txt

# read from errors.txt
less < errors.txt

Pipes

We could redirect standard streams not only in files, but also to other programs. Pipes let us use the output of a program as the input of another.

In the example below, command1 sends its output to command2, which then passes it on to the input of command3:

command1 | command2 | command3

Constructions like this are called pipelines.

In practice, this can be used to process data through several programs. For example, here the output of ls -l is sent to the grep program, which prints only files with a .md extension, and this output is finally sent to the less program:

ls -l | grep .md$ | less

The exit status of a pipeline is normally the exit status of the last command in the pipeline. The shell will not return a status until all the commands in the pipeline have completed. If you want your pipelines to be considered a failure if any of the commands in the pipeline fail, you should set the pipefail option with:

set -o pipefail

Lists of commands

A list of commands is a sequence of one or more pipelines separated by ;, &, && or || operator.

If a command is terminated by the control operator &, the shell executes the command asynchronously in a subshell. In other words, this command will be executed in the background.

Commands separated by a ; are executed sequentially: one after another. The shell waits for the finish of each command.

# command2 will be executed after command1
command1 ; command2

# which is the same as
command1
command2

Lists separated by && and || are called AND and OR lists, respectively.

The AND-list looks like this:

# command2 will be executed if, and only if, command1 finishes successfully (returns 0 exit status)
command1 && command2

The OR-list has the form:

# command2 will be executed if, and only if, command1 finishes unsuccessfully (returns code of error)
command1 || command2

The return code of an AND or OR list is the exit status of the last executed command.