How to use pipes on Linux

Use Linux pipes to choreograph command line utilities to work together. Simplify complex processes and increase your productivity by using a collection of stand-alone commands and turning them into a single-minded team. We’ll show you how.

Pipes are everywhere

Pipes are one of the most useful command line features of Linux and Unix-like operating systems. Pipes are used in a myriad of ways. Check out any Linux command line article – any website, not just ours – and you will find that pipes pop up more often than not. I’ve read some of the How-To Geek Linux articles, and all of them use pipes in one way or another.

Linux pipes allow you to perform actions that are not specified by default by the Sleeve. But because the Linux design philosophy is to have lots of little utilities that their dedicated function very well, and without unnecessary functions – the mantra of “do one thing and do it well” – you can pipe sequences of commands so that the output of one command becomes the input of another. Every command you put brings their unique talent to the team, and soon you will find you have put together a winning squad.

A simple example

Suppose we have a directory with many different types of files. We want to know how many files of a certain type are in this directory. There are other ways to do this, but the goal of this exercise is to put pipes in, so we’re going to do it with pipes.

We can easily get a listing of the files with ls:

ls


To distinguish the file type of interest we use grep. We want to find files whose filename or extension contains the word “page”.

We use the shell special character “|”To direct the output of ls into it grep.

ls | grep "page"

grep prints lines that match its search pattern. So this gives us a listing that contains only “.page” files.

Even this trivial one example shows the functionality of pipes. The output of ls was not sent to the terminal window. It was sent to grep as data for the grep Command to work. The issue we’re seeing comes from grep, This is the last command in the chain.

Extension of our chain

Let’s start by expanding our chain of guided commands. We can Count the “.page” files by adding the wc Command. We’ll use that -l (Number of lines) option with wc. Notice we added that too -l (long format) option too ls . We’ll be using this shortly.

ls - | grep "page" | wc -l

grep is no longer the last command in the chain, so we don’t see its output. The output of grep is fed into the wc Command. The output we see in the terminal window is from wc. wc reports that the directory contains 69 “.page” files.


Let’s expand things one more time. We’ll take that wc Command from the command line and replace it with awk. There are nine columns in the output of ls with the -l (long format) option. We use awk to Print columns five, three and nine. This is the size, owner, and name of the file.

ls -l | grep "page" | awk '{print $5 " " $3 " " $9}'

We get a listing of these columns for each of the files that match.

We’ll cover this issue now through the sort Command. We’ll use that -n (numeric) option to leave sort know that the first column should be treated as numbers.

ls -l | grep "page" | awk '{print $5 " " $3 " " $9}' | sort -n

The output is now sorted by file size, with our custom choice of three columns.

Add another command

We’ll conclude by adding that tail Command. We say it to list those last five lines of output only.

ls -l | grep "page" | awk '{print $5 " " $3 " " $9}' | sort -n | tail -5

This means that our command roughly translates as “Show me the five largest” .page “files in this directory, sorted by size”. Of course there is no command to do this, but using pipes we made our own. We could add this – or any other long command – as an alias or shell function to save all input.

Here is the output:


We could reverse the size order by adding that -r (reverse) option for sort Command and use head Instead of tail to select the lines from the top of the issue.

This time the five largest “.page” files are listed from largest to smallest:

Some recent examples

Here are two interesting examples from recent how-to geek articles.

Some commands, such as xargsCommand, are designed to have input to them. Here’s a way we can have wc count them Words, characters and lines in multiple files, by piping ls into it xargs which then feeds the list of file names wc as if they had been passed on wc as a command line parameter.

ls *.page | xargs wc

The total number of words, characters, and lines is listed at the bottom of the terminal window.

This gives you a sorted list of the unique file extensions in the current directory with the number of each type.

ls | rev | cut -d'.' -f1 | rev | sort | uniq -c

There’s a lot going on here.

  • ls: Lists the files in the directory
  • Revolution: Reverses the text in the filename.
  • cut: Cut the cord at the first occurrence of the specified separator “.”. Text after that is discarded.
  • rev: Reverses the rest of the text, i.e. the file name extension.
  • sort: Sorts the list alphabetically.
  • uniq: Counts the number of each unique entry in the list.


The output shows the list of file extensions, sorted alphabetically with the number of each type.

Named pipes

There is another type of pipe that we call named pipes. The pipes in the previous examples are created on the fly by the shell when it processes the command line. The pipes are created, used, and then discarded. They are ephemeral and leave no traces of themselves. They only exist as long as the command using them is executed.

Named pipes appear as persistent objects in the file system, so you can use them with. can see ls. They are persistent because they can withstand a computer restart – although any unread data in them is discarded at that point.

Named pipes were often used all at once to allow different processes to send and receive data, but I haven’t seen them this way in a long time. No doubt there are people out there who are still using them to great effect, but I haven’t met any lately. But for the sake of completeness, or just to satisfy your curiosity, here’s how you can use them.

Named pipes are used with the. created mkfifo Command. This command creates a named pipe called “geek-pipe” in the current directory.

mkfifo geek-pipe

We can see the details of the named pipe if we have the ls Command with the -l (long format) Option:

ls -l geek-pipe


The first character in the listing is a “p”, which means that it is a pipe. If it were a “d” it would mean the filesystem object is a directory and a hyphen “-” would mean it was a normal file.

Use the named pipe

Let’s use our pipe. The unnamed pipes we used in our previous examples passed the data from the sending command to the receiving command immediately. Data sent over a named pipe remains in the pipe until it is read. The data is actually held in memory so the size of the named pipe does not vary ls Lists of whether or not they contain data.

We’ll be using two terminal windows for this example. I use the label:

# Terminal-1

in a terminal window and

# Terminal-2

in the other so that you can distinguish between them. The hash “#” tells the shell that what follows is a comment and should ignore it.

Let us take the entirety of our previous ones example and redirect that to the named pipe. So we’re using both unnamed and named pipes in one command:

ls | rev | cut -d'.' -f1 | rev | sort | uniq -c > geek-pipe

Not much is going to happen. However, you may find that you don’t return to Command Prompt so something goes on.

In the other terminal window, issue this command:

cat < geek-pipe


We redirect the contents of the named pipe to cat, so that cat shows this content in the second terminal window. Here is the output:

And you will see that you have returned to the Command Prompt in the first Terminal window.

So what just happened.

  • We have redirected some output to the named pipe.
  • The first terminal window did not return to the command prompt.
  • The data stayed in the pipe until it was read from the pipe in the second terminal.
  • We were brought back to the command prompt in the first terminal window.

You might think that you could run the command in the first terminal window as a background task by typing a. Add & until the end of the command. And you’d be right. In that case, we would have been immediately returned to the command prompt.

The point of not using background processing was to emphasize that a named pipe is a blocking process. Pasting something into a named pipe only opens one end of the pipe. The other end will not open until the reader extracts the data. The kernel pauses in the first terminal window until the data is read from the other end of the pipe.

The power of the pipes

Nowadays named pipes are something of a novelty.

Plain old Linux pipes, on the other hand, are one of the most useful tools you can have in your terminal window toolkit. The Linux command line comes alive for you and a whole new power-up when you can orchestrate a collection of commands for cohesive performance.

Separation Note: It is best to write your piped commands by adding one command at a time and making that part work and then initiating the next command.

Related Posts