What are pipes in Linux? How does channel redirection work?

You may have used the cmd0 syntax too many times | cmd1 | cmd2 similar to your terminal.

You probably also know what pipe redirection is, which is used to redirect the output of one command as input for the next command.

But do you know what’s underneath? How does channel redirection actually work?

Don’t worry, because today we’re demystifying Unix pipes so the next time you go on a date with those fancy vertical stripes, you know exactly what’s going on.

Note. We have used the term Unix in some places because the concept of pipes (like many other things in Linux) comes from Unix.

Channels in Linux: a general idea

Here’s what you’ll see all over the place regarding “what are pipes in Unix?”:

  • Unix pipes are an IPC (inter-process communication) mechanism that redirects the output of one program to the input of another program.

This is a general explanation that everyone gives. We want to go deeper. Let’s rephrase the previous line in a more technical way by removing the abstractions:

  • Unix pipes are an IPC (inter-process communication) mechanism that takes a stdout program and sends it to another stdin program through a buffer.

Much better. Removing the abstraction made it much cleaner and more accurate. You can look at the following diagram to understand how pipe works.

One of the simplest examples of a pipe command is to pipe some command output to the grep command to search for a specific line.

Be aware: Pipe redirects standard output to standard input, but not as a command argument

It is very important to understand that the pipeline passes the stdout command to another stdin command, but not as a command argument. We will explain this with an example.

If you use cat with no arguments, it will read from stdin by default. Here’s an example:

$ cat
Блог AndreyEx о Linux
^D
Блог AndreyEx о Linux

We used cat here without transferring files, so stdin is the default. We then wrote a line and then used Ctrl + d to signal that we finished writing (Ctrl + d means EOF or end of file). As soon as we finished writing, cat read stdin and wrote this line to stdout.

Now consider the following command:

echo hey | cat

The second command is NOT equivalent to cat hey. Here stdout “hey” is transferred to the buffer and passes stdin to cat. Since there were no command line arguments, cat by default chose stdin to read and something was already in stdin, so the cat command accepted that and printed stdout.

In fact, we created a file called hey and put some content in it.

Channel types in Linux

There are two types of pipes on Linux:

  • An unnamed channel, also called anonymous.
  • Named pipes

Untitled channels

Untitled channels, as the name suggests, have no name. They are created on the fly by your Unix shell whenever you use the | character.

When people talk about pipes in Linux, they usually talk about it. They are useful because you, the end user, don’t need to track anything. Your shell will handle all of this.

Named pipes

This one is slightly different. Named pipes are indeed present in the file system. They exist like a regular file. You can create a named file using the following command:

mkfifo pipe

This will create a file named “pipe”. Run the following command:

$ ls -l pipe
prw-r--r-- 1 ausername ausername 0 Jul 12 09:51 pipe

Notice the “p” at the beginning, it means the file is a pipe. Now let’s use this channel.

As we said earlier, the pipeline forwards the output of one command to the input of another. It’s like a courier service, you give the package to be delivered from one address, and they deliver it to another. So the first step is to provide a package, that is, provide the channel with something.

echo hey > pipe

You will notice that echo has not returned the terminal to us yet. Open a new terminal and try reading from that file.

cat pipe

Both of these commands completed their execution at the same time.

This is one of the fundamental differences between a regular file and a named pipe. Nothing is written to the pipe until some other process reads it.

Why use named pipes? Here is a list of why you want to use named pipes

Named pipes do not consume disk space

If you do a du -s pipe, you’ll see that it doesn’t take up space. This is because named pipes are like endpoints for reading and writing from and to a memory buffer. Anything written to a named pipe is actually stored in a temporary memory buffer that is flushed when a read operation is performed from another process.

Reduced I / O

Since writing to a named pipe means storing data in a memory buffer, large file operations significantly reduce disk I / O.

The connection between two very different processes

The output of an event can be obtained instantly and very efficiently from another process using named pipes. Since reading and writing are occurring simultaneously, the latency is practically zero.

Understanding lower-level channels [для опытных пользователей и разработчиков]

The next section deals with pipes at a deeper level with actual implementations. This section requires you to have a basic understanding:

  • How a C program works
  • What are system calls
  • What are processes
  • What are file descriptors

We won’t go into the details of the examples. We are talking only about “channels”. For most Linux users, this section is unnecessary.

At the end we have provided a sample file for compilation – Makefile. Keep in mind that these programs are for illustrative purposes only, so if you see errors being handled incorrectly.

Consider the following sample program:

// pipe.c
#include <unistd.h>
#include <stdio.h>
#include <sys/types.h>
#include <stdlib.h>
#include <errno.h>

extern int errno;

int main(){
    signed int fd[2];
    pid_t pid;
    static char input[50];
    static char buf[50];

    pipe(fd);

    if((pid=fork())==-1){
        int err=errno;
        perror("ответвление не сработало");
        exit(err);
    }

    if(pid){
        close(fd[1]);
        read(fd[0], buf, 50);
        printf("В сообщении от ребенка: %sn", buf);
    } else {
        close(fd[0]);
        printf("Введите сообщение от родителя: ");
        for(int i=0; (input[i]=getchar())!=EOF && input[i]!='n' && i<49; i++);
        write(fd[1], input, 50);
        exit(0);
    }
    return 0;
}

In line 16, we created an unnamed pipe using the pipe () function. The first thing to notice is that we passed in an array of signed integers of length 2.

This is because a pipe is nothing more than an array of two unsigned integers representing two file descriptors. One for writing, one for reading. And they both point to the location of the buffer in memory, which is typically 1MB.

Here we named the variable fd. fd [0] – input file descriptor, fd [1] Is a descriptor for the output file. In this program, one process writes a line to the file descriptor fd [1]and another process reads from file descriptor fd [0]…

The named pipe is no different. With a named pipe, instead of two file descriptors, you get a file name that you can open from any process and work with it like any other file, while respecting the characteristics of the pipe.

Here is a sample program that does the same thing as the previous program, but instead of an anonymous pipe, creates a named pipe.

// fifo.c
#include <unistd.h>
#include <sys/types.h>
#include <errno.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <fcntl.h>
#include <sys/stat.h>

extern int errno;

#define fifo "npipe"

int main(void){
    pid_t pid;
    static char input[50];
    static char buf[50];
    signed int fd;
    
    mknod(fifo, S_IFIFO|0700, 0);

    if((pid=fork())<0){
        int err=errno;
        perror("ответвление не сработало");
        exit(err);
    }

    if(pid){
        fd=open(fifo, O_RDONLY);
        read(fd, buf, 50);
        close(fd);
        printf("Вывод есть : %s", buf);
        remove(fifo);
        exit(0);
    } else {
        fd=open(fifo, O_WRONLY);
        for(int i=0; (input[i]=getchar())!=EOF && input[i]!='n' && i<49; i++);
        write(fd, input, strlen(input));
        close(fd);
        exit(0);
    }
    return 0;
}

Here we have used the mknod system call to create a named pipe. As you can see, although we removed the pipe when finished, you can easily leave it and use it easily for communication between different programs by simply opening and writing to a file named “npipe” in my case.

You also don’t have to create two different two-way channels as you would with anonymous channels.

Here is a sample Makefile as promised. Place it in the same directory as the previous programs, named “pipe.c” and “fifo.c” respectively.

CFLAGS?=-Wall -g -O2 -Werror
CC?=clang

build:
	$(CC) $(CFLAGS) -o pipe pipe.c
	$(CC) $(CFLAGS) -o fifo fifo.c

clean:
	rm -rf pipe fifo

Like this. That’s really all there is to Unix pipes.

Sidebar