2022-06-01 How fast are Linux pipes anyway?
The challenge, and a slow first version #
First of all, let’s start with measuring the performance of the fabled FizzBuzz program, following the rules laid down by the StackOverflow post: % ./fizzbuzz | pv >/dev/null 422GiB 0:00:16 [36.2GiB/s] pv is “pipe viewer”, a handy utility to measure the throughput of data flowing through a pipe. So fizzbuzz is producing output at a rate of 36GiB/s. fizzbuzz writes the output in blocks as big as the L2 cache, to strike a good balance between cheap access to memory and minimizing IO overhead.
On my machine, the L2 cache is 256KiB. Throughout this post, we’ll also output blocks of 256KiB, but without “computing” anything. Essentially, we’ll try to measure the upper bound for programs writing to a pipe with a reasonable buffer size. While fizzbuzz uses pv to measure speed, our setup will be slightly different: we’ll implement the programs on both ends of the pipe. This is so that we fully control the code involved in pushing and pulling data from the pipe.
The code is available in my pipes-speed-test repo. write.cpp implements the writing, and read.cpp the reading. write repeatedly writes the same 256KiB forever. read reads through 10GiB of data and terminates, printing the throughput in GiB/s. Both executables accept a variety of command line options to change their behavior. The first attempt at reading and writing from pipes will be using the write and read syscalls, using the same buffer size as fizzbuzz . Here’s a view of the writing end:
int main () { main size_t buf_size = 1 << 18 ; // 256KiB buf_size char * buf = ( char *) malloc ( buf_size ); bufmallocbuf_size (( void *) buf , 'X' , buf_size ); // output Xs memsetbufbuf_size while ( true ) { size_t remaining = buf_size ; remainingbuf_size while ( remaining > 0 ) { remaining // Keep invoking `write` until we've written the entirety // of the buffer. Remember that write returns how much // it could write into the destination -- in this case, // our pipe. ssize_t written = write ( writtenwrite , buf + ( buf_size - remaining ), remaining STDOUT_FILENObufbuf_sizeremainingremaining ); -= written ; remainingwritten } } }
This snippet and following ones omit all error checking for brevity. The memset ensures that the output will be printable, but also plays another role, as we’ll discuss later. The work is all done by the write call, the rest is making sure that the whole buffer is written. The read end is very similar, but read ing data into buf , and terminating when enough has been read. After building, the code from the repo can be run as follows:
% ./write | ./read 3.7GiB/s, 256KiB buffer, 40960 iterations (10GiB piped)
We’re writing the same 256KiB buffer filled with 'X' s 40960 times, and measuring the throughput. What’s worrying is that we’re 10 times slower than fizzbuzz ! And we’re not doing any work, just writing bytes to the pipe.
It turns out that we can’t get much faster than this by using write and read .
... continue reading