GoKawiilIf we are running any eBPF workload or writing eBPF code, we want to measure its performance impact, and in this post we will demonstrate an example of how to do it.
In this example, our goal is to measure the performance of file open operations, one of the most critical functions in the OS. Our code used file open hooks in eBPF, we wanted to measure the performance overhead introduced by adding this hook.
To identify likely bottlenecks, we need a simple C test harness with the fewest dependencies, designed to measure file open performance.
#define _GNU_SOURCE #include <fcntl.h> #include <stdio.h> #include <time.h> #include <stdint.h> #include <stdlib.h> #include <unistd.h> #include <sched.h> #include <sys/mman.h> #include <sys/syscall.h> static inline uint64_t now_ns ( void ) { struct timespec ts; clock_gettime (CLOCK_MONOTONIC, & ts); /* VDSO, no syscall */ return ( uint64_t )ts.tv_sec * 1000000000ull + ts.tv_nsec; } int main ( int argc, char ** argv) { const char * path = argv[ 1 ]; uint64_t n = strtoull (argv[ 2 ], NULL, 10 ); uint64_t warm = n / 10 ; /* preallocate + prefault + lock: no page faults in the loop */ uint32_t * d = mmap (NULL, n * sizeof ( uint32_t ), PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE, - 1 , 0 ); mlock (d, n * sizeof ( uint32_t )); for ( uint64_t i = 0 ; i < n; i ++ ) d[i] = 0 ; /* fault everything in */ for ( uint64_t i = 0 ; i < n; i ++ ) { uint64_t t0 = now_ns (); long fd = syscall (SYS_openat, AT_FDCWD, path, O_RDONLY); /* raw, no libc wrapper */ uint64_t t1 = now_ns (); if (fd >= 0 ) close (fd); d[i] = ( uint32_t )(t1 - t0); } /* dump after the loop only */ for ( uint64_t i = warm; i < n; i ++ ) printf ( "%u
" , d[i]); return 0 ; }
The goal of the code example above is to keep it simple and reopen the same file under warm cache conditions, minimizing unrelated filesystem and disk I/O variability and helping identify the p50/p99. The code invokes syscall(SYS_openat, …) instead of the libc openat() wrapper, and the first 10% of the results are discarded as a warmup period.
This test harness would produce results of opening a file x number of times and how long it took to open. So we could use this to measure the before and after of when the eBPF hook attached.
Setup
When profiling the eBPF code, we want perf tool to be able to resolve symbols so we can analyze where the issue is in our code. To do that, we have to run these commands.
sudo sysctl -w net.core.bpf_jit_enable = 1 sudo sysctl -w net.core.bpf_jit_kallsyms = 1
... continue reading