If you have been following the adventures of our hero over the last couple of years, you might remember that we can’t really trust sampling profilers for Java, and it’s even worse for Java’s instrumentation-based profilers.
For sampling profilers, the so-called observer effect gets in the way: when we profile a program, the profiling itself can change the program’s performance behavior. This means we can’t simply increase the sampling frequency to get a more accurate profile, because the sampling causes inaccuracies. So, how could we possibly know whether a profile correctly reflects an execution?
We could try to look at the code and estimate how long each bit takes, and then painstakingly compute what an accurate profile would be. Unfortunately, with the complexity of today’s processors and language runtimes, this would require a cycle-accurate simulator that needs to model everything, from the processor’s pipeline, over the cache hierarchy, to memory and storage. While there are simulators that do this kind of thing, they are generally too slow to simulate a full JVM with JIT compilation for any interesting program within a practical amount of time. This means that simulation is currently impractical, and it is impractical to determine what a ground truth would be.
So, what other approaches might there be to determine whether a profile is accurate?
In 2010, Mytkowicz et al. already checked whether Java profilers were actionable by inserting computations at the Java bytecode level. On today’s VMs, that’s unfortunately an approach that changes performance in fairly unpredictable ways, because it interacts with the compiler optimizations. However, the idea to check whether a profiler accurately reflects the slowdown of a program is sound. For example, an inaccurate profiler is less likely to correctly identify a change in the distribution of where a program spends its time. Similarly, if we change the overall amount of time a program takes, without changing the distribution of where time is spent, it may attribute run time to the wrong parts of a program.
We can detect both of these issues by accurately slowing down a program. And, as you might know from the previous post, we are able to slow down programs fairly accurately. Figure 1 illustrates the idea with a stacked bar chart for a hypothetical distribution of run-time over three methods. This distribution should remain identical, independent of a slowdown observed by the program. So, there’s a linear relation between the absolute time measured and a constant relation between the percentage of time per method, depending on the slowdown.
Figure 1: A stacked bar chart for a hypothetical program execution, showing the absolute time per method. A profiler should see the linear increase in run time taken by each method, but still report the same percentage of run time taken. If a profiler reports something else, we have found an inaccuracy.
With this slowdown approach, we can detect whether the profiler is accurate with respect to the predicted time increase. I’ll leave all the technical details to the paper. We can also slow down individual basic blocks accurately to make a particular method take more time. As it turns out, this is a good litmus test for the accuracy of profilers, and we find a number of examples where they fail to attribute the run time correctly. Figure 2 shows an example for the Havlak benchmark. The bar charts show how much change the four profilers detect after we slowed down Vector.hasSome to the level indicated by the red dashed line. In this particular example, async-profiler detects the change accurately. JFR is probably within the margin of error. However, JProfiler and YourKit are completely off. JProfiler likely can’t deal with inlining and attributes the change to the forEach method that calls hasSome . YourKit does not seem to see the change at all.
Figure 2: Bar chart with the change in run time between the baseline and slowed-down version, for the top 5 methods of the Havlak benchmark. The red dashed line indicates the expected change for the Vector.hasSome method. Only async-profiler and JFR come close to the expectation.
With this slowdown-based approach, we finally have a way to see how accurate sampling profilers are by approximating the ground truth profile. Since we can’t measure the ground truth directly, we found a way to sidestep a fundamental problem and found a reasonably practical solution.
... continue reading