On "Quantifying the Performance of Garbage Collection vs. Explicit Memory Management"

The computer science research paper "Quantifying the Performance of Garbage Collection vs. Explicit Memory Management" by Emery Berger and Matthew Hertz contains an interesting study about memory management. However, the conclusions given in the paper were badly worded and are now being used to justify an anti-GC ideology.

Introduction

That paper describes an experiment that analyzed the performance of a benchmark suite using:

1. Tracing garbage collection.
2. Oracular memory management (precomputing the earliest point free could have been inserted).

The experiment was performed:

• On one VM, the Jikes Research Virtual Machine (RVM).
• Using one programming language, Java, and consequently one programming paradigm, object oriented programming.
• Using each of the five different garbage collection algorithms provided by that VM.

The five GC algorithms are:

• GenMS - Appel-style generational collector (1988)
• GenCopy - two generations with copying mature space
• CopyMS - nursery with whole-heap collection
• Semispace - Cheney semi-space (1970)
• MarkSweep - non-relocating, non-copying single-generation

Note that none of these GC algorithms are in widespread use today. The only one that comes close is Semispace which is one of Erlang's GC algorithms, albeit used in a substantially different way. In fact, the algorithms studied are all at least 30 years old.

The Obvious Discussion

The experimental results presented in the paper are very interesting and immediately raise many questions. Perhaps the most obvious question is: do these results generalize to any other programming paradigms, languages, virtual machines or garbage collection algorithms?
In the absence of evidence let's just think about this first. Functional languages use different data structures (purely functional or persistent collections) in different ways (recursion via tail calls rather than loops and mutation) so I see no reason to assume that they would behave in the same way that Java does in this context. Indeed, lean functional languages like OCaml famously require far less memory than Java to solve the same problem and the relative bloat is usually attributed to object oriented programming.
Closer to Java is C# on .NET but even this is substantially different because reified generics and value types result in aggressive unboxing giving a flatter heap topology and relieving stress from the GC. This manifests particularly in the context of hash tables where .NET provides an efficient generic hash table implementation that the JVM is incapable of expressing. So it seems extremely unlikely that these results would even generalize to .NET languages like C#.
Still, the observations are interesting.

Their Conclusions

Sadly, rather than asking these obvious questions and suggesting further work the authors chose to draw several completely unjustified conclusions. Worse, this paper passed peer review and has been published containing these unjustified conclusions and is now being bandied around on the internet.
Firstly, the paper concludes:

"In particular, when garbage collection has five times as much memory as required, its runtime performance matches or slightly exceeds that of explicit memory management"

The evidence described in the paper justifies the conclusion that at least for these benchmarks in Java on the Jikes RVM the GenMS GC algorithm requires five times more memory than oracular memory management to achieve the same level of performance. This extremely specific observation cannot possibly be generalized to all garbage collectors.

The paper goes on to conclude:

"Practitioners can use these results to guide their choice of explicitly-managed languages like C or C++, or garbage-collected languages like Java or C#"

Not only is there no evidence that these results have any bearing on C# whatsoever but there is also no evidence to substantiate the implication that the oracular memory management studied here is at all representative of hand-written C or C++ code. Remember: the oracle precomputes the optimal place to free every heap allocated block of memory. Is it reasonable to assume that human software developers will do the same? Absolutely not. In practice, software developers using such languages employ pool allocators that amortise the cost of allocation at the expense of dramatically increasing the amount of floating garbage. Furthermore, the program generated by the oracular method is not a general solution that will run correctly on different inputs: it is only correct for and is optimised for the one input it was given! This is akin to hoisting computation out of the benchmark and is completely unrepresentative of real software.
So, again, it seems extremely unlikely that the oracular memory management described here (although interesting) is at all representative of manual memory management in practice.
Finally, there are two major families of garbage collection: tracing and reference counting. All of the garbage collectors covered by this paper are tracing garbage collectors so the results say nothing about reference counting at all, much less the whole of garbage collection in general.

Our Conclusions

This paper is another piece of an interesting puzzle but it is a puzzle that will never be solved. No amount of objective quantitative factual evidence can ever be enough to justify a completely general conclusion about garbage collection. Hopefully someday someone will repeat this experiment using a different VM, like .NET Core, and a different language and paradigm, like F#. Until then, there is no evidence to support the hypothesis that garbage collection requires 5x more memory to achieve the same performance as manual memory management.