Does reference counting really use less memory than tracing garbage collection? Mathematica vs Swift vs OCaml vs F# on .NET and Mono
Our previous post caused some controversy by questioning the validity of some commonly-held beliefs. Specifically, the beliefs that reference counting (RC) always requires less memory than tracing garbage collection (GC) and that tracing GCs require 3-4x more memory in order to perform adequately. We presented a benchmark written in Swift and OCaml and noted that the RC'd Swift implementation ran over 5x slower and required over 3x more memory than the tracing GC'd OCaml implementation. This observation disproves these beliefs in their strong form and even brings into question whether there is even any validity in the weak forms of those beliefs. After all, we have never seen any empirical evidence to support these beliefs in any form.
We received a lot of criticism for that post. The vast majority of the criticism was not constructive but two valid points did arise. Firstly, although our result was anomalous it would be more compelling to see the benchmark repeated across a wider variety of production languages using both RC and tracing GC because this would help to reduce the error caused by differences other than GC strategy. Secondly, it would be more compelling to repeat the study with different benchmark tasks in case this one problem (computing symbolic derivatives) is anomalous. This post addresses the former concern.
We implemented equivalent solutions to the original Swift and OCaml, this time in F# and Mathematica. The F# uses a tracing garbage that is inherited from its runtime (either Mono or .NET). Mathematica is a domain specific language (DSL) designed specifically for computer algebra that uses its own heavily-optimized RC implementation so it stands the best possible chance of being able to compete with the original OCaml. So these additional measurements should usefully extend the applicability of our findings.
The F# 4.0 code was run on Mono 4.6.2 directly on the Raspberry Pi 3 alongside the Swift and OCaml and also on .NET 4.7 on Windows 10 (i.e. different architecture and platform!). Our results for Mathematica represent excess memory consumption over the initial consumption (because Mathematica has a uniquely huge standard library) and were obtained running on Mathematica 11.2 in the cloud and 10.0.1 on the Raspberry Pi 3.
According to our measurements, the solutions using tracing GCs require less memory than the solutions using RC in every single case:
Again, we find that this common belief that RC requires less memory to be unjustified.
Just out of interest, here are the run times too (on Raspberry Pi 3 only):
We received a lot of criticism for that post. The vast majority of the criticism was not constructive but two valid points did arise. Firstly, although our result was anomalous it would be more compelling to see the benchmark repeated across a wider variety of production languages using both RC and tracing GC because this would help to reduce the error caused by differences other than GC strategy. Secondly, it would be more compelling to repeat the study with different benchmark tasks in case this one problem (computing symbolic derivatives) is anomalous. This post addresses the former concern.
We implemented equivalent solutions to the original Swift and OCaml, this time in F# and Mathematica. The F# uses a tracing garbage that is inherited from its runtime (either Mono or .NET). Mathematica is a domain specific language (DSL) designed specifically for computer algebra that uses its own heavily-optimized RC implementation so it stands the best possible chance of being able to compete with the original OCaml. So these additional measurements should usefully extend the applicability of our findings.
The F# 4.0 code was run on Mono 4.6.2 directly on the Raspberry Pi 3 alongside the Swift and OCaml and also on .NET 4.7 on Windows 10 (i.e. different architecture and platform!). Our results for Mathematica represent excess memory consumption over the initial consumption (because Mathematica has a uniquely huge standard library) and were obtained running on Mathematica 11.2 in the cloud and 10.0.1 on the Raspberry Pi 3.
According to our measurements, the solutions using tracing GCs require less memory than the solutions using RC in every single case:
Just out of interest, here are the run times too (on Raspberry Pi 3 only):
Comments
Post a Comment