6 ms·

Microbenchmarking Mono's new GC

20 points by j_baker 16y ago

Roboprog 16y ago
It looks like a promising improvement. From the sound of it, this seems like the same sort of GC methodology that Java used the last few revs (1.5; 1.6): putting objects in "recent" and "old" pools. That said, I would like to see a benchmark that uses a bit more memory, not just temporary stuff to be discarded ASAP. Many of the Java promotion benchmarks do that too (focus on use of transient data only). What happens if you actually need to hold on to some state, rather than merely concatenating somebody's account info into a dynamic index.html page in a web app? Oh well, there's always memcached, I suppose. It would be nice to have languages that support app types other than stateless-as-possible web apps.
- dfox 16y ago
  Generational GC's are what almost every high-performance VM uses and this approach is well-tested in many practical systems. Almost all aplications tend to allocate large amounts of temporary objects, while having large amounts of memory consumed by long-living objects (I would assume that at least one half of used memory in this case are long-living objects). Also, because generational GC is almost invariably copying GC (it is possible to make non-moving generational GC, but that is more of academic interest) it tends to improve caching behavior, which is even more important with each new generation of CPUs. Main reason why copying and generational GCs are not used everywhere is that they tend to generally complicate matters when you want to seamlessly interface with C code (non-managed code in .NET terms), because objects then tend to be moved around by GC, which has to see and be able to update every reference to it's heap. In most applications and runtimes, there are a lot more behind the stages objects allocated by VM and or library routines than objects that you allocate directly in your code. Edit: and I think that Sun's JVM uses generational GC at least since 1.3, if not 1.2.
  - riffraff 16y ago
    as a side note, bohem's gc is also generational http://www.hpl.hp.com/personal/Hans_Boehm/gc/#details http://www.hpl.hp.com/personal/Hans_Boehm/gc/#details
    - dfox 16y ago
      But generational collection has to be explicitly enabled by GC_enable_incremental() and whole thing looks especially experimental. Main problem is that you cannot reasonably maintain remembered set(or grey set for incremental GC) without write barriers and write barriers are not practical when you cannot modify compiler.
dododo 16y ago
why are so many benchmarks done so badly statistically? the benchmark was run once: we have no idea if this was a fluke or not. hopefully the author didn't repeat and pick the best. at best the numbers reported can be considered some mean. the mean is not a robust estimator: http://en.wikipedia.org/wiki/Robust_statistics#Examples_of_robust_and_non-robust_statistics http://en.wikipedia.org/wiki/Robust_statistics#Examples_of_r... one sample is enough to distort the mean. furthmore we've no idea how typical this run was: there should be some measure of spread.
- viraptor 16y ago
  Zed, is that you? :) http://zedshaw.com/essays/programmer_stats.html http://zedshaw.com/essays/programmer_stats.html I think we've seen that already. Mean is "good enough" for most programmers apparently. And to be honest in many cases, it's ok. It was a microbenchmark, not a big evaluation of latency of both algorithms.
  - dododo 16y ago
    you only know the mean is good enough if you examine the spread and/or use a robust estimator. spikes happen in performance (for better and for worse) all the time. one spike is enough to totally distort the mean, no matter how often you repeat. it might be a microbenchmark but to paraphrase: if it's not done well, it's not worth doing. the complete opposite of what the microbenchmark purports to show could be true most of the time.
    - viraptor 16y ago
      I agree with you from the point of view of people doing proper statistics. Mean is not good enough. Then again - let's say I'm a user of some library and I trust the developers to do the right thing and test the new implementation against the old one. The thing that is of interest to me is whether on average on my random test I still see the difference. He did. Sure - the whole test is flawed, may be incorrect, etc. In reality - he just verified something he already knew/expected against his random data. No matter how much we care about correctness, he's still right on some level. There's a visible improvement even on this random test and if he tried it at least 3 times we can assume that the new GC is worth looking at if you had issues with the previous one. "if it's not done well, it's not worth doing"? I'd say - if it answers your question, it's worth doing. If you assume the question was - "can I verify that the new GC is better in my random not-scientific test on a specific dataset", then everything's all right. And I do hope that the person writing the new GC will publish some better stats.
jules 16y ago
Looks good but one bit triggered WTF: > Mostly precise scanning (stacks and registers are scanned conservatively). Why would you code a conservative garbage collector for .NET?
- dfox 16y ago
  Because of i386. i386 has not enough registers to be able to efficiently divide them into pointer-only and non-pointer-only. For stack case it is often worthwhile to use calling convention reasonably similar to what C/OS uses, but examining that mess is not practical for GC, so it's better to just scan stack conservatively (which is by the way done by many other generational GCs). Edit: also you can implement conservative scan of stack and registers on Unix without writing single line of assembly code by scanning result of getcontext(), or even almost completely portably by scanning result of setjmp() + some heuristics for finding stack in memory.
  - kingkilr 16y ago
    It's still possible, PyPy for example post-processes the assembly in the generated binary in order to identify registers that can contain GC pointers (ones into the heap).
  - ssp 16y ago
    .NET is statically typed though, so it's possible to make a static determination at each GC point which registers and stack locations contain pointers.
    - dfox 16y ago
      Possible, but probably not practical. And especially in case of .NET which offers pretty tight integration with unmanaged code, you can have various stack frames with layout completely out of your control on stack (I'm not sure whenever such stack frames must be scanned for pointers, but they will certainly complicate any attempts to analyze stack structure).
      - 16y ago

20 points by j_baker 16y ago

It looks like a promising improvement. From the sound of it, this seems like the same sort of GC methodology that Java used the last few revs (1.5; 1.6): putting objects in "recent" and "old" pools. That said, I would like to see a benchmark that uses a bit more memory, not just temporary stuff to be discarded ASAP. Many of the Java promotion benchmarks do that too (focus on use of transient data only). What happens if you actually need to hold on to some state, rather than merely concatenating somebody's account info into a dynamic index.html page in a web app? Oh well, there's always memcached, I suppose. It would be nice to have languages that support app types other than stateless-as-possible web apps.

dfox 16y ago
Generational GC's are what almost every high-performance VM uses and this approach is well-tested in many practical systems. Almost all aplications tend to allocate large amounts of temporary objects, while having large amounts of memory consumed by long-living objects (I would assume that at least one half of used memory in this case are long-living objects). Also, because generational GC is almost invariably copying GC (it is possible to make non-moving generational GC, but that is more of academic interest) it tends to improve caching behavior, which is even more important with each new generation of CPUs. Main reason why copying and generational GCs are not used everywhere is that they tend to generally complicate matters when you want to seamlessly interface with C code (non-managed code in .NET terms), because objects then tend to be moved around by GC, which has to see and be able to update every reference to it's heap. In most applications and runtimes, there are a lot more behind the stages objects allocated by VM and or library routines than objects that you allocate directly in your code. Edit: and I think that Sun's JVM uses generational GC at least since 1.3, if not 1.2.
- riffraff 16y ago
  as a side note, bohem's gc is also generational http://www.hpl.hp.com/personal/Hans_Boehm/gc/#details http://www.hpl.hp.com/personal/Hans_Boehm/gc/#details
  - dfox 16y ago
    But generational collection has to be explicitly enabled by GC_enable_incremental() and whole thing looks especially experimental. Main problem is that you cannot reasonably maintain remembered set(or grey set for incremental GC) without write barriers and write barriers are not practical when you cannot modify compiler.

dododo 16y ago

why are so many benchmarks done so badly statistically? the benchmark was run once: we have no idea if this was a fluke or not. hopefully the author didn't repeat and pick the best. at best the numbers reported can be considered some mean. the mean is not a robust estimator: http://en.wikipedia.org/wiki/Robust_statistics#Examples_of_robust_and_non-robust_statistics http://en.wikipedia.org/wiki/Robust_statistics#Examples_of_r... one sample is enough to distort the mean. furthmore we've no idea how typical this run was: there should be some measure of spread.

viraptor 16y ago
Zed, is that you? :) http://zedshaw.com/essays/programmer_stats.html http://zedshaw.com/essays/programmer_stats.html I think we've seen that already. Mean is "good enough" for most programmers apparently. And to be honest in many cases, it's ok. It was a microbenchmark, not a big evaluation of latency of both algorithms.
- dododo 16y ago
  you only know the mean is good enough if you examine the spread and/or use a robust estimator. spikes happen in performance (for better and for worse) all the time. one spike is enough to totally distort the mean, no matter how often you repeat. it might be a microbenchmark but to paraphrase: if it's not done well, it's not worth doing. the complete opposite of what the microbenchmark purports to show could be true most of the time.
  - viraptor 16y ago
    I agree with you from the point of view of people doing proper statistics. Mean is not good enough. Then again - let's say I'm a user of some library and I trust the developers to do the right thing and test the new implementation against the old one. The thing that is of interest to me is whether on average on my random test I still see the difference. He did. Sure - the whole test is flawed, may be incorrect, etc. In reality - he just verified something he already knew/expected against his random data. No matter how much we care about correctness, he's still right on some level. There's a visible improvement even on this random test and if he tried it at least 3 times we can assume that the new GC is worth looking at if you had issues with the previous one. "if it's not done well, it's not worth doing"? I'd say - if it answers your question, it's worth doing. If you assume the question was - "can I verify that the new GC is better in my random not-scientific test on a specific dataset", then everything's all right. And I do hope that the person writing the new GC will publish some better stats.

jules 16y ago

Looks good but one bit triggered WTF: > Mostly precise scanning (stacks and registers are scanned conservatively). Why would you code a conservative garbage collector for .NET?

dfox 16y ago
Because of i386. i386 has not enough registers to be able to efficiently divide them into pointer-only and non-pointer-only. For stack case it is often worthwhile to use calling convention reasonably similar to what C/OS uses, but examining that mess is not practical for GC, so it's better to just scan stack conservatively (which is by the way done by many other generational GCs). Edit: also you can implement conservative scan of stack and registers on Unix without writing single line of assembly code by scanning result of getcontext(), or even almost completely portably by scanning result of setjmp() + some heuristics for finding stack in memory.
- kingkilr 16y ago
  It's still possible, PyPy for example post-processes the assembly in the generated binary in order to identify registers that can contain GC pointers (ones into the heap).
- ssp 16y ago
  .NET is statically typed though, so it's possible to make a static determination at each GC point which registers and stack locations contain pointers.
  - dfox 16y ago
    Possible, but probably not practical. And especially in case of .NET which offers pretty tight integration with unmanaged code, you can have various stack frames with layout completely out of your control on stack (I'm not sure whenever such stack frames must be scanned for pointers, but they will certainly complicate any attempts to analyze stack structure).
    - 16y ago