6 ms·

What sort of allocation rates can servers handle?

40 points by r4um 12y ago

cperciva 12y ago
Can we have s/allocation/memory allocation/ in the title here? It doesn't make much sense otherwise.
- aristidb 12y ago
  What other allocation rates would you immediately think of? Of course allocation rate can mean other things, but memory allocation was the first I thought of.
  - cperciva 12y ago
    Virtual memory, filesystem blocks, processes/threads, TCP ephemeral ports...
  - amelius 12y ago
    Well, it still doesn't make much sense to me. The article speaks of an allocation rate of 400 MB/sec. However, AFAIK, a memory allocator takes time proportional to the number of allocations, NOT the total amount of memory allocated. Or does the "B" stand for block, and not byte somehow? Confused.
    - aardvark179 12y ago
      In a modern pausing GC allocation rates are normally limited by the size of your new generation space so are often best considered in terms of MBs. Thread local allocation buffers often allow an allocation to be a simple pointer increment, so the cost per allocation is tiny providing you can collect or promote from that buffer fast enough.
    - TheLoneWolfling 12y ago
      > However, AFAIK, a memory allocator takes time proportional to the number of allocations, NOT the total amount of memory allocated. Depends on the allocator. On a C-style malloc? "Yes". On a Java-style new? No. Because Java is designed to be sandboxed, and as such cannot allow uninitialized memory to exist. And as such has to make sure that when you create that 1000-element array that all 1000 elements are null. And as such, takes time proportional to both the number of allocations and the amount of memory allocated.
      - stormbrew 12y ago
        Er, this is conflating two different things. Initialization is always linear the number of objects unless it's just not done, but generally speaking even in C or C++ you will still need to perform some kind of initialization on those elements. But really, this has nothing to do with the allocator, which is at a lower level. Malloc is linear to the number of allocations. Automatic tracing garbage collectors are linear to the live set (because they tend to only explore live objects, and there are techniques to make disposing of the non-live objects happen in less than linear time[1]). This can really go either way as to which is better, depending on your allocation rate, which is why this is a point of contention. But none of that has to do with how long it takes to initialize the objects. [1] Semi-space and variants, which are usually used in the allocation-rate-heavy young or nursery generation. Of course if there's extensive use of finalizers you still have to touch those objects to deallocate them.
  - newman314 12y ago
    Resource consumption... CPU, mem, storage etc.
thoughtpolice 12y ago
> When they can choose (and there is a choice) to use collectors that don't pause, the pressure to keep allocation rates down changes, moving the "this is too much" lines up by more than an order of magnitude. You mean - like choose the only garbage collector on earth that actually can do this - for the JVM only - which your old company wrote and keeps guarded as a heavily proprietary secret, with the currently published specifications for the design (in the C4 paper) requiring untenable, unmergeable and extensive hacks in the operating system virtual memory layer that will never be largely accepted, with the current design in your product (which alleviates this) again being a heavily guarded secret that will never be revealed, and never have source available, and costs a shitload of money? Which basically means Azul is going to have the only available pauseless GC for the forseeable future? Well, now that you said it like that - you're right! We really do have a choice... a choice of actually using and working with things that exist and are widely available today, or already living in a land of unlimited money for licensing of a single product that might be completely and totally irrelevant to most of us. I guess the 'choice' isn't so clear, when you say it like that. Look, that aside, great blog post, and extremely informative. I'm a giant fan of Gene's work, seriously. But this work is Azul's brainchild and they'll never let it go. And I don't blame them for that. But drop the rhetoric, please - few people have the 'choice' of spending shitloads of money on a proprietary JVM like Azul.
- fleitz 12y ago
  You can buy servers or you can buy software. Welcome to the real world. Another choice is to write your software without a VM, many languages support this.
  - thoughtpolice 12y ago
    Did I ever say otherwise? I don't blame Azul for making this work proprietary. I'm taking a stance against the rhetoric - this isn't a realistic choice for most people. Most people will buy more hardware over licensing expensive software, especially as hardware generally has reusable value, independent of your technology stack. This is especially true at the low end, where not everyone has massive A series or continuous funding. You have to plan a lot up front at that level to pay off your dividends here if you don't already have steady cash. And many people don't use the JVM at all. So the 'choice' of a pauseless collector is mostly a farce, as far as I can see. > Another choice is to write your software without a VM, many languages support this. Thank you for completely and totally missing the point. Didn't I just say something about "Not everyone uses the JVM", and your response is "just use another completely different language and technology stack"? We're going in circles here. Which other platforms support the C4 collector, including its pause times and sustained allocation rates, might I ask? Freely available open source ones, I might add.
ougawouga 12y ago
Bodes well for Rust as a server language. No GC, and almost as pleasant to write as a high level language. And with memory safety and a compiler that catches errors like crazy.
robotresearcher 12y ago
It looks like this is in the specific context of Java application servers. Is that correct? Can anyone explain how the advice generalizes?
- jallmann 12y ago
  > Can anyone explain how the advice generalizes? The factors constrained by the hardware can be generalized -- cacheability, memory bandwidth, etc. The rest is dependent on the performance characteristics of the runtime, especially the GC. While GC is inapplicable to languages with manually managed memory, allocator performance and fragmentation can still be problematic -- but this is usually much easier to tune. OCaml, which has a very well-designed generational GC, is designed for extremely high allocation rates in its minor heap, to the point where the hardware is the bottleneck -- although compaction of the major heap is still a concern sometimes. While a lot of the recent research into "mechanical sympathy" is fascinating, a lot of it seems driven by the problems brought by high-level languages such as Java. If you're at the point where L1/L2 hit rates are a concern, use a language with inherent control of the memory layout of data structures. The kind of engineering done by Azul, etc is really about making applications utilize the full capabilities of the hardware without being saddled by the underlying platform (eg, JVM).

40 points by r4um 12y ago

cperciva 12y ago

Can we have s/allocation/memory allocation/ in the title here? It doesn't make much sense otherwise.

aristidb 12y ago
What other allocation rates would you immediately think of? Of course allocation rate can mean other things, but memory allocation was the first I thought of.
- cperciva 12y ago
  Virtual memory, filesystem blocks, processes/threads, TCP ephemeral ports...
- amelius 12y ago
  Well, it still doesn't make much sense to me. The article speaks of an allocation rate of 400 MB/sec. However, AFAIK, a memory allocator takes time proportional to the number of allocations, NOT the total amount of memory allocated. Or does the "B" stand for block, and not byte somehow? Confused.
  - aardvark179 12y ago
    In a modern pausing GC allocation rates are normally limited by the size of your new generation space so are often best considered in terms of MBs. Thread local allocation buffers often allow an allocation to be a simple pointer increment, so the cost per allocation is tiny providing you can collect or promote from that buffer fast enough.
  - TheLoneWolfling 12y ago
    > However, AFAIK, a memory allocator takes time proportional to the number of allocations, NOT the total amount of memory allocated. Depends on the allocator. On a C-style malloc? "Yes". On a Java-style new? No. Because Java is designed to be sandboxed, and as such cannot allow uninitialized memory to exist. And as such has to make sure that when you create that 1000-element array that all 1000 elements are null. And as such, takes time proportional to both the number of allocations and the amount of memory allocated.
    - stormbrew 12y ago
      Er, this is conflating two different things. Initialization is always linear the number of objects unless it's just not done, but generally speaking even in C or C++ you will still need to perform some kind of initialization on those elements. But really, this has nothing to do with the allocator, which is at a lower level. Malloc is linear to the number of allocations. Automatic tracing garbage collectors are linear to the live set (because they tend to only explore live objects, and there are techniques to make disposing of the non-live objects happen in less than linear time[1]). This can really go either way as to which is better, depending on your allocation rate, which is why this is a point of contention. But none of that has to do with how long it takes to initialize the objects. [1] Semi-space and variants, which are usually used in the allocation-rate-heavy young or nursery generation. Of course if there's extensive use of finalizers you still have to touch those objects to deallocate them.
- newman314 12y ago
  Resource consumption... CPU, mem, storage etc.

thoughtpolice 12y ago

> When they can choose (and there is a choice) to use collectors that don't pause, the pressure to keep allocation rates down changes, moving the "this is too much" lines up by more than an order of magnitude. You mean - like choose the only garbage collector on earth that actually can do this - for the JVM only - which your old company wrote and keeps guarded as a heavily proprietary secret, with the currently published specifications for the design (in the C4 paper) requiring untenable, unmergeable and extensive hacks in the operating system virtual memory layer that will never be largely accepted, with the current design in your product (which alleviates this) again being a heavily guarded secret that will never be revealed, and never have source available, and costs a shitload of money? Which basically means Azul is going to have the only available pauseless GC for the forseeable future? Well, now that you said it like that - you're right! We really do have a choice... a choice of actually using and working with things that exist and are widely available today, or already living in a land of unlimited money for licensing of a single product that might be completely and totally irrelevant to most of us. I guess the 'choice' isn't so clear, when you say it like that. Look, that aside, great blog post, and extremely informative. I'm a giant fan of Gene's work, seriously. But this work is Azul's brainchild and they'll never let it go. And I don't blame them for that. But drop the rhetoric, please - few people have the 'choice' of spending shitloads of money on a proprietary JVM like Azul.

fleitz 12y ago
You can buy servers or you can buy software. Welcome to the real world. Another choice is to write your software without a VM, many languages support this.
- thoughtpolice 12y ago
  Did I ever say otherwise? I don't blame Azul for making this work proprietary. I'm taking a stance against the rhetoric - this isn't a realistic choice for most people. Most people will buy more hardware over licensing expensive software, especially as hardware generally has reusable value, independent of your technology stack. This is especially true at the low end, where not everyone has massive A series or continuous funding. You have to plan a lot up front at that level to pay off your dividends here if you don't already have steady cash. And many people don't use the JVM at all. So the 'choice' of a pauseless collector is mostly a farce, as far as I can see. > Another choice is to write your software without a VM, many languages support this. Thank you for completely and totally missing the point. Didn't I just say something about "Not everyone uses the JVM", and your response is "just use another completely different language and technology stack"? We're going in circles here. Which other platforms support the C4 collector, including its pause times and sustained allocation rates, might I ask? Freely available open source ones, I might add.

ougawouga 12y ago

Bodes well for Rust as a server language. No GC, and almost as pleasant to write as a high level language. And with memory safety and a compiler that catches errors like crazy.

robotresearcher 12y ago

It looks like this is in the specific context of Java application servers. Is that correct? Can anyone explain how the advice generalizes?

jallmann 12y ago
> Can anyone explain how the advice generalizes? The factors constrained by the hardware can be generalized -- cacheability, memory bandwidth, etc. The rest is dependent on the performance characteristics of the runtime, especially the GC. While GC is inapplicable to languages with manually managed memory, allocator performance and fragmentation can still be problematic -- but this is usually much easier to tune. OCaml, which has a very well-designed generational GC, is designed for extremely high allocation rates in its minor heap, to the point where the hardware is the bottleneck -- although compaction of the major heap is still a concern sometimes. While a lot of the recent research into "mechanical sympathy" is fascinating, a lot of it seems driven by the problems brought by high-level languages such as Java. If you're at the point where L1/L2 hit rates are a concern, use a language with inherent control of the memory layout of data structures. The kind of engineering done by Azul, etc is really about making applications utilize the full capabilities of the hardware without being saddled by the underlying platform (eg, JVM).