4 ms·

Unless you are targeting a constrained embedded system, this kind of anti-memory-allocation thinking is counter-productive. Avoiding memory allocation significa

Unless you are targeting a constrained embedded system, this kind of anti-memory-allocation thinking is counter-productive. Avoiding memory allocation significantly increases the complexity of an API -- or worse, leads to shortcuts like using (thread-unsafe) globals or (overrun-prone) fixed buffers. In C, dealing with memory allocation is such a pain that C programmers still tend to avoid it. But C++, especially post-C++11, makes dynamic memory allocation much, much easier to deal with. (And GC'd languages, obviously, are easier still.) There is still a performance cost, obviously, but that cost almost never matters in application programming use cases, and is even negligible in many systems programming cases. I do think Protobuf does too much allocation, but saying that libraries should not allocate memory at all is an outdated view.

vvanders 6y ago
Constrained embedded systems cover a broad range of things up to and including your phone. There's few things I hate to see more than a flat profile from memory allocation or cache misses. The good news is that flatbuffers[1] is a reasonable replacement for most of my use cases. In particular being able to mmap() them directly is a wonderful thing that you can't do with protobufs in addition to being very allocation sparse. [1] https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/
- kentonv 6y ago
  > Constrained embedded systems cover a broad range of things up to and including your phone. No, modern phones are certainly not constrained in the way I meant, and I don't think you could call them "embedded" either. The common programming languages used on phones are very memory-allocation-friendly. > There's few things I hate to see more than a flat profile from memory allocation or cache misses. I think you may be arguing a different point, or a different level of extremity of the point. Reducing memory allocation to optimize performance is a fine thing that everyone does. The person I was replying to, though, seemed to be asserting that libraries should completely avoid allocating memory for themselves. > The good news is that flatbuffers[1] is a reasonable replacement for most of my use cases. In particular being able to mmap() them directly is a wonderful thing that you can't do with protobufs in addition to being very allocation sparse. Yeah... I'm the author of Cap'n Proto, which has the same property, and predates Flatbuffers.
  - vvanders 6y ago
    > The common programming languages used on phones are very memory-allocation-friendly. I would hardly classify Dalvik or ART as "allocation friendly", they don't perform escape analysis and if you do it constantly you'll be in a world of constant hard GC pauses. Multiple times over the years I've had to build free-lists in Java to avoid this specific problem. Same for C++ if you use one of the built-in generic memory allocators. The fastest new/delete are the ones that you don't call. For what it's worth I tend to agree with the grand-parent thread. The lack of awareness of allocations, cache-invalidation via indirection are a significant contributor to why we see software clawing back hardware wins across the years on these platforms.
    - cbsmith 6y ago
      > I would hardly classify Dalvik or ART as "allocation friendly", they don't perform escape analysis and if you do it constantly you'll be in a world of constant hard GC pauses. Multiple times over the years I've had to build free-lists in Java to avoid this specific problem. I don't know what you're doing, but I've generally found free-lists to be a net performance negative in Java libraries. Time and again, I've been called in to "optimize" Java code that uses them, and usually by simply removing them I can get rid of the performance problems entirely.
      - vvanders 6y ago
        Dalvik and ART have different behaviors than traditional Java VMs. Dalvik is aimed at low memory devices, ART trades some memory for speed but still is significantly different(i.e. it doesn't do escape analysis). As always benchmarking on hardware is the way to confirm this but I've had numerous cases where free lists or pre-allocated arrays has gained 10-30x performance improvements. Flatbuffers in particular excels here since once you have the ByteBuffer in memory you can immediately start accessing data without needing to do any extensive parsing. Even the official Android docs are very explicit[1] on the allocation point. Allocations are not cheap and even with generational collection you still will blow the 16.6ms frame window for 60 FPS if any of your operations allocate excessively. [1] https://developer.android.com/training/custom-views/optimizing-view#less https://developer.android.com/training/custom-views/optimizi...
jasonzemos 6y ago
> Avoiding memory allocation significantly increases the complexity of an API -- or worse, leads to shortcuts like using (thread-unsafe) globals or (overrun-prone) fixed buffers. Allow me to unpack this, because the object interface that all of these serialization schemes present is what increases complexity. It makes sense that the simple implementation would tend toward dynamic memory and not further compound such complexity on their interface. This is a flawed architectural premise taken by these library implementations. On the receiving side, there is little reason to ever deviate from zero-copy/zero-allocate. These libraries are far too pro-active in deserializing incoming data into native object structures. It's not necessary to present the user with this; they should pique through the results lazily -- query into the data at their own discretion. This is important because it bounds the minimum requisite cost of receiving messages at ... zero. It costs nothing to discard trash. Whereas with these proactive serialization layers, every single incoming attack becomes an exercise against the deserializer. In a zero-copy/zero-allocate -- let's even say, zero-parse mere lexing of the data, software has more flexibility. In practice, that means passing pointers around to parts of messages straight off the wire to application functionality further up the stack. On the sending side, there is little reason to ever force whole native object representations to conduct a serialization. In other words, don't force the user to build an std::map first and serialize it later. All input is streaming input. Properties do not have to be pre-buffered, they can be streamed. One can model any network serialization this way, sans perhaps canonical representations (sorted JSON keys, etc), and far more efficiently than with requiring arbitrary native structures. This is not really a rebuttal about the merits for or against dynamic memory in and of itself. I know the research, thread-aware allocators can be pretty good -- even darn good, and the state of the art in GC is nothing to shake a stick at. The problem is that with better design it's just not necessary, and in the end it does have a cost that one should want to eliminate if possible. I'm certain it's quite usually possible, at least more than I see in a list like https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats https://en.wikipedia.org/wiki/Comparison_of_data-serializati... etc
- kentonv 6y ago
  What you describe sounds exactly like how Cap'n Proto works. Obviously I like that design since I wrote it. But it does have drawbacks. The encoded size is necessarily a bit larger to allow random access traversal of the raw bytes (though it compresses well). The API to manipulate structures in-place is a little awkward, particularly on the writing side. And, you can't really use the generated types as mutable in-memory state, as people commonly like to do with Protobuf types -- as a result, a common feature request for Cap'n Proto is to support generating "native" structs with the ability to convert between those and the zero-copy types as desired. Everything is full of trade-offs. I don't disagree with your design preferences but I do object to the extreme line you are taking on them. You said: "Protobufs isn't useful for serious network applications." That is plainly contradicted by the existence of a trillion-dollar company built on said technology.
mcguire 6y ago
Once upon a time, we were concerned with handling network traffic at wire speed. I can remember when the AIX folks were happy they could implement the TCP fast path entirely in the interrupt handler.
- kentonv 6y ago
  Certainly you wouldn't use protobuf to encode TCP packets. It's meant for encoding application-layer messages, typically ones that aren't especially large but contain a lot of complex structure.
  - mcguire 6y ago
    Where efficiency doesn't matter because they don't need to be processed quickly and cpu cycles and memory are free.

vvanders 6y ago

Constrained embedded systems cover a broad range of things up to and including your phone. There's few things I hate to see more than a flat profile from memory allocation or cache misses. The good news is that flatbuffers[1] is a reasonable replacement for most of my use cases. In particular being able to mmap() them directly is a wonderful thing that you can't do with protobufs in addition to being very allocation sparse. [1] https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/

kentonv 6y ago
> Constrained embedded systems cover a broad range of things up to and including your phone. No, modern phones are certainly not constrained in the way I meant, and I don't think you could call them "embedded" either. The common programming languages used on phones are very memory-allocation-friendly. > There's few things I hate to see more than a flat profile from memory allocation or cache misses. I think you may be arguing a different point, or a different level of extremity of the point. Reducing memory allocation to optimize performance is a fine thing that everyone does. The person I was replying to, though, seemed to be asserting that libraries should completely avoid allocating memory for themselves. > The good news is that flatbuffers[1] is a reasonable replacement for most of my use cases. In particular being able to mmap() them directly is a wonderful thing that you can't do with protobufs in addition to being very allocation sparse. Yeah... I'm the author of Cap'n Proto, which has the same property, and predates Flatbuffers.
- vvanders 6y ago
  > The common programming languages used on phones are very memory-allocation-friendly. I would hardly classify Dalvik or ART as "allocation friendly", they don't perform escape analysis and if you do it constantly you'll be in a world of constant hard GC pauses. Multiple times over the years I've had to build free-lists in Java to avoid this specific problem. Same for C++ if you use one of the built-in generic memory allocators. The fastest new/delete are the ones that you don't call. For what it's worth I tend to agree with the grand-parent thread. The lack of awareness of allocations, cache-invalidation via indirection are a significant contributor to why we see software clawing back hardware wins across the years on these platforms.
  - cbsmith 6y ago
    > I would hardly classify Dalvik or ART as "allocation friendly", they don't perform escape analysis and if you do it constantly you'll be in a world of constant hard GC pauses. Multiple times over the years I've had to build free-lists in Java to avoid this specific problem. I don't know what you're doing, but I've generally found free-lists to be a net performance negative in Java libraries. Time and again, I've been called in to "optimize" Java code that uses them, and usually by simply removing them I can get rid of the performance problems entirely.
    - vvanders 6y ago
      Dalvik and ART have different behaviors than traditional Java VMs. Dalvik is aimed at low memory devices, ART trades some memory for speed but still is significantly different(i.e. it doesn't do escape analysis). As always benchmarking on hardware is the way to confirm this but I've had numerous cases where free lists or pre-allocated arrays has gained 10-30x performance improvements. Flatbuffers in particular excels here since once you have the ByteBuffer in memory you can immediately start accessing data without needing to do any extensive parsing. Even the official Android docs are very explicit[1] on the allocation point. Allocations are not cheap and even with generational collection you still will blow the 16.6ms frame window for 60 FPS if any of your operations allocate excessively. [1] https://developer.android.com/training/custom-views/optimizing-view#less https://developer.android.com/training/custom-views/optimizi...

jasonzemos 6y ago

> Avoiding memory allocation significantly increases the complexity of an API -- or worse, leads to shortcuts like using (thread-unsafe) globals or (overrun-prone) fixed buffers. Allow me to unpack this, because the object interface that all of these serialization schemes present is what increases complexity. It makes sense that the simple implementation would tend toward dynamic memory and not further compound such complexity on their interface. This is a flawed architectural premise taken by these library implementations. On the receiving side, there is little reason to ever deviate from zero-copy/zero-allocate. These libraries are far too pro-active in deserializing incoming data into native object structures. It's not necessary to present the user with this; they should pique through the results lazily -- query into the data at their own discretion. This is important because it bounds the minimum requisite cost of receiving messages at ... zero. It costs nothing to discard trash. Whereas with these proactive serialization layers, every single incoming attack becomes an exercise against the deserializer. In a zero-copy/zero-allocate -- let's even say, zero-parse mere lexing of the data, software has more flexibility. In practice, that means passing pointers around to parts of messages straight off the wire to application functionality further up the stack. On the sending side, there is little reason to ever force whole native object representations to conduct a serialization. In other words, don't force the user to build an std::map first and serialize it later. All input is streaming input. Properties do not have to be pre-buffered, they can be streamed. One can model any network serialization this way, sans perhaps canonical representations (sorted JSON keys, etc), and far more efficiently than with requiring arbitrary native structures. This is not really a rebuttal about the merits for or against dynamic memory in and of itself. I know the research, thread-aware allocators can be pretty good -- even darn good, and the state of the art in GC is nothing to shake a stick at. The problem is that with better design it's just not necessary, and in the end it does have a cost that one should want to eliminate if possible. I'm certain it's quite usually possible, at least more than I see in a list like https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats https://en.wikipedia.org/wiki/Comparison_of_data-serializati... etc

kentonv 6y ago
What you describe sounds exactly like how Cap'n Proto works. Obviously I like that design since I wrote it. But it does have drawbacks. The encoded size is necessarily a bit larger to allow random access traversal of the raw bytes (though it compresses well). The API to manipulate structures in-place is a little awkward, particularly on the writing side. And, you can't really use the generated types as mutable in-memory state, as people commonly like to do with Protobuf types -- as a result, a common feature request for Cap'n Proto is to support generating "native" structs with the ability to convert between those and the zero-copy types as desired. Everything is full of trade-offs. I don't disagree with your design preferences but I do object to the extreme line you are taking on them. You said: "Protobufs isn't useful for serious network applications." That is plainly contradicted by the existence of a trillion-dollar company built on said technology.

mcguire 6y ago

Once upon a time, we were concerned with handling network traffic at wire speed. I can remember when the AIX folks were happy they could implement the TCP fast path entirely in the interrupt handler.

kentonv 6y ago
Certainly you wouldn't use protobuf to encode TCP packets. It's meant for encoding application-layer messages, typically ones that aren't especially large but contain a lot of complex structure.
- mcguire 6y ago
  Where efficiency doesn't matter because they don't need to be processed quickly and cpu cycles and memory are free.