5 ms·

Since all other comments appear to contradict you and make apologies from authority (after all, Google can't do anything wrong, right?) I'd like to just reassur

Since all other comments appear to contradict you and make apologies from authority (after all, Google can't do anything wrong, right?) I'd like to just reassure you with my 20 years of experience developing C (and the last 10 with C++) in network and system software: that any library -- any library -- that forces internal dynamic memory upon its user smells bad. It's not a universal condemnation, but it begs the question, or rather the skepticism to ask why? In the case of protobufs, there is no real answer. Protobufs is one of the few network libraries that doesn't operate with zero-copy. That steps over a tangible line in the sand. No zero-copy for networking? Forced internal heap allocations with only this arena feature after a decade? Sorry no. Protobufs isn't useful for serious network applications.

tijsvd 6y ago
> No zero-copy for networking? Forced internal heap allocations with only this arena feature after a decade? Sorry no. Protobufs isn't useful for serious network applications. That's a bit harsh. Protobufs deliver smaller wire size than any of the newer "zero-copy" formats. And many receivers of zero-copy formats will... copy the data into some internal representation. If your protobuf implementation delivers classes that are good enough to work with internally (store in maps, forward, etc) then you don't really lose something; instead you gain, due to no manual conversion layer.
- jasonzemos 6y ago
  CBOR also delivers optimal wire sizes with its variable length encoding. Ceteris paribus, there's no technical advantage to using protobufs. The choice is almost always non-technical (business, platform, partnership, etc) or simple naivete. It's not harsh, it's just engineering.
haberman 6y ago
> make apologies from authority (after all, Google can't do anything wrong, right?) That's not my position at all. In my other comment (https://news.ycombinator.com/item?id=25586447 https://news.ycombinator.com/item?id=25586447) I explain how I've spent 10 years trying to improve on protobuf C++ precisely because I agree that some of these limitations are unnecessary. > No zero-copy for networking? Forced internal heap allocations with only this arena feature after a decade? Sorry no. Protobufs isn't useful for serious network applications. I suppose it depends what you are comparing it to. Almost every JSON library has the same limitations you mentioned, and yet many people find JSON useful for network applications. But I agree that giving users full control over allocations makes a library useful in many more situations. I think arena allocation is a pretty reasonable solution to the problem. You can use whatever memory you want for the arena (stack, heap, static buffer) and you can constrain it so that no heap allocations are allowed. Unfortunately protobuf C++ can't live fully within this arena model while it uses std::string for accessors. Hopefully this can be fixed at some point.
- jasonzemos 6y ago
  > I suppose it depends what you are comparing it to. Almost every JSON library has the same limitations you mentioned, and yet many people find JSON useful for network applications. JSON is a human-readable format which is hugely advantageous to develop and operate in many settings. Protobufs doesn't have that advantage. Yet we're paying all of the same costs to structure the data with both. That's enough to posit JSON as a net winner over protobufs. > Unfortunately protobuf C++ can't live fully within this arena model while it uses std::string for accessors. Forcing std::string as a container for core components of a networking API (specifically for an accessor) is an exemplary demonstration of a lack of seriousness in a library.
  - andrekandre 6y ago
    > Forcing std::string as a container for core components of a networking API (specifically for an accessor) is an exemplary demonstration of a lack of seriousness in a library. im not super experienced with c++, so please forgive the maybe obvious question, but why is that? (im guessing it has to do with allowing control over where allocations come from?)
  - arcticbull 6y ago
    In general I agree although I think it's fair to separate the protobuf wire format from the proto3 schema format which had a canonical json representation.
  - kentonv 6y ago
    You can debate the design decisions all you want (and I agree `std::string` was a poor choice here)... but it's kind of absurd to say that a technology underpinning products used by billions of people every day is not "serious".
    - jasonzemos 6y ago
      Why is that absurd? Products used by billions of people may just as well not have serious requirements. I beg not to debate semantics, indeed my own fault, but in the world of network software requirements can start to get very serious, straight through to userspace protocol stacks on DMA'ed device buffers. Would you go through all of the effort to rig an application with userspace networking only to return its results inside an `std::string` -- one of the few std containers where one can't even control the allocator if they wanted to? That's absurd, if anything.
kentonv 6y ago
Unless you are targeting a constrained embedded system, this kind of anti-memory-allocation thinking is counter-productive. Avoiding memory allocation significantly increases the complexity of an API -- or worse, leads to shortcuts like using (thread-unsafe) globals or (overrun-prone) fixed buffers. In C, dealing with memory allocation is such a pain that C programmers still tend to avoid it. But C++, especially post-C++11, makes dynamic memory allocation much, much easier to deal with. (And GC'd languages, obviously, are easier still.) There is still a performance cost, obviously, but that cost almost never matters in application programming use cases, and is even negligible in many systems programming cases. I do think Protobuf does too much allocation, but saying that libraries should not allocate memory at all is an outdated view.
- vvanders 6y ago
  Constrained embedded systems cover a broad range of things up to and including your phone. There's few things I hate to see more than a flat profile from memory allocation or cache misses. The good news is that flatbuffers[1] is a reasonable replacement for most of my use cases. In particular being able to mmap() them directly is a wonderful thing that you can't do with protobufs in addition to being very allocation sparse. [1] https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/
  - kentonv 6y ago
    > Constrained embedded systems cover a broad range of things up to and including your phone. No, modern phones are certainly not constrained in the way I meant, and I don't think you could call them "embedded" either. The common programming languages used on phones are very memory-allocation-friendly. > There's few things I hate to see more than a flat profile from memory allocation or cache misses. I think you may be arguing a different point, or a different level of extremity of the point. Reducing memory allocation to optimize performance is a fine thing that everyone does. The person I was replying to, though, seemed to be asserting that libraries should completely avoid allocating memory for themselves. > The good news is that flatbuffers[1] is a reasonable replacement for most of my use cases. In particular being able to mmap() them directly is a wonderful thing that you can't do with protobufs in addition to being very allocation sparse. Yeah... I'm the author of Cap'n Proto, which has the same property, and predates Flatbuffers.
cbsmith 6y ago
"that any library -- any library -- that forces internal dynamic memory upon its user smells bad" Cough. While with embedded systems there definitely is a big thing about dynamic memory allocation, much as I don't like it, it's not like very popular and successful libraries don't do this. It's a pretty common and accepted practice, and there are standard idioms for how it is done.

tijsvd 6y ago

> No zero-copy for networking? Forced internal heap allocations with only this arena feature after a decade? Sorry no. Protobufs isn't useful for serious network applications. That's a bit harsh. Protobufs deliver smaller wire size than any of the newer "zero-copy" formats. And many receivers of zero-copy formats will... copy the data into some internal representation. If your protobuf implementation delivers classes that are good enough to work with internally (store in maps, forward, etc) then you don't really lose something; instead you gain, due to no manual conversion layer.

jasonzemos 6y ago
CBOR also delivers optimal wire sizes with its variable length encoding. Ceteris paribus, there's no technical advantage to using protobufs. The choice is almost always non-technical (business, platform, partnership, etc) or simple naivete. It's not harsh, it's just engineering.

haberman 6y ago

> make apologies from authority (after all, Google can't do anything wrong, right?) That's not my position at all. In my other comment (https://news.ycombinator.com/item?id=25586447 https://news.ycombinator.com/item?id=25586447) I explain how I've spent 10 years trying to improve on protobuf C++ precisely because I agree that some of these limitations are unnecessary. > No zero-copy for networking? Forced internal heap allocations with only this arena feature after a decade? Sorry no. Protobufs isn't useful for serious network applications. I suppose it depends what you are comparing it to. Almost every JSON library has the same limitations you mentioned, and yet many people find JSON useful for network applications. But I agree that giving users full control over allocations makes a library useful in many more situations. I think arena allocation is a pretty reasonable solution to the problem. You can use whatever memory you want for the arena (stack, heap, static buffer) and you can constrain it so that no heap allocations are allowed. Unfortunately protobuf C++ can't live fully within this arena model while it uses std::string for accessors. Hopefully this can be fixed at some point.

jasonzemos 6y ago
> I suppose it depends what you are comparing it to. Almost every JSON library has the same limitations you mentioned, and yet many people find JSON useful for network applications. JSON is a human-readable format which is hugely advantageous to develop and operate in many settings. Protobufs doesn't have that advantage. Yet we're paying all of the same costs to structure the data with both. That's enough to posit JSON as a net winner over protobufs. > Unfortunately protobuf C++ can't live fully within this arena model while it uses std::string for accessors. Forcing std::string as a container for core components of a networking API (specifically for an accessor) is an exemplary demonstration of a lack of seriousness in a library.
- andrekandre 6y ago
  > Forcing std::string as a container for core components of a networking API (specifically for an accessor) is an exemplary demonstration of a lack of seriousness in a library. im not super experienced with c++, so please forgive the maybe obvious question, but why is that? (im guessing it has to do with allowing control over where allocations come from?)
- arcticbull 6y ago
  In general I agree although I think it's fair to separate the protobuf wire format from the proto3 schema format which had a canonical json representation.
- kentonv 6y ago
  You can debate the design decisions all you want (and I agree `std::string` was a poor choice here)... but it's kind of absurd to say that a technology underpinning products used by billions of people every day is not "serious".
  - jasonzemos 6y ago
    Why is that absurd? Products used by billions of people may just as well not have serious requirements. I beg not to debate semantics, indeed my own fault, but in the world of network software requirements can start to get very serious, straight through to userspace protocol stacks on DMA'ed device buffers. Would you go through all of the effort to rig an application with userspace networking only to return its results inside an `std::string` -- one of the few std containers where one can't even control the allocator if they wanted to? That's absurd, if anything.

kentonv 6y ago

Unless you are targeting a constrained embedded system, this kind of anti-memory-allocation thinking is counter-productive. Avoiding memory allocation significantly increases the complexity of an API -- or worse, leads to shortcuts like using (thread-unsafe) globals or (overrun-prone) fixed buffers. In C, dealing with memory allocation is such a pain that C programmers still tend to avoid it. But C++, especially post-C++11, makes dynamic memory allocation much, much easier to deal with. (And GC'd languages, obviously, are easier still.) There is still a performance cost, obviously, but that cost almost never matters in application programming use cases, and is even negligible in many systems programming cases. I do think Protobuf does too much allocation, but saying that libraries should not allocate memory at all is an outdated view.

vvanders 6y ago
Constrained embedded systems cover a broad range of things up to and including your phone. There's few things I hate to see more than a flat profile from memory allocation or cache misses. The good news is that flatbuffers[1] is a reasonable replacement for most of my use cases. In particular being able to mmap() them directly is a wonderful thing that you can't do with protobufs in addition to being very allocation sparse. [1] https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/
- kentonv 6y ago
  > Constrained embedded systems cover a broad range of things up to and including your phone. No, modern phones are certainly not constrained in the way I meant, and I don't think you could call them "embedded" either. The common programming languages used on phones are very memory-allocation-friendly. > There's few things I hate to see more than a flat profile from memory allocation or cache misses. I think you may be arguing a different point, or a different level of extremity of the point. Reducing memory allocation to optimize performance is a fine thing that everyone does. The person I was replying to, though, seemed to be asserting that libraries should completely avoid allocating memory for themselves. > The good news is that flatbuffers[1] is a reasonable replacement for most of my use cases. In particular being able to mmap() them directly is a wonderful thing that you can't do with protobufs in addition to being very allocation sparse. Yeah... I'm the author of Cap'n Proto, which has the same property, and predates Flatbuffers.

cbsmith 6y ago

"that any library -- any library -- that forces internal dynamic memory upon its user smells bad" Cough. While with embedded systems there definitely is a big thing about dynamic memory allocation, much as I don't like it, it's not like very popular and successful libraries don't do this. It's a pretty common and accepted practice, and there are standard idioms for how it is done.