3 ms·

One of the people who designed Protobuf has criticized it (Edit: to clarify, one of the authors of v2, see below), so that doesn't really inspire much confidenc

One of the people who designed Protobuf has criticized it (Edit: to clarify, one of the authors of v2, see below), so that doesn't really inspire much confidence in it for me.[1] Your general point is correct though, there's much more to a well designed RPC system than what HTTP based systems can do, but protobuf/gRPC is very much lacking in ideas that are decades old at this point, like promise pipelining, etc. Also, I feel like she (intentionally maybe) is conflating concurrency and parallelism. These "green thread" systems provide concurrency, but not parallelism. That should be something people are aware of when they use them. [1] https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers https://capnproto.org/faq.html#how-do-i-make-a-field-require...

kentonv 7y ago
> One of the people who designed Protobuf has said it's awful Hmm, you seem to be citing me. To clarify: 1. I didn't design Protobuf. I just rewrote the implementation (created version 2) and open sourced it. 2. I don't think it's awful. In fact, I think it's best-of-breed for what it is, and certainly a much better choice than JSON-over-HTTP. Yes, in Cap'n Proto I've added a bunch of features which Protobuf doesn't have, like zero-copy and promise pipelining, and I obviously think these features make it better tech. But, to be fair, these ideas make Cap'n Proto more complicated, and whether that complexity is worth it is still very much unproven at the scale that Google uses Protobuf and gRPC/Stubby. > These "green thread" systems provide concurrency, but not parallelism. I'm not sure if these specific definitions of "concurrency" and "parallelism" are universal. I wasn't aware of them, at least.
- anaphor 7y ago
  Thanks for the response, I was being a bit tongue in cheek. I don't think you actually think it's awful :) And thanks for clarifying that. About concurrency vs. parallelism, I think it is fairly standard to think of them as two different concepts that overlap somewhat. You can have concurrency with parallelism (e.g. pthreads, or M:N threading where you map "green threads" on to processes that can run in parallel). You can also have concurrency without parallelism. The difference between the two is that parallelism can be deterministic, whereas concurrency is always going to be non-deterministic. I recognize that it's not very clear, but it isn't a distinction that I made up, e.g. https://existentialtype.wordpress.com/2014/04/09/parallelism-and-concurrency-revisited/ https://existentialtype.wordpress.com/2014/04/09/parallelism...
- kragen 7y ago
  Those definitions have been coming into fashion in the last couple of decades. I think it's useful to have the distinction but I wish we had new words that didn't previously mean both things.
- gen220 7y ago
  Hi kentonv! I have a tangential question for you, if you don’t mind. I brought up your capnproto with a few friends at work, while chatting about the profiling data of our services (mostly CPU-bound, mostly on protobuf de-/encoding). After convincing ourselves that language-agnostic “zero-cost” requests weren’t completely magical, and that the whole promise thing is very useful, we got to wondering... Do you think it’s possible, that gRPC/proto could evolve in a non-total-rewrite way to earn the benefits offered by capnproto? I figure you’d be best positioned to answer that kind of question, having worked so intimately on both! :) We have enjoyed getting to know and using gRPC and proto, I also want to thank you for your work! capnproto is an inspiring solution to a prima facie unsolvable problem, I hope to see it succeed more universally, or at least inspire a proto4. :) Thank you again!
  - Youden 7y ago
    I'm not entirely clear on whether this Github issue would bring feature-parity with Cap'n'proto but apparently Google already has a zero-copy API for Protocol Buffers internally: https://github.com/protocolbuffers/protobuf/issues/1896 https://github.com/protocolbuffers/protobuf/issues/1896
    - kentonv 7y ago
      I originally wrote the "zero copy" support in proto2, long before I created Cap'n Proto. What Protobuf means by "zero copy" is much more limited than what Cap'n Proto means. Protobuf's "zero copy" applies only to individual string (or "bytes") fields. The effect is that when you call the getter for that field, you get a pointer to the string inside the original message buffer, rather than a copy allocated on the heap. The overall message structure still needs to be parsed upfront and converted into an object tree on the heap (which I count as a copy). Cap'n Proto is very different. Every Cap'n Proto object is actually a pointer into the original message buffer. Accessing one element of a large array is O(1) -- the previous (and subsequent) elements don't need to be examined at all. Similarly with structs, each field is located at a known fixed offset from the start of the struct, so can be accessed without examining other fields. Protobuf inherently cannot do this; there is no way to know where a field is located without first parsing all previous fields in the same message.
- Izkata 7y ago
  > > These "green thread" systems provide concurrency, but not parallelism. > I'm not sure if these specific definitions of "concurrency" and "parallelism" are universal. I wasn't aware of them, at least. To be clear, since GP didn't define them: concurrency simulates parallelism through context switching. Context switching itself encompasses both cooperative multitasking (gevent does this) and preemptive multitasking (modern operating system threads when they're sharing a CPU). AFAIK it is universal, but close enough not to matter in most cases so people get lazy with their words.

kentonv 7y ago

> One of the people who designed Protobuf has said it's awful Hmm, you seem to be citing me. To clarify: 1. I didn't design Protobuf. I just rewrote the implementation (created version 2) and open sourced it. 2. I don't think it's awful. In fact, I think it's best-of-breed for what it is, and certainly a much better choice than JSON-over-HTTP. Yes, in Cap'n Proto I've added a bunch of features which Protobuf doesn't have, like zero-copy and promise pipelining, and I obviously think these features make it better tech. But, to be fair, these ideas make Cap'n Proto more complicated, and whether that complexity is worth it is still very much unproven at the scale that Google uses Protobuf and gRPC/Stubby. > These "green thread" systems provide concurrency, but not parallelism. I'm not sure if these specific definitions of "concurrency" and "parallelism" are universal. I wasn't aware of them, at least.

anaphor 7y ago
Thanks for the response, I was being a bit tongue in cheek. I don't think you actually think it's awful :) And thanks for clarifying that. About concurrency vs. parallelism, I think it is fairly standard to think of them as two different concepts that overlap somewhat. You can have concurrency with parallelism (e.g. pthreads, or M:N threading where you map "green threads" on to processes that can run in parallel). You can also have concurrency without parallelism. The difference between the two is that parallelism can be deterministic, whereas concurrency is always going to be non-deterministic. I recognize that it's not very clear, but it isn't a distinction that I made up, e.g. https://existentialtype.wordpress.com/2014/04/09/parallelism-and-concurrency-revisited/ https://existentialtype.wordpress.com/2014/04/09/parallelism...
kragen 7y ago
Those definitions have been coming into fashion in the last couple of decades. I think it's useful to have the distinction but I wish we had new words that didn't previously mean both things.
gen220 7y ago
Hi kentonv! I have a tangential question for you, if you don’t mind. I brought up your capnproto with a few friends at work, while chatting about the profiling data of our services (mostly CPU-bound, mostly on protobuf de-/encoding). After convincing ourselves that language-agnostic “zero-cost” requests weren’t completely magical, and that the whole promise thing is very useful, we got to wondering... Do you think it’s possible, that gRPC/proto could evolve in a non-total-rewrite way to earn the benefits offered by capnproto? I figure you’d be best positioned to answer that kind of question, having worked so intimately on both! :) We have enjoyed getting to know and using gRPC and proto, I also want to thank you for your work! capnproto is an inspiring solution to a prima facie unsolvable problem, I hope to see it succeed more universally, or at least inspire a proto4. :) Thank you again!
- Youden 7y ago
  I'm not entirely clear on whether this Github issue would bring feature-parity with Cap'n'proto but apparently Google already has a zero-copy API for Protocol Buffers internally: https://github.com/protocolbuffers/protobuf/issues/1896 https://github.com/protocolbuffers/protobuf/issues/1896
  - kentonv 7y ago
    I originally wrote the "zero copy" support in proto2, long before I created Cap'n Proto. What Protobuf means by "zero copy" is much more limited than what Cap'n Proto means. Protobuf's "zero copy" applies only to individual string (or "bytes") fields. The effect is that when you call the getter for that field, you get a pointer to the string inside the original message buffer, rather than a copy allocated on the heap. The overall message structure still needs to be parsed upfront and converted into an object tree on the heap (which I count as a copy). Cap'n Proto is very different. Every Cap'n Proto object is actually a pointer into the original message buffer. Accessing one element of a large array is O(1) -- the previous (and subsequent) elements don't need to be examined at all. Similarly with structs, each field is located at a known fixed offset from the start of the struct, so can be accessed without examining other fields. Protobuf inherently cannot do this; there is no way to know where a field is located without first parsing all previous fields in the same message.
Izkata 7y ago
> > These "green thread" systems provide concurrency, but not parallelism. > I'm not sure if these specific definitions of "concurrency" and "parallelism" are universal. I wasn't aware of them, at least. To be clear, since GP didn't define them: concurrency simulates parallelism through context switching. Context switching itself encompasses both cooperative multitasking (gevent does this) and preemptive multitasking (modern operating system threads when they're sharing a CPU). AFAIK it is universal, but close enough not to matter in most cases so people get lazy with their words.