5 ms·

Good points, but can someone articulate what the best alternative to protobuffers would be in 2018, you know with 'hindsight' etc.?

habitue 8y ago
There's Avro: https://avro.apache.org/docs/current/ https://avro.apache.org/docs/current/ Not sure if the best, but maybe has different tradeoffs
k__ 8y ago
I know a few companies who use flatbuffers instead.
amluto 8y ago
Depending on your goals, Cap’n Proto is excellent.
- jeremyjh 8y ago
  Which of the problems listed in the article are solved by it?
  - danharaj 8y ago
    Parameterization of types, off the top of my head.
    - patrec 8y ago
      Also, cap'n proto loses a whole lot of API and semantic weirdness the author complains about: implicit auto-assignment of fields, "repeated" etc. The overall design feels much cleaner, although there are still some surprises.
CobrastanJorji 8y ago
I've heard good things about https://capnproto.org/ https://capnproto.org/ but the author of this piece may strongly disagree, since I think their objections are rather more fundamental, and also because Cap'n Proto went with banning the concept of "required" fields. I'd also be interested in what alternatives the author might suggest.
- weberc2 8y ago
  Capnproto is built by the guy who originally built protobufs, and he cites capnproto as being better because he was able to learn from his mistakes. Makes me optimistic.
  - enneff 8y ago
    Minor clarification: the author of Capnproto worked on protobufs at Google but was not their original creator.
    - nostrademons 8y ago
      Minor clarification of the clarification: the author of Cap'n'proto was the original creator of Proto2, which was a ground-up (but binary compatible) rewrite of Proto1. The original authors of Proto1 were Jeff Dean & Sanjay Ghemawat. The current version of protobufs is Proto3, which AIUI is maintained by a team at Google (it was released after I left), and is an evolution of the Proto2 codebase with many new features.
      - weberc2 8y ago
        Ah, my mistake. Thanks guys. :+1:
- patrec 8y ago
  You should definitely also look at Cap'n Proto – it is in many ways a theoretically more elegant design: far less overhead and the "time-traveling" RPC design is quite clever. Another potential important advantage, esp for crypto could be that Cap'n Proto in theory has a canonical representation, but in practice this was neither fully defined nor usable last I looked. Cap'n Proto also suffers from a number of practical disadvantages, which can be quite severe depending on your use case. The main problem is eco system maturity and rate of adoption. So whilst protobufs have excellent cross-language support and a mature RPC system, in cap'n proto everything but C++ feels at bit second class, even other popular languages. Python and Rust, and I believe Java have workable, but not necessarily great bindings for serialization and in theory RPC, but the RPC support for other languages than C++ didn't look that great last I looked (but that was over a year ago) and even with C++ it looks much easier to get something going with gRPC (tooling, http2 based etc). For other languages I looked at (e.g. Ocaml, Lua and Haskell) the bindings looked very immature. There are a few other minor annoyance with capnproto, such as the lack of a timestamp type (you can build your own, incompatible to everyone else's, but this is still an annoying omission). The C++ library is also saddled with the weird home-grown kj library that no one other than Kenton Varda uses. However, it has to be said that the C++ API and also the Python API for Capnproto feel a lot more natural and idiomatic than the protobuf ones. If protobuf is too high overhead for your use case and you don't need the wide language and eco system support, Cap'n Proto is worth a serious look.
tlarkworthy 8y ago
JSON or graphql. Personally I think the typing, API compatibility and extensive language support make protobuffers currently the winner though, despite the warts
- vthallam 8y ago
  How would graphql help reduce the latency of API calls between services?
  - hardwaresofton 8y ago
    I'm no GraphQL expert/fan but one example is requesting only the data you need -- you can cut down on what you have to send over the wire which is at least guaranteed to help some. Also, the nested query structure can save what would have normally been multiple requests with a by-the-book RESTful HATEOAS setup.
- FridgeSeal 8y ago
  Neither of those are binary formats though, and if I’m picking up a binary format like Protobuffers it’s probably because I want the performance benefits. Personally, these days I’m taking the approach of just outright avoiding pretty much anything google does. Either because it solves a problem only they have, or they’ll probably just drop it out of the blue one day. Or like Tensorflow, it’s fucking impossible to get your head around because their documentation seems to be written with the goal of being maximally confusing.
- sonnyblarney 8y ago
  I like GraphQL schema, but it would have to have a standardized serialization aspect. Moreover, it really is designed for querying and has a bunch of stuff totally irrelevant to data. But I agree ... I almost wish they'd finish it off and go for full on serialization. That said, it may not be that-that hard, a serialization spec can be pretty short if need be. Certainly one could 'roll one's own' for a mid-sized project if need be.
hardwaresofton 8y ago
How about json + jsonschema + gzipped over the wire
mixmastamyk 8y ago
Apache Thrift is another one.
- mlthoughts2018 8y ago
  Thrift suffers pretty much all the same problems as protobufs and has many similar dysfunctional failure modes, like places that rely on serializing into Thrift structs stores in hdfs and treating that like a de facto database, with Thrift struct definitions as the schema. It is so miserable to work in code bases like that.
  - hermanradtke 8y ago
    This is part of the original lambda architecture. What would you recommend the schema be represented in instead? Or are you preferring something like NewSQL?
    - mlthoughts2018 8y ago
      I can’t tell if you mean a schema for the generated RPC or just for data storage. I’m only talking about data storage. But for RPCs, I just think don’t autogenerate them. Just expose an RPC API and let other people write code to consume it in whatever language’s packages for web requests, and don’t ever transmit things that are expected to automatically be treated as any type of object. Just send JSON or an equivalent thing in a more optimized buffer that is never allowed to be anything but a key-value store for primitive types. On the data side, use relational database systems. Yes, even for huge modern webscale event data for apps or services with hundreds of millions of users. Don’t ever use hadoop, period. If you’re bigger than hundreds of millions of users or otherwise are generating hundreds of billions of records per day or more, that’s the size when you might _start_ considering something different than sharded and distributed standard RDBMSs, but you are likely big enough at that point that you need an in-house, highly customized version of a distributed file store that matches your usage patterns and cost optimizations in a way that a one-size-fits-all solution cannot, and so again you should never be using hadoop. As a result, if you find yourself relying on Thrift data serialized on a distribted filesystem as if those “flat files” are a database and map-reduce is like your de facto table scanner, it’s a red alarm bad code smell that you are growing your data scale in a horribly broken way that’s going to cause huge problems the minute you have products which need a data model or some flexibility that cannot be supported when you’ve welded data infrastructure to Thrift objects.
chabsf 8y ago
Furthermore, in the case where I'm using protobufs as a necessary evil of grpc, what's the best alternative here?
- tathougies 8y ago
  JSON/custom binary format + HTTP2?
  - chabsf 8y ago
    HTTP2 is just the transport, there would be a ton of machinery to implement in each language in the stack.
- romed 8y ago
  protos are not necessary for grpc. You can use whatever payload format you want to use.
abernard1 8y ago
Transit: https://github.com/cognitect/transit-format https://github.com/cognitect/transit-format Transit has the following features: - self-describing - allows extension types on top of base types - strong value types + dynamic serialization - can transparently pass through data without consumer knowing about schema - has "repeated key" optimization in the encoder to allow compression in the stream - allows transparent binary encoding with msgpack - is stream-based, so processing can begin immediately unlike JSON which has enclosing {} This set of features basically allow transparent proxying to JSON as well, which is something that proto2/3 cannot do without updating the proxy.

habitue 8y ago

There's Avro: https://avro.apache.org/docs/current/ https://avro.apache.org/docs/current/ Not sure if the best, but maybe has different tradeoffs

k__ 8y ago

I know a few companies who use flatbuffers instead.

amluto 8y ago

Depending on your goals, Cap’n Proto is excellent.

jeremyjh 8y ago
Which of the problems listed in the article are solved by it?
- danharaj 8y ago
  Parameterization of types, off the top of my head.
  - patrec 8y ago
    Also, cap'n proto loses a whole lot of API and semantic weirdness the author complains about: implicit auto-assignment of fields, "repeated" etc. The overall design feels much cleaner, although there are still some surprises.

CobrastanJorji 8y ago

I've heard good things about https://capnproto.org/ https://capnproto.org/ but the author of this piece may strongly disagree, since I think their objections are rather more fundamental, and also because Cap'n Proto went with banning the concept of "required" fields. I'd also be interested in what alternatives the author might suggest.

weberc2 8y ago
Capnproto is built by the guy who originally built protobufs, and he cites capnproto as being better because he was able to learn from his mistakes. Makes me optimistic.
- enneff 8y ago
  Minor clarification: the author of Capnproto worked on protobufs at Google but was not their original creator.
  - nostrademons 8y ago
    Minor clarification of the clarification: the author of Cap'n'proto was the original creator of Proto2, which was a ground-up (but binary compatible) rewrite of Proto1. The original authors of Proto1 were Jeff Dean & Sanjay Ghemawat. The current version of protobufs is Proto3, which AIUI is maintained by a team at Google (it was released after I left), and is an evolution of the Proto2 codebase with many new features.
    - weberc2 8y ago
      Ah, my mistake. Thanks guys. :+1:
patrec 8y ago
You should definitely also look at Cap'n Proto – it is in many ways a theoretically more elegant design: far less overhead and the "time-traveling" RPC design is quite clever. Another potential important advantage, esp for crypto could be that Cap'n Proto in theory has a canonical representation, but in practice this was neither fully defined nor usable last I looked. Cap'n Proto also suffers from a number of practical disadvantages, which can be quite severe depending on your use case. The main problem is eco system maturity and rate of adoption. So whilst protobufs have excellent cross-language support and a mature RPC system, in cap'n proto everything but C++ feels at bit second class, even other popular languages. Python and Rust, and I believe Java have workable, but not necessarily great bindings for serialization and in theory RPC, but the RPC support for other languages than C++ didn't look that great last I looked (but that was over a year ago) and even with C++ it looks much easier to get something going with gRPC (tooling, http2 based etc). For other languages I looked at (e.g. Ocaml, Lua and Haskell) the bindings looked very immature. There are a few other minor annoyance with capnproto, such as the lack of a timestamp type (you can build your own, incompatible to everyone else's, but this is still an annoying omission). The C++ library is also saddled with the weird home-grown kj library that no one other than Kenton Varda uses. However, it has to be said that the C++ API and also the Python API for Capnproto feel a lot more natural and idiomatic than the protobuf ones. If protobuf is too high overhead for your use case and you don't need the wide language and eco system support, Cap'n Proto is worth a serious look.

tlarkworthy 8y ago

JSON or graphql. Personally I think the typing, API compatibility and extensive language support make protobuffers currently the winner though, despite the warts

vthallam 8y ago
How would graphql help reduce the latency of API calls between services?
- hardwaresofton 8y ago
  I'm no GraphQL expert/fan but one example is requesting only the data you need -- you can cut down on what you have to send over the wire which is at least guaranteed to help some. Also, the nested query structure can save what would have normally been multiple requests with a by-the-book RESTful HATEOAS setup.
FridgeSeal 8y ago
Neither of those are binary formats though, and if I’m picking up a binary format like Protobuffers it’s probably because I want the performance benefits. Personally, these days I’m taking the approach of just outright avoiding pretty much anything google does. Either because it solves a problem only they have, or they’ll probably just drop it out of the blue one day. Or like Tensorflow, it’s fucking impossible to get your head around because their documentation seems to be written with the goal of being maximally confusing.
sonnyblarney 8y ago
I like GraphQL schema, but it would have to have a standardized serialization aspect. Moreover, it really is designed for querying and has a bunch of stuff totally irrelevant to data. But I agree ... I almost wish they'd finish it off and go for full on serialization. That said, it may not be that-that hard, a serialization spec can be pretty short if need be. Certainly one could 'roll one's own' for a mid-sized project if need be.

hardwaresofton 8y ago

How about json + jsonschema + gzipped over the wire

mixmastamyk 8y ago

Apache Thrift is another one.

mlthoughts2018 8y ago
Thrift suffers pretty much all the same problems as protobufs and has many similar dysfunctional failure modes, like places that rely on serializing into Thrift structs stores in hdfs and treating that like a de facto database, with Thrift struct definitions as the schema. It is so miserable to work in code bases like that.
- hermanradtke 8y ago
  This is part of the original lambda architecture. What would you recommend the schema be represented in instead? Or are you preferring something like NewSQL?
  - mlthoughts2018 8y ago
    I can’t tell if you mean a schema for the generated RPC or just for data storage. I’m only talking about data storage. But for RPCs, I just think don’t autogenerate them. Just expose an RPC API and let other people write code to consume it in whatever language’s packages for web requests, and don’t ever transmit things that are expected to automatically be treated as any type of object. Just send JSON or an equivalent thing in a more optimized buffer that is never allowed to be anything but a key-value store for primitive types. On the data side, use relational database systems. Yes, even for huge modern webscale event data for apps or services with hundreds of millions of users. Don’t ever use hadoop, period. If you’re bigger than hundreds of millions of users or otherwise are generating hundreds of billions of records per day or more, that’s the size when you might _start_ considering something different than sharded and distributed standard RDBMSs, but you are likely big enough at that point that you need an in-house, highly customized version of a distributed file store that matches your usage patterns and cost optimizations in a way that a one-size-fits-all solution cannot, and so again you should never be using hadoop. As a result, if you find yourself relying on Thrift data serialized on a distribted filesystem as if those “flat files” are a database and map-reduce is like your de facto table scanner, it’s a red alarm bad code smell that you are growing your data scale in a horribly broken way that’s going to cause huge problems the minute you have products which need a data model or some flexibility that cannot be supported when you’ve welded data infrastructure to Thrift objects.

chabsf 8y ago

Furthermore, in the case where I'm using protobufs as a necessary evil of grpc, what's the best alternative here?

tathougies 8y ago
JSON/custom binary format + HTTP2?
- chabsf 8y ago
  HTTP2 is just the transport, there would be a ton of machinery to implement in each language in the stack.
romed 8y ago
protos are not necessary for grpc. You can use whatever payload format you want to use.

abernard1 8y ago

Transit: https://github.com/cognitect/transit-format https://github.com/cognitect/transit-format Transit has the following features: - self-describing - allows extension types on top of base types - strong value types + dynamic serialization - can transparently pass through data without consumer knowing about schema - has "repeated key" optimization in the encoder to allow compression in the stream - allows transparent binary encoding with msgpack - is stream-based, so processing can begin immediately unlike JSON which has enclosing {} This set of features basically allow transparent proxying to JSON as well, which is something that proto2/3 cannot do without updating the proxy.