3 ms·

That's sort of where I've landed too. Protobufs would seem to fit the problem area well, but in practice the space between "big-system non-performance-sensitiv

by ajross 2y ago

That's sort of where I've landed too. Protobufs would seem to fit the problem area well, but in practice the space between "big-system non-performance-sensitive data transfer metaformat"[1] and "super-performance-sensitive custom binary parser"[2] is... actually really small. There are just very few spots that actually "need" protobuf at a level of urgency that would justify walking away from self-describing text formats (which is a big, big disadvantage for binary formats!). [1] Something very well served by JSON [2] Network routing, stateful packet inspection, on-the-fly transcoding. Stuff that you'd never think to use a "standard format" for.

bboygravity 2y ago
Add "everything that communicates with a microcontroller" to 2. That means potentially: the majority of devices in the world.
- thadt 2y ago
  Perhaps surprisingly, I think microcontrollers may be a place where Protobufs are not a bad fit. Using something like Nanopb [1] gives you the size/speed/flexibility advantages of protocol buffers without being too heavyweight. It’ll be a bit slower than your custom binary protocol, but it comes with quite a few advantages, depending on the context. [1] https://github.com/nanopb/nanopb https://github.com/nanopb/nanopb
  - ajross 2y ago
    We carry a nanopb integration in Zephyr. And even there... meh. It's true that there are some really bad binary protocols in the embedded world. And protobufs are for sure a step up from a median command parser or whatever. And they have real size advantages vs. JSON for tiny/sub-megabyte devices, which is real. But even there, I find that really these are very big machines in a historical sense. And text parsing is really not that hard, or that big. The first HTTP server was on a 25MHz 68040! Just use JSON. Anything else in the modern world needs to be presumed to be premature optimization absent a solid analysis with numbers.
    - dpeckett 2y ago
      Amen to that.
    - imtringued 2y ago
      If you use JSON in an embedded capacity and you're stingy with your bytes, you can just send arrays as your main document. There was this one MMO I played, where every packet was just a space separated string, with a fancy variable length number encoding that let them store two digits in a single character. There is not much difference between walk <PlayerId> <x> <y> walk 5 100 300 and ['walk', '<PlayerId>', '<x>', '<y>'] ['walk', 5, 100, 300] in terms of bytes and parsing, this is trivial, but it is a standard JSON document and everyone knows JSON, which is a huge win on the developer side.

by ajross 2y ago

bboygravity 2y ago

Add "everything that communicates with a microcontroller" to 2. That means potentially: the majority of devices in the world.

thadt 2y ago
Perhaps surprisingly, I think microcontrollers may be a place where Protobufs are not a bad fit. Using something like Nanopb [1] gives you the size/speed/flexibility advantages of protocol buffers without being too heavyweight. It’ll be a bit slower than your custom binary protocol, but it comes with quite a few advantages, depending on the context. [1] https://github.com/nanopb/nanopb https://github.com/nanopb/nanopb
- ajross 2y ago
  We carry a nanopb integration in Zephyr. And even there... meh. It's true that there are some really bad binary protocols in the embedded world. And protobufs are for sure a step up from a median command parser or whatever. And they have real size advantages vs. JSON for tiny/sub-megabyte devices, which is real. But even there, I find that really these are very big machines in a historical sense. And text parsing is really not that hard, or that big. The first HTTP server was on a 25MHz 68040! Just use JSON. Anything else in the modern world needs to be presumed to be premature optimization absent a solid analysis with numbers.
  - dpeckett 2y ago
    Amen to that.
  - imtringued 2y ago
    If you use JSON in an embedded capacity and you're stingy with your bytes, you can just send arrays as your main document. There was this one MMO I played, where every packet was just a space separated string, with a fancy variable length number encoding that let them store two digits in a single character. There is not much difference between walk <PlayerId> <x> <y> walk 5 100 300 and ['walk', '<PlayerId>', '<x>', '<y>'] ['walk', 5, 100, 300] in terms of bytes and parsing, this is trivial, but it is a standard JSON document and everyone knows JSON, which is a huge win on the developer side.