4 ms·

We use C and all our nodes are x86... so we just dump packed structs on the wire and the schema is a header file. I suspect this sort of simple approach works i

We use C and all our nodes are x86... so we just dump packed structs on the wire and the schema is a header file. I suspect this sort of simple approach works in most cases... And really that's how the network stack on Linux and Co. is implemented (modulo taking care of endianness)

thadt 4y ago
Yep, that does work quite well in a fixed scope. Till one day someone wakes up and wants a UI in C#. Which is also not a big deal, but you have to either hand roll the serialization code or build a header file parser and generate it. And then someone needs to add a field - so we just tack it onto the end of the struct. This works fine as long as you have the length represented out of band somehow. If not then you get to make struct FooBar2 with your extra field. Then a year later someone has the great idea that it would be great to send text, and now you're making a variable length packet or just always sending around structs with a bunch of empty space and a length field (which is also not too bad). But wait, now the UI team totally needs that data in JavaScript for their new Web UI, so you're back to either generating or hand jamming dozens of structs. All the while tamping out bugs in various places where someone is running old code against new data or old data against new code - all of which have various expectations. Or that microcontroller that is blowing up because it just doesn't like that unaligned integer that is packed into the struct. Not that I'm bitter about that life - it just involved a lot more troubleshooting protocols over the years than I'd have liked. Anyway, those are problems that Protocol Buffers helps solve. But as long as you're using just C and not changing much - packed structs are quite lovely.
- mytailorisrich 4y ago
  Sure, a small header that contains type and length is trivial and pretty much implied in my previous comment. I don't think people should think too much about "what if we change language", etc. because (1) that's unlikely to happen, (2) you have years ahead of you, (3) it may be simpler to convert structs (not the most complicated thing in the world amd supported in most languages in one form or another) into whatever else when/if you actually need it than to overengineer now 'just in case'.
secondcoming 4y ago
Deprecating a field in the struct becomes difficult then without versioning. And versioning only works when someone remembers to increment the version.
__turbobrew__ 4y ago
The main downside I see is performing schema changes. A very common pattern I use with protobufs in distributed systems is: 1.) add a field to the schema 2.) update and deploy the grpc servers to start populating that new field. The old grpc clients will continue to work ignoring the new field 3.) update and deploy the grpc clients to start using the new field With packed structs you don’t get the reverse compatibility, so schemas and endpoints have to be immutable which is not the end of the world but it can make changes more difficult.

thadt 4y ago

Yep, that does work quite well in a fixed scope. Till one day someone wakes up and wants a UI in C#. Which is also not a big deal, but you have to either hand roll the serialization code or build a header file parser and generate it. And then someone needs to add a field - so we just tack it onto the end of the struct. This works fine as long as you have the length represented out of band somehow. If not then you get to make struct FooBar2 with your extra field. Then a year later someone has the great idea that it would be great to send text, and now you're making a variable length packet or just always sending around structs with a bunch of empty space and a length field (which is also not too bad). But wait, now the UI team totally needs that data in JavaScript for their new Web UI, so you're back to either generating or hand jamming dozens of structs. All the while tamping out bugs in various places where someone is running old code against new data or old data against new code - all of which have various expectations. Or that microcontroller that is blowing up because it just doesn't like that unaligned integer that is packed into the struct. Not that I'm bitter about that life - it just involved a lot more troubleshooting protocols over the years than I'd have liked. Anyway, those are problems that Protocol Buffers helps solve. But as long as you're using just C and not changing much - packed structs are quite lovely.

mytailorisrich 4y ago
Sure, a small header that contains type and length is trivial and pretty much implied in my previous comment. I don't think people should think too much about "what if we change language", etc. because (1) that's unlikely to happen, (2) you have years ahead of you, (3) it may be simpler to convert structs (not the most complicated thing in the world amd supported in most languages in one form or another) into whatever else when/if you actually need it than to overengineer now 'just in case'.

secondcoming 4y ago

Deprecating a field in the struct becomes difficult then without versioning. And versioning only works when someone remembers to increment the version.

__turbobrew__ 4y ago

The main downside I see is performing schema changes. A very common pattern I use with protobufs in distributed systems is: 1.) add a field to the schema 2.) update and deploy the grpc servers to start populating that new field. The old grpc clients will continue to work ignoring the new field 3.) update and deploy the grpc clients to start using the new field With packed structs you don’t get the reverse compatibility, so schemas and endpoints have to be immutable which is not the end of the world but it can make changes more difficult.