4 ms·

>Things like passing protobufs/gRPC when C structs, JSON, or CSV files will do These will always get the job done, but at great cost to future maintainers and

>Things like passing protobufs/gRPC when C structs, JSON, or CSV files will do These will always get the job done, but at great cost to future maintainers and callers. Explicitly declared interfaces that give some thought to future evolution aren't that hard, and can save a lot of pain down the road. I vividly remember the outage I caused by changing a struct field name, only to find out it was implicitly a critical part of the API "contract" when serialized to JSON. Another where I relied on a certain field from a dependency's response, only to find it out it was populated in some situations (the examples I used to learn the interface) and not others (revealed by production). Our Thrift services do not have these problems. Anyone can look at a diff and tell a) that the IDL is changing, b) whether the change is breaking, and c) whether a field is required or optional.

deleted 7y ago
[deleted]
pjc50 7y ago
GRPC is great as an API generator, but there's a lot of machinery under the hood in its network layer. It's not small - there was an embedded project where we were considering using it to replace our roll-your-own, then we discovered that the compiled library for GRPC was larger than the entire application. C structs are so brittle they should not be used for any kind of serialisation. You're just setting yourself up for upgrade nightmares. I wish BSON or the ASN.1 toolset were easier to use for this use case.
- fanf2 7y ago
  I hope the CBOR tooling is better https://cbor.io/tools.html https://cbor.io/tools.html
- CoolGuySteve 7y ago
  For C structs, start with a uint32_t 4 char code identifier and a size field. Then you can extend them as long as you only ever append, which is a constraint on protobufs anyways. This is how most media containers have worked since the Amiga days. Meanwhile, their encoding/decoding is infinitely faster than most other serialization formats, allowing you to serialize data back and forth to disk or as IPC faster than anything else.
  - pjc50 7y ago
    .. as long as you never change endianness, structure packing, word alignment requirements, and always use types with explicit sizes. And you have to keep the data in the structure itself, so as soon as you hit a pointer you need to write explicit serialisation.
    - CoolGuySteve 7y ago
      > .. as long as you never change endianness, structure packing, word alignment requirements, and always use types with explicit sizes. #pragma pack(1) and byteswapping solve all this and neither are more work than specifying some intermediate format and calling some external tool in your build system. In C++ you can make types that swap automatically if required. > And you have to keep the data in the structure itself, so as soon as you hit a pointer you need to write explicit serialisation. I have yet to see a serialization format that does this well and quickly. Just make a new message type for it that follows the first.
    - makapuf 7y ago
      Or juste use riff which useful and provide just enough to implement wav, bmp and other file formats (so you'll find heaps of examples(.
    - Matthias247 7y ago
      Indeed. Using plain structs one can't just add a single field with a reasonable default and still be backward compatible - which is a very very common case in API design. You could add completely new revision of the API. But having to maintain it - typically additionally to the old revision since both sides might not updateable in atomic fashion - is a lot of extra work. Serialization which allows to add and remove fields in a backward compatible fashion is a mandatory requirement for any bigger sized project for me. If you need super high performance maybe use Flatbuffers or Cap'N'Proto instead of Protobuf/JSON/CBOR - but I really wouldn't recommend to go for plain structs.
- closeparen 7y ago
  >but there's a lot of machinery under the hood in its network layer GRPC is designed to enable peer-to-peer communication in a microservices mesh, so it has hooks for a number of things like service discovery, authentication, health checking, retry policy, connection pooling, observability, etc. If you're just looking for a serialization format you might prefer Protobuf itself.
deleted 7y ago
[deleted]
dkarl 7y ago
I vividly remember the outage I caused by changing a struct field name, only to find out it was implicitly a critical part of the API "contract" when serialized to JSON. Lack of unit tests for serialization can be deadly for projects where the serialization format is defined implicitly via the application model. I've seen enough projects bitten by this to be convinced it's worth writing ser/des unit tests for every API object. For some reason it's a form of testing that even gung-ho unit testers tend to leave out. Arguably it should be covered by integration tests, but integration tests usually only test round-trip serialization, which rarely breaks when you're testing a codebase against itself. But I think the issue you faced was a documentation issue as well. It should have been evident which objects defined the API. In some projects the objects have to be heavily annotated to make ser/des work at all, which makes it obvious, but when it's possible to separate concerns and define ser/des separately, there should be documentation or naming to make it just as easy to see.
- deleted 7y ago
  [deleted]
- closeparen 7y ago
  I agree, but in this case I moved the interface in question to Thrift because it provides both documentation and compile-time safety for substantially less code than doing it manually with JSON. (There are also various ways to do some or all of it with JSON, such as JSON Schema and Swagger).

deleted 7y ago

[deleted]

pjc50 7y ago

GRPC is great as an API generator, but there's a lot of machinery under the hood in its network layer. It's not small - there was an embedded project where we were considering using it to replace our roll-your-own, then we discovered that the compiled library for GRPC was larger than the entire application. C structs are so brittle they should not be used for any kind of serialisation. You're just setting yourself up for upgrade nightmares. I wish BSON or the ASN.1 toolset were easier to use for this use case.

fanf2 7y ago
I hope the CBOR tooling is better https://cbor.io/tools.html https://cbor.io/tools.html
CoolGuySteve 7y ago
For C structs, start with a uint32_t 4 char code identifier and a size field. Then you can extend them as long as you only ever append, which is a constraint on protobufs anyways. This is how most media containers have worked since the Amiga days. Meanwhile, their encoding/decoding is infinitely faster than most other serialization formats, allowing you to serialize data back and forth to disk or as IPC faster than anything else.
- pjc50 7y ago
  .. as long as you never change endianness, structure packing, word alignment requirements, and always use types with explicit sizes. And you have to keep the data in the structure itself, so as soon as you hit a pointer you need to write explicit serialisation.
  - CoolGuySteve 7y ago
    > .. as long as you never change endianness, structure packing, word alignment requirements, and always use types with explicit sizes. #pragma pack(1) and byteswapping solve all this and neither are more work than specifying some intermediate format and calling some external tool in your build system. In C++ you can make types that swap automatically if required. > And you have to keep the data in the structure itself, so as soon as you hit a pointer you need to write explicit serialisation. I have yet to see a serialization format that does this well and quickly. Just make a new message type for it that follows the first.
  - makapuf 7y ago
    Or juste use riff which useful and provide just enough to implement wav, bmp and other file formats (so you'll find heaps of examples(.
  - Matthias247 7y ago
    Indeed. Using plain structs one can't just add a single field with a reasonable default and still be backward compatible - which is a very very common case in API design. You could add completely new revision of the API. But having to maintain it - typically additionally to the old revision since both sides might not updateable in atomic fashion - is a lot of extra work. Serialization which allows to add and remove fields in a backward compatible fashion is a mandatory requirement for any bigger sized project for me. If you need super high performance maybe use Flatbuffers or Cap'N'Proto instead of Protobuf/JSON/CBOR - but I really wouldn't recommend to go for plain structs.
closeparen 7y ago
>but there's a lot of machinery under the hood in its network layer GRPC is designed to enable peer-to-peer communication in a microservices mesh, so it has hooks for a number of things like service discovery, authentication, health checking, retry policy, connection pooling, observability, etc. If you're just looking for a serialization format you might prefer Protobuf itself.

deleted 7y ago

[deleted]

dkarl 7y ago

I vividly remember the outage I caused by changing a struct field name, only to find out it was implicitly a critical part of the API "contract" when serialized to JSON. Lack of unit tests for serialization can be deadly for projects where the serialization format is defined implicitly via the application model. I've seen enough projects bitten by this to be convinced it's worth writing ser/des unit tests for every API object. For some reason it's a form of testing that even gung-ho unit testers tend to leave out. Arguably it should be covered by integration tests, but integration tests usually only test round-trip serialization, which rarely breaks when you're testing a codebase against itself. But I think the issue you faced was a documentation issue as well. It should have been evident which objects defined the API. In some projects the objects have to be heavily annotated to make ser/des work at all, which makes it obvious, but when it's possible to separate concerns and define ser/des separately, there should be documentation or naming to make it just as easy to see.

deleted 7y ago
[deleted]
closeparen 7y ago
I agree, but in this case I moved the interface in question to Thrift because it provides both documentation and compile-time safety for substantially less code than doing it manually with JSON. (There are also various ways to do some or all of it with JSON, such as JSON Schema and Swagger).