5 ms·

Give me something that isn't based on protobufs at wire / request level. CBOR with CDDL for a fully standards based approach that can work at any size of the st

Give me something that isn't based on protobufs at wire / request level. CBOR with CDDL for a fully standards based approach that can work at any size of the stack

pranay01 3y ago
What's the issue with protobufs?
- reindeerer 3y ago
  The first issue is that protobufs arent a standard. That inherently limits anything built on top of them to not be a standard either, and that limits their applicability Also depending on the environment you run in, can code size bloat vs alternatives can matter
  - tonyarkles 3y ago
    > Aren’t a standard You mean like an IETF standard? That is true, although the specification is quite simple to implement. It is certainly a de-facto standard, even if it hasn’t been standardized by the IETF or IEEE or ANSI or ECMA. > inherently limits anything built on top of them to not be a standard either I’m not sure that strictly follows. https://datatracker.ietf.org/doc/html/rfc9232 https://datatracker.ietf.org/doc/html/rfc9232 for example directly references the protobuf spec at https://protobuf.dev/ https://protobuf.dev/ and includes protobufs as a valid encoding. > depending on the environment I’ve had several projects that ran on wimpy Cortex M0 processors and printf() has generally taken more code space in flash than NanoPB. This is generally with the same device doing both encoding and decoding. If you’re only encoding, the amount of code required to encode a given structure into a PB is very close to trivial. If I recall it can also be done in a streaming fashion so you don’t even need a RAM buffer necessarily to handle the encoded output. Do I love protobufs? Not really. There’s often some issue with protoc when running it in a new environment. The APIs sometimes bother me, especially the callback structure in NanoPB. But it’s been a workhorse for probably 15 years now and as a straightforward TLV encoding it works pretty darned well.
    - withinboredom 3y ago
      Sounds like Stockholm Syndrome. I've work mostly with JSON/CSV/Thrift in the last 10 years, and xml/soap before that, and just recently started interacting with protobuf, so I'd disagree that it is a "de-facto standard." My largest complaint: observability. With almost literally any other protocol, if you can mitm on the wire, your human brain can parse it. You can just take a glance at it and see any issues. With grpc/pbuf ... nope. not happening. Also, I really don't like how it tries to shim data into bitmasks. Going back to debugging two systems talking to each other, I'm not a computer. Needing special tooling just to figure out what two systems are saying to each other to shave a quarter of a packet is barely worth it, if at all, IMHO.
      - FridgeSeal 3y ago
        > You can just take a glance at it and see any issues. With grpc/pbuf ... nope. not happening. Sure, but on the other hand, the number of times I’ve needed to do this, compared to JSON/string/untyped/etc systems is precisely zero. There’s just a whole lot of failure that are just non-issues with typed setups like protobufs. Protobuf still has plenty of flaws and annoying-google-isms, but not being human readable isn’t one of them IMO.
        withinboredom 3y ago
        Nothing is preventing a system from sending you a un-deserializable message disguised as a protobuf, just like with any other encoding. In these cases, you need to diagnose the issue, no? Having something human-readable is simply straightforward. If you haven't needed to do this, perhaps you aren't working on big enough systems? I've primarily needed to do this when dealing with hundreds of millions of disparate clients, not so much on smaller systems.
        FridgeSeal 3y ago
        > Nothing is preventing a system from sending you a un-deserializable message disguised as a protobuf, I guess it depends on where you come down on Postel’s law. If you’re an adherent, and are prepared to be flexible in what you accept, then yeah, you will have extra work on your hands. Personally, I’m not a fan of Postels law, and I’m camp “send correct data, that deserializes and upholds invariants, or get your request rejected”. I’ve played enough games with systems that weren’t strict enough about their boundaries and it’s just not worth the pain.
- jahewson 3y ago
  I used protobufs for a short while and came to the realization that they’re just Go’s opinionated idioms forced on other languages via awkward SDKs. Particularly did not like having to use codegen, rely on Google’s annotations for basic functionality or deal with field masks that are a sort of poor man’s GraphQL. I get it, Google made trade offs that work for them, and I agree with their position - but for someone at a smaller company working in a non-Go/Java/C programming language it was just a ton of friction for no benefit.
  - secondcoming 3y ago
    protobufs have been around a lot longer than Go
  - neonsunset 3y ago
    Annotations are somewhat cursed agree. The code generation part does not have to be painful though. In fact, the tooling in Go isn't even an example of the easiest way to do it and requires you to do more steps than for example .NET where getting server boilerplate, fully working client or just generated POCO contracts from .proto boils down to dotnet add package Grpc.Tools <Protobuf Include="MyApiContracts.proto" /> (in .csproj)
  - hinkley 3y ago
    Are you old enough to remember SOAP? Is this drawing any “history repeats” sentiments for you?
- skywhopper 3y ago
  So many issues, primarily the non-idiomatic (even in golang!) code generation and the requirement for special tooling to do any troubleshooting, and after all that you still don't end up with anything particularly interoperable. At best it works tolerably in a monorepo with tightly controlled deployment scenarios and great tooling. But if you don't have a Google-like operations environment, it's a lot of extra overhead for a mostly meaningless benefit.
- Spivak 3y ago
  I think the issue is that the ergonomics of protobuf kinda suck and the ergonomics of gRPC really suck and having to interface with a gRPC API is likely everyone's introduction to protobufs. Protobufs are a really great idea that's hampered by heinously subpar tooling for everything but Go.
alright2565 3y ago
Here's the example payloads for OTLP over JSON and example of how to ingest them: https://github.com/open-telemetry/opentelemetry-proto/tree/main/examples https://github.com/open-telemetry/opentelemetry-proto/tree/m...
- reindeerer 3y ago
  Yes, but the schema def is still in protobuf

Give me something that isn't based on protobufs at wire / request level. CBOR with CDDL for a fully standards based approach that can work at any size of the stack

pranay01 3y ago

What's the issue with protobufs?

reindeerer 3y ago
The first issue is that protobufs arent a standard. That inherently limits anything built on top of them to not be a standard either, and that limits their applicability Also depending on the environment you run in, can code size bloat vs alternatives can matter
- tonyarkles 3y ago
  > Aren’t a standard You mean like an IETF standard? That is true, although the specification is quite simple to implement. It is certainly a de-facto standard, even if it hasn’t been standardized by the IETF or IEEE or ANSI or ECMA. > inherently limits anything built on top of them to not be a standard either I’m not sure that strictly follows. https://datatracker.ietf.org/doc/html/rfc9232 https://datatracker.ietf.org/doc/html/rfc9232 for example directly references the protobuf spec at https://protobuf.dev/ https://protobuf.dev/ and includes protobufs as a valid encoding. > depending on the environment I’ve had several projects that ran on wimpy Cortex M0 processors and printf() has generally taken more code space in flash than NanoPB. This is generally with the same device doing both encoding and decoding. If you’re only encoding, the amount of code required to encode a given structure into a PB is very close to trivial. If I recall it can also be done in a streaming fashion so you don’t even need a RAM buffer necessarily to handle the encoded output. Do I love protobufs? Not really. There’s often some issue with protoc when running it in a new environment. The APIs sometimes bother me, especially the callback structure in NanoPB. But it’s been a workhorse for probably 15 years now and as a straightforward TLV encoding it works pretty darned well.
  - withinboredom 3y ago
    Sounds like Stockholm Syndrome. I've work mostly with JSON/CSV/Thrift in the last 10 years, and xml/soap before that, and just recently started interacting with protobuf, so I'd disagree that it is a "de-facto standard." My largest complaint: observability. With almost literally any other protocol, if you can mitm on the wire, your human brain can parse it. You can just take a glance at it and see any issues. With grpc/pbuf ... nope. not happening. Also, I really don't like how it tries to shim data into bitmasks. Going back to debugging two systems talking to each other, I'm not a computer. Needing special tooling just to figure out what two systems are saying to each other to shave a quarter of a packet is barely worth it, if at all, IMHO.
    - FridgeSeal 3y ago
      > You can just take a glance at it and see any issues. With grpc/pbuf ... nope. not happening. Sure, but on the other hand, the number of times I’ve needed to do this, compared to JSON/string/untyped/etc systems is precisely zero. There’s just a whole lot of failure that are just non-issues with typed setups like protobufs. Protobuf still has plenty of flaws and annoying-google-isms, but not being human readable isn’t one of them IMO.
      - withinboredom 3y ago
        Nothing is preventing a system from sending you a un-deserializable message disguised as a protobuf, just like with any other encoding. In these cases, you need to diagnose the issue, no? Having something human-readable is simply straightforward. If you haven't needed to do this, perhaps you aren't working on big enough systems? I've primarily needed to do this when dealing with hundreds of millions of disparate clients, not so much on smaller systems.
        FridgeSeal 3y ago
        > Nothing is preventing a system from sending you a un-deserializable message disguised as a protobuf, I guess it depends on where you come down on Postel’s law. If you’re an adherent, and are prepared to be flexible in what you accept, then yeah, you will have extra work on your hands. Personally, I’m not a fan of Postels law, and I’m camp “send correct data, that deserializes and upholds invariants, or get your request rejected”. I’ve played enough games with systems that weren’t strict enough about their boundaries and it’s just not worth the pain.
jahewson 3y ago
I used protobufs for a short while and came to the realization that they’re just Go’s opinionated idioms forced on other languages via awkward SDKs. Particularly did not like having to use codegen, rely on Google’s annotations for basic functionality or deal with field masks that are a sort of poor man’s GraphQL. I get it, Google made trade offs that work for them, and I agree with their position - but for someone at a smaller company working in a non-Go/Java/C programming language it was just a ton of friction for no benefit.
- secondcoming 3y ago
  protobufs have been around a lot longer than Go
- neonsunset 3y ago
  Annotations are somewhat cursed agree. The code generation part does not have to be painful though. In fact, the tooling in Go isn't even an example of the easiest way to do it and requires you to do more steps than for example .NET where getting server boilerplate, fully working client or just generated POCO contracts from .proto boils down to dotnet add package Grpc.Tools <Protobuf Include="MyApiContracts.proto" /> (in .csproj)
- hinkley 3y ago
  Are you old enough to remember SOAP? Is this drawing any “history repeats” sentiments for you?
skywhopper 3y ago
So many issues, primarily the non-idiomatic (even in golang!) code generation and the requirement for special tooling to do any troubleshooting, and after all that you still don't end up with anything particularly interoperable. At best it works tolerably in a monorepo with tightly controlled deployment scenarios and great tooling. But if you don't have a Google-like operations environment, it's a lot of extra overhead for a mostly meaningless benefit.
Spivak 3y ago
I think the issue is that the ergonomics of protobuf kinda suck and the ergonomics of gRPC really suck and having to interface with a gRPC API is likely everyone's introduction to protobufs. Protobufs are a really great idea that's hampered by heinously subpar tooling for everything but Go.

alright2565 3y ago

Here's the example payloads for OTLP over JSON and example of how to ingest them: https://github.com/open-telemetry/opentelemetry-proto/tree/main/examples https://github.com/open-telemetry/opentelemetry-proto/tree/m...

reindeerer 3y ago
Yes, but the schema def is still in protobuf