4 ms·

Folks, if you send raw PB over the wire in a protocol without framing, it _needs_ to be length prefixed somehow. This type of "bug" isn't an unreasonable thing

Folks, if you send raw PB over the wire in a protocol without framing, it _needs_ to be length prefixed somehow. This type of "bug" isn't an unreasonable thing to do, nor uncommon even. If I had a gripe, it's that I hate varints. Lots of wasted cycles to save not that many bytes in the grand scheme of things, especially when you consider other forms of compression layered on top, MTU, etc.

catiopatio 3y ago
It’s absolutely an unreasonable thing to do. You don’t conflate framing with payload by emitting invalid non-standard framed data from a “protobuf” encoder. They’re separate concerns and need to remain that way.
- lowbloodsugar 3y ago
  Except that it’s part of the official implementation from google. It’s how you write multiple such messages to a stream.
  - catiopatio 3y ago
    There’s a non-default `writeDelimitedTo()` that emits a length-prefixed message. The result is not protobuf and doesn’t claim to be. Looking at my employer’s protobuf runtime for our major programming language, we don’t even support it.
    - lowbloodsugar 3y ago
      reading and writing length delimited messages is part of the official implementations. [1] I don't understand the claim of "non-default". There are multiple ways to write protobuf data, and this is one of them. It's actually the only one I've ever used, because why would you not? If your implementation doesn't support something in the reference implementation, that seems like your problem, not anyone else's. [1] https://protobuf.dev/reference/csharp/api-docs/class/google/protobuf/message-parser https://protobuf.dev/reference/csharp/api-docs/class/google/...
      - catiopatio 3y ago
        > It's actually the only one I've ever used, because why would you not? Because it’s not part of the protobuf specification, not part of a valid protobuf message, and framing is a transport/file format concern and should not be performed by default. If someone is expecting to receive a protobuf-encoded payload, it must not include a framing header. > If your implementation doesn't support something in the reference implementation, that seems like your problem, not anyone else's. Someone else’s failure to follow the spec is not our problem.
        whywhywouldyou 3y ago
        OP just told you it was in the spec, and then linked you directly to it. You're just repeating your previous post without adding anything new. Here's a more direct link: https://protobuf.dev/reference/csharp/api-docs/class/google/protobuf/message-parser#class_google_1_1_protobuf_1_1_message_parser_1a47e94dd6d35594250a54858f99bc5b28 https://protobuf.dev/reference/csharp/api-docs/class/google/...
- ninepoints 3y ago
  If you read the post, the conclusion is: > you skip the "helper" function that's breaking things. Yea ok, I'm just going to assume this helper function added framing unless told otherwise. Where in this post did you even read that framing and payload data were conflated (not to mention that there are better protocols that include framing metadata).

catiopatio 3y ago

It’s absolutely an unreasonable thing to do. You don’t conflate framing with payload by emitting invalid non-standard framed data from a “protobuf” encoder. They’re separate concerns and need to remain that way.

lowbloodsugar 3y ago
Except that it’s part of the official implementation from google. It’s how you write multiple such messages to a stream.
- catiopatio 3y ago
  There’s a non-default `writeDelimitedTo()` that emits a length-prefixed message. The result is not protobuf and doesn’t claim to be. Looking at my employer’s protobuf runtime for our major programming language, we don’t even support it.
  - lowbloodsugar 3y ago
    reading and writing length delimited messages is part of the official implementations. [1] I don't understand the claim of "non-default". There are multiple ways to write protobuf data, and this is one of them. It's actually the only one I've ever used, because why would you not? If your implementation doesn't support something in the reference implementation, that seems like your problem, not anyone else's. [1] https://protobuf.dev/reference/csharp/api-docs/class/google/protobuf/message-parser https://protobuf.dev/reference/csharp/api-docs/class/google/...
    - catiopatio 3y ago
      > It's actually the only one I've ever used, because why would you not? Because it’s not part of the protobuf specification, not part of a valid protobuf message, and framing is a transport/file format concern and should not be performed by default. If someone is expecting to receive a protobuf-encoded payload, it must not include a framing header. > If your implementation doesn't support something in the reference implementation, that seems like your problem, not anyone else's. Someone else’s failure to follow the spec is not our problem.
      - whywhywouldyou 3y ago
        OP just told you it was in the spec, and then linked you directly to it. You're just repeating your previous post without adding anything new. Here's a more direct link: https://protobuf.dev/reference/csharp/api-docs/class/google/protobuf/message-parser#class_google_1_1_protobuf_1_1_message_parser_1a47e94dd6d35594250a54858f99bc5b28 https://protobuf.dev/reference/csharp/api-docs/class/google/...
ninepoints 3y ago
If you read the post, the conclusion is: > you skip the "helper" function that's breaking things. Yea ok, I'm just going to assume this helper function added framing unless told otherwise. Where in this post did you even read that framing and payload data were conflated (not to mention that there are better protocols that include framing metadata).