3 ms·

I know it’s utopia, but I still can’t help but ask, why can’t we have protobuffs or some other binary format with readers integrated into normal systems the way

I know it’s utopia, but I still can’t help but ask, why can’t we have protobuffs or some other binary format with readers integrated into normal systems the way everything has a Json reader. I mean, stuff like “double click on .protobuff file and it opens in notepad and looks like json or whatever. When you click save it gets serialized back to protobuff.

MayeulC 5y ago
IIRC (I could be wrong on this one) protobuf isn't really a specification, it's a single implementation, and is quite hard to reimplement. Here's a short comparison of serializing formats: https://drewdevault.com/2020/06/21/BARE-message-encoding.html https://drewdevault.com/2020/06/21/BARE-message-encoding.htm...
- morelisp 5y ago
  Regarding the wire format specifically, it's not fully formalized but there's dozen implementations, not all of them from Google. You could hack together a basic serde for a particular language in an afternoon, in some respects much more easily than you could JSON. Most of the engineering work is in the schema compilers.
  - heavenlyblue 5y ago
    But in order to semantically read protocol buffer files one needs to compile the schema files…
    - morelisp 5y ago
      The degree you "need" to support the full schema format depends entirely on the language you're using. In Go you only need the annotated structures, or in Java you only need a mapping between field number and name, and the languages' own reflection capabilities can handle the rest. Yes, strings, bytes, and substructures all appear in the same in the wire format. Just like strings, bytes, and dates all appear the same in JSON. If you're trying to write a generic protobuf viewer like wodenokoto suggests, you can make reasonable assumptions about the contents 99% of the time based on the data, and show the user multiple options if you're not sure. There are already lots of tools that do this, but none very well integrated into mainstream development workflows.
matja 5y ago
Tricky part about implementing that is that the protobuf wire-format alone doesn't contain enough information to unambiguously represent the real data types, it also needs the schema (.proto file) to do that. For example, in the wire-format, a string and a sub-message are encoded as the same type (a blob - varint + sequence of bytes), but using the schema they are clearly interpreted differently. Sure, it is possible to make a self-describing protobuf message which includes the schema in protobuf representation, but that is a special-case.
- uDontKnowMe 5y ago
  What about Avro then?
- Too 5y ago
  Json schema has a similar challenge, there is no header, in the json files themselves, indicating which schema is used. The slightly dirty solution is https://www.schemastore.org/json/ https://www.schemastore.org/json/, where the IDE looks up schema from a global registry, using a fileMatch pattern.

MayeulC 5y ago

IIRC (I could be wrong on this one) protobuf isn't really a specification, it's a single implementation, and is quite hard to reimplement. Here's a short comparison of serializing formats: https://drewdevault.com/2020/06/21/BARE-message-encoding.html https://drewdevault.com/2020/06/21/BARE-message-encoding.htm...

morelisp 5y ago
Regarding the wire format specifically, it's not fully formalized but there's dozen implementations, not all of them from Google. You could hack together a basic serde for a particular language in an afternoon, in some respects much more easily than you could JSON. Most of the engineering work is in the schema compilers.
- heavenlyblue 5y ago
  But in order to semantically read protocol buffer files one needs to compile the schema files…
  - morelisp 5y ago
    The degree you "need" to support the full schema format depends entirely on the language you're using. In Go you only need the annotated structures, or in Java you only need a mapping between field number and name, and the languages' own reflection capabilities can handle the rest. Yes, strings, bytes, and substructures all appear in the same in the wire format. Just like strings, bytes, and dates all appear the same in JSON. If you're trying to write a generic protobuf viewer like wodenokoto suggests, you can make reasonable assumptions about the contents 99% of the time based on the data, and show the user multiple options if you're not sure. There are already lots of tools that do this, but none very well integrated into mainstream development workflows.

matja 5y ago

Tricky part about implementing that is that the protobuf wire-format alone doesn't contain enough information to unambiguously represent the real data types, it also needs the schema (.proto file) to do that. For example, in the wire-format, a string and a sub-message are encoded as the same type (a blob - varint + sequence of bytes), but using the schema they are clearly interpreted differently. Sure, it is possible to make a self-describing protobuf message which includes the schema in protobuf representation, but that is a special-case.

uDontKnowMe 5y ago
What about Avro then?
Too 5y ago
Json schema has a similar challenge, there is no header, in the json files themselves, indicating which schema is used. The slightly dirty solution is https://www.schemastore.org/json/ https://www.schemastore.org/json/, where the IDE looks up schema from a global registry, using a fileMatch pattern.