4 ms·

I don't think this is as big a problem as the article suggests. There are a substantial number of tradeoffs in serialization protocols, and each application/eco

I don't think this is as big a problem as the article suggests. There are a substantial number of tradeoffs in serialization protocols, and each application/ecosystem can choose their protocol to get the best of these tradeoffs. As long as there are few enough that every popular protocol has a library in almost every language this isn't too bad. One example of a tradeoff that is hard to eliminate is that you can reduce size and increase performance substantially if you pre-specify a schema like Cap'n Proto (and others) do. The downside is then if you just get a message without knowing what it is about it's difficult to figure out. The only way out of this tradeoff that I can see is having a global schema registry and every message having 8 bytes dedicated to schema ID, and that has downsides of its own, especially for small messages. I do agree with the author though that we could do with more binary serialization protocols with tools to easily translate back and forth to a human-readable text format for debugging.

deleted 10y ago
[deleted]
galacticpony 10y ago
>> The downside is then if you just get a message without knowing what it is about it's difficult to figure out. You're saying that as if there was a clerk reading the message at the other end. If a computer program gets a message it doesn't "understand", it's not going to "figure it out" either way. Worse, if you're a programmer that doesn't know the actual schema and you're trying to figure out what the schema might be just by looking at the data, you'll probably run into trouble.
- trishume 10y ago
  I was talking purely about humans finding messages they don't have the schema for during the course of debugging. Computers do equally badly with JSON data and Cap'n Proto data they don't have the schema for. A fake example scenario of what I'm talking about: Someone notices a server is producing way too many errors, they check the logs and see a bunch of "invalid request: ...." messages. The messages are JSON like {"api_version":2,"cpu_usage":0.57}. They figure out that the server monitoring system is forwarding stats to the wrong IP address. Even though they don't know the full schema of the messages, they get the gist and it helped debug. With Cap'n Proto that would have been a few nonsensical bytes. I don't think this is a very big downside, so in general I like protocols like Cap'n Proto. Perhaps a much bigger downside is that in most languages integrating code generation into the build pipeline is a pain in the ass, so it is way easier to use dynamic serialization protocols that don't require compiling a schema.
- dpark 10y ago
  > You're saying that as if there was a clerk reading the message at the other end. If a computer program gets a message it doesn't "understand", it's not going to "figure it out" either way. Self-describing formats do make backwards compatibility a bit easier. With Cap'n Proto, adding a new field in the middle will break consumers.

deleted 10y ago

[deleted]

galacticpony 10y ago

>> The downside is then if you just get a message without knowing what it is about it's difficult to figure out. You're saying that as if there was a clerk reading the message at the other end. If a computer program gets a message it doesn't "understand", it's not going to "figure it out" either way. Worse, if you're a programmer that doesn't know the actual schema and you're trying to figure out what the schema might be just by looking at the data, you'll probably run into trouble.

trishume 10y ago
I was talking purely about humans finding messages they don't have the schema for during the course of debugging. Computers do equally badly with JSON data and Cap'n Proto data they don't have the schema for. A fake example scenario of what I'm talking about: Someone notices a server is producing way too many errors, they check the logs and see a bunch of "invalid request: ...." messages. The messages are JSON like {"api_version":2,"cpu_usage":0.57}. They figure out that the server monitoring system is forwarding stats to the wrong IP address. Even though they don't know the full schema of the messages, they get the gist and it helped debug. With Cap'n Proto that would have been a few nonsensical bytes. I don't think this is a very big downside, so in general I like protocols like Cap'n Proto. Perhaps a much bigger downside is that in most languages integrating code generation into the build pipeline is a pain in the ass, so it is way easier to use dynamic serialization protocols that don't require compiling a schema.
dpark 10y ago
> You're saying that as if there was a clerk reading the message at the other end. If a computer program gets a message it doesn't "understand", it's not going to "figure it out" either way. Self-describing formats do make backwards compatibility a bit easier. With Cap'n Proto, adding a new field in the middle will break consumers.