5 ms·

Yeah... you're only going to save money if you're an absolutely enormous scale. I'd also really question that point at all when you can compress JSON as it's be

Yeah... you're only going to save money if you're an absolutely enormous scale. I'd also really question that point at all when you can compress JSON as it's being transmitted (GZIP, baby). Next, I have yet to see somewhere... anywhere, not at the scale of something like Uber, or a high-performance game server, really need to worry about serialization and deserialization of JSON. Almost all serialization libraries are written in C for MAXIMUM PERFORMANCE regardless of the language you're using, or some sort of hand-tuned-magical bytecode fuckery which has the same effect. I honestly can't imagine this being useful except for obscenely high throughput service to service communication or as an internal serialization protocol (for something like FoundationDB where you're storing the Avro encoded instance to disk). Truly, JSON and REST are good enough for 99.999% of services and the overhead of using something non-standard will likely never pay off due to the increased developer overhead and maintenance. This just smells like tech debt and an architecture that can't be debugged.

tiew9Vii 7y ago
The compression soon mounts up in savings. It’s quicker to send over the network, disks cost money and on the cloud even low terrabyte sized data starts to mount up in cost. JSON is unstructured. Every one who parses the data needs to write a parser. As it’s unstructured how do you manage schema changes? Using a binary format like avro you get strong typing (basic validation), schema change modes, no need to write clients. You can publish your avro schemas to a registry and you then get a catalog of data available anyone can browse to see what data is available and how it looks. > This just smells like tech debt I’d say the opposite. Using JSON is tech debt. Everyone writing parsers/serialisers, not knowing what fields are available, what values the fields can accept, how to handle new fields / deprecating old fields. Debugging is simple, AVRO can be inspected unlike over formats with its Generic record format that can also be represented and dumped as JSON. people should use binary strongly typed formats more. It makes things so much easier for the developer,data evolution and data processing. You also get benefits that your application is no longer running in debug mode.
jacques_chester 7y ago
> you can compress JSON as it's being transmitted (GZIP, baby). Gzip doesn't buy much advantage on small records, because there's not much repetition in the source document. And there's no law against compressing binaries, last I checked. > Next, I have yet to see somewhere... anywhere, not at the scale of something like Uber, or a high-performance game server, really need to worry about serialization and deserialization of JSON. I built a discrete-event simulator[0] which presents its findings with a simple web interface from the localhost. It's about the simplest possible setup. The simulator can process millions of simulated events in a few seconds, but it takes much longer to stream the JSON to the browser for rendering. Even with "GZIP, baby". As an experiment I converted it to stream records in Avro format. There are several javascript libraries and Golang libraries. The conversion took me an afternoon. The amount of data streamed went down by approximately half. I didn't gain on rendering time because it currently completes the entire stream before conversion. If that wasn't the case I expect the Avro version would noticeably faster. So if "it took one random person an afternoon to hook up some libraries" is tech debt, then I guess I better call my bank, because I do that pretty much every day. [0] https://github.com/pivotal/skenario https://github.com/pivotal/skenario
nly 7y ago
I'm toying with Avro right now for a project and our common JSON payload went from 1400 to 70 bytes, which makes it much more suitable for use over UDP Also using JSON in a statically-typed language is tedious and error-prone.
- latchkey 7y ago
  How much of your data is repeated as part of a cyclical graph? You may want to try jsog + compression. [1] https://github.com/jsog/jsog https://github.com/jsog/jsog

tiew9Vii 7y ago

The compression soon mounts up in savings. It’s quicker to send over the network, disks cost money and on the cloud even low terrabyte sized data starts to mount up in cost. JSON is unstructured. Every one who parses the data needs to write a parser. As it’s unstructured how do you manage schema changes? Using a binary format like avro you get strong typing (basic validation), schema change modes, no need to write clients. You can publish your avro schemas to a registry and you then get a catalog of data available anyone can browse to see what data is available and how it looks. > This just smells like tech debt I’d say the opposite. Using JSON is tech debt. Everyone writing parsers/serialisers, not knowing what fields are available, what values the fields can accept, how to handle new fields / deprecating old fields. Debugging is simple, AVRO can be inspected unlike over formats with its Generic record format that can also be represented and dumped as JSON. people should use binary strongly typed formats more. It makes things so much easier for the developer,data evolution and data processing. You also get benefits that your application is no longer running in debug mode.

jacques_chester 7y ago

> you can compress JSON as it's being transmitted (GZIP, baby). Gzip doesn't buy much advantage on small records, because there's not much repetition in the source document. And there's no law against compressing binaries, last I checked. > Next, I have yet to see somewhere... anywhere, not at the scale of something like Uber, or a high-performance game server, really need to worry about serialization and deserialization of JSON. I built a discrete-event simulator[0] which presents its findings with a simple web interface from the localhost. It's about the simplest possible setup. The simulator can process millions of simulated events in a few seconds, but it takes much longer to stream the JSON to the browser for rendering. Even with "GZIP, baby". As an experiment I converted it to stream records in Avro format. There are several javascript libraries and Golang libraries. The conversion took me an afternoon. The amount of data streamed went down by approximately half. I didn't gain on rendering time because it currently completes the entire stream before conversion. If that wasn't the case I expect the Avro version would noticeably faster. So if "it took one random person an afternoon to hook up some libraries" is tech debt, then I guess I better call my bank, because I do that pretty much every day. [0] https://github.com/pivotal/skenario https://github.com/pivotal/skenario

nly 7y ago

I'm toying with Avro right now for a project and our common JSON payload went from 1400 to 70 bytes, which makes it much more suitable for use over UDP Also using JSON in a statically-typed language is tedious and error-prone.

latchkey 7y ago
How much of your data is repeated as part of a cyclical graph? You may want to try jsog + compression. [1] https://github.com/jsog/jsog https://github.com/jsog/jsog