4 ms·
> you can compress JSON as it's being transmitted (GZIP, baby). Gzip doesn't buy much advantage on small records, because there's not much repetition in the so
by jacques_chester 7y ago
> you can compress JSON as it's being transmitted (GZIP, baby).
Gzip doesn't buy much advantage on small records, because there's not much repetition in the source document. And there's no law against compressing binaries, last I checked.
> Next, I have yet to see somewhere... anywhere, not at the scale of something like Uber, or a high-performance game server, really need to worry about serialization and deserialization of JSON.
I built a discrete-event simulator[0] which presents its findings with a simple web interface from the localhost. It's about the simplest possible setup. The simulator can process millions of simulated events in a few seconds, but it takes much longer to stream the JSON to the browser for rendering. Even with "GZIP, baby".
As an experiment I converted it to stream records in Avro format. There are several javascript libraries and Golang libraries. The conversion took me an afternoon. The amount of data streamed went down by approximately half. I didn't gain on rendering time because it currently completes the entire stream before conversion. If that wasn't the case I expect the Avro version would noticeably faster.
So if "it took one random person an afternoon to hook up some libraries" is tech debt, then I guess I better call my bank, because I do that pretty much every day.
[0] https://github.com/pivotal/skenario https://github.com/pivotal/skenario