4 ms·
I largely agree, but most modern binary formats are more rigid with schemas and types. If you want the flexibility of JSON, I'm not sure you'll end up with som
by EwanToo 5y ago
I largely agree, but most modern binary formats are more rigid with schemas and types.
If you want the flexibility of JSON, I'm not sure you'll end up with something massively different from gzipped JSON
- lifthrasiir 5y agoJSON doesn't allow any custom type, so it is not "flexible" per se. Therefore you only need a format that supports the JSON data model and pretty much nothing else; CBOR [1] for example almost surely fits the bill. [1] https://cbor.io/ https://cbor.io/
- Waterluvian 5y agoThat’s an interesting thought. I am now very fascinated by what kind of data would be gigs in size but would need the flexibility of JSON.
- maccard 5y agoDatasets are not always provided by you; if your data source outputs json it doesn't matter why. Also just because your volume of data is measured in gigabytes, that doesn't mean it's a singular stream. As an example you could be handling tens of thousands of small requests.
- the8472 5y agoConverting a text format to a compact in-memory data structure takes extra CPU cycles. (de)compression takes extra cycles. The point of using a binary format is to achieve the same result while avoiding that overhead. For some data formats compression also has the downside that it prevents seeking.
- kevin_thibedeau 5y agoSerialized binary encodings don't require a schema. They can have all the expressability of JSON and more.