3 ms·
Am I reading this right that it's a binary format for real-time streaming data, similar to Avro, but can include arbitrarily deep nested structures unlike Avro?
by mikece 3y ago
Am I reading this right that it's a binary format for real-time streaming data, similar to Avro, but can include arbitrarily deep nested structures unlike Avro?
- jauntywundrkind 3y agoWhy do you single out avro & not any of the hundreds of other ser/de systems? Is that what you know best? Is there something specific about avro that makes it feel particularly similar? https://github.com/maximveksler/awesome-serialization https://github.com/maximveksler/awesome-serialization
- mikece 3y agoI work in an a shop where Kafka and Avro are everywhere. If I worked someone else I might make reference to something else if it was front-of-mind all the time.
- PaulHoule 3y agoPeople keep inventing new ones because the old ones suck or they think the old ones suck. Look at all the discontents around JSON (no comments!), people react violently when people try to apply a little extra like JSON-LD. Then there are all the things like YAML, TOML and such that try to be a little better but are widely thought to be a little worse. (And that's just the human readable data formats) Then there is always https://en.wikipedia.org/wiki/ASN.1 https://en.wikipedia.org/wiki/ASN.1 which is forgotten but not gone.
- fieldcny 3y agoOh man ASN.1, makes me shiver, oh the memories… I hadn’t thought about it in like a decade but yeah it’s still silently in the background…
- abeppu 3y agoIn fairness, in the list you link to, - Avro and Ion are the only two that are labeled Textual/Binary - They are in the same Big Data grouping - They both are schema-embedded, and support some rich nested datastructures, though they deviate on many of the specifics So I think it's reasonable to pick out Avro as an especially similar point of comparison.
- leef 3y agoIon has a binary format but is not specifically about real-time streaming. It is a JSON replacement. Ion originated 10+ years ago from the Amazon catalog team - the team that kept data about the hundreds of millions of items available on Amazon. Nearly every team in the company called the catalog to get information about items all the time - scanning the entire catalog, parts of the catalog, millions of individual item lookups every second, etc. They did the math and some very large percentage of network traffic in Amazon Retails data centers was catalog data. If that data, currently in XML or JSON format, was sent in a more compact format it would save some ridiculous millions of dollars every year. So Ion was born and eventually open sourced.