6 ms·

This reminds me a lot of Avro: https://avro.apache.org/docs/current/ https://avro.apache.org/docs/current/ They both have self-describing schemas, support for

by wyc 10y ago

This reminds me a lot of Avro: https://avro.apache.org/docs/current/ https://avro.apache.org/docs/current/ They both have self-describing schemas, support for binary values, JSON-interoperability, basic type systems (Ion seems to support a few more field types), field annotations, support for schema evolution, code generation not necessary, etc. I think Avro has the additional advantages of being production-tested in many different companies, a fully-JSON schema, support for many languages, RPC baked into the spec, and solid performance numbers found across the web. I can't really see why I'd prefer Ion. It looks like an excellent piece of software with plenty of tests, no doubt, but I think I could do without "clobs", "sexprs", and "symbols" at this level of representation, and it might actually be better if I do. Am I missing something?

umanwizard 10y ago
Amazon invented Ion because yaml, Avro, etc. didn't exist at the time. Ion is actually pretty old. The timing of open-sourcing it mystifies me a bit. Maybe Amazon is trying to become more open-source friendly, like Microsoft did? Perhaps more likely: they're planning on making some internal APIs that use ION heavily public?
- leef 10y ago
  That's not true. Avro at least existed at the time. However they wanted something self-describing to replace JSON/XML usage. Avro is better suited as a data storage format rather than a transit oriented format. Of course, both Ion and Avro can be used for either, but Avro will give you better compression on disk, but Ion is less cumbersome since it doesn't require a schema
- kbenson 10y ago
  Are we talking about a different YAML, or has Ion existed for ~15 years?
  - deleted 10y ago
    [deleted]
  - umanwizard 10y ago
    Woah, you're right. I thought YAML was much younger. I stand corrected. Ion is definitely from last decade, at least. So most of the speculation in my post stands.
    - jonhohle 10y ago
      The JSON RFC was published right around the time Ion was being created making Ion about 10 years old this year. It was not clear at the time that JSON would become so popular ;-) and at the same time Ion fixed a lot of its weak areas (numeric types, dates, struct/object syntax, etc.)
- deleted 10y ago
  [deleted]
cm3 10y ago
> but I think I could do without "clobs", "sexprs", and "symbols" at this level of representation I'm by no means a real Lisp programmer, but even I find S-Expressions more natural to write and process. And the simplicity of it allows for great editing tools too. This may be personal, but I always found JSON clunky.
jcrites 10y ago
What do you mean by they both have self-describing schemas? In order to read or write Avro data, an application needs to possess a schema for that data -- the specific schema that the data was written with, and (when writing) the same schema that a later reader expects to find. This means the data is not self-describing. Ion is designed to be self-describing, meaning that no schema is necessary to deserialize and interact with Ion structures. It's consequently possible to interact with Ion in a dynamic and reflective way, for example, in the same way that you can with JSON and XML. It's possible to write a pretty-printer for a binary Ion structure coming off the wire without having any idea of or schema for what's inside. Ion's advantage over those formats is that it's strongly typed (or richly typed, if you prefer). For example, Ion has types for timestamps, arbitrary-precision decimals like for currency, and can embed binary data directly (without base64 encoding), etc. I wouldn't try to say that one or the other is better across the board. Rather, they have tradeoffs and relative strengths in different circumstances. Ion is in part designed to tackle scenarios like where your data might live a really long time, and needs to be comprehensible decades from now (whether you kept track of the schema or not, or remember which one it was); and needs to be comprehensible in a large distributed environment where not every application might possess the latest schema or where coordinating a single compile-time schema is a challenge (maybe each app only cares about some part of the data), and so on. Ion is well-suited to long-lived, document-type data that's stored at rest and interacted with in a variety of potentially complex ways over time. Data data. In the case of a simple RPC relationship between a single client and service, where the data being exchanged is ephemeral and won't stick around, and it's easy to definitively coordinate a schema across both applications, a typical serialization framework is a fine choice.
- wyc 10y ago
  I think it depends on what level you're referring to. If you mean record-level, then I concede that it's not self-describing. However, looking at the suggested use cases, it seems that it's "self-describing" in that you'll always be able to decode data stored according to what the documentation recommends: "Avro data is always serialized with its schema. Files that store Avro data should always also include the schema for that data in the same file. Avro-based remote procedure call (RPC) systems must also guarantee that remote recipients of data have a copy of the schema used to write that data." https://avro.apache.org/docs/current/spec.html#Data+Serialization https://avro.apache.org/docs/current/spec.html#Data+Serializ...
tjonker 10y ago
A big problem with Avro, BSON, and many other "binary JSON" formats is that they're not isomorphic with JSON, they have a bunch of additional stuff added on. There's Avro documents that don't have direct JSON equivalents. Which means that when a human needs to read the data, you have to transform it into some third format. It's a core feature of Ion that the text and binary representations are isomorphic. You can take any Ion binary document and pretty-print it as an Ion text document that is exactly equivalent. You can edit that document and send it into your application, which will be guaranteed to be able to read it. Or you can take your hand-authored text data and transcode it into binary, and know that any Ion application can handle it without any extra effort.
- amyjess 10y ago
  > A big problem with Avro, BSON, and many other "binary JSON" formats is that they're not isomorphic with JSON, they have a bunch of additional stuff added on. There's Avro documents that don't have direct JSON equivalents. Which means that when a human needs to read the data, you have to transform it into some third format. Also, field order matters in Avro but not in JSON. That bit me pretty hard once... fortunately, I found out that Python's JSON library lets you read a JSON file into an OrderedDict instead of a plain dict, so I was able to get around it.

by wyc 10y ago

umanwizard 10y ago

Amazon invented Ion because yaml, Avro, etc. didn't exist at the time. Ion is actually pretty old. The timing of open-sourcing it mystifies me a bit. Maybe Amazon is trying to become more open-source friendly, like Microsoft did? Perhaps more likely: they're planning on making some internal APIs that use ION heavily public?

leef 10y ago
That's not true. Avro at least existed at the time. However they wanted something self-describing to replace JSON/XML usage. Avro is better suited as a data storage format rather than a transit oriented format. Of course, both Ion and Avro can be used for either, but Avro will give you better compression on disk, but Ion is less cumbersome since it doesn't require a schema
kbenson 10y ago
Are we talking about a different YAML, or has Ion existed for ~15 years?
- deleted 10y ago
  [deleted]
- umanwizard 10y ago
  Woah, you're right. I thought YAML was much younger. I stand corrected. Ion is definitely from last decade, at least. So most of the speculation in my post stands.
  - jonhohle 10y ago
    The JSON RFC was published right around the time Ion was being created making Ion about 10 years old this year. It was not clear at the time that JSON would become so popular ;-) and at the same time Ion fixed a lot of its weak areas (numeric types, dates, struct/object syntax, etc.)
deleted 10y ago
[deleted]

cm3 10y ago

> but I think I could do without "clobs", "sexprs", and "symbols" at this level of representation I'm by no means a real Lisp programmer, but even I find S-Expressions more natural to write and process. And the simplicity of it allows for great editing tools too. This may be personal, but I always found JSON clunky.

jcrites 10y ago

What do you mean by they both have self-describing schemas? In order to read or write Avro data, an application needs to possess a schema for that data -- the specific schema that the data was written with, and (when writing) the same schema that a later reader expects to find. This means the data is not self-describing. Ion is designed to be self-describing, meaning that no schema is necessary to deserialize and interact with Ion structures. It's consequently possible to interact with Ion in a dynamic and reflective way, for example, in the same way that you can with JSON and XML. It's possible to write a pretty-printer for a binary Ion structure coming off the wire without having any idea of or schema for what's inside. Ion's advantage over those formats is that it's strongly typed (or richly typed, if you prefer). For example, Ion has types for timestamps, arbitrary-precision decimals like for currency, and can embed binary data directly (without base64 encoding), etc. I wouldn't try to say that one or the other is better across the board. Rather, they have tradeoffs and relative strengths in different circumstances. Ion is in part designed to tackle scenarios like where your data might live a really long time, and needs to be comprehensible decades from now (whether you kept track of the schema or not, or remember which one it was); and needs to be comprehensible in a large distributed environment where not every application might possess the latest schema or where coordinating a single compile-time schema is a challenge (maybe each app only cares about some part of the data), and so on. Ion is well-suited to long-lived, document-type data that's stored at rest and interacted with in a variety of potentially complex ways over time. Data data. In the case of a simple RPC relationship between a single client and service, where the data being exchanged is ephemeral and won't stick around, and it's easy to definitively coordinate a schema across both applications, a typical serialization framework is a fine choice.

wyc 10y ago
I think it depends on what level you're referring to. If you mean record-level, then I concede that it's not self-describing. However, looking at the suggested use cases, it seems that it's "self-describing" in that you'll always be able to decode data stored according to what the documentation recommends: "Avro data is always serialized with its schema. Files that store Avro data should always also include the schema for that data in the same file. Avro-based remote procedure call (RPC) systems must also guarantee that remote recipients of data have a copy of the schema used to write that data." https://avro.apache.org/docs/current/spec.html#Data+Serialization https://avro.apache.org/docs/current/spec.html#Data+Serializ...

tjonker 10y ago

A big problem with Avro, BSON, and many other "binary JSON" formats is that they're not isomorphic with JSON, they have a bunch of additional stuff added on. There's Avro documents that don't have direct JSON equivalents. Which means that when a human needs to read the data, you have to transform it into some third format. It's a core feature of Ion that the text and binary representations are isomorphic. You can take any Ion binary document and pretty-print it as an Ion text document that is exactly equivalent. You can edit that document and send it into your application, which will be guaranteed to be able to read it. Or you can take your hand-authored text data and transcode it into binary, and know that any Ion application can handle it without any extra effort.

amyjess 10y ago
> A big problem with Avro, BSON, and many other "binary JSON" formats is that they're not isomorphic with JSON, they have a bunch of additional stuff added on. There's Avro documents that don't have direct JSON equivalents. Which means that when a human needs to read the data, you have to transform it into some third format. Also, field order matters in Avro but not in JSON. That bit me pretty hard once... fortunately, I found out that Python's JSON library lets you read a JSON file into an OrderedDict instead of a plain dict, so I was able to get around it.