15 ms·
Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
- trinovantes 5y agoI wonder what's the performance relative to native JSON parsers?
- seniorsassycat 5y agoParsing ion text should be similar to json, it has the same characteristics. All JSON is valid ion text so you can even parse JSON with an ION parser. The binary parser is much faster. All fields are length-prefixed so a parser doesn't have to scan forward for the next syntax element. The ion parsers (lexer? not sure the right vocab) I've worked with have a `JSON.parse` equivalent that returns a fully realized object, a Map, Array, Int, ect but they also have a streaming parser that yields value by value. You can skip over values you don't need, step over structs or into structs without creating a Map or Array. That can be much faster.
- jonwilsdon 5y agoDisclosure: I manage the Ion and PartiQL teams at Amazon. We have done some work on performance comparisons with the ion-java-benchmark-cli tool (https://github.com/amzn/ion-java-benchmark-cli https://github.com/amzn/ion-java-benchmark-cli). Right now you can compare JSON serialized with Jackson and there is a pull request (https://github.com/amzn/ion-java-benchmark-cli/pull/27 https://github.com/amzn/ion-java-benchmark-cli/pull/27) for comparing against CBOR that should be merged soon. We are always happy to hear suggestions for what is useful in this area.
- grouphugs 5y agofuck amazon, and fuck everyone that won't stop promoting them.
- chromatin 5y agoCheck out Ilya Yaroshenko’s Ion library for D, part of the larger ‘mir’ library: http://mir-ion.libmir.org/ http://mir-ion.libmir.org/ https://github.com/libmir/mir-ion https://github.com/libmir/mir-ion
- hatf0 5y agoWeird to see the library I work on show up in HN —- Mir Ion is a pretty complicated library (and admittedly our documentation needs work — I’m working on that!), but I’m very proud of our work. Some fun things about Mir Ion: - We can fully deserialize Ion at compile-time (via D’s CTFE functionality) - We’re one of the fastest JSON parsing libraries (and one of the most memory efficient too — we actually store all JSON data in memory as Ion data, which is vastly more efficient) - We’re nearly 100% compliant to all of the upstream test cases (our main issue is that we’re often too lax on spec, and allow files that are invalid through) - The entire library is (nearly) all `@nogc`, thanks to the Mir standard library If anyone has any questions on Mir Ion, feel free to shoot me a line at harrison (at) 0xcc.pw
- jscholes 5y agoThe latest format for Kindle eBooks, KFX, is based on this.
- loeg 5y agoYep: https://github.com/apprenticeharper/DeDRM_tools/blob/master/DeDRM_plugin/ion.py https://github.com/apprenticeharper/DeDRM_tools/blob/master/...
- yayitswei 5y agoReminds me of Clojure's transit.
- cmancini 5y agoThat was the first thing I thought of! Big fan of transit. Seems very similar.
- programd 5y ago"Zero and negative dates are not valid, so the earliest instant in time that can be represented as a timestamp is Jan 01, 0001" That seems to be...a problem? How do you deal with archeological dates, of which there are many, in Ion?
- elteto 5y agoWhat modern tech service (of the kind that would have use for Ion) is dealing with archaeological dates _at scale_? Honest question.
- pdpi 5y agoThat’s an interesting question. On the one hand, it feels weird that you can’t represent those dates at all. On the other hand, representability of a given date becomes progressively less useful the further back in time you go, and stuff becomes really gnarly once you go past the Julian calendar in 45BC. Also, simplifying to “no dates before Jan 1 0001” has very little impact on applications dealing with the modern-ish world (with “modern” generously defined as “anything after the collapse of the Roman Empire”), and I can only assume applications dealing with earlier times could do with a more specialised representation for dates anyway.
- biztos 5y agoJust to give one example, in Thailand right now it's the year 2564. 1 BC for some is not "-1" for everyone.
- jsnell 5y agoPrevious discussions: https://news.ycombinator.com/item?id=11546098 https://news.ycombinator.com/item?id=11546098 https://news.ycombinator.com/item?id=23921610 https://news.ycombinator.com/item?id=23921610
- dang 5y agoThanks! Macroexpanded: Amazon Ion - https://news.ycombinator.com/item?id=23921610 https://news.ycombinator.com/item?id=23921610 - July 2020 (110 comments) Amazon open-sources Ion – a binary and text interchangable, typed JSON-superset - https://news.ycombinator.com/item?id=11546098 https://news.ycombinator.com/item?id=11546098 - April 2016 (163 comments)
- throwoutway 5y agoWhat do you use for the macroexpansion? There are a hundred odd tasks like this that I need to create macros for!
- dang 5y agoI mean that metaphorically but I do have a bunch of keyboard shortcuts (in a browser extension) that make finding these, and formatting the comments, much faster.
- rjzzleep 5y agoI feel like a lot of file formats came out of companies, but even protocol buffers isn't calling itself google protocol buffers. What is it with modern companies putting their name everywhere they can?
- rp1 5y agoGrpc?
- tjpnz 5y agoThe g is (allegedly) not for Google.
- rp1 5y agoWhat's it for?
- Rebelgecko 5y agogRPC
- travisd 5y ago> What does gRPC stand for? > gRPC Remote Procedure Calls, of course! https://grpc.io/docs/what-is-grpc/faq/ https://grpc.io/docs/what-is-grpc/faq/
- deleted 5y ago[deleted]
- moltenguardian 5y agohttps://grpc.github.io/grpc/core/md_doc_g_stands_for.html https://grpc.github.io/grpc/core/md_doc_g_stands_for.html
- SavantIdiot 5y agoIt's funny, I didn't realize protobuf was a Google thing for a long time because of that. At least `protobuf` is a reasonably-specific search term. `ion` returns too much noise. Almost a good reason to name things weirder, like `iyon`. But then they'd get laughed at. EDIT: oh, its a Tagolog name too, and a light company.
- dorianmariefr 5y agoPretty neat, but isn't it like *two* formats: one binary and one textual?
- echelon 5y agoOne data model, two serializations of it.
- seniorsassycat 5y agoTwo representations of the same data structures. Ion text is like JSON, in fact all JSON is valid ion text. Ion text has comments, trailing commas, dates, and unquoted keys. It's a really good alternative to JSON, YAML, or TOML. Ion binary is compact and fast to parse. Values are length prefixed so the parser can skip over unneeded fields or structs, saving time parsing and memory allocated. Common string values, like struct keys and enum values, are given numeric ids and stored once in a header table.
- ComputerGuru 5y agoDo comments persist in binary serialization or is that a lossy one-way operation?
- the_girabbit 5y agoComments don’t persist in binary. Like white space, they are explicitly not part of the data model.
- seniorsassycat 5y agoI think the ion java library includes a AST parser that includes comments, but the ION data model doesn't. The binary format cannot include comments. I think many text parsers are missing libraries that edit documents in place, preserving formatting and comments.
- OJFord 5y agoConsider that binary, binary coded decimal, Gray code, hexadecimal, octal, etc. are all 'formats' expressing the same (numerical) idea. You can't say the same of, for example, YAML & JSON, since the former (if not the latter?) has constructs unrepresentable in the other. It's slightly confused because an application might 'serialise to' JSON or YAML or Ion equivalently - but really that's saying the application's data being serialised fits a model that's a subset of the intersection between those formats. You could call Ion two, but it's more than that in that it's also a promise that they're 1:1 (err, and onto if you like) - their intersection is their union.
- transfire 5y ago`years::4`? I don't know. What not `4::years`? Also, symbols converted to integers means the receiving end has to already know exactly what they are.
- sokoloff 5y agoThere is provision for encoding a local symbol table: https://amzn.github.io/ion-docs/docs/symbols.html#processing-of-symbol-tables https://amzn.github.io/ion-docs/docs/symbols.html#processing...
- re 5y agoPutting annotations before values is likely to be more useful for streaming parsers than putting them after. Imagine the case where the annotation represents a class that you want to deserialize a large object into.
- stevefan1999 5y agoHow does that differs from the likes of MessagePack and CBOR?
- seniorsassycat 5y agoIons text format is a nice JSON alternative while it's binary format is very compact and allows for efficient sparse parsing. Fields are prefixed with their length so you can skip over unneeded fields or structs while only creating objects for values you'll use.
- hirundo 5y agoIt seems like an odd choice to make the type "metadata" a prefix to the value, rather than a separate field. It feels like overloading. What's the advantage?
- re 5y agoNot sure I understand exactly what "a separate field" would look like, but: 1. Considering that a goal of Ion is to be a strict superset of JSON, separate syntax ensures that any JSON value can be parsed without misinterpreting some field as an annotation--there are no reserved/"magic" field names. 2. Annotations can be applied to any type of value, not just objects, which are the only type that have fields.
- deleted 5y ago[deleted]
- indymike 5y agoIt tells you how to load the value and can be human readable for audit purposes. example: degrees::'celsius'::100
- wisty 5y agoI scanned the docs, and can't see what happens if you alter your data schema. Anyone know?
- travisd 5y agoSeems like you have to handle that yourself. The serialized data includes the type, so your app code might have to have logic a la “if type1: … else: …” after parsing it.
- wisty 5y agoOK, so it's one of the more flexible ones (like those binary jsons) rather than something like protobuf. I guess that should have been obvious from "self-describing".
- AtlasBarfed 5y agoThis is json with relaxed jackson parsing: quote-optional keys, comments, all doable with jackson OOTB for years now.
- quda 5y agoAnother useless transfer data format. It will be forgotten within a year or two.
- oandrew 5y agoSo basically it's Amazon's version of Apache Avro. Avro supports binary/json serialization, schema evolution , logical types (e.g. timestamp) and other cool stuff. https://avro.apache.org/docs/current/spec.html https://avro.apache.org/docs/current/spec.html
- joshka 5y agoAvro didn't exist when Ion started development.
- whimsicalism 5y ago... or thrift ... or protobuf https://xkcd.com/927/ https://xkcd.com/927/
- fnord77 5y agoI wanted to see what the differences are between Ion and Avro. Unlike avro, ion doesn't require a schema.
- tootie 5y agoNo schema validation?
- landonxjames 5y agoI believe that is provided by the Ion Schema Language https://amzn.github.io/ion-schema/docs/spec.html https://amzn.github.io/ion-schema/docs/spec.html
- quiffledwerg 5y agoI just feel deeply disinclined towards supporting anything Amazon because they’ve developed a reputation as such a poor community member.
- setheron 5y agoWow I remember using Ion back at Amazon in 2012. I can’t remember but I think the order data warehouse was using it … I also now remember back to using something that was akin to FaaS but wasn’t called that. I could give them a JAR of some code that would execute on some Ion data for the order data when it changed. Basically FaaS for an ETL pipeline… Crazy how ahead of the times some companies were.
- User23 5y agoThat was a golden age for Amazon engineering. I assume they’re still great, but that stretch from 2004 to 2014 was some incredible advancement.
- vineyardmike 5y agoI wonder why it took 10+ years to share then?
- timdorr 5y agoActually, it only took them 4-ish years: https://amzn.github.io/ion-docs/news/2016/04/21/amazon-open-sources-ion.html https://amzn.github.io/ion-docs/news/2016/04/21/amazon-open-...
- Eelongate 5y agoDid anything ever become of the lispy language that was being built using Ion as its homoiconic syntax? I'm afraid I can't recall what it was called. Fusion maybe?
- garmaine 5y agoI hope you get an answer, because this sounds very intriguing but google is failing me in finding any references to it.
- kayamon 5y agoDunno about that one but if you like that sort of thing, check out Rebol.
- throwaway_sZntK 5y agoYeah, Fusion was the name. Last I heard, they discontinued it, saying essentially "If you really want a full Lisp, there's already Clojure." S-exps continued to be used in Ion for embedded 1-liners but they only supported a handful of operators, not a full language.
- swaranga 5y agoI built a system in my previous team where clients can register "filters" described in Fusion. My system, which was a source of a lot of different notifications, would then run these filters and only send those notifications that passed the filters. It became very popular very quickly because of the easy on-boarding and the fact that clients not get only a fraction of the messages they were interested in. I just checked the Java implementation, it seems to be still active and get commits.
- clhodapp 5y agoIt's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in domain modeling. Apart from that, while you can always sort of fake it with a bunch of optional fields I believe that you kind of need a native encoding to a tagged union if you want to avoid bloating your messages.
- the_girabbit 5y agoGenuine question—why would you need a sum type in a self-describing data format?
- valenterry 5y agoWell, there are already sumtypes, just only specific builtin ones, not custom ones. E.g. booleans are sumtypes (true | false). Everything else that is nullable is also a sumtype (e.g. number | null). I think it should be pretty obvious how these are helpful and why they are needed no?
- the_girabbit 5y agoYeah, but it’s a schema-less, self-describing data format. It’s not like a specific position in a data stream has a requirement to be a specific type. I can see why sum types would be useful in a schema or for the elements of a collection that is required to be homogeneous (ie. List<Foo|Bar>). For what use case would one use custom sum types in a schema-less data format?
- dastbe 5y agodata interchange formats try to encode as little backwards incompatible information as possible. in this case, it would be the restriction that something is a sum type when it could have multiple fields set in the future. another example is protobuf moving to all fields being optional by default. as for the wire format, a variant struct where you've only instantiated a single field will encode down to just about the minimum amount of information required.
- n8ta 5y agoI recently implemented a similar (simpler) format https://baremessages.org/ https://baremessages.org/ in ruby. First thoughts are: ION pros: - easy to skip around while reading a file - no need to write a schema - backed by amazon so major langs will have impls - good date support - better concatenation, probably better suited to logging than bare ION cons - what's the text format even for? BARE pros: - schemas keep things tightly versioned - smaller binaries (not self describing like ion) - simpler to implement so tons of devs have impl'ed for their favorite lang - better suited to small messages (think REST json api) BARE cons: - no skip read - no date support I might do an ion ruby implementation too, to really feel out the difference.
- ozzythecat 5y agoIon text is helpful so you can convert ion binary to text for debugging:
- seniorsassycat 5y agoion text is a good contender for JSON, YAML, TOML usecases. It's also a good way to present the binary to humans.
- imiric 5y ago> what's the text format even for? Configuration files? Not sure if that's an intended use case, but being more flexible than JSON and stricter than YAML seems ideal for configuration.
- the_girabbit 5y agoIon will be even better for (structured) logging if this proposal for templates ever happens. https://github.com/amzn/ion-docs/pull/104 https://github.com/amzn/ion-docs/pull/104 Looks like no one’s even so much as commented on it in the last year, so it might have been abandoned.
- n8ta 5y agoIon is already a little too complex for my taste. It'd be a shame to see it go the same way as yaml where it's so complex that most major implementations are not safely interoperable.
- hliyan 5y agoThis reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serialization/deserialization process by making the memory format of the message and the wire format one and the same. The message class contained the same buffer that you would send/store. The GetInt(fieldID) method of the class simply points to the right place in the buffer and does a cast to int. Application logs contained these messages, rather than plain text. There was a special reader to read logs. Messages were exchanged over raw TCP. They contained their own application layer sequence number so that streams could resume after disconnection. In that world, lantencies were so low that the response to your order submission would land in your front-end before you've had time to lift your finger off the enter key. I now work with web based systems. On days like this, I miss the old ways.
- secondcoming 5y agoI assume you were using C++? I'm not sure what you describe is possible these days due to UB. At the very least just casting bytes received over the wire to a type is UB, so you technically need a memcpy() and hope that the compiler optimises it out.
- hliyan 5y agoYes, it was C++. I was unfamiliar with the acronym "UB" so did a Google search. Does it mean "Undefined Behavior"? If I remember correctly, primitive types other than strings are memcpy'd. GetStr basically returned a char* to the right place in the buffer.
- secondcoming 5y agoApologies, yes Undefined Behaviour
- gpderetta 5y ago
- syspec 5y agoSurprised I have not heard of this before, I'd love something to come along and give JSON a kick in the pants. I do think JSON is the defacto standard, and it really does get the job done, but for some more advances uses something like this could really shine.
- petilon 5y agoI like the fact that you can annotate objects as well, not just literals. So this is valid: animal: Tiger:: { gender: 'F', weight: 450 } This solves the inheritance problem, i.e., if you have multiple subclasses how do you know which type to deserialize as?
- plandis 5y agoI believe that this is exactly how Jackson serialized Ion handles subtype polymorphism.
- Zamicol 5y agoAm I the only one that doesn't like base 64? Hex for when efficiency isn't paramount. Base 85 or BasE91 for when efficiency is more of a concern. http://base91.sourceforge.net/ http://base91.sourceforge.net/
- Aeolun 5y agoI like base64 because it’s the de-facto standard, and data size (in places where I’d use base64) isn’t a main concern for me.
- stjohnswarts 5y agoYeah I get tired of reinvention of everything for tiny gains in size/performance.
- hackcasual 5y agoYou want to use hex whenever byte aligned data is going to be compressed. Base64 quadruples byte level symbols
- ralusek 5y agoI understand the case for Base91, but why hex over Base64? Base64 for readability and sticking to multiples of two, Base91 for maximum efficiency with readable ASCII.
- Zamicol 5y agoBase 64 is good at nothing and bad at some things. - Hex is human readable, case insensitive, not that "inefficient", and always aligns to bytes. - Base 85 and basE91 are efficient. - Bitcoin uses Base58 because they thought base 64 was too human unreadable. Ethereum uses Hex. - Base 256 (bytes) is efficient and the native language of computers. Base 64 is not efficient, not human readable, and not easy to encode. The biggest problem with base 64 is that base 64 is not base 64. Are you doing base 64 with padding? Are you doing base 64 with URL safe characters or URL unsafe characters? Are you following the standard RFC 4648 bucket encoding, or are you using iterative divide by radix? I think a great place where the cracks show is JOSE, where for things like thumbprints there's a ton of conversion steps (UTF-8 key -> base 64 -> ASCII bytes -> digest (bytes) -> base 64 thumbprint). My personal advise for 90% of projects considering looking at base 64 should just use Hex or bytes. If needing human readability, use Hex. Otherwise use binary.
- michalkrupa 5y agoYes, we are all still very excited about JSON. (Edit: and BSON)
- otabdeveloper4 5y agoNice! This thing is actually sane and thought through. A first for serialization formats. They're usually a shitshow. (Should have gone with 'rational' instead of 'decimal', though. Decimal will be too painful to implement accross languages and implementations. Java bias?)
- sirk390 5y agoBut decimal are way more useful as they can represent currency amounts. It would be strange to show a currency amount like "3/4" or "11/12". Personally, the two datatypes I have always been adding manually to json are datetimes and decimals (from python)
- otabdeveloper4 5y agoA currency amount is just a rational number with "1000000" as a denominator. This is the correct representation, and how Google or the blockchain do it.
- sirk390 5y agoI don't thing there is a more "correct" representation. Representing it as a string is equaly correct. Blockchain or banking often represents money as an integer with the smallest divisible unit (cent, or satoshi), but it is not applicable here because there is no smallest divisible unit.
- sirk390 5y agotimestamps and decimal are the two most useful additions compared to json. They would be nice to add to json if that is somehow possible.
- nly 5y agoJSON numbers, just like all human readable formats, are decimal... it's not like binary double values are printed out in to JSON in hex or base64 Sure 99% of decoders convert them to and from binary doubles, but that's purely an implementation choice.
- indymike 5y ago> JSON numbers, just like all human readable formats, are decimal... All JSON numbers are implemented as integers or floating point, and as a result, have to be cast as a decimal (a decimal type is generally something that meets this specification: http://speleotrove.com/decimal/ http://speleotrove.com/decimal/) when you import them. Decimal types differ from floating point types in three ways: they are accurate, and they take into account rounding rules and precision. Decimal math is slower, can have greater precision and is better suited to domains where finite precision is needed. Floating point is faster, but is not as precise, so it's good for some scientific uses... or where perfect precision isn't important but speed is... say 3d graphics. I've billed lots of hours over the years fixing code where a developer used floats where they should have used decimals. For example, if you are dealing with money, you probably want decimal. It's one of those problems like trying to parse email addresses with a regex or rolling your own crypto... it will kind a work until someone finds out it really doesn't (think accounting going, our numbers are off by random amounts, WTF?).
- nly 5y agoA binary double can hold any decimal value to 15 digits of precision, so as a serialisation format it's a bit of a non-issue... you just need to convert to decimal and round appropriately before doing any arithmetic where it matters. And you're confusing JSON the format with typical implementations. Open a JSON file and you see decimal digits. There is no limit to the number of the digits in the grammar. Parsing these digits and converting them to binary doubles, for example, is actually slower than parsing them as decimals, because you have to do the latter anyway to accomplish the former. Almost all JSON libraries convert to binary (e.g. doubles) because of their ubiquitous hardware and software support...but some libraries like RapidJSON expose raw numeric strings out of the parser if you want to plug in a decimal library
- mgamache 5y agomsgpack is near the top for speed and size... Readability is nice. Are there other advantages? https://msgpack.org/index.html https://msgpack.org/index.html
- nprateem 5y agoShame there's no PHP lib :(
- mpfundstein 5y agoyour chance
- jonwilsdon 5y agoDisclosure: I manage the Ion and PartiQL teams at Amazon. If you want to create an issue for it (the best repo is probably the ion-docs one: https://github.com/amzn/ion-docs/issues https://github.com/amzn/ion-docs/issues) that will help to show us there is demand for it. Providing information on your use case helps us prioritize.
- cma 5y ago> Ion supports comments. Thank god.. JSON for config files without comments is so awful.
- silvestrov 5y agoThis is what JSON should have been extended to. But Douglas Crockford just don't want to innovate anything, just like Gruber didn't want to make a proper specification of the Markdown format. Sometimes people are keeping innovation back. Fortunately this did not happend with html. The main thing missing from the text format is a magic and version number. At least the binary format has it.
- kwertyoowiyop 5y agoThe dominance of JSON shows that Crockford made some good decisions, even though we may not agree with them on any given day.
- seanclayton 5y agoThe dominance of JSON just shows that JS is dominant.
- usrusr 5y agoEven JS stopped parsing JSON as a subset JS a long time ago. JSON lineage has been irrelevant in terms of popularity transmission ever since people stopped doing var jsonobj = eval(jsonstring);
- indymike 5y ago> The dominance of JSON just shows that JS is dominant. I don't know that's the case... I've used JSON in lots of non-JS languages because it just works, and errors rarely are caused by mismatches in how JSON behaves in language X and language Y. A lot of that is that it is simple, and rigid.
- ramraj07 5y agoSimple, rigid and limited and everyone’s settled on it so you can’t do anything now.
- tlocke 5y agoOne problem with Ion is that it doesn't have a map type, but instead a struct type that allows duplicate keys. I created Zish https://github.com/tlocke/zish https://github.com/tlocke/zish as a serialization format that addresses the shortcomings of JSON and Ion. Any comments / criticisms welcome.
- CyanLite4 5y agoHow does this compare with MsgPack?
- 95th 5y agoRemind of bencoding used in torrents
- kats 5y agoIt's just another format not better than any of the others.
- deschutes 5y agoI don't like ion. The added features (over json) don't pull their weight. Symbols, annotations and the binary format all add significant complexity but don't make the format much better. As a consequence of the added complexity language support is poor. For RPC the binary encoding compares poorly to external schema formats like protobuf. In this context binary ion is a poorly compressed text format. I don't think the partial document read capability of the binary format is all that important, but I've never worked on an application that would benefit from it either.
- deleted 5y ago[deleted]