3 ms·
JSON is awesome, but it's troublesome that it can't represent binary data (without a separate encoding, which requires metadata, and more code on both sides of
by chubot 6y ago
JSON is awesome, but it's troublesome that it can't represent binary data (without a separate encoding, which requires metadata, and more code on both sides of the wire).
I "discovered" a format that easily solves this, which I call QSN (quoted string notation):
http://www.oilshell.org/release/latest/doc/qsn.html http://www.oilshell.org/release/latest/doc/qsn.html
It's just Rust string literals with single quotes instead of double quotes. Rust string literals have \x01 for arbitrary bytes like C or Python string literals, but without the legacy of say \v and octal, and with just \u{} rather than \u and \U.
I use in Oil to unambiguously parse and print filenames, display argv arrays, etc. All of these are arbitrary binary data.
GNU tools have about 10 different ways to do this, but QSN makes it consistent and precise.
I'm expanding it to QTSV, a variant of TSV, but if you like you could also make some JSON variant. Technically using single quotes rather than double would make it backward compatible with JSON.
- spicybright 6y agoInteresting solution! I usually use base64 encoding unless I'm pushing lots of data, then unfortunately it's easier to make some kind of "file upload" separate from the json if you're going through http.
- jkoudys 6y agoMy demand for that too has slowed in the past couple years, mainly because it's getting easier to do more on the client. e.g. I don't need to upload an entire .docx file, if all my backend needs is ~50kB of values queried out of one of its contained .xml files. Not saying this is a _solution_ to any of the encoding questions, only that it's reduced my immediate need.
- gumby 6y agoLot of json packages support CBOR
- BerislavLopac 6y agoIon? https://amzn.github.io/ion-docs/ https://amzn.github.io/ion-docs/
- chubot 6y agoHm it looks like it would satisfy a lot of binary data use cases, but so would MessagePack, CBOR, and probably a few dozen other formats. A design goal of QSN is to be human readable like JSON. You can view it at the terminal, in an editor, in an e-mail, or in the browser. IMO that fits more naturally with Unix and the web. So I guess I should make clear that it's not only about "binary data", but it's that + a few other constraints.
- BerislavLopac 6y ago> A design goal of QSN is to be human readable like JSON. Well Ion has both text and binary encodings: https://amzn.github.io/ion-docs/docs/spec.html https://amzn.github.io/ion-docs/docs/spec.html
- chubot 6y agoI looked at it more. It has both a string type (which has \xHH escapes oddly specified as code points), and a binary type (which is specified as base64 in text format -- this is bad because base64 isn't human-readable despite being ASCII). The 2 different types makes it unsuitable for the use cases I'm targeting QSN at, which deal only with byte strings, which may be encoded in UTF-8 (like kernel APIs). I don't see what Ion adds over MessagePack, CBOR, or several other such formats... but that's a different discussion :)