6 ms·

JSON is human-readable, why even compare it with this. Is any serialization format now just a "JSON alternative"?

by Levitating 7mo ago

dietr1ch 7mo ago
cat file.whatever | whatever2json | jq ? (Or to avoid using cat to read, whatever2json file.whatever | jq)
- creationix 7mo ago
  Or in this case, just do `rx file.rx` It has jq like queries built in and supports inputs with either rx or json. Also if you prefer jq, you can do `rx file.rx | jq`
  - dietr1ch 7mo ago
    wow, on that case then using `jq` is just a presentation preference at the very last step unless jq is more expressive (which might be the case given how long it has been around?).
    - creationix 7mo ago
      right, the jq query language is much more complex and featureful than the simple selector syntax I added to the rx-cli. But more could be added later as needed or it could just stream JSON output. It would be pretty trivial to hook up a streaming JSON encoder to rx-cli which could then pipe to jq for low-latency lookups. The problem is jq would need to JSON parse all that data which will be expensive.
- Gormo 7mo ago
  That's not really random access, though. You're effectively just searching through the entire dataset for every targeted read you're after. What might be interesting is to have a tool that processes full JSON data and creates a b-tree index on specified keys. Then you could run searches against the index that return byte offsets you can use for actual random access on the original JSON. OTOH, this is basically just recreating a database, just using raw JSON as its storage format.
  - dietr1ch 7mo ago
    Well, JSON had no random access to begin with, so maybe that's on needing JSON. Maybe a query over the random-access file then converted into JSON would work?
    - Gormo 7mo ago
      Right -- that's exactly the point. JSON is sequential data, and has no mechanism for random access. The previous comment described piping the JSON into jq as a solution for querying specific bits of data out of a JSON stream, which is consistent with what you're suggesting here, but still isn't a solution for random access.
  - creationix 7mo ago
    > What might be interesting is to have a tool that processes full JSON data and creates a b-tree index on specified keys. Then you could run searches against the index that return byte offsets you can use for actual random access on the original JSON. I did build that once. But keeping track of the index is a pain. Sometimes I was able to generate the index on-demand and cache it in some ephemeral storage, but overall it didn't work out so well. This system with RX will work better because I get the indexes built-in to the data file and can always convert it back to JSON if needed.
creationix 7mo ago
- this encodes to ASCII text (unless your strings contain unicode themselves) - that means you can copy-paste it (good luck doing that with compressed JSON or CBOR or SQLite - there is a scale where JSON isn't human readable anymore. I've seen files that are 100+MB of minified JSON all on a single very long line. No human is reading that without using some tooling.
- bawolff 7mo ago
  That kind of feels a bit worst of both worlds. None of the space savings/efficiency of binary but also no human readability. Being able to copy/paste a serialization format is not really a feature i think i would care about.
  - creationix 7mo ago
    It's a gradient. I did design several binary formats first, but for my use cases, this is actually better. There is nuance to various use cases. > None of the space savings/efficiency of binary For string heavy datasets, it's nearly the same encoding size as binary. I get 18x smaller sizes compared to JSON for my production datasets. This was originally designed as a binary format years ago (https://github.com/creationix/nibs https://github.com/creationix/nibs) and then later after several iterations, converted to text. > Being able to copy/paste a serialization format is not really a feature i think i would care about Imagine being paged at 3am because some cache in some remote server got poisoned with a bad value (unrelated to the format itself). You load the value in dashboard, but it's encoded as CBOR or some binary format and so you have to download it in a binary safe way, upload that binary file to some tooling or install a cbor reader to your CLI. But then you realize that you don't have exec access to the k8s pods for security reasons, but do have access to a web-based terminal. Again, to extract a binary value you would need to create a shell, hexdump the file and somehow copy-paste that huge hexdump from the web-based terminal to your local machine, un-hex dump it, and finally load it into some CBOR reader. A text format, however is as simple as copy-paste the value from the dashboard and paste into some online tool like https://rx.run/ https://rx.run/ to view the contents.
- rendaw 7mo ago
deleted 7mo ago
[deleted]
jy14898 7mo ago
Came to the same conclusion the moment I had to hunt to see the outputs https://github.com/creationix/rx/tree/main/samples https://github.com/creationix/rx/tree/main/samples
- ok123456 7mo ago
  Reminds me of https://en.wikipedia.org/wiki/PHP_serialization_format https://en.wikipedia.org/wiki/PHP_serialization_format
- SV_BubbleTime 7mo ago
  I was instantly suspicious that a “new better format” for serialization didn’t open with the input/output. And this is why (fucking lol, gtfo): Q^mSat,3^b:d+s+E,4Fri,3^u:h+k+u,6Thu,3^P:j+ If you are effectively going binary, do it. CBOR or Protobuf or any dozen other binary serializations that would be far more efficient. The author claims this is because of copy and pasting… cool, remind me what BASE64 is again?
  - creationix 7mo ago
    It is also a format that can be read as-is without any preprocessing. In some cases base64 can do that, and this format does make heavy use of base64 varints. Sure, you can encode as JSON, then compress with gzip and then base64 encode. You'll probably end up with something smaller than rx and be extremely safe to copy-paste. But your consumers are going to consume orders of magnitude more CPU reading data from this document. RX is usable as-is, is compressed, and is copy-pasteable. It's the unique combination of properties that makes it interesting.
    - SV_BubbleTime 7mo ago
      >It is also a format that can be read as-is without any preprocessing. >Q^mSat,3^b:d+s+E,4Fri,3^u:h+k+u,6Thu,3^P:j+ My man… no. I have no doubt you could kind of figure out what that sample is hot off the heels of writing this, and likely not in six months. And to consider that anyone else would fill their brain with the rules to decipher that, Nah 2.0.
Gormo 7mo ago
It's also quite odd to create a serialization format optimized for random access.
- IshKebab 7mo ago
  Not at all. What makes you say that?
- j16sdiz 7mo ago
  many serialization format are just a memory structure dump.
- creationix 7mo ago
  Serialized just means encoded as a stream of bytes so that it can be transferred between systems. There are absolutely cases where you want to be able to query a value directly like a database instead of parsing the entire thing to memory before you can read it. Think of this as no-sql sqlite.
  - Gormo 7mo ago
    > Serialized just means encoded as a stream of bytes so that it can be transferred between systems. Yes, serially. Which means no random-access across the transfer channel.
    - creationix 7mo ago
      So you're saying that random access formats that are encoded to disk as a stream of bytes are not "serialized" because you don't alway read them in order? Yes, many formats are read start-to-end, but I don't think that's a requirement. The important thing is it can be stored and transmitted as a stream of bytes. The word describes how it is transported and stored, not how it is read.
      - Gormo 7mo ago
        > So you're saying that random access formats that are encoded to disk as a stream of bytes are not "serialized" because you don't alway read them in order? Yes. That's precisely what "random access" means. > The important thing is it can be stored and transmitted as a stream of bytes. What isn't stored and transmitted as a stream of bytes? Memory itself is a sequential array of bytes. The criteria you're using here seem to be all-encompassing. When we're talking about serial access to data, it means that we're sequentially parsing the stream of bytes from its starting point, rather than reading arbitrarily from any point in the stream that we desire.
creationix 7mo ago
Thanks for the feedback. I've improved the framing to make the purpose/value more clear. What do you think about "RX is a read-only embedded store for JSON-shaped data"? https://www.npmjs.com/package/@creationix/rx https://www.npmjs.com/package/@creationix/rx

JSON is human-readable, why even compare it with this. Is any serialization format now just a "JSON alternative"?

dietr1ch 7mo ago

cat file.whatever | whatever2json | jq ? (Or to avoid using cat to read, whatever2json file.whatever | jq)

creationix 7mo ago
Or in this case, just do `rx file.rx` It has jq like queries built in and supports inputs with either rx or json. Also if you prefer jq, you can do `rx file.rx | jq`
- dietr1ch 7mo ago
  wow, on that case then using `jq` is just a presentation preference at the very last step unless jq is more expressive (which might be the case given how long it has been around?).
  - creationix 7mo ago
    right, the jq query language is much more complex and featureful than the simple selector syntax I added to the rx-cli. But more could be added later as needed or it could just stream JSON output. It would be pretty trivial to hook up a streaming JSON encoder to rx-cli which could then pipe to jq for low-latency lookups. The problem is jq would need to JSON parse all that data which will be expensive.
Gormo 7mo ago
That's not really random access, though. You're effectively just searching through the entire dataset for every targeted read you're after. What might be interesting is to have a tool that processes full JSON data and creates a b-tree index on specified keys. Then you could run searches against the index that return byte offsets you can use for actual random access on the original JSON. OTOH, this is basically just recreating a database, just using raw JSON as its storage format.
- dietr1ch 7mo ago
  Well, JSON had no random access to begin with, so maybe that's on needing JSON. Maybe a query over the random-access file then converted into JSON would work?
  - Gormo 7mo ago
    Right -- that's exactly the point. JSON is sequential data, and has no mechanism for random access. The previous comment described piping the JSON into jq as a solution for querying specific bits of data out of a JSON stream, which is consistent with what you're suggesting here, but still isn't a solution for random access.
- creationix 7mo ago
  > What might be interesting is to have a tool that processes full JSON data and creates a b-tree index on specified keys. Then you could run searches against the index that return byte offsets you can use for actual random access on the original JSON. I did build that once. But keeping track of the index is a pain. Sometimes I was able to generate the index on-demand and cache it in some ephemeral storage, but overall it didn't work out so well. This system with RX will work better because I get the indexes built-in to the data file and can always convert it back to JSON if needed.

creationix 7mo ago

- this encodes to ASCII text (unless your strings contain unicode themselves) - that means you can copy-paste it (good luck doing that with compressed JSON or CBOR or SQLite - there is a scale where JSON isn't human readable anymore. I've seen files that are 100+MB of minified JSON all on a single very long line. No human is reading that without using some tooling.

bawolff 7mo ago
That kind of feels a bit worst of both worlds. None of the space savings/efficiency of binary but also no human readability. Being able to copy/paste a serialization format is not really a feature i think i would care about.
- creationix 7mo ago
  It's a gradient. I did design several binary formats first, but for my use cases, this is actually better. There is nuance to various use cases. > None of the space savings/efficiency of binary For string heavy datasets, it's nearly the same encoding size as binary. I get 18x smaller sizes compared to JSON for my production datasets. This was originally designed as a binary format years ago (https://github.com/creationix/nibs https://github.com/creationix/nibs) and then later after several iterations, converted to text. > Being able to copy/paste a serialization format is not really a feature i think i would care about Imagine being paged at 3am because some cache in some remote server got poisoned with a bad value (unrelated to the format itself). You load the value in dashboard, but it's encoded as CBOR or some binary format and so you have to download it in a binary safe way, upload that binary file to some tooling or install a cbor reader to your CLI. But then you realize that you don't have exec access to the k8s pods for security reasons, but do have access to a web-based terminal. Again, to extract a binary value you would need to create a shell, hexdump the file and somehow copy-paste that huge hexdump from the web-based terminal to your local machine, un-hex dump it, and finally load it into some CBOR reader. A text format, however is as simple as copy-paste the value from the dashboard and paste into some online tool like https://rx.run/ https://rx.run/ to view the contents.
rendaw 7mo ago

deleted 7mo ago

[deleted]

jy14898 7mo ago

Came to the same conclusion the moment I had to hunt to see the outputs https://github.com/creationix/rx/tree/main/samples https://github.com/creationix/rx/tree/main/samples

ok123456 7mo ago
Reminds me of https://en.wikipedia.org/wiki/PHP_serialization_format https://en.wikipedia.org/wiki/PHP_serialization_format
SV_BubbleTime 7mo ago
I was instantly suspicious that a “new better format” for serialization didn’t open with the input/output. And this is why (fucking lol, gtfo): Q^mSat,3^b:d+s+E,4Fri,3^u:h+k+u,6Thu,3^P:j+ If you are effectively going binary, do it. CBOR or Protobuf or any dozen other binary serializations that would be far more efficient. The author claims this is because of copy and pasting… cool, remind me what BASE64 is again?
- creationix 7mo ago
  It is also a format that can be read as-is without any preprocessing. In some cases base64 can do that, and this format does make heavy use of base64 varints. Sure, you can encode as JSON, then compress with gzip and then base64 encode. You'll probably end up with something smaller than rx and be extremely safe to copy-paste. But your consumers are going to consume orders of magnitude more CPU reading data from this document. RX is usable as-is, is compressed, and is copy-pasteable. It's the unique combination of properties that makes it interesting.
  - SV_BubbleTime 7mo ago
    >It is also a format that can be read as-is without any preprocessing. >Q^mSat,3^b:d+s+E,4Fri,3^u:h+k+u,6Thu,3^P:j+ My man… no. I have no doubt you could kind of figure out what that sample is hot off the heels of writing this, and likely not in six months. And to consider that anyone else would fill their brain with the rules to decipher that, Nah 2.0.

Gormo 7mo ago

It's also quite odd to create a serialization format optimized for random access.

IshKebab 7mo ago
Not at all. What makes you say that?
j16sdiz 7mo ago
many serialization format are just a memory structure dump.
creationix 7mo ago
Serialized just means encoded as a stream of bytes so that it can be transferred between systems. There are absolutely cases where you want to be able to query a value directly like a database instead of parsing the entire thing to memory before you can read it. Think of this as no-sql sqlite.
- Gormo 7mo ago
  > Serialized just means encoded as a stream of bytes so that it can be transferred between systems. Yes, serially. Which means no random-access across the transfer channel.
  - creationix 7mo ago
    So you're saying that random access formats that are encoded to disk as a stream of bytes are not "serialized" because you don't alway read them in order? Yes, many formats are read start-to-end, but I don't think that's a requirement. The important thing is it can be stored and transmitted as a stream of bytes. The word describes how it is transported and stored, not how it is read.
    - Gormo 7mo ago
      > So you're saying that random access formats that are encoded to disk as a stream of bytes are not "serialized" because you don't alway read them in order? Yes. That's precisely what "random access" means. > The important thing is it can be stored and transmitted as a stream of bytes. What isn't stored and transmitted as a stream of bytes? Memory itself is a sequential array of bytes. The criteria you're using here seem to be all-encompassing. When we're talking about serial access to data, it means that we're sequentially parsing the stream of bytes from its starting point, rather than reading arbitrarily from any point in the stream that we desire.

creationix 7mo ago

Thanks for the feedback. I've improved the framing to make the purpose/value more clear. What do you think about "RX is a read-only embedded store for JSON-shaped data"? https://www.npmjs.com/package/@creationix/rx https://www.npmjs.com/package/@creationix/rx