3 ms·

cat file.whatever | whatever2json | jq ? (Or to avoid using cat to read, whatever2json file.whatever | jq)

creationix 7mo ago
Or in this case, just do `rx file.rx` It has jq like queries built in and supports inputs with either rx or json. Also if you prefer jq, you can do `rx file.rx | jq`
- dietr1ch 7mo ago
  wow, on that case then using `jq` is just a presentation preference at the very last step unless jq is more expressive (which might be the case given how long it has been around?).
  - creationix 7mo ago
    right, the jq query language is much more complex and featureful than the simple selector syntax I added to the rx-cli. But more could be added later as needed or it could just stream JSON output. It would be pretty trivial to hook up a streaming JSON encoder to rx-cli which could then pipe to jq for low-latency lookups. The problem is jq would need to JSON parse all that data which will be expensive.
Gormo 7mo ago
That's not really random access, though. You're effectively just searching through the entire dataset for every targeted read you're after. What might be interesting is to have a tool that processes full JSON data and creates a b-tree index on specified keys. Then you could run searches against the index that return byte offsets you can use for actual random access on the original JSON. OTOH, this is basically just recreating a database, just using raw JSON as its storage format.
- dietr1ch 7mo ago
  Well, JSON had no random access to begin with, so maybe that's on needing JSON. Maybe a query over the random-access file then converted into JSON would work?
  - Gormo 7mo ago
    Right -- that's exactly the point. JSON is sequential data, and has no mechanism for random access. The previous comment described piping the JSON into jq as a solution for querying specific bits of data out of a JSON stream, which is consistent with what you're suggesting here, but still isn't a solution for random access.
- creationix 7mo ago
  > What might be interesting is to have a tool that processes full JSON data and creates a b-tree index on specified keys. Then you could run searches against the index that return byte offsets you can use for actual random access on the original JSON. I did build that once. But keeping track of the index is a pain. Sometimes I was able to generate the index on-demand and cache it in some ephemeral storage, but overall it didn't work out so well. This system with RX will work better because I get the indexes built-in to the data file and can always convert it back to JSON if needed.

Or in this case, just do `rx file.rx` It has jq like queries built in and supports inputs with either rx or json. Also if you prefer jq, you can do `rx file.rx | jq`

dietr1ch 7mo ago
wow, on that case then using `jq` is just a presentation preference at the very last step unless jq is more expressive (which might be the case given how long it has been around?).
- creationix 7mo ago
  right, the jq query language is much more complex and featureful than the simple selector syntax I added to the rx-cli. But more could be added later as needed or it could just stream JSON output. It would be pretty trivial to hook up a streaming JSON encoder to rx-cli which could then pipe to jq for low-latency lookups. The problem is jq would need to JSON parse all that data which will be expensive.

Gormo 7mo ago

That's not really random access, though. You're effectively just searching through the entire dataset for every targeted read you're after. What might be interesting is to have a tool that processes full JSON data and creates a b-tree index on specified keys. Then you could run searches against the index that return byte offsets you can use for actual random access on the original JSON. OTOH, this is basically just recreating a database, just using raw JSON as its storage format.

dietr1ch 7mo ago
Well, JSON had no random access to begin with, so maybe that's on needing JSON. Maybe a query over the random-access file then converted into JSON would work?
- Gormo 7mo ago
  Right -- that's exactly the point. JSON is sequential data, and has no mechanism for random access. The previous comment described piping the JSON into jq as a solution for querying specific bits of data out of a JSON stream, which is consistent with what you're suggesting here, but still isn't a solution for random access.
creationix 7mo ago
> What might be interesting is to have a tool that processes full JSON data and creates a b-tree index on specified keys. Then you could run searches against the index that return byte offsets you can use for actual random access on the original JSON. I did build that once. But keeping track of the index is a pain. Sometimes I was able to generate the index on-demand and cache it in some ephemeral storage, but overall it didn't work out so well. This system with RX will work better because I get the indexes built-in to the data file and can always convert it back to JSON if needed.