3 ms·

Uh ... what about when I don't want to load the entire stream into memory before the next stage starts running? Are these implicit json arrays that stream out?

Uh ... what about when I don't want to load the entire stream into memory before the next stage starts running? Are these implicit json arrays that stream out? Or do we now have incompatible json shell tools and streaming text tools as a permanent fixture? I'm excited about structured output ideas, but json? I'd much rather have streams of whitespace separated words than json. That's in that "No type system is better than a bad type system" metaphorical area. I'll take grepping with a theoretically brittle regex over this jq[1] any day of the week. [1] jq -nc --stream 'inputs | select(length==2) | select( [.[0][0,2,4]] == ["results", "data", "row"]) | [ .[0][6], .[1]] '

selfhoster11 5y ago
For the first point, there's no reason why we can't use JSON array wrapped output. In fact, we likely should use something like this for uniformity. Loading everything into memory is also not too problematic assuming a working swap space and reasonably well-architectured output schema. For the second point, whitespace sensitivity is the one mistake that greatly pisses me off with Unix. I should be able to pass arguments and filenames with as many spaces as I want. We are in the 21st century and occasionally do use spaces in filenames.
kbrazil 5y ago
I agree JSON is probably not right for every type of program output, but the age of web APIs has shown us that is probably great or adequate 90% of the time. If something is spewing out long lines of data I think JSON Lines would be a good option so you don’t need to read the whole structure into RAM. But any other structured output that has a healthy community and ecosystem supporting it would be better than just space delimited lines, or worse - groups of lines you need to deal with.
woodruffw 5y ago
I agree with you in broad strokes, but as a piece of anecdata: I've had a lot of success building tools that emit and consume JSONL[1] instead of entire JSON documents. JSONL preserves the Unix pipeline's inherently parallel design (people tend to forget this, even when waxing about the Unix philosophy!) but gives us all of the nice typing of a JSON stream. That being said, I too will take a `sed` or `awk` one-liner over some of the `jq` monstrosities that I've seen. [1]: https://jsonlines.org/ https://jsonlines.org/
- freshhawk 5y ago
  Yeah, if we can't get something like edn (json but well designed and extensible) then jsonl is at least OK. > people tend to forget this There are people with strong opinions about this that don't even understand why you need something like JSONL in the first place and I'm surprised by that. If they never use unix pipelines I don't get why they feel we want to hear their hot take about how they should be redesigned.

selfhoster11 5y ago

For the first point, there's no reason why we can't use JSON array wrapped output. In fact, we likely should use something like this for uniformity. Loading everything into memory is also not too problematic assuming a working swap space and reasonably well-architectured output schema. For the second point, whitespace sensitivity is the one mistake that greatly pisses me off with Unix. I should be able to pass arguments and filenames with as many spaces as I want. We are in the 21st century and occasionally do use spaces in filenames.

kbrazil 5y ago

I agree JSON is probably not right for every type of program output, but the age of web APIs has shown us that is probably great or adequate 90% of the time. If something is spewing out long lines of data I think JSON Lines would be a good option so you don’t need to read the whole structure into RAM. But any other structured output that has a healthy community and ecosystem supporting it would be better than just space delimited lines, or worse - groups of lines you need to deal with.

woodruffw 5y ago

I agree with you in broad strokes, but as a piece of anecdata: I've had a lot of success building tools that emit and consume JSONL[1] instead of entire JSON documents. JSONL preserves the Unix pipeline's inherently parallel design (people tend to forget this, even when waxing about the Unix philosophy!) but gives us all of the nice typing of a JSON stream. That being said, I too will take a `sed` or `awk` one-liner over some of the `jq` monstrosities that I've seen. [1]: https://jsonlines.org/ https://jsonlines.org/

freshhawk 5y ago
Yeah, if we can't get something like edn (json but well designed and extensible) then jsonl is at least OK. > people tend to forget this There are people with strong opinions about this that don't even understand why you need something like JSONL in the first place and I'm surprised by that. If they never use unix pipelines I don't get why they feel we want to hear their hot take about how they should be redesigned.