5 ms·
A generic (partial) solution to this type of thing is just to sample a number of lines from the large input, and do the investigation on that. shuf -n 1000 fil
by ZeroGravitas 5y ago
A generic (partial) solution to this type of thing is just to sample a number of lines from the large input, and do the investigation on that.
shuf -n 1000 file
This is part of coreutils.
There's also jiq, which is a clone of jid (mentioned elsewhere) but with jq syntax
- dewey 5y agoThat would only work if it's line separated JSON though. If you cut off the first 1000 lines of a big JSON file it will be invalid.
- Someone 5y agoThen, use jq —-compact-output '.' | head -10 | foo That is also useful for grepping to filter on records of interest. jq also has --stream for handling large inputs.