3 ms·
FWIW that's "xsv sample", and it handles headers by default (with an option for headers-less CSV): Randomly samples CSV data uniformly using memory proport
by masklinn 8y ago
FWIW that's "xsv sample", and it handles headers by default (with an option for headers-less CSV):
Randomly samples CSV data uniformly using memory proportional to the size of
the sample.
When an index is present, this command will use random indexing if the sample
size is less than 10% of the total number of records. This allows for efficient
sampling such that the entire CSV file is not parsed.
This command is intended to provide a means to sample from a CSV data set that
is too big to fit into memory (for example, for use with commands like 'xsv
frequency' or 'xsv stats'). It will however visit every CSV record exactly
once, which is necessary to provide a uniform random sample. If you wish to
limit the number of records visited, use the 'xsv slice' command to pipe into
'xsv sample'.
Usage:
xsv sample [options] <sample-size> [<input>]
xsv sample --help
Common options:
-h, --help Display this message
-o, --output <file> Write output to <file> instead of stdout.
-n, --no-headers When set, the first row will be consider as part of
the population to sample from. (When not set, the
first row is the header row and will always appear
in the output.)
-d, --delimiter <arg> The field delimiter for reading CSV data.
Must be a single character. (default: ,)