3 ms·
I didn't know about shuf because it's a GNU specific utility, and not installed on my Mac by default. Homebrew installs the GNU Core Utilities with the g- pref
by eesmith 8y ago
I didn't know about shuf because it's a GNU specific utility, and not installed on my Mac by default.
Homebrew installs the GNU Core Utilities with the g- prefix, and checking now I see it's available as gshuf.
I see a couple of problems with how it's used in this essay:
cat big_csv.csv | shuf | head -n 50 > sample_from_big_csv.csv
Since shuf's -n option is "output at most COUNT lines", there should be no need for the head:
cat big_csv.csv | shuf -n 50 > sample_from_big_csv.csv
In principle this should be faster because there's less I/O, and it should take less memory if shuf implements the -n option using something like reservoir sampling. (EDIT: it does - https://git.savannah.gnu.org/cgit/coreutils.git/tree/src/shuf.c#n175 https://git.savannah.gnu.org/cgit/coreutils.git/tree/src/shu... )
Also, since shuf takes a filename, it could be:
shuf -n 50 < big_csv.csv> sample_from_big_csv.csv
However, in terms of data science, don't most CSV files contain a header? Most of mine do, and the example 'data.csv' has the header "var_x".
Using "shuf" means there's a very high chance that the sampled CSV file either won't include the header line, or will have it somewhere other than the first line.
Doesn't that mean that most data scientists will rarely use 'shuf' for sampling from CSV files?
- sk5t 8y agoThere is no standard for CSV per se, although experience suggests most files have it. Use head/tail as needed to preserve or avoid it in pipelines. The GNU utils are usually the way to go; the Mac/Darwin/BSD variants have some weird quirks and usually aren't worth the bother to fight with. awk and sort are notably deficient. Ease off the "data science" a tad--this is just scripting, a perfectly honorable pastime! Why else would GNU coreutils exist in the first place?
- eesmith 8y agoThere is a CSV standard, RFC 4180. There are also many variants. I used products which specifically say that follow that RFC. It's awkward to preserve a header using shuf, even with head/tail. Here's what I came up with: head -1 x.csv && (awk 'NR>1' x.csv | shuf -n 5) My point is that "shuf" is GNU specific, while the author said it was a Unix tool like the other ones. Linux != Unix. Also, there are no Mac/Darwin/BSD variants of shuf. "Ease off"? Why? The term "data scientist" appear in the title and each of the first three paragraphs, and the comment I replied to was "I would be a little bit shocked if any of the data scientists at my day job didn't know all seven of these". I want to know why data scientists who work with CSV files should be using shuf and not some more appropriate tool like csvkit. That is, the entire topic is about scripting as applied to data science, not "just scripting".
- fiddlerwoaroof 8y agoI do something like: < foo.csv (read -r head; echo “$head”; shuf -n 50) If you do this a lot, you could combine the read/echo into a single shell function.
- eesmith 8y agoOr write a Python program. But a quick test shows that CPython (using a pure Python reservoir sampling) is 1/4th the speed of shuf.
- sk5t 8y agoRFC 4180 is really a well-intentioned attempt to document popular ways CSV is used in the wild, but it does not set a standard. There is no such thing as CSV well-formedness in the way that exists for XML. Can a CSV field value span lines or not? How are linebreaks encoded within a field? Can backslash escape quote (or other) characters, or is the double-doublequote approach contemplated in the RFC proper? How is one to know definitively if the first line is a header or a data row? What about comments? My beef with "data science" is that this is not science--it's even less science than political science--it's munging.
- eesmith 8y agoshrug Sure, yes, I pointed out there are many variants. But you said there wasn't a standard when there is. You meant to say that there is no widely used consistent CSV format, I think, which is a bit different. RFC 4180 says a CSV field can span multiple lines. Section 2.6: > Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes. You asked "Can backslash escape quote (or other) characters". 4180 says "no", and that double-double quoting is used. The grammar in 2.7 is: > escaped = DQUOTE *(TEXTDATA / COMMA / CR / LF / 2DQUOTE) DQUOTE The RFC says that the MIME type should use header=present or header=absent to indicate if the first line is a header. "Implementors choosing not to use this parameter must make their own decisions as to whether the header line is present or absent." I have never seen a CSV file with comments. Note also that the Python csv module, which supports a large number of CSV dialects, also does not support comments. While you have a beef, that doesn't change that you wanted to drag me off topic (from "data science" to "just scripting"), when my point was that that example wasn't a good data science example in the first place. It also isn't a good scripting example.
- deleted 8y ago[deleted]
- bakul 8y ago> don't most CSV files contain a header? tail +2
- eesmith 8y agoD'oh! I forgot that one, and gave an awk description instead.
- masklinn 8y agoFWIW that's "xsv sample", and it handles headers by default (with an option for headers-less CSV): Randomly samples CSV data uniformly using memory proportional to the size of the sample. When an index is present, this command will use random indexing if the sample size is less than 10% of the total number of records. This allows for efficient sampling such that the entire CSV file is not parsed. This command is intended to provide a means to sample from a CSV data set that is too big to fit into memory (for example, for use with commands like 'xsv frequency' or 'xsv stats'). It will however visit every CSV record exactly once, which is necessary to provide a uniform random sample. If you wish to limit the number of records visited, use the 'xsv slice' command to pipe into 'xsv sample'. Usage: xsv sample [options] <sample-size> [<input>] xsv sample --help Common options: -h, --help Display this message -o, --output <file> Write output to <file> instead of stdout. -n, --no-headers When set, the first row will be consider as part of the population to sample from. (When not set, the first row is the header row and will always appear in the output.) -d, --delimiter <arg> The field delimiter for reading CSV data. Must be a single character. (default: ,)
- fwip 8y agoPersonally, I'd use another tool to remove the header before sending it to shuf. As an example: cat myfile | tail -n +2 | shuf -n 50