4 ms·

I would be a little bit shocked if any of the data scientists at my day job didn't know all seven of these, so, I guess that's an accurate title.

by fwip 8y ago

eesmith 8y ago
I didn't know about shuf because it's a GNU specific utility, and not installed on my Mac by default. Homebrew installs the GNU Core Utilities with the g- prefix, and checking now I see it's available as gshuf. I see a couple of problems with how it's used in this essay: cat big_csv.csv | shuf | head -n 50 > sample_from_big_csv.csv Since shuf's -n option is "output at most COUNT lines", there should be no need for the head: cat big_csv.csv | shuf -n 50 > sample_from_big_csv.csv In principle this should be faster because there's less I/O, and it should take less memory if shuf implements the -n option using something like reservoir sampling. (EDIT: it does - https://git.savannah.gnu.org/cgit/coreutils.git/tree/src/shuf.c#n175 https://git.savannah.gnu.org/cgit/coreutils.git/tree/src/shu... ) Also, since shuf takes a filename, it could be: shuf -n 50 < big_csv.csv> sample_from_big_csv.csv However, in terms of data science, don't most CSV files contain a header? Most of mine do, and the example 'data.csv' has the header "var_x". Using "shuf" means there's a very high chance that the sampled CSV file either won't include the header line, or will have it somewhere other than the first line. Doesn't that mean that most data scientists will rarely use 'shuf' for sampling from CSV files?
- sk5t 8y ago
  There is no standard for CSV per se, although experience suggests most files have it. Use head/tail as needed to preserve or avoid it in pipelines. The GNU utils are usually the way to go; the Mac/Darwin/BSD variants have some weird quirks and usually aren't worth the bother to fight with. awk and sort are notably deficient. Ease off the "data science" a tad--this is just scripting, a perfectly honorable pastime! Why else would GNU coreutils exist in the first place?
  - eesmith 8y ago
    There is a CSV standard, RFC 4180. There are also many variants. I used products which specifically say that follow that RFC. It's awkward to preserve a header using shuf, even with head/tail. Here's what I came up with: head -1 x.csv && (awk 'NR>1' x.csv | shuf -n 5) My point is that "shuf" is GNU specific, while the author said it was a Unix tool like the other ones. Linux != Unix. Also, there are no Mac/Darwin/BSD variants of shuf. "Ease off"? Why? The term "data scientist" appear in the title and each of the first three paragraphs, and the comment I replied to was "I would be a little bit shocked if any of the data scientists at my day job didn't know all seven of these". I want to know why data scientists who work with CSV files should be using shuf and not some more appropriate tool like csvkit. That is, the entire topic is about scripting as applied to data science, not "just scripting".
    - fiddlerwoaroof 8y ago
      I do something like: < foo.csv (read -r head; echo “$head”; shuf -n 50) If you do this a lot, you could combine the read/echo into a single shell function.
      - eesmith 8y ago
        Or write a Python program. But a quick test shows that CPython (using a pure Python reservoir sampling) is 1/4th the speed of shuf.
    - sk5t 8y ago
      RFC 4180 is really a well-intentioned attempt to document popular ways CSV is used in the wild, but it does not set a standard. There is no such thing as CSV well-formedness in the way that exists for XML. Can a CSV field value span lines or not? How are linebreaks encoded within a field? Can backslash escape quote (or other) characters, or is the double-doublequote approach contemplated in the RFC proper? How is one to know definitively if the first line is a header or a data row? What about comments? My beef with "data science" is that this is not science--it's even less science than political science--it's munging.
- bakul 8y ago
  > don't most CSV files contain a header? tail +2
  - eesmith 8y ago
    D'oh! I forgot that one, and gave an awk description instead.
- masklinn 8y ago
  FWIW that's "xsv sample", and it handles headers by default (with an option for headers-less CSV): Randomly samples CSV data uniformly using memory proportional to the size of the sample. When an index is present, this command will use random indexing if the sample size is less than 10% of the total number of records. This allows for efficient sampling such that the entire CSV file is not parsed. This command is intended to provide a means to sample from a CSV data set that is too big to fit into memory (for example, for use with commands like 'xsv frequency' or 'xsv stats'). It will however visit every CSV record exactly once, which is necessary to provide a uniform random sample. If you wish to limit the number of records visited, use the 'xsv slice' command to pipe into 'xsv sample'. Usage: xsv sample [options] <sample-size> [<input>] xsv sample --help Common options: -h, --help Display this message -o, --output <file> Write output to <file> instead of stdout. -n, --no-headers When set, the first row will be consider as part of the population to sample from. (When not set, the first row is the header row and will always appear in the output.) -d, --delimiter <arg> The field delimiter for reading CSV data. Must be a single character. (default: ,)
- fwip 8y ago
  Personally, I'd use another tool to remove the header before sending it to shuf. As an example: cat myfile | tail -n +2 | shuf -n 50
snarfy 8y ago
I assumed the bc command would be listed.
quickthrower2 8y ago
I’m a windows .net dev and I know 6 of them. It’s quite useful to run these on windows using Cygwin. Grep I use a lot the others not so much.
dontbenebby 8y ago
I know most of those, but I've never used awk. One command I didn't see mentioned that was surprising was cut[1], especially the -d flag. (Maybe it's less necessary if you know awk?) It's really great for cleaning up the output of poorly structured, wordy scripts. It's also a quick and dirty way to generate csvs with the –output-delimiter flag. [1]https://www.geeksforgeeks.org/cut-command-linux-examples/ https://www.geeksforgeeks.org/cut-command-linux-examples/

I would be a little bit shocked if any of the data scientists at my day job didn't know all seven of these, so, I guess that's an accurate title.

by fwip 8y ago

I would be a little bit shocked if any of the data scientists at my day job didn't know all seven of these, so, I guess that's an accurate title.

eesmith 8y ago

I didn't know about shuf because it's a GNU specific utility, and not installed on my Mac by default. Homebrew installs the GNU Core Utilities with the g- prefix, and checking now I see it's available as gshuf. I see a couple of problems with how it's used in this essay: cat big_csv.csv | shuf | head -n 50 > sample_from_big_csv.csv Since shuf's -n option is "output at most COUNT lines", there should be no need for the head: cat big_csv.csv | shuf -n 50 > sample_from_big_csv.csv In principle this should be faster because there's less I/O, and it should take less memory if shuf implements the -n option using something like reservoir sampling. (EDIT: it does - https://git.savannah.gnu.org/cgit/coreutils.git/tree/src/shuf.c#n175 https://git.savannah.gnu.org/cgit/coreutils.git/tree/src/shu... ) Also, since shuf takes a filename, it could be: shuf -n 50 < big_csv.csv> sample_from_big_csv.csv However, in terms of data science, don't most CSV files contain a header? Most of mine do, and the example 'data.csv' has the header "var_x". Using "shuf" means there's a very high chance that the sampled CSV file either won't include the header line, or will have it somewhere other than the first line. Doesn't that mean that most data scientists will rarely use 'shuf' for sampling from CSV files?

sk5t 8y ago
There is no standard for CSV per se, although experience suggests most files have it. Use head/tail as needed to preserve or avoid it in pipelines. The GNU utils are usually the way to go; the Mac/Darwin/BSD variants have some weird quirks and usually aren't worth the bother to fight with. awk and sort are notably deficient. Ease off the "data science" a tad--this is just scripting, a perfectly honorable pastime! Why else would GNU coreutils exist in the first place?
- eesmith 8y ago
  There is a CSV standard, RFC 4180. There are also many variants. I used products which specifically say that follow that RFC. It's awkward to preserve a header using shuf, even with head/tail. Here's what I came up with: head -1 x.csv && (awk 'NR>1' x.csv | shuf -n 5) My point is that "shuf" is GNU specific, while the author said it was a Unix tool like the other ones. Linux != Unix. Also, there are no Mac/Darwin/BSD variants of shuf. "Ease off"? Why? The term "data scientist" appear in the title and each of the first three paragraphs, and the comment I replied to was "I would be a little bit shocked if any of the data scientists at my day job didn't know all seven of these". I want to know why data scientists who work with CSV files should be using shuf and not some more appropriate tool like csvkit. That is, the entire topic is about scripting as applied to data science, not "just scripting".
  - fiddlerwoaroof 8y ago
    I do something like: < foo.csv (read -r head; echo “$head”; shuf -n 50) If you do this a lot, you could combine the read/echo into a single shell function.
    - eesmith 8y ago
      Or write a Python program. But a quick test shows that CPython (using a pure Python reservoir sampling) is 1/4th the speed of shuf.
  - sk5t 8y ago
    RFC 4180 is really a well-intentioned attempt to document popular ways CSV is used in the wild, but it does not set a standard. There is no such thing as CSV well-formedness in the way that exists for XML. Can a CSV field value span lines or not? How are linebreaks encoded within a field? Can backslash escape quote (or other) characters, or is the double-doublequote approach contemplated in the RFC proper? How is one to know definitively if the first line is a header or a data row? What about comments? My beef with "data science" is that this is not science--it's even less science than political science--it's munging.
bakul 8y ago
> don't most CSV files contain a header? tail +2
- eesmith 8y ago
  D'oh! I forgot that one, and gave an awk description instead.
masklinn 8y ago
FWIW that's "xsv sample", and it handles headers by default (with an option for headers-less CSV): Randomly samples CSV data uniformly using memory proportional to the size of the sample. When an index is present, this command will use random indexing if the sample size is less than 10% of the total number of records. This allows for efficient sampling such that the entire CSV file is not parsed. This command is intended to provide a means to sample from a CSV data set that is too big to fit into memory (for example, for use with commands like 'xsv frequency' or 'xsv stats'). It will however visit every CSV record exactly once, which is necessary to provide a uniform random sample. If you wish to limit the number of records visited, use the 'xsv slice' command to pipe into 'xsv sample'. Usage: xsv sample [options] <sample-size> [<input>] xsv sample --help Common options: -h, --help Display this message -o, --output <file> Write output to <file> instead of stdout. -n, --no-headers When set, the first row will be consider as part of the population to sample from. (When not set, the first row is the header row and will always appear in the output.) -d, --delimiter <arg> The field delimiter for reading CSV data. Must be a single character. (default: ,)
fwip 8y ago
Personally, I'd use another tool to remove the header before sending it to shuf. As an example: cat myfile | tail -n +2 | shuf -n 50

snarfy 8y ago

I assumed the bc command would be listed.

quickthrower2 8y ago

I’m a windows .net dev and I know 6 of them. It’s quite useful to run these on windows using Cygwin. Grep I use a lot the others not so much.

dontbenebby 8y ago

I know most of those, but I've never used awk. One command I didn't see mentioned that was surprising was cut[1], especially the -d flag. (Maybe it's less necessary if you know awk?) It's really great for cleaning up the output of poorly structured, wordy scripts. It's also a quick and dirty way to generate csvs with the –output-delimiter flag. [1]https://www.geeksforgeeks.org/cut-command-linux-examples/ https://www.geeksforgeeks.org/cut-command-linux-examples/