3 ms·
There is no standard for CSV per se, although experience suggests most files have it. Use head/tail as needed to preserve or avoid it in pipelines. The GNU ut
by sk5t 8y ago
There is no standard for CSV per se, although experience suggests most files have it. Use head/tail as needed to preserve or avoid it in pipelines.
The GNU utils are usually the way to go; the Mac/Darwin/BSD variants have some weird quirks and usually aren't worth the bother to fight with. awk and sort are notably deficient.
Ease off the "data science" a tad--this is just scripting, a perfectly honorable pastime! Why else would GNU coreutils exist in the first place?
- eesmith 8y agoThere is a CSV standard, RFC 4180. There are also many variants. I used products which specifically say that follow that RFC. It's awkward to preserve a header using shuf, even with head/tail. Here's what I came up with: head -1 x.csv && (awk 'NR>1' x.csv | shuf -n 5) My point is that "shuf" is GNU specific, while the author said it was a Unix tool like the other ones. Linux != Unix. Also, there are no Mac/Darwin/BSD variants of shuf. "Ease off"? Why? The term "data scientist" appear in the title and each of the first three paragraphs, and the comment I replied to was "I would be a little bit shocked if any of the data scientists at my day job didn't know all seven of these". I want to know why data scientists who work with CSV files should be using shuf and not some more appropriate tool like csvkit. That is, the entire topic is about scripting as applied to data science, not "just scripting".
- fiddlerwoaroof 8y agoI do something like: < foo.csv (read -r head; echo “$head”; shuf -n 50) If you do this a lot, you could combine the read/echo into a single shell function.
- eesmith 8y agoOr write a Python program. But a quick test shows that CPython (using a pure Python reservoir sampling) is 1/4th the speed of shuf.
- sk5t 8y agoRFC 4180 is really a well-intentioned attempt to document popular ways CSV is used in the wild, but it does not set a standard. There is no such thing as CSV well-formedness in the way that exists for XML. Can a CSV field value span lines or not? How are linebreaks encoded within a field? Can backslash escape quote (or other) characters, or is the double-doublequote approach contemplated in the RFC proper? How is one to know definitively if the first line is a header or a data row? What about comments? My beef with "data science" is that this is not science--it's even less science than political science--it's munging.
- eesmith 8y agoshrug Sure, yes, I pointed out there are many variants. But you said there wasn't a standard when there is. You meant to say that there is no widely used consistent CSV format, I think, which is a bit different. RFC 4180 says a CSV field can span multiple lines. Section 2.6: > Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes. You asked "Can backslash escape quote (or other) characters". 4180 says "no", and that double-double quoting is used. The grammar in 2.7 is: > escaped = DQUOTE *(TEXTDATA / COMMA / CR / LF / 2DQUOTE) DQUOTE The RFC says that the MIME type should use header=present or header=absent to indicate if the first line is a header. "Implementors choosing not to use this parameter must make their own decisions as to whether the header line is present or absent." I have never seen a CSV file with comments. Note also that the Python csv module, which supports a large number of CSV dialects, also does not support comments. While you have a beef, that doesn't change that you wanted to drag me off topic (from "data science" to "just scripting"), when my point was that that example wasn't a good data science example in the first place. It also isn't a good scripting example.
- deleted 8y ago[deleted]