7 ms·
ls | grep '.csv$' | xargs cat | grep 'cake' | cut -d, -f2,3 > cakes.csv That's quite a few antipatterns in one go. Unless you have a bajillion files the `xargs
by l0b0 5y ago
ls | grep '.csv$' | xargs cat | grep 'cake' | cut -d, -f2,3 > cakes.csv
That's quite a few antipatterns in one go. Unless you have a bajillion files the `xargs` is unnecessary, the `cat` and `ls` are unnecessary (and `ls` in shell scripts is a whole class of antipatterns by itself). You might want to use something like this instead:
grep cake *.csv | cut -d, -f2,3 > cakes.csv
- XorNot 5y agoI'd go further and say don't parse CSV with plaintext tools because it's barely a plaintext format. Use a CSV library and save yourself heartache when someone drops a quoted string in somewhere.
- oblio 5y agoI don't understand why you're being downvoted. Parsing CSV with simple text-oriented tools is bad of an idea as parsing HTML with regexps.
- devenvdev 5y agoMost likely because it contradicts most people's experience, some CSVs can't be parsed with cli tools, but most of them can be, and it's much easier than writing code that does the same. So what the parent commenter says is true, just not pragmatic.
- oblio 5y agoIt is pragmatic with a minor tweak. https://csvkit.readthedocs.io/en/latest/ https://csvkit.readthedocs.io/en/latest/ https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv Heck, even sqlite has some stuff to help with processing CSV files: https://www.sqlite.org/csv.html https://www.sqlite.org/csv.html
- philwelch 5y agocsvkit and SQLite were always too slow whenever I tried to use them for this sort of thing. I don’t remember if I tried xsv. I do think I ended up converting most of my CSVs to TSV, or exporting them as TSV in the first place.
- ryanianian 5y agoThis isn't parsing CSV, it's generating it. That's not nearly as fraught.
- philwelch 5y agoIn my experience doing this kind of thing, I usually use TSV instead of CSV and end up using Unix tools anyway. None of the CSV tools keep up in terms of performance.
- deleted 5y ago[deleted]
- jrm4 5y agoI really wish people would mostly stop doing this type of utterly and completely necessary "correction." It really misses the point, and exemplifies nearly the exact opposite of the point and value the original article is expressing. This is a tool used to accomplish a thing, and Unix tools can be used to accomplish things in many different ways. This is like complaining about, I don't know, using a metric-labeled screwdriver on an imperial-measured screw that still gets the job done exactly as needed. Cut it out.
- cuu508 5y agoThis is pointing out you can simplify the drill+pliers+plunger+screwdriver contraption to just screwdriver.
- DarylZero 5y ago> using a metric-labeled screwdriver on an imperial-measured screw that still gets the job done exactly as needed This is super ignorant. You risk stripping the screw and getting yourself into a frustrating screw extraction job. Just because you lived, doesn't mean it was safe.
- jrm4 5y agoSure, let's beat up this metaphor. I would say the ignorant one of us is the one who isn't taking into account the nature of the job being done? What makes you think that a) my task is life or death and also b) I'm not equipped to figure out how dangerous it is? To pull out the camera. I'd like to spread Unix tools to everyone, or at least "more people than use them now." I understand though, for a lot of people, they'd rather gatekeep.
- DarylZero 5y agoIt doesn't matter whether it's life or death. If you end up having to extract that screw it's going to be a big headache. You just shouldn't use an "almost"-fitting screwdriver or wrench. I'm not trying to "gatekeep." I'm not trying to talk about the shell metaphorically. I'm literally talking about screws.
- mdoms 5y agoIt will also fall flat on its face for CSV files that contain values which contain escaped commas.
- darrenf 5y agoAlmost, but not quite :) $ ls | grep '.csv$' | xargs cat | grep 'cake' | cut -d, -f2,3 cake cake cake $ grep cake *.csv | cut -d, -f2,3 bar.csv:cake foo.csv:cake quux.csv:cake `grep -h` will do the trick though
- darrenf 5y agoI'm surprised, hours later, that no-one has pointed out my error here! So I'll do it myself - upon revisiting it, I realised my input files weren't CSVs with the number of fields required for `cut` to do the intended thing. Had I made them so, the `-h` isn't required after all. My bad :)
- devenvdev 5y agoYes! Agree. This post is more educational than practical (albeit the title). You need to start somewhere and know the basics to understand these details and caveats to feel what quality code means in this context. I chose to use the long, redundant version to show that chaining a gazillion different commands is ok.
- gbrown_ 5y agoI certainly agree with your points, though the original task is a textbook example of where awk shines. awk 'BEGIN {FS=","} /cake/ {print $2, $3}' *.csv > cakes.csv Unsurprisingly I disagree with the post's description of awk being an "advanced command".
- nickjj 5y ago> Unsurprisingly I disagree with the post's description of awk being an "advanced command". I think it can be pretty advanced, for me awk is one of those tools where I still feel like I need to write a paragraph of comments to explain 1 line of code. For example: https://github.com/nickjj/invoice/blob/75660dce5a29ceb4e47a6cea4a574b45d15548bb/invoice#L11-L30 https://github.com/nickjj/invoice/blob/75660dce5a29ceb4e47a6... Keep in mind I don't really "know" awk. I cobbled that together from a few examples. It will convert times formatted like "2h 30m", "150m" or "2:30" into 2.50. There's a bunch of examples in the test file. NOTE: I wrote that script 2.5 years ago and I know there's questionable patterns in other areas of the script that's not highlighted like using a bunch of separate echo calls instead of a heredoc. Shell scripting is really fun and efficient. I use it all the time for a variety of things.
- nicce 5y agoawk is Turing-complete, so if that is not enough to describe ”advanced” tool, then I don’t know what it is.
- cross 5y agoSo is sed.
- vlovich123 5y agoI’d classify basic search/replace functionality of sed as “basic” - you see such scripts frequently enough in shell scripts. Same for awk and “extract fields 1 and 3”. Anything beyond that quickly escalates into intermediate and advanced usages. Often you can find an SO post for lots of intermediate usages (where you can just include it as a link saying “I’m trying to do X”) but advanced usages that require knowledge of that are best avoided unless you are investing in sed/awk expertise in your team. Usually choosing something like Python, Ruby, Node or even Perl can provide better value because all of those languages can solve the same problem and solve other problems those tools can’t.