7 ms·
> You will not need to open an editor and write a long script and then to have an interpreter like Node.js to run it, sometime the bash command line is just eno
by minimaxir 10y ago
> You will not need to open an editor and write a long script and then to have an interpreter like Node.js to run it, sometime the bash command line is just enough you need!
This is a bad attitude to have for working with data processing, where QA is necessary and the accuracy of the output is important. A 50 LOC scraper with comments and explicitly-defined inputs and output from functions is far preferable to a 8 LOC scraper that those without bash knowledge will be unable to parse.
And the 8 LOC bash script is not much of a time savings as this post demonstrates; you still have to check each function output manually to find which data to parse / handle edge cases.
- jc4p 10y ago100% this. Coming from someone whose made disgusting chained commands to parse access logs for relevant information, you _need_ better documentation than you can do with a single line of commands. I have so many `zgrep ... | awk ... | sort` scripts that I legitimately couldn't tell you what they do anymore, just what the correct output is. I really like the feeling of proving to myself that I know how to use my command line well, but in most cases I end up spending longer trying to remember the differences between OS X built-in `sed` and the `sed` I know and blah. Lots of wasted time. I've started keeping a "scratch" git repo with scripts for one-offs, so the next time I say "oh I want to go through this CSV and run X on each line with Y" I can look at my old code and replace the necessary parts.
- Symbiote 10y agoYou can put comments in Bash scripts, including in multi-line pipelines echo x |\ # Comment here cat | # Or like this cat An interactive shell with the option interactive_comments set (the default) ignores these comments, which is useful if you wish to copy+paste+execute bits of scripts.
- MichaelBurge 10y agoI've used Makefiles to coordinate a zoo of perl/bash scripts before. It's pretty effective.
- black_knight 10y agoMakefiles are underrated. The concept is so simple – it is basically a dependency graph with attached shell scripts – yet so powerful. Not just for building software, but also for everyday tasks where you need some files to be updated under some condition. I recommend mk (original make replacement for Plan 9, available for different OSes through Plan9Port [0,1]). It has a bit more uniform syntax, and can check that the dependency graph is wellfounded. [0] https://github.com/9fans/plan9port https://github.com/9fans/plan9port [1] https://swtch.com/plan9port/ https://swtch.com/plan9port/
- gcb0 10y agoanother advantage of make is that every host have it. like vim. mk is not there yet.
- viraptor 10y agoDid you mean "vi"? Lots of systems don't install vim by default.
- vacri 10y agoI've been getting into Makefiles more over the past year, and still a noob at it. I see them as a necessary evil - they're baroque and they hurt... and yet they make other things easier. I'm using them less to compile stuff and more to package/upload things, but they certainly have their place.
- Symbiote 10y ago> far preferable Well, that depends. Shell scripts can be commented, and they can be built progressively and interactively by building a pipeline. That's a great choice for a one-off task, and in my experience much faster than many other approaches. It also works well for tasks close to the system. For example, our users are able to download large archives of data, and we keep over 99% of such downloads indefinitely. We delete downloads > 100GB after they're more than 6 months old. With a shell script run by cron that's achieved with find + rm + curl|jq (to tell the API the download is deleted).
- minimaxir 10y agoTo clarify, by data processing I mean extract-transform-load, where the outputs of the extract/transform phase(s) may not be immediately obvious. Bash as an simple automation tool for simple tasks is fine.
- module0000 10y agoWhich company/industry do you work in, where people who would be parsing your code are unable to grok something as basic as bash?
- 0942v8653 10y agoI don't think it's really bash, more the utilities' inconsistent APIs and single-letter flags (which are not descriptive at all). Each command has its own API with different options, so it takes a lot of exposure to really learn them.
- ktRolster 10y agoIf you don't like single letter flags, most have options for longer, more descriptive flags that do the same thing. For example, you can either do: Make -I dir or Make --include-dir=dir Choose whichever one makes most sense.
- gcb0 10y agothis. anyone who uses single letter options on a script should be punished. single letters are for one time typing.
- viraptor 10y agoSingle letters are sometimes more known than their full option equivalents. Quick, without a man page, what's the short alias for `tar --get` ?
- niftich 10y agoI have no idea because I memorized 'tar zxvf' a long time ago [1] and I have to google everything else. [1] https://xkcd.com/1597/ https://xkcd.com/1597/
- 10y ago
- nostrademons 10y agoMost interesting data-analysis problems require multiple iterations. It's not a bad idea to take a couple quick exploratory passes at your data with interactive command-line tools, look at its shape, figure out what areas you need to take a closer look at, and then write real programs to work specifically with those. I've gotten surprisingly far using just curl/cat/cut/sed/awk/wc & friends. When I need to build on top of that, I go write a real program, but the UNIX-fu tells me what I need to build on top of.
- LukeShu 10y agoWhat? For a one-off, one-time "I wonder who has the most medals" curiosity script? > 8 LOC scraper that those without bash knowledge will be unable to parse. And someone without knowledge of Node.js will be unable to parse the 50-line JavaScript monstrosity. > And the 8 LOC bash script is not much of a time savings as this post demonstrates; you still have to check each function output manually to find which data to parse / handle edge cases. It only took that long because he detailed every step as a tutorial. If you have basic literacy of the shell, it's no time at all. To refute you, I decided to solve it myself, using only the "hint" at the beginning of the article to use ?action=raw to work with the source wikitext (I had not yet read the rest of the article; I had not seen his solution). It took me literally 2 minutes: (Setting in url='https://en.wikipedia.org/wiki/List_of_Olympic_medalists_in_judo?action=raw' https://en.wikipedia.org/wiki/List_of_Olympic_medalists_in_j... for readability on HN) [2016-08-15 17:52] curl -s ${url} [2016-08-15 17:52] curl -s ${url}|grep flagIOCmedalist [2016-08-15 17:52] curl -s ${url}|grep -o '{{flagIOCmedalist.*}}' [2016-08-15 17:53] curl -s ${url}|grep -o '{{flagIOCmedalist.*}}'|cut -d'|' -f2|sed 's/[[\]]//g' [2016-08-15 17:53] curl -s ${url}|grep -o '{{flagIOCmedalist.*}}'|cut -d'|' -f2 [2016-08-15 17:54] curl -s ${url}|grep -o '{{flagIOCmedalist.*}}'|cut -d'|' -f2|sed 's/\(\[\|\]\)//g' [2016-08-15 17:54] curl -s ${url}|grep -o '{{flagIOCmedalist.*}}'|cut -d'|' -f2|sed 's/\(\[\|\]\)//g'|sort |uniq -c [2016-08-15 17:54] curl -s ${url}|grep -o '{{flagIOCmedalist.*}}'|cut -d'|' -f2|sed 's/\(\[\|\]\)//g'|sort |uniq -c|sort -n You can see the only place I fumbled a bit was with escaping brackets inside of brackets in sed, which is admittedly a little wonky. Sure, for something that might need to run repeatedly, it probably doesn't handle future edge cases that might arise. But it's not production, it's a one-off exploring the data script. Not every one-off program you write needs to be production quality. And being worried about "those without bash knowledge"... don't be afraid to use your operating system! That said, this line of solutions has a big edge case: it relies on the editors of Wikipedia being consistent and formatting each row as one line in the source. If I were to solve this totally on my own, choosing to ignore the "hint", and work with the rendered HTML, and just do most of it with a nokogiri one-liner. Knowing that there's an ?action=something to get just the page HTML without the navigation and such. I spent about 3 minutes finding the "render" action (documented here: https://www.mediawiki.org/wiki/Manual:Parameters_to_index.php#Actions https://www.mediawiki.org/wiki/Manual:Parameters_to_index.ph... ). Then anther 3-ish minutes poking around the DOM in my browser to get an idea of what I'm working with, and visually inspecting the layout of the article; each cell with a medalist has two links, the first to the medalist, and the second to the country. Then it took me a whopping 5 minutes to hammer out the rest of the one-liner. (Similarly, url='https://en.wikipedia.org/wiki/List_of_Olympic_medalists_in_judo?action=render' https://en.wikipedia.org/wiki/List_of_Olympic_medalists_in_j... for HN readability) [2016-08-15 18:14] curl -s ${url} [2016-08-15 18:15] curl -s ${url}|nokogiri -e '$_.css("table.wikitable td a:first-child").each{|a| puts a.text}' [2016-08-15 18:16] curl -s ${url}|nokogiri -e '$_.css("table.wikitable td a:first-child").each{|a| puts a.text}'|grep -vE '^[0-9]{4} ' [2016-08-15 18:17] curl -s ${url}|nokogiri -e '$_.css("table.wikitable td a:first-child").each{|a| puts a.text.sub!("\n", " ")}' [2016-08-15 18:18] curl -s ${url}|nokogiri -e '$_.css("table.wikitable td a:first-child").each{|a| puts a.text.sub("\n", " ")}' [2016-08-15 18:18] curl -s ${url}|nokogiri -e '$_.css("table.wikitable td a:first-child").each{|a| puts a.text}'|grep -vE -e '^[0-9]{4} ' -e '^details$' [2016-08-15 18:19] curl -s ${url}|nokogiri -e '$_.css("table.wikitable td a:first-child").each{|a| puts a.text}'|grep -vE -e '^[0-9]{4} ' -e '^details$'|sort |uniq -c|sort -n And most of it was fumbling around with thinking that the year/details links were one link with a newline in them, when they are in fact two separate links. For a grand total of 11 minutes; generously. My point is: Don't be afraid to play around with your tools and data! Have fun! Not everything needs to be production quality! And being written primarily in bash doesn't necessarily mean that it isn't production quality!