5 ms·
It's not just the grounds that they're on a system where that's all they have. The command line tools are largely written in pure C. They are blazingly fast if
by malux85 7y ago
It's not just the grounds that they're on a system where that's all they have. The command line tools are largely written in pure C. They are blazingly fast if used correctly, and robust to all sorts of failures you didn't even think of, they are ready to go and they are QUICK to both use and develop for.
Here's an example:
I was working at a consultancy one day as a data engineer, and across the desk was a junior data scientist. We were preparing for a meetup and the manager said "this website has a bunch of email addresses on it I'd love to have them in a CSV" (they were in a format where it wasn't easy to copy + paste them)
The junior data scientist said "I can do that" and the manager said "only if it takes less than 2 minutes", the junior, thinking he'd have to:
setup a python environment,
find a parsing lib,
write code to pull the page,
parse the dom,
search for the email addresses,
write them to a file,
said it could be done in an hour or two.
I viewed the code on the page, and saw the html wasn't minified, and the "a" tags were 1 per line, so I :
use wget to pull the webpage
cat webpage.html | grep 'mailto:' | cut -d '"' -f 5 | cut -d '"' -f 1 > emails.txt
^ unix wizards will tell me a better way of doing this, but it took me literally 60 seconds to go from "I wonder if" by the manager to "here's your CSV"
THATS WHY WE USE COMMAND LINE TOOLS, they are a swiss army knife
- pjc50 7y agoSee the classic of the genre, "Command-line Tools can be 235x Faster than your Hadoop Cluster" https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
- andorov 7y agoThis one too - 10 pages of Pascal to six shell commands https://leancrew.com/all-this/2011/12/more-shell-less-egg/ https://leancrew.com/all-this/2011/12/more-shell-less-egg/
- AlchemistCamp 7y ago> cat webpage.html | grep 'mailto:' | cut -d '"' -f 5 | cut -d '"' -f 1 > emails.txt This is great! I can't believe I've never run into "cut" before. In similar spots, my workflow would have been filtering the lines with grep and then opening the file in VI to finish the task.
- ptah 7y agoyip, that's why learning vi is essential for comuputational survivalism. most unix systems has it
- pjc50 7y agoAnother alternative would be to do the transformation with sed or awk, or if you require more complex regexes, perl with the "-i" and "-e (single line of code)" options. Cut has a mirror image program, "paste", but I've not yet hit a use case for it.
- ulucs 7y agoI used it once for counting n-grams while making a markov chain generator
- Thiez 7y agoIn that particular situation I imagine that a smart line of Javascript in the developer console would also have worked. Without the email file (so you must copy the result from the dev console), but it is considerably easier than using curl or wget when the page is behind a login page.
- malux85 7y agowe shared files on a mounted NAS. getting him the CSV was just a "cp" can you share the smart line of javascript? You have 60 seconds, go.
- ddkhd 7y agosomething like this: `document.getElementsByTagName('a').map(elem => elem.href)`
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- vxcwvxcw 7y agothis works at least in chrome console (to get link urls, not emails) [...($$('a'))].map(e => e.href)
- vxcwvxcw 7y ago
- JNRowe 7y agoI get that it wasn't the point of your post, but Bert Bos' html-xml-utils¹ are a great set of tools for these tasks. `hxwls` lists links, so `hxwls <URL or file> | grep mailto | ...`. There is a whole heap of tools in the package, and they're definitely worth a look if you like to poke at the web from a term. Plus they handle a lot of the nasty corner cases that will pop up if you try to treat HTML as simple ASCII text. 1. https://www.w3.org/Tools/HTML-XML-utils/ https://www.w3.org/Tools/HTML-XML-utils/
- taneq 7y agoWe played games like this back in university, doing all sorts of simple off-the-cuff data transformation and extraction. It taught me an eternal respect for the Unix command line as a quick way to hack up data into a usable shape, and it still helps me regularly to this day.
- CalRobert 7y agoIt's amazing how handy the console and presence of jquery in basically every webpage is. Got a list of expandable elements and don't feel like clicking every_single_one? $(".whatever").click() (not tested but you get the idea) does wonders.
- Pete_D 7y ago> unix wizards will tell me a better way of doing this Great story. I won't claim to be a wizard, but here's mine: grep -Eo "mailto:[^\"']*" webpage.html | sed 's/^mailto://' | sort -u The pattern means "mailto: and then everything up to a single or double quote", and the -o flag means "only show the matches". The main benefit of this approach is that it works even if you have multiple email addresses on a line.
- malux85 7y agoYou are a wizard, today I learn, thank you!
- yellowapple 7y agoOn top of the other recommendations, you could replace the separate "wget the page" and "cat it into the pipe" steps with a single "curl https://url.to/webpage.html" https://url.to/webpage.html" (assuming you have curl on your machine, which is practically guaranteed nowadays). More to the point, though: the junior data scientist might not have been entirely on the wrong foot there. There's a tradeoff between "I need to do this thing quickly" v. "I need to do this thing repeatably and robustly"; whipping up a quick shell one-liner to do it falls into the former category, and also can drive some ideas for implementing the latter category (where you'd almost certainly want to be properly parsing the HMTL, properly accounting for commas in the email addresses (and yes, email addresses can have commas per RFC 5321), etc.).
- 7thaccount 7y agoYea...knowing basic Linux commands can save you hours of writing scripts in as little as a week for me. Also it saves sanity as having implicit loops is nice.