15 ms·
Computational survivalist
- malux85 7y agoIt's not just the grounds that they're on a system where that's all they have. The command line tools are largely written in pure C. They are blazingly fast if used correctly, and robust to all sorts of failures you didn't even think of, they are ready to go and they are QUICK to both use and develop for. Here's an example: I was working at a consultancy one day as a data engineer, and across the desk was a junior data scientist. We were preparing for a meetup and the manager said "this website has a bunch of email addresses on it I'd love to have them in a CSV" (they were in a format where it wasn't easy to copy + paste them) The junior data scientist said "I can do that" and the manager said "only if it takes less than 2 minutes", the junior, thinking he'd have to: setup a python environment, find a parsing lib, write code to pull the page, parse the dom, search for the email addresses, write them to a file, said it could be done in an hour or two. I viewed the code on the page, and saw the html wasn't minified, and the "a" tags were 1 per line, so I : use wget to pull the webpage cat webpage.html | grep 'mailto:' | cut -d '"' -f 5 | cut -d '"' -f 1 > emails.txt ^ unix wizards will tell me a better way of doing this, but it took me literally 60 seconds to go from "I wonder if" by the manager to "here's your CSV" THATS WHY WE USE COMMAND LINE TOOLS, they are a swiss army knife
- pjc50 7y agoSee the classic of the genre, "Command-line Tools can be 235x Faster than your Hadoop Cluster" https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
- andorov 7y agoThis one too - 10 pages of Pascal to six shell commands https://leancrew.com/all-this/2011/12/more-shell-less-egg/ https://leancrew.com/all-this/2011/12/more-shell-less-egg/
- AlchemistCamp 7y ago> cat webpage.html | grep 'mailto:' | cut -d '"' -f 5 | cut -d '"' -f 1 > emails.txt This is great! I can't believe I've never run into "cut" before. In similar spots, my workflow would have been filtering the lines with grep and then opening the file in VI to finish the task.
- ptah 7y agoyip, that's why learning vi is essential for comuputational survivalism. most unix systems has it
- pjc50 7y agoAnother alternative would be to do the transformation with sed or awk, or if you require more complex regexes, perl with the "-i" and "-e (single line of code)" options. Cut has a mirror image program, "paste", but I've not yet hit a use case for it.
- ulucs 7y agoI used it once for counting n-grams while making a markov chain generator
- Thiez 7y agoIn that particular situation I imagine that a smart line of Javascript in the developer console would also have worked. Without the email file (so you must copy the result from the dev console), but it is considerably easier than using curl or wget when the page is behind a login page.
- malux85 7y agowe shared files on a mounted NAS. getting him the CSV was just a "cp" can you share the smart line of javascript? You have 60 seconds, go.
- ddkhd 7y agosomething like this: `document.getElementsByTagName('a').map(elem => elem.href)`
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- vxcwvxcw 7y agothis works at least in chrome console (to get link urls, not emails) [...($$('a'))].map(e => e.href)
- vxcwvxcw 7y ago
- JNRowe 7y agoI get that it wasn't the point of your post, but Bert Bos' html-xml-utils¹ are a great set of tools for these tasks. `hxwls` lists links, so `hxwls <URL or file> | grep mailto | ...`. There is a whole heap of tools in the package, and they're definitely worth a look if you like to poke at the web from a term. Plus they handle a lot of the nasty corner cases that will pop up if you try to treat HTML as simple ASCII text. 1. https://www.w3.org/Tools/HTML-XML-utils/ https://www.w3.org/Tools/HTML-XML-utils/
- taneq 7y agoWe played games like this back in university, doing all sorts of simple off-the-cuff data transformation and extraction. It taught me an eternal respect for the Unix command line as a quick way to hack up data into a usable shape, and it still helps me regularly to this day.
- CalRobert 7y agoIt's amazing how handy the console and presence of jquery in basically every webpage is. Got a list of expandable elements and don't feel like clicking every_single_one? $(".whatever").click() (not tested but you get the idea) does wonders.
- Pete_D 7y ago> unix wizards will tell me a better way of doing this Great story. I won't claim to be a wizard, but here's mine: grep -Eo "mailto:[^\"']*" webpage.html | sed 's/^mailto://' | sort -u The pattern means "mailto: and then everything up to a single or double quote", and the -o flag means "only show the matches". The main benefit of this approach is that it works even if you have multiple email addresses on a line.
- malux85 7y agoYou are a wizard, today I learn, thank you!
- yellowapple 7y agoOn top of the other recommendations, you could replace the separate "wget the page" and "cat it into the pipe" steps with a single "curl https://url.to/webpage.html" https://url.to/webpage.html" (assuming you have curl on your machine, which is practically guaranteed nowadays). More to the point, though: the junior data scientist might not have been entirely on the wrong foot there. There's a tradeoff between "I need to do this thing quickly" v. "I need to do this thing repeatably and robustly"; whipping up a quick shell one-liner to do it falls into the former category, and also can drive some ideas for implementing the latter category (where you'd almost certainly want to be properly parsing the HMTL, properly accounting for commas in the email addresses (and yes, email addresses can have commas per RFC 5321), etc.).
- 7thaccount 7y agoYea...knowing basic Linux commands can save you hours of writing scripts in as little as a week for me. Also it saves sanity as having implicit loops is nice.
- Udo 7y ago> "command line tools on the grounds that someday they may be in an environment where that’s all they have. I think of this as a sort of computational survivalism." I don't think this is a helpful characterization, because this invokes post-apocalyptic associations of a world where there are somehow only text terminals left. In the article itself, the author goes on to describe a scenario where he used generic text/command line tools to achieve functionality that had not been explicitly granted to him by a restrictive software policy at work. As opposed to survivalism, I think it's more fitting to describe it as user empowerment. I'd argue that the premier usage scenario for these tools is not as a fallback in hard times, instead it's an important type of toolset integral to the craft.
- teekert 7y agoI hate this attitude by clients/employers. "I have a job for you, here are the tools you may use" ... Gives me a Windows 10 laptop and the only way to get into the network is some Windows only Cisco vpn app. So here I am, expressing all my enthusiasm, all my creativity through Windows on an HP (1280x768 display :S) which I consider to be a very narrow channel for my creativity and it kills my enthusiasm. I want a self chosen laptop that runs Linux and a distro chosen by me. And I want a nice Logitech keyboard/mouse. I want my tools to serve me, not the IT department. I mean, even my friend the electrician gets money ech year to buy tools. And he buys Makita and DeWalt, not that 10 euro for 50 tools package at Ikea. Yeah I'm not forced to work there, anyone saying that is right. And I dream of starting my own business just because of this.
- numlock86 7y agoWhile I agree up to a certain level, this opinion needs to stay within its limits. I like the analogy of a photographer: Talents aside, a pro will always make better pictures using a cheap low-budget camera than any amateur will be able to do with the newest and best camera available. IMHO the same holds true for any IT related job.
- varjag 7y ago> Talents aside, a pro will always make better pictures using a cheap low-budget camera than any amateur will be able to do with the newest and best camera available. Unfortunately it doesn't work like this. Photography, as a low threshold occupation has tons of professional hacks who are bad with any equipment. At the same time there are countless amateurs who are top talent. And it's not just a recent trend. Aside from photojournalism, formal portraiture and weddings, photography was always dominated and advanced by talented amateurs. There's no such thing as "professional landscape photographer" for example, even if some of them are able to make a modest living off their works after years of effort. Ansel Adams is a great example. Also, no professional photographer would use a toy camera for anything other than a gag/vanity/show-off project. They infallibly use reliable, quality equipment day in day out.
- 7y ago
- _Nat_ 7y agoThis argument seems to pop up with respect to proprietary software, where folks can be reluctant to acquire a dependence on a commercial vendor making acceptable offerings in the future. In more extreme cases, I've seen it as an argument against Microsoft software like Windows and Word. In less extreme cases, I've adopted this attitude with respect to some tightly restricted engineering (CAD) software used back in grad school. I mean, it was really useful software, but I didn't feel that I could reasonably rely on having access to it whenever I'd need it.
- gumby 7y agoThis is also why I try to avoid customization and try to use everything with the “out of the box” defaults. Not absolutely, but as much as possible.
- cardiffspaceman 7y agoAgreed. "After the apocalypse, I can't use this editor or shell because my macros or aliases are on github and the net was blown up"
- gumby 7y agoNow everything has gone back to serial, after the apocalypse you should be able to toggle in the boot loader by scraping two wires together...really fast!
- jevgeni 7y agoThis is not much of a computational survivalism, if you need to install extra software on target machine. I mean, you might as well install Python. Do your task with PowerShell or DOS batch, and then this might cut it as computational survivalism on that box. EDIT: Here, the solution to the author's problem without needing to pull a *nix tool chain in. ((Get-Content "<file name>").ToCharArray() ` | Where-Object { $_ -eq '<the character that you need>' } ` | Measure-Object).Count
- AnIdiotOnTheNet 7y agoI mostly really like PowerShell, and it works for Windows where the CLR is now part of the OS. I do wish there was a Linux equivalent but lighter weight, but so far all I've seen are attempts to shoe-horn existing unix programs into that model. I wish there was a ground-up rewrite, an entire environment that used a PowerShell-like object model with no legacy dependencies.
- jevgeni 7y agoYes, you are right. It would be a cool thing to have - an object bash. 8O
- rapind 7y agoNu Shell looks promising (inspired by PowerShell). https://github.com/nushell/nushell https://github.com/nushell/nushell
- AnIdiotOnTheNet 7y agoEh, it looks decent, but it isn't really what I want. For one, its cross-platform nature means it has eschewed greater integration with the system. One of the great strengths of PowerShell is how it treats pretty much every system interface as the same kind of object as everything else. Linux has these weird text based interfaces in /sys/ and /proc/ that it'd be nice to wrap. Secondly it looks like it depends on a fair amount of typical userland being available. It'd prefer something I could slap on top of busybox and be done. Something that only depended on the kernel. But, maybe with some tweaking...
- k2enemy 7y agoA similar, but different, concept that I've seen is "slow computing." It also involves the use of terminal applications over GUIs but with the goal of having an environment that changes less often, is easier to investigate and modify the source code, and is easier to understand exactly what is happening with your data. After so many frustrating experiences with modern GUI apps significantly changing interfaces and functionality, phoning home and creating privacy concerns, or being just plain buggy, I've moved to "slow computing" and have found it a huge relief. And incidentally, not slow at all -- I'm far more productive.
- AnIdiotOnTheNet 7y agoThe annoying thing is that GUIs don't have to be made this way, the complaints you have are relatively recent inventions.
- Nasrudith 7y agoI am pleasantly surprised that this wasn't about the daft Collapse OS. I understand where he is coming from and why he may have gotten that impression but it is more like "accubus ergonomics" essentially for reasons I will explain at the end. The point of the command line is that the shell is very transportable, efficient, and powerful. Compare updating a syatem - you can just write or pass about a single shellscript to do what woukd otherwise require clicking through a ton of menus, watching bars grow, and waiting for it to need clicked. Not that big a deal for one person one system, a bigger deal for 5000. Its main weakness is the required experience and knowledge to use so if you already have those capabilities the added costs are marginal. This the name - it is like dealing with decimals or integers with an accubus vs a calculator and scratch paper. Surprisingly despite a millenia gap the calculator is slower - the user needs to read symbols and enter them one at a time with a hunt and peckish sort of mechanism. Meanwhile a skilled user will have its operation essentially in muscle memory and only need to read the end state. The accubus is rare in comparison because it generally isn't worth it on its own vs the easier alternatives. Plus the paper has a less volatile storage format than bead position. Command line operation isn't for everyone and in most cases and that is okay but it still has a valid purpose.
- svachalek 7y agoabacus?
- sixtypoundhound 7y agoAgree with minimalism. The older I get, the more I value mastery of few things over a superficial grasp of many. In some cases, this can feel a bit regressive. Stepping back to a bone-head simple pattern with fewer moving parts and less dependencies on complicated "others" I cannot see. Part of why I hate WordPress with a passion. It's complex mess of plugins and themes, slapped together without the tight moderation and insistence on good style that we see in a place like the Python standard library.
- alleycat5000 7y agoReminds me of this great talk from Dave Beazley where "I talk about my experience serving as an expert-witness in a patent infringement lawsuit. In this work I was locked away in a hidden vault with 1.5 TB of C++ source code and a Python interpreter. Great fun ensued." http://pyvideo.org/video/2645/discovering-python http://pyvideo.org/video/2645/discovering-python
- absorber 7y agoNotice how the author had looked up how to do his example in Stack Overflow. I found that foraging through different man pages looking for a specific thing to do only once is just very time consuming. Luckily looking up information on places like Stack Exchange or even different blogs and forums (which I usually land on after using a search engine) is much more efficient. Yes, the information contained might not be completely up-to-date or accurate, but at least it sets you up on the right path to search further.