9 ms·
Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monit
by threeseed 12y ago
Oh right the "cool kids" approach.
Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform easily to do X, Y, Z after the crawling ?
I can't stand developers who come up with the xargs/wget approach, hack something together and then walk away from it. I've seen it far too often and it's great for the short term. Dreadful for the long term.
- x0x0 12y agoSure -- you can level multiple complaints. What about the failed pages? How about shoving those on a queue and retrying n times with an exponential backoff between. What about the total number of failed pages? What about failed pages by site? etc etc etc But so what -- the principle is still sound. All I described is still a 100 line python script, written in an afternoon, instead of 3 weeks of working bringing up emr, installing and configuring nutch, figuring out network issues around emr nodes talking to commodity internet, installing a persistent queue, performing remote debugging, building a task dag in either code or (god help you) oozie/xml, and on and on.
- threeseed 12y agoAnybody can throw some crap together and make it stick. And it's a perfectly valid solution. My issue is when there is criticism laid against those solutions which are actually engineered in a way that allows for supportability and extensibility. They are arguably far more important than execution time.
- angrybits 12y agoI can't figure out why you're arguing against standard unix tools/idioms in the name of supportability and extensibility? It defies logic.
- x0x0 12y agoI think, in many peoples minds, extensibility == pain; either lots of code configuration (hello java, ejb), or xml (hello hadoop, java, spring, ejb), or tons of code (hello java, c++), etc. When nice languages don't make things painful, it sometimes feels like it's wrong, or not really enough work, or in some other way, insufficient. But people can mistake the rituals of programming for getting actual work accomplished. :shrug: just .02
- nilbot 12y agoSimple: because std utils are programs that do what they supposed to do. if problems bound are well within the definition domain of a std util then its all good. Supportability and Extensibility is way too generic for you to draw a line saying std utils can handle them all. After all, they are programs, not programming languages.
- shitgoose 12y agoI imagine what would have happened to Linux if Linus designed it with supportability and extensibility in mind.
- saidajigumi 12y ago> Dreadful for the long term. Here comes a bubble-bursting: I've lead a team that built data processing tools exactly like this, and the performance and ease of manipulating vast amounts of text using classic shell tools is hard to beat. We had no problems with any of: operational supportability, restart recovery, or maintainability. Highly testable, even. No, it's not just cowboy-coded crappy shell scripts and pipelines. Sure, there's a discipline to building pipelined tooling well, just as with any other kind of software. Your problems seem to stem from a lack of disciplined software engineering rather than the tools, or maybe just an environment that encouraged high technical debt. The kicker? We were using pipeline-based tooling ... running on a Hadoop cluster. Honestly, I'm a bit surprised to see such an apparent mindshare split (judging by some recent HN posts) between performant single-system approaches and techniques used in-cluster. The point that "be sure your data is really, truly big data" is obviously well made, and still bears repetition. Yet the logical follow-on is that these technique are even more applicable to cluster usage. Why would anyone throw away multiple orders of magnitude performance going to a cluster-based approach?
- threeseed 12y agoThis "orders of magnitude" less performance going to a Hadoop cluster approach is nonsense. There are plenty of options for Hadoop that make it dramatically faster than the naive example in the article. Spark ? MR-Redis ? Storm ?
- virmundi 12y agoSomething to keep in mind is that while a single app might be best served on a single machine piping data, multiple apps working the same data set probably wouldn't scale. Hadoop for all its faults does provide a nice, relativily simple programing platform to support multiple data processes. Edited for phone swipe mistakes.
- developer1 12y agoThe problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example, but it's very common for shell scripts to have horrible error handling and checks for important things. The shell is just not suitable for extremely robust programs. I would bet that 80%+ of people who think they're good at shell scripting... aren't.
- AnthonyMouse 12y agoThe Unix people have thought of these things. You can easily do them with command line tools. > Operational Supportability: How do you monitor the operation ? Downloading files with wget will create files and directories as it proceeds. You can observe and count them to determine progress, or pass a shell script to xargs that writes whatever progress data you like to a file before/after calling wget. > Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? wget has command line options to skip downloading files that already exist. Or you can use tail to skip the number of lines in the input file as there exist complete entries in the destination directory. > Maintainability: Can we run the same application on our desktop as on our production servers ? I'm not sure how this is supposed to be an argument against using the standard utilities that are on everybody's machine already. > Extensibility: Can we extend the platform easily to do X, Y, Z after the crawling ? Again, what? Extensibility is the wheelhouse of the thing you're complaining about.
- sauere 12y agoI could not agree more. And even with the things you mentioned, such a script will still be tiny and very readable. You just have to love the simplicity.
- nextos 12y agoI love Unix, but it's just a local minima in the design space. For example, it's typical text processing pipelines are hard to branch. I have hacked up some solutions, but never found them very elegant. I would love to hear some solutions to this. Ended up switching to Clojure (Prismatic's) Graph.
- LeoPanthera 12y ago> For example, it's typical text processing pipelines are hard to branch. I'm not entirely sure what you mean by this, but it sounds like you should use "tee" pointing at a fifo.
- 12y ago
- gpapilion 12y agoI look at this article as a criticism of the hadoop being the wrong tool for small data sets. This starts to become a question of data locality, and size. 1.75 GB isn't enough data to justify a hadoop solution. That data size fits easily in memory, and without doubt on a single system. From that point you only need some degree of parallelism to maximize the performance. That being said when its 35TB of data, the answer starts to change. The fact that shell commands were used makes for an easy demo that might be hard to support, but if a solution were written using a traditional language with threading or IPC instead of relying on hadoop you should always be faster, since you don't incur the latency costs of the network.
- lloeki 12y ago> That data size fits easily in memory, and without doubt on a single system. From that point you only need some degree of parallelism to maximize the performance. That being said when its 35TB of data, the answer starts to change. Not at all, because data is being streamed. It could just as easily be 35TB and only use a few MB of RAM.
- gpapilion 12y agoThe IO bandwidth of the system will limit you more loading 35TB of data in ram on a single system, even if it is streamed. You'll need more than one disk, and network card to do this in a timely fashion.
- danieldk 12y ago1.75 GB isn't enough data to justify a hadoop solution. That data size fits easily in memory, and without doubt on a single system. It depends on what you do with the data. If you are processing the data in 512KB chunks and each chunk takes a day to process (because expensive computation), you probably do want to spread the work over some cluster.
- jbergens 12y agoI think the article said that you don't need to use Hadoop for everything and that it might be much faster to just use command line tools on a single computer. Of course you might find a use case where the total computing time is massive and in that case a cluster is better. I still don't think many use cases have that problem. We are doing some simple statistics at work for much smaller data sizes and the computing time is usually around 10-100 ms so it could probably compute small batches at almost network speed.
- Symbiote 12y ago'pv' shows a progress bar, something like '&& touch $x.success' can help restart recovery. I'd probably pick the shell approach for something I expect to be a one-off, but reconsider each time the task is repeated. I printed http://xkcd.com/1205/ http://xkcd.com/1205/ and stuck it to the wall. It's a useful reference when someone seems to be under or overengineering something.
- slagfart 12y agoThat chart assumes 24 hours days. The reality of (my?) productivity is that I have perhaps six productive hours in a day. If I can save eight productive hours per month, that's sixteen days a year, not four.
- lloeki 12y agoWhile it drives some point home, the chart eludes the question of robustness (a written script will run twice the same way, whereas human error, especially on routine tasks, may hit one hard) and documentation (writing even a lightly commented script to do yearly maintenance is guaranteed to help your future self remember things one year from now).
- coldtea 12y ago>Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform easily to do X, Y, Z after the crawling ? Yeah, and then they produce some over-engineered monstrocity, late, over-budget and barely able to run...
- pacala 12y agoI fully agree with all the operational / restart / features comments. However, I've often been surprised on how a little thought / research can build all these requirements on top of off-the-shelf components. I also agree that it is likely that one will eventually outgrow wget, but, for example, one may run out of business / pivot before that.
- JustSomeNobody 12y agoI hate developers who over engineer everything and then when it's time to perform some of that support and extensibility, they leave because maintenance is beneath them. They put this behemoth together with a thousand moving parts and then walk away from it. This, too, happens often.
- deleted 12y ago[deleted]
- na85 12y agoI agree, it's much better to re-invent the wheel.
- mwotton 12y agoI've implemented this. It isn't too bad up to a certain point. You have to be a bit careful about your filesystem/layout of files, lots of filesystems don't particularly like it when you have a few hundred million files in one directory.
- Kiro 12y agoAnd I can't stand developers who overengineer things. We have a couple of them at my company and something that should take a few hours always take several weeks just because of all the reasons you mention. Most things don't need that kind of features and maintainability and if they do in the future we can just rewrite them from scratch. The overall expected return on investment is still better since we seldom need to.
- nilbot 12y agoQuite the opposite, and, quite simple: engineers over-engineer thing in order to make things generic. and generic make solutions robust. that's basic science. Unless the problem and solution are well understood, your investment won't guarantee a return at all.
- gaius 12y agoNo, developers over-engineer because setting up a 20-node Hadoop cluster is fun, whereas doing the same task in an hour in Excel means you have to move onto some other boring task. Generic doesn't mean robust either, I don't know where you got that from,the two concepts are entirely unrelated.
- nilbot 12y agoGeneric -> robust. i... i dont know how to explain that. honestly i haven't thought about the necessity of explaining things like this. its... basic mathematics.
- gaius 12y agoI don't think these words mean what you think they mean. Like "science" and "mathematics".
- TickleSteve 12y agoGeneric != Robust. It can quite easily be the opposite, they are certainly orthogonal concepts.
- charltones 12y agoThere are command line tools available to help the transition from 'hack' one liner to a more maintainable / supportable solution. For instance drake (https://github.com/Factual/drake https://github.com/Factual/drake) a 'Make for data' which does dependency checking would allow for sensible restarts of the pipeline. The O'Reilly Data Science at the Command Line book (linked elsewhere in the comments) has a good deal to say on the subject: turning one liners into extensible shell scripts, using drake, using Gnu Parallel.
- andyidsinga 12y agoI've been using GNU Parallel for orchestrating running remote scripts/tools on a bunch of machines in my compute and storage cluster. Its now my goto tool for almost any remote ssh task that needs to hit a bunch of machines at once. An excellent tool ...apparently an improvement on xargs even for local parallel tasks ( see http://unix.stackexchange.com/questions/104778/gnu-parallel-vs-i-mean-background-vs-xargs-p http://unix.stackexchange.com/questions/104778/gnu-parallel-... )
- 2ion 12y agoJust these pointed remarks: - There are things like pv(1) which allow you to monitor pipes. Things like systemd open other interesting possibilities for implementing, grouping and monitoring your processes. - Recovery could be implemented by keeping a logfile of completed steps like a list of completely processed files or moving processed files to elsewhere in the file system (could be done in memory only using ramfs or tmpfs). Of course, it depends on the case whether it's feasible or not. - Extensibility: Scripts and configurations can be done in shell syntax. Hook systems and frameworks of varying complexity exist. I agree that doing extensibility in shell code is going to turn out to be hazardous when done without proper concept and understanding of the tool at hand.
- debacle 12y agoWhat kind of operational supportability do you need for a script that took a few hours to write and takes <5 minutes to run?
- module0000 12y agoWe don't really "come up" with the xargs/wget approach. The approach is already there, waiting to be utilized by someone who understands the tools. The "cool kids" don't like(or are not able) to understand the tools. The author (I think) is trying to point out that these problems are already solved, decades ago, with existing UNIX tools.