13 ms·

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later p

To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network connection, add some split and rsync. A "distributed crawler" is really only like 10 lines of shell script." [0] since his blog is gone: http://readwrite.com/2011/01/22/data-mining-and-taco-bell-prog http://readwrite.com/2011/01/22/data-mining-and-taco-bell-pr...

make3 12y ago
you know, or the real world reasonnable mature engineering answer, a Java/C#/C++ scalable parallel tool using modern libraries and MPI if it ever needs to scale.
- jessaustin 12y ago
  Yeah if you're Google. Most people are not, and wget is plenty. After all it's written in C.
  - eru 12y ago
    Or use curl, for a slightly better engineered wget.
    - copsarebastards 12y ago
      I'm pretty proficient in both, and I think that's a mischaracterization of the two tools. wget is more suited to pulling down large files, groups of files, etc. curl is more suited to API calls where you might need to do something complicated at the protocol level. Each has their use.
- untog 12y ago
  You usually don't need that though, that's the point. If you're building an entire service around page crawling, sure. If you're doing a one-off task, don't bother.
- MrBuddyCasino 12y ago
  That would be my first gut reaction too, but if it is as simple as downloading webpages, this is actually a really great solution. I suspect he used that when he built Milo, a now defunct startup sold to eBay where they had to update prices and inventory data regularly. A startup should make different choices than Google.
threeseed 12y ago
Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform easily to do X, Y, Z after the crawling ? I can't stand developers who come up with the xargs/wget approach, hack something together and then walk away from it. I've seen it far too often and it's great for the short term. Dreadful for the long term.
- x0x0 12y ago
  Sure -- you can level multiple complaints. What about the failed pages? How about shoving those on a queue and retrying n times with an exponential backoff between. What about the total number of failed pages? What about failed pages by site? etc etc etc But so what -- the principle is still sound. All I described is still a 100 line python script, written in an afternoon, instead of 3 weeks of working bringing up emr, installing and configuring nutch, figuring out network issues around emr nodes talking to commodity internet, installing a persistent queue, performing remote debugging, building a task dag in either code or (god help you) oozie/xml, and on and on.
  - threeseed 12y ago
    Anybody can throw some crap together and make it stick. And it's a perfectly valid solution. My issue is when there is criticism laid against those solutions which are actually engineered in a way that allows for supportability and extensibility. They are arguably far more important than execution time.
    - angrybits 12y ago
      I can't figure out why you're arguing against standard unix tools/idioms in the name of supportability and extensibility? It defies logic.
profquail 12y ago
archive.org has Ted's blog post, "Taco Bell Programming": https://web.archive.org/web/20101025124303/http://teddziuba.com/2010/10/taco-bell-programming.html https://web.archive.org/web/20101025124303/http://teddziuba....
lloeki 12y ago
Alternative, real life scenario: navigate through 6 months of daily MySQL dumps, assorted YAML files and Rails production.log, looking for some cross product between tables, requests and serialised entities, for analysis and/or data recovery (pinpoint or retrieval). zcat/cut/sed/grep/awk/perl crawled through it in a couple of minutes and required less than half an hour to craft a reliable enough implementation (including looking up relations from foreign keys straight from the SQL dumps). My colleagues, who still don't get the point of a command line, would still be restoring each dump individually and running SQL requests manually to this day (or more probably declare it "too complex" and mark it as a loss). Side note: I'm torn between leaving this place where nobody seems to understand the point of anything remotely like engineering or keeping this job where I'm obviously being extremely useful to our customers.
- sebastianavina 12y ago
  to be fair, you could have achieved all that with a simple python script. sometimes i feel python is the new bash.
  - coldpie 12y ago
    Yeah. This is a great article. While I was reading it, I was thinking about what I would have done, and my answer was Python. Bash is just too easy to do wrong (see the recent Steam rm -rf bug), and I don't code in it often enough to know the pitfalls by heart. I'd be interested to see another article about doing this job in Python and how its performance compares to this simple one-liner.
  - falcor84 12y ago
    Python is a great tool for this, and is even better when used with the pythonpy tool, which allows for convenient integration of python commands inside unix pipelines - https://github.com/Russell91/pythonpy https://github.com/Russell91/pythonpy
  - tokenrove 12y ago
    Note that you lose some of the parallelism you get effortlessly from the Unix pipeline if you do it as a single (simple) Python script.
101914 12y ago
Are these "web pages" all on the same website? If so, using wget is a poor solution. I have not used wget in over a decade but as I recall it does not do HTTP pipelining; I could be wrong on that - please correct me. I do recall with certainty that when wget was first written and disseminated in the 1990's, "webmasters" wanted to ban it. httpd's were not as resilient then as they are today, nor was bandwidth and hardware as inexpensive. HTTP pipelining is a smarter alternative than burdening the remote host with thousands of consecutive or simultaneous connections. Depending on the remote host's httpd settings, HTTP pipelining usually lets you make 100 or maybe more requests using a single connection. It can be acomplished with only a simple tcpclient like the original nc and the shell. In any event, the line about a "distributed crawler" is spot on. Never understimate the power of marketing to suspend common sense. Also, I find that I can often speed my scripts up a little by using exec in shell pipelines, e.g., util1 |exec util2 or exec util1 |exec util2. There are other, better approaches besides using the builtin exec, but I will leave those for another day.
juliangregorian 12y ago
Having written distributed crawlers, saturating the network connection is quite easy to do and is the main reason for even distributing that type of work in the first place.
mercurial 12y ago
You could just use Scrapy [1]. Easy to setup, and plenty of options you can activate if needed. Likely more robust than shell scripts as well. No Hadoop involved. 1: http://doc.scrapy.org/en/0.24/intro/tutorial.html http://doc.scrapy.org/en/0.24/intro/tutorial.html

make3 12y ago

you know, or the real world reasonnable mature engineering answer, a Java/C#/C++ scalable parallel tool using modern libraries and MPI if it ever needs to scale.

jessaustin 12y ago
Yeah if you're Google. Most people are not, and wget is plenty. After all it's written in C.
- eru 12y ago
  Or use curl, for a slightly better engineered wget.
  - copsarebastards 12y ago
    I'm pretty proficient in both, and I think that's a mischaracterization of the two tools. wget is more suited to pulling down large files, groups of files, etc. curl is more suited to API calls where you might need to do something complicated at the protocol level. Each has their use.
untog 12y ago
You usually don't need that though, that's the point. If you're building an entire service around page crawling, sure. If you're doing a one-off task, don't bother.
MrBuddyCasino 12y ago
That would be my first gut reaction too, but if it is as simple as downloading webpages, this is actually a really great solution. I suspect he used that when he built Milo, a now defunct startup sold to eBay where they had to update prices and inventory data regularly. A startup should make different choices than Google.

threeseed 12y ago

Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform easily to do X, Y, Z after the crawling ? I can't stand developers who come up with the xargs/wget approach, hack something together and then walk away from it. I've seen it far too often and it's great for the short term. Dreadful for the long term.

x0x0 12y ago
Sure -- you can level multiple complaints. What about the failed pages? How about shoving those on a queue and retrying n times with an exponential backoff between. What about the total number of failed pages? What about failed pages by site? etc etc etc But so what -- the principle is still sound. All I described is still a 100 line python script, written in an afternoon, instead of 3 weeks of working bringing up emr, installing and configuring nutch, figuring out network issues around emr nodes talking to commodity internet, installing a persistent queue, performing remote debugging, building a task dag in either code or (god help you) oozie/xml, and on and on.
- threeseed 12y ago
  Anybody can throw some crap together and make it stick. And it's a perfectly valid solution. My issue is when there is criticism laid against those solutions which are actually engineered in a way that allows for supportability and extensibility. They are arguably far more important than execution time.
  - angrybits 12y ago
    I can't figure out why you're arguing against standard unix tools/idioms in the name of supportability and extensibility? It defies logic.

profquail 12y ago

archive.org has Ted's blog post, "Taco Bell Programming": https://web.archive.org/web/20101025124303/http://teddziuba.com/2010/10/taco-bell-programming.html https://web.archive.org/web/20101025124303/http://teddziuba....

lloeki 12y ago

Alternative, real life scenario: navigate through 6 months of daily MySQL dumps, assorted YAML files and Rails production.log, looking for some cross product between tables, requests and serialised entities, for analysis and/or data recovery (pinpoint or retrieval). zcat/cut/sed/grep/awk/perl crawled through it in a couple of minutes and required less than half an hour to craft a reliable enough implementation (including looking up relations from foreign keys straight from the SQL dumps). My colleagues, who still don't get the point of a command line, would still be restoring each dump individually and running SQL requests manually to this day (or more probably declare it "too complex" and mark it as a loss). Side note: I'm torn between leaving this place where nobody seems to understand the point of anything remotely like engineering or keeping this job where I'm obviously being extremely useful to our customers.

sebastianavina 12y ago
to be fair, you could have achieved all that with a simple python script. sometimes i feel python is the new bash.
- coldpie 12y ago
  Yeah. This is a great article. While I was reading it, I was thinking about what I would have done, and my answer was Python. Bash is just too easy to do wrong (see the recent Steam rm -rf bug), and I don't code in it often enough to know the pitfalls by heart. I'd be interested to see another article about doing this job in Python and how its performance compares to this simple one-liner.
- falcor84 12y ago
  Python is a great tool for this, and is even better when used with the pythonpy tool, which allows for convenient integration of python commands inside unix pipelines - https://github.com/Russell91/pythonpy https://github.com/Russell91/pythonpy
- tokenrove 12y ago
  Note that you lose some of the parallelism you get effortlessly from the Unix pipeline if you do it as a single (simple) Python script.

101914 12y ago

Are these "web pages" all on the same website? If so, using wget is a poor solution. I have not used wget in over a decade but as I recall it does not do HTTP pipelining; I could be wrong on that - please correct me. I do recall with certainty that when wget was first written and disseminated in the 1990's, "webmasters" wanted to ban it. httpd's were not as resilient then as they are today, nor was bandwidth and hardware as inexpensive. HTTP pipelining is a smarter alternative than burdening the remote host with thousands of consecutive or simultaneous connections. Depending on the remote host's httpd settings, HTTP pipelining usually lets you make 100 or maybe more requests using a single connection. It can be acomplished with only a simple tcpclient like the original nc and the shell. In any event, the line about a "distributed crawler" is spot on. Never understimate the power of marketing to suspend common sense. Also, I find that I can often speed my scripts up a little by using exec in shell pipelines, e.g., util1 |exec util2 or exec util1 |exec util2. There are other, better approaches besides using the builtin exec, but I will leave those for another day.

juliangregorian 12y ago

Having written distributed crawlers, saturating the network connection is quite easy to do and is the main reason for even distributing that type of work in the first place.

mercurial 12y ago

You could just use Scrapy [1]. Easy to setup, and plenty of options you can activate if needed. Likely more robust than shell scripts as well. No Hadoop involved. 1: http://doc.scrapy.org/en/0.24/intro/tutorial.html http://doc.scrapy.org/en/0.24/intro/tutorial.html