Having had direct contact with all the issue of massive crawls (talking about tens of thousands of websites daily), I can say that it depends on the scale and "

Having had direct contact with all the issue of massive crawls (talking about tens of thousands of websites daily), I can say that it depends on the scale and "business-criticality" of the crawl. For the types of crawls that their product seems to be targeted at (quite small scale and with a very precise "point and gather" approach), it might be efficient to go the service way. But the author fails to mention that the outsourcing decision holds true only for all non-mission critical crawls (whether it's for marketing, accounting or any other operations) while it's totally unfeasible if your company is doing big number crunching. When the product is the data, you need to control the accuracy and timeliness of the crawl without a doubt.