3 ms·

Web Crawling: Why is Service a Better Option than Software?

9 points by grepsr 12y ago

lesingerouge 12y ago
Having had direct contact with all the issue of massive crawls (talking about tens of thousands of websites daily), I can say that it depends on the scale and "business-criticality" of the crawl. For the types of crawls that their product seems to be targeted at (quite small scale and with a very precise "point and gather" approach), it might be efficient to go the service way. But the author fails to mention that the outsourcing decision holds true only for all non-mission critical crawls (whether it's for marketing, accounting or any other operations) while it's totally unfeasible if your company is doing big number crunching. When the product is the data, you need to control the accuracy and timeliness of the crawl without a doubt.
- grepsr 12y ago
  Right now most of our revenue is from bigger projects servicing products which are data centric. We cater both type of projects -- small and big. Not all companies (consultant firms, investment banks, equity researchers) have the expertise to do a in-house solution for bigger projects. This is where a service has an upper hand over using a tool.
weland 12y ago
This kind of article is honestly insulting to the intelligence of anyone who has written more than a dozen lines of code. It literally says nothing while "magically" hiding the fact that "services" are just "software" handled by someone else. Seriously?
- icebraining 12y ago
  Hmm? "Grepsr (...) uses software as a service" "With service mode, however, the software that runs the service (...)" "A powerful crawling software runs at the back of our system (...)"
known 12y ago
http://www.bbc.co.uk/news/technology-23866614 http://www.bbc.co.uk/news/technology-23866614 Google handles more than 100 billion searches each month or more than three billion per day. Google has to crawl more than 20 billion pages a day to keep up with the ever-changing web. Google has found 60 trillion web addresses so far, up from one trillion in 2008. Google has indexed pages from over 230 million domains. A query on Google travels 750 miles each way, to and from the data centre.

9 points by grepsr 12y ago

Having had direct contact with all the issue of massive crawls (talking about tens of thousands of websites daily), I can say that it depends on the scale and "business-criticality" of the crawl. For the types of crawls that their product seems to be targeted at (quite small scale and with a very precise "point and gather" approach), it might be efficient to go the service way. But the author fails to mention that the outsourcing decision holds true only for all non-mission critical crawls (whether it's for marketing, accounting or any other operations) while it's totally unfeasible if your company is doing big number crunching. When the product is the data, you need to control the accuracy and timeliness of the crawl without a doubt.

grepsr 12y ago
Right now most of our revenue is from bigger projects servicing products which are data centric. We cater both type of projects -- small and big. Not all companies (consultant firms, investment banks, equity researchers) have the expertise to do a in-house solution for bigger projects. This is where a service has an upper hand over using a tool.

weland 12y ago

This kind of article is honestly insulting to the intelligence of anyone who has written more than a dozen lines of code. It literally says nothing while "magically" hiding the fact that "services" are just "software" handled by someone else. Seriously?

icebraining 12y ago
Hmm? "Grepsr (...) uses software as a service" "With service mode, however, the software that runs the service (...)" "A powerful crawling software runs at the back of our system (...)"

known 12y ago

http://www.bbc.co.uk/news/technology-23866614 http://www.bbc.co.uk/news/technology-23866614 Google handles more than 100 billion searches each month or more than three billion per day. Google has to crawl more than 20 billion pages a day to keep up with the ever-changing web. Google has found 60 trillion web addresses so far, up from one trillion in 2008. Google has indexed pages from over 230 million domains. A query on Google travels 750 miles each way, to and from the data centre.