dor_jack

4 ms·

1. 1.▲
   by dor_jack 9y ago

   If you need to perform a web-scale crawl I strongly recommend https://www.mixnode.com .
2. 2.▲
   by dor_jack 9y ago

   The platform we used provided their own seedlist and took it from there.
3. 3.▲
   by dor_jack 9y ago

   It was an all inclusive deal: 420 TB at 0.06 per GB = $25,804
4. 4.▲
   by dor_jack 9y ago

   If our company approves I would like to publish some general statistics that may be of interest to others.
5. 5.▲
   by dor_jack 9y ago

   It took us about 13 days. We recorded reources of all types: text/ , image/ , application/* As one would expect the vast majority of data recorded is text/* (html,...)
6. 6.▲
   by dor_jack 9y ago

   We are processing the data as we speak. However the movement of technology based on where your company is based is truly incredible. Will update this in a few days with more data.
7. 7.▲
   by dor_jack 9y ago

   Our company is in the Marketing Intelligence (MI) industry. We needed to measure the penetration of multiple technologies in different countries.
8. 8.▲
   I just finished crawling 5.19B web pages, Ask Me Anything

   19 points by dor_jack 9y ago|19 comments