4 ms·

Being the CEO of a firm that offers web-crawling services, I found this post very interesting. On 80legs, the cost for a similar crawl would be $500, so it's n

by jdrock 14y ago

Being the CEO of a firm that offers web-crawling services, I found this post very interesting. On 80legs, the cost for a similar crawl would be $500, so it's nice to know we're competitive on cost.

neurotech1 14y ago
Wouldn't the cost of transferring the crawled data (compressed/archived) to AWS/EC2 partially offset the cost saving? Edit: AWS still has free imbound BW.. my bad
- brey 14y ago
  nice thing about running crawlers from EC2 - all inbound bandwidth is free
  - zerop 14y ago
    Like running crawlers what can be some other use cases where free inbound is a blessing?
    - bigiain 14y ago
      Backup/archive, where uploading data is frequent and downloading it is only very occasionally required.
- jdrock 14y ago
  No - inbound data transfer cost to AWS is free.
BarnabasLAL 14y ago
Came here to suggest 80legs too, happy to see you're on it. I would add that if the project was being done for actual work and not just curiosity, then 80legs has the added advantage of not spending all that dev time building and testing it.
joshu 14y ago
(Completely aside: Your front page's colors look strange on my Mac. https://skitch.com/joshua/ecykp/80legs-custom-web-crawlers-powerful-web-crawling-and-data-extraction https://skitch.com/joshua/ecykp/80legs-custom-web-crawlers-p... -- see the blue border?)
- showerst 14y ago
  Looks like http://80legs.com/_images/the-most-powerful-web-crawler-ever-welcome-to-web-scale.jpg http://80legs.com/_images/the-most-powerful-web-crawler-ever... uses slightly different gradient stops than http://80legs.com/_images/bg-home.jpg http://80legs.com/_images/bg-home.jpg
- ecaron 14y ago
  Not just your Mac - my Firefox & Chrome on Linux. Seems the color palettes in http://80legs.com/_images/bg-home.jpg http://80legs.com/_images/bg-home.jpg vs http://80legs.com/_images/the-most-powerful-web-crawler-ever-welcome-to-web-scale.jpg http://80legs.com/_images/the-most-powerful-web-crawler-ever... don't match up. Looks great on all browsers on Windows, so I'm guessing the jpgs have a color profile attached.
vlad 14y ago
I notice the welcome message and action buttons on the home page are all jpeg images rather than text, which makes it harder for search engines to make a judgement about the content. (The text appears fuzzy as well.) As a developer of a crawler, is there a reason you found this to be beneficial?

by jdrock 14y ago

Being the CEO of a firm that offers web-crawling services, I found this post very interesting. On 80legs, the cost for a similar crawl would be $500, so it's nice to know we're competitive on cost.

neurotech1 14y ago

Wouldn't the cost of transferring the crawled data (compressed/archived) to AWS/EC2 partially offset the cost saving? Edit: AWS still has free imbound BW.. my bad

brey 14y ago
nice thing about running crawlers from EC2 - all inbound bandwidth is free
- zerop 14y ago
  Like running crawlers what can be some other use cases where free inbound is a blessing?
  - bigiain 14y ago
    Backup/archive, where uploading data is frequent and downloading it is only very occasionally required.
jdrock 14y ago
No - inbound data transfer cost to AWS is free.

BarnabasLAL 14y ago

Came here to suggest 80legs too, happy to see you're on it. I would add that if the project was being done for actual work and not just curiosity, then 80legs has the added advantage of not spending all that dev time building and testing it.

joshu 14y ago

(Completely aside: Your front page's colors look strange on my Mac. https://skitch.com/joshua/ecykp/80legs-custom-web-crawlers-powerful-web-crawling-and-data-extraction https://skitch.com/joshua/ecykp/80legs-custom-web-crawlers-p... -- see the blue border?)

showerst 14y ago
Looks like http://80legs.com/_images/the-most-powerful-web-crawler-ever-welcome-to-web-scale.jpg http://80legs.com/_images/the-most-powerful-web-crawler-ever... uses slightly different gradient stops than http://80legs.com/_images/bg-home.jpg http://80legs.com/_images/bg-home.jpg
ecaron 14y ago
Not just your Mac - my Firefox & Chrome on Linux. Seems the color palettes in http://80legs.com/_images/bg-home.jpg http://80legs.com/_images/bg-home.jpg vs http://80legs.com/_images/the-most-powerful-web-crawler-ever-welcome-to-web-scale.jpg http://80legs.com/_images/the-most-powerful-web-crawler-ever... don't match up. Looks great on all browsers on Windows, so I'm guessing the jpgs have a color profile attached.

vlad 14y ago

I notice the welcome message and action buttons on the home page are all jpeg images rather than text, which makes it harder for search engines to make a judgement about the content. (The text appears fuzzy as well.) As a developer of a crawler, is there a reason you found this to be beneficial?