4 ms·

Err, if I was writing a scraper today, I'd just ignore the robots.txt I dunno if Geocities had a similar robots.txt, but it didn't stop several groups from arc

by tibbon 16y ago

Err, if I was writing a scraper today, I'd just ignore the robots.txt I dunno if Geocities had a similar robots.txt, but it didn't stop several groups from archiving it (which was the right thing to do in either scenario).

apgwoz 16y ago
It's only a matter of time before some sysadmin, or automated log analyzer sees you and stops it. You'd be better served to start a distributed scraping and archiving mission going, but, I'm not sure how to start that.
- StavrosK 16y ago
  I'm not sure if JS can request pages and send them to you (XSF protection), but it might be worth writing a small script to request URLs from your server, spider them and send the content back. Then, put it on a website, and tell people "by staying on this page, you are donating bandwidth and helping archive delicious". It's so no-hassle that I bet you could get a huge following.
  - apgwoz 16y ago
    I guess this might actually be a great use of 80Legs. Has anyone used them in the past, and if so, did it actually work?
  - toolate 16y ago
    This would be possible with Flash.
    - StavrosK 16y ago
      Ah, yes! Come to think of it, JS probably couldn't get around the same-origin restrictions of browsers...
PaulHoule 16y ago
Well, I've worked for organizations that had active defenses against crawlers... Make too many HTTP request and hour and ~poof~ a deny directive goes in the .htaccess file, or if they really like to play a rough game they'll firewall you. I know delicious had active defenses because I ran afoul of them.
- deleted 16y ago
  [deleted]
- tibbon 16y ago
  We are legio.... oh, wait, wrong forum. What I mean to say is, I can imagine quite a few ways to get around those :) But yes, I'd imagine they have something in place. Too bad someone inside doesn't just 'happen' to turn those defenses off. opps.
- kevindication 16y ago
  And really, slowing your connections down to rates that are measured in baud would be the most fun.
- jamiequint 16y ago
  True, there are plenty of relatively cheap proxy services made for exactly this reason though that will roll your IP on each request.
deleted 16y ago
[deleted]

by tibbon 16y ago

apgwoz 16y ago

It's only a matter of time before some sysadmin, or automated log analyzer sees you and stops it. You'd be better served to start a distributed scraping and archiving mission going, but, I'm not sure how to start that.

StavrosK 16y ago
I'm not sure if JS can request pages and send them to you (XSF protection), but it might be worth writing a small script to request URLs from your server, spider them and send the content back. Then, put it on a website, and tell people "by staying on this page, you are donating bandwidth and helping archive delicious". It's so no-hassle that I bet you could get a huge following.
- apgwoz 16y ago
  I guess this might actually be a great use of 80Legs. Has anyone used them in the past, and if so, did it actually work?
- toolate 16y ago
  This would be possible with Flash.
  - StavrosK 16y ago
    Ah, yes! Come to think of it, JS probably couldn't get around the same-origin restrictions of browsers...

PaulHoule 16y ago

Well, I've worked for organizations that had active defenses against crawlers... Make too many HTTP request and hour and ~poof~ a deny directive goes in the .htaccess file, or if they really like to play a rough game they'll firewall you. I know delicious had active defenses because I ran afoul of them.

deleted 16y ago
[deleted]
tibbon 16y ago
We are legio.... oh, wait, wrong forum. What I mean to say is, I can imagine quite a few ways to get around those :) But yes, I'd imagine they have something in place. Too bad someone inside doesn't just 'happen' to turn those defenses off. opps.
kevindication 16y ago
And really, slowing your connections down to rates that are measured in baud would be the most fun.
jamiequint 16y ago
True, there are plenty of relatively cheap proxy services made for exactly this reason though that will roll your IP on each request.

deleted 16y ago

[deleted]