6 ms·

I wonder if this works every time... And if it did, how long until a defense is made...

cliffy 18y ago
You could make it very onerous to move around the site even if you're logged in. For example, after every few links you follow you have to pass another Turing test. That would be completely unreasonable though. I think once a user is logged into your site, you'd have an extremely hard time defeating this sort of behavior without degrading the quality of user interaction or treading on legitimate use of your site. You may be able to defeat egregious abuses, such as scraping entire photo galleries in seconds, but even that can be defeated by a script that randomizes requests/times between requests.
- geoscripting 18y ago
  More and more websites are adding Turing tests after relatively short intervals of time. I consider that to be a good thing, and there are some good ways to stop/identify spiders. You can check out this site: http://stackoverflow.com/questions/450835/how-do-you-stop-scripters-from-slamming-your-website-hundreds-of-times-a-second http://stackoverflow.com/questions/450835/how-do-you-stop-sc... . It shows some pretty good techniques that one can use to defend from spiders.
Retric 18y ago
There is nothing magic about web browsers, telent to port 80 at www.google.com and with a simple get request they will spit back their website. You can make it a little harder to do this stuff but a packet sniffer is always going to let you pretend to be any software you want unless they are using encryption. Also because Firefox is open source you can't prevent people from scripting using it anyway. PS: I recommend all aspiring coders to telent to www.google.com at least once just to feel the magic.
- timf 18y ago
  "Firefox is open source you can't prevent people from scripting using it" There are frameworks for doing that already, check out Selenium: http://seleniumhq.org/ http://seleniumhq.org/
- geoscripting 18y ago
  That's how it all starts :)
- radu_floricica 18y ago
  > PS: I recommend all aspiring coders to telent to www.google.com at least once just to feel the magic. I'm pretty sure it's illegal in a couple of ways. </bitter> But yes, the magic is there.
  - xenophanes 18y ago
    illegal!? why!?
    - geoscripting 18y ago
      It is recommended that you use the API that google provides for searching, but I fail to see why telnetting would be illegal. After all, both firefox and telnet use sockets to do their job.
      - radu_floricica 18y ago
        Again, the comment above wasn't meant to be taken literally, but just because they both use sockets doesn't mean they're just as legal. Imagine a web site which terms of service state you cannot use software to circumvent ads. Or where part of the security is done client-side (stupid, yes, but not impossible). Skipping the browser breaches at least the terms of service, and may be constructed as hacking. I think even Google discourages automated searching and prefers you use its api, which (at least some years ago) wasn't free for commercial use. I may be wrong in this particular case, but the important point is you may want to check the specific TOS before skipping the browser.
    - radu_floricica 18y ago
      DMCA, of course. The reply was meant to be exaggerated, I hope telnet itself isn't illegal.
philh 18y ago
Some sites will expire the cookie after a period of inactivity, in which case this will stop working. But it can easily be refreshed by getting a new set of cookies, which as other people have pointed out is a process that can be automated.

You could make it very onerous to move around the site even if you're logged in. For example, after every few links you follow you have to pass another Turing test. That would be completely unreasonable though. I think once a user is logged into your site, you'd have an extremely hard time defeating this sort of behavior without degrading the quality of user interaction or treading on legitimate use of your site. You may be able to defeat egregious abuses, such as scraping entire photo galleries in seconds, but even that can be defeated by a script that randomizes requests/times between requests.

geoscripting 18y ago
More and more websites are adding Turing tests after relatively short intervals of time. I consider that to be a good thing, and there are some good ways to stop/identify spiders. You can check out this site: http://stackoverflow.com/questions/450835/how-do-you-stop-scripters-from-slamming-your-website-hundreds-of-times-a-second http://stackoverflow.com/questions/450835/how-do-you-stop-sc... . It shows some pretty good techniques that one can use to defend from spiders.

Retric 18y ago

There is nothing magic about web browsers, telent to port 80 at www.google.com and with a simple get request they will spit back their website. You can make it a little harder to do this stuff but a packet sniffer is always going to let you pretend to be any software you want unless they are using encryption. Also because Firefox is open source you can't prevent people from scripting using it anyway. PS: I recommend all aspiring coders to telent to www.google.com at least once just to feel the magic.

timf 18y ago
"Firefox is open source you can't prevent people from scripting using it" There are frameworks for doing that already, check out Selenium: http://seleniumhq.org/ http://seleniumhq.org/
geoscripting 18y ago
That's how it all starts :)
radu_floricica 18y ago
> PS: I recommend all aspiring coders to telent to www.google.com at least once just to feel the magic. I'm pretty sure it's illegal in a couple of ways. </bitter> But yes, the magic is there.
- xenophanes 18y ago
  illegal!? why!?
  - geoscripting 18y ago
    It is recommended that you use the API that google provides for searching, but I fail to see why telnetting would be illegal. After all, both firefox and telnet use sockets to do their job.
    - radu_floricica 18y ago
      Again, the comment above wasn't meant to be taken literally, but just because they both use sockets doesn't mean they're just as legal. Imagine a web site which terms of service state you cannot use software to circumvent ads. Or where part of the security is done client-side (stupid, yes, but not impossible). Skipping the browser breaches at least the terms of service, and may be constructed as hacking. I think even Google discourages automated searching and prefers you use its api, which (at least some years ago) wasn't free for commercial use. I may be wrong in this particular case, but the important point is you may want to check the specific TOS before skipping the browser.
  - radu_floricica 18y ago
    DMCA, of course. The reply was meant to be exaggerated, I hope telnet itself isn't illegal.

philh 18y ago

Some sites will expire the cookie after a period of inactivity, in which case this will stop working. But it can easily be refreshed by getting a new set of cookies, which as other people have pointed out is a process that can be automated.