3 ms·

> What you're describing would basically require the scraper to hammer an invisible endpoint somewhere That wasn't my point. My point was: a lot of a website'

> What you're describing would basically require the scraper to hammer an invisible endpoint somewhere That wasn't my point. My point was: a lot of a website's costs are hidden from a web scraper (e.g. database load), so a scraper can't claim, based on the variables they can observe (bandwidth), that they're costing the website less than normal traffic. I was basically responding to statements like this: > In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript. There's really no way for a scraper to know that unless the website tells them. Their usage pattern is different than typical users and raw bandwidth (for stuff like static images) may not matter to the website.

cookiecaper 10y ago
It's true that there's no way to know that for sure, but it doesn't make sense that a scraper, by virtue of its being a scraper, is incurring additional load. A scraper is only making requests that a person with a desktop browser or any other appliance that speaks HTTP could make. What's the difference between a user clicking the same button on the page 50 times or holding down F5 and a scraper that pings a page once a minute? Your argument is basically boiling down to "scrapers could hit one load-heavy endpoint too fast", but so could desktop browsers. So I don't see what it has to do with scraping.
- DashRattlesnake 10y ago
  > but it doesn't make sense that a scraper, by virtue of its being a scraper, is incurring additional load It does, because scrapers don't have normal usage patterns. They're robots and behave like robots. > What's the difference between a user clicking the same button on the page 50 times or holding down F5 and a scraper that pings a page once a minute? Typical users aren't usually in the habit of mashing F5, especially not for robotically long periods of time. It's basically the difference between a theoretical activity and an actual activity. Basically, scraping is not regular usage, and I don't think it's correct to pretend that they're equivalent (or more extremely, that scraping is less costly to the website).
  - cookiecaper 10y ago
    Scrapers are usually coded to have as regular of a usage pattern as possible, so that the data they retrieve is as much like the data the end user would receive as possible. For example, Googlebot does everything in its power to ensure that it sees pages the same way that end users sees them, executing JavaScript and performing OCR to try to read information conveyed in images. Google also has non-Googlebot scans to try to determine if a page is serving different content to Googlebot-labeled scans, and they penalize sites that they suspect of doing this. While it is true that someone could write a scraper that obviously behaved robotically, it is also true that someone could use their desktop browser in a robotic way. Mashing F5 is so common that there are many ancient memes referring to and making jokes about that activity. There are extensions that end users use to record browser macros, behaviors they want their browser to repeat over and over again. However, this conversation about whether scrapers behave robotically or not is moot because a web site shouldn't break down under load when someone uses it in a slightly-irregular way. The obvious, crappy scrapers are trivial to block. The ones that blend into the traffic are no harm, no foul. If you can't tell the difference between an optimized browser like a scraper and a general-purpose browser like Chrome, why shouldn't it be allowed to talk to your site?
  - kuschku 10y ago
    > Typical users aren't usually in the habit of mashing F5, especially not for robotically long periods of time. It's basically the difference between a theoretical activity and an actual activity. Just like every university site ever is completely down during signup days because everyone is mashing F5. Link me your site, I’ll treat it like a college student waiting to be able to sign up for their classes.

cookiecaper 10y ago

It's true that there's no way to know that for sure, but it doesn't make sense that a scraper, by virtue of its being a scraper, is incurring additional load. A scraper is only making requests that a person with a desktop browser or any other appliance that speaks HTTP could make. What's the difference between a user clicking the same button on the page 50 times or holding down F5 and a scraper that pings a page once a minute? Your argument is basically boiling down to "scrapers could hit one load-heavy endpoint too fast", but so could desktop browsers. So I don't see what it has to do with scraping.

DashRattlesnake 10y ago
> but it doesn't make sense that a scraper, by virtue of its being a scraper, is incurring additional load It does, because scrapers don't have normal usage patterns. They're robots and behave like robots. > What's the difference between a user clicking the same button on the page 50 times or holding down F5 and a scraper that pings a page once a minute? Typical users aren't usually in the habit of mashing F5, especially not for robotically long periods of time. It's basically the difference between a theoretical activity and an actual activity. Basically, scraping is not regular usage, and I don't think it's correct to pretend that they're equivalent (or more extremely, that scraping is less costly to the website).
- cookiecaper 10y ago
  Scrapers are usually coded to have as regular of a usage pattern as possible, so that the data they retrieve is as much like the data the end user would receive as possible. For example, Googlebot does everything in its power to ensure that it sees pages the same way that end users sees them, executing JavaScript and performing OCR to try to read information conveyed in images. Google also has non-Googlebot scans to try to determine if a page is serving different content to Googlebot-labeled scans, and they penalize sites that they suspect of doing this. While it is true that someone could write a scraper that obviously behaved robotically, it is also true that someone could use their desktop browser in a robotic way. Mashing F5 is so common that there are many ancient memes referring to and making jokes about that activity. There are extensions that end users use to record browser macros, behaviors they want their browser to repeat over and over again. However, this conversation about whether scrapers behave robotically or not is moot because a web site shouldn't break down under load when someone uses it in a slightly-irregular way. The obvious, crappy scrapers are trivial to block. The ones that blend into the traffic are no harm, no foul. If you can't tell the difference between an optimized browser like a scraper and a general-purpose browser like Chrome, why shouldn't it be allowed to talk to your site?
- kuschku 10y ago
  > Typical users aren't usually in the habit of mashing F5, especially not for robotically long periods of time. It's basically the difference between a theoretical activity and an actual activity. Just like every university site ever is completely down during signup days because everyone is mashing F5. Link me your site, I’ll treat it like a college student waiting to be able to sign up for their classes.