4 ms·

Eh, this is pretty weak. Scrapers are no different from other browsing devices. The web speaks HTTP. There's no reason that using another HTTP browser would cau

Eh, this is pretty weak. Scrapers are no different from other browsing devices. The web speaks HTTP. There's no reason that using another HTTP browser would cause any disparate impact just by virtue of not being a conventional desktop browser -- you've thrown out a pretty absurd hypothetical. In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript. I did an analysis and a session browsed with my specialized browser would always consume less than 100K of bandwidth (and often far less), whereas a session browsed with a conventional desktop browser would consume at least 1.2 MB, even if everything was cached, and sometimes up to 5 MB. In addition, on the desktop, a JavaScript heartbeat was sent back every few seconds, so all of that data was conserved too. Because we were a specialized browser used by people looking for a very specific piece of data, we could employ caching mechanisms that meant that each person could get their request fulfilled without having to hit the data source's servers. We also had a regular pacing algorithm that meant our users were contacting the site way less than they would've been if they were using a conventional desktop browser. Our service saved the data source a large amount of resource cost. When we were shut down, their site struggled for about two weeks to return to stability. I think they had anticipated the opposite effect. Our service also saved our users a large amount of time. We were accessing publicly-available factual data that was not copyrightable (but only available from this one source's site). There's no reason that the user should be able to choose between Firefox and Chrome but not a task-specialized browser. It is true that some people will (usually accidentally) cause a DDoS with scrapers because the target site is not properly configured, but the same thing could be done with desktop browsers. It doesn't mean that scrapers should be disadvantaged.

DashRattlesnake 10y ago
> I did an analysis and a session browsed with my specialized browser would always consume less than 100K of bandwidth (and often far less), whereas a session browsed with a conventional desktop browser would consume at least 1.2 MB, even if everything was cached, and sometimes up to 5 MB. In addition, on the desktop, a JavaScript heartbeat was sent back every few seconds, so all of that data was saved too. Bandwidth is certainly part of it, but there's also also database and app-server load (which may be the actual bottleneck) that a scraper isn't necessarily bypassing.
- cookiecaper 10y ago
  Yeah, I just have a hard time buying that a scraper that does less than a conventional desktop browser is going to accidentally stumble across something that causes the server-side to flip out. I'm not really sure in what case your hypothetical is plausible. Scrapers are usually used to get publicly-available data more efficiently. What you're describing would basically require the scraper to hammer an invisible endpoint somewhere, but there's no reason the scraper would do that -- it just wants to get the data displayed by the site in a more efficient manner. I suppose the browser could enforce a cooldown on an expensive callback via JavaScript, which a scraper would circumvent, but IMO that's not a fair reason to say scrapers are disallowed; cooldowns should be enforced server-side. There's no way to ensure that a user is going to execute your script. That's just part of the deal. Everything about scrapers means less server load; no images, no wandering around the site trying to find the right place, no heavy JavaScript callbacks that invoke server-side application load, etc. Scrapers are just highly-optimized browsing devices targeting specific pieces of data; it's logical that they would be cheaper to serve than a desktop user who's concerned about aesthetics and the like. In our specific case, those JavaScripts we didn't download included instructions to make over 100 AJAX requests on every page load. No wonder users were looking for something more efficient. So I agree that a scraper isn't necessarily bypassing some load-heavy operations, but I find it highly implausible that a non-malicious scraper would be invoking operations that cause extra load (beyond just hitting the site too often). Frankly, I'd be surprised if there was a functional scraper that regularly invoked more resource cost per-session than a typical desktop browsing session to get equivalent data.
  - DashRattlesnake 10y ago
    > What you're describing would basically require the scraper to hammer an invisible endpoint somewhere That wasn't my point. My point was: a lot of a website's costs are hidden from a web scraper (e.g. database load), so a scraper can't claim, based on the variables they can observe (bandwidth), that they're costing the website less than normal traffic. I was basically responding to statements like this: > In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript. There's really no way for a scraper to know that unless the website tells them. Their usage pattern is different than typical users and raw bandwidth (for stuff like static images) may not matter to the website.
hchasestevens 10y ago
A small counterpoint to this -- in the airline industry, it's relatively commonplace for seat reservations to be made for a user _before_ payment has occurred. In this case, if you're mirroring normal browser activity, you can (temporarily) reduce availability on a flight, potentially even bumping up the price for other, legitimate users, and almost certainly causing the airline to incur costs beyond normal bandwidth and server costs. I'm sure there are many other domains for which this is also the case, however rare.
- gnud 10y ago
  If they don't do the seat reservation behind a POST, or at least blacklist the reservation page in robots.txt, I have no sympathy.
- Steeeve 10y ago
  I've had this happen regularly enough that I developed the habit of finding a fare in the morning and then returning at 11pm ready to buy.
- Chris2048 10y ago
  Cinemas and some online shops do the same. I've always wondered if it's possible to block any tickets sales for entire flights/screenings/products this way. And if airline tickets are based on supply v demand, it might even be possible to drive down ticket prices by suddenly dropping a load of blocks near to the flight date.
- angry_octet 10y ago
  If you have ever tried to buy hot tickets online and not been able to get any, this is due to bots. Bots are the scalpers friend. This can easily be prevented by requiring ID matching the ticket on entry, but the ticket sellers often don't seem to care.
muglug 10y ago
> you've thrown out a pretty absurd hypothetical Not even remotely absurd. Where is the data your scraper consuming coming from? It's almost always served from some sort of data repository (SQL or otherwise). That data costs far more per MB to serve up quickly than JS/CSS/images. Suppose, for example, you host a blogging platform that has one very popular user. Most accounts on your site don't get a ton of visitors, and that one very popular user's post are all stored in cache. Then along comes a scraper. He thinks, "Hey, this site is serving up a million page impressions a day. It can definitely handle me scraping the site". But when he runs the scraper, he fills up the cache with a ton of data that it doesn't need, causing cache evictions and general performance degradation for everyone else.
- cookiecaper 10y ago
  There are already 6-8 major scrapers that do this constantly, across the whole internet, called search engines. You can't handle that? What if you get a normal user who says "Hey, I wanna see some of the lesser known authors on this platform" and opens up a hundred tabs with rarely-read blogs? What if you get 10 users who decide to do that on the same day? Is it reasonable to sue them? Should there be a legal protection to punish them for making your site slow? Don't blame the user for your scaling issues. If the optimized browser ("scraper") isn't hammering your site at a massively unnatural interval, it's clean. And if it is, you should have server-side controls that prevent one client from asking for too much data. These are just normal problems that are part of being on the web. It's not fair to pin it on non-malicious users, even if they're not using a conventional desktop browser.
  - muglug 10y ago
    Search engines respect robots.txt – not sure many scrapers do.
    - cookiecaper 10y ago
      First, search engines are scrapers. No need to make a distinction. Second, search engines don't always respect robots.txt. They sometimes do. Even Google itself says it may still contact a page that has disallowed it. [0] Third, robots.txt is just a convention. There's no reason to assume it has any binding authority. Users should be able to access public HTTP resources with any non-disruptive HTTP client, regardless of the end server's opinion. [0] "You should not use robots.txt as a means to hide your web pages from Google Search results. This is because other pages might point to your page, and your page could get indexed that way, avoiding the robots.txt file." / http://archive.is/A5zh8 http://archive.is/A5zh8

DashRattlesnake 10y ago

> I did an analysis and a session browsed with my specialized browser would always consume less than 100K of bandwidth (and often far less), whereas a session browsed with a conventional desktop browser would consume at least 1.2 MB, even if everything was cached, and sometimes up to 5 MB. In addition, on the desktop, a JavaScript heartbeat was sent back every few seconds, so all of that data was saved too. Bandwidth is certainly part of it, but there's also also database and app-server load (which may be the actual bottleneck) that a scraper isn't necessarily bypassing.

cookiecaper 10y ago
Yeah, I just have a hard time buying that a scraper that does less than a conventional desktop browser is going to accidentally stumble across something that causes the server-side to flip out. I'm not really sure in what case your hypothetical is plausible. Scrapers are usually used to get publicly-available data more efficiently. What you're describing would basically require the scraper to hammer an invisible endpoint somewhere, but there's no reason the scraper would do that -- it just wants to get the data displayed by the site in a more efficient manner. I suppose the browser could enforce a cooldown on an expensive callback via JavaScript, which a scraper would circumvent, but IMO that's not a fair reason to say scrapers are disallowed; cooldowns should be enforced server-side. There's no way to ensure that a user is going to execute your script. That's just part of the deal. Everything about scrapers means less server load; no images, no wandering around the site trying to find the right place, no heavy JavaScript callbacks that invoke server-side application load, etc. Scrapers are just highly-optimized browsing devices targeting specific pieces of data; it's logical that they would be cheaper to serve than a desktop user who's concerned about aesthetics and the like. In our specific case, those JavaScripts we didn't download included instructions to make over 100 AJAX requests on every page load. No wonder users were looking for something more efficient. So I agree that a scraper isn't necessarily bypassing some load-heavy operations, but I find it highly implausible that a non-malicious scraper would be invoking operations that cause extra load (beyond just hitting the site too often). Frankly, I'd be surprised if there was a functional scraper that regularly invoked more resource cost per-session than a typical desktop browsing session to get equivalent data.
- DashRattlesnake 10y ago
  > What you're describing would basically require the scraper to hammer an invisible endpoint somewhere That wasn't my point. My point was: a lot of a website's costs are hidden from a web scraper (e.g. database load), so a scraper can't claim, based on the variables they can observe (bandwidth), that they're costing the website less than normal traffic. I was basically responding to statements like this: > In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript. There's really no way for a scraper to know that unless the website tells them. Their usage pattern is different than typical users and raw bandwidth (for stuff like static images) may not matter to the website.

hchasestevens 10y ago

A small counterpoint to this -- in the airline industry, it's relatively commonplace for seat reservations to be made for a user _before_ payment has occurred. In this case, if you're mirroring normal browser activity, you can (temporarily) reduce availability on a flight, potentially even bumping up the price for other, legitimate users, and almost certainly causing the airline to incur costs beyond normal bandwidth and server costs. I'm sure there are many other domains for which this is also the case, however rare.

gnud 10y ago
If they don't do the seat reservation behind a POST, or at least blacklist the reservation page in robots.txt, I have no sympathy.
Steeeve 10y ago
I've had this happen regularly enough that I developed the habit of finding a fare in the morning and then returning at 11pm ready to buy.
Chris2048 10y ago
Cinemas and some online shops do the same. I've always wondered if it's possible to block any tickets sales for entire flights/screenings/products this way. And if airline tickets are based on supply v demand, it might even be possible to drive down ticket prices by suddenly dropping a load of blocks near to the flight date.
angry_octet 10y ago
If you have ever tried to buy hot tickets online and not been able to get any, this is due to bots. Bots are the scalpers friend. This can easily be prevented by requiring ID matching the ticket on entry, but the ticket sellers often don't seem to care.

muglug 10y ago

> you've thrown out a pretty absurd hypothetical Not even remotely absurd. Where is the data your scraper consuming coming from? It's almost always served from some sort of data repository (SQL or otherwise). That data costs far more per MB to serve up quickly than JS/CSS/images. Suppose, for example, you host a blogging platform that has one very popular user. Most accounts on your site don't get a ton of visitors, and that one very popular user's post are all stored in cache. Then along comes a scraper. He thinks, "Hey, this site is serving up a million page impressions a day. It can definitely handle me scraping the site". But when he runs the scraper, he fills up the cache with a ton of data that it doesn't need, causing cache evictions and general performance degradation for everyone else.

cookiecaper 10y ago
There are already 6-8 major scrapers that do this constantly, across the whole internet, called search engines. You can't handle that? What if you get a normal user who says "Hey, I wanna see some of the lesser known authors on this platform" and opens up a hundred tabs with rarely-read blogs? What if you get 10 users who decide to do that on the same day? Is it reasonable to sue them? Should there be a legal protection to punish them for making your site slow? Don't blame the user for your scaling issues. If the optimized browser ("scraper") isn't hammering your site at a massively unnatural interval, it's clean. And if it is, you should have server-side controls that prevent one client from asking for too much data. These are just normal problems that are part of being on the web. It's not fair to pin it on non-malicious users, even if they're not using a conventional desktop browser.
- muglug 10y ago
  Search engines respect robots.txt – not sure many scrapers do.
  - cookiecaper 10y ago
    First, search engines are scrapers. No need to make a distinction. Second, search engines don't always respect robots.txt. They sometimes do. Even Google itself says it may still contact a page that has disallowed it. [0] Third, robots.txt is just a convention. There's no reason to assume it has any binding authority. Users should be able to access public HTTP resources with any non-disruptive HTTP client, regardless of the end server's opinion. [0] "You should not use robots.txt as a means to hide your web pages from Google Search results. This is because other pages might point to your page, and your page could get indexed that way, avoiding the robots.txt file." / http://archive.is/A5zh8 http://archive.is/A5zh8