7 ms·

> Till helps you circumvent detected as a web scraper by identifying your scraper as a real web browser. It does this by generating random user-agent headers an

by cube00 5y ago

> Till helps you circumvent detected as a web scraper by identifying your scraper as a real web browser. It does this by generating random user-agent headers and randomizing proxy IPs (that you supply) on every HTTP request. How did we start with consent using robots.txt and end up here? A good neighbor doesn't use circumvention.

JulianWasTaken 5y ago
And only a few paragraphs down from a heading that says > Till was architected to follow best practices that DataHen has accumulated I know there are plenty of sites out there out to prevent any scraping whatsoever, or to improperly prevent some situations most of us would agree is reasonable behavior, but this appears to be blatantly hostile to web admins out there.
- cube00 5y ago
  I agree there are cases where entities don't want things scraped that really should be scraped to preserve the history they'd like to rewrite. My concern is that tools like these that use circumvention by default become the go-to when someone needs scraping and makes life hell for us running sites on hardware that's enough to service our small customer base but not an army of bots.
- paco3346 5y ago
  Admin of a 5000+ website platform (with customer inventory) here. This the exact kind of thing I've been working so hard to block lately. Since Feb 2021 we've seen a substantial increase in scraping of our customers' inventory to the point we now have more bot traffic than human traffic. Annoyingly, only 5-10 requests come from a single IP so we've had to resort to always challenging requests that come from certain ASNs (typically those owned by datacenters). This type of project frustrates me because it knowlingly goes against a site's desired bot action (via robots.txt).
realusername 5y ago
We ended up here since some people wrongly decided that only Google has a right to scrape.
- PaulHoule 5y ago
  I've run sites that have a lot of pages where 80%+ of the traffic is web crawlers. Google sends some traffic so I can afford to let them scrape. Bing crawls almost as much as Google but sends 5% as much traffic. Baidu crawls more than Google and never sends a single visitor. I hate reinforcing the Google monopoly, but a crawler that doesn't send any traffic is expensive to serve.
  - dredmorbius 5y ago
    You might want to ask yourself, or your readers, what it is people are trying to access on your site that they cannot by other means. The interfaces for many sites actively and with brutal effectiveness deny ready access to any content not currently featured on the homepage or stream feed. Search features are frequently nonexistent, crippled, or dysfunctional. Last week I found myself stumbling across a now-archived radio programme on a website which afforded exceedingly poor access to the content. The show ran weekly for over a decade, with 531 episodes. Those are visible ... ten at a time ... through either the Archive or Search features. Scraping the site gives me the full archive listing, all 11 years, in a single webpage, that loads in under a second. I can search that by date, title, or guest to find episodes of interest. The utility of this (a few hours work on my part) is much higher than that of the site itself. Often current web sites / apps are little more than wrappers around a JSON delivery-and-parsing engine. Dumping the raw JSON can be much more useful for a technical user. (Reddit, Diaspora, Mastodon, and Ello are several sites in which this is at least somewhat possible.) Much of the suck is imposed by monetisation schemes. One project of mine, decrufting the Washington Post's website, resulted in article pages with two percent the weight of the originally-delivered payload. The de-cruftified version strips not only extraneous JS and CSS, but nags and teasers which are really nothing but distraction to me. Again, that's typical. I'm aware that many scrapers are not benign. More than you might think are, and the fact that casual scraping is a problem for your delivery system reflects more poorly on it than them.
ok2938 5y ago
We ended up here, because a) the promise of easy to use structured data - that is inherent in data processing systems - has not been fulfilled: you have to scrape HTML to reconstruct some relational table, and b) information, despite being almost free to transmit in large amounts today, is treated as if it were a physical good, and each copy costs money. There is just lots of money in there, because our economic system looks more backwards than forwards.
Nextgrid 5y ago
If a real user can access the content then that user should be able to delegate the work to a machine.
- Doctor_Fegg 5y ago
  But what when there are separate interfaces specifically designed for real users and machines? Case in point: OpenStreetMap (run by volunteer sysadmins) provides completely open, machine-readable data dumps. You can use these to set up your own geocoding, rendering or similar service. There is copious documentation, several post-processed dumps for particular purposes, etc. etc. OSM also provides a friendly, human-scale, user-facing interface for map browsing and search. There are clearly documented limitations/TOUs for automated use of this interface. Does that prevent people from setting up massive scrapers to scrape the human-facing interface, rather than using the machine-facing data dumps? No, it does not; and the volunteer sysadmins have to spend an inordinate amount of time protecting the service against these people. DataHen's proudly admitted practices ("No need to worry about IP bans, we auto rotate IPs on any requests that are made."; "our massive pool of auto-rotating proxies, user agents, and 'secret-sauce' helps get around them") is directly antithetical to this sort of scenario. I find this incredibly irresponsible and unethical.
  - ethbr0 5y ago
    Technically, this is the same rabbit hole that leads to DRM and device intent superceding user intent. Because that's the only way to be sure. It's unfortunate that it also allows havoc and burdens good services. But device-over-user is not a future I want to live in.
  - capableweb 5y ago
    Of course if there is another, unlimited way of getting the data, then it's perfectly fine to "redirect" people to another avenue. What this conversation is about (since we're talking about web scrapers here, not "data downloaders" or whatever you would call it) is when there is no other avenue to get the data, you should be able to access the same data via a machine as you could when you're a person.
CiaTransMatters 5y ago
In order for the CIA to continue its legendary fight against transphobia, it is important for them to use these technologies to gather OSINT on regions that house millions of backwards non-progressives and all of their content. I can't believe HN is such a hotbed for transphobics like this to make comments without moderation or punishment.
danpalmer 5y ago
I work on a lot of web scraping and we have business agreements with every site that we scrape explicitly allowing us to do so and with pre-approval for the scraping rate (which we carefully control). None of this gets around over-eager Cloudflare or Akamai rules set up years ago by some contractor that the businesses have no real ability to change.
- Havoc 5y ago
  Why scrape at all if there are agreements in place. Seems like a API task
  - selcuka 5y ago
    > None of this gets around over-eager Cloudflare or Akamai rules set up years ago by some contractor that the businesses have no real ability to change. If the business has no ability to even change the CloudFlare settings I don't expect them to be able to provide an API.
  - hestefisk 5y ago
    In that case, the web site IS the API… only difference is data is wrapped in HTML not JSON.
  - danpalmer 5y ago
    Only about 1 in 20 of our partners has the in-house tech necessary to provide us an API. In other words, we were able to accerelate our integrations by 20x by scraping.
anigbrowl 5y ago
Good neighbors don't track people's every move online either, but here we are.

by cube00 5y ago

JulianWasTaken 5y ago

And only a few paragraphs down from a heading that says > Till was architected to follow best practices that DataHen has accumulated I know there are plenty of sites out there out to prevent any scraping whatsoever, or to improperly prevent some situations most of us would agree is reasonable behavior, but this appears to be blatantly hostile to web admins out there.

cube00 5y ago
I agree there are cases where entities don't want things scraped that really should be scraped to preserve the history they'd like to rewrite. My concern is that tools like these that use circumvention by default become the go-to when someone needs scraping and makes life hell for us running sites on hardware that's enough to service our small customer base but not an army of bots.
paco3346 5y ago
Admin of a 5000+ website platform (with customer inventory) here. This the exact kind of thing I've been working so hard to block lately. Since Feb 2021 we've seen a substantial increase in scraping of our customers' inventory to the point we now have more bot traffic than human traffic. Annoyingly, only 5-10 requests come from a single IP so we've had to resort to always challenging requests that come from certain ASNs (typically those owned by datacenters). This type of project frustrates me because it knowlingly goes against a site's desired bot action (via robots.txt).

realusername 5y ago

We ended up here since some people wrongly decided that only Google has a right to scrape.

PaulHoule 5y ago
I've run sites that have a lot of pages where 80%+ of the traffic is web crawlers. Google sends some traffic so I can afford to let them scrape. Bing crawls almost as much as Google but sends 5% as much traffic. Baidu crawls more than Google and never sends a single visitor. I hate reinforcing the Google monopoly, but a crawler that doesn't send any traffic is expensive to serve.
- dredmorbius 5y ago
  You might want to ask yourself, or your readers, what it is people are trying to access on your site that they cannot by other means. The interfaces for many sites actively and with brutal effectiveness deny ready access to any content not currently featured on the homepage or stream feed. Search features are frequently nonexistent, crippled, or dysfunctional. Last week I found myself stumbling across a now-archived radio programme on a website which afforded exceedingly poor access to the content. The show ran weekly for over a decade, with 531 episodes. Those are visible ... ten at a time ... through either the Archive or Search features. Scraping the site gives me the full archive listing, all 11 years, in a single webpage, that loads in under a second. I can search that by date, title, or guest to find episodes of interest. The utility of this (a few hours work on my part) is much higher than that of the site itself. Often current web sites / apps are little more than wrappers around a JSON delivery-and-parsing engine. Dumping the raw JSON can be much more useful for a technical user. (Reddit, Diaspora, Mastodon, and Ello are several sites in which this is at least somewhat possible.) Much of the suck is imposed by monetisation schemes. One project of mine, decrufting the Washington Post's website, resulted in article pages with two percent the weight of the originally-delivered payload. The de-cruftified version strips not only extraneous JS and CSS, but nags and teasers which are really nothing but distraction to me. Again, that's typical. I'm aware that many scrapers are not benign. More than you might think are, and the fact that casual scraping is a problem for your delivery system reflects more poorly on it than them.

ok2938 5y ago

We ended up here, because a) the promise of easy to use structured data - that is inherent in data processing systems - has not been fulfilled: you have to scrape HTML to reconstruct some relational table, and b) information, despite being almost free to transmit in large amounts today, is treated as if it were a physical good, and each copy costs money. There is just lots of money in there, because our economic system looks more backwards than forwards.

Nextgrid 5y ago

If a real user can access the content then that user should be able to delegate the work to a machine.

Doctor_Fegg 5y ago
But what when there are separate interfaces specifically designed for real users and machines? Case in point: OpenStreetMap (run by volunteer sysadmins) provides completely open, machine-readable data dumps. You can use these to set up your own geocoding, rendering or similar service. There is copious documentation, several post-processed dumps for particular purposes, etc. etc. OSM also provides a friendly, human-scale, user-facing interface for map browsing and search. There are clearly documented limitations/TOUs for automated use of this interface. Does that prevent people from setting up massive scrapers to scrape the human-facing interface, rather than using the machine-facing data dumps? No, it does not; and the volunteer sysadmins have to spend an inordinate amount of time protecting the service against these people. DataHen's proudly admitted practices ("No need to worry about IP bans, we auto rotate IPs on any requests that are made."; "our massive pool of auto-rotating proxies, user agents, and 'secret-sauce' helps get around them") is directly antithetical to this sort of scenario. I find this incredibly irresponsible and unethical.
- ethbr0 5y ago
  Technically, this is the same rabbit hole that leads to DRM and device intent superceding user intent. Because that's the only way to be sure. It's unfortunate that it also allows havoc and burdens good services. But device-over-user is not a future I want to live in.
- capableweb 5y ago
  Of course if there is another, unlimited way of getting the data, then it's perfectly fine to "redirect" people to another avenue. What this conversation is about (since we're talking about web scrapers here, not "data downloaders" or whatever you would call it) is when there is no other avenue to get the data, you should be able to access the same data via a machine as you could when you're a person.

CiaTransMatters 5y ago

In order for the CIA to continue its legendary fight against transphobia, it is important for them to use these technologies to gather OSINT on regions that house millions of backwards non-progressives and all of their content. I can't believe HN is such a hotbed for transphobics like this to make comments without moderation or punishment.

danpalmer 5y ago

I work on a lot of web scraping and we have business agreements with every site that we scrape explicitly allowing us to do so and with pre-approval for the scraping rate (which we carefully control). None of this gets around over-eager Cloudflare or Akamai rules set up years ago by some contractor that the businesses have no real ability to change.

Havoc 5y ago
Why scrape at all if there are agreements in place. Seems like a API task
- selcuka 5y ago
  > None of this gets around over-eager Cloudflare or Akamai rules set up years ago by some contractor that the businesses have no real ability to change. If the business has no ability to even change the CloudFlare settings I don't expect them to be able to provide an API.
- hestefisk 5y ago
  In that case, the web site IS the API… only difference is data is wrapped in HTML not JSON.
- danpalmer 5y ago
  Only about 1 in 20 of our partners has the in-house tech necessary to provide us an API. In other words, we were able to accerelate our integrations by 20x by scraping.

anigbrowl 5y ago

Good neighbors don't track people's every move online either, but here we are.