5 ms·

You don’t have to fend off anything, you just have to fix your server to support this modest amount of traffic. Everyone else is visiting your site for entirel

by sneak 1y ago

You don’t have to fend off anything, you just have to fix your server to support this modest amount of traffic. Everyone else is visiting your site for entirely self-serving purposes, too. I don’t understand why people are ok with Google scraping their site (when it is called indexing), fine with users scraping their site (when it is called RSS reading), but suddenly not ok with AI startups scraping their site. If you publish data to the public, expect the public to access it. If you don’t want the public (this includes AI startups) to access it, don’t publish it. Your website is not being misused when the data is being downloaded to train AI. That’s literally what public data is for.

red369 1y ago
Is it because people viewed it as Google scraping the site to make an index so that people could find the site, while the AI scraping is intended so people won’t need to visit the site at all?
- ehutch79 1y ago
  Also, google is relatively considerate when crawling.
  - DamonHD 1y ago
    Usually though from time to time over many many years from when G was still wet behind the ears I have had to write specific defences effectively just to deal with it.
    - ehutch79 1y ago
      Yeah, “Relatively” is doing a lot of heavy lifting there. I remember people with lax security and a tags that went to deletion endpoints… we all learned a lot back in the day.
- owebmaster 1y ago
  The AI apps (namely chatgpt and Claude) are evolving to display external data with widgets that will potentially drive more traffic than google has been doing for a long time. Might be worth changing focus as SEO killed google.
  - eitland 1y ago
    Kagi already behave really nicely, giving a small summary with numbered footnotes that links to where Kagi got its information from. Which IMO shows that is already possible to do this and and it has been for a while already.
bentley 1y ago
> I don’t understand why people are ok with Google scraping their site (when it is called indexing), fine with users scraping their site (when it is called RSS reading), but suddenly not ok with AI startups scraping their site. I’m pretty open‐minded about AI, and have no visceral objection to AI scraping in theory. However… the rise of AI scrapers is the first time in twenty years of running tiny websites that my servers—all my servers, consistently—hit CPU and bandwidth usage levels far above baseline, and even brush against my hosts’ monthly limits. When I check server logs, the traffic is largely bots repeatedly trawling infinite variations of version control history pages. Googlebot never crawled these enough to become a problem; Internet Archive never crawled these enough to become a problem. But the current scrapers are, and literally everyone I talk to who hosts their own VCS webservers is having the same problem right now. Many are choosing to reduce the problem by injecting Cloudflare or Anubis, which I as a user hate, and refuse to do myself—but what alternative can I propose to them? People running these servers are just users of Gitea and CGit and whatnot, they’re not capable of rewriting the underlying tools to reduce the CPU usage of diff generation.
- sneak 1y ago
  Seems to me like standard dumb rate limiting (max 100r/min or like 250r/5min) per IP would solve the problem easily without affecting human users at all. Surprised this isn’t a feature in Gitea already (even though it is better done in the reverse proxy web server).
  - DamonHD 1y ago
    Some scummy scrapers are snowshoeing their way through scads of (residential) IPs, just for one example. Assuming that there are simple universal solutions, eg based on IP or UA, suggests that you have been lucky enough not to deal with many varieties of this misbehaviour.
- deleted 1y ago
  [deleted]
isaacremuant 1y ago
Your expectations are arbitrary and you're not hosting it so they can take whatever measures they want. Just like some publish "public websites" and immediately show a "subscribe" button if you want to see more. It's their prerogative.
loloquwowndueo 1y ago
Tell me you haven’t had your site scraped by an abusive AI crawler without telling me you haven’t had your site scraped by an abusive AI crawler. Google and traditional crawlers are fine. This new breed of AI crawlers have no scruples, no rate limits, and will just load pages and follow links as fast as they can, very likely overwhelming smaller web servers in the process. I too have blocked many of them to avoid them filling my logs with nonexistent page accesses, for example.
- sneak 1y ago
  I run a public Gitea at https://git.eeqj.de https://git.eeqj.de and while I see it getting scraped, I haven’t noticed any effects whatsoever from the additional load. No mitigations have been necessary.
  - tomsmeding 1y ago
    The OP notes at some point that the scrapers were generating lots of source tarballs individual commits. That takes server resources, and a small server can't sustain many of those requests per second. I wonder why you (apparently) don't suffer from this — in fact one can even wonder why such tarballs are generated in the first place, it sounds very pointless.
    - bob1029 1y ago
      > The OP notes at some point that the scrapers were generating lots of source tarballs individual commits. I refuse to believe OAI has a scraper that is designed to push commits. It is obvious to me that we are conflating other forms of malicious use with the boogeyman.
      - kassner 1y ago
        They are talking about the standard “Download zip”, which can also be done for commits individually. They aren’t precomputed, which causes the load.
OtherShrezzing 1y ago
>I don’t understand why people are ok with Google scraping their site (when it is called indexing), fine with users scraping their site (when it is called RSS reading), but suddenly not ok with AI startups scraping their site. Google and the publisher share the value derived from that scraping event (somewhat equitably) when a user clicks a link in Google's index. OpenAI, Anthropic, and co extract all of the value, and incur costs to the publisher, from their scraping events.
treve 1y ago
The answer is attribution. also putting things public does not imply no copyright or license

by sneak 1y ago

red369 1y ago

Is it because people viewed it as Google scraping the site to make an index so that people could find the site, while the AI scraping is intended so people won’t need to visit the site at all?

ehutch79 1y ago
Also, google is relatively considerate when crawling.
- DamonHD 1y ago
  Usually though from time to time over many many years from when G was still wet behind the ears I have had to write specific defences effectively just to deal with it.
  - ehutch79 1y ago
    Yeah, “Relatively” is doing a lot of heavy lifting there. I remember people with lax security and a tags that went to deletion endpoints… we all learned a lot back in the day.
owebmaster 1y ago
The AI apps (namely chatgpt and Claude) are evolving to display external data with widgets that will potentially drive more traffic than google has been doing for a long time. Might be worth changing focus as SEO killed google.
- eitland 1y ago
  Kagi already behave really nicely, giving a small summary with numbered footnotes that links to where Kagi got its information from. Which IMO shows that is already possible to do this and and it has been for a while already.

bentley 1y ago

> I don’t understand why people are ok with Google scraping their site (when it is called indexing), fine with users scraping their site (when it is called RSS reading), but suddenly not ok with AI startups scraping their site. I’m pretty open‐minded about AI, and have no visceral objection to AI scraping in theory. However… the rise of AI scrapers is the first time in twenty years of running tiny websites that my servers—all my servers, consistently—hit CPU and bandwidth usage levels far above baseline, and even brush against my hosts’ monthly limits. When I check server logs, the traffic is largely bots repeatedly trawling infinite variations of version control history pages. Googlebot never crawled these enough to become a problem; Internet Archive never crawled these enough to become a problem. But the current scrapers are, and literally everyone I talk to who hosts their own VCS webservers is having the same problem right now. Many are choosing to reduce the problem by injecting Cloudflare or Anubis, which I as a user hate, and refuse to do myself—but what alternative can I propose to them? People running these servers are just users of Gitea and CGit and whatnot, they’re not capable of rewriting the underlying tools to reduce the CPU usage of diff generation.

sneak 1y ago
Seems to me like standard dumb rate limiting (max 100r/min or like 250r/5min) per IP would solve the problem easily without affecting human users at all. Surprised this isn’t a feature in Gitea already (even though it is better done in the reverse proxy web server).
- DamonHD 1y ago
  Some scummy scrapers are snowshoeing their way through scads of (residential) IPs, just for one example. Assuming that there are simple universal solutions, eg based on IP or UA, suggests that you have been lucky enough not to deal with many varieties of this misbehaviour.
deleted 1y ago
[deleted]

isaacremuant 1y ago

Your expectations are arbitrary and you're not hosting it so they can take whatever measures they want. Just like some publish "public websites" and immediately show a "subscribe" button if you want to see more. It's their prerogative.

loloquwowndueo 1y ago

Tell me you haven’t had your site scraped by an abusive AI crawler without telling me you haven’t had your site scraped by an abusive AI crawler. Google and traditional crawlers are fine. This new breed of AI crawlers have no scruples, no rate limits, and will just load pages and follow links as fast as they can, very likely overwhelming smaller web servers in the process. I too have blocked many of them to avoid them filling my logs with nonexistent page accesses, for example.

sneak 1y ago
I run a public Gitea at https://git.eeqj.de https://git.eeqj.de and while I see it getting scraped, I haven’t noticed any effects whatsoever from the additional load. No mitigations have been necessary.
- tomsmeding 1y ago
  The OP notes at some point that the scrapers were generating lots of source tarballs individual commits. That takes server resources, and a small server can't sustain many of those requests per second. I wonder why you (apparently) don't suffer from this — in fact one can even wonder why such tarballs are generated in the first place, it sounds very pointless.
  - bob1029 1y ago
    > The OP notes at some point that the scrapers were generating lots of source tarballs individual commits. I refuse to believe OAI has a scraper that is designed to push commits. It is obvious to me that we are conflating other forms of malicious use with the boogeyman.
    - kassner 1y ago
      They are talking about the standard “Download zip”, which can also be done for commits individually. They aren’t precomputed, which causes the load.

OtherShrezzing 1y ago

>I don’t understand why people are ok with Google scraping their site (when it is called indexing), fine with users scraping their site (when it is called RSS reading), but suddenly not ok with AI startups scraping their site. Google and the publisher share the value derived from that scraping event (somewhat equitably) when a user clicks a link in Google's index. OpenAI, Anthropic, and co extract all of the value, and incur costs to the publisher, from their scraping events.

treve 1y ago

The answer is attribution. also putting things public does not imply no copyright or license