3 ms·

Tell me you haven’t had your site scraped by an abusive AI crawler without telling me you haven’t had your site scraped by an abusive AI crawler. Google and tr

Tell me you haven’t had your site scraped by an abusive AI crawler without telling me you haven’t had your site scraped by an abusive AI crawler. Google and traditional crawlers are fine. This new breed of AI crawlers have no scruples, no rate limits, and will just load pages and follow links as fast as they can, very likely overwhelming smaller web servers in the process. I too have blocked many of them to avoid them filling my logs with nonexistent page accesses, for example.

sneak 1y ago
I run a public Gitea at https://git.eeqj.de https://git.eeqj.de and while I see it getting scraped, I haven’t noticed any effects whatsoever from the additional load. No mitigations have been necessary.
- tomsmeding 1y ago
  The OP notes at some point that the scrapers were generating lots of source tarballs individual commits. That takes server resources, and a small server can't sustain many of those requests per second. I wonder why you (apparently) don't suffer from this — in fact one can even wonder why such tarballs are generated in the first place, it sounds very pointless.
  - bob1029 1y ago
    > The OP notes at some point that the scrapers were generating lots of source tarballs individual commits. I refuse to believe OAI has a scraper that is designed to push commits. It is obvious to me that we are conflating other forms of malicious use with the boogeyman.
    - kassner 1y ago
      They are talking about the standard “Download zip”, which can also be done for commits individually. They aren’t precomputed, which causes the load.
      - bob1029 1y ago
        They don't need to be pre-computed or stored. You can directly stream stdout from git archive to the HTTP response. This is the default behavior of git archive.
        loloquwowndueo 1y ago
        And git archive uses zero cpu therefore it can handle thousands of archive requests per second. Right? Right??
        bob1029 1y ago
        Yes, your CPU should be nearly idle. Git archive is I/O bound.

sneak 1y ago

I run a public Gitea at https://git.eeqj.de https://git.eeqj.de and while I see it getting scraped, I haven’t noticed any effects whatsoever from the additional load. No mitigations have been necessary.

tomsmeding 1y ago
The OP notes at some point that the scrapers were generating lots of source tarballs individual commits. That takes server resources, and a small server can't sustain many of those requests per second. I wonder why you (apparently) don't suffer from this — in fact one can even wonder why such tarballs are generated in the first place, it sounds very pointless.
- bob1029 1y ago
  > The OP notes at some point that the scrapers were generating lots of source tarballs individual commits. I refuse to believe OAI has a scraper that is designed to push commits. It is obvious to me that we are conflating other forms of malicious use with the boogeyman.
  - kassner 1y ago
    They are talking about the standard “Download zip”, which can also be done for commits individually. They aren’t precomputed, which causes the load.
    - bob1029 1y ago
      They don't need to be pre-computed or stored. You can directly stream stdout from git archive to the HTTP response. This is the default behavior of git archive.
      - loloquwowndueo 1y ago
        And git archive uses zero cpu therefore it can handle thousands of archive requests per second. Right? Right??
        bob1029 1y ago
        Yes, your CPU should be nearly idle. Git archive is I/O bound.