7 ms·

Scrapers scrape all sorts of nonsense, notably every possible git diff from git hosts. They clearly aren't concerned with rationality.

• XorNot 2mo ago
  Scrapers scrape every link they can find. The git host example is probably the one which Anubis is the dumbest defense for: the main reason to have those links is for easy machine interaction. So that's dead when you implement it. So why even host the links? It's a git repo: send them a local got client and let them clone the repo in browser or something.
  • mitxela 2mo ago
    The main reason to have those links is for people to click on them. No sensibly programmed machine is scraping diffs from a git web front end.
    • cyphar 2mo ago
      Not to mention it is the least efficient representation of git commits. It seems a lot of people are unaware that git generates the diff representation you see with "git show" on the fly -- internally, commits are Merkle trees representing a snapshot of the repo file contents. As you said, if you actually want to interact with a remote git repo you'd do so with packfiles not diffs.
    • novafunc 2mo ago
      That's exactly what's happening to the Linux git repository. The server constantly has mutltiple CPU cores dedicated to rendering git of various old, dead branches in html for scrapers.
  • utopiah 2mo ago
    > why even host the links? It's a git repo: send them a local got client and let them clone the repo in browser or something. I document my code, in issue and elsewhere on the Web, by referring to specific commits so having a URL for each is useful to me at least.