3 ms·

One of the things that a lot of LLM scrapers are fetching are git repositories. They could just use git clone to fetch everything at once. But instead, they fet

One of the things that a lot of LLM scrapers are fetching are git repositories. They could just use git clone to fetch everything at once. But instead, they fetch them commit by commit. That's about as static as you can get, and it is absolutely NOT a non-issue.

LoganDark 8mo ago
No... Basically all git servers have to generate the file contents, diffs etc. on-demand because they don't store static pages for every single possible combination of view parameters. Git repositories also typically don't store full copies of all versions of a file that have ever existed either; they're incremental. You could pre-render everything statically, but that could take up gigabytes or more for any repo of non-trivial size.
- KolmogorovComp 8mo ago
  > Git repositories also typically don't store full copies of all versions of a file that have ever existed either; they're incremental This is wrong. Git does store full copies.
  - meatmanek 8mo ago
    git stores files as objects, which are stored as full copies, unless those objects are stored in packfiles and are deltified, in which case they're stored as deltas. https://codewords.recurse.com/issues/three/unpacking-git-packfiles https://codewords.recurse.com/issues/three/unpacking-git-pac...
    - KolmogorovComp 8mo ago
      Thank you for the insights.
    - PaulDavisThe1st 8mo ago
      ... which, in the context that is being discussed, is unusual.
neoromantique 8mo ago
that's a pretty niche issue, but fairly easy to solve. Prebuild statically the most common commits (last XX) and heavily rate limit deeper ones
- PaulDavisThe1st 8mo ago
  1. that doesn't appear to match the fetching patterns of the scrapers at all 2. 1M independent IPs hitting random commits from across a 25 year history is not, in fact, "easy to solve". It is addressable, but not easy ... 3. why should I have to do anything at all to deal with these scrapers? why is the onus not on them to do the right thing?
  - neoromantique 8mo ago
    I did not imply that it does, I meant to have a budget allocated for 'unauthenticated deep history queries', when it's over it's over and you only handle dynamic fetching for authorized users until cooldown. Is it pretty? No, but it also is a pretty niche thing overall (git repo storage).

LoganDark 8mo ago

No... Basically all git servers have to generate the file contents, diffs etc. on-demand because they don't store static pages for every single possible combination of view parameters. Git repositories also typically don't store full copies of all versions of a file that have ever existed either; they're incremental. You could pre-render everything statically, but that could take up gigabytes or more for any repo of non-trivial size.

KolmogorovComp 8mo ago
> Git repositories also typically don't store full copies of all versions of a file that have ever existed either; they're incremental This is wrong. Git does store full copies.
- meatmanek 8mo ago
  git stores files as objects, which are stored as full copies, unless those objects are stored in packfiles and are deltified, in which case they're stored as deltas. https://codewords.recurse.com/issues/three/unpacking-git-packfiles https://codewords.recurse.com/issues/three/unpacking-git-pac...
  - KolmogorovComp 8mo ago
    Thank you for the insights.
  - PaulDavisThe1st 8mo ago
    ... which, in the context that is being discussed, is unusual.

neoromantique 8mo ago

that's a pretty niche issue, but fairly easy to solve. Prebuild statically the most common commits (last XX) and heavily rate limit deeper ones

PaulDavisThe1st 8mo ago
1. that doesn't appear to match the fetching patterns of the scrapers at all 2. 1M independent IPs hitting random commits from across a 25 year history is not, in fact, "easy to solve". It is addressable, but not easy ... 3. why should I have to do anything at all to deal with these scrapers? why is the onus not on them to do the right thing?
- neoromantique 8mo ago
  I did not imply that it does, I meant to have a budget allocated for 'unauthenticated deep history queries', when it's over it's over and you only handle dynamic fetching for authorized users until cooldown. Is it pretty? No, but it also is a pretty niche thing overall (git repo storage).