5 ms·

GitHub is at the point where it immediately rate limits me if I try to look at a project's commit history without being logged in, as in the first time I even o

GitHub is at the point where it immediately rate limits me if I try to look at a project's commit history without being logged in, as in the first time I even open a single URL to the commit history, I get "Too Many Requests" from GitHub thrown at me. I don't know if my work's antivirus stack is causing GitHub to be suspicious of me, but it's definitely egregious.

vhcr 6mo ago
My theory is that they rate limit that URL aggressively due to AI scrapers. At this point it's faster to just clone the repo and do your searching locally.
colechristensen 6mo ago
Your work is probably all exiting through the same IP, you competing with others on the same IP is causing the rate limit.
- CupricTea 6mo ago
  I've considered this, but the company is small enough that the number of people who would be on GitHub at any moment (instead of our internal git forge) can be counted on one hand, and when I'm the first one there in the morning it still rate limits me.
  - growse 6mo ago
    Maybe your company's ISP is CGNat'ting you?
  - NewJazz 6mo ago
    Do you have any on-prem cicd jobs that access github? Our's kept failing, had to move over to the ECR release of some stuff.
  - colechristensen 6mo ago
    Hm, I've also noticed sites being more aggressive about verifications after I started using LLMs locally. They think I'm a bot (which... fair), even on completely unrelated sites I seem to be getting prompted for human verification much more often.
- embedding-shape 6mo ago
  The very same thing happen on my residential connection, I can do one search query, then I'm rate limited for 15+ minutes, same if I access any list of commits.
- tyingq 6mo ago
  May explain the ipv6 resistance. Hard to do effective per-ip rate-limiting with v6.
  - nickserv 6mo ago
    I don't understand, wouldn't it make it easier?
    - hkt 6mo ago
      No, IPv6 as it is supposed to be implemented gives (say) a single server a /64, which is for all intents and purposes an inexhaustible supply of IPs. You could in principle have an IP per site you visit and have plenty left to spare. Random Google result with a bit more: https://www.captaindns.com/en/blog/ipv6-subnet-sizes-48-vs-56-vs-64 https://www.captaindns.com/en/blog/ipv6-subnet-sizes-48-vs-5... So if I wanted to annoy GitHub, I could connect to them without ever using the same IP twice. Their response would have to be banning my /64, or possibly /56.
      - Hikikomori 6mo ago
        Its not harder to rate limit a /64 though.
        mmbleh 6mo ago
        It is because the IPv6 rollout has not been consistent. Some assign /64 per machine, some assign /64 per data center. Some even go the other way and do a /56 per machine. We've had to build up a list of overrides to do some ranges by /64 and others by /128 because of how they allocate addresses. This creates extra burden on server operators and it's not surprising that some just choose not to deal with it.
        GoblinSlayer 6mo ago
        This problem exists for ipv4 too: some machines have static address, others have dynamic, so you can implement overrides.
sholladay 6mo ago
It’s not you or your setup. I experience the same behavior. Tried with and without Private Relay, residential and commercial ISPs at different locations, and more to debug it. Same results. I think GitHub has just gotten so aggressive with their rate limit policies that it’s straight up incompatible with their own product. The charitable interpretation is that they aren’t keeping good track of how many requests each page actually performs in order to calibrate rate limiting.
- kdhaskjdhadjk 6mo ago
  On the other side of the coin, they also punish people who have slow connections. The acceptable speed for downloading from github on my connection is 90k/sec. No more, no less. Something prevents the rate from being higher (probably Github), and if the rate drops any lower for any length of time, the connection will suddenly abort right in the middle of the download. Since the dumpster fire that is git doesn't support resume, welcome to hell. If I didn't have a fast server elsewhere to git to then zip up and re-download, I'd be screwed.
- bmicraft 6mo ago
  If you didn't specifically test without it, I'd attribute that to cgnat

vhcr 6mo ago

My theory is that they rate limit that URL aggressively due to AI scrapers. At this point it's faster to just clone the repo and do your searching locally.

colechristensen 6mo ago

Your work is probably all exiting through the same IP, you competing with others on the same IP is causing the rate limit.

CupricTea 6mo ago
I've considered this, but the company is small enough that the number of people who would be on GitHub at any moment (instead of our internal git forge) can be counted on one hand, and when I'm the first one there in the morning it still rate limits me.
- growse 6mo ago
  Maybe your company's ISP is CGNat'ting you?
- NewJazz 6mo ago
  Do you have any on-prem cicd jobs that access github? Our's kept failing, had to move over to the ECR release of some stuff.
- colechristensen 6mo ago
  Hm, I've also noticed sites being more aggressive about verifications after I started using LLMs locally. They think I'm a bot (which... fair), even on completely unrelated sites I seem to be getting prompted for human verification much more often.
embedding-shape 6mo ago
The very same thing happen on my residential connection, I can do one search query, then I'm rate limited for 15+ minutes, same if I access any list of commits.
tyingq 6mo ago
May explain the ipv6 resistance. Hard to do effective per-ip rate-limiting with v6.
- nickserv 6mo ago
  I don't understand, wouldn't it make it easier?
  - hkt 6mo ago
    No, IPv6 as it is supposed to be implemented gives (say) a single server a /64, which is for all intents and purposes an inexhaustible supply of IPs. You could in principle have an IP per site you visit and have plenty left to spare. Random Google result with a bit more: https://www.captaindns.com/en/blog/ipv6-subnet-sizes-48-vs-56-vs-64 https://www.captaindns.com/en/blog/ipv6-subnet-sizes-48-vs-5... So if I wanted to annoy GitHub, I could connect to them without ever using the same IP twice. Their response would have to be banning my /64, or possibly /56.
    - Hikikomori 6mo ago
      Its not harder to rate limit a /64 though.
      - mmbleh 6mo ago
        It is because the IPv6 rollout has not been consistent. Some assign /64 per machine, some assign /64 per data center. Some even go the other way and do a /56 per machine. We've had to build up a list of overrides to do some ranges by /64 and others by /128 because of how they allocate addresses. This creates extra burden on server operators and it's not surprising that some just choose not to deal with it.
        GoblinSlayer 6mo ago
        This problem exists for ipv4 too: some machines have static address, others have dynamic, so you can implement overrides.

sholladay 6mo ago

It’s not you or your setup. I experience the same behavior. Tried with and without Private Relay, residential and commercial ISPs at different locations, and more to debug it. Same results. I think GitHub has just gotten so aggressive with their rate limit policies that it’s straight up incompatible with their own product. The charitable interpretation is that they aren’t keeping good track of how many requests each page actually performs in order to calibrate rate limiting.

kdhaskjdhadjk 6mo ago
On the other side of the coin, they also punish people who have slow connections. The acceptable speed for downloading from github on my connection is 90k/sec. No more, no less. Something prevents the rate from being higher (probably Github), and if the rate drops any lower for any length of time, the connection will suddenly abort right in the middle of the download. Since the dumpster fire that is git doesn't support resume, welcome to hell. If I didn't have a fast server elsewhere to git to then zip up and re-download, I'd be screwed.
bmicraft 6mo ago
If you didn't specifically test without it, I'd attribute that to cgnat