8 ms·

According to some comments in the linked GitHub issue, including [1] from last May, Drew could have simply asked to be excluded from automatic refresh traffic f

by comex 4y ago

According to some comments in the linked GitHub issue, including [1] from last May, Drew could have simply asked to be excluded from automatic refresh traffic from the mirror. If I understand correctly, that would still leave traffic from the mirror when it’s acting as a direct proxy for someone’s request, but that is traffic that would be going to sr.ht regardless. For some reason he did not do this and instead chose an option that causes breakage. [1] https://github.com/golang/go/issues/44577#issuecomment-1137822116 https://github.com/golang/go/issues/44577#issuecomment-11378...

deleted 4y ago
[deleted]
FiloSottile 4y ago
I can confirm asking to be excluded from refreshes (which AFAIK is still a standing offer, but I obviously can't speak for the mirror team because I am not at Google anymore) would stop all automated traffic from the mirror, and that sr.ht could send a simple HTTP GET to proxy.golang.org for new tags if it wished not to wait for users to request new versions.
throwaway_62022 4y ago
I think it is problematic that we are using github issues as "support forum" for asking a git host provider to be excluded from the refresh list. This should not have come to that. Whatever happened to "reasonable defaults", so as a random person hosting a single Go module doesn't get DOSed - https://github.com/golang/go/issues/44577#issuecomment-860877028 https://github.com/golang/go/issues/44577#issuecomment-86087... ?
- FiloSottile 4y ago
  Everyone can make their own assessment of what is a reasonable default and what counts as a DoS (and they are welcome to opt-out of any traffic), but note that 4GB per day is 0.3704 Mbps.
  - nemith 4y ago
    Opt-out is still manual and undocumented process? I am very concerned that "own assessment" of what is a DoS means that source code is expected to be hosted only on large platform or by large corporation which is another way to say that "the little guys don't matter". Self hosting of source code should be an option and the proxy should be there to reduce the traffic load, not amplify or artificially increase that load despite the "level of DoS". One thing Drew is asking for is to respect robots.txt to allow the operator to determine what a reasonable level is for that operator and not apply a github bias to it.
  - jallbrit 4y ago
    Per Go module repository, no?
  - 0cf8612b2e1e 4y ago
    > A single module can produce as much as 4 GiB of daily traffic from Google. That’s (upper bound) 4Gib times 2500 per hour. That’s not nothing.
  - throwaway_62022 4y ago
    Comes around $8-11 of egress monthly traffic on AWS. I would think twice before signing up for a service that charges me $10/month - not sure why this should be any different. Also, how do you opt-out? Imagine a random developer in a startup, running a Gitlab instance and then pushing a Go module there and only to be left with inexplicable traffic pattern(and bill). I have no skin in the game but this default _does not_ sound reasonable to me, whichever way you slice it.
    - morelisp 4y ago
      As an aside, I have done this and I do not have the problem. The most interesting question for me in all of this has always been, what triggers Google's request load?
    - treis 4y ago
      AWS gouges the crap out of people with their egress fees, though. You can get 20TB of transfer for ~$5 a month from Hetzner plus a server to go with it.
stonogo 4y ago
Seems like a weird default policy, until you remember Google's wifi mapping opt-out process: https://support.google.com/maps/answer/1725632?hl=en#zippy=%2Chow-do-i-opt-my-access-point-out-of-google-location-services https://support.google.com/maps/answer/1725632?hl=en#zippy=%... "All this would go away if other people would just do what they're told" is a pretty dystopian policy, but it seems to be a popular choice in Google.
- kramerger 4y ago
  That right there is why i stopped using Google. Anyway, things get a bit kafka-esque when you realise that there is another company doing this WiFi thing and to opt out from that one you need a different SSID suffix. Since you can't have both, you end up with at least one company data mining you. Why GDPR has not put a stop to this is baffling
  - samwyse 4y ago
    I’ve got to ask, which other Wi-Fi mapping provider requires a SSID suffix to opt out? Is it one of the big boys or is it someone like wigle.net or openwifimap.net?
tristan957 4y ago
Should every person that hosts an instance of SourceHut, Gitea, or Forgejo have to opt-in to this? That just doesn't scale at all. Drew is standing up for all independent hosters as much as he is standing up for his own business interests.
- jdiff 4y ago
  No other independent hosters are having issues. I'm on one with no more than a few dozen users. Multiple of us write in Go, including the owner of the service, and yet Gitea and the VPS hosting it have never even blinked. If there were more of an issue, there would be more of a fuss than just two individuals. And one of those individuals was completely satisfied with the temporarily hackjob while a more permanent solution has been in the works. The other just denied it with no reason ever given cause he wants to keep litigating the issue.
  - deelly 4y ago
    > No other independent hosters are having issues. I could be wrong but in comments to https://github.com/golang/go/issues/44577 https://github.com/golang/go/issues/44577 issue there at least 4 hosters that forced to manually disable background refresh because of this exactly issue?
    - jdiff 4y ago
      How are you getting that number? There's Drew, who refused all solutions offered. There's Ben, who took the offer and was satisfied. If you're counting Gentoo, which is linked, you shouldn't because that's an unrelated issue caused by a regression test.

by comex 4y ago

deleted 4y ago

[deleted]

FiloSottile 4y ago

I can confirm asking to be excluded from refreshes (which AFAIK is still a standing offer, but I obviously can't speak for the mirror team because I am not at Google anymore) would stop all automated traffic from the mirror, and that sr.ht could send a simple HTTP GET to proxy.golang.org for new tags if it wished not to wait for users to request new versions.

throwaway_62022 4y ago

I think it is problematic that we are using github issues as "support forum" for asking a git host provider to be excluded from the refresh list. This should not have come to that. Whatever happened to "reasonable defaults", so as a random person hosting a single Go module doesn't get DOSed - https://github.com/golang/go/issues/44577#issuecomment-860877028 https://github.com/golang/go/issues/44577#issuecomment-86087... ?

FiloSottile 4y ago
Everyone can make their own assessment of what is a reasonable default and what counts as a DoS (and they are welcome to opt-out of any traffic), but note that 4GB per day is 0.3704 Mbps.
- nemith 4y ago
  Opt-out is still manual and undocumented process? I am very concerned that "own assessment" of what is a DoS means that source code is expected to be hosted only on large platform or by large corporation which is another way to say that "the little guys don't matter". Self hosting of source code should be an option and the proxy should be there to reduce the traffic load, not amplify or artificially increase that load despite the "level of DoS". One thing Drew is asking for is to respect robots.txt to allow the operator to determine what a reasonable level is for that operator and not apply a github bias to it.
- jallbrit 4y ago
  Per Go module repository, no?
- 0cf8612b2e1e 4y ago
  > A single module can produce as much as 4 GiB of daily traffic from Google. That’s (upper bound) 4Gib times 2500 per hour. That’s not nothing.
- throwaway_62022 4y ago
  Comes around $8-11 of egress monthly traffic on AWS. I would think twice before signing up for a service that charges me $10/month - not sure why this should be any different. Also, how do you opt-out? Imagine a random developer in a startup, running a Gitlab instance and then pushing a Go module there and only to be left with inexplicable traffic pattern(and bill). I have no skin in the game but this default _does not_ sound reasonable to me, whichever way you slice it.
  - morelisp 4y ago
    As an aside, I have done this and I do not have the problem. The most interesting question for me in all of this has always been, what triggers Google's request load?
  - treis 4y ago
    AWS gouges the crap out of people with their egress fees, though. You can get 20TB of transfer for ~$5 a month from Hetzner plus a server to go with it.

stonogo 4y ago

Seems like a weird default policy, until you remember Google's wifi mapping opt-out process: https://support.google.com/maps/answer/1725632?hl=en#zippy=%2Chow-do-i-opt-my-access-point-out-of-google-location-services https://support.google.com/maps/answer/1725632?hl=en#zippy=%... "All this would go away if other people would just do what they're told" is a pretty dystopian policy, but it seems to be a popular choice in Google.

kramerger 4y ago
That right there is why i stopped using Google. Anyway, things get a bit kafka-esque when you realise that there is another company doing this WiFi thing and to opt out from that one you need a different SSID suffix. Since you can't have both, you end up with at least one company data mining you. Why GDPR has not put a stop to this is baffling
- samwyse 4y ago
  I’ve got to ask, which other Wi-Fi mapping provider requires a SSID suffix to opt out? Is it one of the big boys or is it someone like wigle.net or openwifimap.net?

tristan957 4y ago

Should every person that hosts an instance of SourceHut, Gitea, or Forgejo have to opt-in to this? That just doesn't scale at all. Drew is standing up for all independent hosters as much as he is standing up for his own business interests.

jdiff 4y ago
No other independent hosters are having issues. I'm on one with no more than a few dozen users. Multiple of us write in Go, including the owner of the service, and yet Gitea and the VPS hosting it have never even blinked. If there were more of an issue, there would be more of a fuss than just two individuals. And one of those individuals was completely satisfied with the temporarily hackjob while a more permanent solution has been in the works. The other just denied it with no reason ever given cause he wants to keep litigating the issue.
- deelly 4y ago
  > No other independent hosters are having issues. I could be wrong but in comments to https://github.com/golang/go/issues/44577 https://github.com/golang/go/issues/44577 issue there at least 4 hosters that forced to manually disable background refresh because of this exactly issue?
  - jdiff 4y ago
    How are you getting that number? There's Drew, who refused all solutions offered. There's Ben, who took the offer and was satisfied. If you're counting Gentoo, which is linked, you shouldn't because that's an unrelated issue caused by a regression test.