6 ms·

Regardless of whether or not Drew is abrasive (I've never dealt with him so have no opinion), this is on Google. If a single service (Go's module crawler) is d

Regardless of whether or not Drew is abrasive (I've never dealt with him so have no opinion), this is on Google. If a single service (Go's module crawler) is doing this to multiple third parties (and it is, by design) then requiring Drew to implement a workaround is the same thing as relying on every (relevant to the issue) third party to do the same thing. Which is bad design. This should be fixed at source or switched off. As for the workaround itself it's true that going on a crawler exclusion list will not stop SourceHut being usable, only make it a little stale, but that doesn't matter. The problem is not at Drew's end. And once again for this workaround to be an effective solution every (relevant to the issue) third party needs to do the same thing. Which is bad design. As a general rule, with excessive polling the system doing the polling is the one at fault. And as a general rule if one system impacts many fix at the originator. If Google devs don't know this, then why on earth are they getting the salaries they do?

tptacek 4y ago
Can you help me understand the problem with the workaround, which is precisely to have Google's proxy not excessively poll DeVault's service? It really seems like DeVault's real argument here isn't about the impact of this on his service, but that he doesn't like the design of the proxy. He has a lot of standing to complain about impacts on his service, but essentially no standing to complain about designs he finds sub-par.
- WhyNotHugo 4y ago
  The proxy _is_ pretty nasty design. Basically every time `golang` fetches modules, it uses Google's proxy out of the box. This lets Google collect very granular data on who uses which modules, and also makes it opaque to webmasters who's using their services (it's all hidden behind google). It's honestly pretty nasty how Google has hidden this kind of "analytics" within a pretty dark grey area. At the same time, the bot that handles the proxy seems to crawl everything, very often, and excessively, effectively DDOS'ing anyone hosting go modules.
- kcartlidge 4y ago
  I can't really speak for Drew's problem with the workaround, only my own from a design point of view - he may not even agree with my own issue at all for all I know, so I don't want to put words in his mouth. From my own perspective, however, the issue is about impact and responsibility. The excess traffic is their impact as it is caused by their design and is entirely of their choosing. Thus it becomes their responsibility. The workaround they offer (as with so much in tech) means their stuff becomes opt-out and not opt-in. By which I mean their own decisions/designs cause an impact where they require everyone else suffering from it to specifically and individually opt-out of being excessively polled. This is an outrageous requirement they can only impose due to the power imbalance.
  - tptacek 4y ago
    It sounds like DeVault let them know their proxy was hitting his service too hard, and they immediately responded by saying they could dial it down. Seems like a pretty normal sequence of events? I feel like I have to be missing something here.
    - jum1p 4y ago
      I can't speak for Devault, either, but I think that Sourcehut is intended to be selfhosted if you want. So it's probably too much pain to ask every language provider (might be a lot) to dial down the traffic. Imagine youre hosting modules from ~200 langs.
    - philosopher1234 4y ago
      This is probably 80% peoples' existing animosity towards Google & Go, so they're ready to hate, regardless of how the facts land. I think its primarily about the Go team having a different working philosophy than the standard HN soft. dev philosophy, and an intolerance of difference.
      - kcartlidge 4y ago
        > This is probably 80% peoples' existing animosity towards Google & Go, so they're ready to hate, regardless of how the facts land. Go is my second fave after C# and I've used it almost daily for many years. No animosity here.
- voxtech 4y ago
  If the workaround didn't degrade service for Drew's go users, google would just implement it for all repositories.
  - tptacek 4y ago
    How exactly does this workaround degrade service for DeVault's Go users? I'm not saying it doesn't; I'm saying nobody has given a simple explanation of what the impact would actually be, except for some people who have given clearly false explanations. It's OK to just not know, but if you don't know, it's weird to have strong opinions about it.
    - jum1p 4y ago
      > How exactly does this workaround degrade service for DeVault's Go users? I think you don't always get the latest stuff due to caching or so.
    - voxtech 4y ago
      I'm more weirded out by the fact that you believe google is performing this DDOS for no actual benefit and are choosing to defend it anyway.
      - tptacek 4y ago
        Is that an answer to some other question I asked? It doesn't seem to be responsive to what I just wrote. I feel like what I'm sticking up for here is the practice of software development. Building an automated system that generates unexpectedly unwelcome load on someone else's service is... not exactly front-page news? It happens basically all the time? The idea that because the Go team is sponsored by Google, nothing like this should ever happen seems deeply unrealistic. "We can push a button to make this stop happening; the tradeoff is that other hosting services that don't care about this load might currently have fresher cache entries [whatever that means]" seems like a perfectly cromulent response. He should just tell them to push the button. If you think he shouldn't, you should be able to say why. I do have a mild rooting interest here: I think the Go module proxy is pretty neat, and does something interesting for the security of the ecosystem. DeVault disagrees. That's fine, disagreement keeps things interesting.
- smw 4y ago
  You shouldn't have to opt-out of a DOS.
  - tptacek 4y ago
    You don't; this isn't a DOS.

tptacek 4y ago

Can you help me understand the problem with the workaround, which is precisely to have Google's proxy not excessively poll DeVault's service? It really seems like DeVault's real argument here isn't about the impact of this on his service, but that he doesn't like the design of the proxy. He has a lot of standing to complain about impacts on his service, but essentially no standing to complain about designs he finds sub-par.

WhyNotHugo 4y ago
The proxy _is_ pretty nasty design. Basically every time `golang` fetches modules, it uses Google's proxy out of the box. This lets Google collect very granular data on who uses which modules, and also makes it opaque to webmasters who's using their services (it's all hidden behind google). It's honestly pretty nasty how Google has hidden this kind of "analytics" within a pretty dark grey area. At the same time, the bot that handles the proxy seems to crawl everything, very often, and excessively, effectively DDOS'ing anyone hosting go modules.
kcartlidge 4y ago
I can't really speak for Drew's problem with the workaround, only my own from a design point of view - he may not even agree with my own issue at all for all I know, so I don't want to put words in his mouth. From my own perspective, however, the issue is about impact and responsibility. The excess traffic is their impact as it is caused by their design and is entirely of their choosing. Thus it becomes their responsibility. The workaround they offer (as with so much in tech) means their stuff becomes opt-out and not opt-in. By which I mean their own decisions/designs cause an impact where they require everyone else suffering from it to specifically and individually opt-out of being excessively polled. This is an outrageous requirement they can only impose due to the power imbalance.
- tptacek 4y ago
  It sounds like DeVault let them know their proxy was hitting his service too hard, and they immediately responded by saying they could dial it down. Seems like a pretty normal sequence of events? I feel like I have to be missing something here.
  - jum1p 4y ago
    I can't speak for Devault, either, but I think that Sourcehut is intended to be selfhosted if you want. So it's probably too much pain to ask every language provider (might be a lot) to dial down the traffic. Imagine youre hosting modules from ~200 langs.
  - philosopher1234 4y ago
    This is probably 80% peoples' existing animosity towards Google & Go, so they're ready to hate, regardless of how the facts land. I think its primarily about the Go team having a different working philosophy than the standard HN soft. dev philosophy, and an intolerance of difference.
    - kcartlidge 4y ago
      > This is probably 80% peoples' existing animosity towards Google & Go, so they're ready to hate, regardless of how the facts land. Go is my second fave after C# and I've used it almost daily for many years. No animosity here.
voxtech 4y ago
If the workaround didn't degrade service for Drew's go users, google would just implement it for all repositories.
- tptacek 4y ago
  How exactly does this workaround degrade service for DeVault's Go users? I'm not saying it doesn't; I'm saying nobody has given a simple explanation of what the impact would actually be, except for some people who have given clearly false explanations. It's OK to just not know, but if you don't know, it's weird to have strong opinions about it.
  - jum1p 4y ago
    > How exactly does this workaround degrade service for DeVault's Go users? I think you don't always get the latest stuff due to caching or so.
  - voxtech 4y ago
    I'm more weirded out by the fact that you believe google is performing this DDOS for no actual benefit and are choosing to defend it anyway.
    - tptacek 4y ago
      Is that an answer to some other question I asked? It doesn't seem to be responsive to what I just wrote. I feel like what I'm sticking up for here is the practice of software development. Building an automated system that generates unexpectedly unwelcome load on someone else's service is... not exactly front-page news? It happens basically all the time? The idea that because the Go team is sponsored by Google, nothing like this should ever happen seems deeply unrealistic. "We can push a button to make this stop happening; the tradeoff is that other hosting services that don't care about this load might currently have fresher cache entries [whatever that means]" seems like a perfectly cromulent response. He should just tell them to push the button. If you think he shouldn't, you should be able to say why. I do have a mild rooting interest here: I think the Go module proxy is pretty neat, and does something interesting for the security of the ecosystem. DeVault disagrees. That's fine, disagreement keeps things interesting.
smw 4y ago
You shouldn't have to opt-out of a DOS.
- tptacek 4y ago
  You don't; this isn't a DOS.