3 ms·

> No matter what load you have, P99 and especially P99.9 latency will be higher than any other language I strongly call BS on that. Strong claim and evidence

by kjksf 7mo ago

> No matter what load you have, P99 and especially P99.9 latency will be higher than any other language I strongly call BS on that. Strong claim and evidence seems to be a hallucination in your own head. There are several writeups of large backends ported from node/python/ruby to Go which resulted in dramatic speedups, including drop in P99 and P99.9 latencies by 10x That's empirical evidence your claim is BS. What exactly is so unfair about Go scheduler and what do you compare it to? Node's lack of multi-threading? Python's and Ruby's GIL? Just leaving this to OS thread scheduler which, unlike Go, has no idea about i/o and therefore cannot optimize for it? Apparently the source of your claim is https://github.com/php/frankenphp/pull/2016 https://github.com/php/frankenphp/pull/2016 Which is optimizing for a very specific micro-benchmark of hammering std-lib http server with concurrent request. Which is not what 99% of go servers need to handle. And is exercising way more than a scheduler. And is not benchmarking against any other language, so the sweeping statement about "higher than any other language" is literally baseless. And you were able to make a change that trades throughput for P99 latency without changing the scheduler, which kind of shows it wasn't the scheduler but an interaction between a specific implementation of HTTP server and Go scheduler. And there are other HTTP servers in Go that focus on speed. It's just 99.9% of Go servers don't need any of that because the baseline is 10x faster than python/ruby/javascript and on-par with Java or C#.

withinboredom 7mo ago
Do I need to share the TLA+ spec that shows its unfair? Or do you have any actual proof to your claims?
- 9rx 7mo ago
  It would be helpful for you to share a link to the Github issue you created. If the TLA+ spec you no doubt put a lot of time into creating is contained there, that would be additionally amazing, but more relevant will be the responses from the maintainers so that we're not stuck with one side of the story. Of course, expecting you to provide the link would be incredibly onerous. We can look it up ourselves just as easy as you can. Well, in theory we can. The only trouble is that I cannot find the issue you are talking about. I cannot find any issues in the Go issue tracker from your account. So, in the interest of good faith, perhaps you can help us out this one time and point us in the right direction?
  - withinboredom 7mo ago
    I’m not interested in contributing to go. I tried once, was basically ignored. I have contributed to issues there where it has impacted projects I’ve worked on. But even then, it didn’t feel collaborative; mostly felt like dealing with a tech support team instead of other developers. That being said, I love studying go and learning how to use it to the best of my ability because I work on sub-ųs networking in go. When I get home, I’ll dig it up. But if you think it’s a fair scheduler, I invite you to just think about it on a whiteboard for a few minutes. It’s nowhere near fair and should be self-evident from first principles alone.
    - withinboredom 7mo ago
      Here’s a much better write up than I’m willing to do: https://www.cockroachlabs.com/blog/rubbing-control-theory/ https://www.cockroachlabs.com/blog/rubbing-control-theory/ There are also multiple issues about this on GitHub. And an open issue that is basically been ignored. golang/go#51071 Like I said. Go won’t fix this because they’ve optimized for throughput at the expense of everything else, which means higher tail latencies. They’d have to give up throughput for lower latency.
      - 9rx 7mo ago
        > And an open issue that is basically been ignored. golang/go#51071 It doesn't look ignored to me. It explains that the test coverage is currently poor, so they are in a terrible position of not being able to make changes until that is rectified. The first step is to improve the test coverage. Are you volunteering? AI isn't at a point where it is going to magically do it on its own, so it is going to take a willing human hand. You do certainly appear to be the perfect candidate, both having the technical understanding and the need for it.
        withinboredom 7mo ago
        Heh. I've had my fair share of mailing list drama. This is political AND technical. Someone saying "let’s cut throughput" is going to get shot down fast, no matter the technical merit. If someone with the political clout were to be willing to champion the work and guide the discussion appropriately while someone like me does the work, that's different. That's at least how things like this are done in other communities, unless go is different.
jerf 7mo ago
"There are several writeups of large backends ported from node/python/ruby to Go which resulted in dramatic speedups, including drop in P99 and P99.9 latencies by 10x" But that's not comparing apples to apples. When you get a dramatic speedup, you will also see big drops in the P99 and P99.9 latencies because what stressed out the scripting language is a yawn to a compiled language. Just going from stressed->yawning will do wonders for all your latencies, tail latencies included. That doesn't say anything about what will happen when the load increases enough to start stressing the compiled language.

by kjksf 7mo ago

withinboredom 7mo ago

Do I need to share the TLA+ spec that shows its unfair? Or do you have any actual proof to your claims?

9rx 7mo ago
It would be helpful for you to share a link to the Github issue you created. If the TLA+ spec you no doubt put a lot of time into creating is contained there, that would be additionally amazing, but more relevant will be the responses from the maintainers so that we're not stuck with one side of the story. Of course, expecting you to provide the link would be incredibly onerous. We can look it up ourselves just as easy as you can. Well, in theory we can. The only trouble is that I cannot find the issue you are talking about. I cannot find any issues in the Go issue tracker from your account. So, in the interest of good faith, perhaps you can help us out this one time and point us in the right direction?
- withinboredom 7mo ago
  I’m not interested in contributing to go. I tried once, was basically ignored. I have contributed to issues there where it has impacted projects I’ve worked on. But even then, it didn’t feel collaborative; mostly felt like dealing with a tech support team instead of other developers. That being said, I love studying go and learning how to use it to the best of my ability because I work on sub-ųs networking in go. When I get home, I’ll dig it up. But if you think it’s a fair scheduler, I invite you to just think about it on a whiteboard for a few minutes. It’s nowhere near fair and should be self-evident from first principles alone.
  - withinboredom 7mo ago
    Here’s a much better write up than I’m willing to do: https://www.cockroachlabs.com/blog/rubbing-control-theory/ https://www.cockroachlabs.com/blog/rubbing-control-theory/ There are also multiple issues about this on GitHub. And an open issue that is basically been ignored. golang/go#51071 Like I said. Go won’t fix this because they’ve optimized for throughput at the expense of everything else, which means higher tail latencies. They’d have to give up throughput for lower latency.
    - 9rx 7mo ago
      > And an open issue that is basically been ignored. golang/go#51071 It doesn't look ignored to me. It explains that the test coverage is currently poor, so they are in a terrible position of not being able to make changes until that is rectified. The first step is to improve the test coverage. Are you volunteering? AI isn't at a point where it is going to magically do it on its own, so it is going to take a willing human hand. You do certainly appear to be the perfect candidate, both having the technical understanding and the need for it.
      - withinboredom 7mo ago
        Heh. I've had my fair share of mailing list drama. This is political AND technical. Someone saying "let’s cut throughput" is going to get shot down fast, no matter the technical merit. If someone with the political clout were to be willing to champion the work and guide the discussion appropriately while someone like me does the work, that's different. That's at least how things like this are done in other communities, unless go is different.

jerf 7mo ago

"There are several writeups of large backends ported from node/python/ruby to Go which resulted in dramatic speedups, including drop in P99 and P99.9 latencies by 10x" But that's not comparing apples to apples. When you get a dramatic speedup, you will also see big drops in the P99 and P99.9 latencies because what stressed out the scripting language is a yawn to a compiled language. Just going from stressed->yawning will do wonders for all your latencies, tail latencies included. That doesn't say anything about what will happen when the load increases enough to start stressing the compiled language.