5 ms·

Go Production Performance Gotcha – GOMAXPROCS

57 points by cbatt 2y ago

pickledish 2y ago
Another potential avenue for problems like this, which I'm a fan of, is taking advantage of k8s's static CPU policy: https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/ https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana... Using this (plus guaranteed QoS), you end up with containers which can only even "see" a subset of cores of the whole node (and they get those cores all to themselves), which is great for reducing noisy neighbors when big machines are running many different services.
- cbatt 2y ago
  Ah interesting. I'll have to dive in deeper here. If I understand correctly this essentially gives you exclusive core pinning? Do you find that you this reduces total utilization when workloads burst but they don't leverage the other unused cores?
- ewidar 2y ago
  I am assuming that in OP's case, they want their go process to "see" the machine as it is though, to surface more accurate/better stats? Interesting link nonetheless, thanks!
- 0cf8612b2e1e 2y ago
  I thought this was why you are supposed to use ‘nproc’ instead of manually parsing cpuinfo or some other mechanism to determine CPU count. There are various ways in which a given process can be limited to a subset of system resources.
dakiol 2y ago
Rant mode on. This is what software engineering looks like nowadays (at least based on my last jobs). The times I discuss truly software engineering topics (modelling, abstractions, algorithms, etc.) is significantly less than the times I discuss about: k8s shenanigans, Go/Java/etc. intricate details, library incompatibilities, build systems, S3 access rights, etc. I don’t like it. I used to work in more interesting topics when I was junior, precisely because there were at most 3 components involved: the db, the server framework, the client side. So I was spending time in the core of the business (business logic). There was no need to scale, no sharding, no k8s, no aws, no failover. We were making money for the company (not like nowadays where every single company I work for is a “unicorn” and is not profitable)
- andrewstuart2 2y ago
  I think this is what software engineering has almost always looked like. Computer science is definitely more pure and at the cutting edge of knowledge and implementations, but software engineering on the other hand is about understanding the tools available to you for your budget, and picking ones that will leave you with enough tolerances for your workload plus or minus expected variances, while expending the least of your budget. The longer you've been in your career, the more you have experienced the existing tools that are out there and can discuss those topics and weed out irrelevant details that some marketing department might tout because it's their tool's most favorable angle. In short, computer science is algorithms, data structures, etc. Engineering is those things applied for a given set of constraints including time and budgetary constraints. Or at least that's how I've come to define the differences. I generally tell my family my job is more like lego than anything else. It's about knowing the pieces available and how to put them together into something cool. And occasionally, you do get to build a new piece that brings the whole set together nicely if the brick you need doesn't exist yet.
  - jerf 2y ago
    Computer science has never been software engineering, though there's been a lot of cross-pollination. But there's still a world of difference in my opinion between "how can I put all these tools together to deliver a solution to the problem" versus "oh crap, the service is running out of CPU again, argh, my service provider changed how we specify CPUs because the old way wasn't good for someone who isn't me, and okay, I'm upgrading, and oh shit the upgrade also changes how we get to the encryption keys and now it's busted, let's revert, what do you mean it irrevocably upgraded our key store to the new version and now the old version doesn't work, fix that then, oh, we can't because that's all in the cloud and we missed the upgrade emails in our other big pile of emails argh argh argh fine, call an incident and bring in half the teams in the company". The latter was not created in the past 5 years, but it has gotten noticeably worse. When things work, they work better than they did before, but when things fail, they fail harder, in the sense that they can create much more complicated snarls than they used to. I much prefer even dull requirements elicitation meetings to too much of the latter.
- 2y ago
pentaphobe 2y ago
Happened to be experimenting with similar things at day job recently - very keen to see more Alas (OFF TOPIC) clicking the burger menu on the site (mobile) gives a rather novel fullscreen black error > "Application error: a client-side exception has occurred (see the browser console for more information)."
- cbatt 2y ago
  Well that's a little embarrassing, fixing now.
  - cbatt 2y ago
    fixed.
    - pentaphobe 2y ago
      super-fast! Looking forward to having full-brain time, super intriguing to see folk so much further down the eBPF tracing path Best of luck with the offering!
AaronBBrown 2y ago
In busy systems, GOMAXPROCS=cpu.limits will still lead to the process exceeding its cfs quota and be throttled as the Go runtime has additional threads beyond just GOMAXPROCS. IME, the only way to (nearly) guarantee there's no throttling is to set cpu.limits=GOMAXPROCS+1 to leave some room for these system threads. Unfortunately, there's broad adoption of uber/automaxprocs (which makes the same error) and utilizing the downward API to do this for other cases.
neonsunset 2y ago
Unthinkable crutches in Go land continue. This is something that needs to be fixed in the runtime itself to be appropriately container-aware and not require the users to write their own libraries to patch this out. For example: https://github.com/dotnet/runtime/pull/99508 https://github.com/dotnet/runtime/pull/99508 (alternatively, the goroutine runtime could have been made auto-scalable which would have reduced the impact at the cost of implementation complexity, like .net's threadpool is)
deleted 2y ago
[deleted]
Attummm 2y ago
This isn't a gotcha, it's an important aspect of how Go runs. While it's not highlighted enough concurrency isn't magic, and GOMAXPROCS is important to control your Go app, especially within production environments. Although it's not well known benchmarks for programming languages would show even faster results for Go with GOMAXPROCS set to 1. This lets single-threaded benchmarks run with less overhead. A real missed opportunity in communication. Because this isn't the first time we see articles like these pop up.
- binary132 2y ago
  It’s a containers / cgroups / linux problem. In the context of cgroups limiting CPU count usable within a given container, where an application in the container wants to ask the host kernel how many CPUs it can use, the application should have access to a system API that is informed by the cgroups configuration, but it doesn’t, at least not portably by default.

57 points by cbatt 2y ago

Another potential avenue for problems like this, which I'm a fan of, is taking advantage of k8s's static CPU policy: https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/ https://kubernetes.io/docs/tasks/administer-cluster/cpu-mana... Using this (plus guaranteed QoS), you end up with containers which can only even "see" a subset of cores of the whole node (and they get those cores all to themselves), which is great for reducing noisy neighbors when big machines are running many different services.

cbatt 2y ago
Ah interesting. I'll have to dive in deeper here. If I understand correctly this essentially gives you exclusive core pinning? Do you find that you this reduces total utilization when workloads burst but they don't leverage the other unused cores?
ewidar 2y ago
I am assuming that in OP's case, they want their go process to "see" the machine as it is though, to surface more accurate/better stats? Interesting link nonetheless, thanks!
0cf8612b2e1e 2y ago
I thought this was why you are supposed to use ‘nproc’ instead of manually parsing cpuinfo or some other mechanism to determine CPU count. There are various ways in which a given process can be limited to a subset of system resources.

dakiol 2y ago

Rant mode on. This is what software engineering looks like nowadays (at least based on my last jobs). The times I discuss truly software engineering topics (modelling, abstractions, algorithms, etc.) is significantly less than the times I discuss about: k8s shenanigans, Go/Java/etc. intricate details, library incompatibilities, build systems, S3 access rights, etc. I don’t like it. I used to work in more interesting topics when I was junior, precisely because there were at most 3 components involved: the db, the server framework, the client side. So I was spending time in the core of the business (business logic). There was no need to scale, no sharding, no k8s, no aws, no failover. We were making money for the company (not like nowadays where every single company I work for is a “unicorn” and is not profitable)

andrewstuart2 2y ago
I think this is what software engineering has almost always looked like. Computer science is definitely more pure and at the cutting edge of knowledge and implementations, but software engineering on the other hand is about understanding the tools available to you for your budget, and picking ones that will leave you with enough tolerances for your workload plus or minus expected variances, while expending the least of your budget. The longer you've been in your career, the more you have experienced the existing tools that are out there and can discuss those topics and weed out irrelevant details that some marketing department might tout because it's their tool's most favorable angle. In short, computer science is algorithms, data structures, etc. Engineering is those things applied for a given set of constraints including time and budgetary constraints. Or at least that's how I've come to define the differences. I generally tell my family my job is more like lego than anything else. It's about knowing the pieces available and how to put them together into something cool. And occasionally, you do get to build a new piece that brings the whole set together nicely if the brick you need doesn't exist yet.
- jerf 2y ago
  Computer science has never been software engineering, though there's been a lot of cross-pollination. But there's still a world of difference in my opinion between "how can I put all these tools together to deliver a solution to the problem" versus "oh crap, the service is running out of CPU again, argh, my service provider changed how we specify CPUs because the old way wasn't good for someone who isn't me, and okay, I'm upgrading, and oh shit the upgrade also changes how we get to the encryption keys and now it's busted, let's revert, what do you mean it irrevocably upgraded our key store to the new version and now the old version doesn't work, fix that then, oh, we can't because that's all in the cloud and we missed the upgrade emails in our other big pile of emails argh argh argh fine, call an incident and bring in half the teams in the company". The latter was not created in the past 5 years, but it has gotten noticeably worse. When things work, they work better than they did before, but when things fail, they fail harder, in the sense that they can create much more complicated snarls than they used to. I much prefer even dull requirements elicitation meetings to too much of the latter.
2y ago

pentaphobe 2y ago

Happened to be experimenting with similar things at day job recently - very keen to see more Alas (OFF TOPIC) clicking the burger menu on the site (mobile) gives a rather novel fullscreen black error > "Application error: a client-side exception has occurred (see the browser console for more information)."

cbatt 2y ago
Well that's a little embarrassing, fixing now.
- cbatt 2y ago
  fixed.
  - pentaphobe 2y ago
    super-fast! Looking forward to having full-brain time, super intriguing to see folk so much further down the eBPF tracing path Best of luck with the offering!

AaronBBrown 2y ago

In busy systems, GOMAXPROCS=cpu.limits will still lead to the process exceeding its cfs quota and be throttled as the Go runtime has additional threads beyond just GOMAXPROCS. IME, the only way to (nearly) guarantee there's no throttling is to set cpu.limits=GOMAXPROCS+1 to leave some room for these system threads. Unfortunately, there's broad adoption of uber/automaxprocs (which makes the same error) and utilizing the downward API to do this for other cases.

neonsunset 2y ago

Unthinkable crutches in Go land continue. This is something that needs to be fixed in the runtime itself to be appropriately container-aware and not require the users to write their own libraries to patch this out. For example: https://github.com/dotnet/runtime/pull/99508 https://github.com/dotnet/runtime/pull/99508 (alternatively, the goroutine runtime could have been made auto-scalable which would have reduced the impact at the cost of implementation complexity, like .net's threadpool is)

deleted 2y ago

[deleted]

Attummm 2y ago

This isn't a gotcha, it's an important aspect of how Go runs. While it's not highlighted enough concurrency isn't magic, and GOMAXPROCS is important to control your Go app, especially within production environments. Although it's not well known benchmarks for programming languages would show even faster results for Go with GOMAXPROCS set to 1. This lets single-threaded benchmarks run with less overhead. A real missed opportunity in communication. Because this isn't the first time we see articles like these pop up.

binary132 2y ago
It’s a containers / cgroups / linux problem. In the context of cgroups limiting CPU count usable within a given container, where an application in the container wants to ask the host kernel how many CPUs it can use, the application should have access to a system API that is informed by the cgroups configuration, but it doesn’t, at least not portably by default.