4 ms·
This is subtly incorrect - as far as Docker is concerned CFS cgroup extension has several knobs to tune - cfs_quota_us, cfs_period_us (typical default is 100ms
by dilyevsky 3y ago
This is subtly incorrect - as far as Docker is concerned CFS cgroup extension has several knobs to tune - cfs_quota_us, cfs_period_us (typical default is 100ms not a second) and shares. When you set shares you get weighted proportional scheduling (but only when there's contention). The former two enforce strict quota. Don't use Docker's --cpu flag and instead use --cpu-shares to avoid (mostly useless) quota enforcement.
From Linux docs:
- cpu.shares: The weight of each group living in the same hierarchy, that
translates into the amount of CPU it is expected to get. Upon cgroup creation,
each group gets assigned a default of 1024. The percentage of CPU assigned to
the cgroup is the value of shares divided by the sum of all shares in all
cgroups in the same level.
- cpu.cfs_period_us: The duration in microseconds of each scheduler period, for
bandwidth decisions. This defaults to 100000us or 100ms. Larger periods will
improve throughput at the expense of latency, since the scheduler will be able
to sustain a cpu-bound workload for longer. The opposite of true for smaller
periods. Note that this only affects non-RT tasks that are scheduled by the
CFS scheduler.
- cpu.cfs_quota_us: The maximum time in microseconds during each cfs_period_us
in for the current group will be allowed to run. For instance, if it is set to
half of cpu_period_us, the cgroup will only be able to peak run for 50 % of
the time. One should note that this represents aggregate time over all CPUs
in the system. Therefore, in order to allow full usage of two CPUs, for
instance, one should set this value to twice the value of cfs_period_us.
- Thaxll 3y agoPeople using Kubernetes don't tune or change those settings, it's up to the app to behave properly.
- dilyevsky 3y agoFalse. Kubernetes cpu request sets the shares, cpu limit sets the cfs quota
- Thaxll 3y agoYou said to change docker flags. Anyway your post is irrelevant, the goal is to let know the runtime about how many posix threads should it use. If you set request / limit to 1 core but you run on 64 cores node , then you runtime will see that which will bring performance down.
- dilyevsky 3y agoOriginal article is about docker. That’s the point of my comment - dont set cpu limit
- riv991 3y agoI intended it to be applicable to all containerised environments. Docker is just easiest on my local machine. I still believe it's best to set these variables regardless of cpu limits and/or cpu shares
- dilyevsky 3y agoAll you did is kneecapped your app to have lower performance so it fits under your arbitrary limit. Hardly what most people describe as “best” - only useful in small percentage of usecases (like reselling compute)
- riv991 3y agoI've seen significant performance gains from this in production. Other people have encountered it too hence libraries like Automaxprocs existing and issues being open with Go for it.
- dilyevsky 3y agoGains by what metric? Are you sure you didn't trade in better latency for worse overall throughput? Also, sure you didn't hit one of many CFS overaccounting bugs which we've seen a few? Have you compared performance without the limit at all?
- riv991 3y agoHi I'm the blog author, thanks for the feedback I'll try and clarify this. I think this is how the sympton presents but I should be clearer.
- mratsim 3y ago> Don't use Docker's --cpu flag and instead use --cpu-shares to avoid (mostly useless) quota enforcement. One caveat is that an application can detect when --cpu is used as I think it's using cpuset. When quota are used it cannot detect and more threads than necessary will likely be spawned
- cpuguy83 3y agoIt is not using cpuset (there is a separate flag for this). --cpus tweaks the cfs quota based on the number of cpus on the system and the requested amount.
- dilyevsky 3y ago—cpu sets the quota, there is is a —cpuset-cpu flag for cpuset and you can detect both by looking at the /sys/fs/cgroup
- cpuguy83 3y ago> "Don't use Docker's --cpu flag and instead use" This is rather strong language without any real qualifiers. It is definitely not "mostly useless". Shares and quotas are for different use-cases, that's all. Understand your use-case and choose accordingly.
- dilyevsky 3y agoIt doesn’t make any sense to me why —cpu flag is tweaking quota and not shares since quota is useful in tiny minority of usecases. A lot of people waste a ton of time debugging weird latency issues as a result of this decision
- the8472 3y agoWith shares you're going to experience worse latency if all the containers on the system size their thread pool to the maximum that's available during idle periods and then constantly context-switch due to oversubscription under load. With quotas you can do fixed resource allocation and the runtimes (not Go apparently) can fit themselves into that and not try to service more requests than they can currently execute given those resources.
- dilyevsky 3y agoAnd how is that different from worse latency due to cpu throttling from your users’ perspective?
- the8472 3y agoFixed queue, so it'll only take as many as it can process and reject the rest, which can be used to do scaling, if you have a cluster. With shares it would think it has all the CPU cores available and oversize the queue.
- dilyevsky 3y agoDoesn’t answer my q really. At least in kubernetes scaling is done by measuring usage against the request (shares) not the limit (quota)