3 ms·

In the low latency trading world, these concerns are addressed by partitioning resources (for CPU, with affinities). This seems like a simpler mechanism that do

In the low latency trading world, these concerns are addressed by partitioning resources (for CPU, with affinities). This seems like a simpler mechanism that doesn’t require the kernel/daemon to track resource usage and to impose limits. I see only upsides to performance (bandwidth and latency) and availability by partitioning resources — so what are the benefits of the alternative, using limits, beyond being able to stuff more apps onto a machine? That’s not to trivialize that benefit. Does kubernetes even allow for “affinitizing”?

dijit 6y ago
Video Games industry is the same, in fact it was one of the reasons we went with google cloud over alternatives, at the time Amazon was not using KVM (or, HVM as they seem to call it)- and GCP was at least attempting CPU affinity on the VMs, this caused quite a variance in latency when using amazon which did not exist on GCP. To answer your question: I believe there is 'pinning' in Kubernetes which can solve it, but kubernetes has other overheads in terms of latency (iptables pod routing with contrack enabled for instance) so I personally would avoid using it for low latency applications. https://builders.intel.com/docs/networkbuilders/cpu-pin-and-isolation-in-kubernetes-app-note.pdf https://builders.intel.com/docs/networkbuilders/cpu-pin-and-...
- GauntletWizard 6y ago
  For videogames, you should not be subject to the iptables bits - Agones encourages use of the `hostPort` networking mode, which doesn't create or require special iptables routing.
dilyevsky 6y ago
If you’re talking about core pinning (cpuset.cpu_exclusive) Google had famously used this for some workloads and when they turned it off by accident performance got better
- dundarious 6y ago
  I have experience with affinities (I don’t know the k8s name for them) where it is crucial to achieving low latency (way sub microsecond RTT). Depending on your app’s architecture, it can be a major boon to bandwidth as well. EDIT: Sorry, I wrote that thinking you were referring to k8s. Just sched_setaffinity and isolcpus are sufficient. YMMV
  - dilyevsky 6y ago
    Yeah Kubernetes only uses cpusets to pin workload to cores and only when you have this enabled explicitly. Sched_setaffinity is separate mechanism not used by k8s 99% of typical kubernetes workloads dont need those kind of latency requirements and it maybe be detrimental for their throughput to only use subset of cores (classic throughput vs latency tradeoff).

dijit 6y ago

Video Games industry is the same, in fact it was one of the reasons we went with google cloud over alternatives, at the time Amazon was not using KVM (or, HVM as they seem to call it)- and GCP was at least attempting CPU affinity on the VMs, this caused quite a variance in latency when using amazon which did not exist on GCP. To answer your question: I believe there is 'pinning' in Kubernetes which can solve it, but kubernetes has other overheads in terms of latency (iptables pod routing with contrack enabled for instance) so I personally would avoid using it for low latency applications. https://builders.intel.com/docs/networkbuilders/cpu-pin-and-isolation-in-kubernetes-app-note.pdf https://builders.intel.com/docs/networkbuilders/cpu-pin-and-...

GauntletWizard 6y ago
For videogames, you should not be subject to the iptables bits - Agones encourages use of the `hostPort` networking mode, which doesn't create or require special iptables routing.

dilyevsky 6y ago

If you’re talking about core pinning (cpuset.cpu_exclusive) Google had famously used this for some workloads and when they turned it off by accident performance got better

dundarious 6y ago
I have experience with affinities (I don’t know the k8s name for them) where it is crucial to achieving low latency (way sub microsecond RTT). Depending on your app’s architecture, it can be a major boon to bandwidth as well. EDIT: Sorry, I wrote that thinking you were referring to k8s. Just sched_setaffinity and isolcpus are sufficient. YMMV
- dilyevsky 6y ago
  Yeah Kubernetes only uses cpusets to pin workload to cores and only when you have this enabled explicitly. Sched_setaffinity is separate mechanism not used by k8s 99% of typical kubernetes workloads dont need those kind of latency requirements and it maybe be detrimental for their throughput to only use subset of cores (classic throughput vs latency tradeoff).