4 ms·
BTW, since the blog post doesn't make it explicit: we're down to ~40 microseconds round trip between VMs in the same zone. No placement groups or infiniband req
by boulos 9y ago
BTW, since the blog post doesn't make it explicit: we're down to ~40 microseconds round trip between VMs in the same zone. No placement groups or infiniband required :).
Disclosure: I work on Google Cloud.
- acdha 9y agoThanks — I was disappointed not to see any absolute values in the post
- FBISurveillance 9y agoIs this available on GKE? Any action required (e.g. node pool flags) on our side to get this working?
- jsolson 9y agoYes, this is enabled for all VMs running on Compute Engine (which includes GKE VMs), however the in-guest iptables &c. bits add non-trivial overhead (I don't have numbers handy, apologies).
- FBISurveillance 9y agoThanks, good to know. Regarding in-guest iptables -- there's not much we can do on GKE/Kubernetes about that. I serve about 1.2M requests per second on my GKE cluster through GLB and see this overhead very well.
- boulos 9y agoThe new Alias IP stuff should help with that (though I imagine there will still be some iptables shenanigans left, so I'm not sure how much it will help)
- FBISurveillance 9y agoAFAIK for GLB to work with GKE (via GLBC) I would still need NodePort service that routes exclusively via iptables? FWIW I also look forward to IPVS in k8s 1.9 that should improve this slightly.
- boulos 9y agoYep! GKE just runs atop GCE. No work required on your side. That reminds me though: you have to be communicating over the internal IP addresses, not the external ones (I hope the Services stuff does the right thing automatically). Firewall rules and such are different for internal versus external IPs (people often want a simple "deny all external traffic"), so that's sadly still a performance cliff.
- dajonker 9y agoHow would I test this? If I just create two (preemptible) n1-standard-1 VMs with the latest debian image and I use "ping" with the internal IP address then I get on average 0.138 ms I do get impressive bandwidth (1.95 Gbits/sec) using iperf.
- boulos 9y agoping is sadly not a great test program. You should do a TCP_RR run with netperf instead. For best results, try one of our newer regions/zones (with newer fabric and NICs) like us-east4-b or europe-west2-c. An n1-std-1 is also a single hyperthread, so you may want a larger instance as well.