4 ms·
Googler, options are my own. This is a great write-up. As someone who has had to deal with backend subsetting a bunch in the past, this is such an interesting
by kyrra 4y ago
Googler, options are my own.
This is a great write-up. As someone who has had to deal with backend subsetting a bunch in the past, this is such an interesting read.
One thing interesting about Google's scale and this architecture is that if you use classic firewalls that end up having connection tracking in them, you can easily overload that connection tracking table. M x k connections can balloon that connection count given the number of replicas some services have.
If you like write-ups like this, there is a great 2016 acm paper by Google on how it does source control: https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...
- bushbaba 4y agoConntrack tables filling up are also a challenge with k8s at very, very, very large scale.
- jeffbee 4y agoAny kind of "smart" thing in your network will eventually hit a scaling limit. The end-to-end principle has to be given due respect. Only the edge of your system has the necessary resources to participate in it. As the scale becomes more extreme, even the kernel of a machine running 100s of k8s pods (or borg tasks) starts to look more like the center of your network and less like the edge of it. The trend is to move as many things as possible out of the kernel and into user processes, including network policy, quota, pacing, and the whole TCP stack too.