4 ms·

Yes and no. There's a tendency in common distributions to pick complex base components to support seldomly used features at the expense of turning K8s from und

Yes and no. There's a tendency in common distributions to pick complex base components to support seldomly used features at the expense of turning K8s from understandable to magic. I can tell you the main modes of operation and how everything fits together with flannel. Ask me to compare Canal to Calico or explain Cilium and I have no idea anymore. These systems also don't explain themselves well either. I have to dig into the code and docs pretty deeply to see it's BGP and eBPF, two technologies I wouldn't expect your average network-literate admin to know. And don't even get me started on Istio.

sparsely 5y ago
Istio has lots of support for retrying and recovering from network errors, which is fortunate because adding it to our infra introduced a lot of transient network errors
hedora 5y ago
I think this complexity is part of the business model. You either pay the cloud vendor to solve this issue, or you pay enterprise vendors for storage/network licenses, etc. J2EE was similar, from what I saw. In theory, you could run the open source stuff. In practice, IBM and Oracle made $$$ selling and supporting a proprietary version that supposedly worked out of the box.
xemdetia 5y ago
I think that's the other thing that's understated too because a lot of these underlying choose-your-own-adventure pieces are also in wildly different levels of maturity when it comes to observing and debug. You can spend a while pursuing one solution only to find one of the pieces simply doesn't have the functionality or it is not achievable or the big thing for me is not able to be secured in any meaningful way or it's a TODO with no sign of completion. That's still the main thing for me that gives me pause, it ends up being very unreliable when you are trying to secure it or simply put walls up around things. Finding out that your Network plugin doesn't support the thing you need is often not obvious until you have put in a massive amount of resources and work and sometimes it's not reconcilable. The idea of k8s is good, and the model is good, but the observability of features and working through the operational costs are still really raw right now and still has a lot of room for growth.

sparsely 5y ago

Istio has lots of support for retrying and recovering from network errors, which is fortunate because adding it to our infra introduced a lot of transient network errors

hedora 5y ago

I think this complexity is part of the business model. You either pay the cloud vendor to solve this issue, or you pay enterprise vendors for storage/network licenses, etc. J2EE was similar, from what I saw. In theory, you could run the open source stuff. In practice, IBM and Oracle made $$$ selling and supporting a proprietary version that supposedly worked out of the box.

xemdetia 5y ago

I think that's the other thing that's understated too because a lot of these underlying choose-your-own-adventure pieces are also in wildly different levels of maturity when it comes to observing and debug. You can spend a while pursuing one solution only to find one of the pieces simply doesn't have the functionality or it is not achievable or the big thing for me is not able to be secured in any meaningful way or it's a TODO with no sign of completion. That's still the main thing for me that gives me pause, it ends up being very unreliable when you are trying to secure it or simply put walls up around things. Finding out that your Network plugin doesn't support the thing you need is often not obvious until you have put in a massive amount of resources and work and sometimes it's not reconcilable. The idea of k8s is good, and the model is good, but the observability of features and working through the operational costs are still really raw right now and still has a lot of room for growth.