5 ms·

The Istio one hits home. It is the single scariest thing to work with in our kubernetes clusters. We've caused several outages by changing the smallest things.

seneca 6y ago
Indeed. All the flak k8s gets about being overly complex and difficult to manage (maybe rightfully) ought to leveled 5x at Istio.
JMTQp8lwXL 6y ago
Shouldn't even small changes be tested in a staging cluster before promoting to the production cluster?
- coolspot 6y ago
  “You don’t introduce your dirty changes onto perfectly good staging without some testing on production first”.
- cbushko 6y ago
  We go through several stages before it hits prod. Our test environment -> the dev environment -> staging -> prod and still we are bitten by istio. A lot of the problems we have seen do not manifest themselves until you get significant traffic in the cluster. Luckily, it is one of our goals this year to make our setup testable. Our setup is: - an ingressgateway running on each node so that our SIEM can get IP addresses - 1000s of virtual service entries for all the services running (40 per namespace x 25+ namespaces) - 100s of deploys a day which causes pilot to update all the time - the clusters are small with like 50 nodes - we have way too many services and consolidation is slow - we haven't been able to upgrade to the Istio operator yet because there is an outstanding bug that breaks everything
- dennis_jeeves 6y ago
  We never test, in the rare situations we test, we first test it on production. :)
nyellin 6y ago
What do you use Istio for? 90% of the usecases I hear about could be solved with simpler tooling.
- FridgeSeal 6y ago
  I don’t use it yet, but I’m probably going to have to because my team is looking at using Kubeflow and it leverages Istio to do things like traffic-splitting and stuff so you can A/B test models without needing to handle that at your model code level. KNative can also do really cool things like per-request routing, scale-to-zero deployments that it can bring back up when it gets a request, it’s pretty rad.
  - chakspak 6y ago
    My company has deployed Kubeflow for production model training. Early on, we used their big deployment, but we got frustrated trying to manage it with kfctl, so we started using kustomize directly and deploying only what we need, like KFP. So no istio for us! YMMV with your use case.
    - FridgeSeal 6y ago
      Oh this is perfect to hear! Yeah my plan is to use it the same way we use kubernetes: everything via kubernetes configs/justo use is friends and nobody using CLI’s, which I’m convinced are the text version of “click ops” hahaha. So Istio isn’t strictly required? Can I ask which components you deploy? At minimum I’m planning on just the deployment and serving components and just use straight Polyaxon for training.
      - chakspak 6y ago
        Yeah, there are quite a few applications that actively discourage the use of anything other than their CLI, which I find profoundly bizarre. Istio had a big warning in their docs that their Helm charts were deprecated, even though istioctl uses them under the hood. Funny that Kubeflow has its own Istio deployment, and they make you use kfctl to deploy it, haha. It's not strictly required, but might be for you, since A/B deployment actually is a thing Istio does. We only use Kubeflow Pipelines currently; looking into Katib. My feeling about Kubeflow is that it's a package of a lot of things that exist, with nice things on top, and an easy way to deploy those things. Only, it ends up not being that easy, and not easy at all to configure, and various features of the underlying tools are hard to get to or completely unavailable. If you deployed your own Istio, you'd understand it end-to-end and could solve problems with it when it goes south, and you'd even understand what exactly you're using it for and why. The biggest problem I have with kfctl is that it basically asks for system:masters privilege to install everything and then you get to figure out what it did on your own. I don't want Istio, Cert-Manager, Knative, Argo, etc. deployments that I don't understand and can't easily configure because they're buried 5 levels deep in Kustomize overlays. These are all things I can install from elsewhere and with more documentation. Kubeflow Pipelines is still a little mysterious, but the footprint is much smaller and not as invasive. Never used Polyaxon. Gonna be Googling that!
neop1x 6y ago
I recommend staying away from Istio unless you have experienced team of Isto ops and very good reasons for using it. I tried it a year ago and it was not pleasant at all. It looks nice (Kiali) when it works but it's a nightmare once it stops working. Using TLS complicates debugging. It lacked basic functuionality like simple Basic Auth (just recently implemented in wasm) and they deprecated original plugin architecture. At one point it started logging out certificate problems for me, I spent hours on it trying to figure out what happens. Then tried to upgrade to newer Istio and it magically fixed it so I think they shipped a broken build but comparing commits resulted in so many changes that I was unable to figure out what happened. It's unnecessary complicated layer which almost no one really needs! Maybe in the future it will be a standard but maybe not, I strongly don't recommend it unless you know what you are doing and have work/troubleshoot capacity.

seneca 6y ago

Indeed. All the flak k8s gets about being overly complex and difficult to manage (maybe rightfully) ought to leveled 5x at Istio.

JMTQp8lwXL 6y ago

Shouldn't even small changes be tested in a staging cluster before promoting to the production cluster?

coolspot 6y ago
“You don’t introduce your dirty changes onto perfectly good staging without some testing on production first”.
cbushko 6y ago
We go through several stages before it hits prod. Our test environment -> the dev environment -> staging -> prod and still we are bitten by istio. A lot of the problems we have seen do not manifest themselves until you get significant traffic in the cluster. Luckily, it is one of our goals this year to make our setup testable. Our setup is: - an ingressgateway running on each node so that our SIEM can get IP addresses - 1000s of virtual service entries for all the services running (40 per namespace x 25+ namespaces) - 100s of deploys a day which causes pilot to update all the time - the clusters are small with like 50 nodes - we have way too many services and consolidation is slow - we haven't been able to upgrade to the Istio operator yet because there is an outstanding bug that breaks everything
dennis_jeeves 6y ago
We never test, in the rare situations we test, we first test it on production. :)

nyellin 6y ago

What do you use Istio for? 90% of the usecases I hear about could be solved with simpler tooling.

FridgeSeal 6y ago
I don’t use it yet, but I’m probably going to have to because my team is looking at using Kubeflow and it leverages Istio to do things like traffic-splitting and stuff so you can A/B test models without needing to handle that at your model code level. KNative can also do really cool things like per-request routing, scale-to-zero deployments that it can bring back up when it gets a request, it’s pretty rad.
- chakspak 6y ago
  My company has deployed Kubeflow for production model training. Early on, we used their big deployment, but we got frustrated trying to manage it with kfctl, so we started using kustomize directly and deploying only what we need, like KFP. So no istio for us! YMMV with your use case.
  - FridgeSeal 6y ago
    Oh this is perfect to hear! Yeah my plan is to use it the same way we use kubernetes: everything via kubernetes configs/justo use is friends and nobody using CLI’s, which I’m convinced are the text version of “click ops” hahaha. So Istio isn’t strictly required? Can I ask which components you deploy? At minimum I’m planning on just the deployment and serving components and just use straight Polyaxon for training.
    - chakspak 6y ago
      Yeah, there are quite a few applications that actively discourage the use of anything other than their CLI, which I find profoundly bizarre. Istio had a big warning in their docs that their Helm charts were deprecated, even though istioctl uses them under the hood. Funny that Kubeflow has its own Istio deployment, and they make you use kfctl to deploy it, haha. It's not strictly required, but might be for you, since A/B deployment actually is a thing Istio does. We only use Kubeflow Pipelines currently; looking into Katib. My feeling about Kubeflow is that it's a package of a lot of things that exist, with nice things on top, and an easy way to deploy those things. Only, it ends up not being that easy, and not easy at all to configure, and various features of the underlying tools are hard to get to or completely unavailable. If you deployed your own Istio, you'd understand it end-to-end and could solve problems with it when it goes south, and you'd even understand what exactly you're using it for and why. The biggest problem I have with kfctl is that it basically asks for system:masters privilege to install everything and then you get to figure out what it did on your own. I don't want Istio, Cert-Manager, Knative, Argo, etc. deployments that I don't understand and can't easily configure because they're buried 5 levels deep in Kustomize overlays. These are all things I can install from elsewhere and with more documentation. Kubeflow Pipelines is still a little mysterious, but the footprint is much smaller and not as invasive. Never used Polyaxon. Gonna be Googling that!

neop1x 6y ago

I recommend staying away from Istio unless you have experienced team of Isto ops and very good reasons for using it. I tried it a year ago and it was not pleasant at all. It looks nice (Kiali) when it works but it's a nightmare once it stops working. Using TLS complicates debugging. It lacked basic functuionality like simple Basic Auth (just recently implemented in wasm) and they deprecated original plugin architecture. At one point it started logging out certificate problems for me, I spent hours on it trying to figure out what happens. Then tried to upgrade to newer Istio and it magically fixed it so I think they shipped a broken build but comparing commits resulted in so many changes that I was unable to figure out what happened. It's unnecessary complicated layer which almost no one really needs! Maybe in the future it will be a standard but maybe not, I strongly don't recommend it unless you know what you are doing and have work/troubleshoot capacity.