3 ms·

This is a couple of years ago, so I fully expect most of the issues we had back then to be fixed by now, but it was definitely Azure that was the problem. We w

This is a couple of years ago, so I fully expect most of the issues we had back then to be fixed by now, but it was definitely Azure that was the problem. We wanted to use their hosted kubernetes solution(I forget the name) and pods would just randomly lose connection to eachother, everything network related was just ridiculously unstable. Host machines would report their status as healthy, but then be unable to start any pods, making scaling out the cluster unreliable. I also remember a colleague I regarded as a bit of a wizard being very frustrated with cosmosdb, but I cannot for the life of me remember what the specific issue was. Our solution was actually quite well written, if I do say so myself, we had designed it to be cloud agnostic, just on the off chance that something like this would happen (there may have been rumours this acquisition would happen ahead of time). But Azure was just utterly unable to deliver on anything they promised, thus the write-off on my part.

Too 2y ago
Pods loosing connection to each other is still very very common in azure. To see the positive side of it, it’s a Chaos Monkey test for free. Everything you deploy must be hardened with reconnections, something you should be doing anyway. What’s frustrating is that it happens rarely enough to give the illusion of being stable, making it easy for PM to postpone hardening work, yet often enough to put a noticeable dent in your uptime if you don’t address it. Perfect degree of gaslighting.
- motorest 2y ago
  > To see the positive side of it, it’s a Chaos Monkey test for free. Everything you deploy must be hardened with reconnections, something you should be doing anyway. Keep in mind that in Azure it's a must-have, whereas everywhere else it's either a nice-to-have or a sign your system is broken.
- naasking 2y ago
  At that point, aren't you better off actually making your own services regularly disconnect clients so you iron out all of the reconnection bugs?
- deelowe 2y ago
  Having worked on the infra side of azure, I'm not surprised. Network is centrally managed and that team was a nightmare to deal with. Their ticket queue was so bad they only worked on sev 1 and the occasional 0. Nothing else got touched without talking to a VP and even then it often didn't change things.
Epa095 2y ago
Ohh, AKS. I had the 'pleasure' of using it quite early on, 5/6 years ago. We kept killing it by actually using it with more than 50 pods. You know you see one of few serious users when you get through the 3 layers of support and gets to talk to the actual developers in the US. But my impression is that it's better now. In general my experience with azure is that the base services, those who see millions of hours in use, are stable. Think VMs, storage, queue etc. But the higher up you go in the stack, the fewer hours of use do they see, and the lower quality it gets.
stego-tech 2y ago
I appreciate the context! Honestly, every time I start thinking that K8s might be the "universal cloud" orchestrator I've been hoping for/working towards, stories like this remind me just how...tenuous it can be relative to traditional VMs and standard containers-as-appliances. Still working to get my Admin cert for the spec, but it's definitely not something that sparks joy, if you catch my drift. Thank you for the insights!

Too 2y ago

Pods loosing connection to each other is still very very common in azure. To see the positive side of it, it’s a Chaos Monkey test for free. Everything you deploy must be hardened with reconnections, something you should be doing anyway. What’s frustrating is that it happens rarely enough to give the illusion of being stable, making it easy for PM to postpone hardening work, yet often enough to put a noticeable dent in your uptime if you don’t address it. Perfect degree of gaslighting.

motorest 2y ago
> To see the positive side of it, it’s a Chaos Monkey test for free. Everything you deploy must be hardened with reconnections, something you should be doing anyway. Keep in mind that in Azure it's a must-have, whereas everywhere else it's either a nice-to-have or a sign your system is broken.
naasking 2y ago
At that point, aren't you better off actually making your own services regularly disconnect clients so you iron out all of the reconnection bugs?
deelowe 2y ago
Having worked on the infra side of azure, I'm not surprised. Network is centrally managed and that team was a nightmare to deal with. Their ticket queue was so bad they only worked on sev 1 and the occasional 0. Nothing else got touched without talking to a VP and even then it often didn't change things.

Epa095 2y ago

Ohh, AKS. I had the 'pleasure' of using it quite early on, 5/6 years ago. We kept killing it by actually using it with more than 50 pods. You know you see one of few serious users when you get through the 3 layers of support and gets to talk to the actual developers in the US. But my impression is that it's better now. In general my experience with azure is that the base services, those who see millions of hours in use, are stable. Think VMs, storage, queue etc. But the higher up you go in the stack, the fewer hours of use do they see, and the lower quality it gets.

stego-tech 2y ago

I appreciate the context! Honestly, every time I start thinking that K8s might be the "universal cloud" orchestrator I've been hoping for/working towards, stories like this remind me just how...tenuous it can be relative to traditional VMs and standard containers-as-appliances. Still working to get my Admin cert for the spec, but it's definitely not something that sparks joy, if you catch my drift. Thank you for the insights!