14 ms·

Horrors of Using Azure Kubernetes Service in Production

371 points by pdeva1 8y ago

rcconf 8y ago
Hmm, this isn't great. Currently using Azure Kubernetes Service and we haven't had many issues so far, but we just made the shift. Hope I don't have to move over to Google cloud.
- outworlder 8y ago
  Why? GKE works perfectly over there.
  - specialp 8y ago
    Many people (me included) don't trust Google to stay with any business besides advertising. There's been too many times that they have ended services with not much time to get off.
    - brazzledazzle 8y ago
      The Cloud products follow a Deprecation Policy documented here: https://cloud.google.com/terms/ https://cloud.google.com/terms/ I can understand being upset about their consumer products but it doesn’t really apply here.
      - dvasdekis 8y ago
        This is what always puzzled me about the Google-Alphabet strategy, specifically the idea of having all the assets under a single share ticker (GOOGL). The more services you put under one banner, the more the stink of one disaster is going to linger, and hinder adoption of the successes. To me, a far simpler proposition would be a new brand & share issuance for each new sub-company (eg. Waymo), with existing Alphabet shareholders getting pro-rata shares in the new company.
        brazzledazzle 8y ago
        I’d bet on it being about the recognition that Google has been a pioneer and thought leader in the scalable systems/hosting space and they didn’t want to throw out the baby with the bathwater.
    - axitanull 8y ago
      That and countless horror stories about once your google account or service is banned for whatever reason detected by their bot, you have no way to appeal with their nonexistent customer support.
- RantyDave 8y ago
  It's heaps and heaps better than Azure.
- zeroone101 8y ago
  You'll be fine. We run a number of AKS clusters and they have all been rock solid. I think the problem is people hear "managed cluster" and so they don't think they need to understand how k8s works. Follow best practices (resource limits, etc.) and you'll be just fine. We've even tested out the upgrade flow on a live production cluster and it was butter smooth.
minxomat 8y ago
My $DAYJOB is leading a team which develops applications and gateways (for the 1k+ employee B2B market) that integrate deeply with Azure, Azure AD and anything that comes with it. We do have Microsoft employees (who work on Azure) on our payroll, too. I can tell you, as I'm sure anyone in my team can, that Azure is one big alpha-stage amalgation of half-baked services. I would never ever recommend Azure to literally any organization no matter the size. Seeing our customers struggle with it, us struggle with it, and even MS folks struggle with even the most basic tasks gets tiring really fast. We have so many workarounds in our software for inconsistency, unavailability, questionable security and general quirks in Azure that it's not even funny anymore. There are some days where random parts of Azure completely fail, like customers not being able to view resources, role assignments or even their directory config. An automatic integration test of one of our apps, which makes heavy use of Azure Resource Management APIs, just fails dozens of times a week not because we have a bug, but because state within Azure didn't propagate (RBAC changes, resource properties) within a timeout of more than 15 minutes! Two weeks back, the same test managed to reproducibly produce a state within Azure that completely disabled the Azure Portal resource view. All "blades" in Azure just displayed "unable to access data". Only an ultra-specific sequence of UI interactions and API calls could restore Azure (while uncovering a lot of other issues). That is the norm, not the exception. In 1.5 years worth of development, there has never been a single week without an Azure issue robbing us of hours of work just debugging their systems and writing workarounds. /rant On topic though, we've had good experiences with these k8s runtimes: - GKE - Rancher + DO - IBM Cloud k8s (yeah, I know!)
- gamblor956 8y ago
  My company uses Azure quite heavily, but not Kubernetes. No crashes, ever. Way more reliable than AWS ever was. (GCP is our failover.) So it seems that your experience is, from my POV, the exception. Maybe there's something wrong with the way you guys have Azure set up?
  - politician 8y ago
    Which provisioning model are you using -- ASM or ARM? The last time I used Azure we used the (deprecated) ASM model which was pretty stable instead of the newer and often broken ARM model. We ended up staying on the deprecated model until we moved to AWS (for unrelated reasons).
curiousDog 8y ago
This is sad. From what I hear, one of the founders of k8s works on AKS. Only a matter of time before GCP becomes the #1/2 cloud provider.
- h4b4n3r0 8y ago
  I’ve seen at least 4 “founders of Kubernetes” by now. How many are there in total?
  - rhencke 8y ago
    At least three to five. https://en.wikipedia.org/wiki/Kubernetes#History https://en.wikipedia.org/wiki/Kubernetes#History
- cheeze 8y ago
  Agree. The quality of their software is so much better than the other major players.
- swozey 8y ago
  Yeah, Brendan Burns is at ms
spicyusername 8y ago
This is probably why they're releasing OpenShift on Azure. To let Red Hat engineers manage the kubernetes part. https://azure.microsoft.com/en-us/blog/openshift-on-azure-the-easiest-fully-managed-openshift-in-the-cloud/ https://azure.microsoft.com/en-us/blog/openshift-on-azure-th...
- netdur 8y ago
  I like openshift, I am not sure why developers are not hot about it.
  - federicoponzi 8y ago
    What do you like about openshift that lacks in kubernetes?
  - wnsire 8y ago
    > I am not sure why developers are not hot about it. Because it's designed entirely for enterprise customers. If you have a startup you have very little reason to choose OpenShift compared to Heroku or AWS honnestly. I still love Redhat tho.
mgalgs 8y ago
FWIW, Amazon's hosted Kubernetes offering (EKS) isn't stable either (DNS failures, HPA is known to be broken, etc.).
- shamsalmon 8y ago
  I know HPA is a legit issue but DNS failures seem to be fairly normal in kubernetes. Scaling up kube-dns has helped us resolve that particular issue as well as moving away from Alpine and into minimal Debian images. Alpine has its own DNS issues that caused us much pain.
  - praseodym 8y ago
    Kube-dns (or CoreDNS in newer clusters) is pretty stable in my experience. It's still a very good idea to run more than one replica so that you can tolerate a single node failure, but if DNS failures are "fairly normal" that definitely warrants some additional investigation.
    - shamsalmon 8y ago
      Thanks we did look into it but not as thoroughly as probably needed. Switching out from Alpine fixed pretty much all our issues.
    - sleepybrett 8y ago
      Most dns problems in kubernetes, in my experience, can be traced to udp failures due to the iptables kubeproxy backend.
  - atombender 8y ago
    We've had issues with KubeDNS, too. Lots of retries and timeouts on the client side, and lots of conntrack entries. Libc has pretty slow retries (5s, I think) by default, and until 1.11 hits you can't easily set up resolver configs, though you can inject an envvar separately into each. And musl-based distros like Alpine don't even support some of libc's options, iirc. We ended up scaling up KubeDNS to 2 replicas and moving them to a dedicated nodepool just to make sure they weren't competing with other nodes. That fixed our issues for now.
paxys 8y ago
Worth nothing that both Microsoft and Amazon's Kubernetes offerings are very new (literally weeks since GA). While "officially" ready, it is pretty naive to rely on them for production-critical workloads just yet, at least compared to Google Kubernetes Engine which has been running for years. If you absolutely need managed Kubernetes, stick to GCP for now.
QiKe 8y ago
(Eng lead for AKS here) While lots of people have had great success with AKS, we're always concerned when someone has a bad time. In this particular case the AKS engineering team spent over a day helping identify that the user had over scheduled their nodes, by running applications without memory limit, resulting in the kernel oom (out of memory) killer terminating the Docker daemon and kubelet. As part of this investigation we increased the system reservation for both Docker and kubelet to ensure that in the future if a user over schedules their nodes the kernel will only terminate their applications and not the critical system daemons.
- empath75 8y ago
  So when you told him to use less memory you told him to provision less memory to his containers which makes a lot more sense than azure being staffed by complete morons.
  - nickbauman 8y ago
    When I deploy to Amazon ECS the upper limit of the resource geometry of my service is checked and if it exceeds that upper limit available of the underlying cluster, it refuses to deploy. I understand k8s has similar features. It reads like Azure doesn't have their k8s configured correctly.
    - markbnj 8y ago
      If the containers in a pod request more ram than is available on any node in the cluster then the pod will fail to schedule and will remain in pending state, which can be seen in the events for the controller (replicaset, daemonset, etc) using, for example `kubectl describe replicaset myreplicaset.` We've gotten ourselves into this situation a few times on GKE. It's easily resolvable by tuning the resource requests or scaling the nodepool and has no adverse effect on the operation of the cluster.
- specialp 8y ago
  This is a pretty bad mistake from the customer if this is true. If not done already it would probably be good to expose Prometheus metrics on CPU/Memory usage per node.
sabareesh 8y ago
Azure VM in general a joke
ageitgey 8y ago
Here's a fun fact about Azure Kubernetes: 1. Deploy your Linux service on k8s with redundant nodes 2. Create a k8s VolumeClaim and mount it on your nodes to give your application some long-lived or shared disk storage (i.e. for processing user-uploaded files). 3. Wait until the subtle bugs start to appear in your app. Because persistent k8s volumes on Azure are provided by Azure disk storage service behind the scenes, lots of weird Windows-isms apply. And this goes beyond stuff like case insensitivity for file names. For example, if a user tries to upload a file called "COM1" or "PRN1", it will blow up with a disk write error. Yes, that's right, Azure is the only cloud vendor that is 100% compatible with enforcing DOS 1.0 reserved filenames - on your Linux server in 2018!
- AaronFriel 8y ago
  Is this with the managed-disk volume (which IIRC is formatted ext4) or with `AzureFiles`, which is essentially SMB/CIFS?
- colemickens 8y ago
  The only way this would make any sense is if you were using Azure Files, rather than Azure Disks. There's virtually never a time when it makes sense to use Azure Files over Azure Disks, and even when it does, a change in the application would be better advised than using Azure Files. >Yes, that's right, Azure is the only cloud vendor that is 100% compatible with enforcing DOS 1.0 reserved filenames - on your Linux server in 2018! This is hyperbolic bordering on flatly false. This is more reasonable and accurate: "Azure is the only cloud vendor that serves their Samba product from Windows boxes, and thus leak Win/NTFS-isms into their Samba shares [that shouldn't be used anyway]." How would an ext4 filesystem, mounted under Linux, attached as a block device to a VM, be subjected to Windows-isms? What you're implying doesn't even make sense.
  - parasubvert 8y ago
    I really would have to disagree with the statement one should never use Azure Files over Azure disks. 1. Most Azure VM types have very stringent limits on attached disks; a K8s worker can easily blow past this limit. 2. You have tremendous complexity to deal with: pick Azure managed disks vs unmanaged disks on storage accounts (you can’t mix them on the same cluster). You have to understand the trade of of standard vs premium storage and how they bill (premium rounds up and charges by capacity, not consumption). And you need the right VM types for premium. 3. Managed disks each create a resource object in your resource group. A resource group last I checked had hard limits on the number of resources (like 4000?). Each VM is minimum 3 to 4 resources (with a NIC, image, and disk)... at scale this gets difficult. 4. Azure disks require significant time to create. , mount and remount. A StatefulSet pod failure will sometimes take 3-5 minutes for it’s PV to move to a different worker. And worse when your Azure region has allocation problems. Azure files are near instantaneous unmoubt/remount. 5. Azure disks are block storage and thus only ReadWriteOnce. Azure files are RWM. So, sure, if you’re running a cluster database with dedicated per node PVs and limited expected redeployments... use Azure disks. If you need a PV for any other reason... especially for application tiers that churn frequently.. use Azure files.
taherchhabra 8y ago
Had a similar experience with azure cosmos graph API. The API is half baked. Doesn't support all gremlin operations. Even supported operations give non standard output. Switched to aws Neptune immediately when they launched
- wnsire 8y ago
  > The API is half baked Doesn't surprise me. Cosmos was too good too be true : - Serverless - Infinite Scalability - Mongo API or Gremlin API or SQL API It's obvious that it can't hold up to all it's promises.
  - the_new_guy_29 8y ago
    "Noone can give you what i can promise you"
partiallypro 8y ago
It's a very new offering, the Linux App Services are still in beta, I have no idea why you would roll it into production expecting no hiccups. AWS is also new on this. Give it 6 months and let the kinks work out before migrating workloads. Seems like common sense.
- verst 8y ago
  App Service on Linux is an unrelated service that actually runs on top of Service Fabric (a stateful microservice and container orchestration platform).
  - apurvajo 8y ago
    App Service on Linux is not in Beta, it is GA product for over year now with SLA of 99.95%. It does not use Service Fabric in backend, it uses it's own custom Orchestrator (which essentially removes the quirks of learning about orchestration away from the user)
AaronFriel 8y ago
There are definitely growing pains with using Kubernetes on Azure. I've wondered a few times if other platforms have similar issues and have seen more than a few complaints about EKS. Microsoft has some great people working on Azure, but I do feel like AKS was released to GA too soon. Without a published roadmap and scant acknowledgment of issues, I'm not sure I could recommend it to my clients or employer. It's disappointing, because I've had few issues with other Azure services. Full disclosure: I receive a monthly credit through a Microsoft program for Azure.
- markbnj 8y ago
  > I've wondered a few times if other platforms have similar issues and have seen more than a few complaints about EKS. I can't speak to EKS but we've been running production workloads on GKE for over a year with very good results. There have been a very few really troublesome "growing pains" type issues (an early example: loadbalancer services getting stuck with a pending external IP assignment for days) but Google has been awesome about support, even to the extent of getting Tim Hockin and Brendan Burns on the phone with us at various times to gather information about stuff like the example I gave above. I give them high marks and would recommend the service without hesitation.
  - curiousDog 8y ago
    Brendan Burns is the lead engineer on AKS AFAIK
    - mohamedmansour 8y ago
      Brendan Burns was the Lead Engineer on Kubernetes at Google around 3 years ago, then went to Microsoft to lead this entire AKS/ACS/Container effort. https://www.linkedin.com/in/brendan-burns-487aa590 https://www.linkedin.com/in/brendan-burns-487aa590
ubuntunero 8y ago
interesting. thanks
rdl 8y ago
Key Vault (their HSM product) is even worse.
gercheq 8y ago
Azure is not bad but there are definitely some rough edges. We're having trouble with their Bizspark Sponsorship biling https://news.ycombinator.com/item?id=17698948 https://news.ycombinator.com/item?id=17698948
stefanatfrg 8y ago
Couple of questions to the OP: 1. What version of docker / container runtime is being used? 2. What base image for your containers is being used? eg. alpine has known DNS issues [1] [1] https://www.youtube.com/watch?v=ZnW3k6m5AY8 https://www.youtube.com/watch?v=ZnW3k6m5AY8
bsaul 8y ago
Side question : what are the best practices for development ? Are you suppose to run a local kubernetes deployment ( it looks like it's pretty hard to set up) , or do you run everything outside of containers when developping and then deal with k8 packaging and deployment as a completely separated issue ( which looks like it could lead to discovering a lot of issues on the preproduction environment) ?
crazy_monkey 8y ago
This is basically "the horrors of using any relatively new feature on Azure and by Microsoft"
hb3b 8y ago
I joined a healthcare startup in 2014 that had a small infrastructure on Azure. Back then AWS weren't signing BAAs and Azure was the only player in town. Being an early startup, the company didn't purchase a support plan from Azure. One day Azure suffered a major outage (may have been storage related) and over an hour later, I reached out to Microsoft for written confirmation that we could forward to customers. Since we didn't have a support plan they flat-out refused to provide any documentation whatsoever about the issue. They wanted $10,000. Azure - never again. Company moved to AWS within a quarter.
nojvek 8y ago
I believe this is a cultural problem with Microsoft. Probably similar to other companies but it was very evident at Microsoft. People responsible to allocate resources (The management chain) rarely dogfood the product. While the Engineers and PM would complain a lot about quality issues, management wants to prioritize more features. It was a running joke at Microsoft: No one gets promoted for improving existing things, if you want a quick promo, build a new thing. So when you see a bazillion half baked things in Azure. That’s because someone got promoted for building each of those half baked things and moving on to the next big thing. Going from 0-90% is the same amount of work as 90-99% and the same amount of work as 99.0% - 99.99%. Making things insanely great is hard and requires a lot of dedicated focus and a commitment to set a higher bar for yourself.
FlorianRappl 8y ago
We have a larger migration project going on for months. So far not a single failure occurred and our TEST environment is already fully migrated (quite responsive and rock solid) since 2 weeks. However, I do share that Azure indeed has released a lot of half-baked features and services lately (last 1.5 to 2 years). I hope this trend does not continue.
parasubvert 8y ago
DNS failures were almost certainly related to all the k8s system services on the cluster not having CPU or memory reservations, and KubeDNS was flaking. In general AKS is a vanilla k8s cluster and expects you know what you’re doing. MS arguably should enforce some opinions about how things like system services have reservations, etc, but none of this is vanilla. The trouble is that K8s defaults are pretty poor from a security (no seccomp profiles or apparmor/se profiles) and performance perspective (no reservations on key system DaemonSets). We’ve had this interesting industry pendulum swing between extreme poles of “we hate opinionated platforms! Give me all the knobs!” And “this is too hard, we need opinions and guard rails!”. I think the success of K8s is exposing people to the complexity of supplying all of the config details yourself and we will see a new breed of opinionated platforms on top of it very shortly. It reminds me of the early Linux Slackware and SLS and Debian days where people traded X11 configs and window manager configs like they were treasured artifacts before Red Hat, Gnome and KDE, SuSE, and eventually Ubuntu, started to force opinions.

371 points by pdeva1 8y ago

Hmm, this isn't great. Currently using Azure Kubernetes Service and we haven't had many issues so far, but we just made the shift. Hope I don't have to move over to Google cloud.

outworlder 8y ago
Why? GKE works perfectly over there.
- specialp 8y ago
  Many people (me included) don't trust Google to stay with any business besides advertising. There's been too many times that they have ended services with not much time to get off.
  - brazzledazzle 8y ago
    The Cloud products follow a Deprecation Policy documented here: https://cloud.google.com/terms/ https://cloud.google.com/terms/ I can understand being upset about their consumer products but it doesn’t really apply here.
    - dvasdekis 8y ago
      This is what always puzzled me about the Google-Alphabet strategy, specifically the idea of having all the assets under a single share ticker (GOOGL). The more services you put under one banner, the more the stink of one disaster is going to linger, and hinder adoption of the successes. To me, a far simpler proposition would be a new brand & share issuance for each new sub-company (eg. Waymo), with existing Alphabet shareholders getting pro-rata shares in the new company.
      - brazzledazzle 8y ago
        I’d bet on it being about the recognition that Google has been a pioneer and thought leader in the scalable systems/hosting space and they didn’t want to throw out the baby with the bathwater.
  - axitanull 8y ago
    That and countless horror stories about once your google account or service is banned for whatever reason detected by their bot, you have no way to appeal with their nonexistent customer support.
RantyDave 8y ago
It's heaps and heaps better than Azure.
zeroone101 8y ago
You'll be fine. We run a number of AKS clusters and they have all been rock solid. I think the problem is people hear "managed cluster" and so they don't think they need to understand how k8s works. Follow best practices (resource limits, etc.) and you'll be just fine. We've even tested out the upgrade flow on a live production cluster and it was butter smooth.

minxomat 8y ago

My $DAYJOB is leading a team which develops applications and gateways (for the 1k+ employee B2B market) that integrate deeply with Azure, Azure AD and anything that comes with it. We do have Microsoft employees (who work on Azure) on our payroll, too. I can tell you, as I'm sure anyone in my team can, that Azure is one big alpha-stage amalgation of half-baked services. I would never ever recommend Azure to literally any organization no matter the size. Seeing our customers struggle with it, us struggle with it, and even MS folks struggle with even the most basic tasks gets tiring really fast. We have so many workarounds in our software for inconsistency, unavailability, questionable security and general quirks in Azure that it's not even funny anymore. There are some days where random parts of Azure completely fail, like customers not being able to view resources, role assignments or even their directory config. An automatic integration test of one of our apps, which makes heavy use of Azure Resource Management APIs, just fails dozens of times a week not because we have a bug, but because state within Azure didn't propagate (RBAC changes, resource properties) within a timeout of more than 15 minutes! Two weeks back, the same test managed to reproducibly produce a state within Azure that completely disabled the Azure Portal resource view. All "blades" in Azure just displayed "unable to access data". Only an ultra-specific sequence of UI interactions and API calls could restore Azure (while uncovering a lot of other issues). That is the norm, not the exception. In 1.5 years worth of development, there has never been a single week without an Azure issue robbing us of hours of work just debugging their systems and writing workarounds. /rant On topic though, we've had good experiences with these k8s runtimes: - GKE - Rancher + DO - IBM Cloud k8s (yeah, I know!)

gamblor956 8y ago
My company uses Azure quite heavily, but not Kubernetes. No crashes, ever. Way more reliable than AWS ever was. (GCP is our failover.) So it seems that your experience is, from my POV, the exception. Maybe there's something wrong with the way you guys have Azure set up?
- politician 8y ago
  Which provisioning model are you using -- ASM or ARM? The last time I used Azure we used the (deprecated) ASM model which was pretty stable instead of the newer and often broken ARM model. We ended up staying on the deprecated model until we moved to AWS (for unrelated reasons).

curiousDog 8y ago

This is sad. From what I hear, one of the founders of k8s works on AKS. Only a matter of time before GCP becomes the #1/2 cloud provider.

h4b4n3r0 8y ago
I’ve seen at least 4 “founders of Kubernetes” by now. How many are there in total?
- rhencke 8y ago
  At least three to five. https://en.wikipedia.org/wiki/Kubernetes#History https://en.wikipedia.org/wiki/Kubernetes#History
cheeze 8y ago
Agree. The quality of their software is so much better than the other major players.
swozey 8y ago
Yeah, Brendan Burns is at ms

spicyusername 8y ago

This is probably why they're releasing OpenShift on Azure. To let Red Hat engineers manage the kubernetes part. https://azure.microsoft.com/en-us/blog/openshift-on-azure-the-easiest-fully-managed-openshift-in-the-cloud/ https://azure.microsoft.com/en-us/blog/openshift-on-azure-th...

netdur 8y ago
I like openshift, I am not sure why developers are not hot about it.
- federicoponzi 8y ago
  What do you like about openshift that lacks in kubernetes?
- wnsire 8y ago
  > I am not sure why developers are not hot about it. Because it's designed entirely for enterprise customers. If you have a startup you have very little reason to choose OpenShift compared to Heroku or AWS honnestly. I still love Redhat tho.

mgalgs 8y ago

FWIW, Amazon's hosted Kubernetes offering (EKS) isn't stable either (DNS failures, HPA is known to be broken, etc.).

shamsalmon 8y ago
I know HPA is a legit issue but DNS failures seem to be fairly normal in kubernetes. Scaling up kube-dns has helped us resolve that particular issue as well as moving away from Alpine and into minimal Debian images. Alpine has its own DNS issues that caused us much pain.
- praseodym 8y ago
  Kube-dns (or CoreDNS in newer clusters) is pretty stable in my experience. It's still a very good idea to run more than one replica so that you can tolerate a single node failure, but if DNS failures are "fairly normal" that definitely warrants some additional investigation.
  - shamsalmon 8y ago
    Thanks we did look into it but not as thoroughly as probably needed. Switching out from Alpine fixed pretty much all our issues.
  - sleepybrett 8y ago
    Most dns problems in kubernetes, in my experience, can be traced to udp failures due to the iptables kubeproxy backend.
- atombender 8y ago
  We've had issues with KubeDNS, too. Lots of retries and timeouts on the client side, and lots of conntrack entries. Libc has pretty slow retries (5s, I think) by default, and until 1.11 hits you can't easily set up resolver configs, though you can inject an envvar separately into each. And musl-based distros like Alpine don't even support some of libc's options, iirc. We ended up scaling up KubeDNS to 2 replicas and moving them to a dedicated nodepool just to make sure they weren't competing with other nodes. That fixed our issues for now.

paxys 8y ago

Worth nothing that both Microsoft and Amazon's Kubernetes offerings are very new (literally weeks since GA). While "officially" ready, it is pretty naive to rely on them for production-critical workloads just yet, at least compared to Google Kubernetes Engine which has been running for years. If you absolutely need managed Kubernetes, stick to GCP for now.

QiKe 8y ago

(Eng lead for AKS here) While lots of people have had great success with AKS, we're always concerned when someone has a bad time. In this particular case the AKS engineering team spent over a day helping identify that the user had over scheduled their nodes, by running applications without memory limit, resulting in the kernel oom (out of memory) killer terminating the Docker daemon and kubelet. As part of this investigation we increased the system reservation for both Docker and kubelet to ensure that in the future if a user over schedules their nodes the kernel will only terminate their applications and not the critical system daemons.

empath75 8y ago
So when you told him to use less memory you told him to provision less memory to his containers which makes a lot more sense than azure being staffed by complete morons.
- nickbauman 8y ago
  When I deploy to Amazon ECS the upper limit of the resource geometry of my service is checked and if it exceeds that upper limit available of the underlying cluster, it refuses to deploy. I understand k8s has similar features. It reads like Azure doesn't have their k8s configured correctly.
  - markbnj 8y ago
    If the containers in a pod request more ram than is available on any node in the cluster then the pod will fail to schedule and will remain in pending state, which can be seen in the events for the controller (replicaset, daemonset, etc) using, for example `kubectl describe replicaset myreplicaset.` We've gotten ourselves into this situation a few times on GKE. It's easily resolvable by tuning the resource requests or scaling the nodepool and has no adverse effect on the operation of the cluster.
specialp 8y ago
This is a pretty bad mistake from the customer if this is true. If not done already it would probably be good to expose Prometheus metrics on CPU/Memory usage per node.

sabareesh 8y ago

Azure VM in general a joke

ageitgey 8y ago

Here's a fun fact about Azure Kubernetes: 1. Deploy your Linux service on k8s with redundant nodes 2. Create a k8s VolumeClaim and mount it on your nodes to give your application some long-lived or shared disk storage (i.e. for processing user-uploaded files). 3. Wait until the subtle bugs start to appear in your app. Because persistent k8s volumes on Azure are provided by Azure disk storage service behind the scenes, lots of weird Windows-isms apply. And this goes beyond stuff like case insensitivity for file names. For example, if a user tries to upload a file called "COM1" or "PRN1", it will blow up with a disk write error. Yes, that's right, Azure is the only cloud vendor that is 100% compatible with enforcing DOS 1.0 reserved filenames - on your Linux server in 2018!

AaronFriel 8y ago
Is this with the managed-disk volume (which IIRC is formatted ext4) or with `AzureFiles`, which is essentially SMB/CIFS?
colemickens 8y ago
The only way this would make any sense is if you were using Azure Files, rather than Azure Disks. There's virtually never a time when it makes sense to use Azure Files over Azure Disks, and even when it does, a change in the application would be better advised than using Azure Files. >Yes, that's right, Azure is the only cloud vendor that is 100% compatible with enforcing DOS 1.0 reserved filenames - on your Linux server in 2018! This is hyperbolic bordering on flatly false. This is more reasonable and accurate: "Azure is the only cloud vendor that serves their Samba product from Windows boxes, and thus leak Win/NTFS-isms into their Samba shares [that shouldn't be used anyway]." How would an ext4 filesystem, mounted under Linux, attached as a block device to a VM, be subjected to Windows-isms? What you're implying doesn't even make sense.
- parasubvert 8y ago
  I really would have to disagree with the statement one should never use Azure Files over Azure disks. 1. Most Azure VM types have very stringent limits on attached disks; a K8s worker can easily blow past this limit. 2. You have tremendous complexity to deal with: pick Azure managed disks vs unmanaged disks on storage accounts (you can’t mix them on the same cluster). You have to understand the trade of of standard vs premium storage and how they bill (premium rounds up and charges by capacity, not consumption). And you need the right VM types for premium. 3. Managed disks each create a resource object in your resource group. A resource group last I checked had hard limits on the number of resources (like 4000?). Each VM is minimum 3 to 4 resources (with a NIC, image, and disk)... at scale this gets difficult. 4. Azure disks require significant time to create. , mount and remount. A StatefulSet pod failure will sometimes take 3-5 minutes for it’s PV to move to a different worker. And worse when your Azure region has allocation problems. Azure files are near instantaneous unmoubt/remount. 5. Azure disks are block storage and thus only ReadWriteOnce. Azure files are RWM. So, sure, if you’re running a cluster database with dedicated per node PVs and limited expected redeployments... use Azure disks. If you need a PV for any other reason... especially for application tiers that churn frequently.. use Azure files.

taherchhabra 8y ago

Had a similar experience with azure cosmos graph API. The API is half baked. Doesn't support all gremlin operations. Even supported operations give non standard output. Switched to aws Neptune immediately when they launched

wnsire 8y ago
> The API is half baked Doesn't surprise me. Cosmos was too good too be true : - Serverless - Infinite Scalability - Mongo API or Gremlin API or SQL API It's obvious that it can't hold up to all it's promises.
- the_new_guy_29 8y ago
  "Noone can give you what i can promise you"

partiallypro 8y ago

It's a very new offering, the Linux App Services are still in beta, I have no idea why you would roll it into production expecting no hiccups. AWS is also new on this. Give it 6 months and let the kinks work out before migrating workloads. Seems like common sense.

verst 8y ago
App Service on Linux is an unrelated service that actually runs on top of Service Fabric (a stateful microservice and container orchestration platform).
- apurvajo 8y ago
  App Service on Linux is not in Beta, it is GA product for over year now with SLA of 99.95%. It does not use Service Fabric in backend, it uses it's own custom Orchestrator (which essentially removes the quirks of learning about orchestration away from the user)

AaronFriel 8y ago

There are definitely growing pains with using Kubernetes on Azure. I've wondered a few times if other platforms have similar issues and have seen more than a few complaints about EKS. Microsoft has some great people working on Azure, but I do feel like AKS was released to GA too soon. Without a published roadmap and scant acknowledgment of issues, I'm not sure I could recommend it to my clients or employer. It's disappointing, because I've had few issues with other Azure services. Full disclosure: I receive a monthly credit through a Microsoft program for Azure.

markbnj 8y ago
> I've wondered a few times if other platforms have similar issues and have seen more than a few complaints about EKS. I can't speak to EKS but we've been running production workloads on GKE for over a year with very good results. There have been a very few really troublesome "growing pains" type issues (an early example: loadbalancer services getting stuck with a pending external IP assignment for days) but Google has been awesome about support, even to the extent of getting Tim Hockin and Brendan Burns on the phone with us at various times to gather information about stuff like the example I gave above. I give them high marks and would recommend the service without hesitation.
- curiousDog 8y ago
  Brendan Burns is the lead engineer on AKS AFAIK
  - mohamedmansour 8y ago
    Brendan Burns was the Lead Engineer on Kubernetes at Google around 3 years ago, then went to Microsoft to lead this entire AKS/ACS/Container effort. https://www.linkedin.com/in/brendan-burns-487aa590 https://www.linkedin.com/in/brendan-burns-487aa590

ubuntunero 8y ago

interesting. thanks

rdl 8y ago

Key Vault (their HSM product) is even worse.

gercheq 8y ago

Azure is not bad but there are definitely some rough edges. We're having trouble with their Bizspark Sponsorship biling https://news.ycombinator.com/item?id=17698948 https://news.ycombinator.com/item?id=17698948

stefanatfrg 8y ago

Couple of questions to the OP: 1. What version of docker / container runtime is being used? 2. What base image for your containers is being used? eg. alpine has known DNS issues [1] [1] https://www.youtube.com/watch?v=ZnW3k6m5AY8 https://www.youtube.com/watch?v=ZnW3k6m5AY8

bsaul 8y ago

Side question : what are the best practices for development ? Are you suppose to run a local kubernetes deployment ( it looks like it's pretty hard to set up) , or do you run everything outside of containers when developping and then deal with k8 packaging and deployment as a completely separated issue ( which looks like it could lead to discovering a lot of issues on the preproduction environment) ?

crazy_monkey 8y ago

This is basically "the horrors of using any relatively new feature on Azure and by Microsoft"

hb3b 8y ago

I joined a healthcare startup in 2014 that had a small infrastructure on Azure. Back then AWS weren't signing BAAs and Azure was the only player in town. Being an early startup, the company didn't purchase a support plan from Azure. One day Azure suffered a major outage (may have been storage related) and over an hour later, I reached out to Microsoft for written confirmation that we could forward to customers. Since we didn't have a support plan they flat-out refused to provide any documentation whatsoever about the issue. They wanted $10,000. Azure - never again. Company moved to AWS within a quarter.

nojvek 8y ago

I believe this is a cultural problem with Microsoft. Probably similar to other companies but it was very evident at Microsoft. People responsible to allocate resources (The management chain) rarely dogfood the product. While the Engineers and PM would complain a lot about quality issues, management wants to prioritize more features. It was a running joke at Microsoft: No one gets promoted for improving existing things, if you want a quick promo, build a new thing. So when you see a bazillion half baked things in Azure. That’s because someone got promoted for building each of those half baked things and moving on to the next big thing. Going from 0-90% is the same amount of work as 90-99% and the same amount of work as 99.0% - 99.99%. Making things insanely great is hard and requires a lot of dedicated focus and a commitment to set a higher bar for yourself.

FlorianRappl 8y ago

We have a larger migration project going on for months. So far not a single failure occurred and our TEST environment is already fully migrated (quite responsive and rock solid) since 2 weeks. However, I do share that Azure indeed has released a lot of half-baked features and services lately (last 1.5 to 2 years). I hope this trend does not continue.

parasubvert 8y ago

DNS failures were almost certainly related to all the k8s system services on the cluster not having CPU or memory reservations, and KubeDNS was flaking. In general AKS is a vanilla k8s cluster and expects you know what you’re doing. MS arguably should enforce some opinions about how things like system services have reservations, etc, but none of this is vanilla. The trouble is that K8s defaults are pretty poor from a security (no seccomp profiles or apparmor/se profiles) and performance perspective (no reservations on key system DaemonSets). We’ve had this interesting industry pendulum swing between extreme poles of “we hate opinionated platforms! Give me all the knobs!” And “this is too hard, we need opinions and guard rails!”. I think the success of K8s is exposing people to the complexity of supplying all of the config details yourself and we will see a new breed of opinionated platforms on top of it very shortly. It reminds me of the early Linux Slackware and SLS and Debian days where people traded X11 configs and window manager configs like they were treasured artifacts before Red Hat, Gnome and KDE, SuSE, and eventually Ubuntu, started to force opinions.