11 ms·

Microsoft doubles down on Kubernetes for Azure

200 points by CrankyBear 9y ago

gkop 9y ago
Edit: Thanks to replies explaining that in fact AKS doesn't charge for the master! I'm leaving the rest of this comment standing because it's an honest reply to the article posted - Microsoft isn't disingenuous, this article is just wrong. > Be that as it may, Microsoft is offering AKS for free. You will only pay for the virtual machines (VM) that you use for managing your Kubernetes cluster. Microsoft says, “Unlike other cloud providers who charge an hourly rate for the management infrastructure, with AKS you will pay nothing for the management of your Kubernetes cluster, ever. After all, the cloud should be about only paying for what you consume.” This is disingenuous. We're talking about k8s, so GKE is the comparison point. Yes GKE charges a flat hourly fee for cluster management, regardless of the number or size of the nodes[0]. But GKE doesn't charge for the master instances (which are hidden away behind the GKE product). So AKS will not charge for the cluster management, but will charge for the master instances. Neither pricing model is more "for free" than the other. Also, if we have to think about the master instance sizes on AKS because we're paying for them, then that's one thing we have to manage on AKS that we don't have to manage on GKE - GKE adds more value by more completely managing our cluster. [0] pedantic: there's actually no fee for clusters of less than 6 nodes.
- clhodapp 9y ago
  GKE does charge for the masters for clusters of 6 or more nodes. See https://cloud.google.com/kubernetes-engine/pricing https://cloud.google.com/kubernetes-engine/pricing. Microsoft's offering does not seem to charge you for the masters ever, so that does seem like a differentiator.
  - gkop 9y ago
    You write "Microsoft's offering does not seem to charge you for the masters ever". The article says "You will only pay for the virtual machines (VM) that you use for managing your Kubernetes cluster". Do you have a source that disagrees with the article? Edit: to those replying correcting the article, if you were to create a top-level comment on this thread correcting the article, we could upvote your comments and better correct the misinformation. I have edited my own comment explaining what you have dug up. Thanks for your research!
    - Voloskaya 9y ago
      https://docs.microsoft.com/gl-es/azure/aks/intro-kubernetes https://docs.microsoft.com/gl-es/azure/aks/intro-kubernetes > "you pay only for the agent nodes within your clusters, not for the masters."
      - justinjlynn 9y ago
        Which is to be read "the price of the Master is incorporated into the amortised price per unit time of the agent nodes". Microsoft is not giving you something for nothing here.
        plasma 9y ago
        To be fair to MS here, the agent price is the same as the stand-alone VM pricing, so there really does not appear to be a premium/hidden cost here.
        justinjlynn 9y ago
        Either that or everyone is paying some kind of premium to subsidise the service. I would be extremely surprised if it's not rolled into some cost of goods sold metric -- if it's one thing microsoft is not, it's being bad at passing on costs.
    - AaronFriel 9y ago
      Yes, I have played with AKS. You do not pay for the Kubernetes masters, just the nodes. In the prior version of the product (ACS) you would specify a cluster size as a number of masters and a number of workers/nodes. In AKS, you only manage the latter and Azure abstracts away the Kubernetes API. https://azure.microsoft.com/en-us/pricing/details/container-service/ https://azure.microsoft.com/en-us/pricing/details/container-...
      - chimeracoder 9y ago
        > Yes, I have played with AKS. You do not pay for the Kubernetes masters, just the nodes. In the prior version of the product (ACS) you would specify a cluster size as a number of masters and a number of workers/nodes. In AKS, you only manage the latter and Azure abstracts away the Kubernetes API. On GKE, if i run a Kubernetes cluster of size 1, I'm still charged for a single VM. You're saying that if I do the equivalent on Azure, I won't get charged at all?
- 013a 9y ago
  Statement's like Microsoft's are horrible; "the cloud should be about paying for what you consume" well, Kubernetes consumes at least one master node in order to orchestrate a cluster. But, Google/GKE does charge for management of the cluster. A constant fee of $109/mo for 6+ nodes. Its not technically the charge for the master node, but that cost to Google + other things is likely where that $109/mo comes from.
- AaronFriel 9y ago
  > Also, if we have to think about the master instance sizes on AKS because we're paying for them, then that's one thing we have to manage on AKS that we don't have to manage on GKE This is entirely premised on your speculation which turned out to be false. Curious that you're willing to call this blog post disingenuous when you're the one spreading falsehoods?
  - gkop 9y ago
    "You will only pay for the virtual machines (VM) that you use for managing your Kubernetes cluster."
- thockingoog 9y ago
  Google and GKE team member here: The AKS announcement was either poorly worded or intentionally vague. I choose to grant the benefit of the doubt. For small clusters (less than 1-5 nodes) GKE does not currently charge anything "extra" except your own node VMs. That is LITERALLY a $0 master. Beyond 5 nodes we currently charge a flat rate that covers your zonal master(s) regardless of how many or how big they need to be.
  - thockingoog 9y ago
    ...and as of today there is no fee for masters on GKE. https://cloudplatform.googleblog.com/2017/11/Cutting-Cluster-Management-Fees-on-Google-Kubernetes-Engine.html https://cloudplatform.googleblog.com/2017/11/Cutting-Cluster...
rad_gruchalski 9y ago
I really, really hope aws joins the game. re:invent is around the corner.
- fredsted 9y ago
  Well, you can use something like kops, although it's a little more difficult to get started, I guess. This new tool from Azure is basically the same thing. It's a little interesting since they already have ECS, which isn't that bad either, but it seems like everyone just wants Kubernetes. Will they do both, or provide a migration path?
  - rad_gruchalski 9y ago
    ECS is okay, used it quite a bit recently. It definitely has some quirks to it, like having to use CloudWatch events to figure out the port given to the container when using bridge network. Maybe if the service discovery story was a little bit more defined, ECS would be easier to use. There's lots of little manual things one has to do around it to make it work as swift as other solutions available. With regards to k8s on AWS, I do not have a problem running it myself but some backing from AWS definitely help defining k8s as "the solution" for running containerized workflows. Here's one hoping.
- geerlingguy 9y ago
  Didn’t AWS just join the Kubernetes foundation earlier this year? If so, we’ll see what comes. One thing ECS does is fully integrate ALBs and NLBs with containerized services and makes the plumbing and scaling pretty easy to set up (at least IMO. If they can swap out the custom ECS service that manages containers with K8s, that makes sense to me and would mean all the major cloud providers are centralizing on K8s... which in turn means we could stop building a new container orchestration layer every month and move on to some new problem :)
- Saaster 9y ago
  I'm almost sure they will. They don't have much of a NIH problem (look at how many DBs they support, anyone remember SimpleDB?), ECS hasn't exactly taken off, and they're generally very good at giving whatever their customers want.
mephitix 9y ago
This is a good thing; Microsoft will increase competition in this space by applying its expertise in dev tools to Kubernetes. I just hope that MS doesn't add in too many platform-specific pieces that would encourage vendor lock-in. For example k8s ingress right now is based on controllers and GCE controllers support/don't support a wide variety of things compared to nginx ingress... these areas of Kubernetes make me worried about potential future switching costs.
- JoshMnem 9y ago
  > I just hope that MS doesn't add in too many platform-specific pieces that would encourage vendor lock-in. That's Microsoft's main strategy though. Example: Users can't even install Firefox or Chrome on Windows 10 S. Microsoft announced: "Apps that browse the web must use the appropriate HTML and JavaScript engines provided by the Windows Platform." I wouldn't be the slightest bit surprised to see similar things happening to their cloud platform as soon as they are able to do them without users' noticing.
  - Kipters 9y ago
    That's a different product, with different goals. You're right about 10 S, but it's kinda unrelated to Azure.
    - JoshMnem 9y ago
      I've lived through the experience of using non-MS products since the early 2000s, and all of those things are related. It has been one of their core strategies and still is, though the techniques are shifting.
  - lewisl9029 9y ago
    > "Apps that browse the web must use the appropriate HTML and JavaScript engines provided by the Windows Platform." This seems like a thing that platform owners are doing more and more (iOS has a similar clause, and ChromeOS, well, uses Chrome). Is this just purely a move to stifle competition from web apps that could threaten control of the platform? Or perhaps I am missing some nuance behind these kinds of decisions? I'm honestly struggling to find any room to give them benefit of the doubt here.
org3432 9y ago
So restaurants give you ketchup for "free" too, but turns out you're paying for it. This is stupid to say the management node is free, you're just paying for it in another way. Now if they said they were able to run the node so cost effectively that it's just included in the hourly node cost, then that's a plus.
- bonesss 9y ago
  For SMBs and smaller dev shops kicking off their forays into container development, not to mention customer deployments, the simplified cost models make it more management friendly and more understandable to non-IT actors. Whether it is absolutely cheaper than GCE or whatever, and if master management is a requirement, is a separate comparison based on the solution. Cutting out the management node costs removes a barrier to entry and creates a more streamlined product. Oh, it also makes their Kubernetes services usable within the free credits they give out on Azure via their MSDN subscriptions... so suddenly millions of devs can use AKS for demos and temp-hosting "for free", which is huge in creating mindshare in the Enterprise...
  - org3432 9y ago
    I'm sure you're right, but not understand economics to that level wow.
Maledictus 9y ago
I hope they don't embrace, extend, extinguish.
amq 9y ago
Whenever I look at Kubernetes, I can't help myself but think that it's overcomplicated, with too many moving parts. Almost as if to incentivise using the hosted solutions instead of rolling it on your own on some VPSs. I can deploy a HA Swarm Mode cluster anywhere in minutes, while I'm not sure even where to start with Kubernetes. Do I use juju (recommended Ubuntu way), or do I use kubeadm? Which network add-on do I use? How do I upgrade (the procedure seems to be different with each version)?
- koko775 9y ago
  Well...at least it's not OpenStack! Out of the couple of stacks I tried, kops was the easiest to set up. Juju and kubeadm were so-so.
- plesiv 9y ago
  It doesn't help that most of the open source solutions to deploy a cluster are either poorly engineered (e.g. kubespray) or poorly maintained (e.g. kubeadm). You are on to something... organizations/people that have the capability and resources to make the toolings to manage Kubernetes setup complexity seem to think they're much better off if they charge for it.
- vhffm 9y ago
  There's a way of setting up a bare-metal (or VMs) cluster using Ansible. It may be the sort of 'easy in' you may be looking for: https://kubernetes.io/docs/getting-started-guides/fedora/fedora_ansible_config/ https://kubernetes.io/docs/getting-started-guides/fedora/fed... If you manage to automate deployment of the VMs or physical nodes (e.g., PXE), you've got a cherry on top.
- nvarsj 9y ago
  If you’re not prepared to dedicate a small team to running Kubernetes, I’d say it’s too much for your needs. Read the Borg paper for an idea of what k8s is meant to solve. If you have hundreds of services spread across 10s/100s of nodes, then the complexity of k8s starts to become amortized. Similarly if you have 100s of devs without a consistent way to do deployments, k8s starts to make sense. Source: tech lead for a team managing a large on-prem and cloud k8s cluster.

200 points by CrankyBear 9y ago

gkop 9y ago

Edit: Thanks to replies explaining that in fact AKS doesn't charge for the master! I'm leaving the rest of this comment standing because it's an honest reply to the article posted - Microsoft isn't disingenuous, this article is just wrong. > Be that as it may, Microsoft is offering AKS for free. You will only pay for the virtual machines (VM) that you use for managing your Kubernetes cluster. Microsoft says, “Unlike other cloud providers who charge an hourly rate for the management infrastructure, with AKS you will pay nothing for the management of your Kubernetes cluster, ever. After all, the cloud should be about only paying for what you consume.” This is disingenuous. We're talking about k8s, so GKE is the comparison point. Yes GKE charges a flat hourly fee for cluster management, regardless of the number or size of the nodes[0]. But GKE doesn't charge for the master instances (which are hidden away behind the GKE product). So AKS will not charge for the cluster management, but will charge for the master instances. Neither pricing model is more "for free" than the other. Also, if we have to think about the master instance sizes on AKS because we're paying for them, then that's one thing we have to manage on AKS that we don't have to manage on GKE - GKE adds more value by more completely managing our cluster. [0] pedantic: there's actually no fee for clusters of less than 6 nodes.

clhodapp 9y ago
GKE does charge for the masters for clusters of 6 or more nodes. See https://cloud.google.com/kubernetes-engine/pricing https://cloud.google.com/kubernetes-engine/pricing. Microsoft's offering does not seem to charge you for the masters ever, so that does seem like a differentiator.
- gkop 9y ago
  You write "Microsoft's offering does not seem to charge you for the masters ever". The article says "You will only pay for the virtual machines (VM) that you use for managing your Kubernetes cluster". Do you have a source that disagrees with the article? Edit: to those replying correcting the article, if you were to create a top-level comment on this thread correcting the article, we could upvote your comments and better correct the misinformation. I have edited my own comment explaining what you have dug up. Thanks for your research!
  - Voloskaya 9y ago
    https://docs.microsoft.com/gl-es/azure/aks/intro-kubernetes https://docs.microsoft.com/gl-es/azure/aks/intro-kubernetes > "you pay only for the agent nodes within your clusters, not for the masters."
    - justinjlynn 9y ago
      Which is to be read "the price of the Master is incorporated into the amortised price per unit time of the agent nodes". Microsoft is not giving you something for nothing here.
      - plasma 9y ago
        To be fair to MS here, the agent price is the same as the stand-alone VM pricing, so there really does not appear to be a premium/hidden cost here.
        justinjlynn 9y ago
        Either that or everyone is paying some kind of premium to subsidise the service. I would be extremely surprised if it's not rolled into some cost of goods sold metric -- if it's one thing microsoft is not, it's being bad at passing on costs.
  - AaronFriel 9y ago
    Yes, I have played with AKS. You do not pay for the Kubernetes masters, just the nodes. In the prior version of the product (ACS) you would specify a cluster size as a number of masters and a number of workers/nodes. In AKS, you only manage the latter and Azure abstracts away the Kubernetes API. https://azure.microsoft.com/en-us/pricing/details/container-service/ https://azure.microsoft.com/en-us/pricing/details/container-...
    - chimeracoder 9y ago
      > Yes, I have played with AKS. You do not pay for the Kubernetes masters, just the nodes. In the prior version of the product (ACS) you would specify a cluster size as a number of masters and a number of workers/nodes. In AKS, you only manage the latter and Azure abstracts away the Kubernetes API. On GKE, if i run a Kubernetes cluster of size 1, I'm still charged for a single VM. You're saying that if I do the equivalent on Azure, I won't get charged at all?
013a 9y ago
Statement's like Microsoft's are horrible; "the cloud should be about paying for what you consume" well, Kubernetes consumes at least one master node in order to orchestrate a cluster. But, Google/GKE does charge for management of the cluster. A constant fee of $109/mo for 6+ nodes. Its not technically the charge for the master node, but that cost to Google + other things is likely where that $109/mo comes from.
AaronFriel 9y ago
> Also, if we have to think about the master instance sizes on AKS because we're paying for them, then that's one thing we have to manage on AKS that we don't have to manage on GKE This is entirely premised on your speculation which turned out to be false. Curious that you're willing to call this blog post disingenuous when you're the one spreading falsehoods?
- gkop 9y ago
  "You will only pay for the virtual machines (VM) that you use for managing your Kubernetes cluster."
thockingoog 9y ago
Google and GKE team member here: The AKS announcement was either poorly worded or intentionally vague. I choose to grant the benefit of the doubt. For small clusters (less than 1-5 nodes) GKE does not currently charge anything "extra" except your own node VMs. That is LITERALLY a $0 master. Beyond 5 nodes we currently charge a flat rate that covers your zonal master(s) regardless of how many or how big they need to be.
- thockingoog 9y ago
  ...and as of today there is no fee for masters on GKE. https://cloudplatform.googleblog.com/2017/11/Cutting-Cluster-Management-Fees-on-Google-Kubernetes-Engine.html https://cloudplatform.googleblog.com/2017/11/Cutting-Cluster...

rad_gruchalski 9y ago

I really, really hope aws joins the game. re:invent is around the corner.

fredsted 9y ago
Well, you can use something like kops, although it's a little more difficult to get started, I guess. This new tool from Azure is basically the same thing. It's a little interesting since they already have ECS, which isn't that bad either, but it seems like everyone just wants Kubernetes. Will they do both, or provide a migration path?
- rad_gruchalski 9y ago
  ECS is okay, used it quite a bit recently. It definitely has some quirks to it, like having to use CloudWatch events to figure out the port given to the container when using bridge network. Maybe if the service discovery story was a little bit more defined, ECS would be easier to use. There's lots of little manual things one has to do around it to make it work as swift as other solutions available. With regards to k8s on AWS, I do not have a problem running it myself but some backing from AWS definitely help defining k8s as "the solution" for running containerized workflows. Here's one hoping.
geerlingguy 9y ago
Didn’t AWS just join the Kubernetes foundation earlier this year? If so, we’ll see what comes. One thing ECS does is fully integrate ALBs and NLBs with containerized services and makes the plumbing and scaling pretty easy to set up (at least IMO. If they can swap out the custom ECS service that manages containers with K8s, that makes sense to me and would mean all the major cloud providers are centralizing on K8s... which in turn means we could stop building a new container orchestration layer every month and move on to some new problem :)
Saaster 9y ago
I'm almost sure they will. They don't have much of a NIH problem (look at how many DBs they support, anyone remember SimpleDB?), ECS hasn't exactly taken off, and they're generally very good at giving whatever their customers want.

mephitix 9y ago

This is a good thing; Microsoft will increase competition in this space by applying its expertise in dev tools to Kubernetes. I just hope that MS doesn't add in too many platform-specific pieces that would encourage vendor lock-in. For example k8s ingress right now is based on controllers and GCE controllers support/don't support a wide variety of things compared to nginx ingress... these areas of Kubernetes make me worried about potential future switching costs.

JoshMnem 9y ago
> I just hope that MS doesn't add in too many platform-specific pieces that would encourage vendor lock-in. That's Microsoft's main strategy though. Example: Users can't even install Firefox or Chrome on Windows 10 S. Microsoft announced: "Apps that browse the web must use the appropriate HTML and JavaScript engines provided by the Windows Platform." I wouldn't be the slightest bit surprised to see similar things happening to their cloud platform as soon as they are able to do them without users' noticing.
- Kipters 9y ago
  That's a different product, with different goals. You're right about 10 S, but it's kinda unrelated to Azure.
  - JoshMnem 9y ago
    I've lived through the experience of using non-MS products since the early 2000s, and all of those things are related. It has been one of their core strategies and still is, though the techniques are shifting.
- lewisl9029 9y ago
  > "Apps that browse the web must use the appropriate HTML and JavaScript engines provided by the Windows Platform." This seems like a thing that platform owners are doing more and more (iOS has a similar clause, and ChromeOS, well, uses Chrome). Is this just purely a move to stifle competition from web apps that could threaten control of the platform? Or perhaps I am missing some nuance behind these kinds of decisions? I'm honestly struggling to find any room to give them benefit of the doubt here.

org3432 9y ago

So restaurants give you ketchup for "free" too, but turns out you're paying for it. This is stupid to say the management node is free, you're just paying for it in another way. Now if they said they were able to run the node so cost effectively that it's just included in the hourly node cost, then that's a plus.

bonesss 9y ago
For SMBs and smaller dev shops kicking off their forays into container development, not to mention customer deployments, the simplified cost models make it more management friendly and more understandable to non-IT actors. Whether it is absolutely cheaper than GCE or whatever, and if master management is a requirement, is a separate comparison based on the solution. Cutting out the management node costs removes a barrier to entry and creates a more streamlined product. Oh, it also makes their Kubernetes services usable within the free credits they give out on Azure via their MSDN subscriptions... so suddenly millions of devs can use AKS for demos and temp-hosting "for free", which is huge in creating mindshare in the Enterprise...
- org3432 9y ago
  I'm sure you're right, but not understand economics to that level wow.

Maledictus 9y ago

I hope they don't embrace, extend, extinguish.

amq 9y ago

Whenever I look at Kubernetes, I can't help myself but think that it's overcomplicated, with too many moving parts. Almost as if to incentivise using the hosted solutions instead of rolling it on your own on some VPSs. I can deploy a HA Swarm Mode cluster anywhere in minutes, while I'm not sure even where to start with Kubernetes. Do I use juju (recommended Ubuntu way), or do I use kubeadm? Which network add-on do I use? How do I upgrade (the procedure seems to be different with each version)?

koko775 9y ago
Well...at least it's not OpenStack! Out of the couple of stacks I tried, kops was the easiest to set up. Juju and kubeadm were so-so.
plesiv 9y ago
It doesn't help that most of the open source solutions to deploy a cluster are either poorly engineered (e.g. kubespray) or poorly maintained (e.g. kubeadm). You are on to something... organizations/people that have the capability and resources to make the toolings to manage Kubernetes setup complexity seem to think they're much better off if they charge for it.
vhffm 9y ago
There's a way of setting up a bare-metal (or VMs) cluster using Ansible. It may be the sort of 'easy in' you may be looking for: https://kubernetes.io/docs/getting-started-guides/fedora/fedora_ansible_config/ https://kubernetes.io/docs/getting-started-guides/fedora/fed... If you manage to automate deployment of the VMs or physical nodes (e.g., PXE), you've got a cherry on top.
nvarsj 9y ago
If you’re not prepared to dedicate a small team to running Kubernetes, I’d say it’s too much for your needs. Read the Borg paper for an idea of what k8s is meant to solve. If you have hundreds of services spread across 10s/100s of nodes, then the complexity of k8s starts to become amortized. Similarly if you have 100s of devs without a consistent way to do deployments, k8s starts to make sense. Source: tech lead for a team managing a large on-prem and cloud k8s cluster.