11 ms·

Kubernetes on Google, Azure and AWS Compared

228 points by stevenacreman 8y ago

lawrence143 8y ago
Good post, Steven. Keep going..
wskinner 8y ago
> Networking is the other reason. Google is miles ahead of everyone here. Similar story with HA and scaling. Does anyone know what the author is referring to with this claim? I don't see anything in the sheet to back this up. At least from a high level, all three options support network policies via CNI, and GKE and EKS use the same one, Calico.
- stevenacreman 8y ago
  Hi, there's a comment on the cross region networking for GKE. "Each cluster receiving an IP range for nodes and another for the containers inside, which are directly routable across your private network, other clusters, and regions." AWS and Azure don't have a flat global network. You have to setup VPN's and complicated overlay networks.
  - shshhdhs 8y ago
    > AWS and Azure don't have a flat global network. You have to setup VPN's and complicated overlay networks. AWS has cross-region VPC peering, which is simple to setup and makes it feel “flat”. There’s no need for VPNs or overlay networks.
    - nodesocket 8y ago
      Until you have to reference security groups across the peering connection. You can't reference a different region's security group id as a source, thus your forced to just whitelist the entire subnet/cidr. Also, while setting up peering is a 10-20 minute task there are lot of constraints, such as the VPC's have to have different CIDR ranges. Oops, if you go with the AWS default, probably all regions using 172.31.0.0/20. Google got networking right with a flat global any region setup. If you need segmentation then just create a new GCP project. If it turns out projects need to communicate after all, GCP has VPC network peering.
      - p0rkbelly 8y ago
        You can use PrivateLink for this instead of peering. That way you just hit a local IP address/ENI in your VPC. CIDR Ranges don't matter with that. (not sure about the x-region story) Two different design philosophies, there are been a global GCP networking outages due to having the single flat network. A few this year. All providers have had their issues, but, global blast radiuses concern me.
  - deleted 8y ago
    [deleted]
  - halbritt 8y ago
    Couple things: Cross-region networking. To this day, I'm flabbergasted this isn't something that AWS has. All the other networking bits seem more sensible to me as well. Another, is 2Gbps per core up to 8 cores which is way more throughput than I've seen on any AWS or Azure instance. On workloads where I care about Network IO:CPU ratio, I'm using 8 core nodes and seeing 16gbps throughput between them consistently.
    - ti_ranger 8y ago
      > Cross-region networking. To this day, I'm flabbergasted this isn't something that AWS has. All the other networking bits seem more sensible to me as well. AWS tries to avoid letting customers create multi-region or global failure modes, like: * https://status.cloud.google.com/incident/compute/18005 https://status.cloud.google.com/incident/compute/18005 22-hour incident with GLOBAL impact resulting in many VMs getting duplicate IPs (this no network connectivity) on 2018-06-15 * https://status.cloud.google.com/incident/cloud-networking/18001 https://status.cloud.google.com/incident/cloud-networking/18... 3-hour (seemingly GLOBAL) impact to load-balancers (no updates or creations) on 2018-01-03 * https://status.cloud.google.com/incident/compute/16007 https://status.cloud.google.com/incident/compute/16007 - 18-minute GLOBAL outage on 2016-04-11 * https://status.cloud.google.com/incident/compute/15055 https://status.cloud.google.com/incident/compute/15055 - 5-minute GLOBAL outage on 2015-08-04 * https://groups.google.com/forum/#!topic/gce-operations/fynnXnb2OFs https://groups.google.com/forum/#!topic/gce-operations/fynnX... 43-minute multi-region packet loss due to multi-region configuration deployment on 2015-03-07 * https://groups.google.com/forum/#!topic/gce-operations/1uw-qEqjBdo https://groups.google.com/forum/#!topic/gce-operations/1uw-q... unscheduled reboot of 28% of instances in multiple regions in ~1.5h on 2014-09-17 There have been recent changes in many AWS services (e.g. DynamoDB, S3, Aurora) to allow cross-region replication without introducing any multi-region dependency, precisely to make it easier to implement multi-region infrastrucure to tolerate a single-region failure (however rare that is on AWS compared to global outages on GCP). > Another, is 2Gbps per core up to 8 cores which is way more throughput than I've seen on any AWS or Azure instance. Which instance types did use on AWS? Many recent instance types (r4, i3, c5, r5, z1) support up to 10Gbps with once core (2 vCPU instances). However, you may need to use placement groups ( https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placemen... ) in large regions in order to get the full throughput on low TCP connection numbers. The only reason I can think of that would explain why GCP doesn't have this problem is they don't have any regions anywhere near as large as large AWS regions ... Edit: white-space only formatting changes
- regnerba 8y ago
  I am not the author and haven't had time to fully review the spreadsheet they provided. After an initial look I think he is referring not to the kubernetes specific networking support but the GCP networking. From the spreadsheet alone GKE has support for cross region load balancers. The author also points out cross region networking. I don't know the exact advantage the author sees with cross region load balancers since you can only deploy a GKE cluster to a single given region. The GCP LB's are really nice though in that traffic enters Google's network at the closest location to the user and then travel within Google's network, which is wicked fast. I have not played around with federated clusters in multiple zones so don't know how that would play out either. They may also just be referring to just general networking from the providers. With the first two points they at least make mention in the spreadsheet of it though. tl;dr: not sure what the author meant but here here is some food for thought on it.
- dilyevsky 8y ago
  It supports addressable pods natively on vpc. Others (i think) do not. This simplifies ingess options a great deal. Also in general GCP networking is better than aws ime (ever tried to have multiple projects in aws share a network?). Don’t know anything about Azure but I suspect msft still sucks at networking things so it’s not great.
  - marcinzm 8y ago
    EKS pods use the VPC networking layer so I think they meet the criteria. They use secondary IPs attached to instances rather than virtual IPs. I’ve been able to route to them from non-EKS instances in the same VPC.
    - dilyevsky 8y ago
      Ah I see aws has their own cni that does that via eni. I never realized that it allowed for native pod addressing (and it was never brought up during their pitches). Might be worse performance for big clusters because it’s still using calico (iptables)
      - gtaylor 8y ago
        I think Calico is only used for network policy on the AWS VPC CNI plugin.
      - anything4dogs 8y ago
        > Might be worse performance for big clusters because it’s still using calico (iptables) I'm curious why you think that Calico's use of iptables will have performance impacts for large clusters, and what type of performance impacts you expect (bandwidth / latency / cpu / something else?). From my experience, Calico performs rather well in large clusters (e.g. 2k nodes, appx 100 pods / node, several hundred network policies).
Arnavion 8y ago
>Show them how some things on Azure need to be done in a clunky web UI, other things need Powershell and other random stuff uses the CLI. This, at least, is wrong. Everything related to Azure's hosted Kubernetes can be done via the CLI without ever touching the web portal or PS.
GordonS 8y ago
> Azure is something I’ve avoided since using it for a few months last year. I was working as a Microsoft partner so it was unavoidable back then. Parts of it are alright but the user experience coming from an Amazon background is worlds apart. Wat? IMO, the Azure portal is amazing to work with, especially compared to the old-fangled, inconsistent UI that AWS provides. > Show them how some things on Azure need to be done in a clunky web UI Years ago, this was true, but it hasn't been for a long time - the Azure web UI of today is fast, consistent and looks great. > some things on Azure need to be done in a clunky web UI, other things need Powershell and other random stuff uses the CLI I've been using Azure for years, and I'm not aware of anything that can only be done in the UI. I also don't believe there is anything that only works with Powershell, or only works with the CLI. > how that effects the design of DevOps pipelines and automation in general. Yes, you can make it work, but why make life hard for yourself Eh? Azure DevOps pipelines are great to work with, and there's a huge library of tasks available. After reading this, it doesn't sound like the author has actually worked with Azure recently, so I'm really not sure why he bothered including it in this article.
- sofaofthedamned 8y ago
  Agree completely. AWS portal is bizarre, it's like each portion is written by a different company. In ECR for example, clicking into a repo is fine, but then going back doesn't do anything. Why? Even simple favicons don't look like they're from the same company. When you've got 40 tabs open you want an idea that that particular tab is from AWS or not. Azure is nicer, but it's not reliable (see my other post on this thread). It's almost like Azure have done too much lazy-loading which persists through login sessions, you can't rely on a refresh to give you the truth - for all the problems with AWS, you can. And yes I know this isn't CLI stuff, but it's important. Even on fully automated / scripted deployments it's sometimes nice to go into the UI and see what's going on. If it doesn't represent reality, there is a problem. FWIW GC doesn't have any of these problems, but it is slow.
  - GordonS 8y ago
    You only mention Azure DevOps in your other comment - have you used Azure in a wider sense? As another datapoint of one, I've been working with Azure for years, and have never found the UI to be unreliable.
sofaofthedamned 8y ago
Interesting post. I'm a Linux guy but i'm in a new job so having to ramp up on Azure DevOps. Looked good at first, I actually praised Microsoft but: 1. Creating a project timed out after 5 minutes or so. This was in the gui, but had no adblocker or anything similar. Refreshed page = no project. 2. Went to a different machine, logged in, no project still. 3. Went to original machine - refreshed - no project. Logged out and back in again - project was there. Azure DevOps was awesome at first but I can not trust something like this when I work with it all day.
- toomuchtodo 8y ago
  I refuse to work with Azure because of issues like this.
  - sofaofthedamned 8y ago
    I don't have a choice unfortunately. BTW never had a problem bringing up an instance quickly on GC. AWS sometimes just sat there for a while, depending on the instance type and region. Azure just stops because it feels like it. I'm sure GC isn't any better than the other two, it's just less utilised. The UI is my biggest issue. Yes, we don't depend on it because we all script our stuff. But after spending an hour fapping at an SSH terminal it's nice to look at the UI and see stuff happen. Azure is the worst at this, with AWS a close second. That reminded me of my favourite bug with AWS UI / Container Registry - you can order the list of container images any way you like, but it'll only order those it's loaded. So i'm looking for a new image called 'zzzzz' but it'll never appear, I need to search for it. Did anybody involved ever use this?
    - toomuchtodo 8y ago
      > I don't have a choice unfortunately. I won't say you always have a choice, but I'm happy to hook you up with my network if you want to passively seek out a new role. Email in my profile. Sidenote: I have seen the ECR UX shenanigans you mention.
- switch007 8y ago
conradk 8y ago
For "Maximum pods per node", it shows GKE at 100, AKS at 110 but still puts GKE in green. How come?
- cobookman 8y ago
  Does anyone even have more than 20 pods / node? I've not personally seen anything near the 100 pods / node limit in the real world.
  - sofaofthedamned 8y ago
    I didn't have 100, but I did have around 60 of my own plus all the associated K8s pods (about 10 of them). It's a bad idea to have that many if they're all Apache or whatever. But when they're simple Golang containers that handle APIs it's ok.
  - regnerba 8y ago
    We don't have 100, but definitely more than 20. A quick look at a random node from our cluster: - 3x ingress pods (1 for an internal load balancer, 2 for an external load balancer) - 1x cert-manager - 2x production version of internal app01 - 2x staging version of internal app02 - 1x fluentd - 1x elastalert - 1x kubewatch - 1x prometheus node exporter - 1x redis for sentry - 1x sentry web - 2x sentry worker - 4x sourcegraph language servers (there are 9 language servers running across the 4 nodes, this node seems to especially like them) - 1x thelounge - 1x staging version of app02 - 2x production version of app02 - 1x kube proxy - 1x kubedns - 1x couchdb (small footprint just for testing some things) the node is just over half provisioned and we are due to add another node to the cluster soon.
    - deepsun 8y ago
      What does prometheus node exporter collects from within a pod? I was under impression that to collect node stats you absolutely need to be "outside" any container, i.e. using prometheus-k8s integration so that it pulls nodes stats from k8s api, not nodes themselves).
inscrutable 8y ago
Fully agree with his comments as someone who's used GKE, AKS, EKS and Hetzner for kubernetes clusters. GCP's UX is so nice... e.g. compare the equivalent as command line option in the UI vs the automation script that Azure gives you.
- GordonS 8y ago
  Azure actually gives you several ways to do things, of which automation scripts (ARM templates) are just one: 1. Web UI 2. Powershell 3. CLI 4. REST API I know what you mean about ARM templates though - the auto-generated ones in particular are extremely verbose!
- krn 8y ago
  > [...] as someone who's used GKE, AKS, EKS and Hetzner for kubernetes clusters. How does Hetzner compare here? In terms of both, general experience and costs.
  - cardine 8y ago
    >How does Hetzner compare here? In terms of both, general experience and costs. We use Hetzner for this purpose. It's hard to compare it with the other providers since with Hetzner you are renting unmanaged servers, not turning up cloud instances. But it is absolutely the best value you can get. Our bill would be at least 10x higher if we were using Google/Amazon/Microsoft. Considering we spent a lot on servers as is, that makes it well worth it for us. It might not be if your product is very computationally light.
    - segmondy 8y ago
      I don't know your location, but if you are in the US and running your own k8s on baremetals, I would reckon you should have at least 1 full time person on staff to handle all of it. Let's say that person costs only $50k/yr. Will your workload on Google cost more than $50k? Google pretty much really does all of the management for GKE. Just deploy your app and run.
kubenaught 8y ago
My company is in the process of migrating off of a managed kubernetes provider. Sure, it's nice to have someone else manage the operations of the master. At the same time, a single customer is entirely insignificant to them. We've experienced multiple outages from forced upgrades. We usually can find out the reason through github, but it may not be a priority for them to provide the fix. If they do, it could take days or weeks for it to become available. Much revenue has been lost because they can't do something at the speed which we could do it.
nbevans 8y ago
His hatred for Azure ruins the article. There seems to be a massive correlation at the moment between Kubernetes and hyped up magpie egomaniac developers.
- FlorianRappl 8y ago
  I thought 100% the same. While it is known that GKE yields the best managed Kubernetes experience I thought the whole Azure bashing was not only pointless, but also flawed with many inaccurate and even false statements. So much to "honest".
  - halbritt 8y ago
    It correlates with my experience pretty closely. Granted, I haven't really used Azure in six months.
- bk24 8y ago
  The Azure bashing had very little substance to it, just alot of previous MS experience-based dogma.
andyfleming 8y ago
I'm curious how the digital ocean offering compares. I think it's in limited availability now.
bsaul 8y ago
A little of topic because it’s not a comparison : Recently had my first experience using gke ( and kubernetes in general), and although i managed to get something working in the end i would say it’s still pretty rough... Documentation is a mess : the general layout for google cloud is really a pain to read and navigate, but in addition to that you often have to jump between kubernetes doc and google doc, with some information being on both. Don’t do that please : either make it obvious that people need to get familiar with certain chapters of kube doc, or provide all the info ( me preference would actually go to the first option..) It’s quite hard to guess what you can do on the gke web interface and what you can’t. You can feel it’s meant for people that really know kubernetes, and not people who discover both gke and kubernetes at the same time. And for the life of me i couldn’t get the load balancer manage https. I’ve read this was possible, but never saw the actual page explaining how. I ended up using cloudfront but lost the ability to see end user ip in my logs in the process. ( also logging is a real beast to tame on its own, with no obvious way to know what is available by default, what is a paid option, what should be configured on the stackdriver website, and what should be coded )
- alpb 8y ago
  TLS with GKE with BYO cert isn't very clear I agree. It's basically in Kubernetes Ingress docs: https://kubernetes.io/docs/concepts/services-networking/ingress/#tls https://kubernetes.io/docs/concepts/services-networking/ingr... For GKE specific features, multiple-certs per Ingress explained here: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-multi-ssl https://cloud.google.com/kubernetes-engine/docs/how-to/ingre..., and LetsEncrypt here: https://github.com/ahmetb/gke-letsencrypt https://github.com/ahmetb/gke-letsencrypt If you use GKE ingress, you can access end-user IP in X-Forwarded-For header.
  - bsaul 8y ago
    but then does that mean you’ll need to manually log incoming ip from the header ? or is there a way to see them in the default logs of either the cluster or the load balancer ?
paxy 8y ago
The service that launched on Azure in 2017 was ACS (Azure Container Service), which is very different from what AKS is today.
capkutay 8y ago
Slightly unrelated, but what would be the best way to migrate a kubernetes application from one cloud to the other?
- halbritt 8y ago
  If you write your application correctly, it should be portable between kubernetes clusters. Figuring out how to manage kubernetes on different providers can be a little tricky.
- scirocco 8y ago
  OpenShift is a Kubernetes-based container platform that can run on AWS, Azure, Google...
dfischer 8y ago
I tried Azure lately – it felt like I booted into Windows 98. I couldn't stand it. A lot of the CLI UX was buggy for me too. AWS works. A lot of the UI is dated but the new designs are nice. GCP is the best. Google has done a great job on Dev UX both CLI and Web Console.
- amf12 8y ago
  > I tried Azure lately – it felt like I booted into Windows 98. I couldn't stand it. Care to elaborate?
  - dfischer 8y ago
    The sidebar immediately took me back aesthetically. UX wise I felt it was terrible drilling down into resources I was looking for. Of course UX is also based on my bias (AWS and GCP). Nonetheless I was surprised at the presentational bar for the azure UI. It felt very dated. In the end I got a hang of it but again, it still felt like using an old pc. CLI wise I had a lot of errors spinning up resources. My support experience getting it figured out was sucky. I had to abandon azure as a decision quickly with that experience. Too risky.
    - bk24 8y ago
      a man of many subjective complaints
      - dfischer 8y ago
        a person lacking constructive comments
drewmassey 8y ago
After several weeks of swimming upstream on a greenfield EKS project I switched gears to GKE. The paradigm just felt way better. For large enterprises that need IAM integration on EKS I suppose that wouldn’t have been an option but GKE just feels way more paradigmatic, for obvious reasons. Did I mention we are hiring devops engineers? If you are a kubernetes guru email me :-)
api 8y ago
We are preparing to move to Google. Our app is CPU intensive and a killer feature was the ability to spec nodes with a lot of cores and little RAM. We save a lot of money by not paying for RAM we dont need. Google also had a location in LA and we are in LA. 5ms to our cloud is nice.
mverwijs 8y ago
Anyone here want to share their experiences running k8s on IBM Bluemix (formerly Softlayer)?
- po 8y ago
  I use and like IBM Cloud (Softlayer) for k8s stuff... I find they are making improvements to it (including docs) on a regular basis and I can often find and talk directly to their devs in a slack channel. Some parts are a bit clunky and I feel like I can see their legacy stuff poking through the abstractions but overall I find it to be good. I tried Azure and had some issues and didn't really like it. I haven't tried AWS or GKE for k8s yet.
- webwanderings 8y ago
  Try IBM Cloud Private, it is free.
craig_asp 8y ago
"As it stands today I’ve personally used EKS, and AWS in general a lot. I’ve used GKE a bit but only with my own personal credits doing Kubernetes The Hard Way and spinning up a very quick GKE test cluster a while ago." "I’m being serious when I say this: if the company I’m working for decided to migrate to Azure I’d find a new job." "It needs to be fast and bug free so that I can build cool automation on top. Working on something like Azure, especially after having worked on AWS for years, would be extremely depressing." If the article is supposed to be an _unbiased_ comparison between cloud hosted Kubernetes providers, I'd say it's a bit of a fail. For some it would be completely different experience because they have experience with Microsoft technologies. And those people might as well quit if their company moves to AWS or a non-Azure platform.
- bk24 8y ago
  Well said, this article is the furthest thing from unbiased. OP obviously spent alot of years being paid to maintain infrastructure running crappy software and is still feeling the scars. I'm no fan of MS myself but I do acknowledge that they've improved alot from the duct-tape and glued NT days of old.
  - geezerjay 8y ago
    ...yet you've addressed none of the points made in the article, and only resorted ro personal attacks.
- sjellis 8y ago
  > If the article is supposed to be an _unbiased_ comparison between cloud hosted Kubernetes providers, I'd say it's a bit of a fail. For some it would be completely different experience because they have experience with Microsoft technologies. And those people might as well quit if their company moves to AWS or a non-Azure platform. Azure services can be flaky, and slow, and the "blades" UI is a bold design choice that doesn't really work in practice. No amount of experience with Microsoft technologies can help you with those problems, unfortunately.
stevenacreman 8y ago
Author here. I can say that this has pretty much gone as I'd expected. Microsoft have 80,000+ developers. Their partner ecosystem is absolutely massive. I've watched them hire hundreds of developer advocates that talk at events. It's therefore quite hard to write anything online that's critical without at least half of the comments coming from a biased source. It is interesting to see the gap between views in the comments here. Edit: I watch this comment go up and down a lot as the vote battle between those with an agenda and those without click against each other. Weirdly, I did criticise AWS a bit for their EKS offering in the blog but I've not had anywhere near the toxicity from people about that.
- mwj 8y ago
  The cult of Microsoft is the new cult of Mac...
- detaro 8y ago
  It's extremely low to automatically attribute disagreement to other commenters being dishonest. (even if we ignore the fact that AWS likely has a larger network of people highly invested into it...)
  - stevenacreman 8y ago
    Any more of this nonsense and I'll get some Azure credits and start doing Youtube videos of the slowness and the crashes :)
    - romanovcode 8y ago
      Please do.
      - stevenacreman 8y ago
        Done. https://kubedex.com/project-dolos-testing-kubernetes-on-google-and-azure/ https://kubedex.com/project-dolos-testing-kubernetes-on-goog...
- 8y ago

228 points by stevenacreman 8y ago

Good post, Steven. Keep going..

> Networking is the other reason. Google is miles ahead of everyone here. Similar story with HA and scaling. Does anyone know what the author is referring to with this claim? I don't see anything in the sheet to back this up. At least from a high level, all three options support network policies via CNI, and GKE and EKS use the same one, Calico.

stevenacreman 8y ago
Hi, there's a comment on the cross region networking for GKE. "Each cluster receiving an IP range for nodes and another for the containers inside, which are directly routable across your private network, other clusters, and regions." AWS and Azure don't have a flat global network. You have to setup VPN's and complicated overlay networks.
- shshhdhs 8y ago
  > AWS and Azure don't have a flat global network. You have to setup VPN's and complicated overlay networks. AWS has cross-region VPC peering, which is simple to setup and makes it feel “flat”. There’s no need for VPNs or overlay networks.
  - nodesocket 8y ago
    Until you have to reference security groups across the peering connection. You can't reference a different region's security group id as a source, thus your forced to just whitelist the entire subnet/cidr. Also, while setting up peering is a 10-20 minute task there are lot of constraints, such as the VPC's have to have different CIDR ranges. Oops, if you go with the AWS default, probably all regions using 172.31.0.0/20. Google got networking right with a flat global any region setup. If you need segmentation then just create a new GCP project. If it turns out projects need to communicate after all, GCP has VPC network peering.
    - p0rkbelly 8y ago
      You can use PrivateLink for this instead of peering. That way you just hit a local IP address/ENI in your VPC. CIDR Ranges don't matter with that. (not sure about the x-region story) Two different design philosophies, there are been a global GCP networking outages due to having the single flat network. A few this year. All providers have had their issues, but, global blast radiuses concern me.
- deleted 8y ago
  [deleted]
- halbritt 8y ago
  Couple things: Cross-region networking. To this day, I'm flabbergasted this isn't something that AWS has. All the other networking bits seem more sensible to me as well. Another, is 2Gbps per core up to 8 cores which is way more throughput than I've seen on any AWS or Azure instance. On workloads where I care about Network IO:CPU ratio, I'm using 8 core nodes and seeing 16gbps throughput between them consistently.
  - ti_ranger 8y ago
    > Cross-region networking. To this day, I'm flabbergasted this isn't something that AWS has. All the other networking bits seem more sensible to me as well. AWS tries to avoid letting customers create multi-region or global failure modes, like: * https://status.cloud.google.com/incident/compute/18005 https://status.cloud.google.com/incident/compute/18005 22-hour incident with GLOBAL impact resulting in many VMs getting duplicate IPs (this no network connectivity) on 2018-06-15 * https://status.cloud.google.com/incident/cloud-networking/18001 https://status.cloud.google.com/incident/cloud-networking/18... 3-hour (seemingly GLOBAL) impact to load-balancers (no updates or creations) on 2018-01-03 * https://status.cloud.google.com/incident/compute/16007 https://status.cloud.google.com/incident/compute/16007 - 18-minute GLOBAL outage on 2016-04-11 * https://status.cloud.google.com/incident/compute/15055 https://status.cloud.google.com/incident/compute/15055 - 5-minute GLOBAL outage on 2015-08-04 * https://groups.google.com/forum/#!topic/gce-operations/fynnXnb2OFs https://groups.google.com/forum/#!topic/gce-operations/fynnX... 43-minute multi-region packet loss due to multi-region configuration deployment on 2015-03-07 * https://groups.google.com/forum/#!topic/gce-operations/1uw-qEqjBdo https://groups.google.com/forum/#!topic/gce-operations/1uw-q... unscheduled reboot of 28% of instances in multiple regions in ~1.5h on 2014-09-17 There have been recent changes in many AWS services (e.g. DynamoDB, S3, Aurora) to allow cross-region replication without introducing any multi-region dependency, precisely to make it easier to implement multi-region infrastrucure to tolerate a single-region failure (however rare that is on AWS compared to global outages on GCP). > Another, is 2Gbps per core up to 8 cores which is way more throughput than I've seen on any AWS or Azure instance. Which instance types did use on AWS? Many recent instance types (r4, i3, c5, r5, z1) support up to 10Gbps with once core (2 vCPU instances). However, you may need to use placement groups ( https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placemen... ) in large regions in order to get the full throughput on low TCP connection numbers. The only reason I can think of that would explain why GCP doesn't have this problem is they don't have any regions anywhere near as large as large AWS regions ... Edit: white-space only formatting changes
regnerba 8y ago
I am not the author and haven't had time to fully review the spreadsheet they provided. After an initial look I think he is referring not to the kubernetes specific networking support but the GCP networking. From the spreadsheet alone GKE has support for cross region load balancers. The author also points out cross region networking. I don't know the exact advantage the author sees with cross region load balancers since you can only deploy a GKE cluster to a single given region. The GCP LB's are really nice though in that traffic enters Google's network at the closest location to the user and then travel within Google's network, which is wicked fast. I have not played around with federated clusters in multiple zones so don't know how that would play out either. They may also just be referring to just general networking from the providers. With the first two points they at least make mention in the spreadsheet of it though. tl;dr: not sure what the author meant but here here is some food for thought on it.
dilyevsky 8y ago
It supports addressable pods natively on vpc. Others (i think) do not. This simplifies ingess options a great deal. Also in general GCP networking is better than aws ime (ever tried to have multiple projects in aws share a network?). Don’t know anything about Azure but I suspect msft still sucks at networking things so it’s not great.
- marcinzm 8y ago
  EKS pods use the VPC networking layer so I think they meet the criteria. They use secondary IPs attached to instances rather than virtual IPs. I’ve been able to route to them from non-EKS instances in the same VPC.
  - dilyevsky 8y ago
    Ah I see aws has their own cni that does that via eni. I never realized that it allowed for native pod addressing (and it was never brought up during their pitches). Might be worse performance for big clusters because it’s still using calico (iptables)
    - gtaylor 8y ago
      I think Calico is only used for network policy on the AWS VPC CNI plugin.
    - anything4dogs 8y ago
      > Might be worse performance for big clusters because it’s still using calico (iptables) I'm curious why you think that Calico's use of iptables will have performance impacts for large clusters, and what type of performance impacts you expect (bandwidth / latency / cpu / something else?). From my experience, Calico performs rather well in large clusters (e.g. 2k nodes, appx 100 pods / node, several hundred network policies).

Arnavion 8y ago

>Show them how some things on Azure need to be done in a clunky web UI, other things need Powershell and other random stuff uses the CLI. This, at least, is wrong. Everything related to Azure's hosted Kubernetes can be done via the CLI without ever touching the web portal or PS.

GordonS 8y ago

> Azure is something I’ve avoided since using it for a few months last year. I was working as a Microsoft partner so it was unavoidable back then. Parts of it are alright but the user experience coming from an Amazon background is worlds apart. Wat? IMO, the Azure portal is amazing to work with, especially compared to the old-fangled, inconsistent UI that AWS provides. > Show them how some things on Azure need to be done in a clunky web UI Years ago, this was true, but it hasn't been for a long time - the Azure web UI of today is fast, consistent and looks great. > some things on Azure need to be done in a clunky web UI, other things need Powershell and other random stuff uses the CLI I've been using Azure for years, and I'm not aware of anything that can only be done in the UI. I also don't believe there is anything that only works with Powershell, or only works with the CLI. > how that effects the design of DevOps pipelines and automation in general. Yes, you can make it work, but why make life hard for yourself Eh? Azure DevOps pipelines are great to work with, and there's a huge library of tasks available. After reading this, it doesn't sound like the author has actually worked with Azure recently, so I'm really not sure why he bothered including it in this article.

sofaofthedamned 8y ago
Agree completely. AWS portal is bizarre, it's like each portion is written by a different company. In ECR for example, clicking into a repo is fine, but then going back doesn't do anything. Why? Even simple favicons don't look like they're from the same company. When you've got 40 tabs open you want an idea that that particular tab is from AWS or not. Azure is nicer, but it's not reliable (see my other post on this thread). It's almost like Azure have done too much lazy-loading which persists through login sessions, you can't rely on a refresh to give you the truth - for all the problems with AWS, you can. And yes I know this isn't CLI stuff, but it's important. Even on fully automated / scripted deployments it's sometimes nice to go into the UI and see what's going on. If it doesn't represent reality, there is a problem. FWIW GC doesn't have any of these problems, but it is slow.
- GordonS 8y ago
  You only mention Azure DevOps in your other comment - have you used Azure in a wider sense? As another datapoint of one, I've been working with Azure for years, and have never found the UI to be unreliable.

sofaofthedamned 8y ago

Interesting post. I'm a Linux guy but i'm in a new job so having to ramp up on Azure DevOps. Looked good at first, I actually praised Microsoft but: 1. Creating a project timed out after 5 minutes or so. This was in the gui, but had no adblocker or anything similar. Refreshed page = no project. 2. Went to a different machine, logged in, no project still. 3. Went to original machine - refreshed - no project. Logged out and back in again - project was there. Azure DevOps was awesome at first but I can not trust something like this when I work with it all day.

toomuchtodo 8y ago
I refuse to work with Azure because of issues like this.
- sofaofthedamned 8y ago
  I don't have a choice unfortunately. BTW never had a problem bringing up an instance quickly on GC. AWS sometimes just sat there for a while, depending on the instance type and region. Azure just stops because it feels like it. I'm sure GC isn't any better than the other two, it's just less utilised. The UI is my biggest issue. Yes, we don't depend on it because we all script our stuff. But after spending an hour fapping at an SSH terminal it's nice to look at the UI and see stuff happen. Azure is the worst at this, with AWS a close second. That reminded me of my favourite bug with AWS UI / Container Registry - you can order the list of container images any way you like, but it'll only order those it's loaded. So i'm looking for a new image called 'zzzzz' but it'll never appear, I need to search for it. Did anybody involved ever use this?
  - toomuchtodo 8y ago
    > I don't have a choice unfortunately. I won't say you always have a choice, but I'm happy to hook you up with my network if you want to passively seek out a new role. Email in my profile. Sidenote: I have seen the ECR UX shenanigans you mention.
switch007 8y ago

conradk 8y ago

For "Maximum pods per node", it shows GKE at 100, AKS at 110 but still puts GKE in green. How come?

cobookman 8y ago
Does anyone even have more than 20 pods / node? I've not personally seen anything near the 100 pods / node limit in the real world.
- sofaofthedamned 8y ago
  I didn't have 100, but I did have around 60 of my own plus all the associated K8s pods (about 10 of them). It's a bad idea to have that many if they're all Apache or whatever. But when they're simple Golang containers that handle APIs it's ok.
- regnerba 8y ago
  We don't have 100, but definitely more than 20. A quick look at a random node from our cluster: - 3x ingress pods (1 for an internal load balancer, 2 for an external load balancer) - 1x cert-manager - 2x production version of internal app01 - 2x staging version of internal app02 - 1x fluentd - 1x elastalert - 1x kubewatch - 1x prometheus node exporter - 1x redis for sentry - 1x sentry web - 2x sentry worker - 4x sourcegraph language servers (there are 9 language servers running across the 4 nodes, this node seems to especially like them) - 1x thelounge - 1x staging version of app02 - 2x production version of app02 - 1x kube proxy - 1x kubedns - 1x couchdb (small footprint just for testing some things) the node is just over half provisioned and we are due to add another node to the cluster soon.
  - deepsun 8y ago
    What does prometheus node exporter collects from within a pod? I was under impression that to collect node stats you absolutely need to be "outside" any container, i.e. using prometheus-k8s integration so that it pulls nodes stats from k8s api, not nodes themselves).

inscrutable 8y ago

Fully agree with his comments as someone who's used GKE, AKS, EKS and Hetzner for kubernetes clusters. GCP's UX is so nice... e.g. compare the equivalent as command line option in the UI vs the automation script that Azure gives you.

GordonS 8y ago
Azure actually gives you several ways to do things, of which automation scripts (ARM templates) are just one: 1. Web UI 2. Powershell 3. CLI 4. REST API I know what you mean about ARM templates though - the auto-generated ones in particular are extremely verbose!
krn 8y ago
> [...] as someone who's used GKE, AKS, EKS and Hetzner for kubernetes clusters. How does Hetzner compare here? In terms of both, general experience and costs.
- cardine 8y ago
  >How does Hetzner compare here? In terms of both, general experience and costs. We use Hetzner for this purpose. It's hard to compare it with the other providers since with Hetzner you are renting unmanaged servers, not turning up cloud instances. But it is absolutely the best value you can get. Our bill would be at least 10x higher if we were using Google/Amazon/Microsoft. Considering we spent a lot on servers as is, that makes it well worth it for us. It might not be if your product is very computationally light.
  - segmondy 8y ago
    I don't know your location, but if you are in the US and running your own k8s on baremetals, I would reckon you should have at least 1 full time person on staff to handle all of it. Let's say that person costs only $50k/yr. Will your workload on Google cost more than $50k? Google pretty much really does all of the management for GKE. Just deploy your app and run.

kubenaught 8y ago

My company is in the process of migrating off of a managed kubernetes provider. Sure, it's nice to have someone else manage the operations of the master. At the same time, a single customer is entirely insignificant to them. We've experienced multiple outages from forced upgrades. We usually can find out the reason through github, but it may not be a priority for them to provide the fix. If they do, it could take days or weeks for it to become available. Much revenue has been lost because they can't do something at the speed which we could do it.

nbevans 8y ago

His hatred for Azure ruins the article. There seems to be a massive correlation at the moment between Kubernetes and hyped up magpie egomaniac developers.

FlorianRappl 8y ago
I thought 100% the same. While it is known that GKE yields the best managed Kubernetes experience I thought the whole Azure bashing was not only pointless, but also flawed with many inaccurate and even false statements. So much to "honest".
- halbritt 8y ago
  It correlates with my experience pretty closely. Granted, I haven't really used Azure in six months.
bk24 8y ago
The Azure bashing had very little substance to it, just alot of previous MS experience-based dogma.

andyfleming 8y ago

I'm curious how the digital ocean offering compares. I think it's in limited availability now.

bsaul 8y ago

A little of topic because it’s not a comparison : Recently had my first experience using gke ( and kubernetes in general), and although i managed to get something working in the end i would say it’s still pretty rough... Documentation is a mess : the general layout for google cloud is really a pain to read and navigate, but in addition to that you often have to jump between kubernetes doc and google doc, with some information being on both. Don’t do that please : either make it obvious that people need to get familiar with certain chapters of kube doc, or provide all the info ( me preference would actually go to the first option..) It’s quite hard to guess what you can do on the gke web interface and what you can’t. You can feel it’s meant for people that really know kubernetes, and not people who discover both gke and kubernetes at the same time. And for the life of me i couldn’t get the load balancer manage https. I’ve read this was possible, but never saw the actual page explaining how. I ended up using cloudfront but lost the ability to see end user ip in my logs in the process. ( also logging is a real beast to tame on its own, with no obvious way to know what is available by default, what is a paid option, what should be configured on the stackdriver website, and what should be coded )

alpb 8y ago
TLS with GKE with BYO cert isn't very clear I agree. It's basically in Kubernetes Ingress docs: https://kubernetes.io/docs/concepts/services-networking/ingress/#tls https://kubernetes.io/docs/concepts/services-networking/ingr... For GKE specific features, multiple-certs per Ingress explained here: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-multi-ssl https://cloud.google.com/kubernetes-engine/docs/how-to/ingre..., and LetsEncrypt here: https://github.com/ahmetb/gke-letsencrypt https://github.com/ahmetb/gke-letsencrypt If you use GKE ingress, you can access end-user IP in X-Forwarded-For header.
- bsaul 8y ago
  but then does that mean you’ll need to manually log incoming ip from the header ? or is there a way to see them in the default logs of either the cluster or the load balancer ?

paxy 8y ago

The service that launched on Azure in 2017 was ACS (Azure Container Service), which is very different from what AKS is today.

capkutay 8y ago

Slightly unrelated, but what would be the best way to migrate a kubernetes application from one cloud to the other?

halbritt 8y ago
If you write your application correctly, it should be portable between kubernetes clusters. Figuring out how to manage kubernetes on different providers can be a little tricky.
scirocco 8y ago
OpenShift is a Kubernetes-based container platform that can run on AWS, Azure, Google...

dfischer 8y ago

I tried Azure lately – it felt like I booted into Windows 98. I couldn't stand it. A lot of the CLI UX was buggy for me too. AWS works. A lot of the UI is dated but the new designs are nice. GCP is the best. Google has done a great job on Dev UX both CLI and Web Console.

amf12 8y ago
> I tried Azure lately – it felt like I booted into Windows 98. I couldn't stand it. Care to elaborate?
- dfischer 8y ago
  The sidebar immediately took me back aesthetically. UX wise I felt it was terrible drilling down into resources I was looking for. Of course UX is also based on my bias (AWS and GCP). Nonetheless I was surprised at the presentational bar for the azure UI. It felt very dated. In the end I got a hang of it but again, it still felt like using an old pc. CLI wise I had a lot of errors spinning up resources. My support experience getting it figured out was sucky. I had to abandon azure as a decision quickly with that experience. Too risky.
  - bk24 8y ago
    a man of many subjective complaints
    - dfischer 8y ago
      a person lacking constructive comments

drewmassey 8y ago

After several weeks of swimming upstream on a greenfield EKS project I switched gears to GKE. The paradigm just felt way better. For large enterprises that need IAM integration on EKS I suppose that wouldn’t have been an option but GKE just feels way more paradigmatic, for obvious reasons. Did I mention we are hiring devops engineers? If you are a kubernetes guru email me :-)

api 8y ago

We are preparing to move to Google. Our app is CPU intensive and a killer feature was the ability to spec nodes with a lot of cores and little RAM. We save a lot of money by not paying for RAM we dont need. Google also had a location in LA and we are in LA. 5ms to our cloud is nice.

mverwijs 8y ago

Anyone here want to share their experiences running k8s on IBM Bluemix (formerly Softlayer)?

po 8y ago
I use and like IBM Cloud (Softlayer) for k8s stuff... I find they are making improvements to it (including docs) on a regular basis and I can often find and talk directly to their devs in a slack channel. Some parts are a bit clunky and I feel like I can see their legacy stuff poking through the abstractions but overall I find it to be good. I tried Azure and had some issues and didn't really like it. I haven't tried AWS or GKE for k8s yet.
webwanderings 8y ago
Try IBM Cloud Private, it is free.

craig_asp 8y ago

"As it stands today I’ve personally used EKS, and AWS in general a lot. I’ve used GKE a bit but only with my own personal credits doing Kubernetes The Hard Way and spinning up a very quick GKE test cluster a while ago." "I’m being serious when I say this: if the company I’m working for decided to migrate to Azure I’d find a new job." "It needs to be fast and bug free so that I can build cool automation on top. Working on something like Azure, especially after having worked on AWS for years, would be extremely depressing." If the article is supposed to be an _unbiased_ comparison between cloud hosted Kubernetes providers, I'd say it's a bit of a fail. For some it would be completely different experience because they have experience with Microsoft technologies. And those people might as well quit if their company moves to AWS or a non-Azure platform.

bk24 8y ago
Well said, this article is the furthest thing from unbiased. OP obviously spent alot of years being paid to maintain infrastructure running crappy software and is still feeling the scars. I'm no fan of MS myself but I do acknowledge that they've improved alot from the duct-tape and glued NT days of old.
- geezerjay 8y ago
  ...yet you've addressed none of the points made in the article, and only resorted ro personal attacks.
sjellis 8y ago
> If the article is supposed to be an _unbiased_ comparison between cloud hosted Kubernetes providers, I'd say it's a bit of a fail. For some it would be completely different experience because they have experience with Microsoft technologies. And those people might as well quit if their company moves to AWS or a non-Azure platform. Azure services can be flaky, and slow, and the "blades" UI is a bold design choice that doesn't really work in practice. No amount of experience with Microsoft technologies can help you with those problems, unfortunately.

stevenacreman 8y ago

Author here. I can say that this has pretty much gone as I'd expected. Microsoft have 80,000+ developers. Their partner ecosystem is absolutely massive. I've watched them hire hundreds of developer advocates that talk at events. It's therefore quite hard to write anything online that's critical without at least half of the comments coming from a biased source. It is interesting to see the gap between views in the comments here. Edit: I watch this comment go up and down a lot as the vote battle between those with an agenda and those without click against each other. Weirdly, I did criticise AWS a bit for their EKS offering in the blog but I've not had anywhere near the toxicity from people about that.

mwj 8y ago
The cult of Microsoft is the new cult of Mac...
detaro 8y ago
It's extremely low to automatically attribute disagreement to other commenters being dishonest. (even if we ignore the fact that AWS likely has a larger network of people highly invested into it...)
- stevenacreman 8y ago
  Any more of this nonsense and I'll get some Azure credits and start doing Youtube videos of the slowness and the crashes :)
  - romanovcode 8y ago
    Please do.
    - stevenacreman 8y ago
      Done. https://kubedex.com/project-dolos-testing-kubernetes-on-google-and-azure/ https://kubedex.com/project-dolos-testing-kubernetes-on-goog...
8y ago