14 ms·

Kubernetes is one of those rare cases where the industry is rallying around a technologically sound platform that emerged from the trenches, and not something t

Kubernetes is one of those rare cases where the industry is rallying around a technologically sound platform that emerged from the trenches, and not something that was designed to create jobs for consultants. It's great. Right now, there's a bunch of fairly traditional players such as Red Hat, but none of them are in a position to mess it up. What I'm waiting for, though, is for a big player to do a modern, clean "Kubernetes first" cloud offering. We're currently on Google Kubernetes Engine, and I'm disappointed in the lack of integration throughout Cloud Platform. GCP is great in many areas, but the Kubernetes offering is clearly an afterthought. As an example, if you create an ingress, this creates a bunch of load balancer objects (forwarding rules, URL maps, backends). But these are all given obscure names (k8s-um-production-zippediboop-f8da5e6c92f38300), and none of the Kubernetes labels transfer over. Same with disks; GCE disks don't even have labels, so they put a bunch of JSON in the "description", which of course cannot be filtered or queried. Similar things happen with VMs and other things; the "mapping" of Kubernetes to GCP concepts is basically either lossy or non-existent. Many other similar weaknesses exist: Cloud SQL (if you want to use Cloud SQL, you have to manually run the Cloud SQL Proxy as a sidecar), GCP service accounts (different from Kubernetes service accounts!), etc. GKE is solid, but everything seems like it's been put together without any over-arching design; GKE is just another thing buried in a somewhat ramshackle jungle of different services. There's an opportunity for a vendor to come in and offer a platform where you start with Kubernetes. In particular, this means that the platform should be manageable from Kubernetes, through the use of CRDs and pluggable controllers. For example, I should be able to define a persistent disk using a Kubernetes manifest, instead of having to go through some web UI or CLI ("gcloud compute disk create" or whatever). That said, it's hard to compete with GCP at this point. Whoever wants to compete in this space have to compete not just with GKE, but with the technical awesomeness of GCP.

navaati 9y ago
Well RedHat's OpenShift.com is kinda that, have a look at it !
- lobster_johnson 9y ago
  Maybe. I'm skeptical of Red Hat. The openshift.com web site, which is definitely not geared toward engineers, doesn't give me any confidence that this is what I'm after. Not that Google is better here, but I already know what GCP provides; something that wants to compete with GCP/GKE really needs to explain why they're a viable competitor. (And to people who design these things: If you need to have a "Products" dropdown filled with unexplained product names, you're doing something wrong.)
  - ghaff 9y ago
    A lot of people come to websites looking for specific products. When you don’t offer an obvious path to what they’re looking for, they get frustrated. I know I do.
  - organsnyder 9y ago
    Red Hat's marketing is definitely a mess (I say this as an enterprise Red Hat customer in the midst of an OpenShift roll-out), but their engineering is top-notch. Many pieces of k8s originated in OpenShift, and Red Hat is heavily involved in k8s development.
  - jacques_chester 9y ago
    I work for Pivotal, we directly compete with Red Hat. Look: they're Red Hat, I mean c'mon now. If you don't think they have great engineers then I'd hate to hear what you think of the rest of us. These websites are not usually intended for you or I. They're intended to get the interest of the cheque-signers. Enterprise software is a very particular world and it has its particular cadences and norms and approaches. Red Hat understands it well. So does Pivotal. That's why we see so much of each other at a number of places where folks are looking to adopt modern platforms.
    - lobster_johnson 9y ago
      Hence my skepticism. Someone who doesn't know how to engage with engineers is someone I don't have any interest in engaging with, either.
martin-adams 9y ago
Digital Ocean are working on their Kubernetes product offering. https://www.digitalocean.com/products/kubernetes/ https://www.digitalocean.com/products/kubernetes/
- lobster_johnson 9y ago
  We were on DigitalOcean before GCP. They're okay. They're cheap, but can't really compare to GCP in terms of performance, stability, networking, object storage, etc. Their Kubernetes offering is also a GKE competitor, not a "Kubernetes first" cloud.
  - StudentStuff 9y ago
    At least DO supports IPv6, for our application the 40ms latency penalty we encountered for half our customers when on GCP from not supporting IPv6 was not worth it. On that note, HE.net really peers with everyone over IPv6, even ISPs that won't peer with them on IPv4, its quite impressive how much better the routing can be on certain ISPs due to this!
    - jpdb 9y ago
      Genuinely curious: how does the lack of IPv6 cause 40ms of latency?
      - pathseeker 9y ago
        Tunneling to a v6 provider.
        StudentStuff 9y ago
        More like the end user gets routed through bespoke connections hundreds of miles away due to ISPs like Centurylink not openly peering on IPv4 at Internet Exchanges, whereas on IPv6 they seem to have local routes to Hurricane Electric and many others on the West Coast, thus no need for traffic to go back east!
eicnix 9y ago
You might want take a look that service brokers which provide a way to manage Cloud Provider resources like Google Cloud SQL as CRDs from k8s/GKE.
- lobster_johnson 9y ago
  Thanks, I know about service brokers. My point was that there's an opportunity for a vendor to do this better out of the box.
clutchdude 9y ago
Well said. Operationalizing K8s can be fraught with vendor specific parts. It doesn't help documentation wise when many things gloss over a concept that is offloaded into GCE or GKE.
praseodym 9y ago
Actually it is already possible to create persistent disks through Kubernetes by creating a persistent volume claim (pvc) which triggers an actual persistent volume to be created. Cloud provider controllers take care of this, just like they create load balancers for services. Also there’s already work being done on supporting cluster management from Kubernetes itself: https://github.com/kubernetes-sigs/cluster-api https://github.com/kubernetes-sigs/cluster-api
- lobster_johnson 9y ago
  That's not what I meant, though. A Kubernetes PV is an abstraction that causes a controller allocate a GCE disk (which of course is another abstraction), but if you create a GCE disk through the CLI/UI/API, then you don't get a PV.
  - colemickens 9y ago
    I think the point is that I don't have to touch the cloud provider's control plane in 99% of cases. Which brings the portability that first attracted me to Kubernetes.
    - yebyen 9y ago
      The portability is not as attractive as a fully integrated solution though, for types of people that are not already attracted to Kubernetes. Where I work, we use AWS and say that "we love vendor lock-in." The more integration with the rest of the vendor's cloud platform, the better. I think it's a great example to say it is idiosyncratic that I can create a PD by creating a PV, but I can't reciprocally enumerate PDs by listing PVs. Another: Kubernetes secrets are hidden inside K8S and they are not generally accessible through any vendor-provided secrets managers outside of K8S. I love Ingress, because it allows me to create new hostname-driven application endpoints inside of my cluster without any permissions on the AWS side. The cluster has a load balancer with a star certificate attached to it, the DNS has a star record similarly, and I can create new Ingress records without reaching outside of the cluster. My InfoSec department hates stuff like that! Now if I use Kubernetes that means they have to learn Kubernetes too, and maybe even build out additional tooling around it, because their AWS-based automated service inventories are not reliable anymore. I don't really have those star records (I'd really like to have them, it would be really convenient for me as a developer, so I don't have to ask for permission and go through layers of approval when I need to do stupid things, like my job...) The unfortunate reality that I've encountered is that those teams are more likely to see the encapsulation features like this Ingress configuration as a pure security risk with no benefit to them, plain and simple.
rubiquity 9y ago
> and not something that wasn't designed to create jobs for consultants. I wouldn't be so sure about that. Consulting companies are drawn to operational complexity, of which Kubernetes has a lot of, like bears are to honey. https://kubernetes.io/partners/ https://kubernetes.io/partners/ https://www.google.com/search?q=kubernetes+consulting&ie=utf-8&oe=utf-8&client=firefox-b-ab https://www.google.com/search?q=kubernetes+consulting&ie=utf...
- atombender 9y ago
  Kubernetes came out of the engineering team at Google. It was created by engineers for engineers. The fact that it's being exploited by enterprise consulting types doesn't change that premise.
  - daxfohl 9y ago
    But there's this from a fairly influential former googler: https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48c9a600e405 https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48... It's a quick read, but to summarise: it's almost as complex as google's internal borg system but the benefit isn't even close (partially because nobody else has google problems). I can't down K8s personally, as I've never used it myself. But I wonder if there's the possibility of a system that's more 80/20 of google's borg rather than the seeming 20/80 coming from k8s. And I wonder if Grab will release it next year.
    - vorpalhex 9y ago
      Kubernetes does suffer from complexity right now, but that also gives it it's flexibility. I think turnkey kubernetes solutions like Rancher will dominate for a lot of use cases, especially for individual devs and small teams that can't have a dedicated DevOps resource to manage kubernetes.
      - sleepybrett 9y ago
        As someone who has been running kubernetes clusters for the last 1.5-2 years. It's quickly becoming boring (in a good way). I'm at a fairly large enterprise and we're running five clusters (two gke managed, which .. is limiting, three with our own bootstrap). I'd say we spend about 10%-20% of our time actively managing clusters (upgrades, troubleshooting.. some of that time is about maintaining some value adds like automatic log shipping.. that kind of thing), most of our work is writing value-adds, helping customers onboard and consulting with teams on how to best deploy and work with kubernetes (many teams at my company aren't really familiar with containerization). Granted we run our own bootstrap (when we started none of the bootstraps running around we're ready, we started with a terraform/make implementation of hightower's hardway and we've just kept adding. We're thinking about revisiting the space again.
daxfohl 9y ago
Many of the k8s founders have left google. Google doesn't use it internally. I'd actually start looking away from gcp at this point for k8s, given google's history of dropping stuff that they're bored with.
- lobster_johnson 9y ago
  Many of the founders (e.g. Brendan Burns) are still working on Kubernetes, so I don't see how that's relevant. Google is heavily invested in Kubernetes, and so is a bunch of other companies. According to googlers here on HN, Google does use Kubernetes internally via GCP/GKE. But clearly those apps are in the minority, given Google's huge investment in Borg.
  - daxfohl 9y ago
    I thought Brendan Burns went to Microsoft and was working on something unrelated now?
    - lobster_johnson 9y ago
      Last I checked, he's working on Azure's Kubernetes solution.
- jcastro 9y ago
  Google has lots of people still working on k8s. I wouldn't equate Google's consumer facing products with their cloud/enterprise products.
- shaklee3 9y ago
  Brendan Burns and Joe Beda both work at other companies on kubernetes. Joe founded a startup to work on it. Why would they do that if they hated it?
- manigandham 9y ago
  Google does not just drop enterprise products and services. They definitely have a problem with consumer (messaging) apps, but that's completely different.
k8sftl 9y ago
Kube is too complex. To derive any kind of business value from it you need highly skilled people. The winner of the container orchestration battle will be the one that simplifies, not complicates. To me this is just another hype train, like TensorFlow/Caffe before it, and like Hadoop/Spark before that. These tools all have their uses in specific, non-common cases, but for your average business they are a net loss in time and money.
- robbyt 9y ago
  Too complex compared to what? All other commonly used web infrastructure is complex in various ways. In my experience building and running Kubernetes clusters for Fortune 500 companies is that actually it's much simpler, it's just different from traditional infrastructure. Different != Complex.
  - k8sftl 9y ago
    I guess we worked at different Fortune 500 companies. What I see are companies adding complexity to their web infrastructure where none needs to exist. A vanishingly small number of use cases require 100's of nodes, and the accompanying orchestration. Stack Overflow runs a top 25 website on 9 webservers, and most of those are just redundancy. Nothing Ansible/Bash or Powershell/MSDeploy can't solve in orders of magnitude less time. And debugging/diagnosing problems is much easier.
  - dsnuh 9y ago
    Were the Fortune 500 companies hosting k8s on-prem or on a cloud provider? From an operational standpoint, I can see where GP is coming from, especially if you are hosting on-prem, for non-greenfield projects. Trying to integrate a k8s cluster into legacy infrastructure is non-trivial, and requires knowledgeable operations teams to deploy. Troubleshooting k8s requires extensive domain knowledge, in addition to foundational understanding of fairly advanced networking and distributed systems concepts. In my experience, working with k8s also requires much more vigilant watching of the project space for (breaking) changes, updated docs, SIG news, and github issues. Even if Google or Amazon is managing the k8s management, troubleshooting your app using kubectl, etc. is a new skill to learn. I actually like a lot about using k8s, but I understand when people complain about the complexity.
kitd 9y ago
What I'm waiting for, though, is for a big player to do a modern, clean "Kubernetes first" cloud offering. IBM Cloud Private is a Kubernetes-based private cloud offering, though from your other points I don't think it will have everything you want. https://www.ibm.com/cloud/private https://www.ibm.com/cloud/private
adamm255 9y ago
Pivotal and VMware are working on this. From what I’ve seen (yet to play around too deep with this yet) looks promising. You’ve got it spot on though, just needs to be simple. https://pivotal.io/containers/deploying https://pivotal.io/containers/deploying
- hodgesrm 9y ago
  Check out also VMware's container service extension (CSE) on Github. CSE makes it easy to pop up clusters on vCloud Director, which is the VMware cloud stack used by a lot of service providers. Disclaimer: I work on CSE and use it for K8s dev/test. It's really handy. https://github.com/vmware/container-service-extension https://github.com/vmware/container-service-extension
- jacques_chester 9y ago
  I work for Pivotal. PKS is our product in the Kubernetes space, co-developed with VMWare and Google. The opensource project we based PKS on is Cloud Foundry Container Runtime, CFCR. We are basically bringing all the stuff we learned building Cloud Foundry to bear on Kubernetes. There's also an increasing amount of systems being shared. Right now Istio is seen as the long-term north-south and east-west routing solution for Cloud Foundry App Runtime (CFAR, what folks traditionally thought of as Cloud Foundry). We're looking at introducing Loggregator to the Kubernetes world as Oratos, we have factored buildpacks into standalone systems that can have a single image builder for any OCI-compliant platform, we led the formation of the Open Service Broker API effort that creates a common service catalogue approach for both CFAR and Kubernetes, we've contributed to runC and I lose track of all the rest. When I say "we" I should say this isn't just Pivotal. IBM, SAP and SUSE are deeply involved as Cloud Foundry community members.
hodgesrm 9y ago
Are you proposing that cloud vendors should do a better job of integrating Kubernetes with underlying infrastructure or are you saying there should be a Kubernetes-only cloud platform? If the former that's clearly desirable; you should be able for instance to set up networking between services without having to worry (much) about how it is implemented. If the latter, I am not so sure (or else I don't understand your point.) A lot of things don't/won't run well on Kubernetes--database management systems are a good example as well as any legacy application, which includes a lot of Windows OS, so you still have to address those. More subtly Kubernetes is not going to rewrite the whole world or implement distributed firewalling, network attached storage, VLANs, etc. So you are always going to have a non-K8s layer under there that is more or less foreign to the Kubernetes model. The best you can do is make the layering relatively efficient.
- bonesss 9y ago
  > A lot of things don't/won't run well on Kubernetes--database management systems are a good example as well as any legacy application, which includes a lot of Windows OS, so you still have to address those. Not saying I've tried it, but some people are having ok luck pinning RDBMS pods in Kubernetes to specific nodes. It goes a bit against k8s principles, but it makes operational sense. Kubernetes is baking in windows container support, for some legacy scenarios. More interestingly (and posted on HNs front page today), are solutions that allow independent VMs to be run as though they were k8s pods. This provides hybrid models ideal for legacy packaging and maintenance while moving onto new hardware. > Kubernetes is not going to rewrite the whole world or implement distributed firewalling, network attached storage, VLANs, etc. So you are always going to have a non-K8s layer under there that is more or less foreign to the Kubernetes model There will always be some natural impedance between hardware and software... I think this picture is ripe for improvements though, and we're already seeing the edges of it take shape :) Kubernetes will be moving towards smarter networking solutions to handle more and better use-cases with better performance (BPF), and is incorporating better network abstractions slowly but surely. From the under layers: software defined networking (SDN), and kubernetes providers for major virtualization platforms (VMware, for example), have commercial offerings that could readily support integrated or "hyperconverged" operations... Microsegmentation and microservices go hand-in-hand. Empowered by a platform connected with role based access controls top-to-bottom there's a lot of potential to harmonize those distributed firewalling/VLAN needs through the same declarative YAML. We're not there, yet. We are closer than ever though :)
outworlder 9y ago
> For example, I should be able to define a persistent disk using a Kubernetes manifest, instead of having to go through some web UI or CLI ("gcloud compute disk create" or whatever). What do you mean? You can do exactly that today, with persistent volume claims. K8s will create the storage volume for you and make it available for containers. This is "dynamic provisioning". https://kubernetes.io/docs/concepts/storage/persistent-volumes/ https://kubernetes.io/docs/concepts/storage/persistent-volum... What is missing is being able to create other resources. For instance, define a EC2 instance as a k8s object and have K8s provision it, for legacy or anything you haven't moved to K8s yet(databases, etc). I see no technical reason why it cannot be implemented today.
- lobster_johnson 9y ago
  See my reply here: https://news.ycombinator.com/item?id=17015475 https://news.ycombinator.com/item?id=17015475.
manigandham 9y ago
>> Whoever wants to compete in this space have to compete not just with GKE, but with the technical awesomeness of GCP. After years of using all the major clouds - Google Cloud has the best performance and primitives, but the worst support and operational/business features. If you want the fastest VMs/storage/networking, and are fine with non-standard APIs and beta SDKs, then it's a great platform and has super simple billing. AWS and Azure are much better suited for most companies and startups though in terms of actually getting your business running with standard tooling, large ecosystems, and a massive portfolio of services.
- humanfromearth 9y ago
  Funny, I had the exact opposite experience when it comes to APIs. Doesn't AWS have a PM for every product with different API each time? Tried to build something like a terraform using boto only to give up after dealing with inconsistent APIs for about 2 weeks.
- ghthor 9y ago
  It's probably AWS Fargate, part of the Elastic Container Service. But even then, it's still now a vendor lockin of the past, if you got your services deployable via containers you're 90% there to running your own Kubes cluster to launch them.
  - manigandham 9y ago
    I think you're replying to the wrong comment...
lugg 9y ago
> What I'm waiting for, though, is for a big player to do a modern, clean "Kubernetes first" cloud offering. As someone who is feeling burned over picking swarm over k8s, I'm not super into the idea of trusting a 3rd party to do k8s first. Anything anyone implements well will be ported to gke in due time, no?
- n00tz 9y ago
  The good news is Docker EE 2.0 brings Kubernetes to you. However, I'm not sure if it's operating split-brain or if they made it such that you can pick your control surface.
mlrtime 9y ago
The biggest problem with GCP cannot create a private kubernetes cluster, it must have public interfaces. Why is this a requirement?
- lobster_johnson 9y ago
  Private clusters are in open beta: https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters https://cloud.google.com/kubernetes-engine/docs/how-to/priva....

navaati 9y ago

Well RedHat's OpenShift.com is kinda that, have a look at it !

lobster_johnson 9y ago
Maybe. I'm skeptical of Red Hat. The openshift.com web site, which is definitely not geared toward engineers, doesn't give me any confidence that this is what I'm after. Not that Google is better here, but I already know what GCP provides; something that wants to compete with GCP/GKE really needs to explain why they're a viable competitor. (And to people who design these things: If you need to have a "Products" dropdown filled with unexplained product names, you're doing something wrong.)
- ghaff 9y ago
  A lot of people come to websites looking for specific products. When you don’t offer an obvious path to what they’re looking for, they get frustrated. I know I do.
- organsnyder 9y ago
  Red Hat's marketing is definitely a mess (I say this as an enterprise Red Hat customer in the midst of an OpenShift roll-out), but their engineering is top-notch. Many pieces of k8s originated in OpenShift, and Red Hat is heavily involved in k8s development.
- jacques_chester 9y ago
  I work for Pivotal, we directly compete with Red Hat. Look: they're Red Hat, I mean c'mon now. If you don't think they have great engineers then I'd hate to hear what you think of the rest of us. These websites are not usually intended for you or I. They're intended to get the interest of the cheque-signers. Enterprise software is a very particular world and it has its particular cadences and norms and approaches. Red Hat understands it well. So does Pivotal. That's why we see so much of each other at a number of places where folks are looking to adopt modern platforms.
  - lobster_johnson 9y ago
    Hence my skepticism. Someone who doesn't know how to engage with engineers is someone I don't have any interest in engaging with, either.

martin-adams 9y ago

Digital Ocean are working on their Kubernetes product offering. https://www.digitalocean.com/products/kubernetes/ https://www.digitalocean.com/products/kubernetes/

lobster_johnson 9y ago
We were on DigitalOcean before GCP. They're okay. They're cheap, but can't really compare to GCP in terms of performance, stability, networking, object storage, etc. Their Kubernetes offering is also a GKE competitor, not a "Kubernetes first" cloud.
- StudentStuff 9y ago
  At least DO supports IPv6, for our application the 40ms latency penalty we encountered for half our customers when on GCP from not supporting IPv6 was not worth it. On that note, HE.net really peers with everyone over IPv6, even ISPs that won't peer with them on IPv4, its quite impressive how much better the routing can be on certain ISPs due to this!
  - jpdb 9y ago
    Genuinely curious: how does the lack of IPv6 cause 40ms of latency?
    - pathseeker 9y ago
      Tunneling to a v6 provider.
      - StudentStuff 9y ago
        More like the end user gets routed through bespoke connections hundreds of miles away due to ISPs like Centurylink not openly peering on IPv4 at Internet Exchanges, whereas on IPv6 they seem to have local routes to Hurricane Electric and many others on the West Coast, thus no need for traffic to go back east!

eicnix 9y ago

You might want take a look that service brokers which provide a way to manage Cloud Provider resources like Google Cloud SQL as CRDs from k8s/GKE.

lobster_johnson 9y ago
Thanks, I know about service brokers. My point was that there's an opportunity for a vendor to do this better out of the box.

clutchdude 9y ago

Well said. Operationalizing K8s can be fraught with vendor specific parts. It doesn't help documentation wise when many things gloss over a concept that is offloaded into GCE or GKE.

praseodym 9y ago

Actually it is already possible to create persistent disks through Kubernetes by creating a persistent volume claim (pvc) which triggers an actual persistent volume to be created. Cloud provider controllers take care of this, just like they create load balancers for services. Also there’s already work being done on supporting cluster management from Kubernetes itself: https://github.com/kubernetes-sigs/cluster-api https://github.com/kubernetes-sigs/cluster-api

lobster_johnson 9y ago
That's not what I meant, though. A Kubernetes PV is an abstraction that causes a controller allocate a GCE disk (which of course is another abstraction), but if you create a GCE disk through the CLI/UI/API, then you don't get a PV.
- colemickens 9y ago
  I think the point is that I don't have to touch the cloud provider's control plane in 99% of cases. Which brings the portability that first attracted me to Kubernetes.
  - yebyen 9y ago
    The portability is not as attractive as a fully integrated solution though, for types of people that are not already attracted to Kubernetes. Where I work, we use AWS and say that "we love vendor lock-in." The more integration with the rest of the vendor's cloud platform, the better. I think it's a great example to say it is idiosyncratic that I can create a PD by creating a PV, but I can't reciprocally enumerate PDs by listing PVs. Another: Kubernetes secrets are hidden inside K8S and they are not generally accessible through any vendor-provided secrets managers outside of K8S. I love Ingress, because it allows me to create new hostname-driven application endpoints inside of my cluster without any permissions on the AWS side. The cluster has a load balancer with a star certificate attached to it, the DNS has a star record similarly, and I can create new Ingress records without reaching outside of the cluster. My InfoSec department hates stuff like that! Now if I use Kubernetes that means they have to learn Kubernetes too, and maybe even build out additional tooling around it, because their AWS-based automated service inventories are not reliable anymore. I don't really have those star records (I'd really like to have them, it would be really convenient for me as a developer, so I don't have to ask for permission and go through layers of approval when I need to do stupid things, like my job...) The unfortunate reality that I've encountered is that those teams are more likely to see the encapsulation features like this Ingress configuration as a pure security risk with no benefit to them, plain and simple.

rubiquity 9y ago

> and not something that wasn't designed to create jobs for consultants. I wouldn't be so sure about that. Consulting companies are drawn to operational complexity, of which Kubernetes has a lot of, like bears are to honey. https://kubernetes.io/partners/ https://kubernetes.io/partners/ https://www.google.com/search?q=kubernetes+consulting&ie=utf-8&oe=utf-8&client=firefox-b-ab https://www.google.com/search?q=kubernetes+consulting&ie=utf...

atombender 9y ago
Kubernetes came out of the engineering team at Google. It was created by engineers for engineers. The fact that it's being exploited by enterprise consulting types doesn't change that premise.
- daxfohl 9y ago
  But there's this from a fairly influential former googler: https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48c9a600e405 https://medium.com/@steve.yegge/honestly-i-cant-stand-k8s-48... It's a quick read, but to summarise: it's almost as complex as google's internal borg system but the benefit isn't even close (partially because nobody else has google problems). I can't down K8s personally, as I've never used it myself. But I wonder if there's the possibility of a system that's more 80/20 of google's borg rather than the seeming 20/80 coming from k8s. And I wonder if Grab will release it next year.
  - vorpalhex 9y ago
    Kubernetes does suffer from complexity right now, but that also gives it it's flexibility. I think turnkey kubernetes solutions like Rancher will dominate for a lot of use cases, especially for individual devs and small teams that can't have a dedicated DevOps resource to manage kubernetes.
    - sleepybrett 9y ago
      As someone who has been running kubernetes clusters for the last 1.5-2 years. It's quickly becoming boring (in a good way). I'm at a fairly large enterprise and we're running five clusters (two gke managed, which .. is limiting, three with our own bootstrap). I'd say we spend about 10%-20% of our time actively managing clusters (upgrades, troubleshooting.. some of that time is about maintaining some value adds like automatic log shipping.. that kind of thing), most of our work is writing value-adds, helping customers onboard and consulting with teams on how to best deploy and work with kubernetes (many teams at my company aren't really familiar with containerization). Granted we run our own bootstrap (when we started none of the bootstraps running around we're ready, we started with a terraform/make implementation of hightower's hardway and we've just kept adding. We're thinking about revisiting the space again.

daxfohl 9y ago

Many of the k8s founders have left google. Google doesn't use it internally. I'd actually start looking away from gcp at this point for k8s, given google's history of dropping stuff that they're bored with.

lobster_johnson 9y ago
Many of the founders (e.g. Brendan Burns) are still working on Kubernetes, so I don't see how that's relevant. Google is heavily invested in Kubernetes, and so is a bunch of other companies. According to googlers here on HN, Google does use Kubernetes internally via GCP/GKE. But clearly those apps are in the minority, given Google's huge investment in Borg.
- daxfohl 9y ago
  I thought Brendan Burns went to Microsoft and was working on something unrelated now?
  - lobster_johnson 9y ago
    Last I checked, he's working on Azure's Kubernetes solution.
jcastro 9y ago
Google has lots of people still working on k8s. I wouldn't equate Google's consumer facing products with their cloud/enterprise products.
shaklee3 9y ago
Brendan Burns and Joe Beda both work at other companies on kubernetes. Joe founded a startup to work on it. Why would they do that if they hated it?
manigandham 9y ago
Google does not just drop enterprise products and services. They definitely have a problem with consumer (messaging) apps, but that's completely different.

k8sftl 9y ago

Kube is too complex. To derive any kind of business value from it you need highly skilled people. The winner of the container orchestration battle will be the one that simplifies, not complicates. To me this is just another hype train, like TensorFlow/Caffe before it, and like Hadoop/Spark before that. These tools all have their uses in specific, non-common cases, but for your average business they are a net loss in time and money.

robbyt 9y ago
Too complex compared to what? All other commonly used web infrastructure is complex in various ways. In my experience building and running Kubernetes clusters for Fortune 500 companies is that actually it's much simpler, it's just different from traditional infrastructure. Different != Complex.
- k8sftl 9y ago
  I guess we worked at different Fortune 500 companies. What I see are companies adding complexity to their web infrastructure where none needs to exist. A vanishingly small number of use cases require 100's of nodes, and the accompanying orchestration. Stack Overflow runs a top 25 website on 9 webservers, and most of those are just redundancy. Nothing Ansible/Bash or Powershell/MSDeploy can't solve in orders of magnitude less time. And debugging/diagnosing problems is much easier.
- dsnuh 9y ago
  Were the Fortune 500 companies hosting k8s on-prem or on a cloud provider? From an operational standpoint, I can see where GP is coming from, especially if you are hosting on-prem, for non-greenfield projects. Trying to integrate a k8s cluster into legacy infrastructure is non-trivial, and requires knowledgeable operations teams to deploy. Troubleshooting k8s requires extensive domain knowledge, in addition to foundational understanding of fairly advanced networking and distributed systems concepts. In my experience, working with k8s also requires much more vigilant watching of the project space for (breaking) changes, updated docs, SIG news, and github issues. Even if Google or Amazon is managing the k8s management, troubleshooting your app using kubectl, etc. is a new skill to learn. I actually like a lot about using k8s, but I understand when people complain about the complexity.

kitd 9y ago

What I'm waiting for, though, is for a big player to do a modern, clean "Kubernetes first" cloud offering. IBM Cloud Private is a Kubernetes-based private cloud offering, though from your other points I don't think it will have everything you want. https://www.ibm.com/cloud/private https://www.ibm.com/cloud/private

adamm255 9y ago

Pivotal and VMware are working on this. From what I’ve seen (yet to play around too deep with this yet) looks promising. You’ve got it spot on though, just needs to be simple. https://pivotal.io/containers/deploying https://pivotal.io/containers/deploying

hodgesrm 9y ago
Check out also VMware's container service extension (CSE) on Github. CSE makes it easy to pop up clusters on vCloud Director, which is the VMware cloud stack used by a lot of service providers. Disclaimer: I work on CSE and use it for K8s dev/test. It's really handy. https://github.com/vmware/container-service-extension https://github.com/vmware/container-service-extension
jacques_chester 9y ago
I work for Pivotal. PKS is our product in the Kubernetes space, co-developed with VMWare and Google. The opensource project we based PKS on is Cloud Foundry Container Runtime, CFCR. We are basically bringing all the stuff we learned building Cloud Foundry to bear on Kubernetes. There's also an increasing amount of systems being shared. Right now Istio is seen as the long-term north-south and east-west routing solution for Cloud Foundry App Runtime (CFAR, what folks traditionally thought of as Cloud Foundry). We're looking at introducing Loggregator to the Kubernetes world as Oratos, we have factored buildpacks into standalone systems that can have a single image builder for any OCI-compliant platform, we led the formation of the Open Service Broker API effort that creates a common service catalogue approach for both CFAR and Kubernetes, we've contributed to runC and I lose track of all the rest. When I say "we" I should say this isn't just Pivotal. IBM, SAP and SUSE are deeply involved as Cloud Foundry community members.

hodgesrm 9y ago

Are you proposing that cloud vendors should do a better job of integrating Kubernetes with underlying infrastructure or are you saying there should be a Kubernetes-only cloud platform? If the former that's clearly desirable; you should be able for instance to set up networking between services without having to worry (much) about how it is implemented. If the latter, I am not so sure (or else I don't understand your point.) A lot of things don't/won't run well on Kubernetes--database management systems are a good example as well as any legacy application, which includes a lot of Windows OS, so you still have to address those. More subtly Kubernetes is not going to rewrite the whole world or implement distributed firewalling, network attached storage, VLANs, etc. So you are always going to have a non-K8s layer under there that is more or less foreign to the Kubernetes model. The best you can do is make the layering relatively efficient.

bonesss 9y ago
> A lot of things don't/won't run well on Kubernetes--database management systems are a good example as well as any legacy application, which includes a lot of Windows OS, so you still have to address those. Not saying I've tried it, but some people are having ok luck pinning RDBMS pods in Kubernetes to specific nodes. It goes a bit against k8s principles, but it makes operational sense. Kubernetes is baking in windows container support, for some legacy scenarios. More interestingly (and posted on HNs front page today), are solutions that allow independent VMs to be run as though they were k8s pods. This provides hybrid models ideal for legacy packaging and maintenance while moving onto new hardware. > Kubernetes is not going to rewrite the whole world or implement distributed firewalling, network attached storage, VLANs, etc. So you are always going to have a non-K8s layer under there that is more or less foreign to the Kubernetes model There will always be some natural impedance between hardware and software... I think this picture is ripe for improvements though, and we're already seeing the edges of it take shape :) Kubernetes will be moving towards smarter networking solutions to handle more and better use-cases with better performance (BPF), and is incorporating better network abstractions slowly but surely. From the under layers: software defined networking (SDN), and kubernetes providers for major virtualization platforms (VMware, for example), have commercial offerings that could readily support integrated or "hyperconverged" operations... Microsegmentation and microservices go hand-in-hand. Empowered by a platform connected with role based access controls top-to-bottom there's a lot of potential to harmonize those distributed firewalling/VLAN needs through the same declarative YAML. We're not there, yet. We are closer than ever though :)

outworlder 9y ago

> For example, I should be able to define a persistent disk using a Kubernetes manifest, instead of having to go through some web UI or CLI ("gcloud compute disk create" or whatever). What do you mean? You can do exactly that today, with persistent volume claims. K8s will create the storage volume for you and make it available for containers. This is "dynamic provisioning". https://kubernetes.io/docs/concepts/storage/persistent-volumes/ https://kubernetes.io/docs/concepts/storage/persistent-volum... What is missing is being able to create other resources. For instance, define a EC2 instance as a k8s object and have K8s provision it, for legacy or anything you haven't moved to K8s yet(databases, etc). I see no technical reason why it cannot be implemented today.

lobster_johnson 9y ago
See my reply here: https://news.ycombinator.com/item?id=17015475 https://news.ycombinator.com/item?id=17015475.

manigandham 9y ago

>> Whoever wants to compete in this space have to compete not just with GKE, but with the technical awesomeness of GCP. After years of using all the major clouds - Google Cloud has the best performance and primitives, but the worst support and operational/business features. If you want the fastest VMs/storage/networking, and are fine with non-standard APIs and beta SDKs, then it's a great platform and has super simple billing. AWS and Azure are much better suited for most companies and startups though in terms of actually getting your business running with standard tooling, large ecosystems, and a massive portfolio of services.

humanfromearth 9y ago
Funny, I had the exact opposite experience when it comes to APIs. Doesn't AWS have a PM for every product with different API each time? Tried to build something like a terraform using boto only to give up after dealing with inconsistent APIs for about 2 weeks.
ghthor 9y ago
It's probably AWS Fargate, part of the Elastic Container Service. But even then, it's still now a vendor lockin of the past, if you got your services deployable via containers you're 90% there to running your own Kubes cluster to launch them.
- manigandham 9y ago
  I think you're replying to the wrong comment...

lugg 9y ago

> What I'm waiting for, though, is for a big player to do a modern, clean "Kubernetes first" cloud offering. As someone who is feeling burned over picking swarm over k8s, I'm not super into the idea of trusting a 3rd party to do k8s first. Anything anyone implements well will be ported to gke in due time, no?

n00tz 9y ago
The good news is Docker EE 2.0 brings Kubernetes to you. However, I'm not sure if it's operating split-brain or if they made it such that you can pick your control surface.

mlrtime 9y ago

The biggest problem with GCP cannot create a private kubernetes cluster, it must have public interfaces. Why is this a requirement?

lobster_johnson 9y ago
Private clusters are in open beta: https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters https://cloud.google.com/kubernetes-engine/docs/how-to/priva....