17 ms·

Four years after its release, Kubernetes has come a long way

233 points by CrankyBear 8y ago

tnolet 8y ago
The only part missing in most Kubernetes write ups is that you probably don't need it. I'm hopeful most engineers recognise this, although my day to day experience paints a darker picture. I know of at least one hot startup deploying Kubernetes just to attract engineering talent.
- stemcc 8y ago
  In your experience with these deployments, which K8s features aren't needed? Is it the orchestration as such that's not necessary, or K8s approach wrt to the workloads deployed?
  - tnolet 8y ago
    Sorry for not being clear. There is nothing wrong with K8s features. It solves a specific problem really well. However, most people & companies I interact with don't have that problem. Or that problem is just a very marginal piece of their full set of problems. Running K8s, maintaining it and making your apps work with it is pretty big undertaking. Only go there if the upside is so great it blows any alternative away. I guess the popular term is "cargo culting". Or maybe Heroku is not hip anymore although it probably is the best advice for many companies.
    - meddlepal 8y ago
      I strongly disagree. I've built a couple in house PaaS type systems in the past with various degrees of success on more classical tech like straight up EC2 using a variety of Terraform/Ansible/CloudFormation/Scripting but the dev experience has always been painful compared to Kubernetes. Developers really love working with Kubernetes. It doesn't take a lot to get started and if you're not doing anything too crazy it is a huge productivity boon IMO. Where things get hairy is persistence, but that's the case regardless of Kubernetes or not.
      - auslander 8y ago
        >Terraform/Ansible/CloudFormation/Script This number of tools speaks for itself. Yeah, you would be better off with out of the box stuff due to lack of skills.
        meddlepal 8y ago
        Ah yes, you're right. I am sure you have a much better understanding of the requirements each of these build outs sitting behind your keyboard over there.
    - rvanmil 8y ago
      Heroku is still amazing indeed. Our team recently moved all Node.js based systems from AWS to Heroku and it has given us a tremendous productivity gain and reduction in complexity. Before we made that decision we spent about 8 months experimenting with K8s, only to discover it actually made things way more complex for us and reducing productivity without any clear benefits.
- outside1234 8y ago
  You probably don't need the cloud either, but everyone recognizes that it is a productivity gain. To me, this is the analogous case for K8s, you can probably do the same with VMs, but everything is a lot more productive with containers and K8s.
  - cc81 8y ago
    I don't have much practical experience with Kubernetes but I think one argument is that there are other alternatives to it that might give equal or better productivity gains for a lot of people. For example Azure App Service or equivalent from the other vendors. You don't have to manage a VM.
    - slow_donkey 8y ago
      App service is alright but I'd recommend Heroku over app service any day. However, there is some visible work being done to improve app service like supporting docker compose files to run multiple containers but the basics are still very painful. Most notably, swapping servers (slots) always caused downtime, accessing logs was flaky and slow, azure interface is just plain slow, continuous deploy was flaky (or it just didn't kick in until 20 min later?) and I had to manually restart servers many times
  - bartread 8y ago
    > You probably don't need the cloud either, but everyone recognizes that it is a productivity gain. Care to quantify that? Because we don't use the cloud (in fact, in my current role, we don't even make use of virtualisation for production infrastructure) and we have no issues with productivity. Or rather, let's be a little more honest, we have no issues with productivity that would be solved by a move to AWS/Azure/$_some_other_cloud_provider.
    - outside1234 8y ago
      You forgot about all the folks managing / maintaining / cabling / air conditioning all your servers. That is the productivity gain you are missing.
- brodsky 8y ago
  what would you recommend to use for container orchestration and service discovery instead of Kubernetes?
  - oldmanhorton 8y ago
    I believe the parent comment was trying to say that most people don't need service orchestration or discovery
    - tnolet 8y ago
      Yes. That is exactly what I meant. Mea culpa for not being more clear.
      - brodsky 8y ago
        no worries; that makes sense and I agree with you. (it's strange that my question resulted in downvotes).
      - pacala 8y ago
        My team runs a grand total of 1 server, 1 daemon and up to 5 concurrent batch jobs. Sometimes we bring up 1 extra server for experimentation. It is handy to not break the bank and use GKE autoscaling for the batch jobs. Surely we could setup some batch job manager over some raw VMs, or maybe spend some time to select and learn to use yet-another AWS service. It is handy to go to the GKE "Workloads" page and see how many servers are currently active, using how many pods. Surely we could pull that off with raw VMs, but it would take me a few minutes extra / a brittle script to pull it off. Do we _need_ Kubernetes? No. It makes us more productive? Yes. Am I confident that in the next assignment I'll be able to reuse my skills, regardless of what VM provider they use? 90% yes, pending EKS becoming usable. Echoes of "you don't need C, just use Assembly, it's simpler and faster" from early '90s gaming industry.
  - firebacon 8y ago
    I think the recommendation is not to use another orchestration service, but to keep the infrastructure simple and not use any orchestration or service discovery at all. Adding service discovery and container orchestration will probably not make your product better. Instead it will add more moving parts that can fail to your system and make operations more complex. So IMO a "containerized microservice architecture" is not a "feature" that you should add to your stack just because. It is a feature you should add to your stack once the benefits outweigh the costs, which IMO only happens at huge scale. Most people know that "Google {does, invented} containers". What not so many developers seem to realize is that a Google Borg "container" is a fundamentally different thing from a Docker container. Borg containers at Google are not really a dependency management mechanism; they solve the problem of scheduling heterogenous workloads developed by tens of thousands of engineers on the same shared compute infrastructure. This however, is a problem that most companies simply do not have, as they are just not running at the required scale. At reasonable scale, buying a bunch of servers will always be cheaper than employing a Kubernetes team. And if you do decide to add a clever cluster scheduler to your system it will probably not improve reliability, but will actually do the opposite. Even something like borg is not a panacea; you will occasionally have weird interactions from two incompatible workloads that happen to get scheduled on the same hardware, i.e. operational problems you would not have had without the "clever" scheduler. So again, unless you need it, you shouldn't use it. I think the problem that Docker does solve for small startups is that it gives you a a repeatable and portable environment. It makes it easy to create an image of your software that you are sure will run on the target server without having talk to the sysadmins or ops departments. But you don't need kubernetes to get that. And while I can appreciate this benefit of using docker, I still think shipping a whole linux environment for every application is not the right long-term way to do it. It does not really "solve" the problem of reliable deployments on linux; it is just a (pretty nice) workaround.
- rb808 8y ago
  > I know of at least one hot startup deploying Kubernetes just to attract engineering talent. Interesting proposition - if it helps attract better talent does that mean they needed k8s?
- swyx 8y ago
  i am a mainly js engineer and so far have lived entirely within the firebase world as far as my backend goes. in your (and everyone else's) opinion, then: when -do- you need it? set some quantifiable bars for me please?
  - tnolet 8y ago
    I could throw the "you'll know it when you see it" but I'll try not to! 1. You have a large amount of underused hardware/VM 's you want to cluster. 2. You have an eloborate microservices system with a dynamic service discovery mechanism. 3. You have the above over many regions / teams. 4. All your services are mostly stateless. 5. Your team of rockstar ops engineers are building their own crappy version of Kubernetes just to deal with the engineering challenges your business faces.
    - cogman10 8y ago
      Funnily, my current employer is just that. We predate K8s, so it can be forgiven somewhat. Transitioning, though, is just awful. We have every reason to transition, yet getting the system setup, running, and working with the current infrastructure is a daunting task. You don't need K8s, until you do, at which point switching can be a nightmare scenario.
    - swyx 8y ago
      thanks, i'm pretty far from that world then, haha. appreciate the list of concerns, i hope to someday worry about these things.
  - firebacon 8y ago
    When you notice that you spend such significant money (i.e. multiple salaries) on hardware capacity planning that employing a team to operate kubernetes might be cheaper instead. Probably not going to happen unless you run on hundreds to thousands of machines.
    - barrkel 8y ago
      Or vmware licensing is killing you and you can't use PaaS for contractual reasons.
- slow_donkey 8y ago
  What if you're on a hosted service like gke/aks? The work needed to maintain the system has been minimal for us with a lot of positives
  - bauerd 8y ago
    Yeah +1. I was super skeptical 1,5 months ago when I started my new job on a 22 node GKE cluster. "Surely K8s is overkill", "could've just used Nomad + Consul + Vault + X", "why not just use Fargate", "K8s seems like a distributed systemd", etc pp. Long story short, K8s on GKE just works, saves us tons of time, makes service-oriented architectures/microservices easy as pie and has led to a way better service level than the previous setup (Chef deployed on baremetal). Yes, it probably is overkill for your 10req/s app, granted. But I highly advise everyone to at least give it a try out of curiosity, because a lot of hard problems at medium scale and above just go poof with K8s.
    - firebacon 8y ago
      > because a lot of hard problems at medium scale and above just go poof with K8s. No, they don't? I don't know why anybody would just assume that something as complex as kubernetes would just run flawlessly once you actually try to run it on thousands of servers. Must be something to do with google PR because people definitely don't seem to assume the same for e.g. hadoop or openstack. Make a guess at how many people large companies have to employ to actually keep their smart cluster scheduler running?
      - bauerd 8y ago
        I was commenting on a specific usage context: >when I started my new job on a 22 node GKE cluster >problems at medium scale vs >Make a guess at how many people large companies have to employ to actually keep their smart cluster scheduler running K8s obviously is not a silver bullet and of course there's ops work to be done. I can't comment on whether operating K8s clusters in other contexts makes economic sense, but I know it does for our current team.
- tlarkworthy 8y ago
  I don't think that's true. If you are using bash or chef to interact with the cloud then k8s can replace those with a much cleaner model. My biggest fear is helm becomes chef cookbooks
  - ridruejo 8y ago
    Can you elaborate on the concerns with helm?
    - temikus 8y ago
      It’s probably referring to people using Chef cookbooks blindly without any knowledge of how to configure or maintain the service itself. I have interacted with people who run production workloads via Helm but struggle to do very basic things with kubectl (like recreating a service). This is an argument that can be used for any high level abstraction, however. I guess Chef is used since they have very good testing tools so Cookbooks are relatively more stable than other CM definitions. Until things go wrong with the service that is.
    - tlarkworthy 8y ago
      That community sourced cookbooks are security nightmares or under maintained. I cannot see how helm charts can stay quality without a curated app store.
      - geggam 8y ago
        Any unmaintained infrastructure as code piece will quit working. Success I have had with chef has been generating my own cookbooks that dont try to do everything for every OS
        tlarkworthy 8y ago
        Yes exactly. Everything tends towards rolling your own.
      - ridruejo 8y ago
- bruinjoe 8y ago
  I see a lot of value in Kubernetes. For one thing I don't want to wake up at 1 AM if one of my node goes down and brings down with it important containers. If I scaled the environment with excess capacity then Kubernetes should start up new pods on the remaining nodes.
  - tnolet 8y ago
    That is an awesome feature of K8s and similar orchestators. It is however just an HA (high availabilty) problem already solved many times over in many ways. Almost every PaaS system has it. All cloud providers have it. Getting K8s (or any of its competitors) for HA is total overkill.
    - kuschku 8y ago
      The point is that I can rent the cheapest $10/month servers I can find, a bunch of them, throw kubernetes on that, and it’ll still be up. Same reliability as AWS, similar performance, much cheaper.
      - yongjik 8y ago
        Unless you're renting from multiple $10/month services, your service can be only as reliable as your service provider, which is probably less than AWS.
        bruinjoe 8y ago
        You hit it on the head. Weave Net is participating in the Istio project. I hope they can bring WeaveNet to Istio. That might be able to provide a multi cloud vendor service mesh.
        kuschku 8y ago
        But that’s easily possible – you can use all the French hosters at once. Latency between them is practically nil because their data centers are only a few hundred meters away from each other. Sure, in case all of Paris goes down, you’re down, too. But in day-to-day usage you can have good reliability on a shoestring budget.
- indescions_2018 8y ago
  "Kubernetes" is an Ancient Greek word meaning "more containers than customers" https://twitter.com/srbaker/status/1002286820078571532 https://twitter.com/srbaker/status/1002286820078571532
  - smnrchrds 8y ago
    I assume it is written in the style of "Ubuntu is an ancient african word, meaning 'I can't configure Debian."
    - StavrosK 8y ago
      It is, "kubernetes" is actually the root of the word "governor", with similar meaning in ancient and modern Greek. The naming of the software makes for very unfortunate conversations if you're talking to other Greeks about it.
  - kbenson 8y ago
    This totally could have come from DevOps Borat. [1] 1: https://mobile.twitter.com/devops_borat https://mobile.twitter.com/devops_borat
- deleted 8y ago
  [deleted]
- ShabbosGoy 8y ago
  In that specific case, Kubernetes is approaching YAGNI territory. But as others have stated, it’s easy to get started with something like Amazon EKS and not have to worry about a thing. Edit: Put another way, give me APIs instead of config file hell.
- AndyNemmity 8y ago
  Depends on your definition of "need", it enables me to manage a ton of stuff across multiple dcs and regions easily. It would feel like the dark ages to be without it as this point.
- collyw 8y ago
  Sadly I doubt it. Look at this question and top answer from Quora. It asks how you would build a website.The answer dives into a host of engineered technologies that would suit something you expect to need to scale massively. https://www.quora.com/If-you-were-to-build-a-website-right-now-from-scratch-what-language-and-tools-would-you-use https://www.quora.com/If-you-were-to-build-a-website-right-n...
  - tnolet 8y ago
    Wow. The top answer on that one is terrible. I would advise the exact opposite. Istio!? Nice idea, but have you tried using it? That is some serious alpha stage software almost noone needs till very late in their tech maturity.
state_less 8y ago
It's nice for users to have a common platform, you don't have rewrite nearly as much code or config (approaching zero) when changing providers. Maybe it'll spur new developments due to the lower costs and provide bedrock for big investments to build on. It's still early days. It'd be nice to see more P2P cluster building tools for individuals and groups of friends.
- markovbot 8y ago
  >It'd be nice to see more P2P cluster building tools for individuals and groups of friends. I've been wanting to build a k8s cluster for myself and a few friends. I didn't feel there was a lack of tooling to facilitate this, but I haven't really looked into it much. Could you share some of your specific concerns, and any tips if you've done this?
  - state_less 8y ago
    There is kubeadm [1] that is part of the project and it does make it easier than in the past to get a cluster running on hosts you control. You can create VMs (Ubuntu, Debian, RHEL, etc...) and run kubeadm on each to join nodes together on the cluster. I'd like it to be even easier though, so your friend just downloads the P2P Kubernetes application, run it and they can see your cluster and join it (with some key that you give them) using a GUI. Similar process if they are starting the cluster, except this time they name the cluster and hand out the invite (with key) to their friend. [1] https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ https://kubernetes.io/docs/setup/independent/create-cluster-...
    - markovbot 8y ago
      ah okay, that's different from what I was thinking of a cluster shared by a group of friends. I'm running my on VMs from a cheap VPS provider, was going to just split the bill with others who used it. Your use case is very interesting though. It would be interesting to see that eventually.
bg4 8y ago
Agreed. And to follow that thought, you probably don't need microservices either.
dopeboy 8y ago
I consult and I also run a small SaaS startup. All of my deploys thus far service an audience of ~100 users per month---fairly small. I git push my monolith to Heroku and call it a day. Because of that, I don't know anything about containers, Kubernetes, Docker, etc. As a forward thinking engineer, what should I do to stay up with the times? Is it worth my time dockerizing my projects? Should I be using kubernetes when deploying my projects?
- oreganoz 8y ago
  I don't think using kubernetes is useful for small projects. It's currently still a hastle, even though it has evolved a lot. Not much in terms of benefits to show for it either. I would read up on Docker though. Having your apps in docker does provide some nice benefits like better isolation, keeps you up with newer tech and you can then easily integrate with Kubernetes. It's also rather relatively easy to achieve.
- vincentmarle 8y ago
  Nope, you're good.
- d4l3k 8y ago
  Probably not. Container services really shine when you have a lot of different teams that all need to launch services and not worry about what happens at the hardware level or where they run. This makes sense at a lot of large companies like Facebook and Google where they have a dedicated team running the container layer. GKE makes it a bit easier but then you're locked into Google Cloud. If you're only running a few services it really doesn't make much sense to setup Kubernetes. I setup a small cluster a few weeks ago and it certainly was a lot more involved than I expected. I ended up deciding not to use it, since it didn't really add much (for my use case) and I felt like if anything went wrong I'd be in a world of hurt trying to debug it. Took a lot more time to setup Kubernetes than my entire current deployment system which uses https://github.com/pressly/sup https://github.com/pressly/sup. However, dockerizing your projects might not be a bad idea depending on what you're doing. I dockerize a lot of my projects since it makes it super easy to deploy when using the automatic Docker Hub builds. Though I did have a problem where Docker upgraded and make all of my existing containers unrunnable.
w8rbt 8y ago
IMO, Kubernetes is another indicator of the success of Go (Docker is too). Go is simple and fast like C, yet safe like Python. It really combines great performance and safety.
- kraig 8y ago
  sometimes i wonder how openstack would have done if it was written in go
  - wmf 8y ago
    It would be much faster but would still be extremely flaky and all the architectural problems (e.g. RPC over MQ) would still be there.
- piano 8y ago
  Is it though? I believe the success of Kubernetes is mostly due to Docker whose success is mostly due to features provided by the Linux kernel, to which Docker is a frontend... Docker's code is seems not very nice to me (when I last read it), I especially don't like the error handling, which provides very little context to go on.
firebacon 8y ago
I think the kubernetes project is heavily driven by Google marketing, and that they are not doing this out of charity, but because they are trying to get you to use their cloud platform in the long run. They know getting somebody to build their stuff on open-source kubernetes is a win for them. After some time people will realize that running kubernetes yourself is actually harder and more fragile than just running your app without it and at that point the obvious move will be to use a hosted kubernetes service, like Google's cloud. And I really think just handing over all our apps to google to run them for us is not in our (developers) interest in the long term. It would further solidify Google's "monopoly of the internet" position and also means that in the future - once we have succeeded in convincing our bosses to just rent every interesting bit of technology from google - that the only interesting jobs left will be... at Google. So please go ahead and downvote me, but please also try to consider my point of view (that there is a ton of very aggressive marketing with a financial incentive going on here) next time you read and defend some kubernetes hype piece.
- pacala 8y ago
  You are correct on Google self-interest in the matter. Not so sure that Google's self-interest is so divergent from the average developer interest. Kubernetes is hard to run by oneself. Worse, self-cobbled setups are also hard to run by oneself. Distributed systems, that is systems that require uptime and/or more than one computer, are hard. The fact of the matter is that computing is the new electricity. While one can run oneself's own generator in the backyard, it is more efficient to buy electricity from a big provider, small-scale solar notwithstanding. Kubernetes offers a well designed abstraction that simplifies managing distributed systems at most scales. Bonus, it has become a shared standard among providers of computing. That is a big step beyond proprietary lock-in abstractions [AWS, etc.], and likely close to the best one can hope for, technically speaking. Economically, perhaps a future solution is regulating computing providers as utilities.
  - firebacon 8y ago
    I agree that running kubernetes ourselves does actually serve our own interests. But I still think it's much more of a slippery slope than alternative solutions (being that some of them are not developed with the intention to eventually get you to use the hosted version)
vshan 8y ago
It's astounding how much of Kubernetes is simply marketing. Google doesn't even use it internally. Service Fabric[0] is leagues ahead of any cluster orchestration framework, heck, it's a complete distributed systems platform which powers all critical Azure services (Cosmos DB, Cortana, SQL DB). It is the only framework that allows native stateful applications; your state doesn't need to be delegated to another service. It offers Reliable Collections (Java and C#) which are persistent, replicated and sharded native collections. I wish more devs knew about SF. [0] https://github.com/Microsoft/service-fabric https://github.com/Microsoft/service-fabric
- the_new_guy_29 8y ago
  This one here is also a pure marketing...
- jasonvorhe 8y ago
  What an insult to one of the most impressive communities in open source. Shameful.
sidcool 8y ago
A lot of displeasure around K8S here, are there any simpler alternatives?
- kross 8y ago
  I get the sense that the naysayers outweight the supporters, simply because the headline alone is not news. Don't let them deter you, spend a few hours and decide for yourself.
- parasubvert 8y ago
  HN tends to follow tech hype cycles, Kubernetes is entering the “trough of disillusionment” now that people are starting to use it in anger. This reminds me of prior fights like Solaris vs Linux. Kubernetes is not that bad, its actually quite elegant in many ways. It’s just that it is tackling a hard problem in the hardest way possible: by providing a new set of bottom up abstractions to replace traditional virtual machines, config management, and network and storage virtualization in the image of the Google model of massive scale workload scheduling. That means it is a whole new world to learn but you can’t quite forget about the old world yet, and need to map between the two. Is it complicated? Yeah, mainly because it’s evolving so fast and serving many audiences. It will get easier. Simpler alternatives? Any PaaS like Heroku or Google App Engine or Cloud Foundry. Or a serverless function platform like AWS lambda (not sure if it’s “simpler” but it is getting there) Disclaimer I work for Pivotal who sells cloud foundry and a distro of Kubernetes
  - aphexairlines 8y ago
    AWS Lambda seems much simpler. Bundle your code in a zip file. Describe it in a yaml template with the name, runtime, zip filename, memory needed, entry point in your code, and external HTTP path. Deploy it with "aws cloudformation package" and "aws cloudformation deploy". The platform will give you load balancing, scaling, logging, metrics, internal and external endpoints, DNS, and CDN. The PaaS platforms you mentioned are basically the same (you provide code + a bit of config and they do the rest), except priced by the hour.
streblo 8y ago
There's a lot of FUD in the comments here about kubernetes, so let me chime in with a success story -- my company has been using kubernetes to operate a reasonably large and complex set of services (>100 nodes/pods, >5B requests/day) and we're doing it with only 3 engineers. Our experience has been great, and if I had to do it again, I'd happily choose k8s without reservation.
- timeSl 8y ago
  Can I ask if you have a bare metal cluster or you do this through a cloud service?
  - streblo 8y ago
    We are using Google Kubernetes Engine
alexk 8y ago
I am writing this for folks who are reading this thread and wondering what's of said here is true and what's not true. * Google conspiracy theories - (Google is doing it to lock everyone in). Not true. Kubernetes was created as a best-practices fully open source Borg, the very reason it's using Docker and Etcd is the desire of Googlers to work with OSS community and it paid back by Red Hat and later CoreOs and others joining the project. Kubernetes is complex, yes, but so is Mesos and OpenStack, and pretty much any other production grade orchestration system I've seen, so I would disregard this argument as well. * Google is not even using Kubernetes. Google is a big company, they can't switch to Kubernetes overnight or even in 5 years. I don't have this info, but I'm pretty sure many teams are using GKE and the internal adoption grows. * What if Google goes away? Everyone will get locked in! Google is not the only major contributor to Kubernetes core these days. Red Hat is another one, and many others, so this lock in point is not valid. Google is not even BDFL - the project is governed by CNCF (OSS foundation similar to Linux foundation) and the project was donated by Google several years ago. Kubernetes development process is organized in the most fascinating open process I have ever seen - SIGs (special interest groups) are fully open and anyone can participate and help develop the project. I have learned a lot about openness just looking at the organisation of the dev process. * You don't need Kubernetes if you have one service. Not true. You can benefit from Kubernetes even if you have just one container. It solves many problems you would have to solve otherwise - service discovery, configuration management, load balancing, secrets management, fail-over, routing, publishing new releases and packaging and many others. You probably don't want to self-host Kubernetes, because it is complex unless you have experience and desire to do so, that point is valid. But thankfully, you can use GKE, AKS or EKS nowadays. Disclaimer: Our company works with Kubernetes so I am biased, but we have no affiliation with Red Hat or Google.
- jacques_chester 8y ago
  > Google conspiracy theories - (Google is doing it to lock everyone in). My own conspiracy theory is slightly different: Google's aim is to scorch the earth so that AWS cannot form a locked-in position. I don't think they immediately saw this possibility, but that they moved swiftly and energetically to support the growth of Kubernetes once they did see it. The alternatives for Google were all strategically painful. ECS might have become successful, meaning a loss for Google. Mesos or Docker Swarm might have taken off, leading to snap acquisitions of the relevant companies by Microsoft or AWS, again a downside for Google. As the plurality contributor to the commoditising winner, Google is able to prevent the other two cloud giants from pulling ahead on container orchestration. Disclosure: I work for Pivotal, we have a Kubernetes-based product (PKS) we co-develop with Google and VMWare.
nstart 8y ago
Maybe off topic. But before Kubernetes AND docker, how did people manage microservices? Was it a machine stack per service? Were there tools that helped keep all the services running together in a single machine? I haven't been in software for long enough to know what this past looked like so courious if any of the HN community might know more about this especially at a personal experience level.
- firebacon 8y ago
  > Were there tools that helped keep all the services running together in a single machine? Traditionally, services on unix system are started and supervised directly by the init process ("init system"). That has been SysV init for most of the time, but in the last decade we got two major new systems: upstart (dead now) and the infamous systemd. Using systemd can still be a very good alternative to using docker today. > Was it a machine stack per service? That still sounds like a good approach to me today. If your service is not big/expensive enough to need at least one full machine, why bother cutting your app up into such small services in the first place? For example, for web applications, a traditional setup would have been to have a couple of dedicated servers that run the database and a larger pool of webservers that serve the (stateless) application. In front of that you could have two linux boxes doing the network load balancing and failover or a commercial load balancer "box" product.
- wmf 8y ago
  Here's Netflix from 2011 (pre-Docker): https://www.slideshare.net/adrianco/global-netflix-platform/19-Ne9lix_Global_PaaS_Architecture_Features https://www.slideshare.net/adrianco/global-netflix-platform/... This was widely considered the gold standard in cloud native architecture. Twitter from 2013: https://www.slideshare.net/caniszczyk/twitter-opensourcestacklinuxcon2013 https://www.slideshare.net/caniszczyk/twitter-opensourcestac... They had a similar in-process JVM-based approach as Netflix, but they also used Mesos as a scheduler.
- jacques_chester 8y ago
  > But before Kubernetes AND docker, how did people manage microservices? Cloud Foundry and other opensource PaaSes. Heroku. Chef, Puppet et al. Java app servers with WARs. Before that was a time of heroic mythology. Shell scripts, bailing wire and raw genius. Disclosure: I work for Pivotal, we work on Cloud Foundry.
throw2016 8y ago
Devops has added a web of complexity that distance users from the underlying technology. Suddenly mounting filesystems is a 'storage driver', using networking technology built into the kernel is a 'network driver' or using an Nginx reverse proxy or Haproxy load balancer is an 'ingress controller'. This new vocabulary confuses rather than informs and ends up adding more layers of complexity. What happens when things break? Simply knowing the json/yaml layers above is not going to help without an understanding of networking and the underlying technologies. Facebook, Google, Netflix are not 2 engineers running devops, they all have unique architectures and an army of experts to run their infrastructure. The idea that you can be 'webscale' without ops experts is complete fantasy.

233 points by CrankyBear 8y ago

The only part missing in most Kubernetes write ups is that you probably don't need it. I'm hopeful most engineers recognise this, although my day to day experience paints a darker picture. I know of at least one hot startup deploying Kubernetes just to attract engineering talent.

stemcc 8y ago
In your experience with these deployments, which K8s features aren't needed? Is it the orchestration as such that's not necessary, or K8s approach wrt to the workloads deployed?
- tnolet 8y ago
  Sorry for not being clear. There is nothing wrong with K8s features. It solves a specific problem really well. However, most people & companies I interact with don't have that problem. Or that problem is just a very marginal piece of their full set of problems. Running K8s, maintaining it and making your apps work with it is pretty big undertaking. Only go there if the upside is so great it blows any alternative away. I guess the popular term is "cargo culting". Or maybe Heroku is not hip anymore although it probably is the best advice for many companies.
  - meddlepal 8y ago
    I strongly disagree. I've built a couple in house PaaS type systems in the past with various degrees of success on more classical tech like straight up EC2 using a variety of Terraform/Ansible/CloudFormation/Scripting but the dev experience has always been painful compared to Kubernetes. Developers really love working with Kubernetes. It doesn't take a lot to get started and if you're not doing anything too crazy it is a huge productivity boon IMO. Where things get hairy is persistence, but that's the case regardless of Kubernetes or not.
    - auslander 8y ago
      >Terraform/Ansible/CloudFormation/Script This number of tools speaks for itself. Yeah, you would be better off with out of the box stuff due to lack of skills.
      - meddlepal 8y ago
        Ah yes, you're right. I am sure you have a much better understanding of the requirements each of these build outs sitting behind your keyboard over there.
  - rvanmil 8y ago
    Heroku is still amazing indeed. Our team recently moved all Node.js based systems from AWS to Heroku and it has given us a tremendous productivity gain and reduction in complexity. Before we made that decision we spent about 8 months experimenting with K8s, only to discover it actually made things way more complex for us and reducing productivity without any clear benefits.
outside1234 8y ago
You probably don't need the cloud either, but everyone recognizes that it is a productivity gain. To me, this is the analogous case for K8s, you can probably do the same with VMs, but everything is a lot more productive with containers and K8s.
- cc81 8y ago
  I don't have much practical experience with Kubernetes but I think one argument is that there are other alternatives to it that might give equal or better productivity gains for a lot of people. For example Azure App Service or equivalent from the other vendors. You don't have to manage a VM.
  - slow_donkey 8y ago
    App service is alright but I'd recommend Heroku over app service any day. However, there is some visible work being done to improve app service like supporting docker compose files to run multiple containers but the basics are still very painful. Most notably, swapping servers (slots) always caused downtime, accessing logs was flaky and slow, azure interface is just plain slow, continuous deploy was flaky (or it just didn't kick in until 20 min later?) and I had to manually restart servers many times
- bartread 8y ago
  > You probably don't need the cloud either, but everyone recognizes that it is a productivity gain. Care to quantify that? Because we don't use the cloud (in fact, in my current role, we don't even make use of virtualisation for production infrastructure) and we have no issues with productivity. Or rather, let's be a little more honest, we have no issues with productivity that would be solved by a move to AWS/Azure/$_some_other_cloud_provider.
  - outside1234 8y ago
    You forgot about all the folks managing / maintaining / cabling / air conditioning all your servers. That is the productivity gain you are missing.
brodsky 8y ago
what would you recommend to use for container orchestration and service discovery instead of Kubernetes?
- oldmanhorton 8y ago
  I believe the parent comment was trying to say that most people don't need service orchestration or discovery
  - tnolet 8y ago
    Yes. That is exactly what I meant. Mea culpa for not being more clear.
    - brodsky 8y ago
      no worries; that makes sense and I agree with you. (it's strange that my question resulted in downvotes).
    - pacala 8y ago
      My team runs a grand total of 1 server, 1 daemon and up to 5 concurrent batch jobs. Sometimes we bring up 1 extra server for experimentation. It is handy to not break the bank and use GKE autoscaling for the batch jobs. Surely we could setup some batch job manager over some raw VMs, or maybe spend some time to select and learn to use yet-another AWS service. It is handy to go to the GKE "Workloads" page and see how many servers are currently active, using how many pods. Surely we could pull that off with raw VMs, but it would take me a few minutes extra / a brittle script to pull it off. Do we _need_ Kubernetes? No. It makes us more productive? Yes. Am I confident that in the next assignment I'll be able to reuse my skills, regardless of what VM provider they use? 90% yes, pending EKS becoming usable. Echoes of "you don't need C, just use Assembly, it's simpler and faster" from early '90s gaming industry.
- firebacon 8y ago
  I think the recommendation is not to use another orchestration service, but to keep the infrastructure simple and not use any orchestration or service discovery at all. Adding service discovery and container orchestration will probably not make your product better. Instead it will add more moving parts that can fail to your system and make operations more complex. So IMO a "containerized microservice architecture" is not a "feature" that you should add to your stack just because. It is a feature you should add to your stack once the benefits outweigh the costs, which IMO only happens at huge scale. Most people know that "Google {does, invented} containers". What not so many developers seem to realize is that a Google Borg "container" is a fundamentally different thing from a Docker container. Borg containers at Google are not really a dependency management mechanism; they solve the problem of scheduling heterogenous workloads developed by tens of thousands of engineers on the same shared compute infrastructure. This however, is a problem that most companies simply do not have, as they are just not running at the required scale. At reasonable scale, buying a bunch of servers will always be cheaper than employing a Kubernetes team. And if you do decide to add a clever cluster scheduler to your system it will probably not improve reliability, but will actually do the opposite. Even something like borg is not a panacea; you will occasionally have weird interactions from two incompatible workloads that happen to get scheduled on the same hardware, i.e. operational problems you would not have had without the "clever" scheduler. So again, unless you need it, you shouldn't use it. I think the problem that Docker does solve for small startups is that it gives you a a repeatable and portable environment. It makes it easy to create an image of your software that you are sure will run on the target server without having talk to the sysadmins or ops departments. But you don't need kubernetes to get that. And while I can appreciate this benefit of using docker, I still think shipping a whole linux environment for every application is not the right long-term way to do it. It does not really "solve" the problem of reliable deployments on linux; it is just a (pretty nice) workaround.
rb808 8y ago
> I know of at least one hot startup deploying Kubernetes just to attract engineering talent. Interesting proposition - if it helps attract better talent does that mean they needed k8s?
swyx 8y ago
i am a mainly js engineer and so far have lived entirely within the firebase world as far as my backend goes. in your (and everyone else's) opinion, then: when -do- you need it? set some quantifiable bars for me please?
- tnolet 8y ago
  I could throw the "you'll know it when you see it" but I'll try not to! 1. You have a large amount of underused hardware/VM 's you want to cluster. 2. You have an eloborate microservices system with a dynamic service discovery mechanism. 3. You have the above over many regions / teams. 4. All your services are mostly stateless. 5. Your team of rockstar ops engineers are building their own crappy version of Kubernetes just to deal with the engineering challenges your business faces.
  - cogman10 8y ago
    Funnily, my current employer is just that. We predate K8s, so it can be forgiven somewhat. Transitioning, though, is just awful. We have every reason to transition, yet getting the system setup, running, and working with the current infrastructure is a daunting task. You don't need K8s, until you do, at which point switching can be a nightmare scenario.
  - swyx 8y ago
    thanks, i'm pretty far from that world then, haha. appreciate the list of concerns, i hope to someday worry about these things.
- firebacon 8y ago
  When you notice that you spend such significant money (i.e. multiple salaries) on hardware capacity planning that employing a team to operate kubernetes might be cheaper instead. Probably not going to happen unless you run on hundreds to thousands of machines.
  - barrkel 8y ago
    Or vmware licensing is killing you and you can't use PaaS for contractual reasons.
slow_donkey 8y ago
What if you're on a hosted service like gke/aks? The work needed to maintain the system has been minimal for us with a lot of positives
- bauerd 8y ago
  Yeah +1. I was super skeptical 1,5 months ago when I started my new job on a 22 node GKE cluster. "Surely K8s is overkill", "could've just used Nomad + Consul + Vault + X", "why not just use Fargate", "K8s seems like a distributed systemd", etc pp. Long story short, K8s on GKE just works, saves us tons of time, makes service-oriented architectures/microservices easy as pie and has led to a way better service level than the previous setup (Chef deployed on baremetal). Yes, it probably is overkill for your 10req/s app, granted. But I highly advise everyone to at least give it a try out of curiosity, because a lot of hard problems at medium scale and above just go poof with K8s.
  - firebacon 8y ago
    > because a lot of hard problems at medium scale and above just go poof with K8s. No, they don't? I don't know why anybody would just assume that something as complex as kubernetes would just run flawlessly once you actually try to run it on thousands of servers. Must be something to do with google PR because people definitely don't seem to assume the same for e.g. hadoop or openstack. Make a guess at how many people large companies have to employ to actually keep their smart cluster scheduler running?
    - bauerd 8y ago
      I was commenting on a specific usage context: >when I started my new job on a 22 node GKE cluster >problems at medium scale vs >Make a guess at how many people large companies have to employ to actually keep their smart cluster scheduler running K8s obviously is not a silver bullet and of course there's ops work to be done. I can't comment on whether operating K8s clusters in other contexts makes economic sense, but I know it does for our current team.
tlarkworthy 8y ago
I don't think that's true. If you are using bash or chef to interact with the cloud then k8s can replace those with a much cleaner model. My biggest fear is helm becomes chef cookbooks
- ridruejo 8y ago
  Can you elaborate on the concerns with helm?
  - temikus 8y ago
    It’s probably referring to people using Chef cookbooks blindly without any knowledge of how to configure or maintain the service itself. I have interacted with people who run production workloads via Helm but struggle to do very basic things with kubectl (like recreating a service). This is an argument that can be used for any high level abstraction, however. I guess Chef is used since they have very good testing tools so Cookbooks are relatively more stable than other CM definitions. Until things go wrong with the service that is.
  - tlarkworthy 8y ago
    That community sourced cookbooks are security nightmares or under maintained. I cannot see how helm charts can stay quality without a curated app store.
    - geggam 8y ago
      Any unmaintained infrastructure as code piece will quit working. Success I have had with chef has been generating my own cookbooks that dont try to do everything for every OS
      - tlarkworthy 8y ago
        Yes exactly. Everything tends towards rolling your own.
    - ridruejo 8y ago
bruinjoe 8y ago
I see a lot of value in Kubernetes. For one thing I don't want to wake up at 1 AM if one of my node goes down and brings down with it important containers. If I scaled the environment with excess capacity then Kubernetes should start up new pods on the remaining nodes.
- tnolet 8y ago
  That is an awesome feature of K8s and similar orchestators. It is however just an HA (high availabilty) problem already solved many times over in many ways. Almost every PaaS system has it. All cloud providers have it. Getting K8s (or any of its competitors) for HA is total overkill.
  - kuschku 8y ago
    The point is that I can rent the cheapest $10/month servers I can find, a bunch of them, throw kubernetes on that, and it’ll still be up. Same reliability as AWS, similar performance, much cheaper.
    - yongjik 8y ago
      Unless you're renting from multiple $10/month services, your service can be only as reliable as your service provider, which is probably less than AWS.
      - bruinjoe 8y ago
        You hit it on the head. Weave Net is participating in the Istio project. I hope they can bring WeaveNet to Istio. That might be able to provide a multi cloud vendor service mesh.
      - kuschku 8y ago
        But that’s easily possible – you can use all the French hosters at once. Latency between them is practically nil because their data centers are only a few hundred meters away from each other. Sure, in case all of Paris goes down, you’re down, too. But in day-to-day usage you can have good reliability on a shoestring budget.
indescions_2018 8y ago
"Kubernetes" is an Ancient Greek word meaning "more containers than customers" https://twitter.com/srbaker/status/1002286820078571532 https://twitter.com/srbaker/status/1002286820078571532
- smnrchrds 8y ago
  I assume it is written in the style of "Ubuntu is an ancient african word, meaning 'I can't configure Debian."
  - StavrosK 8y ago
    It is, "kubernetes" is actually the root of the word "governor", with similar meaning in ancient and modern Greek. The naming of the software makes for very unfortunate conversations if you're talking to other Greeks about it.
- kbenson 8y ago
  This totally could have come from DevOps Borat. [1] 1: https://mobile.twitter.com/devops_borat https://mobile.twitter.com/devops_borat
deleted 8y ago
[deleted]
ShabbosGoy 8y ago
In that specific case, Kubernetes is approaching YAGNI territory. But as others have stated, it’s easy to get started with something like Amazon EKS and not have to worry about a thing. Edit: Put another way, give me APIs instead of config file hell.
AndyNemmity 8y ago
Depends on your definition of "need", it enables me to manage a ton of stuff across multiple dcs and regions easily. It would feel like the dark ages to be without it as this point.
collyw 8y ago
Sadly I doubt it. Look at this question and top answer from Quora. It asks how you would build a website.The answer dives into a host of engineered technologies that would suit something you expect to need to scale massively. https://www.quora.com/If-you-were-to-build-a-website-right-now-from-scratch-what-language-and-tools-would-you-use https://www.quora.com/If-you-were-to-build-a-website-right-n...
- tnolet 8y ago
  Wow. The top answer on that one is terrible. I would advise the exact opposite. Istio!? Nice idea, but have you tried using it? That is some serious alpha stage software almost noone needs till very late in their tech maturity.

state_less 8y ago

It's nice for users to have a common platform, you don't have rewrite nearly as much code or config (approaching zero) when changing providers. Maybe it'll spur new developments due to the lower costs and provide bedrock for big investments to build on. It's still early days. It'd be nice to see more P2P cluster building tools for individuals and groups of friends.

markovbot 8y ago
>It'd be nice to see more P2P cluster building tools for individuals and groups of friends. I've been wanting to build a k8s cluster for myself and a few friends. I didn't feel there was a lack of tooling to facilitate this, but I haven't really looked into it much. Could you share some of your specific concerns, and any tips if you've done this?
- state_less 8y ago
  There is kubeadm [1] that is part of the project and it does make it easier than in the past to get a cluster running on hosts you control. You can create VMs (Ubuntu, Debian, RHEL, etc...) and run kubeadm on each to join nodes together on the cluster. I'd like it to be even easier though, so your friend just downloads the P2P Kubernetes application, run it and they can see your cluster and join it (with some key that you give them) using a GUI. Similar process if they are starting the cluster, except this time they name the cluster and hand out the invite (with key) to their friend. [1] https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ https://kubernetes.io/docs/setup/independent/create-cluster-...
  - markovbot 8y ago
    ah okay, that's different from what I was thinking of a cluster shared by a group of friends. I'm running my on VMs from a cheap VPS provider, was going to just split the bill with others who used it. Your use case is very interesting though. It would be interesting to see that eventually.

bg4 8y ago

Agreed. And to follow that thought, you probably don't need microservices either.

dopeboy 8y ago

I consult and I also run a small SaaS startup. All of my deploys thus far service an audience of ~100 users per month---fairly small. I git push my monolith to Heroku and call it a day. Because of that, I don't know anything about containers, Kubernetes, Docker, etc. As a forward thinking engineer, what should I do to stay up with the times? Is it worth my time dockerizing my projects? Should I be using kubernetes when deploying my projects?

oreganoz 8y ago
I don't think using kubernetes is useful for small projects. It's currently still a hastle, even though it has evolved a lot. Not much in terms of benefits to show for it either. I would read up on Docker though. Having your apps in docker does provide some nice benefits like better isolation, keeps you up with newer tech and you can then easily integrate with Kubernetes. It's also rather relatively easy to achieve.
vincentmarle 8y ago
Nope, you're good.
d4l3k 8y ago
Probably not. Container services really shine when you have a lot of different teams that all need to launch services and not worry about what happens at the hardware level or where they run. This makes sense at a lot of large companies like Facebook and Google where they have a dedicated team running the container layer. GKE makes it a bit easier but then you're locked into Google Cloud. If you're only running a few services it really doesn't make much sense to setup Kubernetes. I setup a small cluster a few weeks ago and it certainly was a lot more involved than I expected. I ended up deciding not to use it, since it didn't really add much (for my use case) and I felt like if anything went wrong I'd be in a world of hurt trying to debug it. Took a lot more time to setup Kubernetes than my entire current deployment system which uses https://github.com/pressly/sup https://github.com/pressly/sup. However, dockerizing your projects might not be a bad idea depending on what you're doing. I dockerize a lot of my projects since it makes it super easy to deploy when using the automatic Docker Hub builds. Though I did have a problem where Docker upgraded and make all of my existing containers unrunnable.

w8rbt 8y ago

IMO, Kubernetes is another indicator of the success of Go (Docker is too). Go is simple and fast like C, yet safe like Python. It really combines great performance and safety.

kraig 8y ago
sometimes i wonder how openstack would have done if it was written in go
- wmf 8y ago
  It would be much faster but would still be extremely flaky and all the architectural problems (e.g. RPC over MQ) would still be there.
piano 8y ago
Is it though? I believe the success of Kubernetes is mostly due to Docker whose success is mostly due to features provided by the Linux kernel, to which Docker is a frontend... Docker's code is seems not very nice to me (when I last read it), I especially don't like the error handling, which provides very little context to go on.

firebacon 8y ago

I think the kubernetes project is heavily driven by Google marketing, and that they are not doing this out of charity, but because they are trying to get you to use their cloud platform in the long run. They know getting somebody to build their stuff on open-source kubernetes is a win for them. After some time people will realize that running kubernetes yourself is actually harder and more fragile than just running your app without it and at that point the obvious move will be to use a hosted kubernetes service, like Google's cloud. And I really think just handing over all our apps to google to run them for us is not in our (developers) interest in the long term. It would further solidify Google's "monopoly of the internet" position and also means that in the future - once we have succeeded in convincing our bosses to just rent every interesting bit of technology from google - that the only interesting jobs left will be... at Google. So please go ahead and downvote me, but please also try to consider my point of view (that there is a ton of very aggressive marketing with a financial incentive going on here) next time you read and defend some kubernetes hype piece.

pacala 8y ago
You are correct on Google self-interest in the matter. Not so sure that Google's self-interest is so divergent from the average developer interest. Kubernetes is hard to run by oneself. Worse, self-cobbled setups are also hard to run by oneself. Distributed systems, that is systems that require uptime and/or more than one computer, are hard. The fact of the matter is that computing is the new electricity. While one can run oneself's own generator in the backyard, it is more efficient to buy electricity from a big provider, small-scale solar notwithstanding. Kubernetes offers a well designed abstraction that simplifies managing distributed systems at most scales. Bonus, it has become a shared standard among providers of computing. That is a big step beyond proprietary lock-in abstractions [AWS, etc.], and likely close to the best one can hope for, technically speaking. Economically, perhaps a future solution is regulating computing providers as utilities.
- firebacon 8y ago
  I agree that running kubernetes ourselves does actually serve our own interests. But I still think it's much more of a slippery slope than alternative solutions (being that some of them are not developed with the intention to eventually get you to use the hosted version)

vshan 8y ago

It's astounding how much of Kubernetes is simply marketing. Google doesn't even use it internally. Service Fabric[0] is leagues ahead of any cluster orchestration framework, heck, it's a complete distributed systems platform which powers all critical Azure services (Cosmos DB, Cortana, SQL DB). It is the only framework that allows native stateful applications; your state doesn't need to be delegated to another service. It offers Reliable Collections (Java and C#) which are persistent, replicated and sharded native collections. I wish more devs knew about SF. [0] https://github.com/Microsoft/service-fabric https://github.com/Microsoft/service-fabric

the_new_guy_29 8y ago
This one here is also a pure marketing...
jasonvorhe 8y ago
What an insult to one of the most impressive communities in open source. Shameful.

sidcool 8y ago

A lot of displeasure around K8S here, are there any simpler alternatives?

kross 8y ago
I get the sense that the naysayers outweight the supporters, simply because the headline alone is not news. Don't let them deter you, spend a few hours and decide for yourself.
parasubvert 8y ago
HN tends to follow tech hype cycles, Kubernetes is entering the “trough of disillusionment” now that people are starting to use it in anger. This reminds me of prior fights like Solaris vs Linux. Kubernetes is not that bad, its actually quite elegant in many ways. It’s just that it is tackling a hard problem in the hardest way possible: by providing a new set of bottom up abstractions to replace traditional virtual machines, config management, and network and storage virtualization in the image of the Google model of massive scale workload scheduling. That means it is a whole new world to learn but you can’t quite forget about the old world yet, and need to map between the two. Is it complicated? Yeah, mainly because it’s evolving so fast and serving many audiences. It will get easier. Simpler alternatives? Any PaaS like Heroku or Google App Engine or Cloud Foundry. Or a serverless function platform like AWS lambda (not sure if it’s “simpler” but it is getting there) Disclaimer I work for Pivotal who sells cloud foundry and a distro of Kubernetes
- aphexairlines 8y ago
  AWS Lambda seems much simpler. Bundle your code in a zip file. Describe it in a yaml template with the name, runtime, zip filename, memory needed, entry point in your code, and external HTTP path. Deploy it with "aws cloudformation package" and "aws cloudformation deploy". The platform will give you load balancing, scaling, logging, metrics, internal and external endpoints, DNS, and CDN. The PaaS platforms you mentioned are basically the same (you provide code + a bit of config and they do the rest), except priced by the hour.

streblo 8y ago

There's a lot of FUD in the comments here about kubernetes, so let me chime in with a success story -- my company has been using kubernetes to operate a reasonably large and complex set of services (>100 nodes/pods, >5B requests/day) and we're doing it with only 3 engineers. Our experience has been great, and if I had to do it again, I'd happily choose k8s without reservation.

timeSl 8y ago
Can I ask if you have a bare metal cluster or you do this through a cloud service?
- streblo 8y ago
  We are using Google Kubernetes Engine

alexk 8y ago

I am writing this for folks who are reading this thread and wondering what's of said here is true and what's not true. * Google conspiracy theories - (Google is doing it to lock everyone in). Not true. Kubernetes was created as a best-practices fully open source Borg, the very reason it's using Docker and Etcd is the desire of Googlers to work with OSS community and it paid back by Red Hat and later CoreOs and others joining the project. Kubernetes is complex, yes, but so is Mesos and OpenStack, and pretty much any other production grade orchestration system I've seen, so I would disregard this argument as well. * Google is not even using Kubernetes. Google is a big company, they can't switch to Kubernetes overnight or even in 5 years. I don't have this info, but I'm pretty sure many teams are using GKE and the internal adoption grows. * What if Google goes away? Everyone will get locked in! Google is not the only major contributor to Kubernetes core these days. Red Hat is another one, and many others, so this lock in point is not valid. Google is not even BDFL - the project is governed by CNCF (OSS foundation similar to Linux foundation) and the project was donated by Google several years ago. Kubernetes development process is organized in the most fascinating open process I have ever seen - SIGs (special interest groups) are fully open and anyone can participate and help develop the project. I have learned a lot about openness just looking at the organisation of the dev process. * You don't need Kubernetes if you have one service. Not true. You can benefit from Kubernetes even if you have just one container. It solves many problems you would have to solve otherwise - service discovery, configuration management, load balancing, secrets management, fail-over, routing, publishing new releases and packaging and many others. You probably don't want to self-host Kubernetes, because it is complex unless you have experience and desire to do so, that point is valid. But thankfully, you can use GKE, AKS or EKS nowadays. Disclaimer: Our company works with Kubernetes so I am biased, but we have no affiliation with Red Hat or Google.

jacques_chester 8y ago
> Google conspiracy theories - (Google is doing it to lock everyone in). My own conspiracy theory is slightly different: Google's aim is to scorch the earth so that AWS cannot form a locked-in position. I don't think they immediately saw this possibility, but that they moved swiftly and energetically to support the growth of Kubernetes once they did see it. The alternatives for Google were all strategically painful. ECS might have become successful, meaning a loss for Google. Mesos or Docker Swarm might have taken off, leading to snap acquisitions of the relevant companies by Microsoft or AWS, again a downside for Google. As the plurality contributor to the commoditising winner, Google is able to prevent the other two cloud giants from pulling ahead on container orchestration. Disclosure: I work for Pivotal, we have a Kubernetes-based product (PKS) we co-develop with Google and VMWare.

nstart 8y ago

Maybe off topic. But before Kubernetes AND docker, how did people manage microservices? Was it a machine stack per service? Were there tools that helped keep all the services running together in a single machine? I haven't been in software for long enough to know what this past looked like so courious if any of the HN community might know more about this especially at a personal experience level.

firebacon 8y ago
> Were there tools that helped keep all the services running together in a single machine? Traditionally, services on unix system are started and supervised directly by the init process ("init system"). That has been SysV init for most of the time, but in the last decade we got two major new systems: upstart (dead now) and the infamous systemd. Using systemd can still be a very good alternative to using docker today. > Was it a machine stack per service? That still sounds like a good approach to me today. If your service is not big/expensive enough to need at least one full machine, why bother cutting your app up into such small services in the first place? For example, for web applications, a traditional setup would have been to have a couple of dedicated servers that run the database and a larger pool of webservers that serve the (stateless) application. In front of that you could have two linux boxes doing the network load balancing and failover or a commercial load balancer "box" product.
wmf 8y ago
Here's Netflix from 2011 (pre-Docker): https://www.slideshare.net/adrianco/global-netflix-platform/19-Ne9lix_Global_PaaS_Architecture_Features https://www.slideshare.net/adrianco/global-netflix-platform/... This was widely considered the gold standard in cloud native architecture. Twitter from 2013: https://www.slideshare.net/caniszczyk/twitter-opensourcestacklinuxcon2013 https://www.slideshare.net/caniszczyk/twitter-opensourcestac... They had a similar in-process JVM-based approach as Netflix, but they also used Mesos as a scheduler.
jacques_chester 8y ago
> But before Kubernetes AND docker, how did people manage microservices? Cloud Foundry and other opensource PaaSes. Heroku. Chef, Puppet et al. Java app servers with WARs. Before that was a time of heroic mythology. Shell scripts, bailing wire and raw genius. Disclosure: I work for Pivotal, we work on Cloud Foundry.

throw2016 8y ago

Devops has added a web of complexity that distance users from the underlying technology. Suddenly mounting filesystems is a 'storage driver', using networking technology built into the kernel is a 'network driver' or using an Nginx reverse proxy or Haproxy load balancer is an 'ingress controller'. This new vocabulary confuses rather than informs and ends up adding more layers of complexity. What happens when things break? Simply knowing the json/yaml layers above is not going to help without an understanding of networking and the underlying technologies. Facebook, Google, Netflix are not 2 engineers running devops, they all have unique architectures and an army of experts to run their infrastructure. The idea that you can be 'webscale' without ops experts is complete fantasy.