16 ms·

Maybe You Don't Need Kubernetes (2019)

281 points by WolfOliver 7y ago

nomadnomad 7y ago
Nomad is awesome. Just like the original author, our tiny team went with nomad for on-prem container orchestration instead of k8s. Since then we haven't spent much time on any nomad/consul maintenance, and all that time we got back - we spent on developing the product. Basically, I agree with others that k8s is both complex and not fully mature yet.
fermigier 7y ago
Previous discussion: https://news.ycombinator.com/item?id=19467067 https://news.ycombinator.com/item?id=19467067 (315 comments).
- alpb 7y ago
  Also https://hn.algolia.com/?q=don%27t+need+Kubernetes+ https://hn.algolia.com/?q=don%27t+need+Kubernetes+
pjmlp 7y ago
Yep, still doing plain old VMs. Just like NoSQL, BigData,..., in a couple of years k8s will be slowly forgotten as everyone updates their CVs to the Next Big Thing™.
- alharith 7y ago
  If you've forgotten about NoSQL or BigData, chances are your shop really just overshot how much data and traffic they have.
  - geofft 7y ago
    The vast majority of shops have less data than the NoSQL vendors wanted to convince them they had. (With the ability to handle big data, some shops have started creating huge datasets e.g. as part of the targeted advertising arms race, but it's not clear they really need to.)
  - flir 7y ago
    Scale: the problem (almost) nobody has, and (almost) everybody wants.
    - skrowl 7y ago
      Devs working at the vast majority of shops: Easily handle production load on 3-5 servers / VMs while reading all about how to script quickly spinning up 1,000 VMs if they really needed to.
      - flir 7y ago
        My servers are named after my pets and I'm not ashamed to admit it!
    - yashap 7y ago
      There’s different kinds of scale. I work for a company that has “only” hundreds of thousands of daily active users, but we ingest and persist a lot of data for them, on the order of millions of events per day for some of the larger users. We definitely run into big data problems in select services, despite not having a crazy number of users. Many companies are similar.
  - snaky 7y ago
    One modern PC server with PostgreSQL can handle about 1 mln transactions per second. http://akorotkov.github.io/blog/2016/05/09/scalability-towards-millions-tps/ http://akorotkov.github.io/blog/2016/05/09/scalability-towar...
    - james_s_tayler 7y ago
      In the comments the guy says the hardware the benchmark was running on had 3TB of RAM. That's not average hardware. It's a couple hundred connections max running select statements as fast as possible on the best hardware they could afford to test against. That doesn't translate into your CRUD app handling a million requests per second.
- LoSboccacc 7y ago
  idk, there's some value in kubernet way beyond instancing vms, it provides a nice system that ties beautifully into configuration management in a way that just vm, dockerfiles or even good old puppet scripts can't
- msla 7y ago
  Some technologies come and go. Williams-Kilburn tubes, for example, are now fully and finally obsolete. Ditto mercury delay line memories. Other technologies merely expand and shrink. Big Data went from being a buzzword to a simple, established fact... albeit established in places you don't work. Does Google have Big Data? Does Amazon have Big Data? Do most of us work at either of those companies? Big Data isn't obsolete in the same way dump trucks aren't obsolete.
- twic 7y ago
  Never bought into the VM fad myself. The physical hardware is chugging along quite nicely over here.
  - aww_dang 7y ago
    Agreed. At this point HN could have a post about the merits of plain old dedicated servers and it would seem novel.
    - hotsauceror 7y ago
      Just wait. In ten more years IBM and Deloitte will re-brand fat clients and on-prem infrastructure as something else and they'll start selling it to everyone. Replace "mainframe" and "dumb terminal" with "the cloud" and "web app" and we had this same conversation 40 years ago. Then in another 20 years we'll swing back to some version of consolidated+remote servers and lightweight client access portals.
  - dvfjsdhgfv 7y ago
    Depends on your use case. VMs give you more flexibility and a reasonable level of isolation that is needed in some scenarios.
  - freehunter 7y ago
    Owning physical hardware is a fad. I rent big iron mainframes from IBM and pay their consulting fees to punch new cards every time I want to modify a program.
  - deleted 7y ago
    [deleted]
  - sickygnar 7y ago
    Wondering if things will swing back to physical hardware with new chips like the amd epyc rome... 64 core ridiculous i/o on a single chip. that, compounded with the fact that people seem to be gravitating back towards compiled and higher performance languages. could run a medium traffic site on one machine. hello mainframe era 2.0
- avip 7y ago
  Just like NoSql, BigData, renewables, and the Fax machine.
- bob1029 7y ago
  Same deal over here. We wrote a little bit of tooling over the last few years to help automate various tasks regarding getting our code into production. We have found that minimal touch is the best approach when it comes to infrastructure and tooling around it. The core of our business value is ultimately in our codebase. The infrastructure only serves to host it to our end customers, so we don't like to spend too much time on it. By keeping things very simple, we can hop between instances and even cloud vendors without much headache. We built self-contained deployments at the code level, not at the infrastructure-level (I.e. docker). How many businesses (aside from hyperscalers) derive their principal business value from infrastructure itself? Is it worth spending all this time and frustration on various competing declarative infrastructure approaches? Surely with the amount of time it takes to get all this mess set up, someone in your organization could have purpose-built something from zero that accomplishes the same or better for your specific business concerns.
- rjkennedy98 7y ago
  NoSQL is definitely not obsolete. When you go to Amazon its not looking in PostgreSQL for your shopping cart, its looking in Dynamo. As someone who previously worked in Fortune 10 company, I can tell you that most production APIs were being driven off of NoSQL or were being migrated to NoSQL (specifically Cassandra and ElasicSearch).
  - pjmlp 7y ago
    You said it quite clearly, Fortune 10, which rules out 99% of all companies out there.
    - snaky 7y ago
      And 5 if not 9 out of that Fortune 10' NoSQL dbs were just tied with a duck tape ('Our frontend team needs something they could understand - Ah okay then') to the mainframe with a previous generation of NoSQL - like ADABAS, IMS or z/TPF - and a ton of COBOL code where all the work is done since forever.
  - SahAssar 7y ago
    Not for OLTP I assume? Or are they using transactional NoSQL?
- hello_moto 7y ago
  Kubernetes is akin to writing unit-test: plenty naysayers due to the fact that the benefit requires the user to use it everyday to be part of their dev/working cycle. k8s won't go away and this comes from a dev who hates infrastructure.
- streblo 7y ago
  They've been "forgotten" because they've become so mainstream that they're now part of the norm.
  - nailer 7y ago
    Containers are a mild improvement over existing technology. Nobody cares whether your cloud provider uses containers, older tech, or newer tech.
    - nailer 7y ago
      Clarifying above: VMs, containers, and MicroVMs are all a box to run code and some surrounding orchestration mechanisms to standardize and make configuration portable. They're all part of the same generation of tech - portable instances. This is a leap over having physical servers, but a mild improvement over each other. Think git vs mercurial - the difference isn't significant enough to move from one to the other. Serverless, for example, would be a leap over the instance model.
- t0astbread 7y ago
  Why not containers? They seem more efficient than VMs. ("Process with restrictions" vs "full operating system and kernel with repeated context switching".)
  - cookiecaper 7y ago
    "Not containers" for lots of reasons, depending on what you mean by "container". Docker is a crime against the computing industry. Other systems have implemented containers more competently, both within Linux (LXC et al) and without (FreeBSD jails), but per usual, without the cute mascot and the bonfire of VC money, these things go unheeded by the hype cycle. Still, we have enough security issues with traditional virtualization, which only worsen every year as new side channel attacks are discovered. You'll find that container platforms like Fargate actually run everything in its own micro-VM because there's no other way for them to provide the basic isolation guarantees they peddle.
    - t0astbread 7y ago
      I wouldn't run untrusted code in a container either. I'm talking about the case where you control and trust the software. Also, what do you mean by "Docker is a crime against the computer industry"?
- overgard 7y ago
  Can't tell if you're joking, but it's not like NoSQL, Big Data, etc, went anywhere. The hype died down but the uptake is probably even bigger now.
chrismmay 7y ago
There are lots of options out there. I'm currently exploring Cloud Foundry for our platform, since our core API is written in Spring Boot. This page from AquaSec has lots of links to articles comparing Cloud Foundry to Kubernetes. https://www.aquasec.com/wiki/display/containers/Kubernetes+vs+Cloud+Foundry https://www.aquasec.com/wiki/display/containers/Kubernetes+v... I also found this book to be helpful https://www.amazon.com/gp/product/B07T1Y2JRJ/ref=ppx_yo_dt_b_d_asin_title_o00?ie=UTF8&psc=1 https://www.amazon.com/gp/product/B07T1Y2JRJ/ref=ppx_yo_dt_b...
siffland 7y ago
A good friend and old coworker has been hitting me up for how to do stuff with docker since his job now uses containers. He has about 15 containers (on 2 hosts) that do not change to much and was asking about setting up k8s (it was a buzzword his manager heard). i talked him into just setting up swarm. It took all of about an hour (over text messaging) to get it set up and all stacks/services running. They could not be happier. It comes down to the right tool for the job. If you don't need all the bells and whistles, then keep it stupid simple. I realize swarm is not a 100% "enterprise" solution, however before they they were just issuing docker command after each reboot.
- seneca 7y ago
  > i talked him into just setting up swarm. Not contesting the heart of your comment but, given the current state of Docker, recommending Swarm to someone strikes me as bad advice. Nomad may be a better call. Mirantis has openly said Swarm's future is a two year sunset with a transition path to k8s.
  - deleted 7y ago
    [deleted]
  - LoSboccacc 7y ago
    I agree, looked into swarm and routing was a mess, even discounting the bugs, lockups and moments of total disconnections, you had to rely on ugly dns hacks to know the live endpoints, making any kind of stateful service a nightmare to set up, while kubernet just give you an endpoint api, and even then, there were no guarantees for local services that swarm mesh would route the calls to the local instances, while in kubernet you can control precisely how services are resolved grouping them in pods, so that they don't needlessly clog the pipes. I find very hard to find swarm use case that a wan, vpn, private segment or kubernet cluster can't handle better.
    - netsectoday 7y ago
      I've been using Docker Swarm in production for about 2 years now... processing about 1TB of data / month across 30+ containers. The networking and routing has been rock solid except for that one day that the Docker dev team, in one release, accidentally added random hashes to the internal DNS names of services. Ever since that day I've used the docker-compose network alias for internal routing https://docs.docker.com/compose/compose-file/#aliases https://docs.docker.com/compose/compose-file/#aliases Discovering bugs in a technology you just started "looking into" actually sounds like the learning curve.
stevefan1999 7y ago
Agreed. k8s is too bloated at this point, yet alternatives like nomad and swarm are missing some fundamental features, so we have had to adopt to k8s unfortunately. For example, swarm still have no fault tolerance and nomad relies on Vault, another product from Hashicorp and is also in the same limited state wrt. documentation
- carlsborg 7y ago
  Try ECS on AWS. Rock solid and reasonably simple and does everything you need.
  - anaganisk 7y ago
    I dont think, the correct answer is to change your stack to fit some vendor
    - carlsborg 7y ago
      Its just docker, your containers remain exactly the same only the orchestration changes!
      - tekkk 7y ago
        You are pushing ECS quite hard, but why so? It's nothing particularly amazing in my opinion, good for what it does but requires all your other infra too to reside in AWS. Which is not that cheap if you're really concerned with cost. Eg NAT is 5 cents per hour and you need at least two of them if you want to use private subnets.
  - xur17 7y ago
    Maybe ECS has gotten better, but when I used it a few years back it had a lot of rough edges, and wasn't really "simple" to use. On top of that, you're locking yourself into a single provider. A managed k8s service (all 3 big providers offer this) really isn't that much more complex, and has much better documentation / no vendor lock-in.
dustinmoris 7y ago
All these "maybe you don't need this or that X" posts die in an instant when the user already knows how to do X (when the learning curve argument is gone). Let's get it right: Kubernetes is really really cheap. I can run 20 low volume apps in a kubes cluster with a single VM. This is cheaper than any other hosting solution in the cloud if you want the same level of stability and isolation. It's even cheaper when you need something like a Redis cache. If my cache goes down and the container needs to be spun up again then it's not a big issue, so for cheap projects I can even save more cost by running some infra like a Redis instance as a container too. Nothing beats that. It gets even better, I can run my services in different namespaces, and have different environments (dev/staging/etc.) isolated from each other and still running on the same amount of VMs. When you caculate the total cost saving here to traditional deployments it's just ridiculously cheap. Kubernetes makes deployments really easy. docker build + kubectl apply. That's literally it. Deployments are two commands and it's live, running in the cloud. It's elastic, it can scale, etc. Kubernetes requires very little maintenance. Kubernetes takes care of itself. A container crashes? Kubes will bring it up. Do I want to roll out a new version? Kubes will do a rolling update on its own. I am running apps in kubes and for almost 2 years I haven't looked at my cluster or vms. They just run. Once every 6 months I log into my console and see that I can upgrade a few nodes. I just click ok and everything happens automatically with zero downtime. I mean yes, theoretically nothing needs Kubernetes, because the internet was the same before we had Kubernetes, so it's certainly not needed, but it makes life a lot easier. Especially as a cheap lazy developer who doesn't want to spend time on any ops Kubernetes is really the best option out there next to serverless. If learning Kubernetes is the reason why it's "not needed" then nothing is needed. Why use a new programming language? Why use a new db technology? Why use anything except HTML 4 + PHP, right? BTW, learning Kubernetes can be done in a few days.
- brodouevencode 7y ago
  Can second this. For most cases it's very low touch. When things break it has always been because I deployed bad code or configs. My weekends have freed up.
- carlsborg 7y ago
  Your single VM is a single point of failure. You probably want to run this in 3 VMs, one in each data center. ECS gives this to you out of the box, rolling deployments and health checks included.
nautilus12 7y ago
I get really sick of these types of posts because it helps to fuel a weird contrarian and/or fear based avoidance of these technologies. Then the company goes down a rabbit hole of inventing their own thing that basically is an in house version of kubernetes but worse. I think kubernetes is pretty easy, I'm not sure why companies are so against it other than wanting to be "even cooler than the cool kids".
- metadaemon 7y ago
  Most likely they themselves adopted a previous "cool" technology that later bit them after dying off, or being realized as just another fad. I think that apprehension of a new technology is a healthy, pragmatic response in most cases.
- superkuh 7y ago
  To start: not everyone is a company. Not everything is about companies. When people bringing overly complex, mostly useless in this context, tech like kubernetes to a personal environment it's silly. But whatever, it's what they know. But when they start telling other people to learn kubernetes because it's so useful then it's annoying. And there are lots of people advocating for using kubernetes on small personal projects. When they get pressed they all fall back to the justification, "You should use kubernetes in personal projects to learn kubernetes." which is back to silly again. That's where these kinds of critical articles come from.
jaequery 7y ago
docker compose and a jwilder proxy is all I need for most of my projects
draw_down 7y ago
Wow, people using Kubernetes must not have any idea what they’re doing! The absolute fools.
Vosporos 7y ago
no fucking joke
Thaxll 7y ago
It's funny we use GKE and coudn't be more happy, once your CI/CD is setup you don't have to think about it and makes your developper life much much easier. I don't even think for one second to go back to uploading tar.gz or DEB / RPM using scripts / puppets ect ...
schmichael 7y ago
This post should probably have (2019) in the title. I'm the Nomad Team Lead and can try to answer any questions. Since this post was made the team has expanded, and the task/job restarting issue they link has been (mostly) addressed. Also new since this post is our Consul Connect integration which can accomplish similar goals to k8s network policies, albeit opt in and with the actual discovery/networking code living in Consul/Envoy respectively.
- scarface74 7y ago
  I absolutely loved Nomad when I used it. We were stuck with both Windows, legacy apps and command line scripts and no time to learn Docker or anything like K8s. We were already using Consul so bringing in Nomad to schedule everything - batch files, .Net executables, some legacy programs, etc was a godsend. Unfortunately, as much as I loved the Nomad+Consul combination, I really couldn’t suggest it today. It is so much easier to find qualified K8s people than Nomad+Consul people I couldn’t in good conscience recommend it. But this is all a moot point to me. While if I were leading another on prem project I would use K8s. We are all in on AWS+ECS+Fargate where I work now and we really don’t care about the lock in boogie man. Given a choice, I would still say at least if you’re on AWS, use the native offerings. The value of hypothetically being able to migrate a large infrastructure “seamlessly” is vastly overrated.
  - schmichael 7y ago
    > I absolutely loved Nomad when I used it. Glad to hear it! > Unfortunately, as much as I loved the Nomad+Consul combination, I really couldn’t suggest it today. This is a fair critique and a problem any project living in a world defined by a strong incumbent suffers. You made me realize we need resources to help k8s users translate their existing knowledge to Nomad as many people looking at Nomad will have k8s experience these days. So thanks for this comment. Maybe with the right docs/resources we can at least minimize the very real cost of using a non-dominant tool. > But this is all a moot point to me. We are all in on AWS+ECS+Fargate where I work now and we really don’t care about the lock in boogie man. This was me in a past lives/jobs! HashiCorp's entire product line (with the exception of maybe Packer and Vagrant) become much more compelling for multi-cloud, hybrid-cloud, and on-prem orgs.
Eikon 7y ago
I know it's an unpopular opinion but I don't see the point of kubernetes. It's a complexity monster which is not providing any real advantage over way simpler solutions. Even pod to pod communication, which would be trivial to do using any sane solution is a huge pain in kubernetes. I would actually be frightened by running kubernetes in production. The simplest things are so hard to do that I don't even want to think about how to fix a weird issue when something goes wrong...
- q3k 7y ago
  > I know it's an unpopular opinion but I don't see the point of kubernetes. It's a complexity monster which is not providing any real advantage over way simpler solutions. The main selling point of k8s over any other tool is that it provides a single, unified, secured, multi-tenant API endpoint for managing all of production. Your developers updating their production app use the same API as a CI system that wants to spawn some worker containers, and the same API as an operator service maintaining a Redis cluster. All of this results in a single view into production. If things go well, the end result is that you swap daily interaction with a handful of different tools with disparate states (Terraform, Ansible/Chef/Salt/Puppet, shell scripts and proprietary tools) into just managing payloads on k8s. > Even pod to pod communication, which would be trivial to do using any sane solution is a huge pain in kubernetes. How is it a pain? A pod behind a service provides a DNS name that allows running requests to it - this handles the bulk of production traffic. If you want to contact a particular pod that is not behind a service just use the k8s API to retrieve details about it (like Prometheus does via k8s pod service discovery).
- emerongi 7y ago
  > Even pod to pod communication, which would be trivial to do using any sane solution is a huge pain in kubernetes. You create a service and then use the DNS name? How is that hard?
- EdwardDiego 7y ago
  The killer feature for us is autoscaling. Allows us to easily adapt to changing load throughout the day without having to overprovision. The ability to scale on custom metrics is really nice.
segmondy 7y ago
If you don't need container orchestration, you don't need kubernetes. If you need reliability and instead of using k8s, orchestrate your container manually you have issues.
deleted 7y ago
[deleted]
puhi 7y ago
Kubernetes is here to stay. What will change or what will be enhanced are: - Minimum requirements to actually run it (see k3s) - More managed services (gke, azure and aws exists but also digitalocean) - More/better handling of stateful services - Simple solution for write once read many (relevant for caching and for ci/cd) At that pace we are already with such a jung project, yeah this is great. This is huge. And no one needs to migrate already to kubernetes! But it already does a few things out of the box which reduces the complexity: - easy cert management - internal loadbalancing - autoscaling - green/blue deployment - deployment But you do see how the industry is struggling with certain problems: We are now with kubernetes moving into a cloud native area. Everyone know has kubernetes available. There was no mesos managed service from google, azure and aws. There was no docker swarm from google, azure and aws.
- nailer 7y ago
  > But you do see how the industry is struggling with certain problems: We are now with kubernetes moving into a cloud native area. We were were moving into a cloud native era way before Kubernetes.
nailer 7y ago
Of course most software companies (ie, where most of HN work) don't need Kubernetes. At this point it's become a meme for wasted engineering. Are you creating a cloud provider? - If you are creating a cloud provider, then yes you might want Kubernetes. If you are making your own cloud provider, you should question why you are doing that. - If you're not creating a cloud provider (for example, you're a software company), then use whatever VM / container / MicroVM / etc your cloud provider gives you rather than layering your own unnecessary complexity that adds questionable value on top of what your pay for from your cloud provider.
- UK-Al05 7y ago
  Normally cloud providers provide k8s to their customers.
  - nailer 7y ago
    Yes, that's the biggest use of Kubernetes. Most cloud providers provide Xen VMs, containers (via whatever orchestration mechanism) or MicroVMs. Most people making software shouldn't care which. It's a box to run code.
Fiahil 7y ago
I'm usually fairly buzzword-adverse, and I didn't follow the NoSQL/Spark/Big-Data phase when it was hip. I was also hostile to kubernetes until I had to work with it against will for a project. Since then, I completely changed my mind about kubernetes. This is a very good technology for only one reason -and it's NOT about container orchestration-. Portability. K8s is the missing piece that allows you to create a network of cooperating computers independently of hardware, OS, and even architecture. If I have an home-made k8s cluster on my raspberry pies at home, it's not because it's lightweight and easy to manage -k8s adds a significant overhead-, it's because it gives me the ability to unplug one of them, take the sdcard, format it and plug another board without any interruption or configuration. I could plug my intel laptop to that network and have some pods running on it without having to change a single line in my configuration. Finally, I can zip a folder and email a bunch of yaml files to my friends (or have them git clone the repo), and they will be able to replicate an exact copy of my home cluster with all -or some- of the services. This is truly amazing.
- pnako 7y ago
  I literally don't understand what you are achieving with your k8s raspberry cluster thing.
  - optimuspaul 7y ago
    automatic failover obviously
frou_dh 7y ago
I heard the word Kubernetes one too many times from listening to podcasts put out by The Changelog. The overhype bit got flipped in my head and now I'm predisposed against it. Marketing kills!
jugg1es 7y ago
I wish this article was titled 'Nomad is pretty great' since that was really the point being made here. Nomad is not a true alternative to Kubernetes. It does not support autoscaling out of the box, it does not have most of the bells/whistles that Kubernetes has. However, it is just a single executable and it just works. Combined with Consul and Fabio, you can run a container orchestration cluster with very little fuss that has service discovery, internal load balancing and cluster-wide logging. It is a feature-rich task scheduler with a pretty good CLI. I highly recommend it, but if you need stuff like autoscaling, you need to use something else.
- schmichael 7y ago
  (Full disclosure: Nomad Team Lead here, so I'm biased!) > Nomad is not a true alternative to Kubernetes. You're right! We explicitly try not to be a standalone dropin replacement for k8s. Our comparison page goes into this a little bit (but now I realize it's in need of a refresh!): https://www.nomadproject.io/intro/vs/kubernetes.html https://www.nomadproject.io/intro/vs/kubernetes.html - Nomad relies on Consul for service discovery - Nomad relies on Consul Connect (Envoy) for network policies - Nomad relies on Vault for secrets - Nomad relies on third party autoscalers - https://github.com/jippi/awesome-nomad#auto-scalers https://github.com/jippi/awesome-nomad#auto-scalers - although there's more we can do to enable them. - Nomad relies on Consul Connect (Envoy) or other third parties for load balancing - Nomad does not provide any cluster log aggregation (although logging plugins are planned which should make third party log aggregators easier to use) Nomad still has many missing features such as CSI (coming in 0.11!), logging plugins, audit logging, etc, but we never intend to be as monolithic a solution as k8s. We always hope to feel like "just a scheduler" that you compose with orthogonal tools.
moshloop 7y ago
I think this post misses the point of where kubernetes brings value the most: multi-team environments. If you are a single-team, then nomad, non-distributed docker, systemd units, etc is probably a better choice today due to the operational complexity of running Kubernetes (GKE,AKS,EKS do lower the complexity, but you still need kubernetes experts on call) However if you are in an organization with multiple teams (as are the vast majority of developers) then Kubernetes provides a common language for deploying, operating and securing your applications which enables you to go from a process which could takes days, weeks or even months to provision and configure a VM to minutes or hours to provision a kubernetes namespace.
vi-mode 7y ago
Before I started with k8s, I've been checking Nomad many times but eventually went with k8s. Nomad's learning curve didn't seem to be better, docs felt sparse and the much bigger ecosystem with k8s shouldn't be undervalued.
candiddevmike 7y ago
Anecdotally, about the worst place I've seen Kubernetes used is for CI/CD pipelines where every pipeline gets a fresh cluster. It's truly a special kind of hell--it takes at least 10 minutes to get a cluster up and running (depending on cloud provider), and it causes all kinds of indeterminism and frustration. You wait for a cluster to be spun up, wait for all of the extra services to be added, have one of them fail (completely unrelated to your PR), and have to do the whole thing over again. It's truly the new version of the "compiling" XKCD. If you ever want to see how brittle and bizarre Kubernetes can be, use it in a 100% automated fashion and hope for the best.
vicjicama 7y ago
I am very happy working with Kubernetes in a very small team, I really like the abstraction of the different entities and the declarative ways to configure them. I think that the learning curve is not that steep if you already had to do the same than with Kubernetes with other alternatives, in my on experience I have been discovering a lot of features that are very helpful not just during production but also during the development environments that were a real pain before. I am have a lab/cluster/blog running on Kubernetes, I am in charge of this one alone, it is opensource [1], I version everything that goes to the cluster so you can see the evolution of the kubernetes entities, the config, the containers and the code. I started this from scratch and improving it feature to feature, I think that this might be a big factor with my positive experience with Kubernetes. I wonder if an issue with adopting kubernetes is to try to migrate a big system into kubernetes in a very limited time lapse and trying to push/force features as they were working/handled in the previous approach? [1]: https://github.com/vicjicaman/microservice-realm https://github.com/vicjicaman/microservice-realm
flowerlad 7y ago
I am a one-person team running Kubernetes on Google Cloud. It costs around $62 per month before tax. (One n1-standard-1 node for $35.34 per month, HTTP Load Balancing $18.60 per month and SSD persistent disks for $8.50 per month). Kubernetes gives me the ability to scale up quickly when the need comes (which is soon, hopefully). I evaluated AWS Fargate & Kubernetes, Azure and GKE before settling on GKE. Amazon charges $148 per month just for cluster management alone. Google and Azure charge $0 for cluster management. This ruled out AWS for me. AWS appears to be a reluctant adopter of Kubernetes. They seem to want you to use Fargate instead. I tried it and found it to be crap--very hard to get things running. I was able to get Kubernetes running on Azure and GKE fairly easily. There were minor hiccups on both those clouds. With Azure initial creation of a AKS cluster failed because some of the resource providers weren't "registered" on my subscription. On GKE it was hard to get ingress working. Static IPs take a long time to take effect, and in the mean time you are fiddling with your yaml files trying to figure out what you may have done wrong, not realizing it is a GKE issue. The awesome part of Kubernetes is that I didn't need to learn almost anything about Google cloud to get everything working. I only had to learn Kubernetes. If Google raises prices I can easily switch to Azure without learning any Azure technologies. My knowledge as well as my application is completely portable. I can't imagine doing any of this as a one-person team without Kubernetes.
- mpmpmpmp 7y ago
  Second the comment about ingress. By the way, if you have any idea how to get ingress to work without taking down the services that are in it while it updates that would be amazing. I think it would obviously work if we used a separate ingress/load balancer for each host, but that seemed kinda wasteful since we are ok with doing downtime in off hours scheduling with our project.
  - nouveaux 7y ago
    I believe the Kubernetes/Docker way is to not update in place but to create new instances. Can you spin up a new node pool/cluster and redirect traffic there?
- gum_ina_package 7y ago
overgard 7y ago
I agree with the sentiment, but I think the hardest part of kubernetes is dealing the complexity of the resulting cloud. You end up having to manage that complexity no matter what you're doing; the upfront cost of figuring out kubernetes is that maybe it takes a couple days longer to learn than something else, but now you have a huge ecosystem of technology and people that are solving the same problems you have.
gfodor 7y ago
Personally, I've found Chef Habitat to be a sane alternative. I don't have any experience with k8s, but if you are not using containers especially, Habitat gets you: - packaging - deploys - configuration management - supervision - service discovery - isolation in a more classic "VM" environment, using unix jails. For anyone who isn't interested in learning k8s, but also wants a relatively modern, all-in-one solution to these problems, I recommend checking it out.
spicyramen 7y ago
I see Kubernetes also being used for ML workflows, yes...while you haven't figured out how to run a ML pipeline yet, add K8s (KubeFlow recommendation) As a ML researcher solutions like KubeFlow has been very painful as I don't want to learn k8s, I need to continue focusing on ML work and figure out my pipeline before I deploy K8s
ehutch79 7y ago
What are you talking about, my blog that has an astounding 2 visitors a month (one or both of which are accidental clicks) definitely needs to be on kubernetes!
sgt 7y ago
I think most Nomad users are well aware that Kubernetes can do all it can - and vastly more. If anything is wrong with Kubernetes, it would be the complexity of it and that it has a steep learning curve. It seems it's best to have a small team of people to manage it and to solve solutions to problems that arise. We started using Nomad last year and one thing I can say is that it's relatively easy to use and works well for its intended purpose, especially for us hybrid-cloud folks.
irq 7y ago
Given all the talk here about Kubernetes’ learning curve, does anyone have any resources for learning it for those that haven’t yet? Whether blogs, books, video courses, etc.
jordanpg 7y ago
The part about all of this containerization technology I don't understand is that it was originally sold as a way (among other benefits) to unburden devs of having to think about ops stuff and just code. Well, my experience with Docker, k8s, and related technologies, is that I now need to be an expert with these things just to get through my day. It's exhausting.
errantspark 7y ago
I feel like a lot of the problems K8s solves are fake problems that we have because all our software links/imports so many libraries/packages it's impossible to tell what's going on. It feels like have built an entire infrastructure around hopefully capturing that one time we got it working and being able to reboot to that state when things go wrong.
- koheripbal 7y ago
  Reminds me of the excitement around Java Beans
- erulabs 7y ago
  You’re not wrong but containerization is pragmatic - not idealistic. You can still write simple, clean software packaged in a SCRATCH container - but you can also isolate whatever crap you have to work with the get the job done. Both the elegant app and the horrible one now share a reliable init system.
luord 7y ago
It's really odd that around 30% of the people who commented defending kubernetes are merely echoing stuff that the article itself mentioned. Also, it's weird that the argument of another 30% of people defending kubernetes boils down to: "Using kubernetes is really easy, just hire Google/Azure/etc to do it for you." Can't begin to wrap my head around that one. But what do I know, I prefer to KISS and I like nomad. In fact, I'd be using swarm if its future wasn't spotty.
kresten 7y ago
I built a big globally scalable system without Kubernetes or containers. It’s very easy to understand and is highly reliable.

281 points by WolfOliver 7y ago

Nomad is awesome. Just like the original author, our tiny team went with nomad for on-prem container orchestration instead of k8s. Since then we haven't spent much time on any nomad/consul maintenance, and all that time we got back - we spent on developing the product. Basically, I agree with others that k8s is both complex and not fully mature yet.

fermigier 7y ago

Previous discussion: https://news.ycombinator.com/item?id=19467067 https://news.ycombinator.com/item?id=19467067 (315 comments).

alpb 7y ago
Also https://hn.algolia.com/?q=don%27t+need+Kubernetes+ https://hn.algolia.com/?q=don%27t+need+Kubernetes+

pjmlp 7y ago

Yep, still doing plain old VMs. Just like NoSQL, BigData,..., in a couple of years k8s will be slowly forgotten as everyone updates their CVs to the Next Big Thing™.

alharith 7y ago
If you've forgotten about NoSQL or BigData, chances are your shop really just overshot how much data and traffic they have.
- geofft 7y ago
  The vast majority of shops have less data than the NoSQL vendors wanted to convince them they had. (With the ability to handle big data, some shops have started creating huge datasets e.g. as part of the targeted advertising arms race, but it's not clear they really need to.)
- flir 7y ago
  Scale: the problem (almost) nobody has, and (almost) everybody wants.
  - skrowl 7y ago
    Devs working at the vast majority of shops: Easily handle production load on 3-5 servers / VMs while reading all about how to script quickly spinning up 1,000 VMs if they really needed to.
    - flir 7y ago
      My servers are named after my pets and I'm not ashamed to admit it!
  - yashap 7y ago
    There’s different kinds of scale. I work for a company that has “only” hundreds of thousands of daily active users, but we ingest and persist a lot of data for them, on the order of millions of events per day for some of the larger users. We definitely run into big data problems in select services, despite not having a crazy number of users. Many companies are similar.
- snaky 7y ago
  One modern PC server with PostgreSQL can handle about 1 mln transactions per second. http://akorotkov.github.io/blog/2016/05/09/scalability-towards-millions-tps/ http://akorotkov.github.io/blog/2016/05/09/scalability-towar...
  - james_s_tayler 7y ago
    In the comments the guy says the hardware the benchmark was running on had 3TB of RAM. That's not average hardware. It's a couple hundred connections max running select statements as fast as possible on the best hardware they could afford to test against. That doesn't translate into your CRUD app handling a million requests per second.
LoSboccacc 7y ago
idk, there's some value in kubernet way beyond instancing vms, it provides a nice system that ties beautifully into configuration management in a way that just vm, dockerfiles or even good old puppet scripts can't
msla 7y ago
Some technologies come and go. Williams-Kilburn tubes, for example, are now fully and finally obsolete. Ditto mercury delay line memories. Other technologies merely expand and shrink. Big Data went from being a buzzword to a simple, established fact... albeit established in places you don't work. Does Google have Big Data? Does Amazon have Big Data? Do most of us work at either of those companies? Big Data isn't obsolete in the same way dump trucks aren't obsolete.
twic 7y ago
Never bought into the VM fad myself. The physical hardware is chugging along quite nicely over here.
- aww_dang 7y ago
  Agreed. At this point HN could have a post about the merits of plain old dedicated servers and it would seem novel.
  - hotsauceror 7y ago
    Just wait. In ten more years IBM and Deloitte will re-brand fat clients and on-prem infrastructure as something else and they'll start selling it to everyone. Replace "mainframe" and "dumb terminal" with "the cloud" and "web app" and we had this same conversation 40 years ago. Then in another 20 years we'll swing back to some version of consolidated+remote servers and lightweight client access portals.
- dvfjsdhgfv 7y ago
  Depends on your use case. VMs give you more flexibility and a reasonable level of isolation that is needed in some scenarios.
- freehunter 7y ago
  Owning physical hardware is a fad. I rent big iron mainframes from IBM and pay their consulting fees to punch new cards every time I want to modify a program.
- deleted 7y ago
  [deleted]
- sickygnar 7y ago
  Wondering if things will swing back to physical hardware with new chips like the amd epyc rome... 64 core ridiculous i/o on a single chip. that, compounded with the fact that people seem to be gravitating back towards compiled and higher performance languages. could run a medium traffic site on one machine. hello mainframe era 2.0
avip 7y ago
Just like NoSql, BigData, renewables, and the Fax machine.
bob1029 7y ago
Same deal over here. We wrote a little bit of tooling over the last few years to help automate various tasks regarding getting our code into production. We have found that minimal touch is the best approach when it comes to infrastructure and tooling around it. The core of our business value is ultimately in our codebase. The infrastructure only serves to host it to our end customers, so we don't like to spend too much time on it. By keeping things very simple, we can hop between instances and even cloud vendors without much headache. We built self-contained deployments at the code level, not at the infrastructure-level (I.e. docker). How many businesses (aside from hyperscalers) derive their principal business value from infrastructure itself? Is it worth spending all this time and frustration on various competing declarative infrastructure approaches? Surely with the amount of time it takes to get all this mess set up, someone in your organization could have purpose-built something from zero that accomplishes the same or better for your specific business concerns.
rjkennedy98 7y ago
NoSQL is definitely not obsolete. When you go to Amazon its not looking in PostgreSQL for your shopping cart, its looking in Dynamo. As someone who previously worked in Fortune 10 company, I can tell you that most production APIs were being driven off of NoSQL or were being migrated to NoSQL (specifically Cassandra and ElasicSearch).
- pjmlp 7y ago
  You said it quite clearly, Fortune 10, which rules out 99% of all companies out there.
  - snaky 7y ago
    And 5 if not 9 out of that Fortune 10' NoSQL dbs were just tied with a duck tape ('Our frontend team needs something they could understand - Ah okay then') to the mainframe with a previous generation of NoSQL - like ADABAS, IMS or z/TPF - and a ton of COBOL code where all the work is done since forever.
- SahAssar 7y ago
  Not for OLTP I assume? Or are they using transactional NoSQL?
hello_moto 7y ago
Kubernetes is akin to writing unit-test: plenty naysayers due to the fact that the benefit requires the user to use it everyday to be part of their dev/working cycle. k8s won't go away and this comes from a dev who hates infrastructure.
streblo 7y ago
They've been "forgotten" because they've become so mainstream that they're now part of the norm.
- nailer 7y ago
  Containers are a mild improvement over existing technology. Nobody cares whether your cloud provider uses containers, older tech, or newer tech.
  - nailer 7y ago
    Clarifying above: VMs, containers, and MicroVMs are all a box to run code and some surrounding orchestration mechanisms to standardize and make configuration portable. They're all part of the same generation of tech - portable instances. This is a leap over having physical servers, but a mild improvement over each other. Think git vs mercurial - the difference isn't significant enough to move from one to the other. Serverless, for example, would be a leap over the instance model.
t0astbread 7y ago
Why not containers? They seem more efficient than VMs. ("Process with restrictions" vs "full operating system and kernel with repeated context switching".)
- cookiecaper 7y ago
  "Not containers" for lots of reasons, depending on what you mean by "container". Docker is a crime against the computing industry. Other systems have implemented containers more competently, both within Linux (LXC et al) and without (FreeBSD jails), but per usual, without the cute mascot and the bonfire of VC money, these things go unheeded by the hype cycle. Still, we have enough security issues with traditional virtualization, which only worsen every year as new side channel attacks are discovered. You'll find that container platforms like Fargate actually run everything in its own micro-VM because there's no other way for them to provide the basic isolation guarantees they peddle.
  - t0astbread 7y ago
    I wouldn't run untrusted code in a container either. I'm talking about the case where you control and trust the software. Also, what do you mean by "Docker is a crime against the computer industry"?
overgard 7y ago
Can't tell if you're joking, but it's not like NoSQL, Big Data, etc, went anywhere. The hype died down but the uptake is probably even bigger now.

chrismmay 7y ago

There are lots of options out there. I'm currently exploring Cloud Foundry for our platform, since our core API is written in Spring Boot. This page from AquaSec has lots of links to articles comparing Cloud Foundry to Kubernetes. https://www.aquasec.com/wiki/display/containers/Kubernetes+vs+Cloud+Foundry https://www.aquasec.com/wiki/display/containers/Kubernetes+v... I also found this book to be helpful https://www.amazon.com/gp/product/B07T1Y2JRJ/ref=ppx_yo_dt_b_d_asin_title_o00?ie=UTF8&psc=1 https://www.amazon.com/gp/product/B07T1Y2JRJ/ref=ppx_yo_dt_b...

siffland 7y ago

A good friend and old coworker has been hitting me up for how to do stuff with docker since his job now uses containers. He has about 15 containers (on 2 hosts) that do not change to much and was asking about setting up k8s (it was a buzzword his manager heard). i talked him into just setting up swarm. It took all of about an hour (over text messaging) to get it set up and all stacks/services running. They could not be happier. It comes down to the right tool for the job. If you don't need all the bells and whistles, then keep it stupid simple. I realize swarm is not a 100% "enterprise" solution, however before they they were just issuing docker command after each reboot.

seneca 7y ago
> i talked him into just setting up swarm. Not contesting the heart of your comment but, given the current state of Docker, recommending Swarm to someone strikes me as bad advice. Nomad may be a better call. Mirantis has openly said Swarm's future is a two year sunset with a transition path to k8s.
- deleted 7y ago
  [deleted]
- LoSboccacc 7y ago
  I agree, looked into swarm and routing was a mess, even discounting the bugs, lockups and moments of total disconnections, you had to rely on ugly dns hacks to know the live endpoints, making any kind of stateful service a nightmare to set up, while kubernet just give you an endpoint api, and even then, there were no guarantees for local services that swarm mesh would route the calls to the local instances, while in kubernet you can control precisely how services are resolved grouping them in pods, so that they don't needlessly clog the pipes. I find very hard to find swarm use case that a wan, vpn, private segment or kubernet cluster can't handle better.
  - netsectoday 7y ago
    I've been using Docker Swarm in production for about 2 years now... processing about 1TB of data / month across 30+ containers. The networking and routing has been rock solid except for that one day that the Docker dev team, in one release, accidentally added random hashes to the internal DNS names of services. Ever since that day I've used the docker-compose network alias for internal routing https://docs.docker.com/compose/compose-file/#aliases https://docs.docker.com/compose/compose-file/#aliases Discovering bugs in a technology you just started "looking into" actually sounds like the learning curve.

stevefan1999 7y ago

Agreed. k8s is too bloated at this point, yet alternatives like nomad and swarm are missing some fundamental features, so we have had to adopt to k8s unfortunately. For example, swarm still have no fault tolerance and nomad relies on Vault, another product from Hashicorp and is also in the same limited state wrt. documentation

carlsborg 7y ago
Try ECS on AWS. Rock solid and reasonably simple and does everything you need.
- anaganisk 7y ago
  I dont think, the correct answer is to change your stack to fit some vendor
  - carlsborg 7y ago
    Its just docker, your containers remain exactly the same only the orchestration changes!
    - tekkk 7y ago
      You are pushing ECS quite hard, but why so? It's nothing particularly amazing in my opinion, good for what it does but requires all your other infra too to reside in AWS. Which is not that cheap if you're really concerned with cost. Eg NAT is 5 cents per hour and you need at least two of them if you want to use private subnets.
- xur17 7y ago
  Maybe ECS has gotten better, but when I used it a few years back it had a lot of rough edges, and wasn't really "simple" to use. On top of that, you're locking yourself into a single provider. A managed k8s service (all 3 big providers offer this) really isn't that much more complex, and has much better documentation / no vendor lock-in.

dustinmoris 7y ago

All these "maybe you don't need this or that X" posts die in an instant when the user already knows how to do X (when the learning curve argument is gone). Let's get it right: Kubernetes is really really cheap. I can run 20 low volume apps in a kubes cluster with a single VM. This is cheaper than any other hosting solution in the cloud if you want the same level of stability and isolation. It's even cheaper when you need something like a Redis cache. If my cache goes down and the container needs to be spun up again then it's not a big issue, so for cheap projects I can even save more cost by running some infra like a Redis instance as a container too. Nothing beats that. It gets even better, I can run my services in different namespaces, and have different environments (dev/staging/etc.) isolated from each other and still running on the same amount of VMs. When you caculate the total cost saving here to traditional deployments it's just ridiculously cheap. Kubernetes makes deployments really easy. docker build + kubectl apply. That's literally it. Deployments are two commands and it's live, running in the cloud. It's elastic, it can scale, etc. Kubernetes requires very little maintenance. Kubernetes takes care of itself. A container crashes? Kubes will bring it up. Do I want to roll out a new version? Kubes will do a rolling update on its own. I am running apps in kubes and for almost 2 years I haven't looked at my cluster or vms. They just run. Once every 6 months I log into my console and see that I can upgrade a few nodes. I just click ok and everything happens automatically with zero downtime. I mean yes, theoretically nothing needs Kubernetes, because the internet was the same before we had Kubernetes, so it's certainly not needed, but it makes life a lot easier. Especially as a cheap lazy developer who doesn't want to spend time on any ops Kubernetes is really the best option out there next to serverless. If learning Kubernetes is the reason why it's "not needed" then nothing is needed. Why use a new programming language? Why use a new db technology? Why use anything except HTML 4 + PHP, right? BTW, learning Kubernetes can be done in a few days.

brodouevencode 7y ago
Can second this. For most cases it's very low touch. When things break it has always been because I deployed bad code or configs. My weekends have freed up.
carlsborg 7y ago
Your single VM is a single point of failure. You probably want to run this in 3 VMs, one in each data center. ECS gives this to you out of the box, rolling deployments and health checks included.

nautilus12 7y ago

I get really sick of these types of posts because it helps to fuel a weird contrarian and/or fear based avoidance of these technologies. Then the company goes down a rabbit hole of inventing their own thing that basically is an in house version of kubernetes but worse. I think kubernetes is pretty easy, I'm not sure why companies are so against it other than wanting to be "even cooler than the cool kids".

metadaemon 7y ago
Most likely they themselves adopted a previous "cool" technology that later bit them after dying off, or being realized as just another fad. I think that apprehension of a new technology is a healthy, pragmatic response in most cases.
superkuh 7y ago
To start: not everyone is a company. Not everything is about companies. When people bringing overly complex, mostly useless in this context, tech like kubernetes to a personal environment it's silly. But whatever, it's what they know. But when they start telling other people to learn kubernetes because it's so useful then it's annoying. And there are lots of people advocating for using kubernetes on small personal projects. When they get pressed they all fall back to the justification, "You should use kubernetes in personal projects to learn kubernetes." which is back to silly again. That's where these kinds of critical articles come from.

jaequery 7y ago

docker compose and a jwilder proxy is all I need for most of my projects

draw_down 7y ago

Wow, people using Kubernetes must not have any idea what they’re doing! The absolute fools.

Vosporos 7y ago

no fucking joke

Thaxll 7y ago

It's funny we use GKE and coudn't be more happy, once your CI/CD is setup you don't have to think about it and makes your developper life much much easier. I don't even think for one second to go back to uploading tar.gz or DEB / RPM using scripts / puppets ect ...

schmichael 7y ago

This post should probably have (2019) in the title. I'm the Nomad Team Lead and can try to answer any questions. Since this post was made the team has expanded, and the task/job restarting issue they link has been (mostly) addressed. Also new since this post is our Consul Connect integration which can accomplish similar goals to k8s network policies, albeit opt in and with the actual discovery/networking code living in Consul/Envoy respectively.

scarface74 7y ago
I absolutely loved Nomad when I used it. We were stuck with both Windows, legacy apps and command line scripts and no time to learn Docker or anything like K8s. We were already using Consul so bringing in Nomad to schedule everything - batch files, .Net executables, some legacy programs, etc was a godsend. Unfortunately, as much as I loved the Nomad+Consul combination, I really couldn’t suggest it today. It is so much easier to find qualified K8s people than Nomad+Consul people I couldn’t in good conscience recommend it. But this is all a moot point to me. While if I were leading another on prem project I would use K8s. We are all in on AWS+ECS+Fargate where I work now and we really don’t care about the lock in boogie man. Given a choice, I would still say at least if you’re on AWS, use the native offerings. The value of hypothetically being able to migrate a large infrastructure “seamlessly” is vastly overrated.
- schmichael 7y ago
  > I absolutely loved Nomad when I used it. Glad to hear it! > Unfortunately, as much as I loved the Nomad+Consul combination, I really couldn’t suggest it today. This is a fair critique and a problem any project living in a world defined by a strong incumbent suffers. You made me realize we need resources to help k8s users translate their existing knowledge to Nomad as many people looking at Nomad will have k8s experience these days. So thanks for this comment. Maybe with the right docs/resources we can at least minimize the very real cost of using a non-dominant tool. > But this is all a moot point to me. We are all in on AWS+ECS+Fargate where I work now and we really don’t care about the lock in boogie man. This was me in a past lives/jobs! HashiCorp's entire product line (with the exception of maybe Packer and Vagrant) become much more compelling for multi-cloud, hybrid-cloud, and on-prem orgs.

Eikon 7y ago

I know it's an unpopular opinion but I don't see the point of kubernetes. It's a complexity monster which is not providing any real advantage over way simpler solutions. Even pod to pod communication, which would be trivial to do using any sane solution is a huge pain in kubernetes. I would actually be frightened by running kubernetes in production. The simplest things are so hard to do that I don't even want to think about how to fix a weird issue when something goes wrong...

q3k 7y ago
> I know it's an unpopular opinion but I don't see the point of kubernetes. It's a complexity monster which is not providing any real advantage over way simpler solutions. The main selling point of k8s over any other tool is that it provides a single, unified, secured, multi-tenant API endpoint for managing all of production. Your developers updating their production app use the same API as a CI system that wants to spawn some worker containers, and the same API as an operator service maintaining a Redis cluster. All of this results in a single view into production. If things go well, the end result is that you swap daily interaction with a handful of different tools with disparate states (Terraform, Ansible/Chef/Salt/Puppet, shell scripts and proprietary tools) into just managing payloads on k8s. > Even pod to pod communication, which would be trivial to do using any sane solution is a huge pain in kubernetes. How is it a pain? A pod behind a service provides a DNS name that allows running requests to it - this handles the bulk of production traffic. If you want to contact a particular pod that is not behind a service just use the k8s API to retrieve details about it (like Prometheus does via k8s pod service discovery).
emerongi 7y ago
> Even pod to pod communication, which would be trivial to do using any sane solution is a huge pain in kubernetes. You create a service and then use the DNS name? How is that hard?
EdwardDiego 7y ago
The killer feature for us is autoscaling. Allows us to easily adapt to changing load throughout the day without having to overprovision. The ability to scale on custom metrics is really nice.

segmondy 7y ago

If you don't need container orchestration, you don't need kubernetes. If you need reliability and instead of using k8s, orchestrate your container manually you have issues.

deleted 7y ago

[deleted]

puhi 7y ago

Kubernetes is here to stay. What will change or what will be enhanced are: - Minimum requirements to actually run it (see k3s) - More managed services (gke, azure and aws exists but also digitalocean) - More/better handling of stateful services - Simple solution for write once read many (relevant for caching and for ci/cd) At that pace we are already with such a jung project, yeah this is great. This is huge. And no one needs to migrate already to kubernetes! But it already does a few things out of the box which reduces the complexity: - easy cert management - internal loadbalancing - autoscaling - green/blue deployment - deployment But you do see how the industry is struggling with certain problems: We are now with kubernetes moving into a cloud native area. Everyone know has kubernetes available. There was no mesos managed service from google, azure and aws. There was no docker swarm from google, azure and aws.

nailer 7y ago
> But you do see how the industry is struggling with certain problems: We are now with kubernetes moving into a cloud native area. We were were moving into a cloud native era way before Kubernetes.

nailer 7y ago

Of course most software companies (ie, where most of HN work) don't need Kubernetes. At this point it's become a meme for wasted engineering. Are you creating a cloud provider? - If you are creating a cloud provider, then yes you might want Kubernetes. If you are making your own cloud provider, you should question why you are doing that. - If you're not creating a cloud provider (for example, you're a software company), then use whatever VM / container / MicroVM / etc your cloud provider gives you rather than layering your own unnecessary complexity that adds questionable value on top of what your pay for from your cloud provider.

UK-Al05 7y ago
Normally cloud providers provide k8s to their customers.
- nailer 7y ago
  Yes, that's the biggest use of Kubernetes. Most cloud providers provide Xen VMs, containers (via whatever orchestration mechanism) or MicroVMs. Most people making software shouldn't care which. It's a box to run code.

Fiahil 7y ago

I'm usually fairly buzzword-adverse, and I didn't follow the NoSQL/Spark/Big-Data phase when it was hip. I was also hostile to kubernetes until I had to work with it against will for a project. Since then, I completely changed my mind about kubernetes. This is a very good technology for only one reason -and it's NOT about container orchestration-. Portability. K8s is the missing piece that allows you to create a network of cooperating computers independently of hardware, OS, and even architecture. If I have an home-made k8s cluster on my raspberry pies at home, it's not because it's lightweight and easy to manage -k8s adds a significant overhead-, it's because it gives me the ability to unplug one of them, take the sdcard, format it and plug another board without any interruption or configuration. I could plug my intel laptop to that network and have some pods running on it without having to change a single line in my configuration. Finally, I can zip a folder and email a bunch of yaml files to my friends (or have them git clone the repo), and they will be able to replicate an exact copy of my home cluster with all -or some- of the services. This is truly amazing.

pnako 7y ago
I literally don't understand what you are achieving with your k8s raspberry cluster thing.
- optimuspaul 7y ago
  automatic failover obviously

frou_dh 7y ago

I heard the word Kubernetes one too many times from listening to podcasts put out by The Changelog. The overhype bit got flipped in my head and now I'm predisposed against it. Marketing kills!

jugg1es 7y ago

I wish this article was titled 'Nomad is pretty great' since that was really the point being made here. Nomad is not a true alternative to Kubernetes. It does not support autoscaling out of the box, it does not have most of the bells/whistles that Kubernetes has. However, it is just a single executable and it just works. Combined with Consul and Fabio, you can run a container orchestration cluster with very little fuss that has service discovery, internal load balancing and cluster-wide logging. It is a feature-rich task scheduler with a pretty good CLI. I highly recommend it, but if you need stuff like autoscaling, you need to use something else.

schmichael 7y ago
(Full disclosure: Nomad Team Lead here, so I'm biased!) > Nomad is not a true alternative to Kubernetes. You're right! We explicitly try not to be a standalone dropin replacement for k8s. Our comparison page goes into this a little bit (but now I realize it's in need of a refresh!): https://www.nomadproject.io/intro/vs/kubernetes.html https://www.nomadproject.io/intro/vs/kubernetes.html - Nomad relies on Consul for service discovery - Nomad relies on Consul Connect (Envoy) for network policies - Nomad relies on Vault for secrets - Nomad relies on third party autoscalers - https://github.com/jippi/awesome-nomad#auto-scalers https://github.com/jippi/awesome-nomad#auto-scalers - although there's more we can do to enable them. - Nomad relies on Consul Connect (Envoy) or other third parties for load balancing - Nomad does not provide any cluster log aggregation (although logging plugins are planned which should make third party log aggregators easier to use) Nomad still has many missing features such as CSI (coming in 0.11!), logging plugins, audit logging, etc, but we never intend to be as monolithic a solution as k8s. We always hope to feel like "just a scheduler" that you compose with orthogonal tools.

moshloop 7y ago

I think this post misses the point of where kubernetes brings value the most: multi-team environments. If you are a single-team, then nomad, non-distributed docker, systemd units, etc is probably a better choice today due to the operational complexity of running Kubernetes (GKE,AKS,EKS do lower the complexity, but you still need kubernetes experts on call) However if you are in an organization with multiple teams (as are the vast majority of developers) then Kubernetes provides a common language for deploying, operating and securing your applications which enables you to go from a process which could takes days, weeks or even months to provision and configure a VM to minutes or hours to provision a kubernetes namespace.

vi-mode 7y ago

Before I started with k8s, I've been checking Nomad many times but eventually went with k8s. Nomad's learning curve didn't seem to be better, docs felt sparse and the much bigger ecosystem with k8s shouldn't be undervalued.

candiddevmike 7y ago

Anecdotally, about the worst place I've seen Kubernetes used is for CI/CD pipelines where every pipeline gets a fresh cluster. It's truly a special kind of hell--it takes at least 10 minutes to get a cluster up and running (depending on cloud provider), and it causes all kinds of indeterminism and frustration. You wait for a cluster to be spun up, wait for all of the extra services to be added, have one of them fail (completely unrelated to your PR), and have to do the whole thing over again. It's truly the new version of the "compiling" XKCD. If you ever want to see how brittle and bizarre Kubernetes can be, use it in a 100% automated fashion and hope for the best.

vicjicama 7y ago

I am very happy working with Kubernetes in a very small team, I really like the abstraction of the different entities and the declarative ways to configure them. I think that the learning curve is not that steep if you already had to do the same than with Kubernetes with other alternatives, in my on experience I have been discovering a lot of features that are very helpful not just during production but also during the development environments that were a real pain before. I am have a lab/cluster/blog running on Kubernetes, I am in charge of this one alone, it is opensource [1], I version everything that goes to the cluster so you can see the evolution of the kubernetes entities, the config, the containers and the code. I started this from scratch and improving it feature to feature, I think that this might be a big factor with my positive experience with Kubernetes. I wonder if an issue with adopting kubernetes is to try to migrate a big system into kubernetes in a very limited time lapse and trying to push/force features as they were working/handled in the previous approach? [1]: https://github.com/vicjicaman/microservice-realm https://github.com/vicjicaman/microservice-realm

flowerlad 7y ago

I am a one-person team running Kubernetes on Google Cloud. It costs around $62 per month before tax. (One n1-standard-1 node for $35.34 per month, HTTP Load Balancing $18.60 per month and SSD persistent disks for $8.50 per month). Kubernetes gives me the ability to scale up quickly when the need comes (which is soon, hopefully). I evaluated AWS Fargate & Kubernetes, Azure and GKE before settling on GKE. Amazon charges $148 per month just for cluster management alone. Google and Azure charge $0 for cluster management. This ruled out AWS for me. AWS appears to be a reluctant adopter of Kubernetes. They seem to want you to use Fargate instead. I tried it and found it to be crap--very hard to get things running. I was able to get Kubernetes running on Azure and GKE fairly easily. There were minor hiccups on both those clouds. With Azure initial creation of a AKS cluster failed because some of the resource providers weren't "registered" on my subscription. On GKE it was hard to get ingress working. Static IPs take a long time to take effect, and in the mean time you are fiddling with your yaml files trying to figure out what you may have done wrong, not realizing it is a GKE issue. The awesome part of Kubernetes is that I didn't need to learn almost anything about Google cloud to get everything working. I only had to learn Kubernetes. If Google raises prices I can easily switch to Azure without learning any Azure technologies. My knowledge as well as my application is completely portable. I can't imagine doing any of this as a one-person team without Kubernetes.

mpmpmpmp 7y ago
Second the comment about ingress. By the way, if you have any idea how to get ingress to work without taking down the services that are in it while it updates that would be amazing. I think it would obviously work if we used a separate ingress/load balancer for each host, but that seemed kinda wasteful since we are ok with doing downtime in off hours scheduling with our project.
- nouveaux 7y ago
  I believe the Kubernetes/Docker way is to not update in place but to create new instances. Can you spin up a new node pool/cluster and redirect traffic there?
gum_ina_package 7y ago

overgard 7y ago

I agree with the sentiment, but I think the hardest part of kubernetes is dealing the complexity of the resulting cloud. You end up having to manage that complexity no matter what you're doing; the upfront cost of figuring out kubernetes is that maybe it takes a couple days longer to learn than something else, but now you have a huge ecosystem of technology and people that are solving the same problems you have.

gfodor 7y ago

Personally, I've found Chef Habitat to be a sane alternative. I don't have any experience with k8s, but if you are not using containers especially, Habitat gets you: - packaging - deploys - configuration management - supervision - service discovery - isolation in a more classic "VM" environment, using unix jails. For anyone who isn't interested in learning k8s, but also wants a relatively modern, all-in-one solution to these problems, I recommend checking it out.

spicyramen 7y ago

I see Kubernetes also being used for ML workflows, yes...while you haven't figured out how to run a ML pipeline yet, add K8s (KubeFlow recommendation) As a ML researcher solutions like KubeFlow has been very painful as I don't want to learn k8s, I need to continue focusing on ML work and figure out my pipeline before I deploy K8s

ehutch79 7y ago

What are you talking about, my blog that has an astounding 2 visitors a month (one or both of which are accidental clicks) definitely needs to be on kubernetes!

sgt 7y ago

I think most Nomad users are well aware that Kubernetes can do all it can - and vastly more. If anything is wrong with Kubernetes, it would be the complexity of it and that it has a steep learning curve. It seems it's best to have a small team of people to manage it and to solve solutions to problems that arise. We started using Nomad last year and one thing I can say is that it's relatively easy to use and works well for its intended purpose, especially for us hybrid-cloud folks.

irq 7y ago

Given all the talk here about Kubernetes’ learning curve, does anyone have any resources for learning it for those that haven’t yet? Whether blogs, books, video courses, etc.

jordanpg 7y ago

The part about all of this containerization technology I don't understand is that it was originally sold as a way (among other benefits) to unburden devs of having to think about ops stuff and just code. Well, my experience with Docker, k8s, and related technologies, is that I now need to be an expert with these things just to get through my day. It's exhausting.

errantspark 7y ago

I feel like a lot of the problems K8s solves are fake problems that we have because all our software links/imports so many libraries/packages it's impossible to tell what's going on. It feels like have built an entire infrastructure around hopefully capturing that one time we got it working and being able to reboot to that state when things go wrong.

koheripbal 7y ago
Reminds me of the excitement around Java Beans
erulabs 7y ago
You’re not wrong but containerization is pragmatic - not idealistic. You can still write simple, clean software packaged in a SCRATCH container - but you can also isolate whatever crap you have to work with the get the job done. Both the elegant app and the horrible one now share a reliable init system.

luord 7y ago

It's really odd that around 30% of the people who commented defending kubernetes are merely echoing stuff that the article itself mentioned. Also, it's weird that the argument of another 30% of people defending kubernetes boils down to: "Using kubernetes is really easy, just hire Google/Azure/etc to do it for you." Can't begin to wrap my head around that one. But what do I know, I prefer to KISS and I like nomad. In fact, I'd be using swarm if its future wasn't spotty.

kresten 7y ago

I built a big globally scalable system without Kubernetes or containers. It’s very easy to understand and is highly reliable.