25 ms·

All these "maybe you don't need this or that X" posts die in an instant when the user already knows how to do X (when the learning curve argument is gone). Let

All these "maybe you don't need this or that X" posts die in an instant when the user already knows how to do X (when the learning curve argument is gone). Let's get it right: Kubernetes is really really cheap. I can run 20 low volume apps in a kubes cluster with a single VM. This is cheaper than any other hosting solution in the cloud if you want the same level of stability and isolation. It's even cheaper when you need something like a Redis cache. If my cache goes down and the container needs to be spun up again then it's not a big issue, so for cheap projects I can even save more cost by running some infra like a Redis instance as a container too. Nothing beats that. It gets even better, I can run my services in different namespaces, and have different environments (dev/staging/etc.) isolated from each other and still running on the same amount of VMs. When you caculate the total cost saving here to traditional deployments it's just ridiculously cheap. Kubernetes makes deployments really easy. docker build + kubectl apply. That's literally it. Deployments are two commands and it's live, running in the cloud. It's elastic, it can scale, etc. Kubernetes requires very little maintenance. Kubernetes takes care of itself. A container crashes? Kubes will bring it up. Do I want to roll out a new version? Kubes will do a rolling update on its own. I am running apps in kubes and for almost 2 years I haven't looked at my cluster or vms. They just run. Once every 6 months I log into my console and see that I can upgrade a few nodes. I just click ok and everything happens automatically with zero downtime. I mean yes, theoretically nothing needs Kubernetes, because the internet was the same before we had Kubernetes, so it's certainly not needed, but it makes life a lot easier. Especially as a cheap lazy developer who doesn't want to spend time on any ops Kubernetes is really the best option out there next to serverless. If learning Kubernetes is the reason why it's "not needed" then nothing is needed. Why use a new programming language? Why use a new db technology? Why use anything except HTML 4 + PHP, right? BTW, learning Kubernetes can be done in a few days.

brodouevencode 7y ago
Can second this. For most cases it's very low touch. When things break it has always been because I deployed bad code or configs. My weekends have freed up.
carlsborg 7y ago
Your single VM is a single point of failure. You probably want to run this in 3 VMs, one in each data center. ECS gives this to you out of the box, rolling deployments and health checks included.
- shantly 7y ago
  An awful lot of server systems can tolerate a hardware failure on their one server every couple years given 1) good backups, 2) "shit's broken" alerts, and 3) reliable push-button re-deploy-from-scratch capability, all of which you should have anyway. Lots of smaller shops trying to run to k8s and The Cloud probably have at least that much downtime (maybe an hour or two a year, on average) due to configuration fuck-ups on their absurd Rube Goldberg deployment processes anyway. [EDIT] oh and of course The Cloud itself dies from time to time, too. Usually due to configuration fuck-ups on their absurd Rube Goldberg deployment processes :-) I don't think one safely-managed (see above points) server is a ton worse than the kind of cloud use any mid-sized-or-smaller business can afford, outside certain special requirements. Your average CRUD app? Just rent a server from some place with a good reputation, once you have paying customers (just host on a VPS or two until then). All the stuff you need to do to run it safely you should be doing with your cloud shit anyway (testing your backups, testing your re-deploy-from-scratch capability, "shit's broken" alerts) so it's not like it takes more time or expertise. Less, really.
  - abraxas 7y ago
    Not to mention there are now servers available for purchase today that have 128 x86 cores. And 2-4 TB of RAM. That's a lot of "cloud" right there in a single server.
  - carlsborg 7y ago
    Business services generally need high availability goals, so often that doesn't cut it. And your single server doesn't autoscale to load. AWS gives you availability zones, which are usually physically distinct datacenters in a region, and multiple regions. Well designed cloud apps failover between them. Very very rarely have we seen an outage across regions in AWS, if ever.
falcolas 7y ago
If you're running on a single host anyways, why not just use init scripts or unit files? All Kubernetes is giving you is another 5-6 layers of indirection and abstraction. EDIT: Quick clarification: still use containers. However, running containers doesn't require running Kubernetes. > learning Kubernetes can be done in a few days The basic commands, perhaps. But with Kubernetes' development velocity, the learning will never stop - you really do need (someone) dedicated part time to it to ensure that a version upgrade doesn't break automation/compliance (something that's happened to my company a few times now).
- moduspol 7y ago
  It gives you good abstractions for your apps. I know exactly what directories each of my apps can write to, and that they can't step on each others' toes. Backing up all of their data is easy because I know the persistent data for all of them is stored in the same parent directory. Even if the whole node caught on fire, I can restore it by just creating a new Kubernetes box from scratch, re-applying the YAML, and restoring the persistent volume contents from backup. To me there's a lot of value over init scripts or unit files.
  - falcolas 7y ago
    > I know exactly what directories each of my apps can write to, and that they can't step on each others' toes You can do this with docker commands too. Ultimately, that's all that Kubernetes is doing, just with a YAML based DSL instead of command line flags. > Even if the whole node caught on fire, I can restore it So, what's different from init/unit files? Just rebuild the box and put in the unit files, and you get the same thing you had running before. Again, for a single node there's nothing that Kubernetes does that init/unit files can't do.
    - moduspol 7y ago
      > You can do this with docker commands too. Ultimately, that's all that Kubernetes is doing, just with a YAML based DSL instead of command line flags. Well, I mean, mostly. You're gonna be creating your own directories and mapping them into your docker-compose YAMLs or Docker CLI commands. And if you have five running and you're ready to add your sixth, you're gonna be SSHing in to do it again. Not quite as clean as "kubectl apply" remotely and the persistent volume gets created for you, since you specified that you needed it in your YAML. > So, what's different from init/unit files? Just rebuild the box and put in the unit files, and you get the same thing you had running before. Again, for a single node there's nothing that Kubernetes does that init/unit files can't do. Well you kinda just partially quoted my statement and then attacked it. You can do it with init/unit files, but you've got a higher likelihood of apps conflicting with each other, storing things in places you're not aware of, and missing important files in your backups. It's not about what you "can't" do. It's about what you can do more easily, and treat bare metal servers like dumb container farms (cattle).
WnZ39p0Dgydaz1 7y ago
I couldn't agree more. I've built a lot of sideprojects on Kubernetes and couldn't be happier with it. It's incredibly cheap (basically free in terms of resource usage), low-touch, and abstracts away so many pesky problems that I had to deal with before. My cluster has been running for 12+ months without a single application-level outage - containers simply restart and get rescheduled when something goes wrong, and external logging and analytics solutions are a breeze to integrate.
- Legogris 7y ago
  I disagree here - there's definitely some minimum amount of resources you want to use to make k8s worth it memory- and CPU-wise. You may be able to get by with 1 CPU and 1GB of RAM for your master (which is just coordinating, not running any workloads), and there's some overhead on the workers as well. I've been looking at running k8s at some raspberry pis at home, but anything smaller than the recently released 4 is just not worth it IMO (though I've seen several people run clusters of 3B+s).
  - byteshock 7y ago
    You can’t really compare you deploying kubernetes to a bunch of raspberry pis to people normally deploying them to cloud hosting services or using managed services..... Totally different use case
    - Legogris 7y ago
      I am not. GP doesn’t mention running managed. When talking about “basically free” and for side-projects I don’t really get your point. Minimum cost for a one-node n1-standard-1 cluster is a subsidized ~25 USD per month on GKE (assuming not using preembtibles). After 1 year you’re above the price of comparable hardware (there are pretty decent 300 $ laptops these days). I’m a happy customer of GKE but it’s not for everyone and everything. Like you say, different use-cases.
avip 7y ago
You nailed it in the first sentence. The blog post pretty much boils down to "k8s looks good but we were too lazy to learn how to use the thing, so we opted into something else". Fair enough, legit argument, but trying to make a "counter-k8s" case based on that is not very convincing.
- politelemon 7y ago
  > but we were too lazy This is unnecessarily dismissive and contributes to what makes HN a toxic place for discussions. They've already addressed the reasons.
  - avip 7y ago
    Point taken. However, the part about "why not kubernetes" reads: [...] we started adding ever-more complex layers of logic to operate our services. As an example, Kubernetes allows [...] this can get quite confusing [...]. [...] this can lead to tight, implicit coupling between your project and Kubernetes. [...] it’s tempting to go down that path and build unnecessary abstractions that can later bite you. [...] It takes a fair amount of time and energy to stay up-to-date with the best practices and latest tooling. [...] the learning curve is quite steep. So in short "it is complex so this and that may happen if you don't learn it properly". Ok, this reasoning applies a-priori to any tool.
    - yjftsjthsd-h 7y ago
      Not every tool is equally complex or requires the same amount of learning. K8s has a reputation for being really high on the scale, so a reasonable team could consider it and then decide to use something less complex.
rooam-dev 7y ago
"BTW, learning Kubernetes can be done in a few days. " - Learn something in a few days and the ability to run it in production are completely 2 different things. Security, upgrades and troubleshooting cannot be learned in couple of days.
- DuskStar 7y ago
  Sure, but "how to toss existing docker containers into GKE" is IMO less than three days. Unless you have a reason to manage your own k8s cluster, k8s is extremely easy.
- pas 7y ago
  Anyone who starts doing anything "in production" based on a "maybe you don't need k8s" article should step back and think about whether they are the right person to put things into production. Are you bootstrapping your stealth-mode side-project? Pick whatever you think is best, but think about the time value of operations. (So maybe just pick a managed k8s.) Are you responsible for a system that handles transactions worth millions of dollars every day? Then maybe, again you should seek the counsel of professionals. Otherwise these articles are just half-empty fuel cans for an (educated?) dumpster fire. That said HashiCorp stuff is almost guaranteed to be amazing. I haven't even looked at Nomad, but I think anybody starting out with orchestration stuff should give it a go, and when they think they have outgrown it they will know what's next. Maybe k8s, maybe something else.
davedx 7y ago
> Kubernetes is really really cheap In terms of infra costs, I can believe it. But what about engineer resources to setup and maintain your infra in k8s clusters? Where I work, we have a full time DevOps team that's almost the same size as the main product teams. That's really not cheap.
- hello_moto 7y ago
  Seems like companies that began to adopt k8s come with less experience so the task looks humongous hence the need to hire "more people".
- aeyes 7y ago
  Weird, we are 3 working on infra/CI/tooling supporting some 80+ engineers working on the product which didn't write a single line of config for Docker, CICD, Infra automation, K8s. In fact, it was done by one person until the product teams reached some 40 engineers. We aren't 10x guys either. What's different though is that we don't believe in the microservice hypetrain so I don't have 200 codebases to watch after.
  - Legogris 7y ago
    Do you run on managed k8s from a cloud provider or do you maintain you own cluster(s)? The former that sounds reasonable, the latter sounds near implausible.
- bloopernova 7y ago
  For whatever accounting/asset/tax reason, corporations seem happy to spend vast sums on monthly costs such as AWS, yet loathe purchasing physical computers to do the same job.
candiddevmike 7y ago
All of this glosses over the biggest issue with Kubernetes: it's still ridiculously complex, and troubleshooting issues that arise (and they will arise), can leave you struggling for days pouring over docs, code, GitHub issues, stackoverflow... All of the positives you listed rely on super complex abstractions that can easily blow up without a clear answer as to "why". Compared to something like scp and restarting services, I would personally not pay the Kubernetes tax unless I absolutely had to.
- dustinmoris 7y ago
  Maybe, but the point with containers and kubernets is to treat it like cattle, not pets. If something blows up or dies, then with Kubernetes it's often faster to just tear down the entire namespace and bring it up again. If the entire cluster is dead, then just spin up a new cluster and run your yaml files on it and kill your old cluster. Treat it like cattle, when it doesn't serve your purpose anymore then shoot it. This is one of the biggest advantages of Kubes, but often overlooked because traditional Ops people keep treating infrastructure like a pet. Only thing you should treat like a pet is your persistence layer, which is presumably outside Kubes, somehting like DynamoDb, Firestore, CosmosDb, SQL server, whatever.
  - reacharavindh 7y ago
    This is not good engineering. If somebody told me this at a business, I’d not trust them anymore with my infrastructure. So, you say that problems happen, and you consciously don’t want to know/solve them. A recurring problem in you view is solved with constantly building new K8s clusters and your whole infrastructure in it every time!?! Simple example - A microservice that leaks memory.... let it keep restarting as it crashes?! I remember at one of my first jobs, at a healthcare system for a hospital in India, their Java app was so poorly written that it kept leaking memory and bloated beyond GC could help and will crash every morning at around 11 AM and then again at around 3 PM. The end users - Doctors, nurses, pharmacists knew about this behavior and had breaks during that time. Absolutely bullshit engineering! It’s a shame on those that wrote that shitty code, and shame on whoever reckless to suggest a ever rebuilding K8s clusters.
peterwwillis 7y ago
If you are running K8s on a single VM, you by definition do not need K8s. Run your apps with Docker and save yourself 50 layers of complexity. > Kubernetes makes deployments really easy. docker build + kubectl apply. The parts between "docker build" and "kubectl apply" is literally CI versus CD; they're more complicated than 2 steps. And when there's a problem with either, K8s is not going to fix it up for you. You'll have to be notified via monitoring and begin picking through the 50 layers of complexity to find the error and fix it. Which is why we have deployment systems to do things like validation of all the steps and dependencies in the pipeline, so you don't end up with a broken prod deploy. > Kubernetes requires very little maintenance Whatever you're smoking, pass it down here... Have you ever had to refresh all the certs on a K8s cluster? Have you ever had to move between breaking changes in the K8s backend during version upgrade? Crafted security policies, RBACs, etc to make sure when you run the thing you're not leaving gaping backdoors into your system? There's like 50 million different custom solutions out there just for K8s maintenance. Entire businesses are built around it.
boobePhuu7iet7i 7y ago
Easy deployments/upgrades, and automagic external DNS and certificate management sealed the deal for me.
- Legogris 7y ago
  Which network fabric do you use and how did you set up DNS/cert management? For me certificates has been one of the pain points - have been using cert-manager with LetsEncrypt for some time but it has been notoriously unstable and they have introduced plenty of breaking changes between releases. (That being said I haven't tried the more recent releases, maybe things have gotten more stable in the past couple of months) Google recently release managed certs for those running on GKE, but those are limited to a single domain per cert.
  - boobePhuu7iet7i 7y ago
    I use the external-dns and cert-manager tools. cert-manager uses lets-encrypt but fully automates everything, you just add an annotation to your ingress resource. Been using it in prod for around 6 months now with no problems.
    - Legogris 7y ago
      Ah, sounds like they’re stabilizing then - I’ve had a lot of stability and upgrading issues with older versions. Just the fact that you couldn’t configure it for automatic renewal with anything else than 24h before expiry and these renewals would fail half the time... But I will give it another try at some point.
dragonsh 7y ago
So let’s take step by step: 1. Learn shell scripts. 2. Learn Docker conf. 4. Learn yaml. (May be helm and plethora of acronyms specially designed for k8s). 5. Combine spaghetti of 1,2,3 to build container image scripts. 6. Tie it into CI/CD process for adventurous. 7. Learn to install and manage k8s cluster or learn proprietary non open source api of google or amazon or azure. 8. Constantly patch and manage plethora of infrastructure software besides application code. Now with all this use a tool designed for million user application on an application which will be used by 100’s to 1000’s of users. I think k8s is designed for google kind of problem and is an overkill for over 80-90% of deployments and applications. May be just use simple deployment with Ansible, puppet,chef, nix, gnu guix etc. to deploy and manage software based on necessity on a single vm and extend it to a large cluster if necessary of bare metal or vm or container in a cloud agnostic manner. Not sure when the technology fashion overtook the infrastructure area like complex web app tooling in JavaScript world. K8s has its own place at scale required by handful of Organization with google level traffic and load for most traditional companies Simple cluster management and configuration management tool will work wonders with less moving parts and cognitive load.
- curryst 7y ago
  Then you still have to learn Ansible, so you're still learning a tool. And you probably want to use Docker anyways, unless you're shipping statically linked binaries. Also, you said VM which implies that instead of Kubernetes you're going to use a VM platform, which comes with every bit of the complexity Kubernetes has. I agree with you that it's a simpler deployment method when you don't need HA. As soon as you need HA, then all of a sudden you need to be able to fail over to new nodes and manage a load balancer and the backends for that load balancer. Kubernetes makes that easy. Kubernetes makes easy things harder than they should be, and hard things easier than they should be. The number of things I've managed that don't need HA is vanishingly low.
  - ar_lan 7y ago
    I think you're starting to get at what has annoyed me about a lot of the anti-k8s folks. I really just don't think they understand the tool, because any production environment should have many of the features Kubernetes helps provide. So the argument becomes "I know how to do it this other way, so learning a new tool is too complex." Kubernetes helps standardize a lot of these things - I can very easily hop between different clusters running completely different apps/topologies and have a good sense of what's going on. A mish-mash of custom solutions re-inventing these wheels is, in my opinion, far more confusing. > Kubernetes makes easy things harder than they should be, and hard things easier than they should be. This is really the crux, I think. I think a lot of people look at Kubernetes, try to learn it by running their blog on a K8s platform, and decide it's overly complex to run something that Docker probably solves (alone) for them. When you need HA for many services, don't want to have to handle the hassle of your networking management, design your applications with clear abstractions, etc., and really need to start worrying about scale, Kubernetes starts to really shine. /rant
jen20 7y ago
> Kubernetes is really really cheap. I can run 20 low volume apps in a kubes cluster with a single VM. Nomad is just as cheap, if not cheaper.
MrUnderhill 7y ago
I'm not very familiar with Kubernetes, but from what I've seen, to me it looks quite complicated. I'm wondering, when people say they use Kubernetes and consider it easy/simple, does that typically include operating the underlying Kubernetes container infrastructure as well? Or does "using Kubernetes" usually mean deploying containers to someone else's Kubernetes hosting (like Google)?
- mpmpmpmp 7y ago
  When I say it (and I would guess outside of people who either contribute to Kubernetes or are using it to build their own PaaS) I mean a managed service like GKE. When I attempted to set my own cluster up all the confusion was around all of the choices to make, what container networking stack, etcd, and so on. GKE gives you an easy button for simpler architectures where you just have some containers and you want them to go in a cluster.
- vorpalhex 7y ago
  You will probably use someone's recommended Kubernetes stack, like k3s. It doesn't make a ton of difference in practice whether you let someone else host your control plane or do it on your own hardware, but your actual nodes you probably want instance level control on. Over time you'll probably grow your own customizations and go-tos on kubernetes that you layer on top of k3s or what have you.
tonetheman 7y ago
K8 is only cheap if someone else is running it for you and you never have problems.
notyourday 7y ago
> Kubernetes is really really cheap. I can run 20 low volume apps in a kubes cluster with a single VM. This is cheaper than any other hosting solution in the cloud if you want the same level of stability and isolation. It's even cheaper when you need something like a Redis cache. If my cache goes down and the container needs to be spun up again then it's not a big issue, so for cheap projects I can even save more cost by running some infra like a Redis instance as a container too. Nothing beats that. It gets even better, I can run my services in different namespaces, and have different environments (dev/staging/etc.) isolated from each other and still running on the same amount of VMs. When you caculate the total cost saving here to traditional deployments it's just ridiculously cheap. If you're running 20 apps on a kubes cluster with a single VM you are running twenty apps on a single VM. There's no backup, scalability or anything else. There's no orchestration. your deployment is a hipster version of rsync -avH myapp mybox:/my/location/myapp followed by a restart done via http to tell monit/systemd to restart your apps. It is a perfectly fine way of handling apps. k8s shines when you have a fleet of VMs and a fleet of applications that depend on each other and have dynamic constrains but that's not what most of k8s installations work > I mean yes, theoretically nothing needs Kubernetes, because the internet was the same before we had Kubernetes, so it's certainly not needed, but it makes life a lot easier. Especially as a cheap lazy developer who doesn't want to spend time on any ops Kubernetes is really the best option out there next to serverless. Only in a throw production code over the fence sense.
jordanbeiber 7y ago
One side of this - our base k8s config is 44k lines of yaml, leading to a bunch of controllers and pods running, in a rather complex fashion. Not to mention k8s conplexity and codebase itself. It blackboxes so many things. I still chuckle at the fact that ansible is a first class citizen of the operator framwork. It can certainly implode on you! In my experience running nomad & consul is a more lightweight and simple way of doing many of the same things. It’s a bit like the discussion raging in regards to systemd and the fact that it’s not “unixy”. I get the same feeling with k8s, whereas the hashicorp stuff, albeit less features, adheres more to the unix philosophy. Thus easier to maintain and integrate. Just my ¢2.
- mst 7y ago
  Sort of similar to the fact that I basically always have both tmux and dtach installed, the latter for "just make this detachable", the former for "actually I'd like some features today".
  - jordanbeiber 7y ago
    Something like that. I want service discovery, where Consul really shines - cause containers ain't all we're doing, mkay. K8s forces etcd on you, for service discovery, and only within the cluster. So, sync it then... more code more complexity, no "pipe" ("pipe" not to be taken literally in this context) and simple integrations. Not to mention, consul is already a full DNS server, but in k8s, we need yet another pod for coredns. Is YAP a thing? =) For example, I love how easily envoy integrates to any service discovery of you liking, even your own - simply present it with a specific JSON response from an API. Much like how the Ansible inventory works. It makes integrating, picking and choosing as well as maintaining and extending your configuration management just so much more pleasant.
- vorpalhex 7y ago
  > our base k8s config is 44k lines of yaml Something has gone horribly wrong. > In my experience running nomad & consul is a more lightweight and simple way of doing many of the same things. If you can replace what you're getting out of your 44k lines of yaml with those two products, then you are using the wrong tools.
etxm 7y ago
Agree / disagree. I’ve been using k8s in large deployments (1000s of nodes) for about 3 years. It’s easy to get started with using GKE, EKS, etc. it’s difficult to maintain if you’re bootstrapping your own cluster. 3 years in, and despite working with k8s at a pretty low level, I still learn more about functionality and function every single day. I do agree it’s great tooling wise. I personally deploy on docker for desktop k8s day one when starting a new project. I understand all the tooling, it’s easier than writing a script and figuring out where to store my secrets every damn time. The big caveat is - kubernetes should be _my_ burden as someone in the Ops/SRE team, but I feel like you frequently see it bleed out into application developer land. I think that the CloudRuns and Fartgates* of the world are better suited to the average developer and I think it’s Ops responsibility to make k8s as transparent as possible within the organization. Application developers want a Heroku. Edit: * LOL
t0astbread 7y ago
I've looked at k8s a while ago and it looked insanely complex to me. Maybe it's one of those things that you need to look at several times to understand it?
- sjellis 7y ago
  Kubernetes addresses a bunch of infrastructure concerns, so IMO it's more equivalent to learning a new cloud platform (AWS, Azure) than anything else. People who already know one or more cloud platforms, and are familiar with the kind of issues that come up with distributed and HA systems on cloud (e.g. networking, secrets management) may find Kubernetes much easier.
Rantenki 7y ago
> BTW, learning Kubernetes can be done in a few days. As someone who has deployed K8S at scale several times, this is nonsense. Learning K8S deeply enough to deploy AND MAINTAIN IT is a huge undertaking that requires an entire team to do right. Sure, you can deploy onto AWS via KOPS in a day, and you can deploy a hello world app in another day. Easy. But that only gets you to deployment, it doesn't mean you can maintain it. There are TONS of potential failure modes, and at this point you don't understand ANY of them. When one of them crops up, who do you page at 3AM, how do you know it's even down (monitoring/alerting isn't "batteries included"), how do you diagnose and fix it once you _do_ know what's broken? Not to mention the fact that you _have_ to continuously upgrade K8S as old releases go out of maintenance in under a year. If you're not continuously testing upgrades against production equivalent deploys, you're going to footgun spectacularly at some point, or be stuck with an old release that new/updated helm charts won't work against. TL;DR: If you can afford a team to deploy and maintain K8S, and have a complex enough application stack to need it, it's awesome; but it's not free in either time or staff.
- UK-Al05 7y ago
  How much effort is it to maintain k8s provided by a cloud service provider instead of something you have set up yourself?
  - xemdetia 7y ago
    As far as I have seen there is still updating overhead where you have to initiate upgrades to a known stable/supported version at a regular cadence. EKS suggests having an E2E CI pipeline to test it before updating production, and I feel like that's the only way to do it. There is nonzero churn even though it's managed.
kissgyorgy 7y ago
It definitely can't be don in a few days. It took me 2-3 months to go through everything and run my own cluster at home. And yes, you need to go through about every major concept (architecture, object type, API, authentication, services, deployments, updates, etc, etc, etc) in order to make sense of Kubernetes overall. It's a very complex software. You can learn a simple programming language like Go in a couple of days, but definitely not Kubernetes.
Legogris 7y ago
> Kubernetes requires very little maintenance. Kubernetes takes care of itself. A container crashes? Kubes will bring it up. Do I want to roll out a new version? Kubes will do a rolling update on its own. I am running apps in kubes and for almost 2 years I haven't looked at my cluster or vms. They just run. Once every 6 months I log into my console and see that I can upgrade a few nodes. I just click ok and everything happens automatically with zero downtime. > BTW, learning Kubernetes can be done in a few days. This is true if you are using managed k8s from a provider or have an in-house team taking care of this. Far, far from the truth if you also need to set up and maintain your own clusters. I'm very comfortable with k8s as a user and developer, would not be comfortable setting up a scalable and highly available cluster for production use for anything more serious than my homelab.
gigatexal 7y ago
Docker swarm gets you this without all the complexity of Kubernetes. Hell even a largish docker-compose setup could get you there.
JeremyNT 7y ago
You can be able to use Kubernetes very quickly. If you have a manged k8s cluster available to you, that somebody manages for you, then sure it is all upside. All the stuff you describe is great! The thing is, if you have to manage the complexity and lifecycle of the cluster yourself, the balance tips dramatically. How do you provision it? How do you maintain it? How do you secure it? How do you upgrade it? So I agree, k8s is great for running all manner of project, big and small. If you already have a k8s you will find yourself wanting to use it for everything! However if you don't have one, and you aren't interested in paying somebody to run one for you, then you should think long and hard about whether you're better off just launching a docker-compose from systemd or something.
- pas 7y ago
  > How do you provision it? How do you maintain it? How do you secure it? How do you upgrade it? https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ https://kubernetes.io/docs/setup/production-environment/tool... https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network https://kubernetes.io/docs/setup/production-environment/tool... https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/ https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/... https://kubernetes.io/docs/concepts/security/overview/ https://kubernetes.io/docs/concepts/security/overview/ Of course, it's not as easy as a managed solution, but it's not exactly black magic. Docker compose from systemd is not bad, but maybe then instead of that using k3s is a better middle ground: https://github.com/rancher/k3s https://github.com/rancher/k3s
erling 7y ago
Have you looked into Erlang/the BEAM VM? It provides many of these same benefits. > This is cheaper than any other hosting solution in the cloud if you want the same level of stability and isolation. Erlang provides total process isolation and can theoretically also run on only a single machine. > It's even cheaper when you need something like a Redis cache. If my cache goes down and the container needs to be spun up again then it's not a big issue, so for cheap projects I can even save more cost by running some infra like a Redis instance as a container too. Nothing beats that. In Erlang, each process keeps its own state without a single point of failure (ie a single Redis instance) and can be restarted on failure by the VM. > Kubernetes takes care of itself. A container crashes? Kubes will bring it up. Erlang VM takes care of itself. A process crashes? The VM will bring it up. > Do I want to roll out a new version? Kubes will do a rolling update on its own. Ditto for Erlang VM, with zero-time deployments with hot code reloading. > I just click ok and everything happens automatically with zero downtime. Erlang is famous for its fault tolerance and nine nines of downtime! And it’s been doing so for quite a bit longer than K8s.
brozaman 7y ago
"BTW, learning Kubernetes can be done in a few days." How do you troubleshoot an api-group that is failing intermittently? How do you troubleshoot a CSI issue? Because CSI isn't a simple protocol like CNI. What do you look at if kubectl gets randomly stuck? What do you do if a node becomes notReady? What do you do if the container runtime of a several nodes starts failing? What do you do if one of the etcd nodes doesn't start? What if you are misisng logs in the log aggregator or metrics? What if when you create a project it's missing the service accounts? What if kube-proxy randomly doesn't work? What if the upgrade fails and ends up in an inconsistent state? What if your pod is actually running but shows as pending? Sure, you can learn how to deploy kubernetes and an application on top of it in a couple days, but learning how to run a production will take way longer than that.

brodouevencode 7y ago

Can second this. For most cases it's very low touch. When things break it has always been because I deployed bad code or configs. My weekends have freed up.

carlsborg 7y ago

Your single VM is a single point of failure. You probably want to run this in 3 VMs, one in each data center. ECS gives this to you out of the box, rolling deployments and health checks included.

shantly 7y ago
An awful lot of server systems can tolerate a hardware failure on their one server every couple years given 1) good backups, 2) "shit's broken" alerts, and 3) reliable push-button re-deploy-from-scratch capability, all of which you should have anyway. Lots of smaller shops trying to run to k8s and The Cloud probably have at least that much downtime (maybe an hour or two a year, on average) due to configuration fuck-ups on their absurd Rube Goldberg deployment processes anyway. [EDIT] oh and of course The Cloud itself dies from time to time, too. Usually due to configuration fuck-ups on their absurd Rube Goldberg deployment processes :-) I don't think one safely-managed (see above points) server is a ton worse than the kind of cloud use any mid-sized-or-smaller business can afford, outside certain special requirements. Your average CRUD app? Just rent a server from some place with a good reputation, once you have paying customers (just host on a VPS or two until then). All the stuff you need to do to run it safely you should be doing with your cloud shit anyway (testing your backups, testing your re-deploy-from-scratch capability, "shit's broken" alerts) so it's not like it takes more time or expertise. Less, really.
- abraxas 7y ago
  Not to mention there are now servers available for purchase today that have 128 x86 cores. And 2-4 TB of RAM. That's a lot of "cloud" right there in a single server.
- carlsborg 7y ago
  Business services generally need high availability goals, so often that doesn't cut it. And your single server doesn't autoscale to load. AWS gives you availability zones, which are usually physically distinct datacenters in a region, and multiple regions. Well designed cloud apps failover between them. Very very rarely have we seen an outage across regions in AWS, if ever.

falcolas 7y ago

If you're running on a single host anyways, why not just use init scripts or unit files? All Kubernetes is giving you is another 5-6 layers of indirection and abstraction. EDIT: Quick clarification: still use containers. However, running containers doesn't require running Kubernetes. > learning Kubernetes can be done in a few days The basic commands, perhaps. But with Kubernetes' development velocity, the learning will never stop - you really do need (someone) dedicated part time to it to ensure that a version upgrade doesn't break automation/compliance (something that's happened to my company a few times now).

moduspol 7y ago
It gives you good abstractions for your apps. I know exactly what directories each of my apps can write to, and that they can't step on each others' toes. Backing up all of their data is easy because I know the persistent data for all of them is stored in the same parent directory. Even if the whole node caught on fire, I can restore it by just creating a new Kubernetes box from scratch, re-applying the YAML, and restoring the persistent volume contents from backup. To me there's a lot of value over init scripts or unit files.
- falcolas 7y ago
  > I know exactly what directories each of my apps can write to, and that they can't step on each others' toes You can do this with docker commands too. Ultimately, that's all that Kubernetes is doing, just with a YAML based DSL instead of command line flags. > Even if the whole node caught on fire, I can restore it So, what's different from init/unit files? Just rebuild the box and put in the unit files, and you get the same thing you had running before. Again, for a single node there's nothing that Kubernetes does that init/unit files can't do.
  - moduspol 7y ago
    > You can do this with docker commands too. Ultimately, that's all that Kubernetes is doing, just with a YAML based DSL instead of command line flags. Well, I mean, mostly. You're gonna be creating your own directories and mapping them into your docker-compose YAMLs or Docker CLI commands. And if you have five running and you're ready to add your sixth, you're gonna be SSHing in to do it again. Not quite as clean as "kubectl apply" remotely and the persistent volume gets created for you, since you specified that you needed it in your YAML. > So, what's different from init/unit files? Just rebuild the box and put in the unit files, and you get the same thing you had running before. Again, for a single node there's nothing that Kubernetes does that init/unit files can't do. Well you kinda just partially quoted my statement and then attacked it. You can do it with init/unit files, but you've got a higher likelihood of apps conflicting with each other, storing things in places you're not aware of, and missing important files in your backups. It's not about what you "can't" do. It's about what you can do more easily, and treat bare metal servers like dumb container farms (cattle).

WnZ39p0Dgydaz1 7y ago

I couldn't agree more. I've built a lot of sideprojects on Kubernetes and couldn't be happier with it. It's incredibly cheap (basically free in terms of resource usage), low-touch, and abstracts away so many pesky problems that I had to deal with before. My cluster has been running for 12+ months without a single application-level outage - containers simply restart and get rescheduled when something goes wrong, and external logging and analytics solutions are a breeze to integrate.

Legogris 7y ago
I disagree here - there's definitely some minimum amount of resources you want to use to make k8s worth it memory- and CPU-wise. You may be able to get by with 1 CPU and 1GB of RAM for your master (which is just coordinating, not running any workloads), and there's some overhead on the workers as well. I've been looking at running k8s at some raspberry pis at home, but anything smaller than the recently released 4 is just not worth it IMO (though I've seen several people run clusters of 3B+s).
- byteshock 7y ago
  You can’t really compare you deploying kubernetes to a bunch of raspberry pis to people normally deploying them to cloud hosting services or using managed services..... Totally different use case
  - Legogris 7y ago
    I am not. GP doesn’t mention running managed. When talking about “basically free” and for side-projects I don’t really get your point. Minimum cost for a one-node n1-standard-1 cluster is a subsidized ~25 USD per month on GKE (assuming not using preembtibles). After 1 year you’re above the price of comparable hardware (there are pretty decent 300 $ laptops these days). I’m a happy customer of GKE but it’s not for everyone and everything. Like you say, different use-cases.

avip 7y ago

You nailed it in the first sentence. The blog post pretty much boils down to "k8s looks good but we were too lazy to learn how to use the thing, so we opted into something else". Fair enough, legit argument, but trying to make a "counter-k8s" case based on that is not very convincing.

politelemon 7y ago
> but we were too lazy This is unnecessarily dismissive and contributes to what makes HN a toxic place for discussions. They've already addressed the reasons.
- avip 7y ago
  Point taken. However, the part about "why not kubernetes" reads: [...] we started adding ever-more complex layers of logic to operate our services. As an example, Kubernetes allows [...] this can get quite confusing [...]. [...] this can lead to tight, implicit coupling between your project and Kubernetes. [...] it’s tempting to go down that path and build unnecessary abstractions that can later bite you. [...] It takes a fair amount of time and energy to stay up-to-date with the best practices and latest tooling. [...] the learning curve is quite steep. So in short "it is complex so this and that may happen if you don't learn it properly". Ok, this reasoning applies a-priori to any tool.
  - yjftsjthsd-h 7y ago
    Not every tool is equally complex or requires the same amount of learning. K8s has a reputation for being really high on the scale, so a reasonable team could consider it and then decide to use something less complex.

rooam-dev 7y ago

"BTW, learning Kubernetes can be done in a few days. " - Learn something in a few days and the ability to run it in production are completely 2 different things. Security, upgrades and troubleshooting cannot be learned in couple of days.

DuskStar 7y ago
Sure, but "how to toss existing docker containers into GKE" is IMO less than three days. Unless you have a reason to manage your own k8s cluster, k8s is extremely easy.
pas 7y ago
Anyone who starts doing anything "in production" based on a "maybe you don't need k8s" article should step back and think about whether they are the right person to put things into production. Are you bootstrapping your stealth-mode side-project? Pick whatever you think is best, but think about the time value of operations. (So maybe just pick a managed k8s.) Are you responsible for a system that handles transactions worth millions of dollars every day? Then maybe, again you should seek the counsel of professionals. Otherwise these articles are just half-empty fuel cans for an (educated?) dumpster fire. That said HashiCorp stuff is almost guaranteed to be amazing. I haven't even looked at Nomad, but I think anybody starting out with orchestration stuff should give it a go, and when they think they have outgrown it they will know what's next. Maybe k8s, maybe something else.

davedx 7y ago

> Kubernetes is really really cheap In terms of infra costs, I can believe it. But what about engineer resources to setup and maintain your infra in k8s clusters? Where I work, we have a full time DevOps team that's almost the same size as the main product teams. That's really not cheap.

hello_moto 7y ago
Seems like companies that began to adopt k8s come with less experience so the task looks humongous hence the need to hire "more people".
aeyes 7y ago
Weird, we are 3 working on infra/CI/tooling supporting some 80+ engineers working on the product which didn't write a single line of config for Docker, CICD, Infra automation, K8s. In fact, it was done by one person until the product teams reached some 40 engineers. We aren't 10x guys either. What's different though is that we don't believe in the microservice hypetrain so I don't have 200 codebases to watch after.
- Legogris 7y ago
  Do you run on managed k8s from a cloud provider or do you maintain you own cluster(s)? The former that sounds reasonable, the latter sounds near implausible.
bloopernova 7y ago
For whatever accounting/asset/tax reason, corporations seem happy to spend vast sums on monthly costs such as AWS, yet loathe purchasing physical computers to do the same job.

candiddevmike 7y ago

All of this glosses over the biggest issue with Kubernetes: it's still ridiculously complex, and troubleshooting issues that arise (and they will arise), can leave you struggling for days pouring over docs, code, GitHub issues, stackoverflow... All of the positives you listed rely on super complex abstractions that can easily blow up without a clear answer as to "why". Compared to something like scp and restarting services, I would personally not pay the Kubernetes tax unless I absolutely had to.

dustinmoris 7y ago
Maybe, but the point with containers and kubernets is to treat it like cattle, not pets. If something blows up or dies, then with Kubernetes it's often faster to just tear down the entire namespace and bring it up again. If the entire cluster is dead, then just spin up a new cluster and run your yaml files on it and kill your old cluster. Treat it like cattle, when it doesn't serve your purpose anymore then shoot it. This is one of the biggest advantages of Kubes, but often overlooked because traditional Ops people keep treating infrastructure like a pet. Only thing you should treat like a pet is your persistence layer, which is presumably outside Kubes, somehting like DynamoDb, Firestore, CosmosDb, SQL server, whatever.
- reacharavindh 7y ago
  This is not good engineering. If somebody told me this at a business, I’d not trust them anymore with my infrastructure. So, you say that problems happen, and you consciously don’t want to know/solve them. A recurring problem in you view is solved with constantly building new K8s clusters and your whole infrastructure in it every time!?! Simple example - A microservice that leaks memory.... let it keep restarting as it crashes?! I remember at one of my first jobs, at a healthcare system for a hospital in India, their Java app was so poorly written that it kept leaking memory and bloated beyond GC could help and will crash every morning at around 11 AM and then again at around 3 PM. The end users - Doctors, nurses, pharmacists knew about this behavior and had breaks during that time. Absolutely bullshit engineering! It’s a shame on those that wrote that shitty code, and shame on whoever reckless to suggest a ever rebuilding K8s clusters.

peterwwillis 7y ago

If you are running K8s on a single VM, you by definition do not need K8s. Run your apps with Docker and save yourself 50 layers of complexity. > Kubernetes makes deployments really easy. docker build + kubectl apply. The parts between "docker build" and "kubectl apply" is literally CI versus CD; they're more complicated than 2 steps. And when there's a problem with either, K8s is not going to fix it up for you. You'll have to be notified via monitoring and begin picking through the 50 layers of complexity to find the error and fix it. Which is why we have deployment systems to do things like validation of all the steps and dependencies in the pipeline, so you don't end up with a broken prod deploy. > Kubernetes requires very little maintenance Whatever you're smoking, pass it down here... Have you ever had to refresh all the certs on a K8s cluster? Have you ever had to move between breaking changes in the K8s backend during version upgrade? Crafted security policies, RBACs, etc to make sure when you run the thing you're not leaving gaping backdoors into your system? There's like 50 million different custom solutions out there just for K8s maintenance. Entire businesses are built around it.

boobePhuu7iet7i 7y ago

Easy deployments/upgrades, and automagic external DNS and certificate management sealed the deal for me.

Legogris 7y ago
Which network fabric do you use and how did you set up DNS/cert management? For me certificates has been one of the pain points - have been using cert-manager with LetsEncrypt for some time but it has been notoriously unstable and they have introduced plenty of breaking changes between releases. (That being said I haven't tried the more recent releases, maybe things have gotten more stable in the past couple of months) Google recently release managed certs for those running on GKE, but those are limited to a single domain per cert.
- boobePhuu7iet7i 7y ago
  I use the external-dns and cert-manager tools. cert-manager uses lets-encrypt but fully automates everything, you just add an annotation to your ingress resource. Been using it in prod for around 6 months now with no problems.
  - Legogris 7y ago
    Ah, sounds like they’re stabilizing then - I’ve had a lot of stability and upgrading issues with older versions. Just the fact that you couldn’t configure it for automatic renewal with anything else than 24h before expiry and these renewals would fail half the time... But I will give it another try at some point.

dragonsh 7y ago

So let’s take step by step: 1. Learn shell scripts. 2. Learn Docker conf. 4. Learn yaml. (May be helm and plethora of acronyms specially designed for k8s). 5. Combine spaghetti of 1,2,3 to build container image scripts. 6. Tie it into CI/CD process for adventurous. 7. Learn to install and manage k8s cluster or learn proprietary non open source api of google or amazon or azure. 8. Constantly patch and manage plethora of infrastructure software besides application code. Now with all this use a tool designed for million user application on an application which will be used by 100’s to 1000’s of users. I think k8s is designed for google kind of problem and is an overkill for over 80-90% of deployments and applications. May be just use simple deployment with Ansible, puppet,chef, nix, gnu guix etc. to deploy and manage software based on necessity on a single vm and extend it to a large cluster if necessary of bare metal or vm or container in a cloud agnostic manner. Not sure when the technology fashion overtook the infrastructure area like complex web app tooling in JavaScript world. K8s has its own place at scale required by handful of Organization with google level traffic and load for most traditional companies Simple cluster management and configuration management tool will work wonders with less moving parts and cognitive load.

curryst 7y ago
Then you still have to learn Ansible, so you're still learning a tool. And you probably want to use Docker anyways, unless you're shipping statically linked binaries. Also, you said VM which implies that instead of Kubernetes you're going to use a VM platform, which comes with every bit of the complexity Kubernetes has. I agree with you that it's a simpler deployment method when you don't need HA. As soon as you need HA, then all of a sudden you need to be able to fail over to new nodes and manage a load balancer and the backends for that load balancer. Kubernetes makes that easy. Kubernetes makes easy things harder than they should be, and hard things easier than they should be. The number of things I've managed that don't need HA is vanishingly low.
- ar_lan 7y ago
  I think you're starting to get at what has annoyed me about a lot of the anti-k8s folks. I really just don't think they understand the tool, because any production environment should have many of the features Kubernetes helps provide. So the argument becomes "I know how to do it this other way, so learning a new tool is too complex." Kubernetes helps standardize a lot of these things - I can very easily hop between different clusters running completely different apps/topologies and have a good sense of what's going on. A mish-mash of custom solutions re-inventing these wheels is, in my opinion, far more confusing. > Kubernetes makes easy things harder than they should be, and hard things easier than they should be. This is really the crux, I think. I think a lot of people look at Kubernetes, try to learn it by running their blog on a K8s platform, and decide it's overly complex to run something that Docker probably solves (alone) for them. When you need HA for many services, don't want to have to handle the hassle of your networking management, design your applications with clear abstractions, etc., and really need to start worrying about scale, Kubernetes starts to really shine. /rant

jen20 7y ago

> Kubernetes is really really cheap. I can run 20 low volume apps in a kubes cluster with a single VM. Nomad is just as cheap, if not cheaper.

MrUnderhill 7y ago

I'm not very familiar with Kubernetes, but from what I've seen, to me it looks quite complicated. I'm wondering, when people say they use Kubernetes and consider it easy/simple, does that typically include operating the underlying Kubernetes container infrastructure as well? Or does "using Kubernetes" usually mean deploying containers to someone else's Kubernetes hosting (like Google)?

mpmpmpmp 7y ago
When I say it (and I would guess outside of people who either contribute to Kubernetes or are using it to build their own PaaS) I mean a managed service like GKE. When I attempted to set my own cluster up all the confusion was around all of the choices to make, what container networking stack, etcd, and so on. GKE gives you an easy button for simpler architectures where you just have some containers and you want them to go in a cluster.
vorpalhex 7y ago
You will probably use someone's recommended Kubernetes stack, like k3s. It doesn't make a ton of difference in practice whether you let someone else host your control plane or do it on your own hardware, but your actual nodes you probably want instance level control on. Over time you'll probably grow your own customizations and go-tos on kubernetes that you layer on top of k3s or what have you.

tonetheman 7y ago

K8 is only cheap if someone else is running it for you and you never have problems.

notyourday 7y ago

> Kubernetes is really really cheap. I can run 20 low volume apps in a kubes cluster with a single VM. This is cheaper than any other hosting solution in the cloud if you want the same level of stability and isolation. It's even cheaper when you need something like a Redis cache. If my cache goes down and the container needs to be spun up again then it's not a big issue, so for cheap projects I can even save more cost by running some infra like a Redis instance as a container too. Nothing beats that. It gets even better, I can run my services in different namespaces, and have different environments (dev/staging/etc.) isolated from each other and still running on the same amount of VMs. When you caculate the total cost saving here to traditional deployments it's just ridiculously cheap. If you're running 20 apps on a kubes cluster with a single VM you are running twenty apps on a single VM. There's no backup, scalability or anything else. There's no orchestration. your deployment is a hipster version of rsync -avH myapp mybox:/my/location/myapp followed by a restart done via http to tell monit/systemd to restart your apps. It is a perfectly fine way of handling apps. k8s shines when you have a fleet of VMs and a fleet of applications that depend on each other and have dynamic constrains but that's not what most of k8s installations work > I mean yes, theoretically nothing needs Kubernetes, because the internet was the same before we had Kubernetes, so it's certainly not needed, but it makes life a lot easier. Especially as a cheap lazy developer who doesn't want to spend time on any ops Kubernetes is really the best option out there next to serverless. Only in a throw production code over the fence sense.

jordanbeiber 7y ago

One side of this - our base k8s config is 44k lines of yaml, leading to a bunch of controllers and pods running, in a rather complex fashion. Not to mention k8s conplexity and codebase itself. It blackboxes so many things. I still chuckle at the fact that ansible is a first class citizen of the operator framwork. It can certainly implode on you! In my experience running nomad & consul is a more lightweight and simple way of doing many of the same things. It’s a bit like the discussion raging in regards to systemd and the fact that it’s not “unixy”. I get the same feeling with k8s, whereas the hashicorp stuff, albeit less features, adheres more to the unix philosophy. Thus easier to maintain and integrate. Just my ¢2.

mst 7y ago
Sort of similar to the fact that I basically always have both tmux and dtach installed, the latter for "just make this detachable", the former for "actually I'd like some features today".
- jordanbeiber 7y ago
  Something like that. I want service discovery, where Consul really shines - cause containers ain't all we're doing, mkay. K8s forces etcd on you, for service discovery, and only within the cluster. So, sync it then... more code more complexity, no "pipe" ("pipe" not to be taken literally in this context) and simple integrations. Not to mention, consul is already a full DNS server, but in k8s, we need yet another pod for coredns. Is YAP a thing? =) For example, I love how easily envoy integrates to any service discovery of you liking, even your own - simply present it with a specific JSON response from an API. Much like how the Ansible inventory works. It makes integrating, picking and choosing as well as maintaining and extending your configuration management just so much more pleasant.
vorpalhex 7y ago
> our base k8s config is 44k lines of yaml Something has gone horribly wrong. > In my experience running nomad & consul is a more lightweight and simple way of doing many of the same things. If you can replace what you're getting out of your 44k lines of yaml with those two products, then you are using the wrong tools.

etxm 7y ago

Agree / disagree. I’ve been using k8s in large deployments (1000s of nodes) for about 3 years. It’s easy to get started with using GKE, EKS, etc. it’s difficult to maintain if you’re bootstrapping your own cluster. 3 years in, and despite working with k8s at a pretty low level, I still learn more about functionality and function every single day. I do agree it’s great tooling wise. I personally deploy on docker for desktop k8s day one when starting a new project. I understand all the tooling, it’s easier than writing a script and figuring out where to store my secrets every damn time. The big caveat is - kubernetes should be _my_ burden as someone in the Ops/SRE team, but I feel like you frequently see it bleed out into application developer land. I think that the CloudRuns and Fartgates* of the world are better suited to the average developer and I think it’s Ops responsibility to make k8s as transparent as possible within the organization. Application developers want a Heroku. Edit: * LOL

t0astbread 7y ago

I've looked at k8s a while ago and it looked insanely complex to me. Maybe it's one of those things that you need to look at several times to understand it?

sjellis 7y ago
Kubernetes addresses a bunch of infrastructure concerns, so IMO it's more equivalent to learning a new cloud platform (AWS, Azure) than anything else. People who already know one or more cloud platforms, and are familiar with the kind of issues that come up with distributed and HA systems on cloud (e.g. networking, secrets management) may find Kubernetes much easier.

Rantenki 7y ago

> BTW, learning Kubernetes can be done in a few days. As someone who has deployed K8S at scale several times, this is nonsense. Learning K8S deeply enough to deploy AND MAINTAIN IT is a huge undertaking that requires an entire team to do right. Sure, you can deploy onto AWS via KOPS in a day, and you can deploy a hello world app in another day. Easy. But that only gets you to deployment, it doesn't mean you can maintain it. There are TONS of potential failure modes, and at this point you don't understand ANY of them. When one of them crops up, who do you page at 3AM, how do you know it's even down (monitoring/alerting isn't "batteries included"), how do you diagnose and fix it once you _do_ know what's broken? Not to mention the fact that you _have_ to continuously upgrade K8S as old releases go out of maintenance in under a year. If you're not continuously testing upgrades against production equivalent deploys, you're going to footgun spectacularly at some point, or be stuck with an old release that new/updated helm charts won't work against. TL;DR: If you can afford a team to deploy and maintain K8S, and have a complex enough application stack to need it, it's awesome; but it's not free in either time or staff.

UK-Al05 7y ago
How much effort is it to maintain k8s provided by a cloud service provider instead of something you have set up yourself?
- xemdetia 7y ago
  As far as I have seen there is still updating overhead where you have to initiate upgrades to a known stable/supported version at a regular cadence. EKS suggests having an E2E CI pipeline to test it before updating production, and I feel like that's the only way to do it. There is nonzero churn even though it's managed.

kissgyorgy 7y ago

It definitely can't be don in a few days. It took me 2-3 months to go through everything and run my own cluster at home. And yes, you need to go through about every major concept (architecture, object type, API, authentication, services, deployments, updates, etc, etc, etc) in order to make sense of Kubernetes overall. It's a very complex software. You can learn a simple programming language like Go in a couple of days, but definitely not Kubernetes.

Legogris 7y ago

> Kubernetes requires very little maintenance. Kubernetes takes care of itself. A container crashes? Kubes will bring it up. Do I want to roll out a new version? Kubes will do a rolling update on its own. I am running apps in kubes and for almost 2 years I haven't looked at my cluster or vms. They just run. Once every 6 months I log into my console and see that I can upgrade a few nodes. I just click ok and everything happens automatically with zero downtime. > BTW, learning Kubernetes can be done in a few days. This is true if you are using managed k8s from a provider or have an in-house team taking care of this. Far, far from the truth if you also need to set up and maintain your own clusters. I'm very comfortable with k8s as a user and developer, would not be comfortable setting up a scalable and highly available cluster for production use for anything more serious than my homelab.

gigatexal 7y ago

Docker swarm gets you this without all the complexity of Kubernetes. Hell even a largish docker-compose setup could get you there.

JeremyNT 7y ago

You can be able to use Kubernetes very quickly. If you have a manged k8s cluster available to you, that somebody manages for you, then sure it is all upside. All the stuff you describe is great! The thing is, if you have to manage the complexity and lifecycle of the cluster yourself, the balance tips dramatically. How do you provision it? How do you maintain it? How do you secure it? How do you upgrade it? So I agree, k8s is great for running all manner of project, big and small. If you already have a k8s you will find yourself wanting to use it for everything! However if you don't have one, and you aren't interested in paying somebody to run one for you, then you should think long and hard about whether you're better off just launching a docker-compose from systemd or something.

pas 7y ago
> How do you provision it? How do you maintain it? How do you secure it? How do you upgrade it? https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ https://kubernetes.io/docs/setup/production-environment/tool... https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network https://kubernetes.io/docs/setup/production-environment/tool... https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/ https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/... https://kubernetes.io/docs/concepts/security/overview/ https://kubernetes.io/docs/concepts/security/overview/ Of course, it's not as easy as a managed solution, but it's not exactly black magic. Docker compose from systemd is not bad, but maybe then instead of that using k3s is a better middle ground: https://github.com/rancher/k3s https://github.com/rancher/k3s

erling 7y ago

Have you looked into Erlang/the BEAM VM? It provides many of these same benefits. > This is cheaper than any other hosting solution in the cloud if you want the same level of stability and isolation. Erlang provides total process isolation and can theoretically also run on only a single machine. > It's even cheaper when you need something like a Redis cache. If my cache goes down and the container needs to be spun up again then it's not a big issue, so for cheap projects I can even save more cost by running some infra like a Redis instance as a container too. Nothing beats that. In Erlang, each process keeps its own state without a single point of failure (ie a single Redis instance) and can be restarted on failure by the VM. > Kubernetes takes care of itself. A container crashes? Kubes will bring it up. Erlang VM takes care of itself. A process crashes? The VM will bring it up. > Do I want to roll out a new version? Kubes will do a rolling update on its own. Ditto for Erlang VM, with zero-time deployments with hot code reloading. > I just click ok and everything happens automatically with zero downtime. Erlang is famous for its fault tolerance and nine nines of downtime! And it’s been doing so for quite a bit longer than K8s.

brozaman 7y ago

"BTW, learning Kubernetes can be done in a few days." How do you troubleshoot an api-group that is failing intermittently? How do you troubleshoot a CSI issue? Because CSI isn't a simple protocol like CNI. What do you look at if kubectl gets randomly stuck? What do you do if a node becomes notReady? What do you do if the container runtime of a several nodes starts failing? What do you do if one of the etcd nodes doesn't start? What if you are misisng logs in the log aggregator or metrics? What if when you create a project it's missing the service accounts? What if kube-proxy randomly doesn't work? What if the upgrade fails and ends up in an inconsistent state? What if your pod is actually running but shows as pending? Sure, you can learn how to deploy kubernetes and an application on top of it in a couple days, but learning how to run a production will take way longer than that.