20 ms·

Maybe You Don't Need Kubernetes

500 points by ra7 8y ago

Daishiman 8y ago
For most small deployments you are far better off using Ansible playbooks or similar solutions. Declarative orchestration management seems, for most relatively small and medium-sized deployments, a black box with an obscenely high learning curve which isn't justified if your deployment and scaling needs fall within the vast majority of use cases.
- deleted 8y ago
  [deleted]
- clhodapp 8y ago
  If you already know how the declarative management system works (and if there are other people out in the world that also know it, as is the case for Kubernetes), it seems like that is just creating yourself a less-reliable tarpit for no real reason, especially if you have to learn Ansible to do it.
- hosh 8y ago
  If someone on a small team brings K8S expertise to the table, it is worth considering. Otherwise, yeah, if you're at the stage where your team is focusing on building the product, then K8S is going to be a distraction.
  - zerkten 8y ago
    > If someone on a small team brings K8S expertise to the table, it is worth considering. Perhaps you have some unstated assumptions here like a small team managing a relatively large infrastructure for their team size? Or, a small team in a much larger infrastructure department. If it was me, I'd put lots of disclaimers around a single person bringing any kind of specialist expertise to the average small team. I think you need to lean on that person to level the whole team up. If there is any chance of that individual leaving you risk leaving the team high and dry.
    - richardknop 8y ago
      You can use managed Kubernetes. And that person with expertise would mostly influence design decisions in order to write software that is easily deployed to kubernetes and orchestrated in containers. They wouldn’t be a dev ops person taking care of k8s cluster obviously that makes no sense for a small team.
    - hosh 8y ago
      It is what I am doing for my team. And yes, if I leave, it would leave the team in a bad state. I have been incrementally cross-training two of the other engineers on this. We’re not managing our own K8S cluster. I did that once by hand at another place (before kubeadm), enough to know that managed GKE on GCP is a great thing. Also did my own single-node K8S for dev work. Nanokube and later, minikube later mare that much easier. I used to do stuff with Capistrano and Chef. Enough to know that once you know the core primitives, it is easier just to use K8S.
bribroder 8y ago
One argument that I think has been propelling kubernetes is that projects using orchestrated containers benefit from a consistent application environment, and such orchestrated deploys can be much more easily distributed when the entire stack configuration can be packaged, and it enables greater out-of-the-box functionality like autoscaling for the teams deploying these projects--the zero to jupyterhub project is a great example: https://zero-to-jupyterhub.readthedocs.io/en/latest/ https://zero-to-jupyterhub.readthedocs.io/en/latest/
AaronM 8y ago
Some fair points are made here, and I think running kubernetes the hard way, without a managed service, does introduce a lot of complexity. However, its not that bad when using a managed service such as EKS or GKE
- drdaeman 8y ago
  Is there any significant difference between e.g. `kubeadm` and `gcloud container clusters`? I believe when the cluster malfunctions, you're still on your own figuring that out. I had a stuck node on GKE just recently, which broke our CI. The GCE machine was there, but the node wasn't visible with kubeadm and quick SSH onto the VM hadn't discovered any immediately visible obvious issues. Auto-repair was enabled but hadn't worked - and TBH I have no idea how should I've diagnosed why it didn't (if there's even a way). Thankfully, this was issue with the node, not master, and the CI nodes are all epheremal, so I've quickly dismissed the idea of debugging what went wrong and had just reinitialized the pool. Could've done the same with bare metal.
  - AaronM 8y ago
    I don't know. I haven't had any random failures using EKS. Anything crashing was of my own doing.
chucky_z 8y ago
I've started heavily using Nomad recently and it's a real joy. Adding things in is just so easy because it doesn't do much and (mostly) everything is very well defined. I cannot suggest it highly enough. An example is I setup a log pipeline recently spanning multiple data centers with full mTLS. It wasn't that hard and thanks to Vault all the certs are refreshed at regular intervals all over the place. Pretty great!
- SteveNuts 8y ago
  +1 for Nomad, I'm extremely impressed with all of the Hashicorp suite. Single binary deployments and upgrades are helpful.
  - mirceal 8y ago
    even Terraform?
    - chucky_z 8y ago
      I manage everything soup to nuts with Terraform. If you know something vastly better please let me know, especially considering the changes to 0.12. It's a very well integrated system.
      - mirceal 8y ago
        what kind of infrastructure are you managing? if you’re in the aws cloud, CloudFormation is the gold standard for “infrastructure as code”.
        jugg1es 8y ago
        says you
        mirceal 8y ago
        Terraform does not automatically rollback in the face of errors. Instead, your Terraform state file has been partially updated with any resources that successfully completed. do you know who can rollback and leaves your infra in a consistent state? can you guess?
nukeop 8y ago
Pretty much just an advertisement for Nomad.
- csomar 8y ago
  I'm not sure why you are getting downvoted but I'm a bit suspicious. Either he really likes Nomad that he went ahead and put their logo on a scooter or a disclaimer is lacking.
  - omn1 8y ago
    I put the logo on a scooter. Not affiliated in any way with HashiCorp.
ekidd 8y ago
Over the years, I've deployed applications using various combinations of custom RPMs, Chef, Heroku, Docker, ECS and Kubernetes. If you can, you should probably deploy to Heroku. (Or a similar service.) It's far cheaper than spending time on devops. Just run "git push" and you're running. When I've deployed on ECS (or other "simple" orchestrators), I've found that I ultimately wound up re-inventing lots of Kubernetes features. So if you can't use Heroku, but you do know basic Docker, then it's worth considering somebody's fully-managed Kubernetes. Google's is nice. Amazon's is a considerably more work. I hear Microsoft's is still a bit sketchy. And I'd love to take a look at Digital Ocean's. But do not attempt to host your own Kubernetes if you can possibly avoid it. If you do try Kubernetes, then read a book like Kubernetes: Up & Running first. Kubernetes is not self-explanatory, but it's pretty straightforward if you're willing to spend a few days reading. Finally, don't overcomplicate it. Just use the basic stuff for as long as you can before trying to layer all sorts of other tools over it.
- nickbauman 8y ago
  Managed k8s or Heroku? I don't understand why people don't try to do AppEngine. It's the oldest "Serverless" platform out there and wickedly mature. If you're worried about vendor lock in (and you should be) I have deployed AppEngine apps on the F/OSS AppScale unmodified.
  - ngrilly 8y ago
    > I have deployed AppEngine apps on the F/OSS AppScale unmodified. How did you manage secrets on Google App Engine?
    - nickbauman 8y ago
      In the flex environment you can inject secrets at container build time. In App Engine standard I've used a deploy wrapper around ansible vault to do it.
      - ngrilly 8y ago
fred909 8y ago
+1 Nomad is great. You can run services that aren't dockerized easily.
erulabs 8y ago
It's a bit strange to say the best part of Nomad is its ecosystem, then to go on to say it has 10x less contributors than Kubernetes. Additionally, one of the benefits is... a lack of features? > The entire point of Nomad is that it does less: it doesn’t include fine-grained rights management or advanced network policies, and that’s by design. PodSecurityPolicies and advanced Network Policies are not a default or required part of Kubernetes, either...
- omn1 8y ago
  10x less contributors than Kubernetes is still over 200 contributors (actually more like 300 at this point). That's one of the bigger Open Source projects in my book. This number pales in comparison to Kubernetes, but show me a project which doesn't. That doesn't mean that the ecosystem is not vibrant, however. The HashiCorp stack is just more modular than Kubernetes. Consul and Vault are standalone products for example. With less functionality, it's harder to build a system which depends on features that only exist in a particular orchestrator. This makes it easier to replace Nomad with something better in the future. I'm not sure if I'd want to work on migrating Kubernetes clusters in five years. :)
  - erulabs 8y ago
    Sure, I get that, but saying its benefit is the ecosystem and then in the next paragraph saying its ~9x smaller than what its being compared to (hence where its benefits should shine...) Kube links services together via DNS. I'm not sure what part of it is tightly coupled really. Additionally, I make really great use of consul-k8s _and_ Vault. Not sure what they have to do with Nomad v K8s is all.
  - shaklee3 8y ago
    There's almost nothing proprietary about kubernetes. As the next post said, it uses dns. Which part are you referring to that it locks you into?
ericsoderstrom 8y ago
> Kubernetes is an Ancient Greek word meaning "More containers than customers." [1] [1] https://twitter.com/srbaker/status/1002286820078571532?lang=en https://twitter.com/srbaker/status/1002286820078571532?lang=...
guiriduro 8y ago
Any reason you didn't consider Docker swarm?
- omn1 8y ago
  I like swarm. Seems like Docker is moving away from it however. At least, that's what the commit numbers say [1]. The fact that Docker integrated Kubernetes into their desktop solutions is also telling. At this point, I would rather have a closer look at either Nomad or k3s for smaller clusters. [1]: https://github.com/docker/swarm/graphs/contributors https://github.com/docker/swarm/graphs/contributors
  - sandGorgon 8y ago
    Wrong repo. https://stackoverflow.com/questions/38474424/the-relation-between-docker-swarm-and-docker-swarmkit https://stackoverflow.com/questions/38474424/the-relation-be... Swarmkit is what is being used in production and is fairly popular. https://www.digitalocean.com/currents/june-2018/ https://www.digitalocean.com/currents/june-2018/ "While @kubernetesio was most popular overall, the smallest companies (1-5 employees) use @Docker Swarm more often (41 percent use Swarm vs. 31 percent that use Kubernetes)."
  - BretFisher 8y ago
    Yea that's the legacy "classic" swarm from pre-2016. Docker isn't moving away from Swarm, in fact they keep adding features. See https://www.bretfisher.com/the-future-of-docker-swarm/ https://www.bretfisher.com/the-future-of-docker-swarm/ Last month they released https://github.com/docker/stacks https://github.com/docker/stacks for Swarm. Also, every industry survey I've seen in the last year on orchestrator uses shows increases in overall usage of K8s and Swarm, which I consider the better measure of a project's health. (CNCF, Digital Ocean, F5, Sysdig)
- richardknop 8y ago
  Because it’s a dead technology not really maintained anymore. Docker has pivoted to k8s so it’s a bad idea to build new projects to run on swarm imho.
nerdbaggy 8y ago
At work we have about 50 containers, which total RAM usage less than 32GB and about 10 CPU cores. Docker swarm works great for us
- sandGorgon 8y ago
  Same here. We use Swarm and it works great for us.
- marmaduke 8y ago
  I was scrolling down the comments for Swarm. Do you run multimachine? How does storage work? (I’m looking for an container thingy for bare metal)
  - nerdbaggy 8y ago
    Yup multimachine. It works fantastic. We have a netapp san https://hub.docker.com/plugins/trident https://hub.docker.com/plugins/trident But we don’t really use it except for databases
echohack5 8y ago
Have you looked at https://habitat.sh https://habitat.sh ?
- omn1 8y ago
  I actually did and I love the idea! In my opinion, that's gonna be the next wave of container management solutions. If you think about it, you shouldn't care about container orchestration at all; you should care about services. Habitat ships the orchestrator together with every service. That sounds strange and redundant at first, but if you think about it, it has some nice advantages: * Services are fully self-contained and can run anywhere (in dev, stage, prod) without any changes. * By tagging services, one can create a service-mesh quite easily without having to think about network-topologies. Services find each other automatically (I think via a Raft/Gossip protocol). I haven't looked at it too closely yet, but it's on my list. Fair disclaimer: I know the folks behind it from the Rust community.
- mr_tristan 8y ago
  I took a look at it, and if you were starting from scratch, I'd give it a serious look. The main reason I didn't push for it were mostly internal politics - we'd have to setup an on-premise installation of the Habitat Builder. And we have rather (ridiculous) requirements that would have made that a massive hurdle. Habitat is very new, so I'd expect to spend some time nailing down exactly what you want for your stuff. But, it's very easy to use.
doublerebel 8y ago
I've been running SmartOS containers for 6+ years and it's the best of all worlds. Easy service management, works with regular binaries or Dockerized deploys, SDN always built-in, service discovery built-in (CNS), even Windows VMs available. Best kept secret in the cloud even though the entire stack is open-source. On the orchestration side I use Puppet Pipelines (formally Distelli) which works with standard VMs, SmartOS containers, Docker, Linux images, from packaging through testing and deployment. And they just lowered pricing (!??!) There's no reason to be locked into Kubernetes unless you're sure you need it. SmartOS scales in seconds. As CTO I'm always checking out the next platform, but all the newer solutions look much more complicated. EDIT: Heroku has weird errors when you push CPU or RAM, also the CPUs aren't so fast, Linode can't be trusted, Digital Ocean is OK but is still very manual roll-your-own, AWS is a behemoth and medium fast, Google is fine but specialized and hard to migrate away from, OpenStack isn't for small orgs... I've run production services on them all. It's easy to try a new service with a Pipelines-style orchestration because it doesn't care which service its talking to. And makes it obvious when platform-specific instructions or concessions are required.
- sansnomme 8y ago
  This looks interesting, I have a few questions. How do you deal with Linux dependencies etc.? All your applications must be BSD compatible? Any advice for getting started with it on DigitalOcean? Or is nested out of the story?
  - doublerebel 8y ago
    Oh, SmartOS containers run Linux seamlessly too. I guess I take that knowledge for granted now. But many many of the popular cloud tools (hashicorp consul and vault, redis, couchdb, arangodb, postgres) run natively on SmartOS and have better resource and user management from the underlying system. I haven't used Digital Ocean's kube offering. Otherwise it's much like Linode or other services from that era -- install your own distro and binaries, which needs some kind of orchestration automation to scale.
    - 8y ago
freedomben 8y ago
The author mentions coupling with kubernetes due to config maps and such. That is certainly a risk, but if you use environment variables for config (which IMHO you should) then k8s makes it trivial to just map your ConfigMap to Environment Variables. I've got a few apps running in k8s and none of them know anything about k8s. If avoiding coupling with k8s is what you want (and I agree it's the way to go) then it really isn't that bad. The hard part is not knowing what you don't know.
- xyzzy123 8y ago
  Or just mount the config map as a file.
- snupples 8y ago
  I think the real question then is why are people running apps that need to be k8s-aware at all? K8s already provides a number of primitives to solve many problems without reinventing the wheel and going "cloud-native" within the application. There should be a very strong reason for any need to access the api directly.
jjeaff 8y ago
The thing that all the "you don't need kubernetes" articles seem to ignore or skim over, is the huge network effect of a popular setup like k8s. Not only is it much easier to find services (like helm) and articles for k8s, but there are so many ready to go configurations. For most major services, there is already a ready to go helm chart or at least some well vetted yml config on GitHub to start from. And most any issue I have come across, I have been able to search and find solutions.
- omn1 8y ago
  Haven't looked into helm-charts deeply, but the ones that I found often needed a considerable amount of time to configure for production. For example, the Logstash chart uses a YAML config whereas we use Logstash's standard config format with many filters and options. Porting these features over to YAML is not a lot of fun. And in the end we would also have to maintain these configs and keep them in sync with upstream changes. Speaking of documentation, I sometimes feel a bit overwhelmed by the amount of literature on Kubernetes these days. Many articles are quite outdated already and new ones get written every day. It's hard to know what the best practices are - and the ecosystem is still rapidly evolving. So I don't think I was skimming over these parts, I just think that inertia can also become a disadvantage.
- paxys 8y ago
  As well as hiring engineers who will be familiar with it from day 1, as opposed to having to learn whatever custom system you use.
  - JMTQp8lwXL 8y ago
    This. In a perfect world, it should be more useful to say to an employer, "I know Kubernetes" than "I know AWS". It's a unified abstraction for cloud resources. They all, more or less, do the same thing. Why learn the proprietary, vendor-specific way to do it, when you learn it once, and apply your knowledge to any cloud, e.g, if you switch jobs and your new one uses a different cloud provider.
    - 8y ago
openloop 8y ago
Maybe you do.
sdan 8y ago
I spent a few days developing a Kubernetes Cluster with some Docker knowledge. Other than creating the config file, I didn't see too much of a problem. It definitely has a learning curve and takes a lot of time to setup (even with GKE). From reading this article, I'm liking Nomad.
ilaksh 8y ago
I guess I am a little behind the times, but no one uses the orchestration stuff that comes with Docker?
- twblalock 8y ago
  I know people who use Docker Compose to run Docker containers on their laptops. I don't think Compose or Swarm were ever nearly as popular as Kubernetes for running services in production. One of the signs that Kubernetes became the industry standard is that its competitors feel the need to promote their compatibility with it. Docker for Mac now ships with a built-in Kubernetes cluster. (I prefer it to Minikube).
- andbberger 8y ago
  I use it a fair bit for development, provides a seemingly simple way to keep things organized and do 'infra-as-code' and have your ducks in a row with reproducible stack deployments, without the overwhelming overhead of k8s. It's an absolute dumpster fire though. A serious convoluted mess. They try to indelicately cram something with incredible intrinsic complexity into a happy little (incredibly leaky) abstraction. docker compose is really what I want, simple service management on one machine. Except, it can't do really basic shit like restart services (this isn't all that obvious at first). So soon you'll find that you need to use docker swarm on the one node, which is similarly chock full of inexplicable and unforeseeable shortcomings. docker swarm/compose is an interim step on your journey to a better solution. in my judgement it's fine while you're developing your core apps. but once you need to do get serious and go into production, it's time to find a big boy tool
  - ngrilly 8y ago
    I did the same thing. Docker Compose was what I wanted for a single-machine deployment, but I soon discovered it was unable to manage a rolling deploy (start new containers before stopping the old ones). So I switched to Docker Swarm on a single node, but I have to agree there is a lot of unexpected shortcomings. I'd like it to be more polished. And I'd like Docker Compose and Docker swarm mode to converge/merge at some point. I guess the "big boy tool" you adopted after that is k8s?
gouggoug 8y ago
The argument that Kubernetes adds complexity is, in my opinion, bogus. Kubernetes is a "define once, forget about it" type of infrastructure. You define the state you want your infrastructure to and Kubernetes takes care of maintaining that state. Tools like Ansible and Puppet, as great as they are, do not guarantee your infrastructure will end up in the state you defined and you easily end up with broken services. The only complexity in kubernetes is the fact that it forces you to think and carefully design your infra in a way people aren't used to, yet. More upfront, careful thinking isn't complexity. It can only benefit you in the long run. There is, however, a learning curve to Kubernetes, but it isn't this sharp. It does require you to sit down and read the doc for 8 hours, but that a small price to pay. A few month back I wrote a blog post[1] that, through walking through the few different infrastructures my company experimented with over the years, surfaces many reasons one would want to use [a managed] Kubernetes. (For a shorter read, you can probably start at reading at [2]) [1]: https://boxunix.com/post/bare_metal_to_kube https://boxunix.com/post/bare_metal_to_kube [2]: https://boxunix.com/post/bare_metal_to_kube/#_hardware_infrastructure https://boxunix.com/post/bare_metal_to_kube/#_hardware_infra...
- JMTQp8lwXL 8y ago
  This has been my experience. It isn't that complex, just a steep learning curve.
  - TooCleverByHalf 8y ago
    If complexity isn't the reason for the steep learning curve what is?
    - JMTQp8lwXL 8y ago
      The documentation around it isn't that great. Once you know the key actors -- which are a dozen or so key types of resources, and how they play together, you understand how to create the configuration for your environment. The YAML files are quite intimidating at first, if you look at examples out there. I'm probably not a great teacher, but I think I could boil it down to a slide deck of no more than 25, and an hour's worth of time, and clearly teach how to interface with Kubernetes for a basic web app w/ horizontal scaling, Let's Encrypt SSL certificates, etc.
jugg1es 8y ago
So many comments here are touting the benefits of this, that or the other, but aren't mentioning Nomad at all. Nomad is definitely worth a try even for large deployments. Yea, it doesn't autoscale without 3rd party tools, but you don't always need that. If you just need to keep your jobs running, load balance to those apps, use externalized config and need a simple UI view system status, then Nomad/Consul/Fabio really works great.
mancerayder 8y ago
I'm attempting to 'transition' -- as a DevOps / infra guy professionally rather than pure dev -- does anyone have any recommended resources for reading non-dry, fairly example-filled docs to get started? The Kubernetes O'Reilly book seems to be very broad and overly high-level from the reviews (and I remember best if I read "why" and "how"). Thanks in advance
- dankohn1 8y ago
  This is particularly well-liked for a more in-depth review of Kubernetes: https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way
xaduha 8y ago
Docker Compose is great, Docker Swarm isn't.
ashton314 8y ago
Rancher [1] looks really nice for a guided use of Kubernetes. I saw a demo on how to use this with Digital Ocean's Kubernetes clusters; the presenter went from having a blog in a container to having the blog on a K8s cluster behind a domain name in 30-ish minutes. Anybody using Rancher? [1]: https://rancher.com https://rancher.com
- dmlittle 8y ago
  Do you have any info regarding Rancher pricing? There is no info on their website other than reaching out to them...
  - nhumrich 8y ago
    Its free. You pay for support.
  - V99 8y ago
    (Rancher employee) All the software is free to use and open source (Apache 2). Customers run the same software that's on GitHub/DockerHub. We sell "enterprise support", which means something like "starting at mid-5 figures", and occasionally professional services engagements around/outside the product.
- seabrookmx 8y ago
  We're doing a sizable migration from Rancher 1.6 (which uses their internal orchestration system called "Cattle") to Rancher 2.x (which sits ontop of k8s). Rancher 2 definitely streamlines the setup of an on-premise cluster. We use AWS for 80% of our stuff and have in-house servers for the remaining 20%, so it allows us to have a single pane of glass for all of our workloads (the AWS stuff being an EKS cluster "imported" into Rancher). As a company they definitely seem to follow the "move fast and break things" mantra though. We've seen quite a few boneheaded bugs in the API and CLI tool.. and the fact that 1.x -> 2.x has no real migration path other than "rewrite all your app configurations as helm charts and you'll be fine!" is kind of a bitter pill. In reality the networking and storage side of things is vastly different between Cattle and k8s. We really felt the limitations of Cattle though, so I understand that this move is necessary. Had they not hopped on the k8s bandwagon we probably would have eventually without them. I guess it's just the price you pay for being an early adopter!
rbjorklin 8y ago
I’ve come to much the same conclusions as the author in this blog post and have put together a small Nomad demo in case someone is interested: https://github.com/rbjorklin/nomad-demo https://github.com/rbjorklin/nomad-demo
sbhn 8y ago
Yes i do, https://homeidea3d.com https://homeidea3d.com runs on kubernetes
joeblow21 8y ago
Sounds like you could just move to a PaaS like App Engine and either downsize or free up a lot of engineering staff for better uses of their time than devops
kissgyorgy 8y ago
I'm pretty sure you don't need Kubernetes.
bifrost 8y ago
I'm very much in the camp of "Kubernetes is a problem factory". Personally, I've found very few problems that it solves and more problems that it creates. That said, its probably a great time to be a Kube consultant!
- mirceal 8y ago
  if your team is only a handful of devs who work on the thing that pays the bills + do k8s, yes. bad time. the only way i could justify running this is if you’re using a managed solution like gke or eks. otoh, if you’re in the cloud, you already have the primitives to build your stuff w/o the k8s headache (vms, autoscalling, managed svcs). but that’s just my opinion. the koolaid is strong.
  - bifrost 8y ago
    Honestly, if you have a $GENERIC_SYSTEM that you deploy your code to, you don't need the wasteful overhead. If you need to (vs want to) rebuild your OS/Environment regularly, you're doing it wrong. IE: PaaS is for you. That said, I've found replicating OS & Environment extremely challenging on certain operating systems so I understand the appeal. I also don't build products on those operating systems and I've found myself much happier :)
    - mirceal 8y ago
      yeah. it depends a lot and k8s has its place in some scenarios, but in most cases you don’t need it and its use is purely driven by koolaid
squid3 8y ago
Maybe all you need is NodeChef. Saves you time and money. https://www.nodechef.com/ https://www.nodechef.com/
deleted 8y ago
[deleted]
eric_khun 8y ago
anyone got recommendation on how to properly set CPU limits? I've set limits about 2x higher than the max cpu usage, but still seeing some throttling, what cause performance degradation. I've ended up setting deployments without limits to avoid throttling but that's definitely not nice if one pod goes crazy for a reason ...
- oso2k 8y ago
  Not sure what you mean by throttling. I tell my customers to do 4 things 1) Set a Request at about 50% of max CPU usage 2) Set a Limit at about 90% of max CPU usage 3) Configure a HorizontalPodAutoscaler that kicks in at 70% 4) Scale out, in advance, to 2 or 3 pod replicas
kissgyorgy 8y ago
> It takes a fair amount of time and energy to stay up-to-date with the best practices and latest tooling. Kubectl, minikube, kubeadm, helm, tiller, kops, oc - the list goes on and on. When you start with a technology, you need to have a solid basis. You need to fully grasp the given technology in it's most basic forms, stuck with the problems for days so you can fully grasp what any of these can solve what problem. If you know "naked" Kubernetes inside and out, then you can add other things to your toolset. Until then, you don't need no helm, tiller, kops, Knative or any of those distractions.
- tarikjn 8y ago
  Isn't knative exactly supposed to solve this problem by giving a portable Heroku-like dev experience?
  - kissgyorgy 8y ago
    Doesn't matter. I talked about the initial learning about Kubernetes. It's just too much to learn everything at once.
    - jacques_chester 8y ago
      Restating: the point is to not have to "learn everything". If your platform makes you do hours of homework, then it's not a platform, it's a pallet of bricks. Heroku got this right with `git push heroku master`. IMO Cloud Foundry got it even more right with `cf push`. Knative is likely to get to that point, probably with the same means as Heroku and Cloud Foundry: Buildpacks. I'm wildly biased of course, having worked on Cloud Foundry, Buildpacks and Knative.
draw_down 8y ago
It’s interesting to see the differing reactions to these articles here. Maybe you don’t need Kubernetes = good, maybe you don’t need jQuery = bad.
TomBombadildoze 8y ago
This starts with a lot of "you don't need Kubernetes" and then concludes with a pretty compelling argument in favor of using Kubernetes. From the "The Nomad ecosystem of loosely coupled components" section: > It integrates very well with other - completely optional - products like Consul (a key-value store) or Vault (for secrets handling). > At trivago, we tag all services, which expose metrics, with trv-metrics. This way, Prometheus finds the services via Consul and periodically scrapes the /metrics endpoint for new data. > The same can be done for logs by integrating Loki for example. > Trigger a Jenkins job using a webhook and Consul watches to redeploy your Nomad job on service config changes. shudder > Use Ceph to add a distributed file system to Nomad. > Use fabio for load balancing. And the icing on the cake: > All of this allowed us to grow our infrastructure organically without too much up-front commitment. So if I understand correctly, the author (and his team) preferred to do all the work of integrating/testing/debugging those components, rather than using a tool that provides every single on of those features, and more, out of the box. Kubernetes isn't a trivial lift but it's a damn sight easier than trying to roll a cheap imitation yourself.
- millettjon 8y ago
  We use nomad because it's flexibility in this regard lets us slowly migrate away from our old infrastructure. Moving fully to Kubernetes would not be politically feasible in the organization.
  - lugg 8y ago
    This is a post about why I don't need kubernetes. Not a post about why kubernetes isn't politically feasible in not my organisation.
- omn1 8y ago
  The point is that Nomad allows to add more components as they are needed while Kubernetes is an all-or-nothing solution. Also, we used tools like Jenkins and Ceph before. They have been tested and we trust them so there's no point in replacing them. Quite the contrary: a lot of internal processes depend on them so it would have been painful to migrate away from them. Nomad is not a cheap imitation of Kubernetes, it is a simple orchestrator which favors composability over an all-in-one approach.
humbleMouse 8y ago
If people would spend 1/10th the time they spent complaining about kubernetes actually learning kubernetes, maybe more people would know kubernetes and realize its not even that difficult to administrate.
trpc 8y ago
just another provocative post title by someone wants to bash something he has no clue about
inopinatus 8y ago
All any of us really need to do is put files on a server and start a process. The question is how much scaffolding you want to build and manage to achieve it. The answer should be “as little as possible”.
AnthonyWnC 8y ago
The overall trend of IT infrastructure goes from tradition DC -> IaaS -> PaaS -> FaaS. k8 sits at PaaS and the logically next step is FaaS/serverless. In fact, today you can surely do purely FaaS without the need k8 at all by leveraging cloud provider managed services for any long-live type of workload. Of course, the tradeoffs will the usual suspects: flexibility, control, cost etc. There are still cases where you would want a cluster manager like k8 (at least for now).
namelosw 8y ago
1. Always think about Heroku first, seriously. Because your need is deploy application. 2. If you thought about 1 but Heroku is really not for you, go Kubernetes directly. It's not that hard at all. 3. Choose any other stuff only when you know what you're doing. Kubernetes is not simple or good. The api is mostly cumbersome. But it's the first thing that makes sense in this field. Before Kubernetes there are just toys or rocket science.
jordanbeiber 8y ago
I did an eval about 4 years ago of schedulers and service discovery software. Lot’s of cool stuff around (dc/os, triton, hashi, k8s, flynn etc)! After 2 days with k8s and not so much as a bootstrapped cluster the turn went to hashicorp. Within 10 min I had a container running, registered in service disc (consul) on my laptop. We ended up using the hashicorp stack in production with great success. Sure, some custom services was needed (service managment, scaling and such). Running primarily on prem the complete simpleness and un-opiniated approach is an edge. It allowed us to implement MACVLAN at first, and the ubuntu FAN-network and integrate totally with existing infrastructure. Now having spent the last 8 months implementing k8s I’m torn. I’ve built from scratch, used kubeadm, kops and now EKS. From 1.13 kubeadm honestly works really well and is my prefered way of deploying k8s. Still it’s a beast... running large deployments with many teams... there’s just so much... stuff. One GH issue leading down a rabbit hole of other GH and gists with conflicting or not working configurations. I’ve had co-workers bail on the project after mere hours of digging through code and GH discussions and SIG docs. Nomad/consul and it’s concise docs is a breeze in comparision. Torn. Cause I see the point of k8s, just not sure about the project. :)
marmaduke 8y ago
I’m curious how this compares to pacemaker and corosync for core infra devices like a active/passive failover for SAN or a virtual IP. It doesn’t seem like they’re in the same category in terms of typical usage, but both can handle the notion of a cluster.
auslander 8y ago
Should we use containers at all is the first question. Why people skip over this one ? :)
gerbilly 8y ago
I currently use Ansible to provision and configure about 50 instances with various home grown services deployed on them, and I'm quite happy with that. I don't use Docker, or Kubernetes. What am I missing here? This is an honest question. I don't particularly see the point of docker in production. I guess it can help resolve clashes between dependencies, but I'm doing ok with Ansible. One guy in another team just tells his application developers to throw everything into a docker container and then deploys whatever they give him. I guess that prevents dependency clashes, sure, but to me that seems like it's just inviting different problems though. (Containers with a lot of disorganized junk in them). And we don't need to grow the cluster for now, so I don't see the point of k8. We already have log aggregation, and service restarting, service monitring/metrics. To me, Kubernetes feels like a brand new alternative OS with crappy documentation. On the other hand I've been managing UNIX machines for 20+ years, and that I do know how to fix (mostly, since they keep changing linux so much these days, it's almost as hard to keep up with as javascript). If somebody could tell me what I'm missing by using this approach, I'd be grateful.
- auslander 8y ago
  Because it's cool, new and good for your blog/CV. I don't see the need too, if stuff is in cloud.
- originalvichy 8y ago
  You aren't "missing" anything, you are just doing things differently. Containers are a way of storing code and configurations in a single deployable entity. Let's say I want to change a services version from 1.2 to 1.3. If I did this in Ansible, I might do it by commanding all of my services to pull the code of version 1.3 from a git repo and restarting to make the change. If I did this with containers, I'd tell them to pull the image version 1.3 from docker registry and after it's pulled to shut down v.1.2 and start v.1.3. Those things just have some smaller differences that are good in terms of A and worse in terms of B. Ansible helps you by keeping it closer to the traditional way of just pushing code changes to a machine and that's that, and Docker opens up this world where the infrastructure can scale up or down (elasticity) quicker horizontally. Kubernetes is just an extrapolation of that elasticity. Ansible is "good enough" if you don't need elasticity or you have programmed another custom solution to handle that. Docker/Kube and Ansible both have Git at their hearts, so if your app components have all they need as code, and you don't worry about elasticity, you won't gain a lot from wrapping it up in a Dockerfile (as opposed to just git cloning).
jpomykala 8y ago
I only use Docker, because I'm too lazy to learn Kuberneres or Ansible. :)
reilly3000 8y ago
Its usually when these kinds of articles come out that their subjects are finally worth paying attention to. Immutable infrastructure is the right thing to do. Maybe K8's isn't the best or most convenient way to achieve that, but it gets you almost all the way there to fully declarative ops. I really think there is a lot of room to improve the ergonimics. That said, there is a business/project opportunity. I want to be able to build a values.yml file in a gui, on my phone, with bounded sliders and smart recommendations for values, along with contextual help and community templates. We have chart repos, but the default values suck for production, and learning what the values are and should be set to for your use case leads to a painful traversal of abstractions. Maybe there is already a project or vendor out there I'm missing?
he0001 8y ago
I’ve been using Kubernetes for a while (1,5y). And I can’t say I’m very impressed. For anything simple it works fairly well. But as soon you get outside the “normal” workflow it gets downright hard. Certain areas are opaque AF and if things doesn’t work it almost impossible to find out why. And if it’s impossible to fix something fast if there’s something burning. I think that, there’s an irony to k8s, it’s supposed to fix the “hard problems” of clusters. But it really only fixes the easy part of them, and obscures the rest.
hjacobs 8y ago
Triggered by Matthias' blog post, I wrote down my thoughts on the topic here: https://srcco.de/posts/why-kubernetes.html https://srcco.de/posts/why-kubernetes.html

500 points by ra7 8y ago

For most small deployments you are far better off using Ansible playbooks or similar solutions. Declarative orchestration management seems, for most relatively small and medium-sized deployments, a black box with an obscenely high learning curve which isn't justified if your deployment and scaling needs fall within the vast majority of use cases.

deleted 8y ago
[deleted]
clhodapp 8y ago
If you already know how the declarative management system works (and if there are other people out in the world that also know it, as is the case for Kubernetes), it seems like that is just creating yourself a less-reliable tarpit for no real reason, especially if you have to learn Ansible to do it.
hosh 8y ago
If someone on a small team brings K8S expertise to the table, it is worth considering. Otherwise, yeah, if you're at the stage where your team is focusing on building the product, then K8S is going to be a distraction.
- zerkten 8y ago
  > If someone on a small team brings K8S expertise to the table, it is worth considering. Perhaps you have some unstated assumptions here like a small team managing a relatively large infrastructure for their team size? Or, a small team in a much larger infrastructure department. If it was me, I'd put lots of disclaimers around a single person bringing any kind of specialist expertise to the average small team. I think you need to lean on that person to level the whole team up. If there is any chance of that individual leaving you risk leaving the team high and dry.
  - richardknop 8y ago
    You can use managed Kubernetes. And that person with expertise would mostly influence design decisions in order to write software that is easily deployed to kubernetes and orchestrated in containers. They wouldn’t be a dev ops person taking care of k8s cluster obviously that makes no sense for a small team.
  - hosh 8y ago
    It is what I am doing for my team. And yes, if I leave, it would leave the team in a bad state. I have been incrementally cross-training two of the other engineers on this. We’re not managing our own K8S cluster. I did that once by hand at another place (before kubeadm), enough to know that managed GKE on GCP is a great thing. Also did my own single-node K8S for dev work. Nanokube and later, minikube later mare that much easier. I used to do stuff with Capistrano and Chef. Enough to know that once you know the core primitives, it is easier just to use K8S.

bribroder 8y ago

One argument that I think has been propelling kubernetes is that projects using orchestrated containers benefit from a consistent application environment, and such orchestrated deploys can be much more easily distributed when the entire stack configuration can be packaged, and it enables greater out-of-the-box functionality like autoscaling for the teams deploying these projects--the zero to jupyterhub project is a great example: https://zero-to-jupyterhub.readthedocs.io/en/latest/ https://zero-to-jupyterhub.readthedocs.io/en/latest/

AaronM 8y ago

Some fair points are made here, and I think running kubernetes the hard way, without a managed service, does introduce a lot of complexity. However, its not that bad when using a managed service such as EKS or GKE

drdaeman 8y ago
Is there any significant difference between e.g. `kubeadm` and `gcloud container clusters`? I believe when the cluster malfunctions, you're still on your own figuring that out. I had a stuck node on GKE just recently, which broke our CI. The GCE machine was there, but the node wasn't visible with kubeadm and quick SSH onto the VM hadn't discovered any immediately visible obvious issues. Auto-repair was enabled but hadn't worked - and TBH I have no idea how should I've diagnosed why it didn't (if there's even a way). Thankfully, this was issue with the node, not master, and the CI nodes are all epheremal, so I've quickly dismissed the idea of debugging what went wrong and had just reinitialized the pool. Could've done the same with bare metal.
- AaronM 8y ago
  I don't know. I haven't had any random failures using EKS. Anything crashing was of my own doing.

chucky_z 8y ago

I've started heavily using Nomad recently and it's a real joy. Adding things in is just so easy because it doesn't do much and (mostly) everything is very well defined. I cannot suggest it highly enough. An example is I setup a log pipeline recently spanning multiple data centers with full mTLS. It wasn't that hard and thanks to Vault all the certs are refreshed at regular intervals all over the place. Pretty great!

SteveNuts 8y ago
+1 for Nomad, I'm extremely impressed with all of the Hashicorp suite. Single binary deployments and upgrades are helpful.
- mirceal 8y ago
  even Terraform?
  - chucky_z 8y ago
    I manage everything soup to nuts with Terraform. If you know something vastly better please let me know, especially considering the changes to 0.12. It's a very well integrated system.
    - mirceal 8y ago
      what kind of infrastructure are you managing? if you’re in the aws cloud, CloudFormation is the gold standard for “infrastructure as code”.
      - jugg1es 8y ago
        says you
        mirceal 8y ago
        Terraform does not automatically rollback in the face of errors. Instead, your Terraform state file has been partially updated with any resources that successfully completed. do you know who can rollback and leaves your infra in a consistent state? can you guess?

nukeop 8y ago

Pretty much just an advertisement for Nomad.

csomar 8y ago
I'm not sure why you are getting downvoted but I'm a bit suspicious. Either he really likes Nomad that he went ahead and put their logo on a scooter or a disclaimer is lacking.
- omn1 8y ago
  I put the logo on a scooter. Not affiliated in any way with HashiCorp.

ekidd 8y ago

Over the years, I've deployed applications using various combinations of custom RPMs, Chef, Heroku, Docker, ECS and Kubernetes. If you can, you should probably deploy to Heroku. (Or a similar service.) It's far cheaper than spending time on devops. Just run "git push" and you're running. When I've deployed on ECS (or other "simple" orchestrators), I've found that I ultimately wound up re-inventing lots of Kubernetes features. So if you can't use Heroku, but you do know basic Docker, then it's worth considering somebody's fully-managed Kubernetes. Google's is nice. Amazon's is a considerably more work. I hear Microsoft's is still a bit sketchy. And I'd love to take a look at Digital Ocean's. But do not attempt to host your own Kubernetes if you can possibly avoid it. If you do try Kubernetes, then read a book like Kubernetes: Up & Running first. Kubernetes is not self-explanatory, but it's pretty straightforward if you're willing to spend a few days reading. Finally, don't overcomplicate it. Just use the basic stuff for as long as you can before trying to layer all sorts of other tools over it.

nickbauman 8y ago
Managed k8s or Heroku? I don't understand why people don't try to do AppEngine. It's the oldest "Serverless" platform out there and wickedly mature. If you're worried about vendor lock in (and you should be) I have deployed AppEngine apps on the F/OSS AppScale unmodified.
- ngrilly 8y ago
  > I have deployed AppEngine apps on the F/OSS AppScale unmodified. How did you manage secrets on Google App Engine?
  - nickbauman 8y ago
    In the flex environment you can inject secrets at container build time. In App Engine standard I've used a deploy wrapper around ansible vault to do it.
    - ngrilly 8y ago

fred909 8y ago

+1 Nomad is great. You can run services that aren't dockerized easily.

erulabs 8y ago

It's a bit strange to say the best part of Nomad is its ecosystem, then to go on to say it has 10x less contributors than Kubernetes. Additionally, one of the benefits is... a lack of features? > The entire point of Nomad is that it does less: it doesn’t include fine-grained rights management or advanced network policies, and that’s by design. PodSecurityPolicies and advanced Network Policies are not a default or required part of Kubernetes, either...

omn1 8y ago
10x less contributors than Kubernetes is still over 200 contributors (actually more like 300 at this point). That's one of the bigger Open Source projects in my book. This number pales in comparison to Kubernetes, but show me a project which doesn't. That doesn't mean that the ecosystem is not vibrant, however. The HashiCorp stack is just more modular than Kubernetes. Consul and Vault are standalone products for example. With less functionality, it's harder to build a system which depends on features that only exist in a particular orchestrator. This makes it easier to replace Nomad with something better in the future. I'm not sure if I'd want to work on migrating Kubernetes clusters in five years. :)
- erulabs 8y ago
  Sure, I get that, but saying its benefit is the ecosystem and then in the next paragraph saying its ~9x smaller than what its being compared to (hence where its benefits should shine...) Kube links services together via DNS. I'm not sure what part of it is tightly coupled really. Additionally, I make really great use of consul-k8s _and_ Vault. Not sure what they have to do with Nomad v K8s is all.
- shaklee3 8y ago
  There's almost nothing proprietary about kubernetes. As the next post said, it uses dns. Which part are you referring to that it locks you into?

ericsoderstrom 8y ago

> Kubernetes is an Ancient Greek word meaning "More containers than customers." [1] [1] https://twitter.com/srbaker/status/1002286820078571532?lang=en https://twitter.com/srbaker/status/1002286820078571532?lang=...

guiriduro 8y ago

Any reason you didn't consider Docker swarm?

omn1 8y ago
I like swarm. Seems like Docker is moving away from it however. At least, that's what the commit numbers say [1]. The fact that Docker integrated Kubernetes into their desktop solutions is also telling. At this point, I would rather have a closer look at either Nomad or k3s for smaller clusters. [1]: https://github.com/docker/swarm/graphs/contributors https://github.com/docker/swarm/graphs/contributors
- sandGorgon 8y ago
  Wrong repo. https://stackoverflow.com/questions/38474424/the-relation-between-docker-swarm-and-docker-swarmkit https://stackoverflow.com/questions/38474424/the-relation-be... Swarmkit is what is being used in production and is fairly popular. https://www.digitalocean.com/currents/june-2018/ https://www.digitalocean.com/currents/june-2018/ "While @kubernetesio was most popular overall, the smallest companies (1-5 employees) use @Docker Swarm more often (41 percent use Swarm vs. 31 percent that use Kubernetes)."
- BretFisher 8y ago
  Yea that's the legacy "classic" swarm from pre-2016. Docker isn't moving away from Swarm, in fact they keep adding features. See https://www.bretfisher.com/the-future-of-docker-swarm/ https://www.bretfisher.com/the-future-of-docker-swarm/ Last month they released https://github.com/docker/stacks https://github.com/docker/stacks for Swarm. Also, every industry survey I've seen in the last year on orchestrator uses shows increases in overall usage of K8s and Swarm, which I consider the better measure of a project's health. (CNCF, Digital Ocean, F5, Sysdig)
richardknop 8y ago
Because it’s a dead technology not really maintained anymore. Docker has pivoted to k8s so it’s a bad idea to build new projects to run on swarm imho.

nerdbaggy 8y ago

At work we have about 50 containers, which total RAM usage less than 32GB and about 10 CPU cores. Docker swarm works great for us

sandGorgon 8y ago
Same here. We use Swarm and it works great for us.
marmaduke 8y ago
I was scrolling down the comments for Swarm. Do you run multimachine? How does storage work? (I’m looking for an container thingy for bare metal)
- nerdbaggy 8y ago
  Yup multimachine. It works fantastic. We have a netapp san https://hub.docker.com/plugins/trident https://hub.docker.com/plugins/trident But we don’t really use it except for databases

echohack5 8y ago

Have you looked at https://habitat.sh https://habitat.sh ?

omn1 8y ago
I actually did and I love the idea! In my opinion, that's gonna be the next wave of container management solutions. If you think about it, you shouldn't care about container orchestration at all; you should care about services. Habitat ships the orchestrator together with every service. That sounds strange and redundant at first, but if you think about it, it has some nice advantages: * Services are fully self-contained and can run anywhere (in dev, stage, prod) without any changes. * By tagging services, one can create a service-mesh quite easily without having to think about network-topologies. Services find each other automatically (I think via a Raft/Gossip protocol). I haven't looked at it too closely yet, but it's on my list. Fair disclaimer: I know the folks behind it from the Rust community.
mr_tristan 8y ago
I took a look at it, and if you were starting from scratch, I'd give it a serious look. The main reason I didn't push for it were mostly internal politics - we'd have to setup an on-premise installation of the Habitat Builder. And we have rather (ridiculous) requirements that would have made that a massive hurdle. Habitat is very new, so I'd expect to spend some time nailing down exactly what you want for your stuff. But, it's very easy to use.

doublerebel 8y ago

I've been running SmartOS containers for 6+ years and it's the best of all worlds. Easy service management, works with regular binaries or Dockerized deploys, SDN always built-in, service discovery built-in (CNS), even Windows VMs available. Best kept secret in the cloud even though the entire stack is open-source. On the orchestration side I use Puppet Pipelines (formally Distelli) which works with standard VMs, SmartOS containers, Docker, Linux images, from packaging through testing and deployment. And they just lowered pricing (!??!) There's no reason to be locked into Kubernetes unless you're sure you need it. SmartOS scales in seconds. As CTO I'm always checking out the next platform, but all the newer solutions look much more complicated. EDIT: Heroku has weird errors when you push CPU or RAM, also the CPUs aren't so fast, Linode can't be trusted, Digital Ocean is OK but is still very manual roll-your-own, AWS is a behemoth and medium fast, Google is fine but specialized and hard to migrate away from, OpenStack isn't for small orgs... I've run production services on them all. It's easy to try a new service with a Pipelines-style orchestration because it doesn't care which service its talking to. And makes it obvious when platform-specific instructions or concessions are required.

sansnomme 8y ago
This looks interesting, I have a few questions. How do you deal with Linux dependencies etc.? All your applications must be BSD compatible? Any advice for getting started with it on DigitalOcean? Or is nested out of the story?
- doublerebel 8y ago
  Oh, SmartOS containers run Linux seamlessly too. I guess I take that knowledge for granted now. But many many of the popular cloud tools (hashicorp consul and vault, redis, couchdb, arangodb, postgres) run natively on SmartOS and have better resource and user management from the underlying system. I haven't used Digital Ocean's kube offering. Otherwise it's much like Linode or other services from that era -- install your own distro and binaries, which needs some kind of orchestration automation to scale.
  - 8y ago

freedomben 8y ago

The author mentions coupling with kubernetes due to config maps and such. That is certainly a risk, but if you use environment variables for config (which IMHO you should) then k8s makes it trivial to just map your ConfigMap to Environment Variables. I've got a few apps running in k8s and none of them know anything about k8s. If avoiding coupling with k8s is what you want (and I agree it's the way to go) then it really isn't that bad. The hard part is not knowing what you don't know.

xyzzy123 8y ago
Or just mount the config map as a file.
snupples 8y ago
I think the real question then is why are people running apps that need to be k8s-aware at all? K8s already provides a number of primitives to solve many problems without reinventing the wheel and going "cloud-native" within the application. There should be a very strong reason for any need to access the api directly.

jjeaff 8y ago

The thing that all the "you don't need kubernetes" articles seem to ignore or skim over, is the huge network effect of a popular setup like k8s. Not only is it much easier to find services (like helm) and articles for k8s, but there are so many ready to go configurations. For most major services, there is already a ready to go helm chart or at least some well vetted yml config on GitHub to start from. And most any issue I have come across, I have been able to search and find solutions.

omn1 8y ago
Haven't looked into helm-charts deeply, but the ones that I found often needed a considerable amount of time to configure for production. For example, the Logstash chart uses a YAML config whereas we use Logstash's standard config format with many filters and options. Porting these features over to YAML is not a lot of fun. And in the end we would also have to maintain these configs and keep them in sync with upstream changes. Speaking of documentation, I sometimes feel a bit overwhelmed by the amount of literature on Kubernetes these days. Many articles are quite outdated already and new ones get written every day. It's hard to know what the best practices are - and the ecosystem is still rapidly evolving. So I don't think I was skimming over these parts, I just think that inertia can also become a disadvantage.
paxys 8y ago
As well as hiring engineers who will be familiar with it from day 1, as opposed to having to learn whatever custom system you use.
- JMTQp8lwXL 8y ago
  This. In a perfect world, it should be more useful to say to an employer, "I know Kubernetes" than "I know AWS". It's a unified abstraction for cloud resources. They all, more or less, do the same thing. Why learn the proprietary, vendor-specific way to do it, when you learn it once, and apply your knowledge to any cloud, e.g, if you switch jobs and your new one uses a different cloud provider.
  - 8y ago

openloop 8y ago

Maybe you do.

sdan 8y ago

I spent a few days developing a Kubernetes Cluster with some Docker knowledge. Other than creating the config file, I didn't see too much of a problem. It definitely has a learning curve and takes a lot of time to setup (even with GKE). From reading this article, I'm liking Nomad.

ilaksh 8y ago

I guess I am a little behind the times, but no one uses the orchestration stuff that comes with Docker?

twblalock 8y ago
I know people who use Docker Compose to run Docker containers on their laptops. I don't think Compose or Swarm were ever nearly as popular as Kubernetes for running services in production. One of the signs that Kubernetes became the industry standard is that its competitors feel the need to promote their compatibility with it. Docker for Mac now ships with a built-in Kubernetes cluster. (I prefer it to Minikube).
andbberger 8y ago
I use it a fair bit for development, provides a seemingly simple way to keep things organized and do 'infra-as-code' and have your ducks in a row with reproducible stack deployments, without the overwhelming overhead of k8s. It's an absolute dumpster fire though. A serious convoluted mess. They try to indelicately cram something with incredible intrinsic complexity into a happy little (incredibly leaky) abstraction. docker compose is really what I want, simple service management on one machine. Except, it can't do really basic shit like restart services (this isn't all that obvious at first). So soon you'll find that you need to use docker swarm on the one node, which is similarly chock full of inexplicable and unforeseeable shortcomings. docker swarm/compose is an interim step on your journey to a better solution. in my judgement it's fine while you're developing your core apps. but once you need to do get serious and go into production, it's time to find a big boy tool
- ngrilly 8y ago
  I did the same thing. Docker Compose was what I wanted for a single-machine deployment, but I soon discovered it was unable to manage a rolling deploy (start new containers before stopping the old ones). So I switched to Docker Swarm on a single node, but I have to agree there is a lot of unexpected shortcomings. I'd like it to be more polished. And I'd like Docker Compose and Docker swarm mode to converge/merge at some point. I guess the "big boy tool" you adopted after that is k8s?

gouggoug 8y ago

The argument that Kubernetes adds complexity is, in my opinion, bogus. Kubernetes is a "define once, forget about it" type of infrastructure. You define the state you want your infrastructure to and Kubernetes takes care of maintaining that state. Tools like Ansible and Puppet, as great as they are, do not guarantee your infrastructure will end up in the state you defined and you easily end up with broken services. The only complexity in kubernetes is the fact that it forces you to think and carefully design your infra in a way people aren't used to, yet. More upfront, careful thinking isn't complexity. It can only benefit you in the long run. There is, however, a learning curve to Kubernetes, but it isn't this sharp. It does require you to sit down and read the doc for 8 hours, but that a small price to pay. A few month back I wrote a blog post[1] that, through walking through the few different infrastructures my company experimented with over the years, surfaces many reasons one would want to use [a managed] Kubernetes. (For a shorter read, you can probably start at reading at [2]) [1]: https://boxunix.com/post/bare_metal_to_kube https://boxunix.com/post/bare_metal_to_kube [2]: https://boxunix.com/post/bare_metal_to_kube/#_hardware_infrastructure https://boxunix.com/post/bare_metal_to_kube/#_hardware_infra...

JMTQp8lwXL 8y ago
This has been my experience. It isn't that complex, just a steep learning curve.
- TooCleverByHalf 8y ago
  If complexity isn't the reason for the steep learning curve what is?
  - JMTQp8lwXL 8y ago
    The documentation around it isn't that great. Once you know the key actors -- which are a dozen or so key types of resources, and how they play together, you understand how to create the configuration for your environment. The YAML files are quite intimidating at first, if you look at examples out there. I'm probably not a great teacher, but I think I could boil it down to a slide deck of no more than 25, and an hour's worth of time, and clearly teach how to interface with Kubernetes for a basic web app w/ horizontal scaling, Let's Encrypt SSL certificates, etc.

jugg1es 8y ago

So many comments here are touting the benefits of this, that or the other, but aren't mentioning Nomad at all. Nomad is definitely worth a try even for large deployments. Yea, it doesn't autoscale without 3rd party tools, but you don't always need that. If you just need to keep your jobs running, load balance to those apps, use externalized config and need a simple UI view system status, then Nomad/Consul/Fabio really works great.

mancerayder 8y ago

I'm attempting to 'transition' -- as a DevOps / infra guy professionally rather than pure dev -- does anyone have any recommended resources for reading non-dry, fairly example-filled docs to get started? The Kubernetes O'Reilly book seems to be very broad and overly high-level from the reviews (and I remember best if I read "why" and "how"). Thanks in advance

dankohn1 8y ago
This is particularly well-liked for a more in-depth review of Kubernetes: https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way

xaduha 8y ago

Docker Compose is great, Docker Swarm isn't.

ashton314 8y ago

Rancher [1] looks really nice for a guided use of Kubernetes. I saw a demo on how to use this with Digital Ocean's Kubernetes clusters; the presenter went from having a blog in a container to having the blog on a K8s cluster behind a domain name in 30-ish minutes. Anybody using Rancher? [1]: https://rancher.com https://rancher.com

dmlittle 8y ago
Do you have any info regarding Rancher pricing? There is no info on their website other than reaching out to them...
- nhumrich 8y ago
  Its free. You pay for support.
- V99 8y ago
  (Rancher employee) All the software is free to use and open source (Apache 2). Customers run the same software that's on GitHub/DockerHub. We sell "enterprise support", which means something like "starting at mid-5 figures", and occasionally professional services engagements around/outside the product.
seabrookmx 8y ago
We're doing a sizable migration from Rancher 1.6 (which uses their internal orchestration system called "Cattle") to Rancher 2.x (which sits ontop of k8s). Rancher 2 definitely streamlines the setup of an on-premise cluster. We use AWS for 80% of our stuff and have in-house servers for the remaining 20%, so it allows us to have a single pane of glass for all of our workloads (the AWS stuff being an EKS cluster "imported" into Rancher). As a company they definitely seem to follow the "move fast and break things" mantra though. We've seen quite a few boneheaded bugs in the API and CLI tool.. and the fact that 1.x -> 2.x has no real migration path other than "rewrite all your app configurations as helm charts and you'll be fine!" is kind of a bitter pill. In reality the networking and storage side of things is vastly different between Cattle and k8s. We really felt the limitations of Cattle though, so I understand that this move is necessary. Had they not hopped on the k8s bandwagon we probably would have eventually without them. I guess it's just the price you pay for being an early adopter!

rbjorklin 8y ago

I’ve come to much the same conclusions as the author in this blog post and have put together a small Nomad demo in case someone is interested: https://github.com/rbjorklin/nomad-demo https://github.com/rbjorklin/nomad-demo

sbhn 8y ago

Yes i do, https://homeidea3d.com https://homeidea3d.com runs on kubernetes

joeblow21 8y ago

Sounds like you could just move to a PaaS like App Engine and either downsize or free up a lot of engineering staff for better uses of their time than devops

kissgyorgy 8y ago

I'm pretty sure you don't need Kubernetes.

bifrost 8y ago

I'm very much in the camp of "Kubernetes is a problem factory". Personally, I've found very few problems that it solves and more problems that it creates. That said, its probably a great time to be a Kube consultant!

mirceal 8y ago
if your team is only a handful of devs who work on the thing that pays the bills + do k8s, yes. bad time. the only way i could justify running this is if you’re using a managed solution like gke or eks. otoh, if you’re in the cloud, you already have the primitives to build your stuff w/o the k8s headache (vms, autoscalling, managed svcs). but that’s just my opinion. the koolaid is strong.
- bifrost 8y ago
  Honestly, if you have a $GENERIC_SYSTEM that you deploy your code to, you don't need the wasteful overhead. If you need to (vs want to) rebuild your OS/Environment regularly, you're doing it wrong. IE: PaaS is for you. That said, I've found replicating OS & Environment extremely challenging on certain operating systems so I understand the appeal. I also don't build products on those operating systems and I've found myself much happier :)
  - mirceal 8y ago
    yeah. it depends a lot and k8s has its place in some scenarios, but in most cases you don’t need it and its use is purely driven by koolaid

squid3 8y ago

Maybe all you need is NodeChef. Saves you time and money. https://www.nodechef.com/ https://www.nodechef.com/

deleted 8y ago

[deleted]

eric_khun 8y ago

anyone got recommendation on how to properly set CPU limits? I've set limits about 2x higher than the max cpu usage, but still seeing some throttling, what cause performance degradation. I've ended up setting deployments without limits to avoid throttling but that's definitely not nice if one pod goes crazy for a reason ...

oso2k 8y ago
Not sure what you mean by throttling. I tell my customers to do 4 things 1) Set a Request at about 50% of max CPU usage 2) Set a Limit at about 90% of max CPU usage 3) Configure a HorizontalPodAutoscaler that kicks in at 70% 4) Scale out, in advance, to 2 or 3 pod replicas

kissgyorgy 8y ago

> It takes a fair amount of time and energy to stay up-to-date with the best practices and latest tooling. Kubectl, minikube, kubeadm, helm, tiller, kops, oc - the list goes on and on. When you start with a technology, you need to have a solid basis. You need to fully grasp the given technology in it's most basic forms, stuck with the problems for days so you can fully grasp what any of these can solve what problem. If you know "naked" Kubernetes inside and out, then you can add other things to your toolset. Until then, you don't need no helm, tiller, kops, Knative or any of those distractions.

tarikjn 8y ago
Isn't knative exactly supposed to solve this problem by giving a portable Heroku-like dev experience?
- kissgyorgy 8y ago
  Doesn't matter. I talked about the initial learning about Kubernetes. It's just too much to learn everything at once.
  - jacques_chester 8y ago
    Restating: the point is to not have to "learn everything". If your platform makes you do hours of homework, then it's not a platform, it's a pallet of bricks. Heroku got this right with `git push heroku master`. IMO Cloud Foundry got it even more right with `cf push`. Knative is likely to get to that point, probably with the same means as Heroku and Cloud Foundry: Buildpacks. I'm wildly biased of course, having worked on Cloud Foundry, Buildpacks and Knative.

draw_down 8y ago

It’s interesting to see the differing reactions to these articles here. Maybe you don’t need Kubernetes = good, maybe you don’t need jQuery = bad.

TomBombadildoze 8y ago

This starts with a lot of "you don't need Kubernetes" and then concludes with a pretty compelling argument in favor of using Kubernetes. From the "The Nomad ecosystem of loosely coupled components" section: > It integrates very well with other - completely optional - products like Consul (a key-value store) or Vault (for secrets handling). > At trivago, we tag all services, which expose metrics, with trv-metrics. This way, Prometheus finds the services via Consul and periodically scrapes the /metrics endpoint for new data. > The same can be done for logs by integrating Loki for example. > Trigger a Jenkins job using a webhook and Consul watches to redeploy your Nomad job on service config changes. shudder > Use Ceph to add a distributed file system to Nomad. > Use fabio for load balancing. And the icing on the cake: > All of this allowed us to grow our infrastructure organically without too much up-front commitment. So if I understand correctly, the author (and his team) preferred to do all the work of integrating/testing/debugging those components, rather than using a tool that provides every single on of those features, and more, out of the box. Kubernetes isn't a trivial lift but it's a damn sight easier than trying to roll a cheap imitation yourself.

millettjon 8y ago
We use nomad because it's flexibility in this regard lets us slowly migrate away from our old infrastructure. Moving fully to Kubernetes would not be politically feasible in the organization.
- lugg 8y ago
  This is a post about why I don't need kubernetes. Not a post about why kubernetes isn't politically feasible in not my organisation.
omn1 8y ago
The point is that Nomad allows to add more components as they are needed while Kubernetes is an all-or-nothing solution. Also, we used tools like Jenkins and Ceph before. They have been tested and we trust them so there's no point in replacing them. Quite the contrary: a lot of internal processes depend on them so it would have been painful to migrate away from them. Nomad is not a cheap imitation of Kubernetes, it is a simple orchestrator which favors composability over an all-in-one approach.

humbleMouse 8y ago

If people would spend 1/10th the time they spent complaining about kubernetes actually learning kubernetes, maybe more people would know kubernetes and realize its not even that difficult to administrate.

trpc 8y ago

just another provocative post title by someone wants to bash something he has no clue about

inopinatus 8y ago

All any of us really need to do is put files on a server and start a process. The question is how much scaffolding you want to build and manage to achieve it. The answer should be “as little as possible”.

AnthonyWnC 8y ago

The overall trend of IT infrastructure goes from tradition DC -> IaaS -> PaaS -> FaaS. k8 sits at PaaS and the logically next step is FaaS/serverless. In fact, today you can surely do purely FaaS without the need k8 at all by leveraging cloud provider managed services for any long-live type of workload. Of course, the tradeoffs will the usual suspects: flexibility, control, cost etc. There are still cases where you would want a cluster manager like k8 (at least for now).

namelosw 8y ago

1. Always think about Heroku first, seriously. Because your need is deploy application. 2. If you thought about 1 but Heroku is really not for you, go Kubernetes directly. It's not that hard at all. 3. Choose any other stuff only when you know what you're doing. Kubernetes is not simple or good. The api is mostly cumbersome. But it's the first thing that makes sense in this field. Before Kubernetes there are just toys or rocket science.

jordanbeiber 8y ago

I did an eval about 4 years ago of schedulers and service discovery software. Lot’s of cool stuff around (dc/os, triton, hashi, k8s, flynn etc)! After 2 days with k8s and not so much as a bootstrapped cluster the turn went to hashicorp. Within 10 min I had a container running, registered in service disc (consul) on my laptop. We ended up using the hashicorp stack in production with great success. Sure, some custom services was needed (service managment, scaling and such). Running primarily on prem the complete simpleness and un-opiniated approach is an edge. It allowed us to implement MACVLAN at first, and the ubuntu FAN-network and integrate totally with existing infrastructure. Now having spent the last 8 months implementing k8s I’m torn. I’ve built from scratch, used kubeadm, kops and now EKS. From 1.13 kubeadm honestly works really well and is my prefered way of deploying k8s. Still it’s a beast... running large deployments with many teams... there’s just so much... stuff. One GH issue leading down a rabbit hole of other GH and gists with conflicting or not working configurations. I’ve had co-workers bail on the project after mere hours of digging through code and GH discussions and SIG docs. Nomad/consul and it’s concise docs is a breeze in comparision. Torn. Cause I see the point of k8s, just not sure about the project. :)

marmaduke 8y ago

I’m curious how this compares to pacemaker and corosync for core infra devices like a active/passive failover for SAN or a virtual IP. It doesn’t seem like they’re in the same category in terms of typical usage, but both can handle the notion of a cluster.

auslander 8y ago

Should we use containers at all is the first question. Why people skip over this one ? :)

gerbilly 8y ago

I currently use Ansible to provision and configure about 50 instances with various home grown services deployed on them, and I'm quite happy with that. I don't use Docker, or Kubernetes. What am I missing here? This is an honest question. I don't particularly see the point of docker in production. I guess it can help resolve clashes between dependencies, but I'm doing ok with Ansible. One guy in another team just tells his application developers to throw everything into a docker container and then deploys whatever they give him. I guess that prevents dependency clashes, sure, but to me that seems like it's just inviting different problems though. (Containers with a lot of disorganized junk in them). And we don't need to grow the cluster for now, so I don't see the point of k8. We already have log aggregation, and service restarting, service monitring/metrics. To me, Kubernetes feels like a brand new alternative OS with crappy documentation. On the other hand I've been managing UNIX machines for 20+ years, and that I do know how to fix (mostly, since they keep changing linux so much these days, it's almost as hard to keep up with as javascript). If somebody could tell me what I'm missing by using this approach, I'd be grateful.

auslander 8y ago
Because it's cool, new and good for your blog/CV. I don't see the need too, if stuff is in cloud.
originalvichy 8y ago
You aren't "missing" anything, you are just doing things differently. Containers are a way of storing code and configurations in a single deployable entity. Let's say I want to change a services version from 1.2 to 1.3. If I did this in Ansible, I might do it by commanding all of my services to pull the code of version 1.3 from a git repo and restarting to make the change. If I did this with containers, I'd tell them to pull the image version 1.3 from docker registry and after it's pulled to shut down v.1.2 and start v.1.3. Those things just have some smaller differences that are good in terms of A and worse in terms of B. Ansible helps you by keeping it closer to the traditional way of just pushing code changes to a machine and that's that, and Docker opens up this world where the infrastructure can scale up or down (elasticity) quicker horizontally. Kubernetes is just an extrapolation of that elasticity. Ansible is "good enough" if you don't need elasticity or you have programmed another custom solution to handle that. Docker/Kube and Ansible both have Git at their hearts, so if your app components have all they need as code, and you don't worry about elasticity, you won't gain a lot from wrapping it up in a Dockerfile (as opposed to just git cloning).

jpomykala 8y ago

I only use Docker, because I'm too lazy to learn Kuberneres or Ansible. :)

reilly3000 8y ago

Its usually when these kinds of articles come out that their subjects are finally worth paying attention to. Immutable infrastructure is the right thing to do. Maybe K8's isn't the best or most convenient way to achieve that, but it gets you almost all the way there to fully declarative ops. I really think there is a lot of room to improve the ergonimics. That said, there is a business/project opportunity. I want to be able to build a values.yml file in a gui, on my phone, with bounded sliders and smart recommendations for values, along with contextual help and community templates. We have chart repos, but the default values suck for production, and learning what the values are and should be set to for your use case leads to a painful traversal of abstractions. Maybe there is already a project or vendor out there I'm missing?

he0001 8y ago

I’ve been using Kubernetes for a while (1,5y). And I can’t say I’m very impressed. For anything simple it works fairly well. But as soon you get outside the “normal” workflow it gets downright hard. Certain areas are opaque AF and if things doesn’t work it almost impossible to find out why. And if it’s impossible to fix something fast if there’s something burning. I think that, there’s an irony to k8s, it’s supposed to fix the “hard problems” of clusters. But it really only fixes the easy part of them, and obscures the rest.

hjacobs 8y ago

Triggered by Matthias' blog post, I wrote down my thoughts on the topic here: https://srcco.de/posts/why-kubernetes.html https://srcco.de/posts/why-kubernetes.html