13 ms·

For me, and many others: infrastructure as code. Kubernetes is very complex and took a long time to learn properly. And there have been fires among the way. I

For me, and many others: infrastructure as code. Kubernetes is very complex and took a long time to learn properly. And there have been fires among the way. I plan to write extensively on my blog about it. But at the end of the day: having my entire application stack as YAML files, fully reproducible [1] is invaluable. Even cron jobs. Note: I don't use micro services, service meshes, or any fancy stuff. Just a plain ol' Django monolith. Maybe there's room for a simpler IAC solution out there. Swarm looked promising then fizzled. But right now the leader is k8s[2] and for that alone it's worth it. [1] Combined with Terraform [2] There are other proprietary solutions. But k8s is vendor agnostic. I can and have repointed my entire infrastructure with minimal fuss.

dustym 6y ago
Yup, even monoliths can benefit from certain k8s tooling (HPAs, batch jobs, etc).
- silviogutierrez 6y ago
  "What cron jobs does this app run?" Open cron.yaml and see. With schedule. Self documented. Amazing. Every time. Even when some as my k8s battle wounds are still healing (or permanently scarred). See other replies for more info.
  - StreamBright 6y ago
    Just like my Ansible repo.
sp332 6y ago
I've just started to look into it, but it seems like the project has been focusing on improving the onboarding experience since it has a reputation for being a huge pain to set up. Do you think it has gotten easier lately?
- silviogutierrez 6y ago
  No. Not easier in my opinion. And some of the fires you only learn after getting burnt badly. [1] Note: my experience was all with cloud-provided Kubernetes, never running my own. So it was already an order of magnitude easier. Can't even imagine rolling my own. [2] [1] My personal favorite. Truly egregious, despite how amazing k8s is. https://github.com/kubernetes/kubernetes/issues/63371#issuecomment-633263227 https://github.com/kubernetes/kubernetes/issues/63371#issuec... [2] https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way
  - cyberpunk 6y ago
    Rancher makes this quite painless; worth a look if you have to run on prem, or in an iaas cloud for some reason.
mfer 6y ago
Out of curiosity, are you using terraform to deploy k8s, your app stack on k8s, or both?
- throw1234651234 6y ago
  To follow this - what do you feel K8S provides on top of terraform? We used K8S on a large project and I felt like it really, really wasn't necessary.
  - verdverm 6y ago
    Consistency and standardized interfaces for AppOps regardless of the hyper-cloud I use. Kubernetes basically has an equivalent learning curve, but you only have to do it once
  - nvarsj 6y ago
    They operate at different layers. K8s sits on top of the infrastructure which terraform provisions. It's far more dynamic and operates at runtime, compared to terraform which you execute ad-hoc from an imperative tool (and so only makes sense for the low level things that don't change often).
  - StreamBright 6y ago
    Nothing. We use Terraform to provision a simple auto-scaling cluster with loadbalancers and certs, does exactly the same thing but there is no Docker and k8s. Few million lines less Go code turning yaml filed into seggfaults.
- silviogutierrez 6y ago
  So terraform is a higher-order, meta-Kubernetes. It's very rarely used, but who provisions the cluster itself? That's terraform. So terraform creates the cluster, DNS and VPC. Then k8s runs pretty much everything.
  - pvorb 6y ago
    Why would you want to provision your own k8s cluster, if you can use EKS, AKS or similar?
herval 6y ago
Have you tried or considered Nomad (from the makers of Terraform)?
- silviogutierrez 6y ago
  I haven't, and since I've sunk the cost into Kubernetes and know it very well now, likely won't end up there. In retrospect though, maybe it's exactly what I needed. Great suggestion.
- p_l 6y ago
  can it handle networking (including load balancing and reverse proxies with automatic TLS) or virtualized persistent storage? Make it easy to integrate common logging system? Cause those are the parts that I miss probably the most when dealing with non-k8s deployment, and I haven't had the occasion to use Nomad.
  - adadgar 6y ago
    For load balancing you can just run one of the common LB solutions (nginx, haproxy, Traefik) and pick up the services from the Consul service catalog. Traefik makes it quite nice since it integrates with LetsEncrypt and you can setup the routing with tags in your Nomad jobs: https://learn.hashicorp.com/nomad/load-balancing/traefik https://learn.hashicorp.com/nomad/load-balancing/traefik What Nomad doesn’t do is setup a cloud provider load balancer for you. For persistent storage, Nomad uses CSI which is the same technology K8s does: https://learn.hashicorp.com/nomad/stateful-workloads/csi-volumes https://learn.hashicorp.com/nomad/stateful-workloads/csi-vol... Logging should be very similar to K8S. Both Nomad and K8S log to a file and a logging agent tails and ships the logs. Disclosure, I am a HashiCorp employee.
    - p_l 6y ago
      Thanks, definitely widened my understanding of Nomad in pretty short time :) Kinda feels bad that I don't have anything to use it on right now.
    - VectorLock 6y ago
jcastro 6y ago
> I can and have repointed my entire infrastructure with minimal fuss. When you get to that blog post please consider going in depth on this. Would love to see actual battletested information vs. the usual handwavy "it works everywhere".
- silviogutierrez 6y ago
  I sure will. 99% of the work was ingress handling and SSL cert generation. Everything else was fairly seamless. Even ingress is trivial if you use a cloud balancer per ingress. But I wanted to save money so use a single cloud balancer for multiple ingresses. So you need something like ingress-nginx, which has a few vendor-specific subtleties.
bosswipe 6y ago
According to the article you are wrong about "infrastructure as code". Kubernetes is infrastructure as data, specifically YAML files. Puppet and Chef are infrastructure as code. Edit: not sure why the down votes, I was just trying to point out what seems like a big distinction that the article is trying to make.
- monadic2 6y ago
  What’s the material difference between well-formatted data and a DSL? Why does this matter?
- danaur 6y ago
  Seems like a nitpick? Infra as data seems like a subset of infrastructure as code
- silviogutierrez 6y ago
  Maybe? I'm too lazy to formally verify if the YAML files k8s accepts are Turing complete. With kustomize they might very well be. How about "infrastructure-as-some-sort-of-text-file-versioned-in-my-repository". It's a mouthful, but maybe it'll catch on.
  - lmkg 6y ago
    Infrastructure-as-config?
  - geofft 6y ago
    They don't do loops or recursion. They don't even do iterative steps in the way that Ansible YAML has plays/tasks. Yes, higher-level tools like Kustomize or Jsonnet or whatever else you use for templating the files are Turing-complete - but that's at the level of you on your machine generating input to Kubernetes, not at the level of Kubernetes itself. That's a valuable distinction - it means you can't have a Kubernetes manifest get halfway through and fail the way that you can have an Ansible playbook get halfway through and fail; there's no "halfway." If something fails halfway through your Jsonnet, it fails in template expansion without actually doing anything to your infrastructure. (You can, of course, have it run out of resources or hit quota issues partway through deploying some manifest, but there's no ordering constraint - it won't refuse to run the "rest" of the "steps" because an "earlier step" failed, there's no such thing. You can address the issue, and Kubernetes will resume trying to shape reality to match your manifest just as if some hardware failed at runtime and you were recovering, or whatever.)
bdcravens 6y ago
Of course it doesn't have the advantage of #2, but I've found ECS to be far easier to grok and implement.
UweSchmidt 6y ago
With "Swarm", do you mean Docker Swarm? Why has it "fizzled"? The way I learned it in Bret Fisher's Udemy course, Swarm is very much relevant, and will be supported indefinitely. It seems to be a much simpler version of Kubernetes. It has both composition in YAML files (i.e. all your containers together) and the distribution over nodes. What else do you need before you hit corporation-scale requirements?
- morelisp 6y ago
  > What else do you need before you hit corporation-scale requirements? Cronjobs, configmaps, and dynamically allocated persistent volumes have been big ones for our small corporation. Access control also, but I'm less aware of the details here, other than that our ops is happier to hand out credentials with limited access, which was somehow much more difficult with swarm Swarm has frankly also been buggy. "Dead" but still running containers - sometimes visible to swarm, sometimes only the local Docker daemon - happen every 1-2 months, and it takes forever to figure out what's going on each time.
- sdan 6y ago
  I use Swarm in production and am learning k8s as fast as possible because of how bad Swarm is: 1. Swarm is dead in the water. No big releases/development afaik recently 2. Swarm for me has been a disaster because after a couple of days some of my nodes slowly start failing (although they’re perfectly normal) and I have to manually remove each node from the swarm, join them, and start everything up again. I think this might be because of some WireGuard incompatibility, but the strange thing is that it works for a week sometimes and other times just a few hours 3. Lack of GPU support
  - GordonS 6y ago
    To add another side, I use Swarm in production and continue to do so because of how good it is. I've had clusters running for years without issue. I've even used it for packaging B2B software, where customers use it both in cloud and on-prem - no issues whatsoever. I've looked at k8s a few times, but it's vastly more complex than Swarm (which is basically Docker Compose with cluster support), and would add nothing for my use case. I'm sure a lot of people need the functionality that k8s brings, but I'm also sure that many would be better suited to Swarm.
McAtNite 6y ago
Out of curiosity why do you feel swarm fizzled out? I’ve deployed swarm in a home lab and found it really simple to work with, and enjoyable to use. I haven’t tried k8, but I often see view points like yours stating that k8 is vastly superior.
lifeisstillgood 6y ago
So are you saying that, no matter what, if you want to reply your whole infrastructure as code (networks, dmz, hosts, services, apps, backups etc ) that you are going to have to reproduce that somehow (whatever the combo of AWS services are, OR just learn K8S Effectively, "every infrastructure as code project will reimplement Kubernetes in Bash"
- silviogutierrez 6y ago
  Exactly. That's a great way to put it. A bunch of Bash glue reimplementing what Kubernetes already does. Poorly.
  - roman_sf 6y ago
    Kubernetes is outstanding because proving ol bash scripts were bad and re-writing all Bash glue in Go glue, slapping 150 APIs on top (60 "new" and about hundred "old versions" https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18 https://kubernetes.io/docs/reference/generated/kubernetes-ap...), adding few dozen opensource must have projects - so finally it can be called "cloud native" - boom, a new cloud native bash for the cloud is born!
    - tannhaeuser 6y ago
      Actually it's worse than that - k8s was ported from Java, and the codebase is an incoherent mess, as k8s devs say themselves
  - gfodor 6y ago
    Bash glue has rarely failed me. Never used k8s, but the horror stories I hear ensure I won’t feel dirty for writing a little bash to solve problems anytime soon.
- aprdm 6y ago
  Not necessarily. You can have all of above with Terraform, Ansible, Puppet, Chef... etc.
tyingq 6y ago
There's an offshoot of this that I see from developers, especially at old, stodgy companies. Once everything is "infrastructure as code", the app team becomes less dependent on other teams in the org. People like to own their own destiny. Of course, that also removes a lot of potential scapegoats, so you now mostly own all outages, tech debt, etc.
- duxup 6y ago
  I think that's been coming since ... well ever. I worked in networking for the longest time. When I started there network guys and server guys (at least where I was). They were different people who did different things who kinda worked together. Then there were storage area networks and similar, networks really FOR the server and storage guys.... that kind of extended the server world over some of the network. Then comes VMware and such things and now there was a network in a box somewhere that was entirely the server guy's deal (well except when we had to help them... always). Then we also had load balances who in their own way were a sort of code for networks ... depending on how you looked at it (open ticket #11111 of 'please stop hard coding ip addresses'). You also had a lot of software defined networking type things and so forth brewing up in dozens of different ways. Granted these descriptions are not exact, there were ebs and flows and some tech that sort of did this (or tired) all along. It all starts to evolve slowly into one entity.
gigatexal 6y ago
imo, all swarm needed was a decent way to handle secrets and it would have been the k8s of its day
tombert 6y ago
Swarm is still supported and works. I have it running on my home server and love it. Kubernetes is fine, but setting it up kind of feels like I'm trying to earn a PhD thesis. Swarm is dog-simple to get working and I've really had no issues in the three years that I've been running it. The configs aren't as elaborate or as modular as Kubernetes, and that's a blessing as well as a curse; it's easy to set up and administer, but you have less control. Still, for small-to-mid-sized systems, I would still recommend Swarm.
- theptip 6y ago
  > setting it up kind of feels like I'm trying to earn a PhD thesis. Are you following "k8s the hard way"? I've never had this problem; either: `gcloud container clusters create` Or `install docker-for-mac` And you have a k8s cluster up and running. Maybe it's more work on AWS?
  - kronin 6y ago
    Ekscli (if you want managed) and kops (if you want to manage it yourself) are just as straightforward.
- ithkuil 6y ago
  > setting it up kind of feels like I'm trying to earn a PhD thesis. The kind of people who has to both set the cluster up and keep it up and also has to develop the application and deploy it and keep it up etc is not the target audience. K8s shines when the roles of managing the cluster and running workloads on it are separated. It defines a good contract between infrastructure and workload. It lets different people focus on different aspects. Yes it still has rough edges, things that are either not there yet, or vestigial complexity of wrong turns that happened through it's history. But if you look at it through the lense of this corporate scenario it starts making more sense than when you just think of what a full-stack dev in a two person startup would rather use and fully own/understand.
  - DelightOne 6y ago
    What are Kubernetes' rough edges?
archsurface 6y ago
I'm not sure "a plain ol' Django monolith" with none of the "fancy stuff" is either what people are referring to when they say "kubernetes", or a great choice for that. I could run hello world on a Cray but that doesn't mean I can say I do supercomputing. Our team does use it for all the fancy stuff, and spends all day everyday for years now yamling, terraforming, salting, etc so theoretically our setup is "entire application stack as YAML files, fully reproducible", but if it fell apart tomorrow, I'd run for the hills. Basically, I think you're selling it from a position which doesn't use it to any degree which gives sufficient experience required to give in-depth assessment of it. You're selling me a Cray based on your helloworld.
- silviogutierrez 6y ago
  Reading this charitably: I guess I agree. k8s is definitely overpowered for my needs. And I'm almost certain my blog or my business will never need that full power. Fully aware of that. But I'm not sure one can find something of "the right power" that has the same support from cloud providers, the open source community, the critical mass, etc. [1] Eventually, a standard "simplified" abstraction over k8s will emerge. Many already exist, but they're all over the place. And some are vendor specific (Google Cloud Run is basically just running k8s for you). Then if you need the power, you can eject. Something like Create React App, but by Kubernetes. Create Kubernetes App. [1] Though Nomad looks promising.
  - wolco 6y ago
    Curious why run it at all? The cost must be 10 times more this way. It is mostly for the fun of learning. I come from the opposite approach. I have 4 servers two digital ocean $5 and two vulr $2.50 instances. One holds the db. One server as the frontend/code. One server to do heavy work and another to server a heavy site and holds backups. For $15 I'm hosting hundreds of sites, running so many background processes. I couldn't imagine hitting that point where k8s would make sense just for myself unless for fun.
    - StavrosK 6y ago
bulldoa 6y ago
Do you have a recommended tutorial for engineer with backend background to setup a simple k8 infra in ec2?
- hartem_ 6y ago
  Take a look at https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way. That’d be a great first step if the purpose is to learn Kubernetes. If, however, you want to set up a cluster for real use then you will need much more than bare bones Kubernetes (something that solves networking, monitoring, logging, security, backups and more) so consider using a distribution or a managed cloud service instead.
  - physicles 6y ago
    Setting up your own k8s from scratch is kind of like writing your own string class in C++: it’s a good exercise (if it’s valuable for your learning path) but you probably don’t want to use it for actual work. Maintaining a cluster set up like that is a ton of work. And if you don’t perform an upgrade perfectly, you’ll have downtime. Tools like kops help a lot but you’ll still spend far more time than the $70/month it costs for a managed cluster.
- turtlebits 6y ago
  I find that K3s is great for getting started. It has traefik included and its less of a learning curve to actually be productive, vs diving in with K8s and having to figure out way more pieces.
deleted 6y ago
[deleted]
kasey_junk 6y ago
Conversely I think the trend of writing infrastructure as yaml may be the worst part of modern ops. It’s really hard to think of a worse language for this.
- banifo 6y ago
  Why? We have plenty of yaml 'code' which is simple and does exactly what it needs to do. For all other usecases, there are plenty of alternatives, including libs for your preferred language. Most people use the yaml way because its easy and does exactly what it needs to do. Everyone else has plenty of well supported and well working alternatives.
  - kasey_junk 6y ago
    Its typing is too weak. It’s block scoping is dangerous. References, classes and attachments are all pretty bad for reuse. Schemas are bolt on and there is no standard query language for it.
- p_l 6y ago
  Fortunately YAML is not actually necessary in k8s, and only provided as a convenience because writing JSON by hand (or Proto3, lol) is annoying verging on insane. We can build higher level abstractions easily having a schema to target and we can build them in whatever we want. That's a big boon for me :)
  - kasey_junk 6y ago
    For kubernetes thats absolutely true and I think more people should do that (disclaimer I work with this https://github.com/stripe/skycfg https://github.com/stripe/skycfg daily). I think it actually would be easier for people to understand the k8s system if they did. But, yaml is now everywhere in the ops space. Config management systems use it, metrics systems use it, its the defacto configuration format right now and that is unfortunate cause its bad.
collyw 6y ago
Do you have a good tutorial for doing Django or a standard 3 tier web app on Kubernetes? We are using kubernetes at my workspace, but it seems way too complicated to consider for something like that. Maybe if I can bridge the gap between architectures it will help.

dustym 6y ago

Yup, even monoliths can benefit from certain k8s tooling (HPAs, batch jobs, etc).

silviogutierrez 6y ago
"What cron jobs does this app run?" Open cron.yaml and see. With schedule. Self documented. Amazing. Every time. Even when some as my k8s battle wounds are still healing (or permanently scarred). See other replies for more info.
- StreamBright 6y ago
  Just like my Ansible repo.

sp332 6y ago

I've just started to look into it, but it seems like the project has been focusing on improving the onboarding experience since it has a reputation for being a huge pain to set up. Do you think it has gotten easier lately?

silviogutierrez 6y ago
No. Not easier in my opinion. And some of the fires you only learn after getting burnt badly. [1] Note: my experience was all with cloud-provided Kubernetes, never running my own. So it was already an order of magnitude easier. Can't even imagine rolling my own. [2] [1] My personal favorite. Truly egregious, despite how amazing k8s is. https://github.com/kubernetes/kubernetes/issues/63371#issuecomment-633263227 https://github.com/kubernetes/kubernetes/issues/63371#issuec... [2] https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way
- cyberpunk 6y ago
  Rancher makes this quite painless; worth a look if you have to run on prem, or in an iaas cloud for some reason.

mfer 6y ago

Out of curiosity, are you using terraform to deploy k8s, your app stack on k8s, or both?

throw1234651234 6y ago
To follow this - what do you feel K8S provides on top of terraform? We used K8S on a large project and I felt like it really, really wasn't necessary.
- verdverm 6y ago
  Consistency and standardized interfaces for AppOps regardless of the hyper-cloud I use. Kubernetes basically has an equivalent learning curve, but you only have to do it once
- nvarsj 6y ago
  They operate at different layers. K8s sits on top of the infrastructure which terraform provisions. It's far more dynamic and operates at runtime, compared to terraform which you execute ad-hoc from an imperative tool (and so only makes sense for the low level things that don't change often).
- StreamBright 6y ago
  Nothing. We use Terraform to provision a simple auto-scaling cluster with loadbalancers and certs, does exactly the same thing but there is no Docker and k8s. Few million lines less Go code turning yaml filed into seggfaults.
silviogutierrez 6y ago
So terraform is a higher-order, meta-Kubernetes. It's very rarely used, but who provisions the cluster itself? That's terraform. So terraform creates the cluster, DNS and VPC. Then k8s runs pretty much everything.
- pvorb 6y ago
  Why would you want to provision your own k8s cluster, if you can use EKS, AKS or similar?

herval 6y ago

Have you tried or considered Nomad (from the makers of Terraform)?

silviogutierrez 6y ago
I haven't, and since I've sunk the cost into Kubernetes and know it very well now, likely won't end up there. In retrospect though, maybe it's exactly what I needed. Great suggestion.
p_l 6y ago
can it handle networking (including load balancing and reverse proxies with automatic TLS) or virtualized persistent storage? Make it easy to integrate common logging system? Cause those are the parts that I miss probably the most when dealing with non-k8s deployment, and I haven't had the occasion to use Nomad.
- adadgar 6y ago
  For load balancing you can just run one of the common LB solutions (nginx, haproxy, Traefik) and pick up the services from the Consul service catalog. Traefik makes it quite nice since it integrates with LetsEncrypt and you can setup the routing with tags in your Nomad jobs: https://learn.hashicorp.com/nomad/load-balancing/traefik https://learn.hashicorp.com/nomad/load-balancing/traefik What Nomad doesn’t do is setup a cloud provider load balancer for you. For persistent storage, Nomad uses CSI which is the same technology K8s does: https://learn.hashicorp.com/nomad/stateful-workloads/csi-volumes https://learn.hashicorp.com/nomad/stateful-workloads/csi-vol... Logging should be very similar to K8S. Both Nomad and K8S log to a file and a logging agent tails and ships the logs. Disclosure, I am a HashiCorp employee.
  - p_l 6y ago
    Thanks, definitely widened my understanding of Nomad in pretty short time :) Kinda feels bad that I don't have anything to use it on right now.
  - VectorLock 6y ago

jcastro 6y ago

> I can and have repointed my entire infrastructure with minimal fuss. When you get to that blog post please consider going in depth on this. Would love to see actual battletested information vs. the usual handwavy "it works everywhere".

silviogutierrez 6y ago
I sure will. 99% of the work was ingress handling and SSL cert generation. Everything else was fairly seamless. Even ingress is trivial if you use a cloud balancer per ingress. But I wanted to save money so use a single cloud balancer for multiple ingresses. So you need something like ingress-nginx, which has a few vendor-specific subtleties.

bosswipe 6y ago

According to the article you are wrong about "infrastructure as code". Kubernetes is infrastructure as data, specifically YAML files. Puppet and Chef are infrastructure as code. Edit: not sure why the down votes, I was just trying to point out what seems like a big distinction that the article is trying to make.

monadic2 6y ago
What’s the material difference between well-formatted data and a DSL? Why does this matter?
danaur 6y ago
Seems like a nitpick? Infra as data seems like a subset of infrastructure as code
silviogutierrez 6y ago
Maybe? I'm too lazy to formally verify if the YAML files k8s accepts are Turing complete. With kustomize they might very well be. How about "infrastructure-as-some-sort-of-text-file-versioned-in-my-repository". It's a mouthful, but maybe it'll catch on.
- lmkg 6y ago
  Infrastructure-as-config?
- geofft 6y ago
  They don't do loops or recursion. They don't even do iterative steps in the way that Ansible YAML has plays/tasks. Yes, higher-level tools like Kustomize or Jsonnet or whatever else you use for templating the files are Turing-complete - but that's at the level of you on your machine generating input to Kubernetes, not at the level of Kubernetes itself. That's a valuable distinction - it means you can't have a Kubernetes manifest get halfway through and fail the way that you can have an Ansible playbook get halfway through and fail; there's no "halfway." If something fails halfway through your Jsonnet, it fails in template expansion without actually doing anything to your infrastructure. (You can, of course, have it run out of resources or hit quota issues partway through deploying some manifest, but there's no ordering constraint - it won't refuse to run the "rest" of the "steps" because an "earlier step" failed, there's no such thing. You can address the issue, and Kubernetes will resume trying to shape reality to match your manifest just as if some hardware failed at runtime and you were recovering, or whatever.)

bdcravens 6y ago

Of course it doesn't have the advantage of #2, but I've found ECS to be far easier to grok and implement.

UweSchmidt 6y ago

With "Swarm", do you mean Docker Swarm? Why has it "fizzled"? The way I learned it in Bret Fisher's Udemy course, Swarm is very much relevant, and will be supported indefinitely. It seems to be a much simpler version of Kubernetes. It has both composition in YAML files (i.e. all your containers together) and the distribution over nodes. What else do you need before you hit corporation-scale requirements?

morelisp 6y ago
> What else do you need before you hit corporation-scale requirements? Cronjobs, configmaps, and dynamically allocated persistent volumes have been big ones for our small corporation. Access control also, but I'm less aware of the details here, other than that our ops is happier to hand out credentials with limited access, which was somehow much more difficult with swarm Swarm has frankly also been buggy. "Dead" but still running containers - sometimes visible to swarm, sometimes only the local Docker daemon - happen every 1-2 months, and it takes forever to figure out what's going on each time.
sdan 6y ago
I use Swarm in production and am learning k8s as fast as possible because of how bad Swarm is: 1. Swarm is dead in the water. No big releases/development afaik recently 2. Swarm for me has been a disaster because after a couple of days some of my nodes slowly start failing (although they’re perfectly normal) and I have to manually remove each node from the swarm, join them, and start everything up again. I think this might be because of some WireGuard incompatibility, but the strange thing is that it works for a week sometimes and other times just a few hours 3. Lack of GPU support
- GordonS 6y ago
  To add another side, I use Swarm in production and continue to do so because of how good it is. I've had clusters running for years without issue. I've even used it for packaging B2B software, where customers use it both in cloud and on-prem - no issues whatsoever. I've looked at k8s a few times, but it's vastly more complex than Swarm (which is basically Docker Compose with cluster support), and would add nothing for my use case. I'm sure a lot of people need the functionality that k8s brings, but I'm also sure that many would be better suited to Swarm.

McAtNite 6y ago

Out of curiosity why do you feel swarm fizzled out? I’ve deployed swarm in a home lab and found it really simple to work with, and enjoyable to use. I haven’t tried k8, but I often see view points like yours stating that k8 is vastly superior.

lifeisstillgood 6y ago

So are you saying that, no matter what, if you want to reply your whole infrastructure as code (networks, dmz, hosts, services, apps, backups etc ) that you are going to have to reproduce that somehow (whatever the combo of AWS services are, OR just learn K8S Effectively, "every infrastructure as code project will reimplement Kubernetes in Bash"

silviogutierrez 6y ago
Exactly. That's a great way to put it. A bunch of Bash glue reimplementing what Kubernetes already does. Poorly.
- roman_sf 6y ago
  Kubernetes is outstanding because proving ol bash scripts were bad and re-writing all Bash glue in Go glue, slapping 150 APIs on top (60 "new" and about hundred "old versions" https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18 https://kubernetes.io/docs/reference/generated/kubernetes-ap...), adding few dozen opensource must have projects - so finally it can be called "cloud native" - boom, a new cloud native bash for the cloud is born!
  - tannhaeuser 6y ago
    Actually it's worse than that - k8s was ported from Java, and the codebase is an incoherent mess, as k8s devs say themselves
- gfodor 6y ago
  Bash glue has rarely failed me. Never used k8s, but the horror stories I hear ensure I won’t feel dirty for writing a little bash to solve problems anytime soon.
aprdm 6y ago
Not necessarily. You can have all of above with Terraform, Ansible, Puppet, Chef... etc.

tyingq 6y ago

There's an offshoot of this that I see from developers, especially at old, stodgy companies. Once everything is "infrastructure as code", the app team becomes less dependent on other teams in the org. People like to own their own destiny. Of course, that also removes a lot of potential scapegoats, so you now mostly own all outages, tech debt, etc.

duxup 6y ago
I think that's been coming since ... well ever. I worked in networking for the longest time. When I started there network guys and server guys (at least where I was). They were different people who did different things who kinda worked together. Then there were storage area networks and similar, networks really FOR the server and storage guys.... that kind of extended the server world over some of the network. Then comes VMware and such things and now there was a network in a box somewhere that was entirely the server guy's deal (well except when we had to help them... always). Then we also had load balances who in their own way were a sort of code for networks ... depending on how you looked at it (open ticket #11111 of 'please stop hard coding ip addresses'). You also had a lot of software defined networking type things and so forth brewing up in dozens of different ways. Granted these descriptions are not exact, there were ebs and flows and some tech that sort of did this (or tired) all along. It all starts to evolve slowly into one entity.

gigatexal 6y ago

imo, all swarm needed was a decent way to handle secrets and it would have been the k8s of its day

tombert 6y ago

Swarm is still supported and works. I have it running on my home server and love it. Kubernetes is fine, but setting it up kind of feels like I'm trying to earn a PhD thesis. Swarm is dog-simple to get working and I've really had no issues in the three years that I've been running it. The configs aren't as elaborate or as modular as Kubernetes, and that's a blessing as well as a curse; it's easy to set up and administer, but you have less control. Still, for small-to-mid-sized systems, I would still recommend Swarm.

theptip 6y ago
> setting it up kind of feels like I'm trying to earn a PhD thesis. Are you following "k8s the hard way"? I've never had this problem; either: `gcloud container clusters create` Or `install docker-for-mac` And you have a k8s cluster up and running. Maybe it's more work on AWS?
- kronin 6y ago
  Ekscli (if you want managed) and kops (if you want to manage it yourself) are just as straightforward.
ithkuil 6y ago
> setting it up kind of feels like I'm trying to earn a PhD thesis. The kind of people who has to both set the cluster up and keep it up and also has to develop the application and deploy it and keep it up etc is not the target audience. K8s shines when the roles of managing the cluster and running workloads on it are separated. It defines a good contract between infrastructure and workload. It lets different people focus on different aspects. Yes it still has rough edges, things that are either not there yet, or vestigial complexity of wrong turns that happened through it's history. But if you look at it through the lense of this corporate scenario it starts making more sense than when you just think of what a full-stack dev in a two person startup would rather use and fully own/understand.
- DelightOne 6y ago
  What are Kubernetes' rough edges?

archsurface 6y ago

I'm not sure "a plain ol' Django monolith" with none of the "fancy stuff" is either what people are referring to when they say "kubernetes", or a great choice for that. I could run hello world on a Cray but that doesn't mean I can say I do supercomputing. Our team does use it for all the fancy stuff, and spends all day everyday for years now yamling, terraforming, salting, etc so theoretically our setup is "entire application stack as YAML files, fully reproducible", but if it fell apart tomorrow, I'd run for the hills. Basically, I think you're selling it from a position which doesn't use it to any degree which gives sufficient experience required to give in-depth assessment of it. You're selling me a Cray based on your helloworld.

silviogutierrez 6y ago
Reading this charitably: I guess I agree. k8s is definitely overpowered for my needs. And I'm almost certain my blog or my business will never need that full power. Fully aware of that. But I'm not sure one can find something of "the right power" that has the same support from cloud providers, the open source community, the critical mass, etc. [1] Eventually, a standard "simplified" abstraction over k8s will emerge. Many already exist, but they're all over the place. And some are vendor specific (Google Cloud Run is basically just running k8s for you). Then if you need the power, you can eject. Something like Create React App, but by Kubernetes. Create Kubernetes App. [1] Though Nomad looks promising.
- wolco 6y ago
  Curious why run it at all? The cost must be 10 times more this way. It is mostly for the fun of learning. I come from the opposite approach. I have 4 servers two digital ocean $5 and two vulr $2.50 instances. One holds the db. One server as the frontend/code. One server to do heavy work and another to server a heavy site and holds backups. For $15 I'm hosting hundreds of sites, running so many background processes. I couldn't imagine hitting that point where k8s would make sense just for myself unless for fun.
  - StavrosK 6y ago

bulldoa 6y ago

Do you have a recommended tutorial for engineer with backend background to setup a simple k8 infra in ec2?

hartem_ 6y ago
Take a look at https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way. That’d be a great first step if the purpose is to learn Kubernetes. If, however, you want to set up a cluster for real use then you will need much more than bare bones Kubernetes (something that solves networking, monitoring, logging, security, backups and more) so consider using a distribution or a managed cloud service instead.
- physicles 6y ago
  Setting up your own k8s from scratch is kind of like writing your own string class in C++: it’s a good exercise (if it’s valuable for your learning path) but you probably don’t want to use it for actual work. Maintaining a cluster set up like that is a ton of work. And if you don’t perform an upgrade perfectly, you’ll have downtime. Tools like kops help a lot but you’ll still spend far more time than the $70/month it costs for a managed cluster.
turtlebits 6y ago
I find that K3s is great for getting started. It has traefik included and its less of a learning curve to actually be productive, vs diving in with K8s and having to figure out way more pieces.

deleted 6y ago

[deleted]

kasey_junk 6y ago

Conversely I think the trend of writing infrastructure as yaml may be the worst part of modern ops. It’s really hard to think of a worse language for this.

banifo 6y ago
Why? We have plenty of yaml 'code' which is simple and does exactly what it needs to do. For all other usecases, there are plenty of alternatives, including libs for your preferred language. Most people use the yaml way because its easy and does exactly what it needs to do. Everyone else has plenty of well supported and well working alternatives.
- kasey_junk 6y ago
  Its typing is too weak. It’s block scoping is dangerous. References, classes and attachments are all pretty bad for reuse. Schemas are bolt on and there is no standard query language for it.
p_l 6y ago
Fortunately YAML is not actually necessary in k8s, and only provided as a convenience because writing JSON by hand (or Proto3, lol) is annoying verging on insane. We can build higher level abstractions easily having a schema to target and we can build them in whatever we want. That's a big boon for me :)
- kasey_junk 6y ago
  For kubernetes thats absolutely true and I think more people should do that (disclaimer I work with this https://github.com/stripe/skycfg https://github.com/stripe/skycfg daily). I think it actually would be easier for people to understand the k8s system if they did. But, yaml is now everywhere in the ops space. Config management systems use it, metrics systems use it, its the defacto configuration format right now and that is unfortunate cause its bad.

collyw 6y ago

Do you have a good tutorial for doing Django or a standard 3 tier web app on Kubernetes? We are using kubernetes at my workspace, but it seems way too complicated to consider for something like that. Maybe if I can bridge the gap between architectures it will help.