9 ms·

> If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. What is the alternative to k8s that does not ne

> If you want k8s, you really do need people that know how to maintain it on a more or less full time schedule. What is the alternative to k8s that does not need people to have any technical knowledge? To me Kubernetes is extremely attractive because it helps me avoid learning cloud vendors' proprietary technologies. K8s is learn once, use everywhere, which is fantastic. I am a 1-person venture doing everything from JavaScript/React to maintaining backend infra, and I couldn't have done it without k8s.

lifeisstillgood 6y ago
Firstly tell me more about your 1-person full-stack venture, but second how comes you, with barely any time for sitting down can use k8s happily but it falls over for others. I am struggling to see truth amount the comments here :-(
- dashwav 6y ago
  TBF with a completely greenfield project and managed K8s (GKE or EKS) you can absolutely get a pretty well set up infra very quickly if you are willing to learn how to do so. I often get the feeling a lot of the negativity comes (rightfully so) from trying to replicate a current existing project into kubernetes. This is true of almost any paradigm - try replicating a Java EE monolith into Erlang and you are going to have a lot of problems. The big thing to note is that starting a project on Erlang very well might solve the problems that a Java EE project ran into, but that is because they were able to solve them at the ground floor, and just popping a Java EE project with all of it's architecture into an Erlang project will probably end up in a worse spot. I think that this is what often happens with k8s as well - if you or your company have a currently working implementation that isn't on k8s, of course you won't be able to just easily plop it into a k8s cluster and everything be all well and good, but I think the problem is that people are equating that issue with k8s itself, which is a completely different paradigm.
  - solidasparagus 6y ago
    > managed K8s (GKE or EKS) you can absolutely get a pretty well set up infra very quickly And then tear your hair out when something doesn't work for some reason and root causing it requires learning a stupid number of layers. k8s is easy until it goes wrong.
    - harpratap 6y ago
      Isn't this the same for every software? How is debugging issues with Linux, NGINX, any complex framework any easier?
      - Aeolun 6y ago
        Well, for one thing, all logs are in /var/log. If it doesn’t fail just right in kubernetes there might be no logs at all. I’m thinking on particular about trying to mount filesystems into pods.
      - lenkite 6y ago
        Debugging k8s issues is like debugging a vast distributed monolith with a vague guesses on where the problem occurred.
- snuxoll 6y ago
  Not the GP, but I honestly couldn't tell you. A lot probably comes down to tooling, the applications you are deploying, security requirements, etc., as well as how familiar you are with k8s itself. I migrated PCGamingWiki from running on some Hetzner boxes to DigitalOcean Kubernetes in a few days of work creating Dockerfiles and k8s manifests. I run a Kubernetes cluster at work fairly hands-free that hosts applications critical to our billing operations, and developers on the team deploy new applications with little or no support. Any of the issues I've hit are an artifact of migrating legacy applications not designed to run in more-or-less stateless environments, which is why the PCGW Community site still runs on its own server (Invision Community sucks). I really don't see all the issues people have that aren't due to a mismatch of application design vs target environment (and no, it's not monolith vs microservice - monoliths run just fine on k8s; but you should be designing your application with a 12-factor environment in mind) or a misguided notion that you will be drowning in YAML hell (it's real, but you can manage it - and it's directly related to the complexity of the services you are deploying).
  - ramraj07 6y ago
    Seems like you are a thoughtful engineer who can also make sure you don't make any fundamental design flaws while building these systems well. Whenever I have seen kubernetes fail it's often also because the engineer(s) who built it were not thoughtful at all and often didn't fully understand what they were doing. Perhaps k8s's failing is it makes people who dont know enough think that they do.
    - snuxoll 6y ago
      I mean, I’m far from perfect - the trick has always been to KISS. I don’t use istio, it’s absolute overkill for my needs. I use nginx-ingress because it fits the bill, I know nginx as do enough other people that they could exec into a pod to debug it. I don’t run stateful applications that aren’t prepared to have servers randomly vanishing because it take a LOT of work to get these running in-cluster. I don’t use public helm charts because they often suck and making your own container is something you can do quickly if you were able to deploy the software on a traditional server. Every choice I make is done with the day 2 operations in mind, not what is hot, what gets initially deployed fastest - but what makes it so I can touch the thing as frequently as possible. PCGW is a great example - installing a new Mediawiki extension, changing a config file, upgrading to a newer MW release is just updating a file or a git submodule and committing. I don’t get paid thousands a month to manage the site for the owner, so I make my time spent as efficient as can be done.
- banifo 6y ago
  Might also be the amount of experience a person has. I have done Softwaredevelopment for ~15 years, swiched now to infrastructure and build a gke cluster. It was awesome, very logical, nice and easy to use. Now i read stories from coinbase and don't get it.
- lmm 6y ago
  People who don't like - or don't "get" - declarative systems tend to spend an inordinate amount of time and effort fighting them. I've seen the same thing with a delcarative build system (maven), or with adopting an ORM - if you're willing to work with the tool then it will save you a lot of effort, but if you're determined to do things your own way then you can make it almost arbitrarily difficult.
  - antonvs 6y ago
    > if you're determined to do things your own way then you can make it almost arbitrarily difficult. This is true of software development in general, if not life itself! But you're right that declarative systems amplify this issue.
- antonvs 6y ago
  This is pretty typical when a product is still climbing the adoption curve. I've helped small companies set up (managed) k8s clusters and migrate their apps to them, and when you know what you're doing, it's a super smooth experience that's basically all upside. But, if you're approaching it for the first time with no assistance, there are lots of things that can trip you up, and lots to learn. That's not a reflection on k8s, it's just the nature of the large set of problems it's solving. K8s is succeeding because it's very well designed, has a large and diverse ecosystem, and solves set of important problems that very few other tools even try to tackle. Apache Mesos perhaps comes closest, but it's not quite as pragmatic, and its adoption level reflects that. Also, because of k8s' scope, many people may not fully appreciate the range of problems it's solving, seeing it through the lens of their own background and focus.
ponyous 6y ago
I feel like what 98% companies need is really just barebones linux with some good documentation how to spawn new nodes. To use k8s you need to know linux anyways, but to use linux you don’t need k8s knowledge. Most of things are as easy to setup with linux and the things where k8s really shines are not needed most often. What I really wonder is where did you learn k8s, which parts did you learn the most? It seems huge and I would love to be in your position
- joana035 6y ago
  K8s is a nice api on top of gnu/linux. Want a iptables rule? Write a yaml (network policy). Want a storage for you app? Write a yaml (persistent volume) etc etc. For people who already know Linux, kubernetes comes naturally because it is pretty obvious. But indeed, by experience, many companies can go to "unicorn scale" with two or three boxes.
  - hwbehrens 6y ago
    Couldn't you get nearly the same behavior using some basic Ansible playbooks? My impression was that the killer feature of k8s was scaling, automatic failover, etc., although to be fair it's been several years since I last looked into it.
    - joana035 6y ago
      For me k8s kinda extends what you can do with ansible. For example, if you use just kubelet (the main daemon) without the api, drop some static yaml files (ala unit files) under /etc/kubernetes it will be something like systemd is. No big deal so far, but using the kubernetes api (just another daemon) it allows you to run the all the things anywhere. I mean, you terminate one computer, k8s will figure out that the program should be running and it will be started in another computer. (Including moving detaching/attaching the block device from/into computers, this kind of things) There is no much secret. For me it really feels like an (well designed/integrated) API on top of standard Linux technology (ipvs, netfilter, mount, process mgmt, virtual ip, etc) via yaml running your stuff in 1..N computers but you manage it as one big computer :) EDIT: added more context within "()"
mmgutz 6y ago
I'm in the same boat opting for docker-compose instead. docker-compose is much simpler to manage. Obviously it doesn't have feature parity with k8s but docker-compose does the basics well. An inexpensive VPS runs compose well with more resources at a lower cost than the managed costs of k8s.
- threeseed 6y ago
  I don't understand this comment at all. Kubernetes is designed for a cluster of nodes not a single one. Of course a single VPS is lower cost and easier to use but what happens if you want a second or third. Or any form of redundancy.
  - j4ah4n 6y ago
    I presume they are using docker swarm in this instance.
  - imtringued 6y ago
    You just run scp with the compose file and run docker compose down; docker compose up Bonus points if you just mount the compose file on an NFS share. I've wasted enough man months on kubernetes that unless you tell me to manage 1000 nodes this approach will never cost me more time than the time I spent using and learning kubernetes.
    - temptemptemp111 6y ago
      docker-compose.yml on an NFS share? What's the benefit? Or you mean sharing the files between containers?
  - folmar 6y ago
    A lot of people use it for deployment standard and don't care about multinode.
- temptemptemp111 6y ago
  Agreed. Good support for compose currently, and swarm is usually overkill. With a bit more elegant HA functionality, docker-compose could be the go-to for many more. The comment below claiming he doesn't understand your comment reminds me of all of those who will say "well its not FOR production" - seems more like superstition than science.
zaptheimpaler 6y ago
Plain old linux is the alternative, which is also "learn once, use everywhere" whether its AWS EC2 or GCP Instances or nearly any machine under the sun. I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS without k8s - read their docs, figure out the API. Maybe i'm an "old man yelling at new tech" but meh i just see very little value in k8s because you inevitably need to understand the layer beneath - linux (k8s is far from a non-leaky abstraction imo), PLUS all the complexity of k8s itself. I do see the value when managing a big and complex infra with 100s of servers or something, but very few people have that problem.
- threeseed 6y ago
  > I don't see how k8s avoids the need to learn about cloud vendor specific tech e.g searching "aws RDS k8s" Not sure what you mean here. Kubernetes is designed to be cloud and vendor agnostic. And RDS is a hosted database that you connect to from your application. Whether that applications runs standalone, in a container or VM is irrelevant.
  - deleted 6y ago
    [deleted]
  - Aeolun 6y ago
    > Kubernetes is designed to be cloud and vendor agnostic. But it’s not. Connecting to EKS is completely different from connecting to GCP. Setting up worker nodes is completely different too, oh and load balancers. It’s only the last mile that’s similar.
    - 15155 6y ago
      Provisioning a new cluster on each of these platforms is tens of lines of Terraform.
    - dtech 6y ago
peterwwillis 6y ago
> What is the alternative to k8s that does not need people to have any technical knowledge? Paying for a managed provider. Heroku, Elastic Beanstalk, GAE, GCP, Fargate, etc. You push some buttons, they manage your cluster/services. People still think they can get a free lunch by downloading some free software. If that were so, Windows and Mac would be dead and Linux would be the only desktop OS. But good news: I hear 2030 will be the year of the Linux Desktop!
hardwaresofton 6y ago
Check out Dokku[0] [0]: https://github.com/dokku/dokku https://github.com/dokku/dokku
OOPMan 6y ago
Couldn't have done it without k8s? Ahahahahaha, kids say the darndest things these days...

lifeisstillgood 6y ago

Firstly tell me more about your 1-person full-stack venture, but second how comes you, with barely any time for sitting down can use k8s happily but it falls over for others. I am struggling to see truth amount the comments here :-(

dashwav 6y ago
TBF with a completely greenfield project and managed K8s (GKE or EKS) you can absolutely get a pretty well set up infra very quickly if you are willing to learn how to do so. I often get the feeling a lot of the negativity comes (rightfully so) from trying to replicate a current existing project into kubernetes. This is true of almost any paradigm - try replicating a Java EE monolith into Erlang and you are going to have a lot of problems. The big thing to note is that starting a project on Erlang very well might solve the problems that a Java EE project ran into, but that is because they were able to solve them at the ground floor, and just popping a Java EE project with all of it's architecture into an Erlang project will probably end up in a worse spot. I think that this is what often happens with k8s as well - if you or your company have a currently working implementation that isn't on k8s, of course you won't be able to just easily plop it into a k8s cluster and everything be all well and good, but I think the problem is that people are equating that issue with k8s itself, which is a completely different paradigm.
- solidasparagus 6y ago
  > managed K8s (GKE or EKS) you can absolutely get a pretty well set up infra very quickly And then tear your hair out when something doesn't work for some reason and root causing it requires learning a stupid number of layers. k8s is easy until it goes wrong.
  - harpratap 6y ago
    Isn't this the same for every software? How is debugging issues with Linux, NGINX, any complex framework any easier?
    - Aeolun 6y ago
      Well, for one thing, all logs are in /var/log. If it doesn’t fail just right in kubernetes there might be no logs at all. I’m thinking on particular about trying to mount filesystems into pods.
    - lenkite 6y ago
      Debugging k8s issues is like debugging a vast distributed monolith with a vague guesses on where the problem occurred.
snuxoll 6y ago
Not the GP, but I honestly couldn't tell you. A lot probably comes down to tooling, the applications you are deploying, security requirements, etc., as well as how familiar you are with k8s itself. I migrated PCGamingWiki from running on some Hetzner boxes to DigitalOcean Kubernetes in a few days of work creating Dockerfiles and k8s manifests. I run a Kubernetes cluster at work fairly hands-free that hosts applications critical to our billing operations, and developers on the team deploy new applications with little or no support. Any of the issues I've hit are an artifact of migrating legacy applications not designed to run in more-or-less stateless environments, which is why the PCGW Community site still runs on its own server (Invision Community sucks). I really don't see all the issues people have that aren't due to a mismatch of application design vs target environment (and no, it's not monolith vs microservice - monoliths run just fine on k8s; but you should be designing your application with a 12-factor environment in mind) or a misguided notion that you will be drowning in YAML hell (it's real, but you can manage it - and it's directly related to the complexity of the services you are deploying).
- ramraj07 6y ago
  Seems like you are a thoughtful engineer who can also make sure you don't make any fundamental design flaws while building these systems well. Whenever I have seen kubernetes fail it's often also because the engineer(s) who built it were not thoughtful at all and often didn't fully understand what they were doing. Perhaps k8s's failing is it makes people who dont know enough think that they do.
  - snuxoll 6y ago
    I mean, I’m far from perfect - the trick has always been to KISS. I don’t use istio, it’s absolute overkill for my needs. I use nginx-ingress because it fits the bill, I know nginx as do enough other people that they could exec into a pod to debug it. I don’t run stateful applications that aren’t prepared to have servers randomly vanishing because it take a LOT of work to get these running in-cluster. I don’t use public helm charts because they often suck and making your own container is something you can do quickly if you were able to deploy the software on a traditional server. Every choice I make is done with the day 2 operations in mind, not what is hot, what gets initially deployed fastest - but what makes it so I can touch the thing as frequently as possible. PCGW is a great example - installing a new Mediawiki extension, changing a config file, upgrading to a newer MW release is just updating a file or a git submodule and committing. I don’t get paid thousands a month to manage the site for the owner, so I make my time spent as efficient as can be done.
banifo 6y ago
Might also be the amount of experience a person has. I have done Softwaredevelopment for ~15 years, swiched now to infrastructure and build a gke cluster. It was awesome, very logical, nice and easy to use. Now i read stories from coinbase and don't get it.
lmm 6y ago
People who don't like - or don't "get" - declarative systems tend to spend an inordinate amount of time and effort fighting them. I've seen the same thing with a delcarative build system (maven), or with adopting an ORM - if you're willing to work with the tool then it will save you a lot of effort, but if you're determined to do things your own way then you can make it almost arbitrarily difficult.
- antonvs 6y ago
  > if you're determined to do things your own way then you can make it almost arbitrarily difficult. This is true of software development in general, if not life itself! But you're right that declarative systems amplify this issue.
antonvs 6y ago
This is pretty typical when a product is still climbing the adoption curve. I've helped small companies set up (managed) k8s clusters and migrate their apps to them, and when you know what you're doing, it's a super smooth experience that's basically all upside. But, if you're approaching it for the first time with no assistance, there are lots of things that can trip you up, and lots to learn. That's not a reflection on k8s, it's just the nature of the large set of problems it's solving. K8s is succeeding because it's very well designed, has a large and diverse ecosystem, and solves set of important problems that very few other tools even try to tackle. Apache Mesos perhaps comes closest, but it's not quite as pragmatic, and its adoption level reflects that. Also, because of k8s' scope, many people may not fully appreciate the range of problems it's solving, seeing it through the lens of their own background and focus.

ponyous 6y ago

I feel like what 98% companies need is really just barebones linux with some good documentation how to spawn new nodes. To use k8s you need to know linux anyways, but to use linux you don’t need k8s knowledge. Most of things are as easy to setup with linux and the things where k8s really shines are not needed most often. What I really wonder is where did you learn k8s, which parts did you learn the most? It seems huge and I would love to be in your position

joana035 6y ago
K8s is a nice api on top of gnu/linux. Want a iptables rule? Write a yaml (network policy). Want a storage for you app? Write a yaml (persistent volume) etc etc. For people who already know Linux, kubernetes comes naturally because it is pretty obvious. But indeed, by experience, many companies can go to "unicorn scale" with two or three boxes.
- hwbehrens 6y ago
  Couldn't you get nearly the same behavior using some basic Ansible playbooks? My impression was that the killer feature of k8s was scaling, automatic failover, etc., although to be fair it's been several years since I last looked into it.
  - joana035 6y ago
    For me k8s kinda extends what you can do with ansible. For example, if you use just kubelet (the main daemon) without the api, drop some static yaml files (ala unit files) under /etc/kubernetes it will be something like systemd is. No big deal so far, but using the kubernetes api (just another daemon) it allows you to run the all the things anywhere. I mean, you terminate one computer, k8s will figure out that the program should be running and it will be started in another computer. (Including moving detaching/attaching the block device from/into computers, this kind of things) There is no much secret. For me it really feels like an (well designed/integrated) API on top of standard Linux technology (ipvs, netfilter, mount, process mgmt, virtual ip, etc) via yaml running your stuff in 1..N computers but you manage it as one big computer :) EDIT: added more context within "()"

mmgutz 6y ago

I'm in the same boat opting for docker-compose instead. docker-compose is much simpler to manage. Obviously it doesn't have feature parity with k8s but docker-compose does the basics well. An inexpensive VPS runs compose well with more resources at a lower cost than the managed costs of k8s.

threeseed 6y ago
I don't understand this comment at all. Kubernetes is designed for a cluster of nodes not a single one. Of course a single VPS is lower cost and easier to use but what happens if you want a second or third. Or any form of redundancy.
- j4ah4n 6y ago
  I presume they are using docker swarm in this instance.
- imtringued 6y ago
  You just run scp with the compose file and run docker compose down; docker compose up Bonus points if you just mount the compose file on an NFS share. I've wasted enough man months on kubernetes that unless you tell me to manage 1000 nodes this approach will never cost me more time than the time I spent using and learning kubernetes.
  - temptemptemp111 6y ago
    docker-compose.yml on an NFS share? What's the benefit? Or you mean sharing the files between containers?
- folmar 6y ago
  A lot of people use it for deployment standard and don't care about multinode.
temptemptemp111 6y ago
Agreed. Good support for compose currently, and swarm is usually overkill. With a bit more elegant HA functionality, docker-compose could be the go-to for many more. The comment below claiming he doesn't understand your comment reminds me of all of those who will say "well its not FOR production" - seems more like superstition than science.

zaptheimpaler 6y ago

Plain old linux is the alternative, which is also "learn once, use everywhere" whether its AWS EC2 or GCP Instances or nearly any machine under the sun. I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS without k8s - read their docs, figure out the API. Maybe i'm an "old man yelling at new tech" but meh i just see very little value in k8s because you inevitably need to understand the layer beneath - linux (k8s is far from a non-leaky abstraction imo), PLUS all the complexity of k8s itself. I do see the value when managing a big and complex infra with 100s of servers or something, but very few people have that problem.

threeseed 6y ago
> I don't see how k8s avoids the need to learn about cloud vendor specific tech e.g searching "aws RDS k8s" Not sure what you mean here. Kubernetes is designed to be cloud and vendor agnostic. And RDS is a hosted database that you connect to from your application. Whether that applications runs standalone, in a container or VM is irrelevant.
- deleted 6y ago
  [deleted]
- Aeolun 6y ago
  > Kubernetes is designed to be cloud and vendor agnostic. But it’s not. Connecting to EKS is completely different from connecting to GCP. Setting up worker nodes is completely different too, oh and load balancers. It’s only the last mile that’s similar.
  - 15155 6y ago
    Provisioning a new cluster on each of these platforms is tens of lines of Terraform.
  - dtech 6y ago

peterwwillis 6y ago

> What is the alternative to k8s that does not need people to have any technical knowledge? Paying for a managed provider. Heroku, Elastic Beanstalk, GAE, GCP, Fargate, etc. You push some buttons, they manage your cluster/services. People still think they can get a free lunch by downloading some free software. If that were so, Windows and Mac would be dead and Linux would be the only desktop OS. But good news: I hear 2030 will be the year of the Linux Desktop!

hardwaresofton 6y ago

Check out Dokku[0] [0]: https://github.com/dokku/dokku https://github.com/dokku/dokku

OOPMan 6y ago

Couldn't have done it without k8s? Ahahahahaha, kids say the darndest things these days...