5 ms·

At this point I'm pretty much convinced these repeated "Kubernetes is bad for startups" rants are some kind of FUD campaign (probably grass roots from people wh

At this point I'm pretty much convinced these repeated "Kubernetes is bad for startups" rants are some kind of FUD campaign (probably grass roots from people who've missed the containerisation and declarative infra train). Kubernetes is actually pretty great for the vast majority of use-cases. Sure if you don't have experience with kubernetes leave learning it until after you've hit product-market fit. But rejecting the advances made in this ecosystem is a red flag upon itself. My ultimate MVP starter pack is probably a big fat VM running a single node K3s instance and a cloud provider managed database. Software wise nginx-ingress / cert-manager / argo-cd is a pretty good default set.

pas 4y ago
could you elaborate on the "ideal" argocd workflow? let's say devs push to gitlab, which starts a pipeline, tests run, container images get built, then in the last step in the gitops repo a version bump happens and then argocd will pick it up automatically? (and if one wants to be able to rollback then that's the same workflow just instead of version "bump" the pipeline tags images with the commit hash and at the last step sets the version to the right tag?) are there any gotchas, best practices? :) thanks!
- Chilinot 4y ago
  For deployments using Argo, yea that's pretty much it. You can set it to automatically deploy to your target environment as soon as it detects changes to your k8s mainfests, or you can require a manual "approve" step which is a push of a button for configs that are out of sync with what is applied to k8s. For rollbacks, you have a button in argo that lists previous deployments and you can just choose which previous deployment you want to go back to. It works pretty well those (thankfully few) times i have had to rollback any deployments.
  - isbvhodnvemrwvn 4y ago
    Can it run smoke and acceptance tests? Or does it require calling external services?
    - grumpy-de-sre 4y ago
      For automated testing the argo-workflows project is amazing and worth looking into. I think when you look at the entire argo suite you start seeing something that could really disrupt the way we use products like gitlab, particularly for startups.
    - wavesquid 4y ago
      For smoke and acceptance tests you can use the sister project argo-rollouts: https://argoproj.github.io/argo-rollouts/ https://argoproj.github.io/argo-rollouts/ But that's going past an MVP/startup toolset.
- jq-r 4y ago
  Not the parent, but this is exactly what happens. ArgoCD is pointed to an "application" chart which just points to a path of a helm chart (of your app/sevice) in Git. So when your CI changes the image hash in the values file in your helm chart, ArgoCD with notice that, and change the deployment resource's image hash. It is just a nice system to have resources in sync between your git repo and what you have live in the cluster. Of course, if you change anything else, ArgoCD will change it too. You're also right, rolling back just includes changing the hash back to whatever you had, or to a new hash which was a result of a git revert or whatever. Also the good thing is that if that newly deployed service is very broken (that it doesn't pass the k8s health check), ArgoCD will hold on to the old ReplicaSet and will not let your service die because of it. What's also nice about ArgoCD is that you can play a bit with some service/application in a branch. Say you have some live service, and you want to adjust some configuration of it. Usually third party services have a lot of options to set. The problem is that you're not 100% sure how to get what you want, so doing a pull request for every small change can be very slow and exhausting. To work around that, you can point your ArgoCD's application chart to a chart which is in a branch, test/dev/fiddle with just pushing to that remote branch, and when you're satisfied, you merge your branch, and at the same time point your ArgoCD application manifest to point to the master/HEAD for that chart. In effect, at this step, only your Git repo will be updated, your service already has all the changes so ArgoCD will do nothing. That way you can iterate faster, and undo whatever regression you've introduced just by pointing ArgoCD to watch the master, not your branch (or you can just reset your branch to be indentical to master).
zozbot234 4y ago
"Containerization and declarative infra" are only a small fraction of what could be done with real network distributed OS's as early as the 1980s and 1990s, and with far lower levels of overall complexity. That "train" has left the station a long time ago, it's going full speed and there are no brakes on it either. Of course you can keep chugging along with clunky K8s and call that a "good default choice". But that's just being complacent about what real progress looks like.
- grumpy-de-sre 4y ago
  So where is this real progress happening today? I will say I think what the fly.io guys are doing with VMs on the edge and in-process distributed databases is pretty exciting.
- Chilinot 4y ago
  Could you give some examples of these network distributed OS's that are going full steam ahead? Also interested in why you think those are less clunky than k8s. :)
  - bheadmaster 4y ago
    Plan9 has basically invented the idea of namespaces, which are fundamental building blocks of containers. Plan9 also invented 9P file protocol which is used in WSL2. Plan9 is the ultimate distributed OS. Running programs on different computers is as easy as mounting a remote /dev/cpu file to your process' namespace. Virtual desktop is as easy as mounting /dev/draw. And so on. Many such complicated things that require ad-hoc solutions on Linux are solved on a fundamental level in Plan9. I wouldn't say that it's going full steam ahead, since its use in the modern world is fairly limited, but the ideas were there, and they have influenced the development of Linux.
    - vbezhenar 4y ago
      Distributed OS hides distributed nature from the application. Kubernetes does not try to hide it. It allows developer to embrace it. Hiding network latency is leaky abstraction. Kubernetes does it right. It does not pretend that there's no network between your services. It's the other way around: there's always network between your services. Sometimes it's loopback network, sometimes it's real network, but you're aware of it.
angio 4y ago
In my latest project I skipped the managed database and started using a postgres operator. So far the dev experience and performance is good.
- grumpy-de-sre 4y ago
  The PostgreSQL operators are definitely usable, pretty much none of their limitations are going to hit you on an MVP. I think a lot of the FUD about "don't run databases on kubernetes" came from a time before we had statefulsets and persistent volumes. Which operator did you go with out of curiousity?
  - angio 4y ago
    I'm using the Zalando one. It only took a couple of hours to get it running (first time running it) and integrated in graphana. Includes incremental backups to S3. It really feels like a cloud independent managed database.
quickthrower2 4y ago
Kubernetes is the React of devops. You wont get fired for picking it. And there will be a tonne of support available online, as well as courses, and experienced people you can hire. Also there is cloud managed k8s which takes most of the pain out of it. And plenty of out the box stuff. It is not a bad choice if you want to do the ops yourself for some reason. If you don't then use a BaaS or PaaS, that might be easier and probably not much more expensive, but will lock you in.
- thathndude 4y ago
  This has always been my observation/thought as well. It’s “ no one ever got fired for picking IBM” all over again. Cute advertisement slogan, but it’s based on a true/real mentality, and a dangerous one at that (in terms of reaching potential and managing burn).
- andybak 4y ago
  You seem to be saying there's nothing inbetween K8s and "use a BaaS or PaaS"? What about something that does the subset of what K8s does that is the simple bit and covers most use-cases outside of complex clusters? I get the strong sense people are just misusing it for the wrong types of task. I've seen this happen countless times in tech. Development by CV.
  - digianarchist 4y ago
    Docker Swarm, Nomad, ECS?
    - andybak 4y ago
      I've seen K8S recommended for single server installs with a couple of services. i.e. people jumping on instead of vanilla Docker/Docker Compose.
      - quickthrower2 4y ago
        Following that argument why use Docker? Just spin up a VM and install it on that.
andybak 4y ago
> Kubernetes is actually pretty great for the vast majority of use-cases. From my brief glances at it, it seems like a great solution for very complex use-cases. But "vast majority"? The cost-benefit seems way off unless you have a very different sense of "typical use-cases" to me. Most people are building simple web sites and apps that run on a single server, surely?
- thathndude 4y ago
  Agreed. OC is overstating it. It’s not that the anti K8s crowd is necessarily anti-containerization, it’s that containerization doesn’t necessarily require Something as complex as K8s. I’m invested in Cycle.io, which is a company that simplifies containerization and infrastructure (major oversimplification of the company). But a lot of customers are refugees from K8s, whose appetite exceeded common sense. Their infrastructure ops teams are costing them 6-7 figs that they can shed with something like Cycle, because K8s is simply really complex. This is going to become really really important in the next 3-4 lean years we have coming our way. If you’re a K8s user, and spending money you can’t afford on an ops team to manage, you owe it to your company, investors, and customers to explore options to reduce this type of burn. P.S. Intentionally not making a hyperlink for Cycle.io because I’m honestly not trying to shill here.
- afiori 4y ago
  > Most people are building simple web sites and apps that run on a single server, surely? I believe so, which makes a comparison between k3s and docker-compose very interesting.
  - andybak 4y ago
    I learned the basics of docker-compose in about an hour and I don't see myself outgrowing it in the near future.
KaiserPro 4y ago
> Kubernetes is actually pretty great for the vast majority of use-cases Its a pain in the arse, and you're on the hook for looking after it. I'm captain infra, so when Managed DBs came along, I felt threatened. But then when I used it, I realised that, yes, they are expensive, but I don't have to fucking worry about them. One touch deploy, proper backups are an option away, as is proper role based auth. K8s is like having a DB admin. Everything revolves around them, and it can be limiting, unless you have a specific use case. K8s is useful for a specific role, but for most people who've outgrown a single machine/bunch of machines, ECS is good enough. Yes, there are fancy things you can't natively do, but you probably shouldn't do those just yet anyway. Where K8s comes in useful is probably limited to: 1) mixture of real steel and cloud 2) A cluster shared with a large number of team, each deploying services that rely on other people's stuff I've been managing clusters in one for or another for many years. K8s is certainly better than swarm, but its really not that great as a general purpose "Datacentre OS". in short, K8s is your generation's XML.
bob1029 4y ago
> probably grass roots from people who've missed the containerisation and declarative infra train I am curious. How much work would it take to get on this train in 2022? Sounds like more of a cult than a valuable technology proposition when presented in this manner.

pas 4y ago

could you elaborate on the "ideal" argocd workflow? let's say devs push to gitlab, which starts a pipeline, tests run, container images get built, then in the last step in the gitops repo a version bump happens and then argocd will pick it up automatically? (and if one wants to be able to rollback then that's the same workflow just instead of version "bump" the pipeline tags images with the commit hash and at the last step sets the version to the right tag?) are there any gotchas, best practices? :) thanks!

Chilinot 4y ago
For deployments using Argo, yea that's pretty much it. You can set it to automatically deploy to your target environment as soon as it detects changes to your k8s mainfests, or you can require a manual "approve" step which is a push of a button for configs that are out of sync with what is applied to k8s. For rollbacks, you have a button in argo that lists previous deployments and you can just choose which previous deployment you want to go back to. It works pretty well those (thankfully few) times i have had to rollback any deployments.
- isbvhodnvemrwvn 4y ago
  Can it run smoke and acceptance tests? Or does it require calling external services?
  - grumpy-de-sre 4y ago
    For automated testing the argo-workflows project is amazing and worth looking into. I think when you look at the entire argo suite you start seeing something that could really disrupt the way we use products like gitlab, particularly for startups.
  - wavesquid 4y ago
    For smoke and acceptance tests you can use the sister project argo-rollouts: https://argoproj.github.io/argo-rollouts/ https://argoproj.github.io/argo-rollouts/ But that's going past an MVP/startup toolset.
jq-r 4y ago
Not the parent, but this is exactly what happens. ArgoCD is pointed to an "application" chart which just points to a path of a helm chart (of your app/sevice) in Git. So when your CI changes the image hash in the values file in your helm chart, ArgoCD with notice that, and change the deployment resource's image hash. It is just a nice system to have resources in sync between your git repo and what you have live in the cluster. Of course, if you change anything else, ArgoCD will change it too. You're also right, rolling back just includes changing the hash back to whatever you had, or to a new hash which was a result of a git revert or whatever. Also the good thing is that if that newly deployed service is very broken (that it doesn't pass the k8s health check), ArgoCD will hold on to the old ReplicaSet and will not let your service die because of it. What's also nice about ArgoCD is that you can play a bit with some service/application in a branch. Say you have some live service, and you want to adjust some configuration of it. Usually third party services have a lot of options to set. The problem is that you're not 100% sure how to get what you want, so doing a pull request for every small change can be very slow and exhausting. To work around that, you can point your ArgoCD's application chart to a chart which is in a branch, test/dev/fiddle with just pushing to that remote branch, and when you're satisfied, you merge your branch, and at the same time point your ArgoCD application manifest to point to the master/HEAD for that chart. In effect, at this step, only your Git repo will be updated, your service already has all the changes so ArgoCD will do nothing. That way you can iterate faster, and undo whatever regression you've introduced just by pointing ArgoCD to watch the master, not your branch (or you can just reset your branch to be indentical to master).

zozbot234 4y ago

"Containerization and declarative infra" are only a small fraction of what could be done with real network distributed OS's as early as the 1980s and 1990s, and with far lower levels of overall complexity. That "train" has left the station a long time ago, it's going full speed and there are no brakes on it either. Of course you can keep chugging along with clunky K8s and call that a "good default choice". But that's just being complacent about what real progress looks like.

grumpy-de-sre 4y ago
So where is this real progress happening today? I will say I think what the fly.io guys are doing with VMs on the edge and in-process distributed databases is pretty exciting.
Chilinot 4y ago
Could you give some examples of these network distributed OS's that are going full steam ahead? Also interested in why you think those are less clunky than k8s. :)
- bheadmaster 4y ago
  Plan9 has basically invented the idea of namespaces, which are fundamental building blocks of containers. Plan9 also invented 9P file protocol which is used in WSL2. Plan9 is the ultimate distributed OS. Running programs on different computers is as easy as mounting a remote /dev/cpu file to your process' namespace. Virtual desktop is as easy as mounting /dev/draw. And so on. Many such complicated things that require ad-hoc solutions on Linux are solved on a fundamental level in Plan9. I wouldn't say that it's going full steam ahead, since its use in the modern world is fairly limited, but the ideas were there, and they have influenced the development of Linux.
  - vbezhenar 4y ago
    Distributed OS hides distributed nature from the application. Kubernetes does not try to hide it. It allows developer to embrace it. Hiding network latency is leaky abstraction. Kubernetes does it right. It does not pretend that there's no network between your services. It's the other way around: there's always network between your services. Sometimes it's loopback network, sometimes it's real network, but you're aware of it.

angio 4y ago

In my latest project I skipped the managed database and started using a postgres operator. So far the dev experience and performance is good.

grumpy-de-sre 4y ago
The PostgreSQL operators are definitely usable, pretty much none of their limitations are going to hit you on an MVP. I think a lot of the FUD about "don't run databases on kubernetes" came from a time before we had statefulsets and persistent volumes. Which operator did you go with out of curiousity?
- angio 4y ago
  I'm using the Zalando one. It only took a couple of hours to get it running (first time running it) and integrated in graphana. Includes incremental backups to S3. It really feels like a cloud independent managed database.

quickthrower2 4y ago

Kubernetes is the React of devops. You wont get fired for picking it. And there will be a tonne of support available online, as well as courses, and experienced people you can hire. Also there is cloud managed k8s which takes most of the pain out of it. And plenty of out the box stuff. It is not a bad choice if you want to do the ops yourself for some reason. If you don't then use a BaaS or PaaS, that might be easier and probably not much more expensive, but will lock you in.

thathndude 4y ago
This has always been my observation/thought as well. It’s “ no one ever got fired for picking IBM” all over again. Cute advertisement slogan, but it’s based on a true/real mentality, and a dangerous one at that (in terms of reaching potential and managing burn).
andybak 4y ago
You seem to be saying there's nothing inbetween K8s and "use a BaaS or PaaS"? What about something that does the subset of what K8s does that is the simple bit and covers most use-cases outside of complex clusters? I get the strong sense people are just misusing it for the wrong types of task. I've seen this happen countless times in tech. Development by CV.
- digianarchist 4y ago
  Docker Swarm, Nomad, ECS?
  - andybak 4y ago
    I've seen K8S recommended for single server installs with a couple of services. i.e. people jumping on instead of vanilla Docker/Docker Compose.
    - quickthrower2 4y ago
      Following that argument why use Docker? Just spin up a VM and install it on that.

andybak 4y ago

> Kubernetes is actually pretty great for the vast majority of use-cases. From my brief glances at it, it seems like a great solution for very complex use-cases. But "vast majority"? The cost-benefit seems way off unless you have a very different sense of "typical use-cases" to me. Most people are building simple web sites and apps that run on a single server, surely?

thathndude 4y ago
Agreed. OC is overstating it. It’s not that the anti K8s crowd is necessarily anti-containerization, it’s that containerization doesn’t necessarily require Something as complex as K8s. I’m invested in Cycle.io, which is a company that simplifies containerization and infrastructure (major oversimplification of the company). But a lot of customers are refugees from K8s, whose appetite exceeded common sense. Their infrastructure ops teams are costing them 6-7 figs that they can shed with something like Cycle, because K8s is simply really complex. This is going to become really really important in the next 3-4 lean years we have coming our way. If you’re a K8s user, and spending money you can’t afford on an ops team to manage, you owe it to your company, investors, and customers to explore options to reduce this type of burn. P.S. Intentionally not making a hyperlink for Cycle.io because I’m honestly not trying to shill here.
afiori 4y ago
> Most people are building simple web sites and apps that run on a single server, surely? I believe so, which makes a comparison between k3s and docker-compose very interesting.
- andybak 4y ago
  I learned the basics of docker-compose in about an hour and I don't see myself outgrowing it in the near future.

KaiserPro 4y ago

> Kubernetes is actually pretty great for the vast majority of use-cases Its a pain in the arse, and you're on the hook for looking after it. I'm captain infra, so when Managed DBs came along, I felt threatened. But then when I used it, I realised that, yes, they are expensive, but I don't have to fucking worry about them. One touch deploy, proper backups are an option away, as is proper role based auth. K8s is like having a DB admin. Everything revolves around them, and it can be limiting, unless you have a specific use case. K8s is useful for a specific role, but for most people who've outgrown a single machine/bunch of machines, ECS is good enough. Yes, there are fancy things you can't natively do, but you probably shouldn't do those just yet anyway. Where K8s comes in useful is probably limited to: 1) mixture of real steel and cloud 2) A cluster shared with a large number of team, each deploying services that rely on other people's stuff I've been managing clusters in one for or another for many years. K8s is certainly better than swarm, but its really not that great as a general purpose "Datacentre OS". in short, K8s is your generation's XML.

bob1029 4y ago

> probably grass roots from people who've missed the containerisation and declarative infra train I am curious. How much work would it take to get on this train in 2022? Sounds like more of a cult than a valuable technology proposition when presented in this manner.