4 ms·

genuine question - why this over ArgoCD ? which is a huge ecosystem and all the declarative setups. Plus has clean github integration for incremental deploys,

genuine question - why this over ArgoCD ? which is a huge ecosystem and all the declarative setups. Plus has clean github integration for incremental deploys, etc

SOLAR_FIELDS 3y ago
Argo is kind of antithetical to what this project is trying to solve. This project wants to be a Dockerfile like approach to Kubernetes. It focuses on simplicity. ArgoCD is decidedly not that, even if longer term something like that is the more correct way to manage production deployments. I could see this project being picked up by some dev who just wants it on the cluster then an SRE ports it to something like Argo. Ideally the dev should learn Argo in the first place but a world where every dev understands what Argo is doing feels pretty unrealistic. Side note (kind of off topic but I’ve been thinking about Argo a lot lately) I’ve been a bit more bearish on Argo for non production environments because it encourages pets and not cattle. I think nowadays the correct way to do environments is a gitops production cluster and dynamic pushed based ephemeral environments that are spawned from that production configuration and templates accordingly. Actually maybe it isn’t off topic: that’s what I’d like to see to offer to developers: some tool that is easily configurable like this that spawns ephemeral environments and deployments based off a checked in production grade configuration template like you would get with Argo
- sandGorgon 3y ago
  that's a fair point, but Argocd + Packer is just that right ? >some tool that is easily configurable like this that spawns ephemeral environments and deployments based off a checked in production grade configuration template like you would get with Argo Totally with you there, but you can build this on top of Argo by having your opinionated manifest language (like Packer or Cuberfile or something else). But there is no joy to maintaining the Kubernetes api interfaces.
  - SOLAR_FIELDS 3y ago
    you could achieve this with packer and some custom tooling yeah. But just saying “and some custom tooling” is like saying “you could do what Dropbox does with a custom rsync script”. There’s no off the shelf tooling to do it. One does not simply “build this on top of Argo by having your opinionated manifest language”. There’s probably 10 tools I could customize to do this, packer being one of them, but no real patterns I’m aware of where people are actually doing it with argocd or fluxcd config as a base. Usually you’re either all in or all out on gitops, so I would be interested in seeing blog posts or libraries of people who have done hybrid approaches (push and pull deployments) and the trade offs they had to make. There is a decent amount of decisions that have to be made to build an abstraction like that, so anyone who has done it before I’m happy to stand on the shoulders of. It’s been on my list of unsolved devops/infra problems to unwrap for a couple of years now In the end there are really three steps that need to happen for true “copy prod” ephemeral environments. You need to provision some stack of infra and then deploy some stuff on top of it and then pull some arbitrary, extremely customizable state into that deployment. I’ve been quite interested in Layerform recently for the first part of that, I think they are getting close to the right abstraction on the infra side. But then you still have to deploy (Argo, helm, flux, something) into that infra stack. Then you have to get the state into that deployment somehow. there is no answer for the latter two from Layerform (terraform is not the right tool for deploys or state backups once you do anything beyond a hello world app). Getting the state in has its own whole Pandora’s box (can I copy customer data from prod into another environment? How do I do so on a way that doesn’t violate SOC2/hipaa/whatever? How will my system’s data and configurations affect the third party systems it connects to?) that I think is simply just not generalizable. But the first two steps should be able to be generalized, and the last step is something that tools can be made available to make it easier with the knowledge that it’s basically business logic at that point
    - sandGorgon 3y ago
      well my intention was not to trivialize the effort. I think i was trying to ask an architecture question: if you were to build something like Cuber...would you build it over Argocd ? or build it entirely from the ground up. It would still take equal amount of effort probably, but you get the benefit of a larger ecosystem and more work on the underlying infra, while you can focus entirely on the abstractions.
- iFire 3y ago
  Can you write about Argo workflow? I'll be interested to read a long form summary of how to raise cattle.
- antonvs 3y ago
  > it encourages pets and not cattle. How so? From the rest of your comment it sounds like you may be saying that target environments are the pets, but ArgoCD makes no assumption about that. It will deploy to any cluster you configure it to deploy to, and spinning up a cluster is a logically (very!) separate operation from deploying an app. Argo only deals with the latter. If you want something to actually provision an ephemeral environment for you, ArgoCD is not the tool. It's more focused than that, and that's its strength. All it does is deploy k8s manifests out of git to whatever cluster you tell it to deploy them to. That cluster could be ephemeral or not, it doesn't matter to Argo.
  - SOLAR_FIELDS 3y ago
    Of course you could use argo to do what I'm describing, but the overhead of checking in code is 1. clunky for dynamic environments (it would require constantly committing to some repo) and 2. not well suited for provisioning dynamically from a local machine. So by virtue of the way argo works it encourages pets, because its setup is not very dynamic. Like you said, in environments when you want to be able to roll back, that is its strength. But my point I was making above is that really you want that for only one environment: prod when it all boils down to it. Every other environment not being a dynamic branch off of prod just creates maintenance and state issues down the line
    - antonvs 3y ago
      It's not just prod, it's any shared environment that needs to live for some time. An example is an integration environment, especially if it's shared by multiple teams. A "pet" is a machine or environment that you can't easily delete and replace. Nothing we're discussing can't easily be deleted and replaced. What you're describing relates to some other property. As for "dynamic", with Argo you can just edit the namespaces of Applications in a branch and deploy a new environment. > it would require constantly committing to some repo If you have an automated CI/CD pipeline, you already do this anyway. It's not clear to me what kind of operation you're thinking of, but in the same sort of way that k8s is overkill for many scenarios, so is ArgoCD. But the issue is not that it encourages pets or can't easily handle dynamic environments.
booleanbetrayal 3y ago
Just chiming in here to say how great of a product ArgoCD is. We've been using it for a few years now and it makes for a highly configurable deployment management layer on top of our clusters. Highly recommend it for managing multiple application deployments where things like RBAC, observability, and configurability matter. Also doesn't hurt that it has a decent UX (although we primarily interface via automatons these days). The project maintainers are also extremely responsive and professional (thanks @alexmt!). All that being said, Cuber definitely feels like a different sort of market fit.
- sandGorgon 3y ago
  >All that being said, Cuber definitely feels like a different sort of market fit. but it could have been built on top of Argo right ? I totally get an opinionated out-of-the-box workflow and manifest file. But i would rather see that on top of the Argo infra (with its millions of person hours of kuberentes fighting).
ransom1538 3y ago
ArgoCD is meh. Its a UI over your k8s clusters. What happened to us is sre/devops types just use k9s or the console. The devs could not be bothered to log into argo. They just merge, and code gets deployed. They don't hang out on argo. With 5k+ pods, the UI doesn't even work. So it isn't designed for large systems which makes it further weird.
- antonvs 3y ago
  > Its a UI over your k8s clusters. Not really. It's a deployment tool, the UI is there to help see what's happening with deployments. It's not intended to be a primary interface to cluster management the way k9s or Lens etc. is. > They just merge, and code gets deployed. How does "code get deployed" though? There has to be something that's actually (1) monitoring branches and (2) running the equivalent of `kubectl apply` to deploy manifests. That's what ArgoCD does. If you're saying devs are using k9s or kubectl to manually trigger the apply from a branch, that has all sorts of drawbacks, which is exactly what Argo is designed to address.

genuine question - why this over ArgoCD ? which is a huge ecosystem and all the declarative setups. Plus has clean github integration for incremental deploys, etc

SOLAR_FIELDS 3y ago

Argo is kind of antithetical to what this project is trying to solve. This project wants to be a Dockerfile like approach to Kubernetes. It focuses on simplicity. ArgoCD is decidedly not that, even if longer term something like that is the more correct way to manage production deployments. I could see this project being picked up by some dev who just wants it on the cluster then an SRE ports it to something like Argo. Ideally the dev should learn Argo in the first place but a world where every dev understands what Argo is doing feels pretty unrealistic. Side note (kind of off topic but I’ve been thinking about Argo a lot lately) I’ve been a bit more bearish on Argo for non production environments because it encourages pets and not cattle. I think nowadays the correct way to do environments is a gitops production cluster and dynamic pushed based ephemeral environments that are spawned from that production configuration and templates accordingly. Actually maybe it isn’t off topic: that’s what I’d like to see to offer to developers: some tool that is easily configurable like this that spawns ephemeral environments and deployments based off a checked in production grade configuration template like you would get with Argo

sandGorgon 3y ago
that's a fair point, but Argocd + Packer is just that right ? >some tool that is easily configurable like this that spawns ephemeral environments and deployments based off a checked in production grade configuration template like you would get with Argo Totally with you there, but you can build this on top of Argo by having your opinionated manifest language (like Packer or Cuberfile or something else). But there is no joy to maintaining the Kubernetes api interfaces.
- SOLAR_FIELDS 3y ago
  you could achieve this with packer and some custom tooling yeah. But just saying “and some custom tooling” is like saying “you could do what Dropbox does with a custom rsync script”. There’s no off the shelf tooling to do it. One does not simply “build this on top of Argo by having your opinionated manifest language”. There’s probably 10 tools I could customize to do this, packer being one of them, but no real patterns I’m aware of where people are actually doing it with argocd or fluxcd config as a base. Usually you’re either all in or all out on gitops, so I would be interested in seeing blog posts or libraries of people who have done hybrid approaches (push and pull deployments) and the trade offs they had to make. There is a decent amount of decisions that have to be made to build an abstraction like that, so anyone who has done it before I’m happy to stand on the shoulders of. It’s been on my list of unsolved devops/infra problems to unwrap for a couple of years now In the end there are really three steps that need to happen for true “copy prod” ephemeral environments. You need to provision some stack of infra and then deploy some stuff on top of it and then pull some arbitrary, extremely customizable state into that deployment. I’ve been quite interested in Layerform recently for the first part of that, I think they are getting close to the right abstraction on the infra side. But then you still have to deploy (Argo, helm, flux, something) into that infra stack. Then you have to get the state into that deployment somehow. there is no answer for the latter two from Layerform (terraform is not the right tool for deploys or state backups once you do anything beyond a hello world app). Getting the state in has its own whole Pandora’s box (can I copy customer data from prod into another environment? How do I do so on a way that doesn’t violate SOC2/hipaa/whatever? How will my system’s data and configurations affect the third party systems it connects to?) that I think is simply just not generalizable. But the first two steps should be able to be generalized, and the last step is something that tools can be made available to make it easier with the knowledge that it’s basically business logic at that point
  - sandGorgon 3y ago
    well my intention was not to trivialize the effort. I think i was trying to ask an architecture question: if you were to build something like Cuber...would you build it over Argocd ? or build it entirely from the ground up. It would still take equal amount of effort probably, but you get the benefit of a larger ecosystem and more work on the underlying infra, while you can focus entirely on the abstractions.
iFire 3y ago
Can you write about Argo workflow? I'll be interested to read a long form summary of how to raise cattle.
antonvs 3y ago
> it encourages pets and not cattle. How so? From the rest of your comment it sounds like you may be saying that target environments are the pets, but ArgoCD makes no assumption about that. It will deploy to any cluster you configure it to deploy to, and spinning up a cluster is a logically (very!) separate operation from deploying an app. Argo only deals with the latter. If you want something to actually provision an ephemeral environment for you, ArgoCD is not the tool. It's more focused than that, and that's its strength. All it does is deploy k8s manifests out of git to whatever cluster you tell it to deploy them to. That cluster could be ephemeral or not, it doesn't matter to Argo.
- SOLAR_FIELDS 3y ago
  Of course you could use argo to do what I'm describing, but the overhead of checking in code is 1. clunky for dynamic environments (it would require constantly committing to some repo) and 2. not well suited for provisioning dynamically from a local machine. So by virtue of the way argo works it encourages pets, because its setup is not very dynamic. Like you said, in environments when you want to be able to roll back, that is its strength. But my point I was making above is that really you want that for only one environment: prod when it all boils down to it. Every other environment not being a dynamic branch off of prod just creates maintenance and state issues down the line
  - antonvs 3y ago
    It's not just prod, it's any shared environment that needs to live for some time. An example is an integration environment, especially if it's shared by multiple teams. A "pet" is a machine or environment that you can't easily delete and replace. Nothing we're discussing can't easily be deleted and replaced. What you're describing relates to some other property. As for "dynamic", with Argo you can just edit the namespaces of Applications in a branch and deploy a new environment. > it would require constantly committing to some repo If you have an automated CI/CD pipeline, you already do this anyway. It's not clear to me what kind of operation you're thinking of, but in the same sort of way that k8s is overkill for many scenarios, so is ArgoCD. But the issue is not that it encourages pets or can't easily handle dynamic environments.

booleanbetrayal 3y ago

Just chiming in here to say how great of a product ArgoCD is. We've been using it for a few years now and it makes for a highly configurable deployment management layer on top of our clusters. Highly recommend it for managing multiple application deployments where things like RBAC, observability, and configurability matter. Also doesn't hurt that it has a decent UX (although we primarily interface via automatons these days). The project maintainers are also extremely responsive and professional (thanks @alexmt!). All that being said, Cuber definitely feels like a different sort of market fit.

sandGorgon 3y ago
>All that being said, Cuber definitely feels like a different sort of market fit. but it could have been built on top of Argo right ? I totally get an opinionated out-of-the-box workflow and manifest file. But i would rather see that on top of the Argo infra (with its millions of person hours of kuberentes fighting).

ransom1538 3y ago

ArgoCD is meh. Its a UI over your k8s clusters. What happened to us is sre/devops types just use k9s or the console. The devs could not be bothered to log into argo. They just merge, and code gets deployed. They don't hang out on argo. With 5k+ pods, the UI doesn't even work. So it isn't designed for large systems which makes it further weird.

antonvs 3y ago
> Its a UI over your k8s clusters. Not really. It's a deployment tool, the UI is there to help see what's happening with deployments. It's not intended to be a primary interface to cluster management the way k9s or Lens etc. is. > They just merge, and code gets deployed. How does "code get deployed" though? There has to be something that's actually (1) monitoring branches and (2) running the equivalent of `kubectl apply` to deploy manifests. That's what ArgoCD does. If you're saying devs are using k9s or kubectl to manually trigger the apply from a branch, that has all sorts of drawbacks, which is exactly what Argo is designed to address.