5 ms·

OpenGitOps

50 points by locknitpicker 10mo ago

coredog64 10mo ago
Serious question: How do organizations deal with having git on the critical path for deployment? Current employer actually prohibits this due to frequent outages in the git plant.
- numbsafari 10mo ago
  Ummm … github is not git … if you must, keep your git stored locally and simply use webhooks to keep it synced whenever changes are merged via your forge of choice… you can, if necessary, make updates to your locally hosted repo in the event of an outage at the forge, but you’ll need a procedure to “sync back” any changes made during the outage. Fortunately, the whole thing is git based, so you have all the tools you need to do it.
- grayhatter 10mo ago
  > Current employer actually prohibits this due to frequent outages in the git plant. Even without knowing what you mean by git plant. I can tell you you're holding it wrong. I'd like to help, but I can't offer useful suggestions if I have no idea what you're even doing, or prohibiting.
- calgoo 10mo ago
  gitlab on internal network works quite well. You can still use K8 / EKS / AKS for runners etc, just run it all on the internal network. Im always surprised at how many orgs use public platforms for their code and ci/cd.
- master_crab 10mo ago
  Git is just the repo store. Everything you are referring to is the CI/CD pipeline. GitHub actions, Gitlab Runners, ArgoCD; they can all do some sort of gitops. Those dependencies existed before gitops anyway, so nothing new is being added.
- locknitpicker 10mo ago
  > Serious question: How do organizations deal with having git on the critical path for deployment? You mean like storing source code? CICD pipeline definitions? Container image specs? IaC manifests? Service configuration files? I don't think that arbitrarily drawing the line on what package version is being deployed is a rational opinion to have. I mean, look at npm's package.json. Is it ok to specify which dependency versions you bundle with your software, but specifying which version you're deploying is unheard of? The only drawback I see in GitOps is that CICD pipelines are still lagging way behind on providing adequate visualization strategies to monitor the current state of multi-environment deployments.
- cortesoft 10mo ago
  We use flux for our gitops, and it is easy to bypass if we need to in an emergency. You can disable reconciliation and then manually apply a change if your git repo is unavailable.
- antonvs 10mo ago
  > frequent outages in the git plant. What are you talking about. And whatever it is, it’s wrong.
gpi 10mo ago
ArgoCD is the defacto gitops standard now and has the lions share of gitops deployments.
- dijit 10mo ago
  Last I saw, ArgoCD was heavily reliant on Kubernetes. Has this changed? (also, it seems like the site is heavily supporting ArgoCD)
  - formerly_proven 10mo ago
    The Argo project exclusively targets Kubernetes.
  - gpi 10mo ago
    The whole premise of opengitops is heavily reliant on kubernetes.
    - locknitpicker 10mo ago
      > The whole premise of opengitops is heavily reliant on kubernetes. There's indeed a fair degree of short-sightedness in some GitOps proponents, who conflate their own personal implementation with the one true GitOps. Back in the real world, the bulk of cloud infrastructure covers resources that go well beyond applying changes to pre-baked Kubernetes cluster. Any service running on the likes of AWS/Google Cloud/Azure/etc require configuring plenty of cloud resources with whatever IaC platform they use, and Kubernetes operators neither cover those nor are a reasonable approach to the problem domain.
      - acedTrex 10mo ago
        > and Kubernetes operators neither cover those nor are a reasonable approach to the problem domain. I mean Crossplane is a pretty popular k8s operator that does exactly that, create cloud infrastructure from K8s objects.
jpillora 10mo ago
The problem with git ops only manifests after it’s become a standard within an org: With 1 git ops pipeline, it’s fine, it’s the human merge gate, it’s doing its job protecting downstream With multiple git ops pipelines however, they start to get in the way of progress - especially when they need to be joined in series The better approach is to build API-first then optionally, add an API client into your git pipeline
- dijit 10mo ago
  > With multiple git ops pipelines however, they start to get in the way of progress - especially when they need to be joined in series Definitely, that's why systems like Zuul exist. They're esoteric and require a lot of engineering discipline in patience- but in my experience most people who reach for gitops aren't doing it for a sense of "everything as code" (for the audibility and theoretical reproducibility of it) it's because they think it will allow them to go faster; and a tool like Zuul is hard to learn and will intentionally slow you down. Because slow is smooth, and smooth is fast.
  - mroche 10mo ago
    Would you mind elaborating on this more, describing the differences and how tools like Zuul introduce degrees of friction that result in smooth operation and pipelines? I know my phrasing may come off wrong, I apologize for that. But I'm asking genuinely; I've only ever seen Zuul in the wild in the Red Hat and OpenStack ecosystems.
    - dijit 10mo ago
      Right, so Zuul is properly interesting if you're dealing with multi-repo setups and want to test changes across them before they merge; that's the key bit that something like GitLab CI doesn't really do. The main thing with Zuul is speculative execution. Say you've got a queue of patches waiting to merge across different repos. Zuul will optimistically test each patch as if all the patches ahead of it in the queue have already merged. So if you've got patches A, B, and C queued up, Zuul tests: * A on its own * B with A already applied * C with both A and B applied If something fails, Zuul rewinds and retests without the failing patch. This means you're not waiting for A to fully merge before you can even start testing B - massive time saver when you've got lots of changes flowing through. With GitLab CI, you're basically testing each MR in isolation against the current state of the target branch. If you've got interdependent changes across repos, you end up with this annoying pattern: * Merge change in repo A * Wait for it to land * Now test change in repo B that depends on it * Merge that * Now test change in repo C... It's serial and slow, and you find out about problems late. If change C reveals an issue with change A, you've already merged A ages ago. Zuul also has this concept of cross-repo dependencies built in. You can explicitly say "this patch in repo A depends on that patch in repo B" and Zuul will test them together. GitLab CI can sort of hack this together with trigger pipelines and artifacts, but it's not the same thing... you're still not getting that speculative testing across the dependency tree. The trade-off is that Zuul is significantly more complex to set up and run. It's designed for the OpenStack-style workflow where you've got dozens of repos and hundreds of patches in flight. For a single repo or even a handful of loosely-coupled repos, GitLab CI (and it's ilk) is probably fine and much simpler. But once you hit that multi-repo, high-velocity scenario, Zuul starts to make proper sense. Yet nobodies using it except hardcore foundational infrastructure providers.
gitttnwmrlg 10mo ago
What’s the (practical) difference between section 3 and 4? Please explain.
- bo0tzz 10mo ago
  The system should undo state drift even if a run hasn't been prompted by changes to the upstream definitions in the repository
Carrett 10mo ago
Does it really make sense to use Kubernetes in 2026? Especially in the cloud? I think it’s just adding unnecessary layers, increasing operational debt, and complicating the developer experience.
- 0xbadcafebee 10mo ago
  It's not a black and white question. For you it doesn't make sense, for others it makes tons of sense.
- locknitpicker 10mo ago
  > Does it really make sense to use Kubernetes in 2026? Especially in the cloud? I can't tell if your comment is a joke or not.
  - pfix 10mo ago
    That in itself is an answer :D
  - antonvs 10mo ago
    It’s a joke whether or not it’s intended as one.
  - Carrett 10mo ago
    No, I’m not joking. To me, it seems pointless to keep using this tool, as it only serves to complicate the architecture. It might make sense for on-premise data centers, but in the cloud—despite the vendor lock-in, it’s still more cost-effective than adding layer after layer of extra complexity.
- nwmcsween 10mo ago
  The whole business model of cloud providers is to charge a premium for their ecosystem and to create lock-in by making everything interdependent. A Kubernetes deployment could cost 100k/yr a similar cloud deployment would be ~1m/yr.
cedws 10mo ago
I’m less positive about GitOps. GitOps is a lie. I’ve never seen software that actually manages to adhere to the ‘repo is the state’ principle. Inevitably you push something, it doesn’t work, now you have to do something out of band or revert to get it working again. Sometimes you revert and it’s still not fixed… Looking at you Argo CD.
- gpi 10mo ago
  You may have to use Kargo as well, also by the makers of Argo
- SOLAR_FIELDS 10mo ago
  This happens a ton when bootstrapping, but in my experience doesn’t really happen that much when modifying existing infrastructure which is typically when you care about it being purely GitOps a bit more. Argo has all sorts of flaws around bootstrapping things that isn’t necessarily Argo’s fault but they manifest themselves (har har) in weird ways. For instance I do a lot of crossplane these days and the fact that Argo doesn’t sync CRD’s and force them to exist before future sync waves is a huge pain point, especially when you start to get to more complex applications that have lots of resources. You find yourself having to manually sync CRD’s because the default ordering logic doesn’t wait for the CRD to exist before creating the downstream resource - even if you’ve specified sync waves!
- zzyzxd 10mo ago
  A gitops repo can never be the reflection of the system's actual state. It's a desired state your humans want the system to reach eventually, sometimes defined very loosely. This is the idea since Weaveworks invented the term years ago. Unfortunately I admit it's not very intuitive, especially to engineers who are not super familiar with declarative systems.
zaps 10mo ago
brb, registering openjjops.org for when Jujutsu overtakes Git as the default scm
almaight 10mo ago
Both FluxCD and ArgoCD, which use CRDs in Gitops, have a serious flaw: will these tools fail when your Kubernetes needs an update? I've encountered incompatibility issues even with simple Helm (which failed due to changes in the HPA API), let alone OPSs that forcibly depend on CRDs. GitLab CI + Pulumi/Kusion is the most stable solution.
- jzebedee 10mo ago
  Interesting. I've used Pulumi but this is the first I've heard of Kusion. From a quick look, it still requires all of the resource specification to be present in the AppConfiguration, and it's written in their own DSL called KCL. Is there more to the use case that I'm missing? It seems like if I'm already specifying the details of the entire workload, I'd either use Terraform, where I probably already know the DSL, or Pulumi, where I could skip the DSLs entirely.
- SOLAR_FIELDS 10mo ago
  Ultimately probably not, under the hood all Argo is doing in its vanilla configuration is pulling stuff from a git repo, calculating a diff, performing some templating to spit out the manifest (helm or kustomize) and then kubectl apply the manifest (or the api equivalent of that). So unless the API’s it uses to calculate deltas or apply manifests change it’s going to be mostly resilient. The underlying things it might apply though, are often much more sensitive to that kind of api version diff

50 points by locknitpicker 10mo ago

Serious question: How do organizations deal with having git on the critical path for deployment? Current employer actually prohibits this due to frequent outages in the git plant.

numbsafari 10mo ago
Ummm … github is not git … if you must, keep your git stored locally and simply use webhooks to keep it synced whenever changes are merged via your forge of choice… you can, if necessary, make updates to your locally hosted repo in the event of an outage at the forge, but you’ll need a procedure to “sync back” any changes made during the outage. Fortunately, the whole thing is git based, so you have all the tools you need to do it.
grayhatter 10mo ago
> Current employer actually prohibits this due to frequent outages in the git plant. Even without knowing what you mean by git plant. I can tell you you're holding it wrong. I'd like to help, but I can't offer useful suggestions if I have no idea what you're even doing, or prohibiting.
calgoo 10mo ago
gitlab on internal network works quite well. You can still use K8 / EKS / AKS for runners etc, just run it all on the internal network. Im always surprised at how many orgs use public platforms for their code and ci/cd.
master_crab 10mo ago
Git is just the repo store. Everything you are referring to is the CI/CD pipeline. GitHub actions, Gitlab Runners, ArgoCD; they can all do some sort of gitops. Those dependencies existed before gitops anyway, so nothing new is being added.
locknitpicker 10mo ago
> Serious question: How do organizations deal with having git on the critical path for deployment? You mean like storing source code? CICD pipeline definitions? Container image specs? IaC manifests? Service configuration files? I don't think that arbitrarily drawing the line on what package version is being deployed is a rational opinion to have. I mean, look at npm's package.json. Is it ok to specify which dependency versions you bundle with your software, but specifying which version you're deploying is unheard of? The only drawback I see in GitOps is that CICD pipelines are still lagging way behind on providing adequate visualization strategies to monitor the current state of multi-environment deployments.
cortesoft 10mo ago
We use flux for our gitops, and it is easy to bypass if we need to in an emergency. You can disable reconciliation and then manually apply a change if your git repo is unavailable.
antonvs 10mo ago
> frequent outages in the git plant. What are you talking about. And whatever it is, it’s wrong.

gpi 10mo ago

ArgoCD is the defacto gitops standard now and has the lions share of gitops deployments.

dijit 10mo ago
Last I saw, ArgoCD was heavily reliant on Kubernetes. Has this changed? (also, it seems like the site is heavily supporting ArgoCD)
- formerly_proven 10mo ago
  The Argo project exclusively targets Kubernetes.
- gpi 10mo ago
  The whole premise of opengitops is heavily reliant on kubernetes.
  - locknitpicker 10mo ago
    > The whole premise of opengitops is heavily reliant on kubernetes. There's indeed a fair degree of short-sightedness in some GitOps proponents, who conflate their own personal implementation with the one true GitOps. Back in the real world, the bulk of cloud infrastructure covers resources that go well beyond applying changes to pre-baked Kubernetes cluster. Any service running on the likes of AWS/Google Cloud/Azure/etc require configuring plenty of cloud resources with whatever IaC platform they use, and Kubernetes operators neither cover those nor are a reasonable approach to the problem domain.
    - acedTrex 10mo ago
      > and Kubernetes operators neither cover those nor are a reasonable approach to the problem domain. I mean Crossplane is a pretty popular k8s operator that does exactly that, create cloud infrastructure from K8s objects.

jpillora 10mo ago

The problem with git ops only manifests after it’s become a standard within an org: With 1 git ops pipeline, it’s fine, it’s the human merge gate, it’s doing its job protecting downstream With multiple git ops pipelines however, they start to get in the way of progress - especially when they need to be joined in series The better approach is to build API-first then optionally, add an API client into your git pipeline

dijit 10mo ago
> With multiple git ops pipelines however, they start to get in the way of progress - especially when they need to be joined in series Definitely, that's why systems like Zuul exist. They're esoteric and require a lot of engineering discipline in patience- but in my experience most people who reach for gitops aren't doing it for a sense of "everything as code" (for the audibility and theoretical reproducibility of it) it's because they think it will allow them to go faster; and a tool like Zuul is hard to learn and will intentionally slow you down. Because slow is smooth, and smooth is fast.
- mroche 10mo ago
  Would you mind elaborating on this more, describing the differences and how tools like Zuul introduce degrees of friction that result in smooth operation and pipelines? I know my phrasing may come off wrong, I apologize for that. But I'm asking genuinely; I've only ever seen Zuul in the wild in the Red Hat and OpenStack ecosystems.
  - dijit 10mo ago
    Right, so Zuul is properly interesting if you're dealing with multi-repo setups and want to test changes across them before they merge; that's the key bit that something like GitLab CI doesn't really do. The main thing with Zuul is speculative execution. Say you've got a queue of patches waiting to merge across different repos. Zuul will optimistically test each patch as if all the patches ahead of it in the queue have already merged. So if you've got patches A, B, and C queued up, Zuul tests: * A on its own * B with A already applied * C with both A and B applied If something fails, Zuul rewinds and retests without the failing patch. This means you're not waiting for A to fully merge before you can even start testing B - massive time saver when you've got lots of changes flowing through. With GitLab CI, you're basically testing each MR in isolation against the current state of the target branch. If you've got interdependent changes across repos, you end up with this annoying pattern: * Merge change in repo A * Wait for it to land * Now test change in repo B that depends on it * Merge that * Now test change in repo C... It's serial and slow, and you find out about problems late. If change C reveals an issue with change A, you've already merged A ages ago. Zuul also has this concept of cross-repo dependencies built in. You can explicitly say "this patch in repo A depends on that patch in repo B" and Zuul will test them together. GitLab CI can sort of hack this together with trigger pipelines and artifacts, but it's not the same thing... you're still not getting that speculative testing across the dependency tree. The trade-off is that Zuul is significantly more complex to set up and run. It's designed for the OpenStack-style workflow where you've got dozens of repos and hundreds of patches in flight. For a single repo or even a handful of loosely-coupled repos, GitLab CI (and it's ilk) is probably fine and much simpler. But once you hit that multi-repo, high-velocity scenario, Zuul starts to make proper sense. Yet nobodies using it except hardcore foundational infrastructure providers.

gitttnwmrlg 10mo ago

What’s the (practical) difference between section 3 and 4? Please explain.

bo0tzz 10mo ago
The system should undo state drift even if a run hasn't been prompted by changes to the upstream definitions in the repository

Carrett 10mo ago

Does it really make sense to use Kubernetes in 2026? Especially in the cloud? I think it’s just adding unnecessary layers, increasing operational debt, and complicating the developer experience.

0xbadcafebee 10mo ago
It's not a black and white question. For you it doesn't make sense, for others it makes tons of sense.
locknitpicker 10mo ago
> Does it really make sense to use Kubernetes in 2026? Especially in the cloud? I can't tell if your comment is a joke or not.
- pfix 10mo ago
  That in itself is an answer :D
- antonvs 10mo ago
  It’s a joke whether or not it’s intended as one.
- Carrett 10mo ago
  No, I’m not joking. To me, it seems pointless to keep using this tool, as it only serves to complicate the architecture. It might make sense for on-premise data centers, but in the cloud—despite the vendor lock-in, it’s still more cost-effective than adding layer after layer of extra complexity.
nwmcsween 10mo ago
The whole business model of cloud providers is to charge a premium for their ecosystem and to create lock-in by making everything interdependent. A Kubernetes deployment could cost 100k/yr a similar cloud deployment would be ~1m/yr.

cedws 10mo ago

I’m less positive about GitOps. GitOps is a lie. I’ve never seen software that actually manages to adhere to the ‘repo is the state’ principle. Inevitably you push something, it doesn’t work, now you have to do something out of band or revert to get it working again. Sometimes you revert and it’s still not fixed… Looking at you Argo CD.

gpi 10mo ago
You may have to use Kargo as well, also by the makers of Argo
SOLAR_FIELDS 10mo ago
This happens a ton when bootstrapping, but in my experience doesn’t really happen that much when modifying existing infrastructure which is typically when you care about it being purely GitOps a bit more. Argo has all sorts of flaws around bootstrapping things that isn’t necessarily Argo’s fault but they manifest themselves (har har) in weird ways. For instance I do a lot of crossplane these days and the fact that Argo doesn’t sync CRD’s and force them to exist before future sync waves is a huge pain point, especially when you start to get to more complex applications that have lots of resources. You find yourself having to manually sync CRD’s because the default ordering logic doesn’t wait for the CRD to exist before creating the downstream resource - even if you’ve specified sync waves!
zzyzxd 10mo ago
A gitops repo can never be the reflection of the system's actual state. It's a desired state your humans want the system to reach eventually, sometimes defined very loosely. This is the idea since Weaveworks invented the term years ago. Unfortunately I admit it's not very intuitive, especially to engineers who are not super familiar with declarative systems.

zaps 10mo ago

brb, registering openjjops.org for when Jujutsu overtakes Git as the default scm

almaight 10mo ago

Both FluxCD and ArgoCD, which use CRDs in Gitops, have a serious flaw: will these tools fail when your Kubernetes needs an update? I've encountered incompatibility issues even with simple Helm (which failed due to changes in the HPA API), let alone OPSs that forcibly depend on CRDs. GitLab CI + Pulumi/Kusion is the most stable solution.

jzebedee 10mo ago
Interesting. I've used Pulumi but this is the first I've heard of Kusion. From a quick look, it still requires all of the resource specification to be present in the AppConfiguration, and it's written in their own DSL called KCL. Is there more to the use case that I'm missing? It seems like if I'm already specifying the details of the entire workload, I'd either use Terraform, where I probably already know the DSL, or Pulumi, where I could skip the DSLs entirely.
SOLAR_FIELDS 10mo ago
Ultimately probably not, under the hood all Argo is doing in its vanilla configuration is pulling stuff from a git repo, calculating a diff, performing some templating to spit out the manifest (helm or kustomize) and then kubectl apply the manifest (or the api equivalent of that). So unless the API’s it uses to calculate deltas or apply manifests change it’s going to be mostly resilient. The underlying things it might apply though, are often much more sensitive to that kind of api version diff