4 ms·

I have been building and running web applications which are used by millions of users for over 20 years now. But reading this post, I feel like I am looking int

by TekMol 3y ago

I have been building and running web applications which are used by millions of users for over 20 years now. But reading this post, I feel like I am looking into a completely different world. None of the following is something which ever crossed my path: - GitOps repository cleanup - yaml manifests that create our namespaces - ArgoCD - Helm deployments - Persistent Volume Claims - Velero - Restic - PVC block volume data - Vultr I simply write data to MariaDB (a fork of MySql) and back it up via mysqldump. I wonder if that would have been an option here too?

sofixa 3y ago
I can see the point you're trying to make, but you're choosing an interesting way of conveying it, and I disagree. Restic is a backup tool. Velero is a backup tool for Kubernetes. Vultr is a low cost but still decent "cloud" provider. GitOps is a philosophy which makes sense even on small projects. None of those are "wrong" or "overcomplicated" options. The elephant in the room is Kubernetes, which is indeed quite complex, and often gets used as the go-to even where it doesn't make sense [1] either because it's popular or because that's what people know, or because of the ready-made tools from others (e.g. if all you want to deploy exists in the form of Helm charts, it can save you lots of time) but it has its place and brings a lot to the table. You just have to be aware of the risks the complexity brings. Disclaimer: I work at HashiCorp, I'm a massive fan of Nomad and think it's a better fit than Kubernetes in many cases, but dismissing Kubernetes outright is wrong. 1 - https://atodorov.me/img/nomad/kubernetes.jpg#center https://atodorov.me/img/nomad/kubernetes.jpg#center
- politelemon 3y ago
  > You just have to be aware of the risks the complexity brings. I'd say most are not aware. It is not enough to say that people should "just" be aware. That is something said from a position of knowledge and awareness, which helps nobody. Completely agree with the parent comment, simplicity is the first thing that should be reached for. K8s ought to be dismissed by default, because then you have to justify its inclusion. That's probably a way to increase awareness before plunging in.
- tannhaeuser 3y ago
  > The elephant in the room is Kubernetes, which [...] often gets used as the go-to even where it doesn't make sense either because it's popular or because that's what people know Or because folks want to pad their resume with k8s, which is what I'm seeing at work most of the time.
- nvm0n2 3y ago
  > GitOps is a philosophy which makes sense even on small projects. Er, does it? Root cause of this catastrophic dataloss incident is that in "GitOps" none of the traditional safety checks can be implemented. In normal sysadmin workflows, attempting to delete all your data will yield an "Are you sure?!" type message and you'll probably have to take explicit steps to confirm that this is really what you intended. There will also be dry run modes and other helpers. Because git is intended for source code and not as a way to make stateful changes to servers, there are no features for that. If you push a commit that didn't do what you mean, it will just blindly do it. It seems like this is a pretty major flaw in the whole "philosophy". The whole point of hacking a VCS into a server admin UI is because people think git will let you roll back infrastructure changes easily. But it cannot, because infrastructure isn't a stateless function of your git repository.
  - sofixa 3y ago
    > In normal sysadmin workflows, attempting to delete all your data will yield an "Are you sure?!" type message and you'll probably have to take explicit steps to confirm that this is really what you intended Depends on how you go about it, a wrong pathed mv could erase your data without a warning. > Because git is intended for source code and not as a way to make stateful changes to servers, there are no features for that. If you push a commit that didn't do what you mean, it will just blindly do it. There are no features in git itself for that, but in the way I usually implement GitOps, with a CI/CD system, you can easily have a manual "check this terraform plan's output to make sure you aren't doing anything crazy, and manually click this button to approve" step.
  - Terretta 3y ago
    > Because git is intended for source code and not as a way to make stateful changes to servers, there are no features for that. If you push a commit that didn't do what you mean, it will just blindly do it. git doesn't do anything (except keep versioned source of declarative state). Instead, have a look at your state engine, and make it as safe as you care to. > because infrastructure isn't a stateless function of your git repository So we agree, there's your problem. Not git, but the state engine function. While you don't need K8s, for GitOps to work you do need a structured approach for operating your infrastructure. These concepts can help: https://operatorframework.io/about/ https://operatorframework.io/about/ “It is relatively easy to manage and scale web apps, mobile backends, and API services right out of the box. Why? Because these applications are generally stateless, so scripts can scale and recover infrastructure from failures without additional knowledge.” “A larger challenge is managing stateful applications, like databases, caches, and monitoring systems. These systems require application domain knowledge to correctly scale, upgrade, and reconfigure while protecting against data loss or unavailability. We want this application-specific operational knowledge encoded into software … to run and manage the application correctly.” Start by looking at the Operator Capability Level diagram here: https://sdk.operatorframework.io/docs/overview/ https://sdk.operatorframework.io/docs/overview/ You can iterate your ability to operate from desired state through those stages, with focus on the ones that hurt the most (by risk or by repetition).
- earthling8118 3y ago
  Nomad is honestly really good. For a long time I've wished that it had a wider reach because I just outright don't want to go back to Kubernetes after using them both. I'm really fearful that the recent events have harmed the chances of that happening though. It's a shame so I hope that isn't how it all happens.
CoolCold 3y ago
You speak is like from one's of that sysadmins breed may speak. It's a heresy nowdays not to use kubernetes/not write yaml. The only even more heresy thing could be - run web applications on baremetal servers!
tamimio 3y ago
It’s just mostly Kubernetes stuff, totally not recommended, just stick with VM or use proxmox in your deployments and containers where you need to.

by TekMol 3y ago

sofixa 3y ago

I can see the point you're trying to make, but you're choosing an interesting way of conveying it, and I disagree. Restic is a backup tool. Velero is a backup tool for Kubernetes. Vultr is a low cost but still decent "cloud" provider. GitOps is a philosophy which makes sense even on small projects. None of those are "wrong" or "overcomplicated" options. The elephant in the room is Kubernetes, which is indeed quite complex, and often gets used as the go-to even where it doesn't make sense [1] either because it's popular or because that's what people know, or because of the ready-made tools from others (e.g. if all you want to deploy exists in the form of Helm charts, it can save you lots of time) but it has its place and brings a lot to the table. You just have to be aware of the risks the complexity brings. Disclaimer: I work at HashiCorp, I'm a massive fan of Nomad and think it's a better fit than Kubernetes in many cases, but dismissing Kubernetes outright is wrong. 1 - https://atodorov.me/img/nomad/kubernetes.jpg#center https://atodorov.me/img/nomad/kubernetes.jpg#center

politelemon 3y ago
> You just have to be aware of the risks the complexity brings. I'd say most are not aware. It is not enough to say that people should "just" be aware. That is something said from a position of knowledge and awareness, which helps nobody. Completely agree with the parent comment, simplicity is the first thing that should be reached for. K8s ought to be dismissed by default, because then you have to justify its inclusion. That's probably a way to increase awareness before plunging in.
tannhaeuser 3y ago
> The elephant in the room is Kubernetes, which [...] often gets used as the go-to even where it doesn't make sense either because it's popular or because that's what people know Or because folks want to pad their resume with k8s, which is what I'm seeing at work most of the time.
nvm0n2 3y ago
> GitOps is a philosophy which makes sense even on small projects. Er, does it? Root cause of this catastrophic dataloss incident is that in "GitOps" none of the traditional safety checks can be implemented. In normal sysadmin workflows, attempting to delete all your data will yield an "Are you sure?!" type message and you'll probably have to take explicit steps to confirm that this is really what you intended. There will also be dry run modes and other helpers. Because git is intended for source code and not as a way to make stateful changes to servers, there are no features for that. If you push a commit that didn't do what you mean, it will just blindly do it. It seems like this is a pretty major flaw in the whole "philosophy". The whole point of hacking a VCS into a server admin UI is because people think git will let you roll back infrastructure changes easily. But it cannot, because infrastructure isn't a stateless function of your git repository.
- sofixa 3y ago
  > In normal sysadmin workflows, attempting to delete all your data will yield an "Are you sure?!" type message and you'll probably have to take explicit steps to confirm that this is really what you intended Depends on how you go about it, a wrong pathed mv could erase your data without a warning. > Because git is intended for source code and not as a way to make stateful changes to servers, there are no features for that. If you push a commit that didn't do what you mean, it will just blindly do it. There are no features in git itself for that, but in the way I usually implement GitOps, with a CI/CD system, you can easily have a manual "check this terraform plan's output to make sure you aren't doing anything crazy, and manually click this button to approve" step.
- Terretta 3y ago
  > Because git is intended for source code and not as a way to make stateful changes to servers, there are no features for that. If you push a commit that didn't do what you mean, it will just blindly do it. git doesn't do anything (except keep versioned source of declarative state). Instead, have a look at your state engine, and make it as safe as you care to. > because infrastructure isn't a stateless function of your git repository So we agree, there's your problem. Not git, but the state engine function. While you don't need K8s, for GitOps to work you do need a structured approach for operating your infrastructure. These concepts can help: https://operatorframework.io/about/ https://operatorframework.io/about/ “It is relatively easy to manage and scale web apps, mobile backends, and API services right out of the box. Why? Because these applications are generally stateless, so scripts can scale and recover infrastructure from failures without additional knowledge.” “A larger challenge is managing stateful applications, like databases, caches, and monitoring systems. These systems require application domain knowledge to correctly scale, upgrade, and reconfigure while protecting against data loss or unavailability. We want this application-specific operational knowledge encoded into software … to run and manage the application correctly.” Start by looking at the Operator Capability Level diagram here: https://sdk.operatorframework.io/docs/overview/ https://sdk.operatorframework.io/docs/overview/ You can iterate your ability to operate from desired state through those stages, with focus on the ones that hurt the most (by risk or by repetition).
earthling8118 3y ago
Nomad is honestly really good. For a long time I've wished that it had a wider reach because I just outright don't want to go back to Kubernetes after using them both. I'm really fearful that the recent events have harmed the chances of that happening though. It's a shame so I hope that isn't how it all happens.

CoolCold 3y ago

You speak is like from one's of that sysadmins breed may speak. It's a heresy nowdays not to use kubernetes/not write yaml. The only even more heresy thing could be - run web applications on baremetal servers!

tamimio 3y ago

It’s just mostly Kubernetes stuff, totally not recommended, just stick with VM or use proxmox in your deployments and containers where you need to.