4 ms·

This is what I was wondering, in the article it looks like kubernetes is just used to launch the node containers - why is the database and rabbitmq outside of k

This is what I was wondering, in the article it looks like kubernetes is just used to launch the node containers - why is the database and rabbitmq outside of kubernetes? This architecture looks like it’s been cobbled together by a junior

robertlagrant 3y ago
Kubernetes loves stateless services. Zero wrong with moving RabbitMQ or a database outside of it.
- malux85 3y ago
  Except kubernetes has a whole storage provisioning system that gives you redundancy and automatic failover, if you’re going to the trouble of running kubernetes why not just run your whole infra on it? I run https://atomictessellator.com https://atomictessellator.com solo, using kubernetes, and my database, Minio object store, application servers, quantum workers, everything is all on kubernetes, it’s self healing and much simpler to run all the infrastructure the same. Recently I had a node failure while I was sleeping and the whole system healed itself while I slept, the monitoring system didn’t even alarm me because the small blip of increased latency while the pods rebalanced wasn’t above the alert threshold so it didn’t even wake me up. What happens in the article infra when the rabbitmq or database nodes fail? The whole system goes offline, which seems very silly setup when you have kubernetes sitting right there, who’s primary function is to handle all of this.
  - robertlagrant 3y ago
    What happens when your storage detaches from your k8s cluster? Your services start 503ing, hopefully, because you didn't design your system thinking that k8s == 100% uptime.
    - malux85 3y ago
      Anybody can invent random problems ad nauseam - that doesn’t prove anything. I’m not claiming that it’s totally bullet proof, I never said that - I’m saying that if you had a kubernetes cluster anyway why not benefit from its abilities? Especially when the alternative is single node, single points of failure, which is clearly inferior. The "what if the storage detaches" argument could easily apply to the single node VMs too, in which case the outcome would be a total system failure. We are discussing the contrast between the articles architecture and running everything on K8s ... and I'm saying that running everything on K8s is clearly better
      - jrockway 3y ago
        Yeah, what happens when someone in AWS clicks "delete database" accidentally? It's the same thing that happens when K8s blows up in some weird way. You restore from your backup. (Fun fact: deleting the instance deletes the backups!)
        robertlagrant 3y ago
        Again, I was replying to: > What happens in the article infra when the rabbitmq or database nodes fail?
      - robertlagrant 3y ago
        > Anybody can invent random problems ad nauseam I agree; I was replying to this invented problem: > What happens in the article infra when the rabbitmq or database nodes fail? It makes sense if you read the reply as a reply.
  - codenesium 3y ago
    Rabbit and most databases have their own failover strategy. Putting it all on k8s is fine for a toy app but idk why anyone would deploy a real system like that.
    - malux85 3y ago
      OK, I can only speak to my personal projects and 20+ years experience at work. We run all of our stateful and stateless workloads on 10+ kubernetes clusters at work in multiple datacenters in multiple continents, and we serve 500 million users a month with it. I wrote the first BORG version of DFP backend systems at Google, where we served billions of users billions of ads a day, and we used stateful infrastructure management on some of the first container runtime systems that inspired k8s during it's development. Using rabbit and "most databases" native fallover strategy is fine for toy projects, but when you're operating at this scale, you need automated infrastructure provisioning and all of the automated tooling around it.
      - codenesium 3y ago
        That is scale I can't comprehend.
      - robertlagrant 3y ago
        There are layers to this. At the simplest level, you only have K8s people (and aren't willing to use cloud services). So you install the RabbitMQ Helm chart, hope for the best, and fix any issues that come up. Then you get a bit worried that the Postgres Helm chart, while good, doesn't do what you want. So you update to use a dedicated clustered Postgres, using some Postgres clustering tech. Finally, you're at so much scale you can throw giant wads of advertising cash at the problem, and you can use anything you like and it'll work. You just need to choose the best thing for your particular problem.
baq 3y ago
There some of us who still perform four extra steps before putting any DB in k8s and we have good reasons.
sass_muffin 3y ago
It is actually the opposite, it is currently considered a good practice to run stateful workloads outside of kubernetes and stateless workloads inside of kubernetes.
- KronisLV 3y ago
  > It is actually the opposite, it is currently considered a good practice to run stateful workloads outside of kubernetes and stateless workloads inside of kubernetes. Is that still true? I wouldn't call the parent comment charitable enough, because there definitely can be some reasons for running stateful workloads even outside of containers altogether (familiarity included), but at the same time it feels like a lot of effort has been invested into making that a non-issue. For example, how many database Operators are now available for Kubernetes: https://operatorhub.io/?category=Database&capabilityLevel=%5B%22Seamless+Upgrades%22%2C%22Full+Lifecycle%22%2C%22Deep+Insights%22%2C%22Auto+Pilot%22%5D https://operatorhub.io/?category=Database&capabilityLevel=%5... Honestly, as long as you have storage and config setup correctly, it's not like you even need an Operator, that's for more advanced setups. I've been running databases in containers (even without Kubernetes) for years, haven't had that many issues at small/medium scale.

robertlagrant 3y ago

Kubernetes loves stateless services. Zero wrong with moving RabbitMQ or a database outside of it.

malux85 3y ago
Except kubernetes has a whole storage provisioning system that gives you redundancy and automatic failover, if you’re going to the trouble of running kubernetes why not just run your whole infra on it? I run https://atomictessellator.com https://atomictessellator.com solo, using kubernetes, and my database, Minio object store, application servers, quantum workers, everything is all on kubernetes, it’s self healing and much simpler to run all the infrastructure the same. Recently I had a node failure while I was sleeping and the whole system healed itself while I slept, the monitoring system didn’t even alarm me because the small blip of increased latency while the pods rebalanced wasn’t above the alert threshold so it didn’t even wake me up. What happens in the article infra when the rabbitmq or database nodes fail? The whole system goes offline, which seems very silly setup when you have kubernetes sitting right there, who’s primary function is to handle all of this.
- robertlagrant 3y ago
  What happens when your storage detaches from your k8s cluster? Your services start 503ing, hopefully, because you didn't design your system thinking that k8s == 100% uptime.
  - malux85 3y ago
    Anybody can invent random problems ad nauseam - that doesn’t prove anything. I’m not claiming that it’s totally bullet proof, I never said that - I’m saying that if you had a kubernetes cluster anyway why not benefit from its abilities? Especially when the alternative is single node, single points of failure, which is clearly inferior. The "what if the storage detaches" argument could easily apply to the single node VMs too, in which case the outcome would be a total system failure. We are discussing the contrast between the articles architecture and running everything on K8s ... and I'm saying that running everything on K8s is clearly better
    - jrockway 3y ago
      Yeah, what happens when someone in AWS clicks "delete database" accidentally? It's the same thing that happens when K8s blows up in some weird way. You restore from your backup. (Fun fact: deleting the instance deletes the backups!)
      - robertlagrant 3y ago
        Again, I was replying to: > What happens in the article infra when the rabbitmq or database nodes fail?
    - robertlagrant 3y ago
      > Anybody can invent random problems ad nauseam I agree; I was replying to this invented problem: > What happens in the article infra when the rabbitmq or database nodes fail? It makes sense if you read the reply as a reply.
- codenesium 3y ago
  Rabbit and most databases have their own failover strategy. Putting it all on k8s is fine for a toy app but idk why anyone would deploy a real system like that.
  - malux85 3y ago
    OK, I can only speak to my personal projects and 20+ years experience at work. We run all of our stateful and stateless workloads on 10+ kubernetes clusters at work in multiple datacenters in multiple continents, and we serve 500 million users a month with it. I wrote the first BORG version of DFP backend systems at Google, where we served billions of users billions of ads a day, and we used stateful infrastructure management on some of the first container runtime systems that inspired k8s during it's development. Using rabbit and "most databases" native fallover strategy is fine for toy projects, but when you're operating at this scale, you need automated infrastructure provisioning and all of the automated tooling around it.
    - codenesium 3y ago
      That is scale I can't comprehend.
    - robertlagrant 3y ago
      There are layers to this. At the simplest level, you only have K8s people (and aren't willing to use cloud services). So you install the RabbitMQ Helm chart, hope for the best, and fix any issues that come up. Then you get a bit worried that the Postgres Helm chart, while good, doesn't do what you want. So you update to use a dedicated clustered Postgres, using some Postgres clustering tech. Finally, you're at so much scale you can throw giant wads of advertising cash at the problem, and you can use anything you like and it'll work. You just need to choose the best thing for your particular problem.

baq 3y ago

There some of us who still perform four extra steps before putting any DB in k8s and we have good reasons.

sass_muffin 3y ago

It is actually the opposite, it is currently considered a good practice to run stateful workloads outside of kubernetes and stateless workloads inside of kubernetes.

KronisLV 3y ago
> It is actually the opposite, it is currently considered a good practice to run stateful workloads outside of kubernetes and stateless workloads inside of kubernetes. Is that still true? I wouldn't call the parent comment charitable enough, because there definitely can be some reasons for running stateful workloads even outside of containers altogether (familiarity included), but at the same time it feels like a lot of effort has been invested into making that a non-issue. For example, how many database Operators are now available for Kubernetes: https://operatorhub.io/?category=Database&capabilityLevel=%5B%22Seamless+Upgrades%22%2C%22Full+Lifecycle%22%2C%22Deep+Insights%22%2C%22Auto+Pilot%22%5D https://operatorhub.io/?category=Database&capabilityLevel=%5... Honestly, as long as you have storage and config setup correctly, it's not like you even need an Operator, that's for more advanced setups. I've been running databases in containers (even without Kubernetes) for years, haven't had that many issues at small/medium scale.