3 ms·

So you don't run any databases in those thousands of clusters? To your point, and I have not used k8s I just started to research it when my former company was

So you don't run any databases in those thousands of clusters? To your point, and I have not used k8s I just started to research it when my former company was thinking about shoehorning cassandra into k8s... But there was dogma around not allowing access to VM command exec via kubectl, while I basically needed it in the basic mode for certain one-off diagnosis needs and nodetool stuff... And yes, some of the floated stuff was "use sidecars" which also seemed to architect complexity for dogma's sake.

voidfunc 2y ago
> So you don't run any databases in those thousands of clusters? We do, but not of the SQL variety (that I am aware of). We have persistent key-value and document store databases hosted in these clusters. SQL databases are off-loaded to managed offering's in the cloud. Admittedly, this does simplify a lot of problems for us.
- tayo42 2y ago
  How much data? I keep hearing k8s isn't usable becasue sometimes there is to much data and it can't be moved around.
  - pletnes 2y ago
    The simplest approach I’m aware of is to create the k8s cluster and databases in the same datacenter / availability zone.
  - darkstar_16 2y ago
    In the managed k8s space, the data is on a PVC in the same availability zone as the node it is being mounted on. If the node dies, the volume is just mounted on to a new node in the same zone. There is no data movement required.
  - eek04_ 2y ago
    While I've not played with k8, I did run stuff in Google's Borg for a very long while, and that has a similar architecture. My team was petabyte scale and we were far from the team with the largest footprint. So it is clearly possible to handle large scale data in this type of architecture.
pas 2y ago
postgresql operators are pretty nice, so it makes sense to run stateful stuff on k8s (ie. for CI, testing, staging, dev, etc.. and probably even for prod if there's a need to orchestrate shards) > exec kubectl exec is good, and it's possible to audit access (ie. get kubectl exec events with arguments logged) and I guess and admissions webhook can filter the allowed commands but IMHO it's shouldn't be necessary, the bastion host where the "kubectl exec" is run from should be accessible only through an SSH session recorder

voidfunc 2y ago

> So you don't run any databases in those thousands of clusters? We do, but not of the SQL variety (that I am aware of). We have persistent key-value and document store databases hosted in these clusters. SQL databases are off-loaded to managed offering's in the cloud. Admittedly, this does simplify a lot of problems for us.

tayo42 2y ago
How much data? I keep hearing k8s isn't usable becasue sometimes there is to much data and it can't be moved around.
- pletnes 2y ago
  The simplest approach I’m aware of is to create the k8s cluster and databases in the same datacenter / availability zone.
- darkstar_16 2y ago
  In the managed k8s space, the data is on a PVC in the same availability zone as the node it is being mounted on. If the node dies, the volume is just mounted on to a new node in the same zone. There is no data movement required.
- eek04_ 2y ago
  While I've not played with k8, I did run stuff in Google's Borg for a very long while, and that has a similar architecture. My team was petabyte scale and we were far from the team with the largest footprint. So it is clearly possible to handle large scale data in this type of architecture.

pas 2y ago

postgresql operators are pretty nice, so it makes sense to run stateful stuff on k8s (ie. for CI, testing, staging, dev, etc.. and probably even for prod if there's a need to orchestrate shards) > exec kubectl exec is good, and it's possible to audit access (ie. get kubectl exec events with arguments logged) and I guess and admissions webhook can filter the allowed commands but IMHO it's shouldn't be necessary, the bastion host where the "kubectl exec" is run from should be accessible only through an SSH session recorder