4 ms·

Ah, the 'ol sunk cost fallacy of infrastructure. We are already investing in supporting K8s, so let's throw the databases in there too. Couldn't possibly be tha

Ah, the 'ol sunk cost fallacy of infrastructure. We are already investing in supporting K8s, so let's throw the databases in there too. Couldn't possibly be that much work. Sure, a decade-old dedicated team at a billion-dollar multinational corporation has honed a solution designed to support hundreds of thousands of customers with high availability, and we could pay a little bit extra money to spin up a new database per tenant that's a little bit less flexible, ..... or we could reinvent everything they do on our own software platform and expect the same results. All it'll cost us is extra expertise, extra staff, extra time, extra money, extra planning, and extra operations. But surely it will improve our product dramatically.

dijit 4y ago
And when it all goes bottoms up it will be much more difficult to resolve.
- throwawaymaths 4y ago
  Depends. A lot of postgres usage is often "things that might as well be redis", like session tokens (but the library we imported came configured for postgres) so if the postgres goes down, as long as it can be restarted it won't be the end of the world even if all the data were wiped. Probably there is also an 80/20 for most users where it's not awful if you can restore from a cold storage, say 12h, backup.
- baq 4y ago
  Fortunately Postgres doesn’t do that often by itself. It usually needs some creative developer’s assistance.
  - dijit 4y ago
    I think you’re triggering the worst case a lot more often when it comes to running Postgres on k8s: the storage can be removed independently from the workload and the pod can be evicted much easier than it would be in traditional database hosting methods. No need for developers to do anything strange at all.
gw99 4y ago
I'm not so sure. All you have is another layer of abstraction between you and the problem that you are facing. And that level of abstraction may violate your SLAs unless you pitch $15k for the enterprise support option. And that may not even be fruitful because it relies on an uncertain network of folk at the other end who may or may not even be able to interpret and/or solve your problem. Also you are at the whim of their changes which may or may not break your shit. Source: AWS user on very very large scale stuff for about 10 years now. It's not magic or perfection. It's just someone else's pile of problems that are lurking. The only consolation is they appear to try slightly harder than the datacentres that we replaced.
xani_ 4y ago
- wbl 4y ago
  Running a statefull service in K8S is its own ball of wax
  - foobarian 4y ago
    Yes, Postgres on K8S... <shudder>
  - patrec 4y ago
    It is, but then I never understood why on earth you'd use k8s if you don't have stateful services. I mean really, what's the point?
    - mijamo 4y ago
      Because it's easy? What alternative would you suggest?
      - janee 4y ago
        Bare metal I'd think is the first choice for a large rdbms where you have skilled dedicated personnel that can manage it. If not rather use a specialist service like RDS for anything with serious uptime/throughput requirements. k8s doesn't really make sense to me unless it's for spinning up lots of instances, like for test or dev envs or like in the article where they host DBs for people.
      - patrec 4y ago
        The idea that something of the monstrous complexity of k8s is easy is pretty funny to me. I think if you have less than than 2 full time experts on k8s at hand, you're basically nuts if you use it for some non-toy project. In my experience, you can and will experience interesting failure scenarios. If you don't have state, why not just either use something serverless/fully-managed (beanstalk, lambda, cloudflare workers whatever) if you really need to scale up and down (or have very limited devops/sysadmin capacity) or deploy like 2 or 3 bare metal machines or VMs? Either sounds like a lot less work to manage and troubleshoot than some freaking k8s cluster.
suggala 4y ago
AWS RDS is 10x slower than BareMetal MySQL (both reads and writes). Slowness is mainly due to the reason that Storage is over network for RDS. Not bad to invest some extra time to get better performance. You are falling to “Appeal to antiquity” fallacy if you think something old is better.
- Nextgrid 4y ago
  It's unlikely running it on K8S (which is itself going to run on underpowered VMs with networked storage) is going to help. If you're gonna spend effort in running Postgres manually, do it on bare-metal and at least get some reward out of it (performance and reduced cost).
  - derefr 4y ago
    > It's unlikely running it on K8S (which is itself going to run on underpowered VMs with networked storage) is going to help. On GCP, at least, you can provision a GKE node-pool where the nodes have direct-attached NVMe storage; deploy a privileged container that formats and RAID0s up the drives; and then make use of the resulting scratch filesystem via host-mounts.
  - qeternity 4y ago
    > It's unlikely running it on K8S (which is itself going to run on underpowered VMs with networked storage) is going to help. What?? We run replicated Patroni on local NVMEs and it's incredibly fast.
    - Nextgrid 4y ago
      If you have a K8S cluster running on bare-metal with directly attached disks, sure, it'll work great. I'm just pointing out that K8S by itself will not give you any performance boost if the underlying K8S nodes are the same, slow VMs with terrible IO performance.
      - qeternity 4y ago

dijit 4y ago

And when it all goes bottoms up it will be much more difficult to resolve.

throwawaymaths 4y ago
Depends. A lot of postgres usage is often "things that might as well be redis", like session tokens (but the library we imported came configured for postgres) so if the postgres goes down, as long as it can be restarted it won't be the end of the world even if all the data were wiped. Probably there is also an 80/20 for most users where it's not awful if you can restore from a cold storage, say 12h, backup.
baq 4y ago
Fortunately Postgres doesn’t do that often by itself. It usually needs some creative developer’s assistance.
- dijit 4y ago
  I think you’re triggering the worst case a lot more often when it comes to running Postgres on k8s: the storage can be removed independently from the workload and the pod can be evicted much easier than it would be in traditional database hosting methods. No need for developers to do anything strange at all.

gw99 4y ago

I'm not so sure. All you have is another layer of abstraction between you and the problem that you are facing. And that level of abstraction may violate your SLAs unless you pitch $15k for the enterprise support option. And that may not even be fruitful because it relies on an uncertain network of folk at the other end who may or may not even be able to interpret and/or solve your problem. Also you are at the whim of their changes which may or may not break your shit. Source: AWS user on very very large scale stuff for about 10 years now. It's not magic or perfection. It's just someone else's pile of problems that are lurking. The only consolation is they appear to try slightly harder than the datacentres that we replaced.

xani_ 4y ago

wbl 4y ago
Running a statefull service in K8S is its own ball of wax
- foobarian 4y ago
  Yes, Postgres on K8S... <shudder>
- patrec 4y ago
  It is, but then I never understood why on earth you'd use k8s if you don't have stateful services. I mean really, what's the point?
  - mijamo 4y ago
    Because it's easy? What alternative would you suggest?
    - janee 4y ago
      Bare metal I'd think is the first choice for a large rdbms where you have skilled dedicated personnel that can manage it. If not rather use a specialist service like RDS for anything with serious uptime/throughput requirements. k8s doesn't really make sense to me unless it's for spinning up lots of instances, like for test or dev envs or like in the article where they host DBs for people.
    - patrec 4y ago
      The idea that something of the monstrous complexity of k8s is easy is pretty funny to me. I think if you have less than than 2 full time experts on k8s at hand, you're basically nuts if you use it for some non-toy project. In my experience, you can and will experience interesting failure scenarios. If you don't have state, why not just either use something serverless/fully-managed (beanstalk, lambda, cloudflare workers whatever) if you really need to scale up and down (or have very limited devops/sysadmin capacity) or deploy like 2 or 3 bare metal machines or VMs? Either sounds like a lot less work to manage and troubleshoot than some freaking k8s cluster.

suggala 4y ago

AWS RDS is 10x slower than BareMetal MySQL (both reads and writes). Slowness is mainly due to the reason that Storage is over network for RDS. Not bad to invest some extra time to get better performance. You are falling to “Appeal to antiquity” fallacy if you think something old is better.

Nextgrid 4y ago
It's unlikely running it on K8S (which is itself going to run on underpowered VMs with networked storage) is going to help. If you're gonna spend effort in running Postgres manually, do it on bare-metal and at least get some reward out of it (performance and reduced cost).
- derefr 4y ago
  > It's unlikely running it on K8S (which is itself going to run on underpowered VMs with networked storage) is going to help. On GCP, at least, you can provision a GKE node-pool where the nodes have direct-attached NVMe storage; deploy a privileged container that formats and RAID0s up the drives; and then make use of the resulting scratch filesystem via host-mounts.
- qeternity 4y ago
  > It's unlikely running it on K8S (which is itself going to run on underpowered VMs with networked storage) is going to help. What?? We run replicated Patroni on local NVMEs and it's incredibly fast.
  - Nextgrid 4y ago
    If you have a K8S cluster running on bare-metal with directly attached disks, sure, it'll work great. I'm just pointing out that K8S by itself will not give you any performance boost if the underlying K8S nodes are the same, slow VMs with terrible IO performance.
    - qeternity 4y ago