3 ms·

This kind of resilience is a form of art, and it's also kind of a full-time job. I would not advise trying this for a "side project at work". Generally we all

This kind of resilience is a form of art, and it's also kind of a full-time job. I would not advise trying this for a "side project at work". Generally we all agree that we move to the cloud and it's "fully managed". If it goes down - that's the price we pay. If you have to ask the question "what if the RDS goes down", then you are really in a different universe. That last guarantee of uptime requires a ton of work, testing, and money, because you are all the way up and to the right on the curve of diminishing returns.

candiddevmike 3y ago
> That last guarantee of uptime requires a ton of work, testing, and money, because you are all the way up and to the right on the curve of diminishing returns. Not when it's a core part of your business...?
williamdclt 3y ago
> If you have to ask the question "what if the RDS goes down", then you are really in a different universe. It does go down though, don’t neglect the possibility because it likely will happen. With very average workloads, I’ve seen RDS databases restart unexpectedly, read replicas being completely out of service, and even databases being completely frozen (can’t even connect as root). I’d still go with managed, but it certainly doesn’t give full reliability :) you still have to consider “what if it goes down” - it will!
- debarshri 3y ago
  Problem is that RDS comes at a price. It is purely about operation cost. When you have 1500+ databases these cost add up. At that point, this kind of techniques are required to self host the databases. Price per DB with HPA na VPA is way lower than what you would pay for managed databases as well as you can hire a full time devops+dbadmin and still be cheaper.
  - SlightlyLeftPad 3y ago
    This is a perfect example of a cost savings opportunity that offers no benefit and is really just a foot gun when it comes to operational complexity. The cloud cost savings almost never justify the operational costs here. There’s typically a very long list of other cost optimizations to be made before this would ever be on the table for me. If you’re talking about 1500+ databases and think a single full time devops+dbadmin in some kind of unicorn person is going to be a better option, I feel bad for both of you.
    - p_l 3y ago
      Except the typical cloud prices are dialed for USA, and it's not that unusual in my experience for the cloud savings to be high enough to add more than one highly skilled engineer who will give large positive impact all around.
    - debarshri 3y ago
      I am not implying hiring single full time dbadmin, you can hire a team of them and still be cheaper that what AWS would cost at that scale.
figassis 3y ago
My company also runs on a k8s cluster, but I agree with this fully and I keep my DB outside the cluster. For me, the order of preference is: RDS, Cloud Instance, Bare metal DB server, something else that is not kubernetes. RDS will fail, but it likely will come back up without much action on my part. K8s will fail, you will spend untold human hours figuring out the k8s failure modes, before figuring out the database failure modes (which likely are quite straight forward). It's just a cost that is not worth it.

candiddevmike 3y ago

> That last guarantee of uptime requires a ton of work, testing, and money, because you are all the way up and to the right on the curve of diminishing returns. Not when it's a core part of your business...?

williamdclt 3y ago

> If you have to ask the question "what if the RDS goes down", then you are really in a different universe. It does go down though, don’t neglect the possibility because it likely will happen. With very average workloads, I’ve seen RDS databases restart unexpectedly, read replicas being completely out of service, and even databases being completely frozen (can’t even connect as root). I’d still go with managed, but it certainly doesn’t give full reliability :) you still have to consider “what if it goes down” - it will!

debarshri 3y ago
Problem is that RDS comes at a price. It is purely about operation cost. When you have 1500+ databases these cost add up. At that point, this kind of techniques are required to self host the databases. Price per DB with HPA na VPA is way lower than what you would pay for managed databases as well as you can hire a full time devops+dbadmin and still be cheaper.
- SlightlyLeftPad 3y ago
  This is a perfect example of a cost savings opportunity that offers no benefit and is really just a foot gun when it comes to operational complexity. The cloud cost savings almost never justify the operational costs here. There’s typically a very long list of other cost optimizations to be made before this would ever be on the table for me. If you’re talking about 1500+ databases and think a single full time devops+dbadmin in some kind of unicorn person is going to be a better option, I feel bad for both of you.
  - p_l 3y ago
    Except the typical cloud prices are dialed for USA, and it's not that unusual in my experience for the cloud savings to be high enough to add more than one highly skilled engineer who will give large positive impact all around.
  - debarshri 3y ago
    I am not implying hiring single full time dbadmin, you can hire a team of them and still be cheaper that what AWS would cost at that scale.

figassis 3y ago

My company also runs on a k8s cluster, but I agree with this fully and I keep my DB outside the cluster. For me, the order of preference is: RDS, Cloud Instance, Bare metal DB server, something else that is not kubernetes. RDS will fail, but it likely will come back up without much action on my part. K8s will fail, you will spend untold human hours figuring out the k8s failure modes, before figuring out the database failure modes (which likely are quite straight forward). It's just a cost that is not worth it.