6 ms·

I might get flak for saying this but if you aren't a postgres expert already: just use RDS or a similar cloud DB. The amount of money you're saving by hosting a

by mjr00 2mo ago

I might get flak for saying this but if you aren't a postgres expert already: just use RDS or a similar cloud DB. The amount of money you're saving by hosting and managing your own postgres instance is absolute peanuts compared to having battle-tested infrastructure for HA, backup and restores, point-in-time recovery, read replicas, etc.

vanviegen 2mo ago
And then.. you're basically trapped inside the AWS cloud (due to egress costs and db latency). No thanks!
dwedge 2mo ago
At $dayjob we have the same mentality and as a result have a load of managed read replicas that are never used for anything (not reporting, not read only queries, not backups because $cloud handles it) that cost every month. Plus managed database restricts what you can do with the database - sometimes in really annoying ways. So while I partly agree with you, a lot of companies don't really need HA, read replicas, or even PITR (though I would argue the last one is so trivial and cheap to enable that why not), but they click the expensive check box, and I would argue that companies who do need these features should consider hiring at least a couple of DBAs and get more flexibility instead of the current status quo of everyone being scared of the database and everyone just hoping cloud support will come to their rescue if ever needed
- throwaway894345 2mo ago
  > At $dayjob we have the same mentality and as a result have a load of managed read replicas that are never used for anything (not reporting, not read only queries, not backups because $cloud handles it) that cost every month. Obviously "let RDS manage your database" doesn't require egregious read replicas. The decision to use read replicas or not is completely orthogonal to whether you use RDS to manage them.
  - dwedge 2mo ago
    > Obviously "let RDS manage your database" doesn't require egregious read replicas Of course not, but an easy checkbox, a best practice AWS or terraform guide and someone doing AWS certified X associate makes it easier to happen without anyone ever really discussing it. > The decision to use read replicas or not is completely orthogonal to whether you use RDS to manage them. Assuming you're talking about letting RDS manage anything, then sure - apart from it being more likely to slip through the net if nobody has to configure them. Database is just an expensive cost nobody necessarily drills into. However if you mean the decision to let $cloud manage the replicas (and keep the primary managed), that totally depends on the cloud and the options. For example have you ever tried having a primary in GCP Cloud SQL but the replica not in cloud SQL?
    - throwaway894345 2mo ago
      > Of course not, but an easy checkbox, a best practice AWS or terraform guide and someone doing AWS certified X associate makes it easier to happen without anyone ever really discussing it. I'm defending your employer or their mindset, I'm just disagreeing with your claim that this follows from the parent's cost analysis claims. It _seems_ like your organization's problems are precisely because they _weren't_ doing the kind of cost analysis that the parent advocated. In other words, nothing in the parent's comment advocated for blindly following some Terraform guide. It feels unfair to the parent to suggest that their mindset caused your organization problems when it seems like your organization's problems were caused by _not having_ the parent's mindset.
    - ktaraszk 2mo ago
      They won't let you. It's part of their business to keep you locked in.
      - throwaway894345 2mo ago
        More likely it’s just not worth going out of their way to support niche deployments like that. I’ve been using various clouds for years, I’ve done half a dozen cloud migrations, currently working at a multicloud org—the vendors aren’t doing much to lock us in. They very much enable us to move platforms by offering things like bulk data transfer tools, standard application runtimes like Kubernetes, workload identity federation (use external identities as principals in the cloud provider’s IAM system, etc). Like I have no doubt that they’re all greedy bastards, but they aren’t doing much to lock people in. The people who complain about lock in are usually talking about “cloud providers making their services so much easier to use than bespoke platforms on bare Linux hosts such that no one will want to go back to the latter”.
- drdexebtjl 2mo ago
  > hiring at least a couple of DBAs and get more flexibility Every place I’ve ever worked at that had DBAs had the complete opposite of more flexibility. You have to do things the DBA’s way, and if their way doesn’t work for your service, you need to fight for their time and priority. Meanwhile every place I worked at where every team completely owned their databases + did periodic data recovery drills had much more flexibility and no data loss.
  - sgarland 2mo ago
    That’s usually because they’ve seen a lot of failure modes over the years, and what seems fine to you can have surprising outcomes later. My personal experience has been the opposite of yours: lots of data inconsistency issues, data loss only resolved by the database team spinning up backups, etc. And somehow, even after yet another incident, there’s never been appetite to properly fix things.
  - skullone 2mo ago
    Now you have lots of DBs all probably operating inefficiently with only periodic recovery testing. This is the stuff DBAs do every day, and you're "hasn't happened to us... yet". You'll be just another lesson someday
    - drdexebtjl 2mo ago
      Are you saying DBAs would be doing _continuous_ recovery testing? That’s not even a thing that exists. Managed databases don’t lose customer data. They would be sued and lose, and if word got out, they would lose their customers. They are optimized for the machines they run on. The proprietary tooling from AWS for managing HA clusters, doing blue-green upgrades and monitoring for slow queries is all miles ahead of the open source tooling available for self-managed databases. There’s nothing left for the DBA to do.
      - dwedge 2mo ago
        > There’s nothing left for the DBA to do. And yet here we still are
- zdc1 2mo ago
  You can use RDS without HA. Many people probably should do this, but no one wants to tell their boss they want to disable RDS HA and wear 1h of downtime a year so they can cut 50% of their spend.
  - williamdclt 2mo ago
    > no one wants to tell their boss they want to disable RDS HA and wear 1h of downtime a year so they can cut 50% of their spend. If you're a SaaS, 1h of downtime costs a lot of money. In reputation, in lost business, in support time, in on-call response, in making a public postmortem and sharing it to customers and dealing with responses... For saving a few thousand bucks, it's often indeed a very bad deal.
    - dwedge 2mo ago
      Most of those are already priced in (support costs, time dealing with a postmortem), and an hour of downtime a year isn't going to meaningfully harm the reputation of most companies, I'd wager a lot of clients wouldn't even notice. Of course this isn't true for some companies and the cost of downtime far exceeds the cost of HA. Most companies think they are that company, most aren't. And even if they are, "AWS had an outage" does a lot of lifting. Every time I get a new client they need constant point in time backups, failover, HA everything. Usually when the costs are explained they change their mind and an hour old snapshot restore is actually fine
- h2aichat 2mo ago
  That's really helpful. Thanks!
deleted 2mo ago
[deleted]
frollogaston 2mo ago
RDS is nice even if you just want basic functionality with backups.
9dev 2mo ago
It’s really not that hard, especially with an AI agent to help you. Running a Postgres server and a read replica in Hetzner with pgBackRest backing up to their S3 buckets and a cloud volume can be had for under 50€, has HA, PITR, 3-2-1 backups, and will carry you through your series A comfortably, with GDPR compliance built-in. What does the equivalent RDS setup cost you?
- OrangeDelonge 2mo ago
  RDS gets you easy integration with the rest of the AWS services, which as a startup, you’re probably using quite a few of. Also, if you are nickel and diming over expenses in the 50€ range, you’re probably at the wrong startup.
  - 9dev 2mo ago
    That was not quite the point I was making, which is that the equivalent RDS setup would come in at around ten times of that.
    - williamdclt 2mo ago
      ten times 50euros? Sounds like a very reasonable deal to not have to think about these things or spend the time setting it up.
      - 9dev 2mo ago
        If expending 6k per year on your database alone sounds reasonable to you, we have very different perspectives on a good use of investor funds. We’re not talking about maintaining your own k8s cluster here mind you, but a basic Postgres setup. If you can’t handle that comfortably in an afternoon, you probably shouldn’t be entrusted with customer data.
        mjr00 2mo ago
        > We’re not talking about maintaining your own k8s cluster here mind you, but a basic Postgres setup. If you can’t handle that comfortably in an afternoon, you probably shouldn’t be entrusted with customer data. Intelligence is being able to set up your own postgres database with backups, PITR, retention policies, HA/replicas, and everything else needed to prevent catastrophic data loss. Wisdom is realizing that's a solved problem completely unrelated to your business, so you don't have to.
manphone 2mo ago
There’s a bunch of features in compatibility, performance, and cost reasons not to choose these cloud hosted managed databases.
inigyou 2mo ago
This only works if you're already on the cloud, otherwise you're paying through the nose for bandwidth.

by mjr00 2mo ago

vanviegen 2mo ago

And then.. you're basically trapped inside the AWS cloud (due to egress costs and db latency). No thanks!

dwedge 2mo ago

At $dayjob we have the same mentality and as a result have a load of managed read replicas that are never used for anything (not reporting, not read only queries, not backups because $cloud handles it) that cost every month. Plus managed database restricts what you can do with the database - sometimes in really annoying ways. So while I partly agree with you, a lot of companies don't really need HA, read replicas, or even PITR (though I would argue the last one is so trivial and cheap to enable that why not), but they click the expensive check box, and I would argue that companies who do need these features should consider hiring at least a couple of DBAs and get more flexibility instead of the current status quo of everyone being scared of the database and everyone just hoping cloud support will come to their rescue if ever needed

throwaway894345 2mo ago
> At $dayjob we have the same mentality and as a result have a load of managed read replicas that are never used for anything (not reporting, not read only queries, not backups because $cloud handles it) that cost every month. Obviously "let RDS manage your database" doesn't require egregious read replicas. The decision to use read replicas or not is completely orthogonal to whether you use RDS to manage them.
- dwedge 2mo ago
  > Obviously "let RDS manage your database" doesn't require egregious read replicas Of course not, but an easy checkbox, a best practice AWS or terraform guide and someone doing AWS certified X associate makes it easier to happen without anyone ever really discussing it. > The decision to use read replicas or not is completely orthogonal to whether you use RDS to manage them. Assuming you're talking about letting RDS manage anything, then sure - apart from it being more likely to slip through the net if nobody has to configure them. Database is just an expensive cost nobody necessarily drills into. However if you mean the decision to let $cloud manage the replicas (and keep the primary managed), that totally depends on the cloud and the options. For example have you ever tried having a primary in GCP Cloud SQL but the replica not in cloud SQL?
  - throwaway894345 2mo ago
    > Of course not, but an easy checkbox, a best practice AWS or terraform guide and someone doing AWS certified X associate makes it easier to happen without anyone ever really discussing it. I'm defending your employer or their mindset, I'm just disagreeing with your claim that this follows from the parent's cost analysis claims. It _seems_ like your organization's problems are precisely because they _weren't_ doing the kind of cost analysis that the parent advocated. In other words, nothing in the parent's comment advocated for blindly following some Terraform guide. It feels unfair to the parent to suggest that their mindset caused your organization problems when it seems like your organization's problems were caused by _not having_ the parent's mindset.
  - ktaraszk 2mo ago
    They won't let you. It's part of their business to keep you locked in.
    - throwaway894345 2mo ago
      More likely it’s just not worth going out of their way to support niche deployments like that. I’ve been using various clouds for years, I’ve done half a dozen cloud migrations, currently working at a multicloud org—the vendors aren’t doing much to lock us in. They very much enable us to move platforms by offering things like bulk data transfer tools, standard application runtimes like Kubernetes, workload identity federation (use external identities as principals in the cloud provider’s IAM system, etc). Like I have no doubt that they’re all greedy bastards, but they aren’t doing much to lock people in. The people who complain about lock in are usually talking about “cloud providers making their services so much easier to use than bespoke platforms on bare Linux hosts such that no one will want to go back to the latter”.
drdexebtjl 2mo ago
> hiring at least a couple of DBAs and get more flexibility Every place I’ve ever worked at that had DBAs had the complete opposite of more flexibility. You have to do things the DBA’s way, and if their way doesn’t work for your service, you need to fight for their time and priority. Meanwhile every place I worked at where every team completely owned their databases + did periodic data recovery drills had much more flexibility and no data loss.
- sgarland 2mo ago
  That’s usually because they’ve seen a lot of failure modes over the years, and what seems fine to you can have surprising outcomes later. My personal experience has been the opposite of yours: lots of data inconsistency issues, data loss only resolved by the database team spinning up backups, etc. And somehow, even after yet another incident, there’s never been appetite to properly fix things.
- skullone 2mo ago
  Now you have lots of DBs all probably operating inefficiently with only periodic recovery testing. This is the stuff DBAs do every day, and you're "hasn't happened to us... yet". You'll be just another lesson someday
  - drdexebtjl 2mo ago
    Are you saying DBAs would be doing _continuous_ recovery testing? That’s not even a thing that exists. Managed databases don’t lose customer data. They would be sued and lose, and if word got out, they would lose their customers. They are optimized for the machines they run on. The proprietary tooling from AWS for managing HA clusters, doing blue-green upgrades and monitoring for slow queries is all miles ahead of the open source tooling available for self-managed databases. There’s nothing left for the DBA to do.
    - dwedge 2mo ago
      > There’s nothing left for the DBA to do. And yet here we still are
zdc1 2mo ago
You can use RDS without HA. Many people probably should do this, but no one wants to tell their boss they want to disable RDS HA and wear 1h of downtime a year so they can cut 50% of their spend.
- williamdclt 2mo ago
  > no one wants to tell their boss they want to disable RDS HA and wear 1h of downtime a year so they can cut 50% of their spend. If you're a SaaS, 1h of downtime costs a lot of money. In reputation, in lost business, in support time, in on-call response, in making a public postmortem and sharing it to customers and dealing with responses... For saving a few thousand bucks, it's often indeed a very bad deal.
  - dwedge 2mo ago
    Most of those are already priced in (support costs, time dealing with a postmortem), and an hour of downtime a year isn't going to meaningfully harm the reputation of most companies, I'd wager a lot of clients wouldn't even notice. Of course this isn't true for some companies and the cost of downtime far exceeds the cost of HA. Most companies think they are that company, most aren't. And even if they are, "AWS had an outage" does a lot of lifting. Every time I get a new client they need constant point in time backups, failover, HA everything. Usually when the costs are explained they change their mind and an hour old snapshot restore is actually fine
h2aichat 2mo ago
That's really helpful. Thanks!

deleted 2mo ago

[deleted]

frollogaston 2mo ago

RDS is nice even if you just want basic functionality with backups.

9dev 2mo ago

It’s really not that hard, especially with an AI agent to help you. Running a Postgres server and a read replica in Hetzner with pgBackRest backing up to their S3 buckets and a cloud volume can be had for under 50€, has HA, PITR, 3-2-1 backups, and will carry you through your series A comfortably, with GDPR compliance built-in. What does the equivalent RDS setup cost you?

OrangeDelonge 2mo ago
RDS gets you easy integration with the rest of the AWS services, which as a startup, you’re probably using quite a few of. Also, if you are nickel and diming over expenses in the 50€ range, you’re probably at the wrong startup.
- 9dev 2mo ago
  That was not quite the point I was making, which is that the equivalent RDS setup would come in at around ten times of that.
  - williamdclt 2mo ago
    ten times 50euros? Sounds like a very reasonable deal to not have to think about these things or spend the time setting it up.
    - 9dev 2mo ago
      If expending 6k per year on your database alone sounds reasonable to you, we have very different perspectives on a good use of investor funds. We’re not talking about maintaining your own k8s cluster here mind you, but a basic Postgres setup. If you can’t handle that comfortably in an afternoon, you probably shouldn’t be entrusted with customer data.
      - mjr00 2mo ago
        > We’re not talking about maintaining your own k8s cluster here mind you, but a basic Postgres setup. If you can’t handle that comfortably in an afternoon, you probably shouldn’t be entrusted with customer data. Intelligence is being able to set up your own postgres database with backups, PITR, retention policies, HA/replicas, and everything else needed to prevent catastrophic data loss. Wisdom is realizing that's a solved problem completely unrelated to your business, so you don't have to.

manphone 2mo ago

There’s a bunch of features in compatibility, performance, and cost reasons not to choose these cloud hosted managed databases.

inigyou 2mo ago

This only works if you're already on the cloud, otherwise you're paying through the nose for bandwidth.