11 ms·

Why we moved from AWS RDS to Postgres in Kubernetes

205 points by elitan 4y ago

xwowsersx 4y ago
I've recently been spending a fair amount of time trying to improve query performance on RDS. This includes reviewing and optimizing particularly nasty queries, tuning PG configuration (min_wal_size, random_page_cost, work_mem, etc). I am using a db.t3.xlarge with general purpose SSD (gp2) for a web server that sees moderate writes and a lot of reads. I know there's no real way to know other than through testing, but I'm not clear on which instance type best serves our needs — I think it may very well be the case that the t3 family isn't fit for our purposes. I'm also unclear on whether we ought to switch to provisioned IOPS SSD. Does anyone have any general pointers here? I know the question is pretty open-ended, but would be great if anyone has general advice from personal experience?
- notac 4y ago
  I'd recommend hopping off of t3 asap if you're searching for performance gains - performance can be extremely variable (by design). M class will even you out. General storage IOPS is governed by your provisioned storage size. You can again get much more consistent performance by using provisioned IOPS. Feel free to email me if you want to chat through things specific to your env - email is in my about:
  - xwowsersx 4y ago
    Thank you so much, will definitely take you up on the offer.
  - brazzledazzle 4y ago
    Provisioned IOPS is much more expensive though, so make sure you really need it. If you use general IOPS you can monitor your burst balance. You can always start with general and then move to provisioned when you need it too.
    - xwowsersx 4y ago
      Thanks. If I'm reading this https://ibb.co/bNGmrCB https://ibb.co/bNGmrCB correctly, it seems like we have plenty burst balance. Does this seem to indicate that provisioned IOPS is unlikely to help us here?
      - brazzledazzle 4y ago
        Yeah you're looking good though I would recommend adding a CloudWatch alert to make sure it doesn't sneak up on you. IIRC in general provisioned IOPS can help with other performance attributes like throughput so I would look at the differences documented by AWS and then take a look at all of the relevant metrics to be certain.
        xwowsersx 4y ago
        Oh interesting, thanks. So PIOPS may help even if we're not getting low on balance. Will dig into the docs some more. Thanks!
  - aeyes 4y ago
    I would advise that you try to fit the working set into memory before spending on provisioned IOPS. Reading a lot of data from network storage constantly should be avoided as much as possible, having more IOPS doesn't necessarily improve read latency.
- Nextgrid 4y ago
  It's hard to say without metrics; what does your CPU load look like? In general, unless your CPU is often maxing out, changing the CPU is unlikely to help, so you're left with either memory or IO. Unused memory on Linux will be automatically used to cache IO operations, and you can also tweak PG itself to use more memory during queries (search for "work_mem", though there are others). If your workload is read-heavy, just giving it more memory so that the majority of your dataset is always in the kernel IO cache will give you an immediate performance boost, without even having to tweak PG's config (though that might help even further). This won't transfer to writes - those still require an actual, uncached IO operation to complete (unless you want to put your data at risk, in which case there are parameters that can be used to override that). For write-heavy workloads, you will need to upgrade IO; there's no way around the "provisioned IOPS" disks.
  - xwowsersx 4y ago
    Thanks very much for the reply. CPU is not often maxing out. Here's a graph of max CPU utilization from the last week https://ibb.co/tzw5p3L https://ibb.co/tzw5p3L
    - Nextgrid 4y ago
      You've got some spikes that could signify some large or unoptimized queries, but otherwise yes, the CPU doesn't look that hot. I suggest upgrading to an instance type which gives you 32GB or more of memory. You'll get a bigger CPU along with it as well, but don't make the CPU your priority, it's not your main bottleneck at the moment.
      - xwowsersx 4y ago
        Makes sense, thank you. Sounds like M class is the way to go as other commenter suggested. Also, yes. There are many awful queries that I'm aware of and working to correct.
        CodesInChaos 4y ago
- paulryanrogers 4y ago
  General storage IOPS scales with disk size, roughly and to a point. It's often cheaper and faster to increase the instance storage than move to EBS, prioritized or not. Of course if you need to recover quickly in a disaster you'll want a hot standby or replica. Still may be cheaper than PIOPs. (Especially if you need HA anyway.)
  - xwowsersx 4y ago
    Thanks for the reply. > It's often cheaper and faster to increase the instance storage than move to EBS, prioritized or not. You're saying it may well be cheaper to increase storage in order to get more IOPS than moving to an EBS-optimized instance type? Regarding HA, not relevant for at this point (assuming I understood you correctly). We've only got a single primary and one replica, the latter being used primarily for analytics.
radimm 4y ago
Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be already stretching it). Always wonder about use cases around 3 - 5k req/sec as minimum. [edit] PS: not really ditching neither k8s pg nor AWS RDS or similar solutions. Just being curious.
- Nextgrid 4y ago
  > 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.
  - rrampage 4y ago
    It depends a lot on the backend architecture. Number of DB requests per web request can also be high due to the pathological cases in some ORMs which can result in N+1 query problems or eagerly fetching entire object hierarchies. Such problems in application code can get brushed under the carpet due to "magical" autoscaling (be it RDS or K8s). There can also be fanout to async services/job queues which will in turn run even more DB queries.
    - AccountAccount1 4y ago
      Hey, this is not a problem for us at Nhost since most of the interfacing with Postgres is through Hasura (a GraphQL SQL-to-GraphQL) it solves the n+1 issue by compiling a performant sql statement from the gql query (it's also written in haskell, you can read more here https://hasura.io/blog/architecture-of-a-high-performance-graphql-to-sql-server-58d9944b8a87/ https://hasura.io/blog/architecture-of-a-high-performance-gr...)
    - 4y ago
0xbadcafebee 4y ago
Ah, the 'ol sunk cost fallacy of infrastructure. We are already investing in supporting K8s, so let's throw the databases in there too. Couldn't possibly be that much work. Sure, a decade-old dedicated team at a billion-dollar multinational corporation has honed a solution designed to support hundreds of thousands of customers with high availability, and we could pay a little bit extra money to spin up a new database per tenant that's a little bit less flexible, ..... or we could reinvent everything they do on our own software platform and expect the same results. All it'll cost us is extra expertise, extra staff, extra time, extra money, extra planning, and extra operations. But surely it will improve our product dramatically.
- dijit 4y ago
  And when it all goes bottoms up it will be much more difficult to resolve.
  - throwawaymaths 4y ago
    Depends. A lot of postgres usage is often "things that might as well be redis", like session tokens (but the library we imported came configured for postgres) so if the postgres goes down, as long as it can be restarted it won't be the end of the world even if all the data were wiped. Probably there is also an 80/20 for most users where it's not awful if you can restore from a cold storage, say 12h, backup.
  - baq 4y ago
    Fortunately Postgres doesn’t do that often by itself. It usually needs some creative developer’s assistance.
    - dijit 4y ago
      I think you’re triggering the worst case a lot more often when it comes to running Postgres on k8s: the storage can be removed independently from the workload and the pod can be evicted much easier than it would be in traditional database hosting methods. No need for developers to do anything strange at all.
geggam 4y ago
I would love to see the monitoring on this. Network IOPs and NAT nastiness or disk IO the bigger issue ?
jmarbach 4y ago
$0.50 per extra GB seems high, especially for a storage-intensive app. Given the cost of cloud Object Storage services it doesn't seem to make much sense. Examples of alternatives for managed Postgres: * Supabase is $0.125 per GB * DigitalOcean managed Postgres is ~$0.35 per GB
- makestuff 4y ago
  SUpabase runs on AWS so they are either losing a ton of money, have some amazing deal with AWS, or the $0.50 is inaccurate.
  - kiwicopple 4y ago
    (supabase ceo) EBS pricing is here: https://aws.amazon.com/ebs/pricing/ https://aws.amazon.com/ebs/pricing/ I'd have to check with the team but I'm 80% sure we're on gp3 ($0.08/GB-month). That said, we have a very generous free tier. With AWS we have an enterprise plan + savings plan + reserved instances. Not all of these affect EBS pricing, but we end up paying a lot less than the average AWS user due to our high-usage.
- claytongulick 4y ago
  I really wish I could use DO, but unless something has changed recently, they don't support delta backups, which is a deal killer for me. For small startups, my DR/HA plan is hourly delta snapshots of the whole volume. GCP, AWS and Azure all make this possible.
MBCook 4y ago
So they switch from one giant RDS instance with all tenants per AZ to per-tenant PG in Kubernetes. So really we don’t know how much RDS was a problem compared to the the tenant distribution. For the purposes of an article like this it would be nice if the two steps were separate or they had synthetic benchmarks of the various options. But I understand why they just moved forward. They said they consulted experts, it would also be nice to discuss some of what they looked or asked about.
- raffraffraff 4y ago
  Yeah. I mean, if you're going to use AWS database service for this use case, something that automatically scales based on load makes more sense, like Aurora Serverless. But that's also expensive. Regardless of cost, plain RDS isn't the right solution here as all.
  - eptcyka 4y ago
    Yes, that's basically the whole point of the article - an assumption they may not have made all too consciously to use RDS turned into a bad decision they sought to rectify.
HL33tibCe7 4y ago
Couldn’t you just spin up an RDS instance for each project (so, single-tenant RDS instances) to avoid the noisy neighbour problem? Or is that too expensive?
- elitan 4y ago
  We could, yes. But way to expensive compared to our current setup. We're offering free projects (Postgres, GraphQL (Hasura), Auth, Storage, Serverless Functions) so we need to optimize costs internally.
techn00 4y ago
So what solution did you end up using? Crunchy operator?
- nesmanrique 4y ago
  We evaluated several operators but at the end decided it would be best to deploy our own setup for the postgres workloads instead using helm.
qubit23 4y ago
I was hoping to see a bit more of an explanation of how this was implemented.
- elitan 4y ago
  We need a follow up: *How* we're running thosands of Postgres databases in Kubernetes.
KaiserPro 4y ago
In this instance I can see the point, being able to give raw access to customer's own psql instance is a good feature. but. It sounds bloody expensive to develop and maintain a reliable psql service on k8s
qeternity 4y ago
These threads are always full of people who have always used an AWS/GCP/Azure service, or have never actually run the service themselves. Running HA Postgres is not easy...but at any sort of scale where this stuff matters, nothing is easy. It's not as if AWS has 100% uptime, nor is it super cheap/performant. There are tradeoffs for everyone's use-case but every thread is full of people at one end of the cloud / roll-your-own spectrum.
- api 4y ago
  I wonder how many people use things like CockroachDB, Yugabyte, or TiDB? They're at least in theory far easier to run in HA configurations at the cost of some additional overhead and in some cases more limited SQL functionality. They seem like a huge step up from the arcane "1980s Unix" nightmare of Postgres clustering but I don't hear about them that much. Are they not used much or are their users just happy and quiet? (These are all "NewSQL" databases.)
  - belmont_sup 4y ago
    New user of cockroach. We’ll find out! If this startup ever makes it to any meaningful user sizd
  - tluyben2 4y ago
    We are testing all our software on yugabyte now to see how well it works. The cockroach license makes it not a fit for us, so we decided to try Yuga. So far works very well for our workloads.
  - KronisLV 4y ago
    > I wonder how many people use things like CockroachDB, Yugabyte, or TiDB? TiDB is a pretty interesting project, but there are a few limitations that should be taken into account when trying to use it: https://docs.pingcap.com/tidb/stable/mysql-compatibility https://docs.pingcap.com/tidb/stable/mysql-compatibility A lot of these are tradeoffs that will affect how a database can be architected, such as having no access to foreign keys and thus needing to think about any sort of consistency and not leaving orphaned data at the application level.
neilv 4y ago
I didn't see "backups" mentioned in that, though I'm sure they have them. Depending on your needs, it's a big thing to keep in mind while weighing options. For a small startup or operation, a managed service having credible snapshots, PITR backups, failover, etc. is going to save a business a lot of ops cost, compared to DIY designing, implementing, testing, and drilling, to the same level of credibility. One recent early startup, I looked at the amount of work for me or a contractor/consultant/hire to upgrade our Postgres recovery capability (including testing and drills) with confidence. I soon decided to move from self-hosted Postgres to RDS Postgres. RDS was a significant chunk of our modest AWS bill (otherwise, almost entirely plain EC2, S3, and traffic), but easy to justify to the founders, just by mentioning the costs it saved us for business existential protection we needed.
- nunopato 4y ago
  Thanks for bringing this up. We do have backups running daily, and we will have "backups on demand" soon as well.
stunt 4y ago
What's the benefit of running Postgres in Kubernetes vs VMs (with replication obviously)?
nunopato 4y ago
(Nhost) Sorry for not answering everyone individually, but I see some confusion duo to the lack of context about what we do as a company. First things first, Nhost falls into the category of backend-as-a-service. We provision and operate infrastructure at scale, and we also provide and run the necessary services for features such as user authentication and file storage, for users creating applications and businesses. A project/backend is comprised of a Postgres Database and the aforementioned services, none of it is shared. You get your own GraphQL engine, your own auth service, etc. We also provide the means to interface with the backend through our official SDKs. Some points I see mentioned below that are worth exploring: - One RDS instance per tenant is prohibited from a cost perspective, obviously. RDS is expensive and we have a very generous free tier. - We run the infrastructure for thousands of projects/backends which we have absolutely no control over what they are used for. Users might be building a simple job board, or the next Facebook (please don't). This means we have no idea what the workloads and access patterns will look like. - RDS is mature and a great product, AWS is a billion dolar company, etc - that is all true. But is it also true that we do not control if a user's project is missing an index and the fact that RDS does not provide any means to limit CPU/memory usage per database/tenant. - We had a couple of discussions with folks at AWS and for the reasons already mentioned, there was no obvious solution to our problem. Let me reiterate this, the folks that own the service didn't have a solution to our problem given our constraints. - Yes, this is a DIY scenario, but this is part of our core business. I hope this clarifies some of the doubts. And I expect to have a more detailed and technical blog post about our experience soon. By the way, we are hiring. If you think what we're doing is interesting and you have experience operating Postgres at scale, please write me an email at nuno@nhost.io. And don't forget to star us at https://github.com/nhost/nhost https://github.com/nhost/nhost.
- cloudbee 4y ago
  And what are your cost savings from RDS perspective. I'd a similar problem where we'd to provision like 5 databases for 5 different teams. RDS is really expensive. And your solution is open source ? I would like to try.
  - nunopato 4y ago
mp3tricord 4y ago
In a production data base why are people executing long running queries on the primary. They should be using a DB replica.
e-clinton 4y ago
Congrats on the launch. Curious to see what else is in store for this week. Do I have to manually upgrade my old instances?
- elitan 4y ago
  Thank you. It's going to be a fun week! We're working on a one-click migration from RDS to a dedicated Postgres instance for older projects. Should be live in the next week or so.
ransom1538 4y ago
I operate a large fleet of mysql db instances. We cannot use Cloudsql (RDS competitor) due mainly to cost. BUT, one thing left out, was the ability to have complex topologies. EG. MasterA <- SlaveA[1..n] <- MasterB <- SlaveB[1..n]. With extremely high writes, being able to cut and shard where you want if very powerful. In this example you could write to MasterB with different data. If i need to filter a table in replication: done. We don't need to beg AWS RDS team for the option to change a db variable (I have done this). Warning: Doing this stuff at scale with massive bills is very stressful. It took about a year to get everything ironed out [snapshots, autoscaling, sharding, custom monitoring, etc].
donedealomg 4y ago
xyzzy_plugh 4y ago
If the cost of operating a postgres database is eating into your margins so much (and you can't simply adjust your prices to eat the difference) then I would suspect the wrong technology is in place. Sure, RDS is expensive, but it's also quite well done. Almost every cloud platform service is more expensive than doing it yourself. No surprise here. In the past I've deployed SQLite over Postgres for cost cutting reasons. It's not too difficult to swap out unless you're heavily bought into database features.
- movedx 4y ago
  > Almost every cloud platform service is more expensive than doing it yourself. No surprise here. In a business environment, this is actually not true unless you consider the extreme long term. A Multi-AZ MySQL RDS instance of size db.m1.large (2x vCPU, 7.5GB of RAM), a 500GB standard disk, and an on-demand pricing model with 100% monthly utilization, will cost you approx. US$7,000 per year (rounding up.) That price gets you almost everything you can imagine from that service. US$7,000 wouldn't get you my services for the time needed to setup a service that came even 30% as close in terms of reliability, feature parity and support. RDS is not expensive (in the right environment.)
  - xyzzy_plugh 4y ago
    Agreed. RDS is a no-brainer, generally. The issue here appears to be with the unit economics per tenant. My argument is if the unit economics matter so much, the technology choice is likely a poor fit.
maxyurk 4y ago
did you consider https://www.pgbouncer.org/ https://www.pgbouncer.org/ ?

205 points by elitan 4y ago

I've recently been spending a fair amount of time trying to improve query performance on RDS. This includes reviewing and optimizing particularly nasty queries, tuning PG configuration (min_wal_size, random_page_cost, work_mem, etc). I am using a db.t3.xlarge with general purpose SSD (gp2) for a web server that sees moderate writes and a lot of reads. I know there's no real way to know other than through testing, but I'm not clear on which instance type best serves our needs — I think it may very well be the case that the t3 family isn't fit for our purposes. I'm also unclear on whether we ought to switch to provisioned IOPS SSD. Does anyone have any general pointers here? I know the question is pretty open-ended, but would be great if anyone has general advice from personal experience?

notac 4y ago
I'd recommend hopping off of t3 asap if you're searching for performance gains - performance can be extremely variable (by design). M class will even you out. General storage IOPS is governed by your provisioned storage size. You can again get much more consistent performance by using provisioned IOPS. Feel free to email me if you want to chat through things specific to your env - email is in my about:
- xwowsersx 4y ago
  Thank you so much, will definitely take you up on the offer.
- brazzledazzle 4y ago
  Provisioned IOPS is much more expensive though, so make sure you really need it. If you use general IOPS you can monitor your burst balance. You can always start with general and then move to provisioned when you need it too.
  - xwowsersx 4y ago
    Thanks. If I'm reading this https://ibb.co/bNGmrCB https://ibb.co/bNGmrCB correctly, it seems like we have plenty burst balance. Does this seem to indicate that provisioned IOPS is unlikely to help us here?
    - brazzledazzle 4y ago
      Yeah you're looking good though I would recommend adding a CloudWatch alert to make sure it doesn't sneak up on you. IIRC in general provisioned IOPS can help with other performance attributes like throughput so I would look at the differences documented by AWS and then take a look at all of the relevant metrics to be certain.
      - xwowsersx 4y ago
        Oh interesting, thanks. So PIOPS may help even if we're not getting low on balance. Will dig into the docs some more. Thanks!
- aeyes 4y ago
  I would advise that you try to fit the working set into memory before spending on provisioned IOPS. Reading a lot of data from network storage constantly should be avoided as much as possible, having more IOPS doesn't necessarily improve read latency.
Nextgrid 4y ago
It's hard to say without metrics; what does your CPU load look like? In general, unless your CPU is often maxing out, changing the CPU is unlikely to help, so you're left with either memory or IO. Unused memory on Linux will be automatically used to cache IO operations, and you can also tweak PG itself to use more memory during queries (search for "work_mem", though there are others). If your workload is read-heavy, just giving it more memory so that the majority of your dataset is always in the kernel IO cache will give you an immediate performance boost, without even having to tweak PG's config (though that might help even further). This won't transfer to writes - those still require an actual, uncached IO operation to complete (unless you want to put your data at risk, in which case there are parameters that can be used to override that). For write-heavy workloads, you will need to upgrade IO; there's no way around the "provisioned IOPS" disks.
- xwowsersx 4y ago
  Thanks very much for the reply. CPU is not often maxing out. Here's a graph of max CPU utilization from the last week https://ibb.co/tzw5p3L https://ibb.co/tzw5p3L
  - Nextgrid 4y ago
    You've got some spikes that could signify some large or unoptimized queries, but otherwise yes, the CPU doesn't look that hot. I suggest upgrading to an instance type which gives you 32GB or more of memory. You'll get a bigger CPU along with it as well, but don't make the CPU your priority, it's not your main bottleneck at the moment.
    - xwowsersx 4y ago
      Makes sense, thank you. Sounds like M class is the way to go as other commenter suggested. Also, yes. There are many awful queries that I'm aware of and working to correct.
      - CodesInChaos 4y ago
paulryanrogers 4y ago
General storage IOPS scales with disk size, roughly and to a point. It's often cheaper and faster to increase the instance storage than move to EBS, prioritized or not. Of course if you need to recover quickly in a disaster you'll want a hot standby or replica. Still may be cheaper than PIOPs. (Especially if you need HA anyway.)
- xwowsersx 4y ago
  Thanks for the reply. > It's often cheaper and faster to increase the instance storage than move to EBS, prioritized or not. You're saying it may well be cheaper to increase storage in order to get more IOPS than moving to an EBS-optimized instance type? Regarding HA, not relevant for at this point (assuming I understood you correctly). We've only got a single primary and one replica, the latter being used primarily for analytics.

radimm 4y ago

Having recently heard a lot of about PostgreSQL in Kubernetes (cloudNativePG for example) it always makes me wonder about the actual load and the complexity of the cluster in the question. > This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be already stretching it). Always wonder about use cases around 3 - 5k req/sec as minimum. [edit] PS: not really ditching neither k8s pg nor AWS RDS or similar solutions. Just being curious.

Nextgrid 4y ago
> 23 req/sec (guess peak 60 - 100 might be already stretching it) That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about. It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.
- rrampage 4y ago
  It depends a lot on the backend architecture. Number of DB requests per web request can also be high due to the pathological cases in some ORMs which can result in N+1 query problems or eagerly fetching entire object hierarchies. Such problems in application code can get brushed under the carpet due to "magical" autoscaling (be it RDS or K8s). There can also be fanout to async services/job queues which will in turn run even more DB queries.
  - AccountAccount1 4y ago
    Hey, this is not a problem for us at Nhost since most of the interfacing with Postgres is through Hasura (a GraphQL SQL-to-GraphQL) it solves the n+1 issue by compiling a performant sql statement from the gql query (it's also written in haskell, you can read more here https://hasura.io/blog/architecture-of-a-high-performance-graphql-to-sql-server-58d9944b8a87/ https://hasura.io/blog/architecture-of-a-high-performance-gr...)
  - 4y ago

0xbadcafebee 4y ago

Ah, the 'ol sunk cost fallacy of infrastructure. We are already investing in supporting K8s, so let's throw the databases in there too. Couldn't possibly be that much work. Sure, a decade-old dedicated team at a billion-dollar multinational corporation has honed a solution designed to support hundreds of thousands of customers with high availability, and we could pay a little bit extra money to spin up a new database per tenant that's a little bit less flexible, ..... or we could reinvent everything they do on our own software platform and expect the same results. All it'll cost us is extra expertise, extra staff, extra time, extra money, extra planning, and extra operations. But surely it will improve our product dramatically.

dijit 4y ago
And when it all goes bottoms up it will be much more difficult to resolve.
- throwawaymaths 4y ago
  Depends. A lot of postgres usage is often "things that might as well be redis", like session tokens (but the library we imported came configured for postgres) so if the postgres goes down, as long as it can be restarted it won't be the end of the world even if all the data were wiped. Probably there is also an 80/20 for most users where it's not awful if you can restore from a cold storage, say 12h, backup.
- baq 4y ago
  Fortunately Postgres doesn’t do that often by itself. It usually needs some creative developer’s assistance.
  - dijit 4y ago
    I think you’re triggering the worst case a lot more often when it comes to running Postgres on k8s: the storage can be removed independently from the workload and the pod can be evicted much easier than it would be in traditional database hosting methods. No need for developers to do anything strange at all.

geggam 4y ago

I would love to see the monitoring on this. Network IOPs and NAT nastiness or disk IO the bigger issue ?

jmarbach 4y ago

$0.50 per extra GB seems high, especially for a storage-intensive app. Given the cost of cloud Object Storage services it doesn't seem to make much sense. Examples of alternatives for managed Postgres: * Supabase is $0.125 per GB * DigitalOcean managed Postgres is ~$0.35 per GB

makestuff 4y ago
SUpabase runs on AWS so they are either losing a ton of money, have some amazing deal with AWS, or the $0.50 is inaccurate.
- kiwicopple 4y ago
  (supabase ceo) EBS pricing is here: https://aws.amazon.com/ebs/pricing/ https://aws.amazon.com/ebs/pricing/ I'd have to check with the team but I'm 80% sure we're on gp3 ($0.08/GB-month). That said, we have a very generous free tier. With AWS we have an enterprise plan + savings plan + reserved instances. Not all of these affect EBS pricing, but we end up paying a lot less than the average AWS user due to our high-usage.
claytongulick 4y ago
I really wish I could use DO, but unless something has changed recently, they don't support delta backups, which is a deal killer for me. For small startups, my DR/HA plan is hourly delta snapshots of the whole volume. GCP, AWS and Azure all make this possible.

MBCook 4y ago

So they switch from one giant RDS instance with all tenants per AZ to per-tenant PG in Kubernetes. So really we don’t know how much RDS was a problem compared to the the tenant distribution. For the purposes of an article like this it would be nice if the two steps were separate or they had synthetic benchmarks of the various options. But I understand why they just moved forward. They said they consulted experts, it would also be nice to discuss some of what they looked or asked about.

raffraffraff 4y ago
Yeah. I mean, if you're going to use AWS database service for this use case, something that automatically scales based on load makes more sense, like Aurora Serverless. But that's also expensive. Regardless of cost, plain RDS isn't the right solution here as all.
- eptcyka 4y ago
  Yes, that's basically the whole point of the article - an assumption they may not have made all too consciously to use RDS turned into a bad decision they sought to rectify.

HL33tibCe7 4y ago

Couldn’t you just spin up an RDS instance for each project (so, single-tenant RDS instances) to avoid the noisy neighbour problem? Or is that too expensive?

elitan 4y ago
We could, yes. But way to expensive compared to our current setup. We're offering free projects (Postgres, GraphQL (Hasura), Auth, Storage, Serverless Functions) so we need to optimize costs internally.

techn00 4y ago

So what solution did you end up using? Crunchy operator?

nesmanrique 4y ago
We evaluated several operators but at the end decided it would be best to deploy our own setup for the postgres workloads instead using helm.

qubit23 4y ago

I was hoping to see a bit more of an explanation of how this was implemented.

elitan 4y ago
We need a follow up: *How* we're running thosands of Postgres databases in Kubernetes.

KaiserPro 4y ago

In this instance I can see the point, being able to give raw access to customer's own psql instance is a good feature. but. It sounds bloody expensive to develop and maintain a reliable psql service on k8s

qeternity 4y ago

These threads are always full of people who have always used an AWS/GCP/Azure service, or have never actually run the service themselves. Running HA Postgres is not easy...but at any sort of scale where this stuff matters, nothing is easy. It's not as if AWS has 100% uptime, nor is it super cheap/performant. There are tradeoffs for everyone's use-case but every thread is full of people at one end of the cloud / roll-your-own spectrum.

api 4y ago
I wonder how many people use things like CockroachDB, Yugabyte, or TiDB? They're at least in theory far easier to run in HA configurations at the cost of some additional overhead and in some cases more limited SQL functionality. They seem like a huge step up from the arcane "1980s Unix" nightmare of Postgres clustering but I don't hear about them that much. Are they not used much or are their users just happy and quiet? (These are all "NewSQL" databases.)
- belmont_sup 4y ago
  New user of cockroach. We’ll find out! If this startup ever makes it to any meaningful user sizd
- tluyben2 4y ago
  We are testing all our software on yugabyte now to see how well it works. The cockroach license makes it not a fit for us, so we decided to try Yuga. So far works very well for our workloads.
- KronisLV 4y ago
  > I wonder how many people use things like CockroachDB, Yugabyte, or TiDB? TiDB is a pretty interesting project, but there are a few limitations that should be taken into account when trying to use it: https://docs.pingcap.com/tidb/stable/mysql-compatibility https://docs.pingcap.com/tidb/stable/mysql-compatibility A lot of these are tradeoffs that will affect how a database can be architected, such as having no access to foreign keys and thus needing to think about any sort of consistency and not leaving orphaned data at the application level.

neilv 4y ago

I didn't see "backups" mentioned in that, though I'm sure they have them. Depending on your needs, it's a big thing to keep in mind while weighing options. For a small startup or operation, a managed service having credible snapshots, PITR backups, failover, etc. is going to save a business a lot of ops cost, compared to DIY designing, implementing, testing, and drilling, to the same level of credibility. One recent early startup, I looked at the amount of work for me or a contractor/consultant/hire to upgrade our Postgres recovery capability (including testing and drills) with confidence. I soon decided to move from self-hosted Postgres to RDS Postgres. RDS was a significant chunk of our modest AWS bill (otherwise, almost entirely plain EC2, S3, and traffic), but easy to justify to the founders, just by mentioning the costs it saved us for business existential protection we needed.

nunopato 4y ago
Thanks for bringing this up. We do have backups running daily, and we will have "backups on demand" soon as well.

stunt 4y ago

What's the benefit of running Postgres in Kubernetes vs VMs (with replication obviously)?

nunopato 4y ago

(Nhost) Sorry for not answering everyone individually, but I see some confusion duo to the lack of context about what we do as a company. First things first, Nhost falls into the category of backend-as-a-service. We provision and operate infrastructure at scale, and we also provide and run the necessary services for features such as user authentication and file storage, for users creating applications and businesses. A project/backend is comprised of a Postgres Database and the aforementioned services, none of it is shared. You get your own GraphQL engine, your own auth service, etc. We also provide the means to interface with the backend through our official SDKs. Some points I see mentioned below that are worth exploring: - One RDS instance per tenant is prohibited from a cost perspective, obviously. RDS is expensive and we have a very generous free tier. - We run the infrastructure for thousands of projects/backends which we have absolutely no control over what they are used for. Users might be building a simple job board, or the next Facebook (please don't). This means we have no idea what the workloads and access patterns will look like. - RDS is mature and a great product, AWS is a billion dolar company, etc - that is all true. But is it also true that we do not control if a user's project is missing an index and the fact that RDS does not provide any means to limit CPU/memory usage per database/tenant. - We had a couple of discussions with folks at AWS and for the reasons already mentioned, there was no obvious solution to our problem. Let me reiterate this, the folks that own the service didn't have a solution to our problem given our constraints. - Yes, this is a DIY scenario, but this is part of our core business. I hope this clarifies some of the doubts. And I expect to have a more detailed and technical blog post about our experience soon. By the way, we are hiring. If you think what we're doing is interesting and you have experience operating Postgres at scale, please write me an email at nuno@nhost.io. And don't forget to star us at https://github.com/nhost/nhost https://github.com/nhost/nhost.

cloudbee 4y ago
And what are your cost savings from RDS perspective. I'd a similar problem where we'd to provision like 5 databases for 5 different teams. RDS is really expensive. And your solution is open source ? I would like to try.
- nunopato 4y ago

mp3tricord 4y ago

In a production data base why are people executing long running queries on the primary. They should be using a DB replica.

e-clinton 4y ago

Congrats on the launch. Curious to see what else is in store for this week. Do I have to manually upgrade my old instances?

elitan 4y ago
Thank you. It's going to be a fun week! We're working on a one-click migration from RDS to a dedicated Postgres instance for older projects. Should be live in the next week or so.

ransom1538 4y ago

I operate a large fleet of mysql db instances. We cannot use Cloudsql (RDS competitor) due mainly to cost. BUT, one thing left out, was the ability to have complex topologies. EG. MasterA <- SlaveA[1..n] <- MasterB <- SlaveB[1..n]. With extremely high writes, being able to cut and shard where you want if very powerful. In this example you could write to MasterB with different data. If i need to filter a table in replication: done. We don't need to beg AWS RDS team for the option to change a db variable (I have done this). Warning: Doing this stuff at scale with massive bills is very stressful. It took about a year to get everything ironed out [snapshots, autoscaling, sharding, custom monitoring, etc].

donedealomg 4y ago

xyzzy_plugh 4y ago

If the cost of operating a postgres database is eating into your margins so much (and you can't simply adjust your prices to eat the difference) then I would suspect the wrong technology is in place. Sure, RDS is expensive, but it's also quite well done. Almost every cloud platform service is more expensive than doing it yourself. No surprise here. In the past I've deployed SQLite over Postgres for cost cutting reasons. It's not too difficult to swap out unless you're heavily bought into database features.

movedx 4y ago
> Almost every cloud platform service is more expensive than doing it yourself. No surprise here. In a business environment, this is actually not true unless you consider the extreme long term. A Multi-AZ MySQL RDS instance of size db.m1.large (2x vCPU, 7.5GB of RAM), a 500GB standard disk, and an on-demand pricing model with 100% monthly utilization, will cost you approx. US$7,000 per year (rounding up.) That price gets you almost everything you can imagine from that service. US$7,000 wouldn't get you my services for the time needed to setup a service that came even 30% as close in terms of reliability, feature parity and support. RDS is not expensive (in the right environment.)
- xyzzy_plugh 4y ago
  Agreed. RDS is a no-brainer, generally. The issue here appears to be with the unit economics per tenant. My argument is if the unit economics matter so much, the technology choice is likely a poor fit.

maxyurk 4y ago

did you consider https://www.pgbouncer.org/ https://www.pgbouncer.org/ ?