4 ms·

The author has gained a significant amount of Postgres-related knowledge when performing this migration. Taking this into consideration, is it still worth usin

The author has gained a significant amount of Postgres-related knowledge when performing this migration. Taking this into consideration, is it still worth using the managed Aurora, instead of the EC2 self-managed instance?

sciguymcq 6y ago
I currently manage, or am "lead dev", for around a half dozen to 10 apps using Postgres as the database. For two of those I use a managed database service (AWS RDS / AWS Aurora Postgres) because they drive mission critical high value products / services and I wouldn't consider using a standalone install of Postgres. For other apps that are not so mission critical I am perfectly happy saving considerable money and running them on a VPS on AWS EC2 or Digital Ocean or Linode. So ultimately it depends on the app and the risk tolerance for Recovery Time Objective and Recovery Point Objective.
- fastball 6y ago
  Worth noting that DO also has a hosted Postgres service.
- rohansahai 6y ago
  That's super interesting! Have heard people do the opposite for performance reasons (run on bare EC2) and storage cost reason. Do you notice a big performance difference between the EC2 apps and RDS apps? Also have you had any EBS volume related issues on RDS or unexplained latency/downtime ?
jrochkind1 6y ago
RDS Postgres (actual postgres, not the Aurora product which can be "mostly postgres compatible") is another option, not just "EC2 self-managed instance". I guess we'd have to look at price comparisons for the size you actually need. I think depending on load, the price differences may be nominal.
- mijoharas 6y ago
  Thought I'd weigh in here, since we moved from heroku to ec2 to aurora to RDS postgres, so I can probably speak to this a little more. * ec2 self managed is easily the cheapest, we had a solid setup, with continuous backups and a read replica, if cost is a factor, it's easily a winner. However, there is a _lot_ of knowledge that goes with it. When it comes down to it, you can pay someone else to handle that. This isn't just the setup cost, you need to factor in ongoing maintenance (the number of people at my company that could have done complicated things with the instance was probably me and another engineer, and we didn't want to be permanently on call for this) and general risk. * EC2 did probably work out to between 1/2 to 3/4 (probably 2/3) the price of the equivalent RDS (tough to say exactly, as I'd need to factor in all the ancillary costs that are more "bundled together" in RDS) * RDS was much cheaper than aurora for our workload. (almost 1/2 the price). I think the main thing to remember, is that ec2 is much cheaper than RDS, but RDS is cheaper than engineers. With that, while we didn't need to do anything complicated other than the migration, the risk and possible engineering time and bus factor didn't feel worth it to stay on ec2.
  - rohansahai 6y ago
    Been on RDS for awhile now but have recently had some issues that are pushing us to heavily consider EC2 - especially since we've heard about considerable performance gains running on some of the bare metal EC2 options out there vs RDS.
    - mijoharas 6y ago
      We had equivalent performance (and we'd benchmarked both) for equivalent RDS and EC2 instances. (EDIT: to note, we didn't dig in heavily, just looked and saw it seemed the same, maybe there's something smart you can do to make EC2 a lot faster) First thing to think about for RDS (or any postgres instance really) is to figure out what the bottlenecks are, if your cache hit ratio is slightly low, or queries are generally feeling a little sluggish but cpu and ram aren't too high, I'd recommend looking into increasing the provisioned IOPS of the machine (requires no downtime, basically you can just make the storage faster). If your issues are ram/cpu being high you might want to consider moving to a larger instance. It can also be worth looking into the number of temp files being created by your queries and tweak your work_mem setting (you can use log_temp_files to see if this is likely to help a bit). Other than that, remember that every connection to the db is a separate process, so you don't want too many. It could be worth looking at RDS proxy or pg_bouncer if you think you have a lot of connections to your db.
takeda 6y ago
If you have a CM infrastructure set up, you're not gaining much. Tools like repmgr/barman (in public cloud you probably would use WAL-E/WAL-G) provide nearly all of the benefits. PostgreSQL is generally low maintenance, the only cases when this might not be true is when you're hitting performance bottlenecks and need to tune your database, optimize your tables, or use bigger instance etc, but RDS doesn't save you from that.

sciguymcq 6y ago

I currently manage, or am "lead dev", for around a half dozen to 10 apps using Postgres as the database. For two of those I use a managed database service (AWS RDS / AWS Aurora Postgres) because they drive mission critical high value products / services and I wouldn't consider using a standalone install of Postgres. For other apps that are not so mission critical I am perfectly happy saving considerable money and running them on a VPS on AWS EC2 or Digital Ocean or Linode. So ultimately it depends on the app and the risk tolerance for Recovery Time Objective and Recovery Point Objective.

fastball 6y ago
Worth noting that DO also has a hosted Postgres service.
rohansahai 6y ago
That's super interesting! Have heard people do the opposite for performance reasons (run on bare EC2) and storage cost reason. Do you notice a big performance difference between the EC2 apps and RDS apps? Also have you had any EBS volume related issues on RDS or unexplained latency/downtime ?

jrochkind1 6y ago

RDS Postgres (actual postgres, not the Aurora product which can be "mostly postgres compatible") is another option, not just "EC2 self-managed instance". I guess we'd have to look at price comparisons for the size you actually need. I think depending on load, the price differences may be nominal.

mijoharas 6y ago
Thought I'd weigh in here, since we moved from heroku to ec2 to aurora to RDS postgres, so I can probably speak to this a little more. * ec2 self managed is easily the cheapest, we had a solid setup, with continuous backups and a read replica, if cost is a factor, it's easily a winner. However, there is a _lot_ of knowledge that goes with it. When it comes down to it, you can pay someone else to handle that. This isn't just the setup cost, you need to factor in ongoing maintenance (the number of people at my company that could have done complicated things with the instance was probably me and another engineer, and we didn't want to be permanently on call for this) and general risk. * EC2 did probably work out to between 1/2 to 3/4 (probably 2/3) the price of the equivalent RDS (tough to say exactly, as I'd need to factor in all the ancillary costs that are more "bundled together" in RDS) * RDS was much cheaper than aurora for our workload. (almost 1/2 the price). I think the main thing to remember, is that ec2 is much cheaper than RDS, but RDS is cheaper than engineers. With that, while we didn't need to do anything complicated other than the migration, the risk and possible engineering time and bus factor didn't feel worth it to stay on ec2.
- rohansahai 6y ago
  Been on RDS for awhile now but have recently had some issues that are pushing us to heavily consider EC2 - especially since we've heard about considerable performance gains running on some of the bare metal EC2 options out there vs RDS.
  - mijoharas 6y ago
    We had equivalent performance (and we'd benchmarked both) for equivalent RDS and EC2 instances. (EDIT: to note, we didn't dig in heavily, just looked and saw it seemed the same, maybe there's something smart you can do to make EC2 a lot faster) First thing to think about for RDS (or any postgres instance really) is to figure out what the bottlenecks are, if your cache hit ratio is slightly low, or queries are generally feeling a little sluggish but cpu and ram aren't too high, I'd recommend looking into increasing the provisioned IOPS of the machine (requires no downtime, basically you can just make the storage faster). If your issues are ram/cpu being high you might want to consider moving to a larger instance. It can also be worth looking into the number of temp files being created by your queries and tweak your work_mem setting (you can use log_temp_files to see if this is likely to help a bit). Other than that, remember that every connection to the db is a separate process, so you don't want too many. It could be worth looking at RDS proxy or pg_bouncer if you think you have a lot of connections to your db.

takeda 6y ago

If you have a CM infrastructure set up, you're not gaining much. Tools like repmgr/barman (in public cloud you probably would use WAL-E/WAL-G) provide nearly all of the benefits. PostgreSQL is generally low maintenance, the only cases when this might not be true is when you're hitting performance bottlenecks and need to tune your database, optimize your tables, or use bigger instance etc, but RDS doesn't save you from that.