4 ms·
We have billions of rows of data and very complicated CTEs doing joins on a dozen tables matching based on GEOS radius data, and it's the primary query for our
by nasalgoat 9y ago
We have billions of rows of data and very complicated CTEs doing joins on a dozen tables matching based on GEOS radius data, and it's the primary query for our application. We're running PostgreSQL 9.5 and need to have all the data in memory for the fastest results.
Part of the problem is that the network disk that EC2 provides as EBS is 100x slower than local disk, so keeping all the indexes and data in memory is the only way to replicate physical hardware performance.
If anyone knows of a better EC2 setup for PostgreSQL, I'm all ears.
- anarazel 9y ago> Part of the problem is that the network disk that EC2 provides as EBS is 100x slower than local disk, so keeping all the indexes and data in memory is the only way to replicate physical hardware performance. > If anyone knows of a better EC2 setup for PostgreSQL, I'm all ears. Have a proper replication & archiving setup, and use instance storage. If you have configured streaming replication to 1-2 other servers, and archive your WAL to s3 (using wal-e or such), you're already above EBS's guarantees (99.9% durability IIRC?).
- nasalgoat 9y agoInstance storage is much more limited and is heavily tied to the specific instance types you pick.
- anarazel 9y agoIndeed. I didn't want to say it's a panacea, just that you can sometimes get a lot better performance for your money. E.g. larger i3's both have decent IO performance and sizes. Still doesn't even remotely compete with what you can get with "normal" hardware.