6 ms·

Amazon Aurora with PostgreSQL Compatibility Supports Serverless

88 points by kesor 7y ago

BenoitP 7y ago
AWS RDS for PostgresSQL supports many popular extensions [1]. I wonder if these are activated here or will be in time. One of pg's unfair advantage is its ecosystem. I have user-triggered batch processing jobs that need PostGIS; Aurora Serverless for pg would be a great fit. [1] https://docs.aws.amazon.com/fr_fr/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.FeatureSupport.Extensions https://docs.aws.amazon.com/fr_fr/AmazonRDS/latest/UserGuide...
- nmussy 7y ago
  I haven't had a chance to try it out myself yet, but the Aurora FAQ[0] seems confirm your wish: Aurora database engine is designed to be wire-compatible with PostgreSQL 9.6 and 10, and supports the same set of PostgreSQL extensions that are supported with RDS for PostgreSQL 9.6 and 10 For reference, full list of PostgeSQL RDS extensions in the documentation[1] [0] https://aws.amazon.com/rds/aurora/faqs/ https://aws.amazon.com/rds/aurora/faqs/ [1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.FeatureSupport.Extensions https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_...
  - BenoitP 7y ago
    Oh, that's great! Thanks
- hbcondo714 7y ago
  Still waiting for AWS RDS for PostgresSQL to support the TimescaleDB extension, 2 years and counting: https://github.com/timescale/timescaledb/issues/65 https://github.com/timescale/timescaledb/issues/65
unixhero 7y ago
Aurora with Postgres is amazing. I don't know if I want to go back to Redshift after using it. Pair it with a nice instance of PGADMIN4 and enjoy.
- eloff 7y ago
  They're pretty different databases with different use cases in mind. If you don't really have big data you can get away with a row oriented database. If you do have big data for reals - well then you almost certainly have to use a column oriented store.
- sk5t 7y ago
  I can't tell if you're being sarcastic, or if your tastes in software are extremely distinct from mine and most--comparing Aurora to Redshift? Pgadmin4, really??
sudhirj 7y ago
This is on HN like just now https://news.ycombinator.com/item?id=20398353 https://news.ycombinator.com/item?id=20398353
mattgibson 7y ago
For clarity, there is a significant warm up time for the DB to load your data when you cold-start, so this is sadly not suitable for use with serverless lambda functions in a web request cycle. A good use case for this would be batch processing on demand e.g. a background job that's infrequently run when records are uploaded somehow. You save the cost of keeping a big DB running in between jobs and the start up time doesn't matter.
- BenoitP 7y ago
  What kind of time are we talking about, 10 seconds or 3-4 minutes? Or very short, but everything will be painfully slow?
  - sudhirj 7y ago
    On the order of tens of seconds, for a cold start from zero. Enough time to load your RAM cache from hibernation and boot up OS and Postgres.
    - boarnoah 7y ago
      Is it really? Been trying out Aurora MySQL for our staging db, and its closer to 1 min+ for a cold start.
      - sudhirj 7y ago
        But isn’t the variance on this completely useless? It’ll depend entirely on the size of your cache, the region you’re in, the traffic at that time and rack you’re lucky enough to be on. Will also depend on AWS configs, which can change daily.
    - jjeaff 7y ago
      But what does it mean to be warm in this case? Last called less than a minute ago? 10 minutes?
- crb002 7y ago
jakozaur 7y ago
Has anybody here used Amazona Aurora serverless? Is it a viable alternative for big RDS instances?
- nh_99 7y ago
  A company that I work for was in the beta of serverless for MySQL, and we currently use it for both dev and production workloads. We're a startup, so it definitely simplifies devops for us, but personally I think you're better off just running a large RDS instance. The main problem with it that we have run into is that if there are active transactions in the database, it can't find a "scale point". So when our application gets hit with thousands of users at a time and they are all generating database transactions, it can't scale up. It's definitely been good for us in development though, and I would highly recommend it for that.
  - davidjnelson 7y ago
    They do allow you to drop connections with forced scale points if that's a tradeoff your application can make https://aws.amazon.com/about-aws/whats-new/2019/04/amazon_aurora_serverless_now_supports_a_minimum_capacity_of_1_unit_and_a_new_scaling_option/ https://aws.amazon.com/about-aws/whats-new/2019/04/amazon_au...
- jjeaff 7y ago
  From a cost perspective, when I last checked, it was quite a bit more expensive to go serverless unless you have very intermittent demand or have very spikey and unpredictable traffic.
quantguy11959 7y ago
How well would this work with something like airflow ? If I have jobs that run only end of day for an hour on airflow, I pay just for an hour of the database ?
- habitue 7y ago
  Depends on how you've configured airflow. It polls its dag directory constantly, and may do some db queries for each poll (don't quote me on this, I definitely don't know everything about the internals of airflow). From what I remember you can reduce the polling period in config
  - quantguy11959 7y ago
    Yeah I’ll need to do some research, if I can only poll the db for just the time I need this will be a huge cost saving for me. I think the db only gets queried if a dag actually runs but I’m not 100% sure.
fabioyy 7y ago
I've cut costs by an order of magnitude by changing back to old school dedicated servers. ( using the same data center that aws/google uses, equinix ) IOPS on aws is ridiculous expensive
- redis_mlc 7y ago
  1) AWS and GCP have their own datacenters with their own custom switches and routers, etc. 2) Sure, as a DBA, I prefer dedicated servers with SSD. If you don't have a DBA, then RDS/Aurora are ok too. For production, ensure you have a replica either way. MySQL 8 has multi-master now (or use Percona Server/Galera), so you can easily enable HA, and I'm sure Postgres does too.
ralusek 7y ago
Is this still capped at data of 64TB?
- iamed2 7y ago
  Smaller, 32 TiB. https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_Limits.html#RDS_Limits.FileSize.Aurora https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...
jjeaff 7y ago
Although I know AWS is not optimizing for this use case, I see a lot of potential with serverless databases for things like open source, home brew style applications where people want to own and control their data but don't want to pay for a monthly service. IoT devices could communicate asynchronously and store important data using just llamdba functions and serverless db instances. The first thought that came to mind was something like the bitwarden open source server. Currently, you can run a nano instance in the cloud that stays on forever but it needs to be maintained and it might go down occasionally. Additionally, it will cost you a few bucks a month assuming you need more than the single nano allowed by the free tier. With serverless functions and a serverless database you could have a central store without needing to run a full server. Cold startup wouldn't be an issue since you only need to sync to the server occasionally.

88 points by kesor 7y ago

AWS RDS for PostgresSQL supports many popular extensions [1]. I wonder if these are activated here or will be in time. One of pg's unfair advantage is its ecosystem. I have user-triggered batch processing jobs that need PostGIS; Aurora Serverless for pg would be a great fit. [1] https://docs.aws.amazon.com/fr_fr/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.FeatureSupport.Extensions https://docs.aws.amazon.com/fr_fr/AmazonRDS/latest/UserGuide...

nmussy 7y ago
I haven't had a chance to try it out myself yet, but the Aurora FAQ[0] seems confirm your wish: Aurora database engine is designed to be wire-compatible with PostgreSQL 9.6 and 10, and supports the same set of PostgreSQL extensions that are supported with RDS for PostgreSQL 9.6 and 10 For reference, full list of PostgeSQL RDS extensions in the documentation[1] [0] https://aws.amazon.com/rds/aurora/faqs/ https://aws.amazon.com/rds/aurora/faqs/ [1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html#PostgreSQL.Concepts.General.FeatureSupport.Extensions https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_...
- BenoitP 7y ago
  Oh, that's great! Thanks
hbcondo714 7y ago
Still waiting for AWS RDS for PostgresSQL to support the TimescaleDB extension, 2 years and counting: https://github.com/timescale/timescaledb/issues/65 https://github.com/timescale/timescaledb/issues/65

unixhero 7y ago

Aurora with Postgres is amazing. I don't know if I want to go back to Redshift after using it. Pair it with a nice instance of PGADMIN4 and enjoy.

eloff 7y ago
They're pretty different databases with different use cases in mind. If you don't really have big data you can get away with a row oriented database. If you do have big data for reals - well then you almost certainly have to use a column oriented store.
sk5t 7y ago
I can't tell if you're being sarcastic, or if your tastes in software are extremely distinct from mine and most--comparing Aurora to Redshift? Pgadmin4, really??

sudhirj 7y ago

This is on HN like just now https://news.ycombinator.com/item?id=20398353 https://news.ycombinator.com/item?id=20398353

mattgibson 7y ago

For clarity, there is a significant warm up time for the DB to load your data when you cold-start, so this is sadly not suitable for use with serverless lambda functions in a web request cycle. A good use case for this would be batch processing on demand e.g. a background job that's infrequently run when records are uploaded somehow. You save the cost of keeping a big DB running in between jobs and the start up time doesn't matter.

BenoitP 7y ago
What kind of time are we talking about, 10 seconds or 3-4 minutes? Or very short, but everything will be painfully slow?
- sudhirj 7y ago
  On the order of tens of seconds, for a cold start from zero. Enough time to load your RAM cache from hibernation and boot up OS and Postgres.
  - boarnoah 7y ago
    Is it really? Been trying out Aurora MySQL for our staging db, and its closer to 1 min+ for a cold start.
    - sudhirj 7y ago
      But isn’t the variance on this completely useless? It’ll depend entirely on the size of your cache, the region you’re in, the traffic at that time and rack you’re lucky enough to be on. Will also depend on AWS configs, which can change daily.
  - jjeaff 7y ago
    But what does it mean to be warm in this case? Last called less than a minute ago? 10 minutes?
crb002 7y ago

jakozaur 7y ago

Has anybody here used Amazona Aurora serverless? Is it a viable alternative for big RDS instances?

nh_99 7y ago
A company that I work for was in the beta of serverless for MySQL, and we currently use it for both dev and production workloads. We're a startup, so it definitely simplifies devops for us, but personally I think you're better off just running a large RDS instance. The main problem with it that we have run into is that if there are active transactions in the database, it can't find a "scale point". So when our application gets hit with thousands of users at a time and they are all generating database transactions, it can't scale up. It's definitely been good for us in development though, and I would highly recommend it for that.
- davidjnelson 7y ago
  They do allow you to drop connections with forced scale points if that's a tradeoff your application can make https://aws.amazon.com/about-aws/whats-new/2019/04/amazon_aurora_serverless_now_supports_a_minimum_capacity_of_1_unit_and_a_new_scaling_option/ https://aws.amazon.com/about-aws/whats-new/2019/04/amazon_au...
jjeaff 7y ago
From a cost perspective, when I last checked, it was quite a bit more expensive to go serverless unless you have very intermittent demand or have very spikey and unpredictable traffic.

quantguy11959 7y ago

How well would this work with something like airflow ? If I have jobs that run only end of day for an hour on airflow, I pay just for an hour of the database ?

habitue 7y ago
Depends on how you've configured airflow. It polls its dag directory constantly, and may do some db queries for each poll (don't quote me on this, I definitely don't know everything about the internals of airflow). From what I remember you can reduce the polling period in config
- quantguy11959 7y ago
  Yeah I’ll need to do some research, if I can only poll the db for just the time I need this will be a huge cost saving for me. I think the db only gets queried if a dag actually runs but I’m not 100% sure.

fabioyy 7y ago

I've cut costs by an order of magnitude by changing back to old school dedicated servers. ( using the same data center that aws/google uses, equinix ) IOPS on aws is ridiculous expensive

redis_mlc 7y ago
1) AWS and GCP have their own datacenters with their own custom switches and routers, etc. 2) Sure, as a DBA, I prefer dedicated servers with SSD. If you don't have a DBA, then RDS/Aurora are ok too. For production, ensure you have a replica either way. MySQL 8 has multi-master now (or use Percona Server/Galera), so you can easily enable HA, and I'm sure Postgres does too.

ralusek 7y ago

Is this still capped at data of 64TB?

iamed2 7y ago
Smaller, 32 TiB. https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_Limits.html#RDS_Limits.FileSize.Aurora https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

jjeaff 7y ago

Although I know AWS is not optimizing for this use case, I see a lot of potential with serverless databases for things like open source, home brew style applications where people want to own and control their data but don't want to pay for a monthly service. IoT devices could communicate asynchronously and store important data using just llamdba functions and serverless db instances. The first thought that came to mind was something like the bitwarden open source server. Currently, you can run a nano instance in the cloud that stays on forever but it needs to be maintained and it might go down occasionally. Additionally, it will cost you a few bucks a month assuming you need more than the single nano allowed by the free tier. With serverless functions and a serverless database you could have a central store without needing to run a full server. Cold startup wouldn't be an issue since you only need to sync to the server occasionally.