8 ms·

Serverless Aurora

114 points by abd12 9y ago

machbio 9y ago
Am struggling to understand the concept of Serverless Aurora and AWS Lambda integration - they highlight that AWS Lambda would scale on traffic and there by Serverless Aurora kicks into support this scale - but there is a limitation on AWS Lambda inside a VPC[1]. So in order to really utilize the Serverless Aurora fully - the Proxy fleet would have special permission to access Aurora? Also, has anyone solved the issue of AWS Lambda (in VPC) scaling beyond the limited IP space ? [1]http://docs.aws.amazon.com/lambda/latest/dg/vpc.html http://docs.aws.amazon.com/lambda/latest/dg/vpc.html ENIs = Projected peak concurrent executions * (Memory in GB / 3GB)
- bni 9y ago
  But would you really need to put Serverless Aurora in a VPC? Is it even possible to do that? I expect it working like DynamoDB in this regard, the tables/database just exists and access is determined by IAM role.
- Prefinem 9y ago
  You probably shouldn't be putting your Lambda Functions in a VPC unless it needs access to a VPC resource. There was a pretty good presentation on that in the "What's new in Serverless" seminar. On top of that, I believe you can do a /2 CIDR which gives you over a million IPs to use. Not sure why you think that wouldn't be enough.
andrewstuart 9y ago
I wonder if this can be accessed from Lambda VPCs?
- simonmales 9y ago
  It will be accessible via HTTP, just like all of Amazon's APIs. Doesn't matter if you are in a VPC or not.
  - zwily 9y ago
    Aurora is accessible via TCP (mysql or Postgres), not HTTP.
moduspwnens14 9y ago
It looks really great. The only concern I have (relative to other serverless features) is that it does have a top-end architectural limit (256 ACUs). It's probably not worth fretting over, but it is worth mentioning when comparing to DynamoDB. That means in building a serverless web app, you might look at starting with Serverless Aurora, graduating to regular Aurora (as usage increases due to pricing), and then perhaps moving off workloads better suited to DynamoDB as you discover them.
paulgb 9y ago
I wish Amazon or Google would just make a simple key/value store to give minimal persistence for their serverless offerings. DynamoDB comes closest but it still overcomplicates things.
- eropple 9y ago
  I'm not super familiar with DynamoDB except as a strict key/value store. You set a partition key, you ignore your range key, and you go--what's particularly overcomplicated about that?
  - paulgb 9y ago
    It's been a while since I played around with it but I recall having to increase the number of write nodes to load data at a reasonable rate, then drop the number back down to avoid overpaying. And there were limits on how often you could lower the number of instances so I had to be careful not to reach the limit. Maybe I'm misremembering or that has changed?
    - sitharus 9y ago
      In June this year AWS shipped DynamoDB autoscaling: https://aws.amazon.com/blogs/aws/new-auto-scaling-for-amazon-dynamodb/ https://aws.amazon.com/blogs/aws/new-auto-scaling-for-amazon...
      - paulgb 9y ago
        Good to know!
  - noobiemcfoob 9y ago
    Setting the number of write nodes and read nodes to match your workload. I was scaling up a system and as more things came online, the traffic became more bursty we were constantly bumping them up and wondering if we were allocating too much (probably useless fretting regardless). Ultimately, we moved to an S3 only solution since we have such a low throughput environment and the simplicity is addictive.
boyter 9y ago
This was also the biggest news of re:Invent for me. Total game changer. No need to provision large R instance types to support the batch processing that happens a 3AM and sits idle most of the time. All environmental tiers can have 100% the same configuration. I know of some projects burning millions a month in database costs because they like to replicate environments for testing branches. Switching to serverless Aurora would reduce that cost to probably low thousands. Of course it all depends on how quickly it can scale up to meet the load and I reserve full judgement till I can try it out, but am very interested in this.
oh_hello 9y ago
The idea is exciting, but the pricing looks too high for side projects. The docs state "Aurora Serverless can scale from a minimum of 1 ACU to a maximum of 256 ACUs." Unless I'm reading this incorrectly, you are paying for 1 ACU 24 hours a day. $0.06 * 24 * 365 = $525.60 per year. This is more expensive than a low-end RDS instance. It's a shame, I immediately thought this would be the perfect solution for side projects and prototypes, but the pricing killed it.
- Mizza 9y ago
  Although not applicable for all use cases, you can use static storage as an alternative for radical cost savings: https://github.com/Miserlou/NoDB https://github.com/Miserlou/NoDB https://github.com/Miserlou/zappa-django-utils#using-an-s3-backed-database-engine https://github.com/Miserlou/zappa-django-utils#using-an-s3-b...
- sjtgraham 9y ago
  I was about to say you don't get encryption at rest with a low-end instance, but that looks like it's no longer the case since this summer.
- jajern 9y ago
  You pay per second of usage and the ACU starts and shuts down automatically. If your DB is only actually accessed a occasionally then you are not paying for 24hrs of usage. I think the auto-scaling is the draw though, not necessarily the price. So you are only getting charged the the database is being accessed. I think the examples on the pricing page make it a little more clear.
  - k__ 9y ago
    Would be strange if it was otherwise. The whole serverless stuff only pays if you have highly varying loads OR want to save of Ops staff.
  - jchanimal 9y ago
    FaunaDB Serverless Cloud has global ACID transactions with per-request pricing. You can optimize your queries for cost by looking at the response headers, or support your organization with an on-premise multitenant cluster. Learn more about pricing and features here https://fauna.com/serverless https://fauna.com/serverless
jondubois 9y ago
Serverless is a bad idea, it was designed to serve big infrastructure providers as a way to increase their lock-in factor but the benefits to software developers is negative. Serverless is simple but opaque; it's great for making demo apps but not ideal for real production systems. It narrows down integration possibilities, complicates large-scale development and deployment, and often, it makes it more difficult to keep track of backend errors... In addition to this, it makes you fully dependent on a single infrastructure provider; it takes away all your leverage when negotiating hosting costs. The control panels of some of these services have become so complex and elaborate that you'll be wishing that you could just SSH into the server directly.
- sokoloff 9y ago
  Serverless is (way?) over-hyped, but has valid use cases I think. As an example, for a tiny side-project, I don't care about the markup or lock-in, I just want my code to run an unknown, small number of times per day for a cost-effective total price and a low cognitive and support overhead on me.
  - jondubois 9y ago
    Sure, but I've found that even some use cases that seem perfect are in fact not as good as you think. For example, I used Amazon Elastic Transcoder to do video compression/transcoding at a previous company thinking that we would save a lot of time but the API wasn't great; a lot of the default settings weren't right for our use case and the API and notification flow was way more complicated than it needed to be and prone to errors (due the the sheer complexity of input parameters it could handle). At one point, we had a problem receiving SNS notifications from the transcoder and we didn't know what was causing it because we couldn't effectively access/browse the error log and was was exposed to us wasn't detailed enough. Basically the visibility in Amazon-land wasn't great. We had to pay Amazon for special support so that they would tell us what the error was (after weeks of back-and-forth). Now that it's all running, it's fine; so long as we don't need to change anything and don't need to start scaling up/sharding the pipelines; no sure how to do this through their UI.
obiefernandez 9y ago
This is super interesting to me on applications where I'm using Heroku's $50/month Postgres plan. Almost inevitably I find that the only reason I've upgraded past hobby level is for performance, not scaling. That it would auto-scale under load is just the icing on the cake.
- le-mark 9y ago
  What performance problems were you having specifically? What are your use cases? Just curious to hear your experience.
  - obiefernandez 9y ago
    Alas, I don't have metrics. Just seemed noticeably faster on the more expensive Postgres plan.
sctb 9y ago
Recent related discussion: https://news.ycombinator.com/item?id=15808271 https://news.ycombinator.com/item?id=15808271
deleted 9y ago
[deleted]
Footkerchief 9y ago
Any ideas what replication model they're using under the hood to bring new instances online so quickly?

114 points by abd12 9y ago

Am struggling to understand the concept of Serverless Aurora and AWS Lambda integration - they highlight that AWS Lambda would scale on traffic and there by Serverless Aurora kicks into support this scale - but there is a limitation on AWS Lambda inside a VPC[1]. So in order to really utilize the Serverless Aurora fully - the Proxy fleet would have special permission to access Aurora? Also, has anyone solved the issue of AWS Lambda (in VPC) scaling beyond the limited IP space ? [1]http://docs.aws.amazon.com/lambda/latest/dg/vpc.html http://docs.aws.amazon.com/lambda/latest/dg/vpc.html ENIs = Projected peak concurrent executions * (Memory in GB / 3GB)

bni 9y ago
But would you really need to put Serverless Aurora in a VPC? Is it even possible to do that? I expect it working like DynamoDB in this regard, the tables/database just exists and access is determined by IAM role.
Prefinem 9y ago
You probably shouldn't be putting your Lambda Functions in a VPC unless it needs access to a VPC resource. There was a pretty good presentation on that in the "What's new in Serverless" seminar. On top of that, I believe you can do a /2 CIDR which gives you over a million IPs to use. Not sure why you think that wouldn't be enough.

andrewstuart 9y ago

I wonder if this can be accessed from Lambda VPCs?

simonmales 9y ago
It will be accessible via HTTP, just like all of Amazon's APIs. Doesn't matter if you are in a VPC or not.
- zwily 9y ago
  Aurora is accessible via TCP (mysql or Postgres), not HTTP.

moduspwnens14 9y ago

It looks really great. The only concern I have (relative to other serverless features) is that it does have a top-end architectural limit (256 ACUs). It's probably not worth fretting over, but it is worth mentioning when comparing to DynamoDB. That means in building a serverless web app, you might look at starting with Serverless Aurora, graduating to regular Aurora (as usage increases due to pricing), and then perhaps moving off workloads better suited to DynamoDB as you discover them.

paulgb 9y ago

I wish Amazon or Google would just make a simple key/value store to give minimal persistence for their serverless offerings. DynamoDB comes closest but it still overcomplicates things.

eropple 9y ago
I'm not super familiar with DynamoDB except as a strict key/value store. You set a partition key, you ignore your range key, and you go--what's particularly overcomplicated about that?
- paulgb 9y ago
  It's been a while since I played around with it but I recall having to increase the number of write nodes to load data at a reasonable rate, then drop the number back down to avoid overpaying. And there were limits on how often you could lower the number of instances so I had to be careful not to reach the limit. Maybe I'm misremembering or that has changed?
  - sitharus 9y ago
    In June this year AWS shipped DynamoDB autoscaling: https://aws.amazon.com/blogs/aws/new-auto-scaling-for-amazon-dynamodb/ https://aws.amazon.com/blogs/aws/new-auto-scaling-for-amazon...
    - paulgb 9y ago
      Good to know!
- noobiemcfoob 9y ago
  Setting the number of write nodes and read nodes to match your workload. I was scaling up a system and as more things came online, the traffic became more bursty we were constantly bumping them up and wondering if we were allocating too much (probably useless fretting regardless). Ultimately, we moved to an S3 only solution since we have such a low throughput environment and the simplicity is addictive.

boyter 9y ago

This was also the biggest news of re:Invent for me. Total game changer. No need to provision large R instance types to support the batch processing that happens a 3AM and sits idle most of the time. All environmental tiers can have 100% the same configuration. I know of some projects burning millions a month in database costs because they like to replicate environments for testing branches. Switching to serverless Aurora would reduce that cost to probably low thousands. Of course it all depends on how quickly it can scale up to meet the load and I reserve full judgement till I can try it out, but am very interested in this.

oh_hello 9y ago

The idea is exciting, but the pricing looks too high for side projects. The docs state "Aurora Serverless can scale from a minimum of 1 ACU to a maximum of 256 ACUs." Unless I'm reading this incorrectly, you are paying for 1 ACU 24 hours a day. $0.06 * 24 * 365 = $525.60 per year. This is more expensive than a low-end RDS instance. It's a shame, I immediately thought this would be the perfect solution for side projects and prototypes, but the pricing killed it.

Mizza 9y ago
Although not applicable for all use cases, you can use static storage as an alternative for radical cost savings: https://github.com/Miserlou/NoDB https://github.com/Miserlou/NoDB https://github.com/Miserlou/zappa-django-utils#using-an-s3-backed-database-engine https://github.com/Miserlou/zappa-django-utils#using-an-s3-b...
sjtgraham 9y ago
I was about to say you don't get encryption at rest with a low-end instance, but that looks like it's no longer the case since this summer.
jajern 9y ago
You pay per second of usage and the ACU starts and shuts down automatically. If your DB is only actually accessed a occasionally then you are not paying for 24hrs of usage. I think the auto-scaling is the draw though, not necessarily the price. So you are only getting charged the the database is being accessed. I think the examples on the pricing page make it a little more clear.
- k__ 9y ago
  Would be strange if it was otherwise. The whole serverless stuff only pays if you have highly varying loads OR want to save of Ops staff.
- jchanimal 9y ago
  FaunaDB Serverless Cloud has global ACID transactions with per-request pricing. You can optimize your queries for cost by looking at the response headers, or support your organization with an on-premise multitenant cluster. Learn more about pricing and features here https://fauna.com/serverless https://fauna.com/serverless

jondubois 9y ago

Serverless is a bad idea, it was designed to serve big infrastructure providers as a way to increase their lock-in factor but the benefits to software developers is negative. Serverless is simple but opaque; it's great for making demo apps but not ideal for real production systems. It narrows down integration possibilities, complicates large-scale development and deployment, and often, it makes it more difficult to keep track of backend errors... In addition to this, it makes you fully dependent on a single infrastructure provider; it takes away all your leverage when negotiating hosting costs. The control panels of some of these services have become so complex and elaborate that you'll be wishing that you could just SSH into the server directly.

sokoloff 9y ago
Serverless is (way?) over-hyped, but has valid use cases I think. As an example, for a tiny side-project, I don't care about the markup or lock-in, I just want my code to run an unknown, small number of times per day for a cost-effective total price and a low cognitive and support overhead on me.
- jondubois 9y ago
  Sure, but I've found that even some use cases that seem perfect are in fact not as good as you think. For example, I used Amazon Elastic Transcoder to do video compression/transcoding at a previous company thinking that we would save a lot of time but the API wasn't great; a lot of the default settings weren't right for our use case and the API and notification flow was way more complicated than it needed to be and prone to errors (due the the sheer complexity of input parameters it could handle). At one point, we had a problem receiving SNS notifications from the transcoder and we didn't know what was causing it because we couldn't effectively access/browse the error log and was was exposed to us wasn't detailed enough. Basically the visibility in Amazon-land wasn't great. We had to pay Amazon for special support so that they would tell us what the error was (after weeks of back-and-forth). Now that it's all running, it's fine; so long as we don't need to change anything and don't need to start scaling up/sharding the pipelines; no sure how to do this through their UI.

obiefernandez 9y ago

This is super interesting to me on applications where I'm using Heroku's $50/month Postgres plan. Almost inevitably I find that the only reason I've upgraded past hobby level is for performance, not scaling. That it would auto-scale under load is just the icing on the cake.

le-mark 9y ago
What performance problems were you having specifically? What are your use cases? Just curious to hear your experience.
- obiefernandez 9y ago
  Alas, I don't have metrics. Just seemed noticeably faster on the more expensive Postgres plan.

sctb 9y ago

Recent related discussion: https://news.ycombinator.com/item?id=15808271 https://news.ycombinator.com/item?id=15808271

deleted 9y ago

[deleted]

Footkerchief 9y ago

Any ideas what replication model they're using under the hood to bring new instances online so quickly?