11 ms·
Improved VPC Networking for AWS Lambda
- EwanToo 7y agoThis is a great improvement for Lambda users, much reduced cold start times!
- gazzini 7y agoThis is huge for Lambda. It allows devs to create “serverless” apps [1], with relational databases, without 10+ second cold-start times. In the article, they measure it as 988ms. I have tried building an API using API Gateway <-> Lambda, but had to choose between using DynamoDB to store data (no-SQL, so challenging to query) or suffering unacceptably long response times whenever a request happens to cause a cold-start. Theoretically, this problem is now going away! [1] https://serverless-stack.com https://serverless-stack.com
- paulddraper 7y ago*It allows devs to create those apps _within a VPC_. You could always have fast startup with Lambda + database outside the VPC.
- deleted 7y ago[deleted]
- k__ 7y agoAlso, wasn't Aurora Serverless created because of that problem?
- nostrebored 7y agoAurora Serverless also handles connections. The problem of having a burst of 1000 concurrent invocations accessing your databases still exists even with VPC access
- reilly3000 7y agoThat limit can be raised, apparently. I've seen mention of limits up to 30K concurrent invocations.
- gazzini 7y agoI think Aurora Serverless has even worse [1] cold-start times (for the DB itself), and it was intended as more of a price-optimization than a performance boost. [1] https://forums.aws.amazon.com/thread.jspa?threadID=288043 https://forums.aws.amazon.com/thread.jspa?threadID=288043
- gazzini 7y agoYou raise a fair point, this was possible, although it seems safe to say it would be a compromise on security. I think it’s best not to expose the DB to outside connections in general, although it is still possible [1] when using RDS instances. I think this is different for things like DynamoDB because, instead of a standard SQL-like db “connection”, they use AWS role-based auth for each request. Of course, one could always configure some type of proxy service between the lambda and the DB... but that seems antithetical to going “serverless” in the first place. [1] https://stackoverflow.com/questions/45227397/publicly-accessing-aws-rds-from-outside-vpc https://stackoverflow.com/questions/45227397/publicly-access... Edit: I thought it was not possible to expose an RDS instance outside of a VPC, but I was wrong (you can place it in a public subnet, linked in [1]).
- jabart 7y agoWhich is how most breach announcements start "A database server was found with an open port exposed to the internet and no or poor authentication, all records were exposed." This also should mean that Lambda's can get stable public IPs through a VPC for firewalls as well. *edit for must to most.
- fulafel 7y agoBut VPC is not an especially efficient additional "defense-in-depth" layer against this kind of "fucked up both firewall and password" configuration mistake. The first 2 obvious ones are passwords, network-level firewalling, host-level firewalling of course, and after that you can add monitoring / port scanning for all your "must be firewalled" services. And you can mandate better-than-passwords authentication methods[1]. Etc. The latter is better because it is more general and doesn't add costly complexity to your networking topology (by way of NAT and/or ambiguous rfc1918 addressing) [1] For example https://www.postgresql.org/docs/current/auth-cert.html https://www.postgresql.org/docs/current/auth-cert.html or https://aws.amazon.com/premiumsupport/knowledge-center/users-connect-rds-iam/ https://aws.amazon.com/premiumsupport/knowledge-center/users...
- Niksko 7y agoYou mention defense in depth, but then immediately decide that an extra layer of defense is unnecessary.
- fulafel 7y agoAre you proposing that by acknowledging defense-in-depth, consistency dictates that one should pile up as many layers per attack vector as possible? Maybe, if you have infinite resources and don't need to make compromises on where you spend effort and resources in your risk management plan. But that's rarely the case in the real world.
- danb232 7y agois that a good tutorial? looks really good on the surface!
- mvanbaak 7y agoIf you put an event bus in the middle (kinesis) your api-lambda functions don't need direct access to your RDS. Subscribe lambda functions to your kinesis stream, and let them handle the link to your RDS. This way you wont notice the cold starts.
- meekins 7y agoThis comment doesn't seem to make sense, could you elaborate a bit? How would you replace the database working as a persistence layer to an API application by polling an event stream?
- dabeeeenster 7y agoIt doesnt replace the DB. It just uses Kineses to be the messaging provider from the lambda to the DB and back. Not sure that's a great idea TBH but who knows?!
- scarface74 7y agoAnd then when you need to read the database?
- anherome21003 7y agoI dont understand why people use AWS Lambda. Here in France, people use PHP with docker and it works just fine. Another scam from amazon?
- paulddraper 7y agoFaaS offers more responsive resource scaling and is "easier" to manage.
- whalesalad 7y agoI don't understand why people drive cars. Here in XYZ we ride horses everywhere and it works just fine.
- k__ 7y agoI think Lambda is more like Uber and Docker more like Rent-A-Car. With Uber you don't have to drive, they just fetch and deliver you and a few things you have on you. With Rent-A-Car, you can transport much more stuff, but you also have to drive yourself.
- anherome21003 7y agoMost websites of the world run php, comparing it to horses is a bit unfair ;-)
- k__ 7y agoYou can use PHP on AWS Lambda if you wish. If you use Lambda instead of Docker, you don't have to mess with container orchestration.
- aledalgrande 7y agoActually now you can deploy your docker layers directly to lambda, so you are developing on exactly the same environment. https://github.com/awslabs/aws-lambda-container-image-converter https://github.com/awslabs/aws-lambda-container-image-conver...
- jfbaro 7y agoWow! That's great. Cold starts are no longer a show stopper! Rust powered APIs running on AWS .. It sounds really exciting
- paulddraper 7y agoIconoclast view ahead (change my mind please): AWS does tons of stuff around VPCs....I feel like they really want me to use them (or their customers really want to use them), but I just don't see why. I just run RDS on the internet. I don't have to muck with the complexity or cost of NATs or peering or Lambda slow start or any other weird networking issues. I know it's "public", but that seems irrelevant in the era of cloud services. This isn't any different than, say, how Firebase or a million other services run. Should I be concerned that my Firebase apps are insecure because someone isn't overlaying a 10.* network on them? EDIT: I should clarify that I understand the legitimacy of security groups, especially for technologies that weren't meant to operate outside a firewall. But that's mostly a different subject; AWS had security groups years before VPCs and subnets and NATs.
- enitihas 7y agoVPCs are very useful when running things like elasticache though( memcache and redis), because AFAIK those don't have an authentication ecosystem so making them public would be a terrible idea.
- saurik 7y agoJust use security groups, which fully solved this problem without all of the overhead and complexity of VPC.
- paulddraper 7y agoMemcache has had reliable authentication (SASL) for some time. Redis has authentication meant to be a secondary protection. But that's a good point. I suppose all the services I use already have security models (usually more complex, multi-user ones, so agent X can read but not modify, etc.). HOWEVER...this could be solved with security groups, but it seems that's not the model AWS has emphasized. Security groups are orthogonal to NAT and private networks; AWS had security groups before it had VPCs.
- saurik 7y agoSo making the actual listening port for a database server "public" is generally a bad idea as that is another attack surface of code that honestly is hardly ever made public... but if when you say "public" you mean you are using security groups (which are super trivial to use and easy to understand) to define which other AWS devices can access the port, then yeah: I have never seen any reason why this entire feature should exist and the concept of having to think about IP address ranges as if they somehow matter is one of the things I was escaping when I moved to cloud in the first place, and somehow they wanted to reintroduce it? Why?!? It doesn't even work well (!!), and introduces tons of latency into everything it touches (not just Lambda) :/.
- ajoy 7y agoThis solves one part of the cold start problem. Starting the container and loading the image on to it is still going to cause some latency.
- nostrebored 7y agoSolves might be strong, but it removes a big portion of the cold start latency that was difficult to optimize for and out of the control of developers. Creating minimal images isn't difficult for a number of environments (e.g. webpacking your node.js lambdas) and barring necessarily large images (think pandas on Lambda) this puts a lot of control for the cold start p99 back in the hands of customers. Overall, definitely a big win!
- StreamBright 7y agoWhich can be mitigated by invoking your own Lambda functions once every minute or 5 minutes. Usually does not blow the budget.
- nostrebored 7y agoWarming functions in the previous VPC architecture was always a questionable practice. You had no guarantee that your environments would be warm across all subnets or which subnets would handle incoming requests. Beyond that, what happens to requests which you receive when the function is being warmed? You still incur cold starts. There has never been a guarantee of environment reuse. Any architecture which isn't capable of incurring cold starts is not a good fit for serverless.
- scarface74 7y agoWhich is a horrible idea.... How many lambdas do you keep warm? 5, 10, 20? Every new connection is a new lambda instance. You're still just delaying the inevitable. Just use Fargate if you want to stay serverless and don't want the cold start times -- well at least before today.
- StreamBright 7y ago
- slovenlyrobot 7y agoThis has been a /major/ sore point for Lambda use, amazing they fixed it, and always great to see they've documented the intense engineering requirements involved to make it happen. AWS is a beautiful mix of business and technology, it's very rare to see such a large engineering-driven organization managing to balance customer friendliness. I'm an unashamed fanboy
- k__ 7y agoMajor is a bit harsh. As far as I know this was only an issue for legacy architectures.
- scarface74 7y agoNo. Using an RDMS instead of DynamoDB is not a “legacy” architecture. You also shouldn’t expose your database publicly.
- k__ 7y agoRDMS is not legacy, but perimeter security certainly is.
- scarface74 7y agoI’m one of the harshest critics of “lift and shifters” - old school net ops people who get one certificates by watching an ACloudGuru video, duplicate their on prem infrastructure and processes to the cloud and don’t go all in on the advantages of it and end up costing their clients more - but nowhere is it considered “legacy” to not use perimeter security.
- jimmychangas 7y agoHonest question: what, in you opinion, is the state-of-the-art approach? Something like BeyondCorp?
- k__ 7y ago
- jmb12686 7y agoAWS announced this enhancement at 2018 re:Invent. It was slated for "sometime in 2019". I was excited, and I'm impressed that they released the feature well ahead of the end of the year (and before the next conference, which would obviously raise a few questions)
- scarface74 7y agoThey did something similar with drift detection and cloud formation. They announced it at reInvent 2017 and released it one week before reInvent 2018.
- reilly3000 7y agoThis is great news, but I'm bummed they didn't bundle the NAT gateway with this service. In a typical function that calls out to get data from a service and reads/writes from a DB in a VPC, that requires the somewhat painful configuration of a NAT gateway and dedicated subnets, as well as a $36/month bill for the NAT gateway service. There are some workarounds that using multiple lambdas, but they have their own gotchas. Still, hooray, this is good news. The Data API is great for Serverless Aurora, but I can't use that with BI tools.
- abhorrence 7y agoYou can run your own gateway instance(s) for a lot cheaper than the nat gateway service. There are definitely some tradeoffs, but if $36/mo is an issue, they can be worthwhile: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_NAT_Instance.html https://docs.aws.amazon.com/vpc/latest/userguide/VPC_NAT_Ins...
- scarface74 7y agoThis is not meant to be a criticism of AWS, I’m an AWS true believer, but the main purpose of going to AWS is to make the “undifferentiated heavy lifting” someone else’s problem not to save money. Going to AWS to save money on resources is about like going to the Apple Store to buy a cheap laptop.
- deleted 7y ago[deleted]
- reilly3000 7y agoI’m not against AWS or $36/mo. It just is kinda a drag when the promise of serverless is pay per user and scaling to zero. You could get a nice EC2 t3.medium and do a lot more RPS for the cost of that NAT and Lambda invocations.
- scarface74 7y agoIf you don’t care about cold starts, there is always Aurora Serverless with the Data API. I don’t believe it requires either a NAT or for the lambda to be attached to your VPC.
- deleted 7y ago[deleted]