9 ms·
Behind the scenes, AWS Lambda
- chews 5y agonice writeup for how the magic really works. lambdas rock!
- abarrak 5y agoA couple of days ago, I tried to search on how AWS operates RDS behind the scenes, since it is a managed stateful service I was wondering whether it runs in a traditional way VM-based or in a fully containerized environment? .. Unfortunately, a simple search will lead you to the consumer/customer resources out there only.
- rorykoehler 5y agoBased on how they bill it, it looks like it's running on VMs
- StratusBen 5y agoAgreed. AWS RDS instance types are just EC2 instance types prefixed with "db." and you're choosing either single-AZ or multi-AZ deployments so presumably AWS is just spinning up 1 to 3 EC2 instances with some preconfigured software on them.
- navaati 5y agoFrom what I know there is a secret sauce beyond a mere AMI and a control plane, based on some EBS volumes magic. I may be mixing things up with Aurora though.
- fierro 5y agothis is very, very safe to assume. There is probably a hundred engineer's worth of "secret sauce" for an entire managed DB product line.
- aeyes 5y agoThere were some comments in the early days that the Multi-AZ magic for classic RDS was just drbd on top of EBS. Aurora is a completely different approach where the RDBMS code is modified to directly interface with EBS instead of going through a traditional OS filesystem layer.
- rrdharan 5y agoAmazon published a paper describing how Aurora works: https://www.allthingsdistributed.com/files/p1041-verbitski.pdf https://www.allthingsdistributed.com/files/p1041-verbitski.p...
- redis_mlc 5y ago> There were some comments in the early days that the Multi-AZ magic for classic RDS was just drbd on top of EBS. If this was true, then you could support multiple database vendors in a generic way, and gradually add the supported products, without knowing the intricacies of each product's replication details. Hmm. :)
- ec109685 5y agoThis is a good paper that talks about Aurora and provides some insight into how RDS operates: https://www.allthingsdistributed.com/files/p1041-verbitski.pdf https://www.allthingsdistributed.com/files/p1041-verbitski.p... It’s nice that AWS builds their own higher level abstractions on the same primitives outside developers use. Feels like they eat their own dogfood much more than Google where they bypass GCP and instead utilize underlying Borg primitives for many services.
- abarrak 5y agoNice. I wonder if the stateful merits provided and marketed by containers orchestrates (e.g. K8S) is something they will consider in the future? ..
- rejectedandsad 5y agoTo build a new service at Amazon, the general path of least resistance these days is to use Lambda. If not Lambda, then ECS. If not ECS (or if it requires bare metal) then EC2.
- valenterry 5y agoYeah, except for the lambda time limit...
- rejectedandsad 5y agoGiven most things are frontended by API Gateway anyway this isn’t too big of a concern since there’s already an effective 30 second timeout there. For ETL like stuff you can use Stepfunctions too.
- valenterry 5y agoIt's still sad, because then you need a lambda for just these one or two things that might run long (or maybe even forever) and now you have to mix lambda and e.g. ECS.
- personlurking 5y agoAside: Lambda, out of Peru, is apparently the next virus variant we need to worry about. Edit: I had never seen the word before then I saw it used twice in two days, in different contexts. Just thought it was odd.
- minitoar 5y agoIt’s not that odd if you consider it is the 11th letter of the Greek alphabet.
- adflux 5y agoCan we agree to leave COVID out of this discussion?
- deleted 5y ago[deleted]
- Exmoor 5y agohttps://en.wikipedia.org/wiki/Frequency_illusion https://en.wikipedia.org/wiki/Frequency_illusion
- nl 5y agoThe Y-combinator is a function in Lambda calculus.
- dr_kretyn 5y agoIs this write up correct? How do they know that? I don't see any references on info source except a talk at re:invent.
- garblegarble 5y agoThere's a decent references section at the bottom, and having watched the talks and briefly scanning the Firecracker paper referenced, they do back up the writer.
- dr_kretyn 5y agoThat's a footnote section and to me the listing is only partially related to the text, i.e. the write up contains a lots more details on multiple components. Thanks though for backing up this write up. That's +1 for confidence.
- dmarinus 5y agoWhen I was at re:invent 2019 I joined some chalk talks which weren't recorded (or not published). Some of the hosts told lot of details of their internal infrastructure.
- bschaatsbergen 5y agoBoth Marc Brooker (lead developer on the AWS Lambda team) giving the talks at Re:Invent as I mentioned in the footnotes, and the official documentation that's out there will provide you with a lot of information.
- chrisweekly 5y agoThis is great! Awesome writeup w thekind of details that are sometimes opaque and hard to find documentation for. I recently deployed a NextJS app using Serverless framework (and serverless-nextjs), so Lambda@Edge... looking fwd to playing more with compute at CDN edgein general (eg fly.io). Amazing how easy it is, esp. as someone who came into webdev in 1998.
- emteycz 5y agoConsidering your long experience, didn't you feel like we lost a lot post-PHP? I also stepped out of the PHP world into JS, and never understood why there isn't any apache2-modnodejs... And to me, the serverless JS movement seems to be just that, but with a lot of unnecessary baggage.
- nuclearnice3 5y agoWe surely picked up some baggage. Much of it vendor specific. But also we jettison some? You stick a lambda into API gateway and you're on the internet. No servers. No linux setup. No apache conf. I'd encourage you to dive in for 20 hours in pure curiosity mode and see what you find.
- throwaway3699 5y agoLambda is a proprietary solution that only works for people on AWS. Linux is open, and I just need to put it on a box. How do I install the AWS Lambda stack on a standard Linux box?
- beckingz 5y agoOpenstack? https://www.openstack.org/ https://www.openstack.org/
- nuclearnice3 5y agoYou can't. Are you actually asking this question? Or you pretending to ask a question because you think the fact that AWS Lambda run on AWS is some huge gotcha that I never imagined and no one would ever tolerate? I explicitly note vendor-specific baggage. AWS revenue is over 45 billion annually and half of customers use lambda.
- mlerner 5y agoIf you're interested in Firecracker, I wrote a summary of the original paper here: https://www.micahlerner.com/2021/06/17/firecracker-lightweight-virtualization-for-serverless-applications.html https://www.micahlerner.com/2021/06/17/firecracker-lightweig...
- bschaatsbergen 5y agoGreat article @mlerner
- daxfohl 5y agoAny idea how much it has diverged from crosvm?
- mjb 5y agoQuite a lot. Initially, a lot of the changes were removing things from crosvm, but adding features like snapshots, and factoring things out into RustVMM, has made them diverge a lot more. There's some data in the paper about how similar they were then, too.
- carlosf 5y agoReally cool post! From the architecture, it's not really clear to me why Lambdas have the 15 min limitation. It seems to me AWS could use the same infrastructure to make a product that competes with Google Cloud Run. Maybe it's a businesses thing?
- kolanos 5y agoThis service exists, it's called AWS Fargate [0]. [0]: https://read.iopipe.com/how-far-out-is-aws-fargate-a2409d2f9bc7 https://read.iopipe.com/how-far-out-is-aws-fargate-a2409d2f9...
- carlosf 5y agoOof Makes sense! I wish Fargate was easier to use and had a scale to 0 feature. If App Runner ends up supporting private deployments then we can have a true Cloud Run competitor.
- kolanos 5y ago> I wish Fargate was easier to use and had a scale to 0 feature. Fargate can be scaled to zero. Also, have you tried the CLI? [0] [0]: https://github.com/aws/copilot-cli https://github.com/aws/copilot-cli
- simonw 5y agoWhen I say "scale to zero" I mean like Cloud Run or AWS Lambda: I define it as the service automatically scaling to zero (and hence costing nothing to run) in between requests, but automatically starting up again when a new request comes in - so the request still gets served, it just suffers from a few seconds of cold-start time. I'm pretty sure Fargate doesn't offer this. It sounds like you're talking about the ability to manually (or automatically through scripting) turn off your Fargate containers, then manually turn them back on again - but not in a way that an incoming request still gets served even though the container wasn't running when the request first arrived.
- tnolet 5y agoGreat write up. Besides the technical parts, AWS Lambda probably created a ton of new businesses/ startups that otherwise would have been hard or at least expensive to get going.
- bschaatsbergen 5y agoGood to know you enjoyed the read!
- simonw 5y agoThis is a great article - I really appreciate when people take the time to assemble details from a bunch of different sources (Firecracker paper, re:Invent talks) and turn them into a useful overview like this. Clearly Bruno got a lot of the details right, Jeff Barr tweeted a link to this a few weeks ago: https://twitter.com/jeffbarr/status/1404512248152825857 https://twitter.com/jeffbarr/status/1404512248152825857
- daxfohl 5y agoOne other thing I learned here is that lambda@edge is not actually run on the edge at all. It is forwarded to the nearest datacenter to execute. Not enough capacity in edges to spin up entire VMs for everything, even with Firecracker.
- mcspiff 5y agoThey do have Cloudfront Functions now for real Edge compute: https://aws.amazon.com/blogs/aws/introducing-cloudfront-functions-run-your-code-at-the-edge-with-low-latency-at-any-scale/ https://aws.amazon.com/blogs/aws/introducing-cloudfront-func...
- floatboth 5y agoWhich are barely "compute", but very nice for redirect-rewrite-add-headers type stuff. Like mruby in h2o, lua/js in various nginx modules and so on. Their JS engine seems quite odd, they selectively have very modern features but not let/const o_0
- Something1234 5y agoFantastic paper. So I've been playing with the java and python runtimes and it's absolutely stunning how much better python is on execution and start up time. Also how does an event actually get to the lambda handler? Because they can come from all kind of sources.
- mdaniel 5y agoI believe they fire up an http server, based on how their local executor behaves, and then do "servlet-y" (or WSGi-y) dispatch into the entry point method
- ben0x539 5y agoJudging from five minutes of browsing the go runtime sources, they poll some lambda API: https://github.com/aws/aws-lambda-go/blob/159d1c69878562cd5409719d4fc4cfa298e2c4bf/lambda/runtime_api_client.go#L71 https://github.com/aws/aws-lambda-go/blob/159d1c69878562cd54... The response is apparently posted to another API endpoint. Googling for the environment variable actually specifying the API endpoint, I found https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom... which seems to spell out some of the details. Of course that just moves the question to "where does the runtime API get the event from?" but at that point they could be doing all kinds of things, I guess. Now I'm curious to see what happens if you ask for more events before returning a response for the first event you got. Can you actually process events in parallel?
- bigodanktime 5y agoPretty sure its just gRPC calls, and firecracker passes events along to its firecracker-containerd services. My guess is java is slow as it works really well when the JIT can optimize your code. Longer running functions will very likely outperform python.
- marp00 5y agoGreat article. Is there an according one about Google Cloud Functions?
- a-dub 5y agoi keep seeing talk about fast firecracker boot times... but as far as i can tell firecracker is something that talks to the KVM apis and does monitoring... wouldn't fast boot times be a result of kvm and the structure of the VM being booted, or does this boot time metric include scheduling (on one or more firecracker hosts) and delivery of the image to the runner host for the VM?
- bigodanktime 5y agoLots of interesting work is being done in this area (Currently doing research around serverless at the moment). Cold start up times still remain a pretty large issue (125ms start up for VM is still quite large) but some interesting papers trying to attack this, through strategies like snapshotting! https://arxiv.org/pdf/2101.09355.pdf https://arxiv.org/pdf/2101.09355.pdf Also predicting function calls to properly schedule and reduce cold start latency https://www.usenix.org/system/files/atc20-shahrad.pdf https://www.usenix.org/system/files/atc20-shahrad.pdf
- Dunedan 5y agoThese 125ms are only the startup time of the MVM and don't include additional latency introduced by optimizing the code package and the involvement of the placement service. You can also avoid the cold start penalties entirely, if you're willing to pay extra for provisioned concurrency [1]. [1]: https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html#configuration-concurrency-provisioned https://docs.aws.amazon.com/lambda/latest/dg/configuration-c...
- bigodanktime 5y agoThis seems to be a solution that comes at the cost of the consumer which is fine if they want to pay for it, and seems to be an option provided for more latency sensitive applications. Obviously, one could eliminate the cold start issue in general by just constantly paying for a running EC2 instance. But cold start is still an issue for the provider as cold start is a cost to them, even reducing cold start of internal runtimes would be a massive benefit (For example pre-warmed JITs). Better cold start times means better bin packing for their services, and overall less cost to everyone.
- robertlagrant 5y agoI'd love to see a similar writeup of Cloudflare Workers.