6 ms·

I have spent the last year and a half building a completely serverless production service on Lambda, API Gateway, and DynamoDB (along with the standard auxiliar

I have spent the last year and a half building a completely serverless production service on Lambda, API Gateway, and DynamoDB (along with the standard auxiliary services like CW, SNS, Route53, S3, CF, X-Ray, etc.). It was a lot of work establishing new patterns for many of the operational aspects, particularly custom CW metrics and A/B deployments with Lambda traffic shifting, but in the end everything is set up nicely and I'm quite pleased with the end result. We're starting to ramp up traffic now by orders of magnitude (with many more to come) and it's soooooo awesome knowing the stack is pretty much bombproof. Another super-nice thing is all internal authentication and networking being controlled by IAM rather than security groups/VPC/traditional networking - that aspect alone eliminates a tremendous number of headaches. My biggest complaints are probably DynamoDB eventual consistency (unavoidable when using GSIs), occasional CloudFormation funkiness (though no urgent prod issues yet, thankfully), CodeDeploy CW alarm rollback jankiness (which doesn't tell you which alarm triggered a rollback!!), and lambda coldstarts. But none of these are too terribly concerning and I have faith they'll get incrementally better over time, hopefully. The biggest cautionary tip I have is we run all our Lambdas with the max 3GB memory both for peace-of-mind and because the underlying EC2 instances have significantly faster CPU. We were seeing weird timeouts and latency initially with <1GB memory, so I'd be hesitant to run the service if the extra cost of using the biggest possible instances is a concern, which for us it is not. Another cost concern I should also mention is that we mitigate cold starts by running multiple canaries using scheduled lambdas (in addition to the standard canary role of generating a baseline of metrics and immediately detecting/alarming on end-to-end issues). We are effectively maintaining a constant warm pool which, in theory anyway, greatly decreases the chances customer traffic will hit cold starts. I'm not intimately involved with the financial aspects but I suspect achieving the same effect with EC2 would be significantly cheaper, at least with respect to infrastructure costs. I would guess, though, that the developer time savings achieved by massively reduced ops burden and overall system simplicity are probably comparable to the increased infrastructure cost, and very possibly hugely outweighing it.

zkim 8y ago
Excellent writeup. Are you serving client web traffic (if applicable) via Lambdas? Or deploying your web infrastructure to traditional instances managed through CF?
- not_kurt_godel 8y ago
  Yes, we're serving client web traffic directly with Lambdas via API Gateway. We have no traditional instances or non-serverless components of any kind in the core stack.
  - bufferoverflow 8y ago
    So how do you store your state? I assume DynamoDB part isn't serverless.
    - sokoloff 8y ago
      DynamoDB is as serverless as Lambda. (There are servers somewhere for both, but in neither case do you operate them.)
      - deanCommie 8y ago
        I'll give you there are no hosts to manage, but Lambda is serverless in the sense of "stateless" and "ephemeral". DynamoDB is still persistent storage. It seems that every every "hosted" solution is now being dubbed "serverless". :\ Oh and of course 5+ if-statements is now "AI".
        lostcolony 8y ago
        Lamdas aren't stateless or ephemeral either. Anything that occurs on code load will persist on that container (i.e., if you initialize something at the module level in Node, it will persist between calls to the same container; this can cause all kinds of weirdness if you aren't aware of it. For instance, I have seen where a dev read some data at load time, and then performing destructive operations on it as part of data transformations in the code, and then wondered why he was getting non-deterministic results back). And there's a half gig of temp space on each container you can write to as well. While the definition of what constitutes 'serverless' is pretty ambiguous, no one includes ephemeral state as a systems requirement, else you have something useless. DynamoDB is generally viewed as serverless because there's no management of an underlying VM, and for some definitions because it can scale out horizontally automatically, without downtime, to meet demand (as compared with RDS, or another managed database solution that can only scale vertically).
everdev 8y ago
Does Lambda have a good version control or CI workflow? My biggest question is how to develop severless functions with a team of developers.
- not_kurt_godel 8y ago
  You can utilize lambda traffic shifting in conjunction with CodeDeploy and Code Pipelines to do safe A/B (aka Blue/Green) deployments. It has some rough edges but overall works well. Check out these docs to get started: https://github.com/awslabs/serverless-application-model/blob/a09929bcdd7cd6083d5c344ce5a64208a56bc4a6/docs/safe_lambda_deployments.rst#traffic-shifting-using-codedeploy https://github.com/awslabs/serverless-application-model/blob... It's also possible to use API Gateway canary deployments and dedicated "preprod" stages as well - the benefit being you can, in theory, use traffic shifting lifecycle hooks to deploy a new version completely isolated from customer traffic and test it before incrementally rolling out your main A/B deployment. I created an experimental proof-of-concept for this but haven't had the time to flesh it out for production usage, but would very much like to at some point. tl;dr The answer to your question is yes, there is a good story around CI with Lambda using the AWS ecosystem. However Lambda alone is not something that should be relied on for long-term or definitive versioning on its own. Edit: thinking about your question more, it seems you are assuming that individual developers will directly edit Lambda function code in the console and you want to track versions or trigger deployments based on that activity. You absolutely should not ever be editing Lambda code manually directly in the console outside of one-off experiments/prototypes that are completely unrelated to dev/test/production. Always keep your Lambda function code under source control and deploy using zips uploaded to S3 and CloudFormation (the serverless framework[1] provides good tooling for this, though we don't use it - instead we use SAM and our own internal tools). [1] https://serverless.com/ https://serverless.com/
- mooreds 8y ago
  This looks useful: https://github.com/awslabs/serverless-application-model/blob/develop/docs/safe_lambda_deployments.rst https://github.com/awslabs/serverless-application-model/blob...
atmosx 8y ago
Your setup, although appealing from an AWS ecosystem perspective, sounds like a bit expensive to me on a first reading. Of course everything depends on the specifics but Lambdas and DynamoDB are expensive at scale. I wonder how it compares cost-wise to a more traditional solution.
- aequitas 8y ago
  Indeed, I always get the feeling you need some sort of exit strategy to a traditional model for when your service starts lifting of. Never did the cost calculations though.
  - jonathan-kosgei 8y ago
    I've felt the same way. Serverless is most appealing when you're starting out and have low traffic. It enabled us at ipdata.co to have the most global infrastructure possible with the lowest latencies at an insignificant cost. At some point I believe when we're big enough we might switch to using servers in all the regions where we currently run APIG+Lambda.
    - aequitas 8y ago
      The advantage of Serverless to me seems that it already forces you in a somewhat sane design and separation of concerns so all work out into the lambda functions should be easily translatable into a different architecture.
      - xrd 8y ago
        Very smart. Good point.
      - not_kurt_godel 8y ago
        Yes, very much this. A lot of the effort was extremely in-depth planning for scalability with respect to both unbounded traffic growth/adoption and expanding the team. In fact I coded an initial prototype in about a month that could have run quite happily on a single instance and subsequently broken out into a typical LB/Autoscaling group/DB architecture without much trouble. Now we have about 7 core microservices which are independenty scalable and deployable, many of which we anticipate handing off to dedicated teams as we expand and hire.
lifeisstillgood 8y ago
what is the business model around this - I mean 18 months to get "set up nicely" strikes me as hard to justify to upper management that is not invested already
- not_kurt_godel 8y ago
  Broadly speaking, it started out as a greenfield/experimental project with buy-in from senior management that is now going through the initial phase of productization and productionization. Although the AWS bill sounds expensive the whole project was effectively done by 3 developers including myself (2 backend/full stack and 1 frontend). There were also some deliberate management decisions that greatly prolonged implementation time - we could have easily shaved 8 months off that figure taking a more straightforward path.
  - lifeisstillgood 8y ago
    That is still a year long, 1/4M or more speculative investment. I would not moan too loudly about mgmt interference prolonging the project - that was a rare piece of long term willingness to invest for future technical gains - rare in my experience. (But imo the only way) well a moan bit loudly...
    - not_kurt_godel 8y ago
      Yes, it was/is a rare opportunity and I'm very grateful I had/have it. (That being said, the potential upsides are in the range of tens to hundreds of millions of dollars, even with only minimal success - so there is definitely a very real business incentive to invest in the project.) The prolongment I'm referring to isn't to do with the project being put on hold or anything like that. Essentially what happened was it was decided we'd build an "alpha" version of the stack to 'validate' the value of the project even though it was plainly clear what we had to do. The alpha stack nominally was supposed to be a cheap, quick version of the real architecture (which we'd already designed) whose supposed savings were gained by substituting manual processes for some of the APIs rather than actually building them. The end result was a. confirming what we already knew in that yes, the proposed functionality is fundamentally useful (very obvious from the outset) and b. a huge diversion of time and effort doing throwaway work that was about 65% as much work as just doing it the right way would have been, with the additional burden of having to perform the manual "API" functions, operational overhead of maintaining that stack while implementing the real one, burden of having to migrate data from the alpha stack to new stack once it was ready, and work to deprecate/tear it down once it was completely out-of-use. The overall wasted time was easily 8 months.
jonathan-kosgei 8y ago
>We're starting to ramp up traffic now by orders of magnitude (with many more to come) and it's soooooo awesome knowing the stack is pretty much bombproof. This does come with additional cost. Serverless pricing doesn't scale, your costs increase linearly with your usage, and there're no discounts for bulk usage or reserved pricing. We were recently on the receiving end of a massive HTTP GET Flood DDoS and although we did not experience any downtime as a result of it, I ended up finding out about it a few days later when billing alarms started going off.
- k__ 8y ago
  I read in many places that you should limit your max parallel executions.
  - jonathan-kosgei 8y ago
    We were wary of limiting paid users. Even with lambda's max concurrent function executions limit, when the function completes in a few milliseconds, the number of invocations per second can still be high.
    - k__ 8y ago
      And Shield wasn't an option?
      - jonathan-kosgei 8y ago
        Waf would've required setting up cloudfront in front of our endpoints which would've increased latency. Basic Shield is supposed to be on for all users but I don't think APIG is covered in the range of services covered by their Advanced Protection plan.
- citrablue 8y ago
  Curious - are you in an industry where you expected bad actors? I'm not blaming you -- far from it! Just wondering how applicable your experience is to the world I normally work in.

zkim 8y ago

Excellent writeup. Are you serving client web traffic (if applicable) via Lambdas? Or deploying your web infrastructure to traditional instances managed through CF?

not_kurt_godel 8y ago
Yes, we're serving client web traffic directly with Lambdas via API Gateway. We have no traditional instances or non-serverless components of any kind in the core stack.
- bufferoverflow 8y ago
  So how do you store your state? I assume DynamoDB part isn't serverless.
  - sokoloff 8y ago
    DynamoDB is as serverless as Lambda. (There are servers somewhere for both, but in neither case do you operate them.)
    - deanCommie 8y ago
      I'll give you there are no hosts to manage, but Lambda is serverless in the sense of "stateless" and "ephemeral". DynamoDB is still persistent storage. It seems that every every "hosted" solution is now being dubbed "serverless". :\ Oh and of course 5+ if-statements is now "AI".
      - lostcolony 8y ago
        Lamdas aren't stateless or ephemeral either. Anything that occurs on code load will persist on that container (i.e., if you initialize something at the module level in Node, it will persist between calls to the same container; this can cause all kinds of weirdness if you aren't aware of it. For instance, I have seen where a dev read some data at load time, and then performing destructive operations on it as part of data transformations in the code, and then wondered why he was getting non-deterministic results back). And there's a half gig of temp space on each container you can write to as well. While the definition of what constitutes 'serverless' is pretty ambiguous, no one includes ephemeral state as a systems requirement, else you have something useless. DynamoDB is generally viewed as serverless because there's no management of an underlying VM, and for some definitions because it can scale out horizontally automatically, without downtime, to meet demand (as compared with RDS, or another managed database solution that can only scale vertically).

everdev 8y ago

Does Lambda have a good version control or CI workflow? My biggest question is how to develop severless functions with a team of developers.

not_kurt_godel 8y ago
You can utilize lambda traffic shifting in conjunction with CodeDeploy and Code Pipelines to do safe A/B (aka Blue/Green) deployments. It has some rough edges but overall works well. Check out these docs to get started: https://github.com/awslabs/serverless-application-model/blob/a09929bcdd7cd6083d5c344ce5a64208a56bc4a6/docs/safe_lambda_deployments.rst#traffic-shifting-using-codedeploy https://github.com/awslabs/serverless-application-model/blob... It's also possible to use API Gateway canary deployments and dedicated "preprod" stages as well - the benefit being you can, in theory, use traffic shifting lifecycle hooks to deploy a new version completely isolated from customer traffic and test it before incrementally rolling out your main A/B deployment. I created an experimental proof-of-concept for this but haven't had the time to flesh it out for production usage, but would very much like to at some point. tl;dr The answer to your question is yes, there is a good story around CI with Lambda using the AWS ecosystem. However Lambda alone is not something that should be relied on for long-term or definitive versioning on its own. Edit: thinking about your question more, it seems you are assuming that individual developers will directly edit Lambda function code in the console and you want to track versions or trigger deployments based on that activity. You absolutely should not ever be editing Lambda code manually directly in the console outside of one-off experiments/prototypes that are completely unrelated to dev/test/production. Always keep your Lambda function code under source control and deploy using zips uploaded to S3 and CloudFormation (the serverless framework[1] provides good tooling for this, though we don't use it - instead we use SAM and our own internal tools). [1] https://serverless.com/ https://serverless.com/
mooreds 8y ago
This looks useful: https://github.com/awslabs/serverless-application-model/blob/develop/docs/safe_lambda_deployments.rst https://github.com/awslabs/serverless-application-model/blob...

atmosx 8y ago

Your setup, although appealing from an AWS ecosystem perspective, sounds like a bit expensive to me on a first reading. Of course everything depends on the specifics but Lambdas and DynamoDB are expensive at scale. I wonder how it compares cost-wise to a more traditional solution.

aequitas 8y ago
Indeed, I always get the feeling you need some sort of exit strategy to a traditional model for when your service starts lifting of. Never did the cost calculations though.
- jonathan-kosgei 8y ago
  I've felt the same way. Serverless is most appealing when you're starting out and have low traffic. It enabled us at ipdata.co to have the most global infrastructure possible with the lowest latencies at an insignificant cost. At some point I believe when we're big enough we might switch to using servers in all the regions where we currently run APIG+Lambda.
  - aequitas 8y ago
    The advantage of Serverless to me seems that it already forces you in a somewhat sane design and separation of concerns so all work out into the lambda functions should be easily translatable into a different architecture.
    - xrd 8y ago
      Very smart. Good point.
    - not_kurt_godel 8y ago
      Yes, very much this. A lot of the effort was extremely in-depth planning for scalability with respect to both unbounded traffic growth/adoption and expanding the team. In fact I coded an initial prototype in about a month that could have run quite happily on a single instance and subsequently broken out into a typical LB/Autoscaling group/DB architecture without much trouble. Now we have about 7 core microservices which are independenty scalable and deployable, many of which we anticipate handing off to dedicated teams as we expand and hire.

lifeisstillgood 8y ago

what is the business model around this - I mean 18 months to get "set up nicely" strikes me as hard to justify to upper management that is not invested already

not_kurt_godel 8y ago
Broadly speaking, it started out as a greenfield/experimental project with buy-in from senior management that is now going through the initial phase of productization and productionization. Although the AWS bill sounds expensive the whole project was effectively done by 3 developers including myself (2 backend/full stack and 1 frontend). There were also some deliberate management decisions that greatly prolonged implementation time - we could have easily shaved 8 months off that figure taking a more straightforward path.
- lifeisstillgood 8y ago
  That is still a year long, 1/4M or more speculative investment. I would not moan too loudly about mgmt interference prolonging the project - that was a rare piece of long term willingness to invest for future technical gains - rare in my experience. (But imo the only way) well a moan bit loudly...
  - not_kurt_godel 8y ago
    Yes, it was/is a rare opportunity and I'm very grateful I had/have it. (That being said, the potential upsides are in the range of tens to hundreds of millions of dollars, even with only minimal success - so there is definitely a very real business incentive to invest in the project.) The prolongment I'm referring to isn't to do with the project being put on hold or anything like that. Essentially what happened was it was decided we'd build an "alpha" version of the stack to 'validate' the value of the project even though it was plainly clear what we had to do. The alpha stack nominally was supposed to be a cheap, quick version of the real architecture (which we'd already designed) whose supposed savings were gained by substituting manual processes for some of the APIs rather than actually building them. The end result was a. confirming what we already knew in that yes, the proposed functionality is fundamentally useful (very obvious from the outset) and b. a huge diversion of time and effort doing throwaway work that was about 65% as much work as just doing it the right way would have been, with the additional burden of having to perform the manual "API" functions, operational overhead of maintaining that stack while implementing the real one, burden of having to migrate data from the alpha stack to new stack once it was ready, and work to deprecate/tear it down once it was completely out-of-use. The overall wasted time was easily 8 months.

jonathan-kosgei 8y ago

>We're starting to ramp up traffic now by orders of magnitude (with many more to come) and it's soooooo awesome knowing the stack is pretty much bombproof. This does come with additional cost. Serverless pricing doesn't scale, your costs increase linearly with your usage, and there're no discounts for bulk usage or reserved pricing. We were recently on the receiving end of a massive HTTP GET Flood DDoS and although we did not experience any downtime as a result of it, I ended up finding out about it a few days later when billing alarms started going off.

k__ 8y ago
I read in many places that you should limit your max parallel executions.
- jonathan-kosgei 8y ago
  We were wary of limiting paid users. Even with lambda's max concurrent function executions limit, when the function completes in a few milliseconds, the number of invocations per second can still be high.
  - k__ 8y ago
    And Shield wasn't an option?
    - jonathan-kosgei 8y ago
      Waf would've required setting up cloudfront in front of our endpoints which would've increased latency. Basic Shield is supposed to be on for all users but I don't think APIG is covered in the range of services covered by their Advanced Protection plan.
citrablue 8y ago
Curious - are you in an industry where you expected bad actors? I'm not blaming you -- far from it! Just wondering how applicable your experience is to the world I normally work in.