3 ms·

At present, we put all incoming page views into SQS Hmm, not the most economical of choices, but... We had decided that we were going to simply increase our l

At present, we put all incoming page views into SQS Hmm, not the most economical of choices, but... We had decided that we were going to simply increase our lambda concurrency limit to 8,000,000 requests per second (800,000 concurrents) and handle the spam attacks ... wait, what? If you're using Lambda functions for something as trivial as logging page views, you evidently have more money than sense. The author is concerned about this story reading like an advert for AWS Shield, but to me it reads like a case study in when "Serverless" is a bad idea.

JackWritesCode 6y ago
Lambda concurrency wasn't actually increased to 800,000 concurrents. It's a stupid idea that someone would think up after fighting attacks all day.
- cperciva 6y ago
  Regardless of what you set Lambda concurrency to: Are you seriously invoking Lambdas for every page view?
  - JackWritesCode 6y ago
    We are indeed. Neither of us enjoy DevOps so we pay a premium to not have to manage servers. It brings a huge mental health benefit (we are a two person company), we're profitable and our monthly Lambda cost isn't that significant. Honestly, the biggest inefficiency (in terms of cost) is our use of SQS/RDS, which we are ditching soon.
    - deleted 6y ago
      [deleted]
    - cperciva 6y ago
      Well, it's your money... I'd be happy to help make this more efficient though (at no charge of course). Offhand I'd say "web server which accumulates data and uploads it to S3 every N requests or M seconds" would probably get you what you need at a tiny fraction of the cost of "lambda which posts to SQS". Create an AMI and toss it at an autoscaling group and you really won't need to worry about scaling issues either.
      - not_kurt_godel 6y ago
        If you were charging, how much would you charge to set up a server that you literally never have to bounce to get it back up?
        cperciva 6y ago
        That's the wrong approach -- you're going to want to be able to reboot (or replace) servers for security patches if nothing else. But if you're regularly rebooting servers because they stop responding, something has gone very wrong.
      - JackWritesCode 6y ago
        I bet you could set up something way cheaper. And I'm certain you know 10x more than me regarding servers, hardening, configuration, etc. And I'm certain you enjoy servers! For us, the cost works and we have appropriate margin for it. The cost savings aren't worth the extra "we have to monitor these servers" thoughts. Our approach is 100% emotional.
        cperciva 6y ago
        Whatever works for you. :-) The offer is open if you ever change your mind (e.g. if the cost becomes annoyingly large when you scale up further).
        iampims 6y ago
        ApiGateway -> kinesis stream -> lambda consumes batch of N page views. You keep the serverless approach you like, and it will cost you a fraction of what you currently spend.
        JackWritesCode 6y ago
        I saw Kinesis stream today with DynamoDB and I will have to review it :)
        xupybd 6y ago
jrockway 6y ago
My main thought here is how important backpressure is for managing queues. We all hate to admit defeat, but sometimes shit is broken and you just have to say "sorry, service not available" and drop the requests on the floor. No buffer is infinitely long.
- JackWritesCode 6y ago
  And to add on this, the plan moving forward is to default to writing data to the appropriate places (aim is < 100-200ms) but fall back to SQS when services are unavailable.

JackWritesCode 6y ago

Lambda concurrency wasn't actually increased to 800,000 concurrents. It's a stupid idea that someone would think up after fighting attacks all day.

cperciva 6y ago
Regardless of what you set Lambda concurrency to: Are you seriously invoking Lambdas for every page view?
- JackWritesCode 6y ago
  We are indeed. Neither of us enjoy DevOps so we pay a premium to not have to manage servers. It brings a huge mental health benefit (we are a two person company), we're profitable and our monthly Lambda cost isn't that significant. Honestly, the biggest inefficiency (in terms of cost) is our use of SQS/RDS, which we are ditching soon.
  - deleted 6y ago
    [deleted]
  - cperciva 6y ago
    Well, it's your money... I'd be happy to help make this more efficient though (at no charge of course). Offhand I'd say "web server which accumulates data and uploads it to S3 every N requests or M seconds" would probably get you what you need at a tiny fraction of the cost of "lambda which posts to SQS". Create an AMI and toss it at an autoscaling group and you really won't need to worry about scaling issues either.
    - not_kurt_godel 6y ago
      If you were charging, how much would you charge to set up a server that you literally never have to bounce to get it back up?
      - cperciva 6y ago
        That's the wrong approach -- you're going to want to be able to reboot (or replace) servers for security patches if nothing else. But if you're regularly rebooting servers because they stop responding, something has gone very wrong.
    - JackWritesCode 6y ago
      I bet you could set up something way cheaper. And I'm certain you know 10x more than me regarding servers, hardening, configuration, etc. And I'm certain you enjoy servers! For us, the cost works and we have appropriate margin for it. The cost savings aren't worth the extra "we have to monitor these servers" thoughts. Our approach is 100% emotional.
      - cperciva 6y ago
        Whatever works for you. :-) The offer is open if you ever change your mind (e.g. if the cost becomes annoyingly large when you scale up further).
      - iampims 6y ago
        ApiGateway -> kinesis stream -> lambda consumes batch of N page views. You keep the serverless approach you like, and it will cost you a fraction of what you currently spend.
        JackWritesCode 6y ago
        I saw Kinesis stream today with DynamoDB and I will have to review it :)
      - xupybd 6y ago

jrockway 6y ago

My main thought here is how important backpressure is for managing queues. We all hate to admit defeat, but sometimes shit is broken and you just have to say "sorry, service not available" and drop the requests on the floor. No buffer is infinitely long.

JackWritesCode 6y ago
And to add on this, the plan moving forward is to default to writing data to the appropriate places (aim is < 100-200ms) but fall back to SQS when services are unavailable.