4 ms·

We run quite a bit of infrastructure via AWS lambda via the Serverless project including a data crawling/ingestion pipeline, data cleaning and enriching, and ev

by Rapzid 9y ago

We run quite a bit of infrastructure via AWS lambda via the Serverless project including a data crawling/ingestion pipeline, data cleaning and enriching, and even a flask API.. I agree with the sentiment of the article, if not the specifics of every point. Definitely feel some pain in areas. Sometimes I feel we are doing architectural contortions. Pain points include: * SQS is not an event source :( * Cold starts can be very problematic for latency sensitive scenarios when using languages/frameworks that startup slow. It's pre-fork without the fork :( * No concurrency in event handling exacerbating some of the above * Artifact size limits can often be rough * Bizarre artifact storage limits and little help in the way of cleaning them up * CloudFormation limits and bugs * The service is rather opaque * Onboarding; good luck :) Learned the hard way: * Created too many individual project repos and serverless services. Classic monolith vs services pains but more to do with too-small service boundaries * Interface to serverless projects too fine grained(too many lambda functions per service) * Didn't start using step functions earlier to bridge the gap between stateful processes and stateless processors * Used SQS as database(long story) * Ran a flask API as a lambda. Works, but there is just no way we would use this if it weren't internal due to cold start/scale latencies with bursty traffic.

andrewstuart 9y ago
You can keep the Lambda function warm by setting up a Cloudwatch event that pings the Lambda function every 5 minutes.
- Rapzid 9y ago
  I believe there is a common misconception that this is an actual solution to the problem of cold start latencies. You can attempt to keep between 1 and X instances of a lambda function running, however the underlying provisioning system is mostly a black box without published details and supposedly not entirely deterministic. Keeping a single instance of the function running isn't going to give great control over the tail on latencies. This is particularly true when faced with bursty, inconsistent traffic patterns.
  - andrewstuart 9y ago
    Hopefully the warm/cold start problem is something they will find a solution for in the end.
    - icebraining 9y ago
      Well, Docker already has (experimental) support for CRIU[1]. Since they control the environment, it should be possible to prevent people from doing stuff that would prevent it from working. [1] https://criu.org/Main_Page https://criu.org/Main_Page
- perlgeek 9y ago
  And now you're back to trying to manage the infrastructure, over which you only have very indirect control. We've come full circle, no?
wickawic 9y ago
> Used SQS as database(long story) I would love to hear it. I have mistakenly used SQS for medium-term storage of state, which is what I am assuming your issue was.
- Rapzid 9y ago
  That's the gist of it; tracking position of a long running process by writing messages back into the SQS queue. Any of the numerous edge cases that will cause a dupe record to appear(some of them documented SQS implementation details) would cause the process to duplicate :| Then triplicate :|
Roodgorf 9y ago
> SQS is not an event source :( I really don't understand why this is the case. Every time we need a queue for something we end up having to explain this again to whoever our newest developers are, which seems to me like a sign of how unintuitive it is. I have yet to encounter a use case where we didn't want to grab items out of the queue relatively frequently or even as they were added, so why force us to mess with scheduled events and long polling to do so?
- vacri 9y ago
  SQS is probably carrying design baggage from when it was introduced in 2004. I've never used it, but would Kinesis be better for what you want to do?
  - Roodgorf 9y ago
    In most cases I think SNS is closer to what I want: multiple producers feeding into a single pipeline that notifies its consumer as soon as possible. The main thing from SQS is the ability to retry a failed message. If SQS were event based, you could potentially try to process a message as soon as it was received and then have a batch job to reprocess any messages that have been left in the queue however often you like. The closest to this we got was by using SNS and feeding it into SQS, or even a table somewhere, but that gets sort of messy pretty quickly and didn't seem worth the extra infrastructure. Edit: I didn't really respond to the suggestion of Kinesis here. I haven't done much with Kinesis, but I get the sense it's overkill for the use cases I'm thinking about. I don't plan to have a large amount of producers and don't expect them to have a consistent or large stream of data. If I'm wrong in that being the main use case for Kinesis please correct me.
    - vacri 9y ago
      I haven't used kinesis myself, I've just heard that it's "SQS with the FIFO issue fixed", and wondered if it might also be suitable for you. Judging from the other comments, it wouldn't be a silver bullet, though.

by Rapzid 9y ago

andrewstuart 9y ago

You can keep the Lambda function warm by setting up a Cloudwatch event that pings the Lambda function every 5 minutes.

Rapzid 9y ago
I believe there is a common misconception that this is an actual solution to the problem of cold start latencies. You can attempt to keep between 1 and X instances of a lambda function running, however the underlying provisioning system is mostly a black box without published details and supposedly not entirely deterministic. Keeping a single instance of the function running isn't going to give great control over the tail on latencies. This is particularly true when faced with bursty, inconsistent traffic patterns.
- andrewstuart 9y ago
  Hopefully the warm/cold start problem is something they will find a solution for in the end.
  - icebraining 9y ago
    Well, Docker already has (experimental) support for CRIU[1]. Since they control the environment, it should be possible to prevent people from doing stuff that would prevent it from working. [1] https://criu.org/Main_Page https://criu.org/Main_Page
perlgeek 9y ago
And now you're back to trying to manage the infrastructure, over which you only have very indirect control. We've come full circle, no?

wickawic 9y ago

> Used SQS as database(long story) I would love to hear it. I have mistakenly used SQS for medium-term storage of state, which is what I am assuming your issue was.

Rapzid 9y ago
That's the gist of it; tracking position of a long running process by writing messages back into the SQS queue. Any of the numerous edge cases that will cause a dupe record to appear(some of them documented SQS implementation details) would cause the process to duplicate :| Then triplicate :|

Roodgorf 9y ago

> SQS is not an event source :( I really don't understand why this is the case. Every time we need a queue for something we end up having to explain this again to whoever our newest developers are, which seems to me like a sign of how unintuitive it is. I have yet to encounter a use case where we didn't want to grab items out of the queue relatively frequently or even as they were added, so why force us to mess with scheduled events and long polling to do so?

vacri 9y ago
SQS is probably carrying design baggage from when it was introduced in 2004. I've never used it, but would Kinesis be better for what you want to do?
- Roodgorf 9y ago
  In most cases I think SNS is closer to what I want: multiple producers feeding into a single pipeline that notifies its consumer as soon as possible. The main thing from SQS is the ability to retry a failed message. If SQS were event based, you could potentially try to process a message as soon as it was received and then have a batch job to reprocess any messages that have been left in the queue however often you like. The closest to this we got was by using SNS and feeding it into SQS, or even a table somewhere, but that gets sort of messy pretty quickly and didn't seem worth the extra infrastructure. Edit: I didn't really respond to the suggestion of Kinesis here. I haven't done much with Kinesis, but I get the sense it's overkill for the use cases I'm thinking about. I don't plan to have a large amount of producers and don't expect them to have a consistent or large stream of data. If I'm wrong in that being the main use case for Kinesis please correct me.
  - vacri 9y ago
    I haven't used kinesis myself, I've just heard that it's "SQS with the FIFO issue fixed", and wondered if it might also be suitable for you. Judging from the other comments, it wouldn't be a silver bullet, though.