3 ms·

What do folks use for something like Meta’s Async? RabbitMQ? > The workloads commonly executed on Async are those that do not require blocking an active user’s

What do folks use for something like Meta’s Async? RabbitMQ? > The workloads commonly executed on Async are those that do not require blocking an active user’s experience with a product and can be performed anywhere from a few seconds to several hours after a user’s action.

rangledangle 3y ago
Sounds like something you could do with SQS and Lambda. They just have massive datacenter infra and compute at disposal.
- salil999 3y ago
  Yeah but what about something that won't make my wallet cry?
  - jjice 3y ago
    Pretty much any queue and a compute will do. SQS paired with lambda an event bridge (I forget if that's the service to trigger the lambda or not) would be solid. I lean towards compute that's a bit easier to run in different settings so just any old code checking the queue and executing as expected also works. SQS is cheap and it's pretty damn solid. I believe it was the first AWS service. But in reality, any managed queue or even a well structured DB table can be fine, depending on your scale.
    - cj 3y ago
      Is using Lambda with an external data store (like MongoDB) a solved problem? I remember this being an issue a long time ago, where Lambda pretty much was a non starter for use cases where you're running 1000 independent tasks that each, on their own, is a small unit of work (just 1 or 2 database queries). Long way of asking: is there a way to share a database connection between lambda functions or does each run need to re-establish / re-authenticate a new connection with the database?
      - josevalerio 3y ago
        Declare your connection outside of your handler function - this post is from 2017 but it shows it clearly: https://www.jeremydaly.com/reuse-database-connections-aws-lambda/ https://www.jeremydaly.com/reuse-database-connections-aws-la... It will get re-used across lambda invocations as long as the instance of it isn't killed which typically happens after a few minutes
      - alexbevi 3y ago
        For MongoDB specifically see the best practices docs for Lambda (https://www.mongodb.com/docs/atlas/manage-connections-aws-lambda/ https://www.mongodb.com/docs/atlas/manage-connections-aws-la...). This outlines creating the handler outside the function context to allow the client to be reused between invocations.
  - gnaman 3y ago
    Spot and any queue
stickfigure 3y ago
I'm an enormous fan of Google Cloud Tasks (previously known as the Google App Engine task service). The low-level mechanism is "call this URL later" but with a tiny bit of application glue the abstraction is "run this code later".
- latchkey 3y ago
  I concur. Why isn't there an open source self hosted version of GCT?
jauntywundrkind 3y ago
Generally I'd suggest combining different messaging/streaming(ex: Kafka) & FaaS/serverless (ex: OpenFaaS, Fission, OpenWhisk) layers to achieve something like Async. Where-as Async seems to be a bit of both. I think over time we'll see streaming systems expand their footprint & keep eating away at the space pure FaaS occupies. Apache Pulsar has been around for a while as a streaming system, and it does have a fairly significant Pulsar Functions layer which might make it a decent all in one candidate. (Pulsar in general feels like a much smarter lower-latency take on Kafka, allowing much more control/flexibility of where data gets backed/replicated/read from that makes it exceedingly more useful for "log/stream is the database" styles.) https://pulsar.apache.org/docs/3.1.x/functions-overview/ https://pulsar.apache.org/docs/3.1.x/functions-overview/ There's so many up and coming streaming things here. Apache Ballista. Netflix's Mantis. RisingWave. A million years ago there was SOA; a startup I was at used Apache Camel. Benthos. Others. Many of these will decidedly remain more streaming oriented, not encroach on FaaS space, which is fine, but I think we'll see overlap grow. Reductionist, but a function call is "run this code when a function name and parameters happen" versus maybe "run this code when an event type happens", which seems not so far apart to me.

rangledangle 3y ago

Sounds like something you could do with SQS and Lambda. They just have massive datacenter infra and compute at disposal.

salil999 3y ago
Yeah but what about something that won't make my wallet cry?
- jjice 3y ago
  Pretty much any queue and a compute will do. SQS paired with lambda an event bridge (I forget if that's the service to trigger the lambda or not) would be solid. I lean towards compute that's a bit easier to run in different settings so just any old code checking the queue and executing as expected also works. SQS is cheap and it's pretty damn solid. I believe it was the first AWS service. But in reality, any managed queue or even a well structured DB table can be fine, depending on your scale.
  - cj 3y ago
    Is using Lambda with an external data store (like MongoDB) a solved problem? I remember this being an issue a long time ago, where Lambda pretty much was a non starter for use cases where you're running 1000 independent tasks that each, on their own, is a small unit of work (just 1 or 2 database queries). Long way of asking: is there a way to share a database connection between lambda functions or does each run need to re-establish / re-authenticate a new connection with the database?
    - josevalerio 3y ago
      Declare your connection outside of your handler function - this post is from 2017 but it shows it clearly: https://www.jeremydaly.com/reuse-database-connections-aws-lambda/ https://www.jeremydaly.com/reuse-database-connections-aws-la... It will get re-used across lambda invocations as long as the instance of it isn't killed which typically happens after a few minutes
    - alexbevi 3y ago
      For MongoDB specifically see the best practices docs for Lambda (https://www.mongodb.com/docs/atlas/manage-connections-aws-lambda/ https://www.mongodb.com/docs/atlas/manage-connections-aws-la...). This outlines creating the handler outside the function context to allow the client to be reused between invocations.
- gnaman 3y ago
  Spot and any queue

stickfigure 3y ago

I'm an enormous fan of Google Cloud Tasks (previously known as the Google App Engine task service). The low-level mechanism is "call this URL later" but with a tiny bit of application glue the abstraction is "run this code later".

latchkey 3y ago
I concur. Why isn't there an open source self hosted version of GCT?

jauntywundrkind 3y ago

Generally I'd suggest combining different messaging/streaming(ex: Kafka) & FaaS/serverless (ex: OpenFaaS, Fission, OpenWhisk) layers to achieve something like Async. Where-as Async seems to be a bit of both. I think over time we'll see streaming systems expand their footprint & keep eating away at the space pure FaaS occupies. Apache Pulsar has been around for a while as a streaming system, and it does have a fairly significant Pulsar Functions layer which might make it a decent all in one candidate. (Pulsar in general feels like a much smarter lower-latency take on Kafka, allowing much more control/flexibility of where data gets backed/replicated/read from that makes it exceedingly more useful for "log/stream is the database" styles.) https://pulsar.apache.org/docs/3.1.x/functions-overview/ https://pulsar.apache.org/docs/3.1.x/functions-overview/ There's so many up and coming streaming things here. Apache Ballista. Netflix's Mantis. RisingWave. A million years ago there was SOA; a startup I was at used Apache Camel. Benthos. Others. Many of these will decidedly remain more streaming oriented, not encroach on FaaS space, which is fine, but I think we'll see overlap grow. Reductionist, but a function call is "run this code when a function name and parameters happen" versus maybe "run this code when an event type happens", which seems not so far apart to me.