3 ms·

I'm firmly in the "I prefer explicit lambda functions for off-request work" camp, with the recognition that you need a lot of operational and organizational mat

I'm firmly in the "I prefer explicit lambda functions for off-request work" camp, with the recognition that you need a lot of operational and organizational maturity to keep a fleet of functions maintainable. I get that isn't everyone's cup of tea or a good fit for every org. That said, I don't understand this bit: > Leaning on your worker queue purely for offloaded execution means writing all the glue code to get the data into and out of the job, and back to the caller or end-user’s device somehow I assumed by "worker queue" they were talking about something akin to Celery in python land, but it actually does handle all this glue. As far as I can tell, Celery provides a very similar developer experience to FLAME but has the added benefit that if you do want durability those knobs are there. The only real downside seems you need redis or rabbit to facilitate it? I don't have any experience with them but I'd assume it's the same story with other languages/frameworks (eg ruby+sidekiq)? Maybe I'm missing something.

jrmiii 3y ago
Yeah, I think this was more inward focusing on things like `Oban` in elixir land. He's made the distinction in the article that those tools are great when you need durability, but this gives you a lower ceremony way to make it Just Work™ when all you're after is passing off the work.
josevalim 3y ago
Wouldn’t you lose, for example, streaming capabilities once you use Celery? You would have to first upload the whole video, then enqueue the job, and then figure out a mechanism to send the thumbnails back to that client, while with FLAME you get a better user experience by streaming thumbnails as soon as the upload starts. I believe the main point though is that background workers and FLAME are orthogonal concepts. You can use FLAME for autoscaling, you can use Celery for durability, and you could use Celery with FLAME to autoscale your background workers based on queue size. So being able to use these components individually will enable different patterns and use cases.
- seabrookmx 3y ago
  Yes fair point. Celery has to pickle parameters so that they can flow through redis or rabbit to the worker pool. It's worth pointing out this transparent remote function call ability is unique to the BEAM. The FLAME pattern in other languages (as described in the article with Javascript) would also require serializable parameters. > I believe the main point though is that background workers and FLAME are orthogonal concepts Yeah I think this is what the author is driving at. I appreciate you helping me try and wrap my brain around it :)
Doxin 3y ago
Personally I feel more like this is pythons multiprocessing, with some spices added to start a server on demand. It's been a while since I last read the multiprocessing docs but last time I did it did a pretty poor job of showing all the fancy tricks multiprocessing supports, like running a function in a different interpreter or on a different server altogether.

jrmiii 3y ago

Yeah, I think this was more inward focusing on things like `Oban` in elixir land. He's made the distinction in the article that those tools are great when you need durability, but this gives you a lower ceremony way to make it Just Work™ when all you're after is passing off the work.

josevalim 3y ago

Wouldn’t you lose, for example, streaming capabilities once you use Celery? You would have to first upload the whole video, then enqueue the job, and then figure out a mechanism to send the thumbnails back to that client, while with FLAME you get a better user experience by streaming thumbnails as soon as the upload starts. I believe the main point though is that background workers and FLAME are orthogonal concepts. You can use FLAME for autoscaling, you can use Celery for durability, and you could use Celery with FLAME to autoscale your background workers based on queue size. So being able to use these components individually will enable different patterns and use cases.

seabrookmx 3y ago
Yes fair point. Celery has to pickle parameters so that they can flow through redis or rabbit to the worker pool. It's worth pointing out this transparent remote function call ability is unique to the BEAM. The FLAME pattern in other languages (as described in the article with Javascript) would also require serializable parameters. > I believe the main point though is that background workers and FLAME are orthogonal concepts Yeah I think this is what the author is driving at. I appreciate you helping me try and wrap my brain around it :)

Doxin 3y ago

Personally I feel more like this is pythons multiprocessing, with some spices added to start a server on demand. It's been a while since I last read the multiprocessing docs but last time I did it did a pretty poor job of showing all the fancy tricks multiprocessing supports, like running a function in a different interpreter or on a different server altogether.