if you need to do some batching operations such as read from a queue, do some processing and publish to s3, I can recommend Broadway https://github.com/dashbitc

if you need to do some batching operations such as read from a queue, do some processing and publish to s3, I can recommend Broadway https://github.com/dashbitco/broadway https://github.com/dashbitco/broadway I have seen lots of codebases in lots of languages do this type of task, but aside from maybe spark on the high end, i haven't see it done better. The beauty of erlang is your code reads like synchronous code so it's easy to read/maintain, but it has all the power of the parallelism of async code.