3 ms·

I don't really understand this sentiment of not needing a queue system. This is not much different from spawning a thread in Java to delay the email sending. F

I don't really understand this sentiment of not needing a queue system. This is not much different from spawning a thread in Java to delay the email sending. For any serious application you want a job like that to be persisted so you can guarantee it runs even if your application is restarted. I know that Erlang/Elixir is designed for stateful applications and if you have a cluster and do hot deploys this is less of a problem, but who does that? Most Elixir applications I've seen are deployed as stateless systems just like any Ruby, Python, Node etc systems.

polmuz 6y ago
Having a queue means you have a distributed system. How do you handle network problems, errors/retries, back pressure? OTP has excellent idiomatic tools for all that and more.
- jeremyjh 6y ago
  Elixir developers who need a queue generally reach for Rabbit (which any language can use), or something backed in a database like rihanna[1] or honeydew[2]. Rolling your own distributed system is very much a last resort, and despite its excellent concurrency characteristics the BEAM still lacks basics such as a battle-tested raft implementation. [1]https://github.com/samsondav/rihanna https://github.com/samsondav/rihanna [2]https://github.com/koudelka/honeydew https://github.com/koudelka/honeydew
  - zambal 6y ago
    > and despite its excellent concurrency characteristics the BEAM still lacks basics such as a battle-tested raft implementation. A couple of years ago the RabbitMQ team has published a raft library[1] which they use in their implementation of persistent queues. It has a flexible API and implementing your own state machines is quite straightforward, as it follows the OTP gen_* behaviour paradigm. And by now, I'd say it's pretty well battle-tested. [1] https://github.com/rabbitmq/ra https://github.com/rabbitmq/ra
    - jeremyjh 6y ago
      I didn’t know about that - thank you for pointing me at it!
moreaccountspls 6y ago
Slightly offtopic, but consider a pattern of persisting the state machine of a given task instead of the task queue itself. For example, for a daily email job, you might have three states: UNSENT, BEGIN(begin_time), SENT. Then you can just have a supervisor job that scans your table or whatever and schedules jobs, retries, etc. as needed. Now you don't have to worry about the queue state at all!

polmuz 6y ago

Having a queue means you have a distributed system. How do you handle network problems, errors/retries, back pressure? OTP has excellent idiomatic tools for all that and more.

jeremyjh 6y ago
Elixir developers who need a queue generally reach for Rabbit (which any language can use), or something backed in a database like rihanna[1] or honeydew[2]. Rolling your own distributed system is very much a last resort, and despite its excellent concurrency characteristics the BEAM still lacks basics such as a battle-tested raft implementation. [1]https://github.com/samsondav/rihanna https://github.com/samsondav/rihanna [2]https://github.com/koudelka/honeydew https://github.com/koudelka/honeydew
- zambal 6y ago
  > and despite its excellent concurrency characteristics the BEAM still lacks basics such as a battle-tested raft implementation. A couple of years ago the RabbitMQ team has published a raft library[1] which they use in their implementation of persistent queues. It has a flexible API and implementing your own state machines is quite straightforward, as it follows the OTP gen_* behaviour paradigm. And by now, I'd say it's pretty well battle-tested. [1] https://github.com/rabbitmq/ra https://github.com/rabbitmq/ra
  - jeremyjh 6y ago
    I didn’t know about that - thank you for pointing me at it!

moreaccountspls 6y ago

Slightly offtopic, but consider a pattern of persisting the state machine of a given task instead of the task queue itself. For example, for a daily email job, you might have three states: UNSENT, BEGIN(begin_time), SENT. Then you can just have a supervisor job that scans your table or whatever and schedules jobs, retries, etc. as needed. Now you don't have to worry about the queue state at all!