2 ms·

I have a few questions after reading. Mostly I'd like to know how they built their dynamic queueing system. How do they signal the workers to refresh their dyn

I have a few questions after reading. Mostly I'd like to know how they built their dynamic queueing system. How do they signal the workers to refresh their dynamic queue lists? Are they using a homebuilt queueing system or piggybacking on top of something like celery? What is the underlying message bus and how is it deployed? Would love to know more. Also kudos to them for stressing idempotency in a message system. It's usually much easier to ensure idempotency than to ensure exactly once delivery.

amz3 8y ago
> How do they signal the workers to refresh their dynamic queue lists? > Are they using a homebuilt queueing system or piggybacking on top of something like celery? They prolly do. Other comments mention erlang. I don't know if braze use Erlang. After reading the dissertation about it http://erlang.org/download/armstrong_thesis_2003.pdf http://erlang.org/download/armstrong_thesis_2003.pdf and my xp with Celery, it is clear that they have their own queueing system, maybe based on erlang. My understanding is that in erlang systems workers have much more knowledge than Celery workers about what happens in the whole system. Celery workers are very simple: they poll for tasks in a single statically defined queue. There is no sens of priority, it's a FIFO queue. You emulate priorities with several queues each of which have a number of worker proportional to their priority. That's not that simple because it depends on the kind of tasks they must execute (e.g. how many times it takes to execute, kind of resources required by the task (RAM, CPU, IO, GPU)). At the end of the day, I am not happy with Celery + RabbitMQ. I am also looking for dynamic queueing systems [0]. [0] https://github.com/celery/celery/issues/4901 https://github.com/celery/celery/issues/4901

2 ms·

I have a few questions after reading. Mostly I'd like to know how they built their dynamic queueing system. How do they signal the workers to refresh their dyn

by CorvusCrypto 8y ago

amz3 8y ago
> How do they signal the workers to refresh their dynamic queue lists? > Are they using a homebuilt queueing system or piggybacking on top of something like celery? They prolly do. Other comments mention erlang. I don't know if braze use Erlang. After reading the dissertation about it http://erlang.org/download/armstrong_thesis_2003.pdf http://erlang.org/download/armstrong_thesis_2003.pdf and my xp with Celery, it is clear that they have their own queueing system, maybe based on erlang. My understanding is that in erlang systems workers have much more knowledge than Celery workers about what happens in the whole system. Celery workers are very simple: they poll for tasks in a single statically defined queue. There is no sens of priority, it's a FIFO queue. You emulate priorities with several queues each of which have a number of worker proportional to their priority. That's not that simple because it depends on the kind of tasks they must execute (e.g. how many times it takes to execute, kind of resources required by the task (RAM, CPU, IO, GPU)). At the end of the day, I am not happy with Celery + RabbitMQ. I am also looking for dynamic queueing systems [0]. [0] https://github.com/celery/celery/issues/4901 https://github.com/celery/celery/issues/4901