3 ms·
I would lean towards maintaining a simple list on redis and letting the workers pop the next job from it whenever they are done.
by nurettin 7y ago
I would lean towards maintaining a simple list on redis and letting the workers pop the next job from it whenever they are done.
- nostrademons 7y agoThat works fine too. The advantage of Postgres is that the master is a single point of failure, and if Postgres goes down all the state is persistent and you can just bring it back up again. Also, ad-hoc query tools are a little better for Postgres, which gives you a really simple interface to inspect the status of the job and the partial results. This architecture originally evolved from an Amazon-SQS based queue system, where I'd stick the jobs in SQS and pop them off. Then I realized that I needed the DB anyway for my reducer-substitute, and if I'm writing SQL/ORM code anyway it was simpler to use the DB to manage the queue than to use SQS, and it gave me some other nifty abilities like being able to record timing & size stats with each shard, being able to query progress while the job is going, and being able to manually restart shards after correcting the errors that led to their failure. If you're running dozens-to-hundreds of workers, the cost of a t2 RDS instance (that's all you need for 1 QPS) is a rounding error, and a lot cheaper than EMR would be.
- nurettin 7y agoTo me, the message queue is simply a substitute for polling the DB, not a way of persisting message status. That is a job for the DB.