3 ms·

>> spin up a bunch of worker processes as EC2 spot instances that check for the next PENDING task Does that mean workers poll the database?

nostrademons 7y ago
Yup, but they do so only once per shard (well, twice, once at the beginning and once at the end). If you've got a big job where each shard takes a few minutes to process and a hundred or so workers, the DB gets about 1 req/sec, which is well within the capabilities of Postgres.
- flavor8 7y ago
  We had an "interesting" bug where we inadvertently ended up polling postgres advisory locks around 6 million times per minute from a cluster of 200 servers. It added about 20% load to the database, enough to be anomalous but not enough to outright trip any alarms. The overall database performance was fine.
  - exikyut 7y ago
    (That's 100,000 req/sec) Was "the database" a single node, or multiple systems?
- nurettin 7y ago
  I would lean towards maintaining a simple list on redis and letting the workers pop the next job from it whenever they are done.
  - nostrademons 7y ago
    That works fine too. The advantage of Postgres is that the master is a single point of failure, and if Postgres goes down all the state is persistent and you can just bring it back up again. Also, ad-hoc query tools are a little better for Postgres, which gives you a really simple interface to inspect the status of the job and the partial results. This architecture originally evolved from an Amazon-SQS based queue system, where I'd stick the jobs in SQS and pop them off. Then I realized that I needed the DB anyway for my reducer-substitute, and if I'm writing SQL/ORM code anyway it was simpler to use the DB to manage the queue than to use SQS, and it gave me some other nifty abilities like being able to record timing & size stats with each shard, being able to query progress while the job is going, and being able to manually restart shards after correcting the errors that led to their failure. If you're running dozens-to-hundreds of workers, the cost of a t2 RDS instance (that's all you need for 1 QPS) is a rounding error, and a lot cheaper than EMR would be.
    - nurettin 7y ago
      To me, the message queue is simply a substitute for polling the DB, not a way of persisting message status. That is a job for the DB.
- wahern 7y ago
  Postgres provides LISTEN/NOTIFY for precisely this sort of use case. https://www.postgresql.org/docs/9.1/sql-notify.html https://www.postgresql.org/docs/9.1/sql-notify.html
  - seunosewa 7y ago
    Yes, any idea about why isn't it used in the original article above?
    - buckminster 7y ago
      It is used in the article.
- blondin 7y ago
  do you mean that there should be something like a controller that can spin up workers and also get rid of those that hanged (or retry) etc?

nostrademons 7y ago

Yup, but they do so only once per shard (well, twice, once at the beginning and once at the end). If you've got a big job where each shard takes a few minutes to process and a hundred or so workers, the DB gets about 1 req/sec, which is well within the capabilities of Postgres.

flavor8 7y ago
We had an "interesting" bug where we inadvertently ended up polling postgres advisory locks around 6 million times per minute from a cluster of 200 servers. It added about 20% load to the database, enough to be anomalous but not enough to outright trip any alarms. The overall database performance was fine.
- exikyut 7y ago
  (That's 100,000 req/sec) Was "the database" a single node, or multiple systems?
nurettin 7y ago
I would lean towards maintaining a simple list on redis and letting the workers pop the next job from it whenever they are done.
- nostrademons 7y ago
  That works fine too. The advantage of Postgres is that the master is a single point of failure, and if Postgres goes down all the state is persistent and you can just bring it back up again. Also, ad-hoc query tools are a little better for Postgres, which gives you a really simple interface to inspect the status of the job and the partial results. This architecture originally evolved from an Amazon-SQS based queue system, where I'd stick the jobs in SQS and pop them off. Then I realized that I needed the DB anyway for my reducer-substitute, and if I'm writing SQL/ORM code anyway it was simpler to use the DB to manage the queue than to use SQS, and it gave me some other nifty abilities like being able to record timing & size stats with each shard, being able to query progress while the job is going, and being able to manually restart shards after correcting the errors that led to their failure. If you're running dozens-to-hundreds of workers, the cost of a t2 RDS instance (that's all you need for 1 QPS) is a rounding error, and a lot cheaper than EMR would be.
  - nurettin 7y ago
    To me, the message queue is simply a substitute for polling the DB, not a way of persisting message status. That is a job for the DB.
wahern 7y ago
Postgres provides LISTEN/NOTIFY for precisely this sort of use case. https://www.postgresql.org/docs/9.1/sql-notify.html https://www.postgresql.org/docs/9.1/sql-notify.html
- seunosewa 7y ago
  Yes, any idea about why isn't it used in the original article above?
  - buckminster 7y ago
    It is used in the article.
blondin 7y ago
do you mean that there should be something like a controller that can spin up workers and also get rid of those that hanged (or retry) etc?