5 ms·

Can you shed a bit more light on this - > job scheduling systems that could be replaced by a postgres table I've never used Postgres in my life ever.

by envolt 7y ago

reledi 7y ago
What Jon Richards said. Also see https://github.com/collectiveidea/delayed_job https://github.com/collectiveidea/delayed_job which is a popular DB-based scheduling library for Ruby.
jon_richards 7y ago
I think the point is to use a DB as a message broker. Schedule a job by adding a row to a table. Have services running that secure a lock on a row, run the task, and then update/remove the job from the DB. Django-Q (the multiprocessing task queue framework, annoyingly named the same as Django Q objects), allows this as one of the configurable brokers. https://django-q.readthedocs.io/en/latest/brokers.html#django-orm https://django-q.readthedocs.io/en/latest/brokers.html#djang...
- liampronan 7y ago
  +1. And I think specifically the SKIP LOCKED Postgres feature is relevant for this -https://tnishimura.github.io/articles/queues-in-postgresql/ https://tnishimura.github.io/articles/queues-in-postgresql/
  - jon_richards 7y ago
    Oh, I hadn't noticed that. I'm not sure how Django Q manages with a non-postgres DB, since it doesn't mention that as a requirement. I have used nowait before to batch scheduled jobs together for a slow API call.
    - orf 7y ago
      Why would you pick django-q over Celery, which also supports a database backend?
      - michaelcampbell 7y ago
        We've had a lot of mysterious "it just stops working" issues with Celery, and their # of open tickets is approaching scary levels. Moving everything to rq, and so far so good.
        orf 7y ago
        We’ve had exactly the opposite experience. RQ was an absolute nightmare in every aspect across two companies I’ve worked at. And when you need something that Celery supports that RQ does not, you are in a pickle. I wish you the best, but I cannot help but think you will regret that decision.
        orf 7y ago
        Also, Kiwi.com gave a talk[1] about their experiences with RQ and how/why they migrated to Celery at EuroPython last week. It's worth a watch. 1. https://youtu.be/SUQYuA9AJT0?t=7546 https://youtu.be/SUQYuA9AJT0?t=7546
        o-__-o 7y ago
        Have never experienced this and we sold a huge number of our app using celery. Based on other replies one should pause, take a step back and consider if their code or design is to blame..
        withinboredom 7y ago
        Are you sure that changing your implementation didn't fix a bug? I've noticed that a lot of "dependency" problems on "complex systems" are usually programmer implementation problems. Changing the implementation forces you to review the original implementation (or throw it out) and fix the bug that caused you to change the implementation in the first place.
- envolt 7y ago
  As long as I want to avoid very edgy race condition cases (low throughput), any DB will work right? Either Mongo, MySQL or List in Redis. Is there anything sepcific about Postgres when it comes to this? (except SKIP TABLES, as mentioned in other comments)
  - CoolGuySteve 7y ago
    I said postgres because it's just all around good at a bunch of stuff. But I've used redis lists in the past to great success. Redis streams might be even better but they weren't ready the last time I needed a queue.
- michaelcampbell 7y ago
  rq ("redis queue") for python does this with, obviously, redis. API is pretty small and comprehensible.
- lrem 7y ago
  Funnily enough, we also use this approach in Google. There's at least once chapter in the first (the unactionable) SRE book about this.
KaiserPro 7y ago
It can't. Not easily. Job scheduling is trivial. Doing it reliably, tracably and in a way that handled failure predictably is really a pain in the arse. There are zillions of ETL frameworks out there, just use those. some are unspeakably complex (airflow I'm looking at you) Some are simple, but require glue (AWS batch) some are trendy (argo on kubernetes) However, most CI tools are effectively job scheduling systems. Jenkins works surprisingly well as a cron replacement for >100 machines

Can you shed a bit more light on this - > job scheduling systems that could be replaced by a postgres table I've never used Postgres in my life ever.

by envolt 7y ago

Can you shed a bit more light on this - > job scheduling systems that could be replaced by a postgres table I've never used Postgres in my life ever.

reledi 7y ago

What Jon Richards said. Also see https://github.com/collectiveidea/delayed_job https://github.com/collectiveidea/delayed_job which is a popular DB-based scheduling library for Ruby.

jon_richards 7y ago

I think the point is to use a DB as a message broker. Schedule a job by adding a row to a table. Have services running that secure a lock on a row, run the task, and then update/remove the job from the DB. Django-Q (the multiprocessing task queue framework, annoyingly named the same as Django Q objects), allows this as one of the configurable brokers. https://django-q.readthedocs.io/en/latest/brokers.html#django-orm https://django-q.readthedocs.io/en/latest/brokers.html#djang...

liampronan 7y ago
+1. And I think specifically the SKIP LOCKED Postgres feature is relevant for this -https://tnishimura.github.io/articles/queues-in-postgresql/ https://tnishimura.github.io/articles/queues-in-postgresql/
- jon_richards 7y ago
  Oh, I hadn't noticed that. I'm not sure how Django Q manages with a non-postgres DB, since it doesn't mention that as a requirement. I have used nowait before to batch scheduled jobs together for a slow API call.
  - orf 7y ago
    Why would you pick django-q over Celery, which also supports a database backend?
    - michaelcampbell 7y ago
      We've had a lot of mysterious "it just stops working" issues with Celery, and their # of open tickets is approaching scary levels. Moving everything to rq, and so far so good.
      - orf 7y ago
        We’ve had exactly the opposite experience. RQ was an absolute nightmare in every aspect across two companies I’ve worked at. And when you need something that Celery supports that RQ does not, you are in a pickle. I wish you the best, but I cannot help but think you will regret that decision.
      - orf 7y ago
        Also, Kiwi.com gave a talk[1] about their experiences with RQ and how/why they migrated to Celery at EuroPython last week. It's worth a watch. 1. https://youtu.be/SUQYuA9AJT0?t=7546 https://youtu.be/SUQYuA9AJT0?t=7546
      - o-__-o 7y ago
        Have never experienced this and we sold a huge number of our app using celery. Based on other replies one should pause, take a step back and consider if their code or design is to blame..
      - withinboredom 7y ago
        Are you sure that changing your implementation didn't fix a bug? I've noticed that a lot of "dependency" problems on "complex systems" are usually programmer implementation problems. Changing the implementation forces you to review the original implementation (or throw it out) and fix the bug that caused you to change the implementation in the first place.
envolt 7y ago
As long as I want to avoid very edgy race condition cases (low throughput), any DB will work right? Either Mongo, MySQL or List in Redis. Is there anything sepcific about Postgres when it comes to this? (except SKIP TABLES, as mentioned in other comments)
- CoolGuySteve 7y ago
  I said postgres because it's just all around good at a bunch of stuff. But I've used redis lists in the past to great success. Redis streams might be even better but they weren't ready the last time I needed a queue.
michaelcampbell 7y ago
rq ("redis queue") for python does this with, obviously, redis. API is pretty small and comprehensible.
lrem 7y ago
Funnily enough, we also use this approach in Google. There's at least once chapter in the first (the unactionable) SRE book about this.

KaiserPro 7y ago

It can't. Not easily. Job scheduling is trivial. Doing it reliably, tracably and in a way that handled failure predictably is really a pain in the arse. There are zillions of ETL frameworks out there, just use those. some are unspeakably complex (airflow I'm looking at you) Some are simple, but require glue (AWS batch) some are trendy (argo on kubernetes) However, most CI tools are effectively job scheduling systems. Jenkins works surprisingly well as a cron replacement for >100 machines