4 ms·

that's a nice read, but does it also apply to MySQL (InnoDB)?

• hparadiz 2y ago
  I've done this with mysql. Never do one at a time if you have jobs per minute over 30. It won't scale. Instead have the job dispatcher reserve 100 at a time and then fire that off to a subprocess which will subsequently fire off a process for each job. A three layer approach makes it much easier to build out multiserver. Or if you don't want the headache just use SQS which is pretty much free under 1 million jobs.
  • est 2y ago
    Yeah it's very basic and limited. However if I am about to use DB as a job queue for budget reasons, I'd make sure the job doesn't get too complicated.
    • hparadiz 2y ago
      For me it was a lot of small jobs. I was able to get it up to 3500 jobs an hour and likely could have gone far past that but the load on the MySQL server was not reasonable
      • jeeybee 2y ago
        I was able to push PqQueuer to 25k jobs a second in my benchmarking script.