4 ms·

Using prime numbers to avoid cron task collision at sametime

2 points by akshayB 9y ago

I would like to get input from you all. We have legacy system (bit of a pain to maintain) where multiple corn jobs runs (8 to be precise) and engineers who created the system put everything to run at 5 minutes interval. Computationally task is heavy and often results into a crash or running out of memory but business owners are fine if things are delayed by like an 1 hour or so. My idea was to run on prime intervals instead of every 5 minutes. Couple of them have depend on output from others tasks but that should not be an issue. Job 1 - Every 5 minutes ( computational less intense ) Job 2 - Every 7 minutes Job 3 - Every 11 minutes Job 4 - Every 13 minutes Job 5 - Every 17 minutes ( computational more intense ) .... and so on This should put a significant dent in avoiding them running at same time and leaving some room for each task to finish itself. I was just wondering if there is any other alternative that I can explore here. Thanks for any suggestions in advance.

CarolineW 9y ago
If the owners don't care then just have */16 * * * * bin/RunJob_1 2-59/16 * * * * bin/RunJob_2 4-59/16 * * * * bin/RunJob_3 6-59/16 * * * * bin/RunJob_4 8-59/16 * * * * bin/RunJob_5 10-59/16 * * * * bin/RunJob_6 12-59/16 * * * * bin/RunJob_7 14-59/16 * * * * bin/RunJob_8 If that doesn't work for you then tell us why. As it is your problem is underspecified. Also, you say some are more compute intensive - have you actually measured the load and time-to-finish? Prime numbers and similar techniques can help, it just depends if you actually just want to kludge something together, or do it properly.
ultrablue 9y ago
Is your goal to avoid all collisions? If so your current scheme won't prevent that. For example, Job 1 and Job 2 will both run at 0035, 0405, 2025, etc. (assuming that both jobs start at 0000).
- CarolineW 9y ago
  That depends - cron time steps always restart from 0 in the hour, so what you say is valid, but the most obvious way of implementing what I think is being suggested doesn't have exactly that problem.
  - ultrablue 9y ago
    Ah, didn't know that, thanks!
- akshayB 9y ago
  yeah I am ultimately trying to avoid collision and because other route for code refactor is not an option
pwg 9y ago
> I was just wondering if there is any other alternative that I can explore here. Investigate task spooler: http://vicerveza.homeunix.net/~viric/soft/ts/ http://vicerveza.homeunix.net/~viric/soft/ts/ This will let you queue up the tasks, and each will run as soon as the last has finished, but you won't get a "thundering herd" where they all are trying to run at once.

2 points by akshayB 9y ago

CarolineW 9y ago

If the owners don't care then just have */16 * * * * bin/RunJob_1 2-59/16 * * * * bin/RunJob_2 4-59/16 * * * * bin/RunJob_3 6-59/16 * * * * bin/RunJob_4 8-59/16 * * * * bin/RunJob_5 10-59/16 * * * * bin/RunJob_6 12-59/16 * * * * bin/RunJob_7 14-59/16 * * * * bin/RunJob_8 If that doesn't work for you then tell us why. As it is your problem is underspecified. Also, you say some are more compute intensive - have you actually measured the load and time-to-finish? Prime numbers and similar techniques can help, it just depends if you actually just want to kludge something together, or do it properly.

ultrablue 9y ago

Is your goal to avoid all collisions? If so your current scheme won't prevent that. For example, Job 1 and Job 2 will both run at 0035, 0405, 2025, etc. (assuming that both jobs start at 0000).

CarolineW 9y ago
That depends - cron time steps always restart from 0 in the hour, so what you say is valid, but the most obvious way of implementing what I think is being suggested doesn't have exactly that problem.
- ultrablue 9y ago
  Ah, didn't know that, thanks!
akshayB 9y ago
yeah I am ultimately trying to avoid collision and because other route for code refactor is not an option

pwg 9y ago

> I was just wondering if there is any other alternative that I can explore here. Investigate task spooler: http://vicerveza.homeunix.net/~viric/soft/ts/ http://vicerveza.homeunix.net/~viric/soft/ts/ This will let you queue up the tasks, and each will run as soon as the last has finished, but you won't get a "thundering herd" where they all are trying to run at once.