6 ms·

Given how taxing the "thundering herd" effect can be on mirrors, websites (RSS readers!), you'd think this sort of thing should've been in cron since at least m

Given how taxing the "thundering herd" effect can be on mirrors, websites (RSS readers!), you'd think this sort of thing should've been in cron since at least mid-90s. Once again, OpenBSD with the simple, obvious solution, that everyone else kinda overlooked. I hope every other cron out there copies and ships this as soon as possible.

dralley 3y ago
>OpenBSD with the simple, obvious solution, that everyone else kinda overlooked Systemd Timers have had this for a while.
- usr1106 3y ago
  RandomizedDelaySec=
- _2z1p 3y ago
  They said simple and obvious, which much of systemd is not. Personally I find the OpenBSD solution far more elegant and UNIXy than the systemd one, but to each their own.
  - ijustlovemath 3y ago
    What's more simple than an ini file?
    - rodgerd 3y ago
      Reinventing the same features many times over in many different places, of course.
  - ratorx 3y ago
    Why does the complexity of the rest of systemd matter when comparing the implementation of this feature? What would you say is wrong with RandomizedDelaySec?
    - mekster 3y ago
      UI is too complicated to type some word concatenated keyword than being visually intuitive.
      - vbezhenar 3y ago
        On the other hand I wouldn't be able to understand what the hell is this cron string. I actually have no idea about cron format despite the fact that I used it multiple times. I have to read man every time I use it. Also different software implements it differently. [Timer] OnCalendar=daily RandomizedDelaySec=12h might take few more seconds to type, but it's definitely readable without any additional documentation.
        panzi 3y ago
        The option says Sec but the value says h? What does that mean?
        rjh29 3y ago
        Sec is a standard suffix for time values, anything ending with Sec accepts a value in seconds. 12h is shorthand for 'the number of seconds in 12 hours'
        Gud 3y ago
        Standard where? I have never seen this before and I’ve been reading *nix configuration files for 20+ years
        rjh29 3y ago
        Standard within systemd (i.e. consistent).
        gkbrk 3y ago
        It's standard to say 12h and expect this to read as 43200? Sounds like a lousy standard if the correct way to use it is to say "I want to delay by 12 hour seconds". What is even "12 hour seconds"?
- trasz4 3y ago
  Yeah, but with systemd timers it's just another ad hoc hack.
  - djbusby 3y ago
    I'm not hot for systemd but this https://www.freedesktop.org/software/systemd/man/systemd.timer.html https://www.freedesktop.org/software/systemd/man/systemd.tim... looks robust, far from ad-hoc hack.
    - trasz4 3y ago
      How does it look robust? Because someone wrote a man page and there's lots of boilerplate in it? Proper design reduces complexity. The above adds a lot of complexity, and thus it's a hack. Not the appearance.
      - 0dayz 3y ago
        That's not a hack though, a hack is gluing things together to fix a certain specific bug that is not easily solved because the bug is related to the design instead of a mere mistake. Otherwise with your standard of hack anything beyond hello world and baby's first input are hacks because everything else requires boilerplate.
        trasz4 3y ago
        It's not a hack because of the boilerplate - it's a hack because it's functionality implemented in a wrong place. I think the boilerplate made you believe it's not a hack by making it look professional(ish), ie someone spend time putting a lot of lipstick on that pig, man page and all.
        dralley 3y ago
        How is it functionality implemented in the wrong place?
- msla 3y ago
  But, see, a number of people hate systemd, so that Doesn't Count. Even if their cron does...
- nazgulsenpai 3y ago
  Even when discussing OpenBSD somehow we end up debating systemd...
toast0 3y ago
freebsd cron(8) has -j for the daemon to add a random sleep of up to 60 seconds on each task. This was added in FreeBSD 5.3, committed 19 years ago. (FreeBSD 5.3 released November 6, 2004) https://github.com/freebsd/freebsd-src/commit/f5896baf9c429cf0c2990456bb7f8fb155d8a223 https://github.com/freebsd/freebsd-src/commit/f5896baf9c429c...
- chaps 3y ago
  A random sleep of up to 60 seconds doesn't really solve the problem the OpenBSD changes do, especially when your jobs take longer than 60s. For example, instead of "0-59/10" in the minutes field, "0~59/10" can be used to run a command every 10 minutes where the first command starts at a random offset in the range [0,9]. The high and low numbers are optional, "~/10" can be used instead.
  - cyberpunk 3y ago
    A random sleep of up to 10 seconds turned a 150gbit/sec spike into a 5gbit/sec spike on the Akamai bill from a newspaper app I once worked on...
    - mlyle 3y ago
      That's surprising. You'd think spreading the workload start over 10 seconds would lower the size of spikes (integrated over a second) by at most a factor of 10. But the above point is still true: many jobs take a few minutes to run. 60s of dispersion in start time is better than nothing, but you really want more. (In this case, things are still quantized to a minute boundary, so you'd really want both).
      - dspillett 3y ago
        > That's surprising. You'd think spreading the workload start over 10 seconds would lower the size of spikes (integrated over a second) by at most a factor of 10. If the delay is at the reading side, away from Akami, through a cache, perhaps 10 concurrent requests for X would result in ten lots of data transfer as it isn't in cache yet, but 10 with a short delay is enough to prime the local cache on the first request before the rest start. There are a number of reasons a sudden glut of activity could balloon bandwidth or CPU/memory costs more than you might expect. Without a chunk more detail about the system in question, this is just random speculation of course.
cratermoon 3y ago
I believe the philosophy was to leave it up to the service to behave sensibly, including things like have a circuit breaker, use some kind of backoff/retry, and generally be robust in the face of resource contention. It kind of feels like this is putting the policy of "don't all go at once" into the cron mechanism, which is just starting jobs at desired times.
throw7 3y ago
I first ran into randomization delays in cronie (the ~ is how it's implemented also and there's a RANDOM_DELAY variable for use too), after redhat had switched to it at some point years ago. Personally, never really used it, but it's nice it's there.
- rollcat 3y ago
  Intriguing! I had no idea! Again, I wish the concept was more popular back when cron and the Internet were younger, and having it built-in and readily available goes a very long way. If someone doesn't introduce you to the concept, you have little chance of knowing better until you find yourself at the receiving end of a spike, yelling at the clouds.
nvy 3y ago
It's a neat idea to be sure but I fail to see how this will have any material impact. Firstly, OpenBSD is a niche OS, meaning the absolute magnitude of OpenBSD cron jobs out "in the wild" is relatively low. Second, my understanding is that this is a client-side feature. I.e. if I run a service, this feature only benefits me if a significant portion of my users opt into it. Third, I have an unsubstantiated suspicion that cron usage relative to systemd usage is also on the decline.
LeoPanthera 3y ago
The lazy solution is to put something like: sleep $((RANDOM%=60)) ...or longer, at the top of the script your cron is running.

dralley 3y ago

>OpenBSD with the simple, obvious solution, that everyone else kinda overlooked Systemd Timers have had this for a while.

usr1106 3y ago
RandomizedDelaySec=
_2z1p 3y ago
They said simple and obvious, which much of systemd is not. Personally I find the OpenBSD solution far more elegant and UNIXy than the systemd one, but to each their own.
- ijustlovemath 3y ago
  What's more simple than an ini file?
  - rodgerd 3y ago
    Reinventing the same features many times over in many different places, of course.
- ratorx 3y ago
  Why does the complexity of the rest of systemd matter when comparing the implementation of this feature? What would you say is wrong with RandomizedDelaySec?
  - mekster 3y ago
    UI is too complicated to type some word concatenated keyword than being visually intuitive.
    - vbezhenar 3y ago
      On the other hand I wouldn't be able to understand what the hell is this cron string. I actually have no idea about cron format despite the fact that I used it multiple times. I have to read man every time I use it. Also different software implements it differently. [Timer] OnCalendar=daily RandomizedDelaySec=12h might take few more seconds to type, but it's definitely readable without any additional documentation.
      - panzi 3y ago
        The option says Sec but the value says h? What does that mean?
        rjh29 3y ago
        Sec is a standard suffix for time values, anything ending with Sec accepts a value in seconds. 12h is shorthand for 'the number of seconds in 12 hours'
        Gud 3y ago
        Standard where? I have never seen this before and I’ve been reading *nix configuration files for 20+ years
        rjh29 3y ago
        Standard within systemd (i.e. consistent).
        gkbrk 3y ago
        It's standard to say 12h and expect this to read as 43200? Sounds like a lousy standard if the correct way to use it is to say "I want to delay by 12 hour seconds". What is even "12 hour seconds"?
trasz4 3y ago
Yeah, but with systemd timers it's just another ad hoc hack.
- djbusby 3y ago
  I'm not hot for systemd but this https://www.freedesktop.org/software/systemd/man/systemd.timer.html https://www.freedesktop.org/software/systemd/man/systemd.tim... looks robust, far from ad-hoc hack.
  - trasz4 3y ago
    How does it look robust? Because someone wrote a man page and there's lots of boilerplate in it? Proper design reduces complexity. The above adds a lot of complexity, and thus it's a hack. Not the appearance.
    - 0dayz 3y ago
      That's not a hack though, a hack is gluing things together to fix a certain specific bug that is not easily solved because the bug is related to the design instead of a mere mistake. Otherwise with your standard of hack anything beyond hello world and baby's first input are hacks because everything else requires boilerplate.
      - trasz4 3y ago
        It's not a hack because of the boilerplate - it's a hack because it's functionality implemented in a wrong place. I think the boilerplate made you believe it's not a hack by making it look professional(ish), ie someone spend time putting a lot of lipstick on that pig, man page and all.
        dralley 3y ago
        How is it functionality implemented in the wrong place?
msla 3y ago
But, see, a number of people hate systemd, so that Doesn't Count. Even if their cron does...
nazgulsenpai 3y ago
Even when discussing OpenBSD somehow we end up debating systemd...

toast0 3y ago

freebsd cron(8) has -j for the daemon to add a random sleep of up to 60 seconds on each task. This was added in FreeBSD 5.3, committed 19 years ago. (FreeBSD 5.3 released November 6, 2004) https://github.com/freebsd/freebsd-src/commit/f5896baf9c429cf0c2990456bb7f8fb155d8a223 https://github.com/freebsd/freebsd-src/commit/f5896baf9c429c...

chaps 3y ago
A random sleep of up to 60 seconds doesn't really solve the problem the OpenBSD changes do, especially when your jobs take longer than 60s. For example, instead of "0-59/10" in the minutes field, "0~59/10" can be used to run a command every 10 minutes where the first command starts at a random offset in the range [0,9]. The high and low numbers are optional, "~/10" can be used instead.
- cyberpunk 3y ago
  A random sleep of up to 10 seconds turned a 150gbit/sec spike into a 5gbit/sec spike on the Akamai bill from a newspaper app I once worked on...
  - mlyle 3y ago
    That's surprising. You'd think spreading the workload start over 10 seconds would lower the size of spikes (integrated over a second) by at most a factor of 10. But the above point is still true: many jobs take a few minutes to run. 60s of dispersion in start time is better than nothing, but you really want more. (In this case, things are still quantized to a minute boundary, so you'd really want both).
    - dspillett 3y ago
      > That's surprising. You'd think spreading the workload start over 10 seconds would lower the size of spikes (integrated over a second) by at most a factor of 10. If the delay is at the reading side, away from Akami, through a cache, perhaps 10 concurrent requests for X would result in ten lots of data transfer as it isn't in cache yet, but 10 with a short delay is enough to prime the local cache on the first request before the rest start. There are a number of reasons a sudden glut of activity could balloon bandwidth or CPU/memory costs more than you might expect. Without a chunk more detail about the system in question, this is just random speculation of course.

cratermoon 3y ago

I believe the philosophy was to leave it up to the service to behave sensibly, including things like have a circuit breaker, use some kind of backoff/retry, and generally be robust in the face of resource contention. It kind of feels like this is putting the policy of "don't all go at once" into the cron mechanism, which is just starting jobs at desired times.

throw7 3y ago

I first ran into randomization delays in cronie (the ~ is how it's implemented also and there's a RANDOM_DELAY variable for use too), after redhat had switched to it at some point years ago. Personally, never really used it, but it's nice it's there.

rollcat 3y ago
Intriguing! I had no idea! Again, I wish the concept was more popular back when cron and the Internet were younger, and having it built-in and readily available goes a very long way. If someone doesn't introduce you to the concept, you have little chance of knowing better until you find yourself at the receiving end of a spike, yelling at the clouds.

nvy 3y ago

It's a neat idea to be sure but I fail to see how this will have any material impact. Firstly, OpenBSD is a niche OS, meaning the absolute magnitude of OpenBSD cron jobs out "in the wild" is relatively low. Second, my understanding is that this is a client-side feature. I.e. if I run a service, this feature only benefits me if a significant portion of my users opt into it. Third, I have an unsubstantiated suspicion that cron usage relative to systemd usage is also on the decline.

LeoPanthera 3y ago

The lazy solution is to put something like: sleep $((RANDOM%=60)) ...or longer, at the top of the script your cron is running.