13 ms·

Avoid 2:00 and 3:00 am cron jobs (2013)

341 points by pera 1y ago

deleted 1y ago
[deleted]
noir_lord 1y ago
I worked at a place that had set the servers to BST not UTC and used cron jobs extensively for reporting, which caused chaos twice a year and twice a year they had to explain to the business again why it caused chaos and ask for the time to fix it. Company went bust before it got fixed. Just use UTC folks unless you have a really good idea why you shouldn't.
- lxgr 1y ago
  "Our customers aren't in UTC" is often a compelling reason. People generally expect things like usage limits to reset overnight, not at UTC midnight, and these are often implicitly tied to (the result of) batch jobs. Financial jobs, including billing, is another big category.
  - latchkey 1y ago
    Always set the servers to UTC. Convert the time to local time for display, but store in UTC. The reporting job should be run from a server on UTC and on UTC time. The report itself can convert to whatever is local time.
    - gruez 1y ago
      That doesn't fix the problem of "my quota reset (or reporting job) gets shifted by 1 hr when daylight saving time changes".
      - latchkey 1y ago
        The reporting job should be run from a server on UTC and on UTC time. The report itself can convert to whatever is local time. This is exactly what I said above.
        lxgr 1y ago
        That works if all your consumers of the report are indifferent to it arriving at a different time half of the year. All of yours might well be, but you can't generalize that assumption to every domain and scenario.
        ipaddr 1y ago
        If you want an email of the report to send out at 9am locally. Generate before and send to everyone based on timezone.
        kjehjrktlhl 1y ago
        You don’t want a report to include 25h or 23h twice a year just to be able to send it at some distinct local time.
        lxgr 1y ago
        I absolutely want some daily business reports to cover 25 hours on one day of the year and 23 on another, since that's how long a calendar day is in local time in every place that has DST shifts.
        lenzm 1y ago
        Yes, that's exactly what I want. Everything shifts with dst - business processes, people's working hours, ach windows, ...
        gruez 1y ago
        > Generate before and send to everyone based on timezone. How are you going to do this? Clearly cron (with timezone set to UTC) doesn't work for this so you'll need another system. In that case why not have that multi-time zone scheduling system run your script?
      - ranger_danger 1y ago
        You could have the reset script check the timezone of the user before actually performing the reset, and have this script run at least once an hour to make sure you get close to accurate for most people (some timezones only differ by less than an hour though!). But also, if the user can change the timezone themselves, now you have a condition where the user could keep their quota at bay for a very long time (if not forever) if they keep changing their timezone accordingly.
      - throwaway150 1y ago
        > That doesn't fix the problem of "my quota reset (or reporting job) gets shifted by 1 hr when daylight saving time changes". How hard is to convert the UTC time to user's local time and perform quota reset? I mean you should be doing this anyway. Even if you store everything in PST or EST or CST, you never know when you'll get customers from another part of the world and then you'll have to do this anyway. So why not do this already?
        lxgr 1y ago
        > How hard is to convert the UTC time to user's local time and perform quota reset? Probably roughly as hard as writing a timezone-aware scheduler that considers all edge cases around daylight savings time, i.e. probably possible but I'd try fairly hard to see if I can get around it. One way of getting around it is to run your batch jobs in your "primary business timezone". Sure, you might outgrow that concept eventually, but many companies never do, and in that sense "running on UTC" seems similarly aspirational in spirit (albeit at a much smaller scale) to starting with high scalability.
        gruez 1y ago
        >Even if you store everything in PST or EST or CST, you never know when you'll get customers from another part of the world and then you'll have to do this anyway. So why not do this already? 1. allowing users to change their timezone and tying it to when their quota reset sounds like a great way to add more edge cases and complexity. For instance, does messing with your time zone mean you can get your quota reset multiple times a day? 2. Not everyone operates some sort of global b2c SaaS that's timezone agnostic. For many enterprise backoffice tasks "6am HQ time" is far more reasonable than "6am/7am, dunno depends on whether daylight saving time is on or not". In this case having a server set to the HQ's timezone makes far more sense than doing UTC and trying to work around daylight saving issues.
      - cweagans 1y ago
        Do you actually care if the requests were made on some specific calendar day? Or do you just want to make sure that heavy users are paying more and/or people aren't abusing your system? Instead of tying your quotas to calendar days in some specific time zone, tie them to a rolling 24 hour usage period. Even better: use a rolling hour.
  - freedomben 1y ago
    Having tried many times to use local time instead of UTC to make life simpler for myself, I really don't think it's ever a good idea. Better to localize times for users on the client-side based on their local settings, IMHO For quotas and usage limit resets it can be a little harder, but if it's a cron job anyway, my initial approach would be to just schedule it for a time that is close to midnight for the area your customers are in.
  - Kwpolska 1y ago
    That does not require setting the server to local time. Run the cron job every hour. Before starting the script, check the time and compare it to the last saved execution time. If the day has changed, do stuff and save the new date, else exit. This also happens to be more resilient to server downtime around midnight.
    - lxgr 1y ago
      > Before starting the script, check the time and compare it to the last saved execution time. If the day has changed, do stuff and save the new date, else exit. That sounds a bit like reimplementing a scheduler inside the scheduled task (which is ok if you don't trust your scheduler, but I'd avoid it if at all possible).
    - bell-cot 1y ago
      Except downtime can start during execution. So for many types of jobs... - Check if run today, exiting on "yes" - Run - Update the last-run-on date
      - latchkey 1y ago
        What is "today"?
        bell-cot 1y ago
        Hopefully just a datetime recent enough that the job does not currently need to be (re-)run. But as soon as one job needs another to complete first, or two jobs shouldn't be run at the same time - things start getting messy.
        lxgr 1y ago
        > Hopefully just a datetime recent enough that the job does not currently need to be (re-)run. I can think of several ways to mess that seemingly trivial calculation up just from the top of my head. (Consider changing timezones, increasing the interval the job is running to within half a day of scheduling period etc.)
  - skissane 1y ago
    > "Our customers aren't in UTC" is often a compelling reason. This doesn’t work as soon as you become a global business… because if you tell a customer in Asia that it resets at midnight in some US timezone, what are they going to think? Many people will accept UTC because it is the international standard. Everyone will accept “let each customer pick the timezone that works best for them”. Saying “I’m going to use a US timezone because that is easiest for our US customers” risks sending a message to your (now or future) global customers that you view them as second-class citizens.
    - lxgr 1y ago
      Really depends on your market and customers. Especially in the financial industry, local time zones (often of a given currency's central bank's main branch, a primary regulator etc.) are often very, very entrenched. > Many people will accept UTC because it is the international standard. I really wish that were true universally... And I do think that UTC is actually somewhat Eurocentric! From personal experience, it's significantly easier to mentally add/subtract one or two hours (modulo 24) than 7 or more. > Saying “I’m going to use a US timezone because that is easiest for our US customers” risks sending a message to your (now or future) global customers that you view them as second-class citizens. Definitely, but on the other hand, I do think that picking your head office's time zone as the "canonical" one for resetting quotas etc. has some merit as well, if only because it makes the concepts of "today" and "tomorrow" work a bit better. (UTC midnight might be very unintuitive to both you and your customers.)
      - skissane 1y ago
        > Definitely, but on the other hand, I do think that picking your head office's time zone as the "canonical" What happens when you move your head office to a different time zone? e.g. Oracle moved their HQ from California to Texas (and have announced a further move to Tennessee, although last I heard that still hasn’t happened) What happens when an M&A happens and you are now part of a much larger enterprise with a different HQ timezone, and systems with wildly inconsistent configured timezones due to having acquired multiple companies which had HQs in different time zones and all of which configured everything to use their HQ time? > UTC midnight might be very unintuitive to both you and your customers I live in eastern Australia, UTC+11 at the moment (UTC+10 during Southern Hemisphere winter). Right now, resetting stuff at UTC midnight means at 11am my time, resetting stuff at US Eastern midnight means at 3pm my time - roughly equal in potential inconvenience, but I’ll be much more forgiving of the first than the second. > And I do think that UTC is actually somewhat Eurocentric! From personal experience, it's significantly easier to mentally add/subtract one or two hours (modulo 24) than 7 or more. For me, add/subtract 10 is pretty easy… and then just one more during DST. Does that make UTC “Australia-centric”? Or Guam-centric or Vladivostok-centric? (both of which are also UTC+10)
- latchkey 1y ago
  At Marin Software, they ran each customer on their own set of servers and set the servers to the customers defined TZ. It was an endless cronjob nightmare. Now they are also bankrupt. Just found out some guy decided to try to restart the company. Good luck. https://x.com/Austen/status/1981904435539280324 https://x.com/Austen/status/1981904435539280324
- ta1243 1y ago
  If I want a report of what happened at a specific time I need that report at local time I get a daily status report of various things from our 24 hour operational management team which runs at 8am UK time every day. That means last week it ran at 0700UTC and this week at 0800UTC This is built around operational events, shift changes, etc. I've got another system which is in operation in Sydney from 0630 to 1630 local time, this means that maintence windows which overlap with UK shift patterns depend on the week but mean the system is operating 2130-0730 UK time at some times, 2030-0630 UK at others, and 1930-0530 at others. UTC is not "the answer". Sometimes you want things running at a UTC time, sometimes you want them running at local time. I have a regular meeting at 10am London time on Tuesday and Thursday. That can't be stored in UTC as it varies depending on the time of the year. It has to have the timezone stored and actioned.
  - latchkey 1y ago
    You can tell the job when to run in UTC time and change that time depending on what the current DST rules are. But if the job is set to run in UK time and the system clock jumps around because of TZ changes, it will run twice or not at all. Having a stable zero based time that doesn't move around by external forces that you can't control is "the answer."
    - ta1243 1y ago
      So I have to manually update the job How about I use some form of library to do it. I tell it I want to run at 0800 London time, and it runs at 0800 London time If I tell it I want to run at 0130 London time (or 0330 Athens time) I still have a problem -- do I run it twice when the clocks go back, do I skip it when clocks go forward? But that's a business logic problem, and defining it as UTC and having another job to update the time twice a year doesn't actually solve the question of "what do I do at this point".
      - latchkey 1y ago
        You don't have any choice in the matter. If you control the time you want the jobs to run externally from the server, then you can have control over it. Otherwise, you're getting jobs that run 2x or not at all.
- jedberg 1y ago
  > Just use UTC folks unless you have a really good idea why you shouldn't. If my whole team is based in Pacific Time, I'm gonna use something close to PT so I don't have to do hard math when reading raw logs. If we're all US based, I still want something in the USA. The math is easier.
  - kjehjrktlhl 1y ago
    Technical debt. Please don’t.
    - jrockway 1y ago
      Yeah. I log in json + unix timestamp nanoseconds, and just convert it to something human-readable on display (github.com/jrockway/json-logs). I am not sure why logs need to be "pretty" at time of logging when they can be made pretty at time of display. Doing it at time of display means you can just use local time, and then nobody is confused.
    - jedberg 1y ago
      [flagged]
      - throwaway150 1y ago
        > Who hurt you? Nobody hurt them. Please don't do this kind of childish rebuttals. It looks really silly on HN. If you don't want to use UTC, that's one thing. You do you. But there are reasons for sane professionals (who weren't hurt by anybody) to expect their coworkers to use UTC when possible. Because using local timezone leads to some problems including the one in TFA.
      - hobs 1y ago
        Well me, I have been burned by fixing this UTC bug at like 8 companies and overall more than four year of my life (between other things, each project took at least six months of people poking at things) because at an eventual scaling point it starts biting your ass everywhere in every service. Just because it worked once doesn't mean it's good.
- yabones 1y ago
  I put a little analog clock on my desk that's set to UTC time. It helps a lot with converting logs to get a quick reference.
  - lozf 1y ago
    Perhaps a shell alias for: TZ="Etc/UTC" xclock -analog -update 1 -norender -hl grey -fg grey -bg black (or similar) would also suffice.
  - derwiki 1y ago
    I do the same! I am in PDT and keep a UTC and EDT clock (most of my team is east coast)
  - schraitle 1y ago
    I have a piece of paper on my desk, each side has the time zone, utc offset, and date when DST will change. Twice a year I flip over the paper. Right now it reads "EDT, UTC-4, until: Sun, Nov 2" Going to add a clock next to it now, that's a great idea
    - ponector 1y ago
      And then with distributed team you realize DST change dates are different around the globe.
      - pseudalopex 1y ago
        Desks around the globe can have different pieces of paper.
    - vdm 1y ago
      $ zdump -Vc 2025,2026 America/New_York America/New_York Sun Mar 9 06:59:59 2025 UT = Sun Mar 9 01:59:59 2025 EST isdst=0 gmtoff=-18000 America/New_York Sun Mar 9 07:00:00 2025 UT = Sun Mar 9 03:00:00 2025 EDT isdst=1 gmtoff=-14400 America/New_York Sun Nov 2 05:59:59 2025 UT = Sun Nov 2 01:59:59 2025 EDT isdst=1 gmtoff=-14400 America/New_York Sun Nov 2 06:00:00 2025 UT = Sun Nov 2 01:00:00 2025 EST isdst=0 gmtoff=-18000 $ zdump -Vc 2025,2026 Europe/London Europe/London Sun Mar 30 00:59:59 2025 UT = Sun Mar 30 00:59:59 2025 GMT isdst=0 gmtoff=0 Europe/London Sun Mar 30 01:00:00 2025 UT = Sun Mar 30 02:00:00 2025 BST isdst=1 gmtoff=3600 Europe/London Sun Oct 26 00:59:59 2025 UT = Sun Oct 26 01:59:59 2025 BST isdst=1 gmtoff=3600 Europe/London Sun Oct 26 01:00:00 2025 UT = Sun Oct 26 01:00:00 2025 GMT isdst=0 gmtoff=0
- nodesocket 1y ago
  I use UTC for all public/production servers, but for my homelab servers in my closet rack they all use local time. Makes grokking times in homelab servers much easier. The exception is database insert/update date/times which are always stored UTC.
- SecondHandTofu 1y ago
  Generally, yes but I there's a surprising amount of cases when it is important, which makes it difficult to generalise e.g. Huge amounts of the financial sector cares because of market times or regulatory reasons.
- jacobsenscott 1y ago
  Also don't use cron full stop. Build (or use an off the shelf) scheduler in your app that's more sophisticated, has access to data and client preferences, etc.
superkuh 1y ago
This isn't a problem with actual cron and crontab. It is a problem with the systemd-timers shim "crontab" which doesn't work the same in many corner cases and often has weird bugs.
- lxgr 1y ago
  "Not a problem" as in these only run these cron job once per day, irrespective of DST changes making a given "hour happen twice"?
- Bratmon 1y ago
  How does actual cron handle this situation?
- LegionMammal978 1y ago
  This post is literally about an issue observed in vixie-cron (as included in some distro c. 2013), not about any systemd implementation.
  - nailer 1y ago
    Exactly! The article makes the point that: > until one of them achieves cron’s level of ubiquity, we have to live with cron at least some places and sometimes systemd could arguably be described as close to (maybe behind, maybe ahead of since it's the default for the most popular Linux distros) cron's level of ubiquity, and doesn't have this bug as far as we know.
- ziml77 1y ago
  What do you consider "actual cron"? Because the post specifically says Vixie cron, which has been the basis for most versions of cron on Linux systems.
- Kwpolska 1y ago
  Which cron is actual cron? There are tons of implementations out there. This post describes vixie-cron, not systemd-timers.
cuu508 1y ago
Apparently there are some timezones (Cuba, Egypt, Lebanon) where DST change happens at midnight, so, also watch out for that! https://mas.to/@mpirnat/115395859892135002 https://mas.to/@mpirnat/115395859892135002
- latchkey 1y ago
  There are TZ rules that do all sorts of wild things.
  - noir_lord 1y ago
    Leading to my favourite web comic of all time. https://www.monkeyuser.com/2018/going-global/ https://www.monkeyuser.com/2018/going-global/ Monkey User is always entertaining.
  - wat10000 1y ago
    Brazil not only had DST at midnight, but until 2008 they also had no standard rule for when DST would begin and end, setting the dates by decree often just a few weeks in advance.
    - deleted 1y ago
      [deleted]
    - 1-more 1y ago
      Kind of funny that it went from a Ramadan model to an Easter-post-computus model https://en.wikipedia.org/wiki/Ramadan#Beginning https://en.wikipedia.org/wiki/Ramadan#Beginning https://en.wikipedia.org/wiki/Date_of_Easter https://en.wikipedia.org/wiki/Date_of_Easter
  - baq 1y ago
    Is there any other person other than Arthur David Olson who needed an RFC written to cover their retirement? https://www.rfc-editor.org/rfc/rfc6557.html https://www.rfc-editor.org/rfc/rfc6557.html
gadders 1y ago
Also, never set jobs to run at midnight (00:00) as nobody will be able to tell what day it is. Set it to 00:01 or something. And tbh, don't run jobs on the hour anyway. Everything kicks off then. Set stuff to run at 01:45 or 02:15 or something instead. (None of these suggestions are time change related)
- tetha 1y ago
  In general I try to use odd times for cronjobs. Backups starting at XX:13, analysis scripts at XX:23, data exports at XX:42 and so on. This simplifies the question of "Why does my system go whack at 23:23? Oh, wait".
  - kibwen 1y ago
    With enough precision in your timestamp, you can base-10 encode arbitrary binary data and shove it in your nanos for debugging.
  - ksenzee 1y ago
    This is a great tip. One of our QA engineers noticed recently that a particularly difficult-to-reproduce bug was affecting records saved at :21, which is when a particular cronjob hits. Without that extra info we’d probably still be scratching our heads over it.
  - gadders 1y ago
    That's a good idea.
- deleted 1y ago
  [deleted]
- ericpauley 1y ago
  Honestly haven’t seen 00:00 cause confusion outside of “dog ate my homework” scenarios. If you’re using 12h clock then clearly it’s a bigger issue.
ttz 1y ago
> Alternatively, where possible set the server timezone to UTC so that no daylight savings changes will happen at all this is the way
- bobmcnamara 1y ago
  Use TAI
ldoughty 1y ago
I try to avoid crons at the top of the hour, partly because of this... but also because (in shared / serverless infrastructure) I assume a lot more people are setting their crons for 'on the hour' so there's more resource contention... I also aim for 'after 4am' where I can as well, or 'before midnight', to avoid this whole range.
- sgc 1y ago
  To incrementally improve that tactic, systemd has RandomizedDelaySec, which is a convenient way to reduce the possibility of scheduling conflicts.
  - tetha 1y ago
    I prefer to combine this with FixedRandomDelay=true. FixedRandomDelay ensures that the randomized delay is an arbitrary number up to RandomizedDelaySec, but it is deterministic per server and timer. I find this useful because this means the timer will always run at XX:12:45 on server01, always run on XX:06:23 on server02 and so on. This combines very simple configuration, while being predictable and spreading out timers well.
- layer8 1y ago
  One trick for cron is to prepend the actual command in the crontab with something like sleep $(( $(od -N1 -tuC -An /dev/urandom) % 60 ))m ; which will delay it by 0 to 59 minutes at random.
throwaway106382 1y ago
bruh, just use UTC
- bobmcnamara 1y ago
  Ugh gross. Same non monotonic time problems just not every year. TAI4LYFE!
jedberg 1y ago
I've told this story before, but it's super relevant here. When I set up all the servers for reddit, I set them all to Arizona time. Arizona does not have DST, and is only one hour off of California (where we all were) for five months, and the same as here the rest of the year. I didn't use UTC because subtracting eight when reading raw logs is a lot harder than subtracting one. They use UTC now, which makes sense since they have a global workforce and log reading interfaces that can automatically change timezones on display.
- latchkey 1y ago
  Computers can do things. Build a UX to read the logs and have it automatically parse/convert the logs to show whatever is local time. https://unix.stackexchange.com/questions/274816/how-to-convert-a-timezone-in-logs-as-they-are-being-tailed https://unix.stackexchange.com/questions/274816/how-to-conve...
  - jedberg 1y ago
    There are many solutions, but when you're actually running a site with millions of daily visitors and one person focused on ops, you do the easy thing, not the right thing, unless those happen to coincide.
    - latchkey 1y ago
      Sure... "I'm too busy fighting fires to come up with a solution" is always a valid answer, but that's not what I was replying to.
      - lxgr 1y ago
        Even with infinite resources (love to see that in real companies!), any such solution adds complexity, which comes at a cost.
        latchkey 1y ago
        If this is a complexity battle, I'd rather that sysadmins have to use a tool to read logs vs. figuring out why the server is melting down because all the jobs are running 2x or answering questions about why jobs didn't run at all. In my old gig, jobs ran for many many HOURS and consumed most of the IO on the server (processing reams of data), which meant that when you got them running 2x because of overlap, it was a trainwreck.
raggles 1y ago
I have a job that deliberately runs at 2am and 3am... to update the time in a bunch of really old PLCs for DST. And check that every other device on my telemetry network has correctly updated its time.
PhilipRoman 1y ago
I read this a while ago and was looking forward to the mess that would happen with these default Alpine configurations (after setting non-UTC localtime) 0 2 * * * run-parts /etc/periodic/daily 0 3 * * 6 run-parts /etc/periodic/weekly It appears that busybox cron is smart enough to handle it and only ran the jobs once.
- latchkey 1y ago
  Google's AI and ChatGPT answer disagrees. https://www.google.com/search?q=busybox+cron+dst https://www.google.com/search?q=busybox+cron+dst "BusyBox’s crond is a lightweight implementation designed for embedded systems, so it doesn’t have all the timezone/DST logic of full vixie-cron or cronie. It just reads /etc/TZ or the system timezone at startup and then wakes up every minute to check if something should run."
  - PhilipRoman 1y ago
    https://github.com/mirror/busybox/blob/master/miscutils/crond.c#L1093 https://github.com/mirror/busybox/blob/master/miscutils/cron...
  - moderateclimate 1y ago
    Please don't do this. AI is not a source and is frequently wrong (as it is in this case).
    - latchkey 1y ago
      You're right. It was honestly just a low effort semi-sarcastic reply that I wasn't really expecting anyone to take seriously. I should have just gone and read the source code as PhilipRoman did below.
matja 1y ago
Seems to me the real problem here is not the timezone (there's legitimate business needs to run something daily at a specific localtime...) but having multiple instances of a cron job that overlap, in which case it should wait until the previous is done or not start at all. At least prefix a job with "flock -n" if it doesn't/can't handle that.
tonyhart7 1y ago
DST is such a dumb mechanism anyway just use UTC
deleted 1y ago
[deleted]
skopje 1y ago
UTC also solves plotting daily time plots in zones with time changes (twice a year when the plots make no sense). But there still isn't no good fix for leap year plots!
- bobmcnamara 1y ago
  Just use TAI?
meonkeys 1y ago
anacron is worth a mention here for sometimes-on computers like desktops, laptops, or personal servers. It's great for ensuring something runs at the set interval, or just whenever the machine comes back online. Minimum anacron interval is one day, so this can't simply replace cron. cron might even be the most sensible thing to use to kick off anacron on your system, and it might already be in use (see if your /etc/crontab fires off anacron). anacron won't have the OP's 2am/3am issue because it locks jobs while they run, so multiple job spawns should be no-ops. Still seems like it would be good practice to, e.g. use UTC for the host system and do whatever else might help avoid the unnecessary re-spawns.
litoE 1y ago
I run a cronjob at 2:30AM every day that uses dirvish (a perl wrapper around rsync) to perform backups of several computers. In the fall, at 2AM the clock is rolled back to 1AM. The cronjob runs at 2:30AM, which is 25 hours after the previous day's run. That's not a problem. However, in the spring, at 2AM the clock jumps to 3AM. Vixie cron seems to be smart enough to run all the cron jobs that were scheduled between 2M and 3AM at that point. However, dirvish is started but exits without running the backups! After some troubleshooting I concluded that dirvish does not make the backups because the previous day's run happened less that 24 hours ago (because that day was shortened by the change to DST) so it sees no need to run a backup yet. This problem would persist even if I moved the cronjob to a different time because it has to do with the day having only 23 hours. I solved this by adding a second entry to the crontab that runs at 3:45AM only on the 2nd Sunday in March.
- deleted 1y ago
  [deleted]
tomkaos 1y ago
I avoid cron job at this time for this reason and for many other random problem because 3:00 am seem to be often choose for a maintenance like automatic update, server reboot, database backup..
- tclancy 1y ago
  Can be tricky: an old project had a job that ran every 15 minutes to poll data from a popular smart thermostat company. I always knew when DST was starting or ending because that job would throw an error email at 2/3am on those days.
hinkley 1y ago
2 am is mid afternoon for the Pune office. If you’re running a 24/7 operation and all of your employees are in PST you’re gonna have a bad time. But the other thing that has changed since 2013 is that most of us are using autoscaling. There were times of day when there is supplemental server capacity to run periodic processes, and it made sense to schedule them during quiet daylight hours if they existed, but now the availability of bus numbers if and when a problem happens are more pressing. And they all outweigh concerns about DST. Instead of seeking the low traffic times of day you mainly need to avoid peak traffic. The batch processing system I renovated to use 7% if the CPU hours of the original design, used the same fixed parallelism trick that crawlers use so as not to knock over your sites. They keep k requests in flight at all times and if your server responds faster, they issue more requests per second. If the production services had excess capacity the job would run in ten minutes. If response times are slow it takes 15. You’re inverting Little’s Law and keeping the amount of servers fixed and varying the duration. It’s a pity that spooling up more servers to handle that job took a substantial fraction of that ten minutes or I could have gotten it to five (originally it was over 30 minutes and climbing toward 35). I tried at one point to move it entirely to a Bamboo agent but we underprovisioned them. If I’d had an agent with 2x the CPUs I could have shut down a service. I was on my way out the door before I realized that upgrading the bamboo agents would have cost us less than running that service and everyone’s builds would have been faster as a consequence. And probably let someone else do the same. False economies.
brettgriffin 1y ago
> Alternatively, where possible set the server timezone to UTC so that no daylight savings changes will happen at all. Is this what people actually want, though? I'm actually working on a scheduled process right now deployed on a cloud scheduler that does not adjust for daylight saving time (it literally says "does not take into consideration daylight saving time" next to the time input). Everybody wants the job to run four hours before the work day starts, every day of the year. I will have to change it on Sunday to ensure the job completes at the time the end users expect it. And again in March. Am I missing something?
- monster_truck 1y ago
  I call this "rake stomping", as long as you're running back and forth in the same places every time, you'll never notice.
- waisbrot 1y ago
  The article is mostly talking about using the `cron` scheduler and running things at 2am on Sunday because there's especially low traffic. These cron jobs are "I don't care when it runs, but it should have minimal chance of causing problems." Your use-case is totally different: "I want this job to run at this time" so the only lesson that applies to you is that the `cron` utility might behave weird during DST switches. No idea if it underlies your cloud provider, so it may be completely irrelevant.
- evil-olive 1y ago
  > Everybody wants the job to run four hours before the work day starts recurring events like this - ones that take place relative to humans in a given location - are the most common snag related to "just use UTC everywhere" advice. but they're not actually in conflict at all. they just require an extra layer of indirection. let's say this is a factory that starts early in the morning, 06:30 local time, because then "four hours before" falls into the window affected by DST. "run at 02:30 local time" is what you want to avoid, because on the "spring forward" day, 02:30 simply does not exist. the clocks jump from 01:59:59 to 03:00:00. [0] likewise, "run at 01:30 local time" is to be avoided, because on the "fall back", that occurs twice. ideally, you want to think of the recurrence as being associated with a given location, and that recurrence generates individual events, which happen in UTC time. for example, if the factory is in San Francisco, the recurrence might specify "once a day, at 02:30 SF time", which would generate a series of daily events in UTC time. during the summer those events are at 09:30Z, during the winter they're at 10:30Z. you also want the scheduler to be smart enough to understand the "once a day" constraint - so that a "once a day, at 01:30 local time" task generates only one daily event, rather than two, even though 01:30 local time happens twice in the same day. and then, because the individual events are all in UTC, and the system clock is as well, there's no ambiguity in the actual execution time of the event. the complexity of the recurrence is handled at a higher level. if you want an even harder version of this problem, imagine a team in SF and a team in London, that have a Zoom meeting every Monday at 8am (SF) / 4pm (London). it's necessary for one of those locations to "own" the recurrence, because the UK does their time changes 1-3 weeks offset from the US [1], creating weeks where it isn't a straightforward 8 hour time difference. 0: https://www.timeanddate.com/time/change/usa https://www.timeanddate.com/time/change/usa 1: https://www.timeanddate.com/time/change/uk https://www.timeanddate.com/time/change/uk
p0w3n3d 1y ago
It depends on tz. 4 am in some countries too
sedatk 1y ago
Redundantly running cron jobs are the least of our problems. Daylight savings adjustment day causes surge in deaths and ER visits[1]. Daylight savings needs to go. https://www.sciencealert.com/daylight-savings-time-change-kills-people https://www.sciencealert.com/daylight-savings-time-change-ki...
HiPhish 1y ago
DST was a mistake and it needs to die. It solves no problems and only creates them. And not even the type of problem that someone could profit from, just plain old complete waste of time for everyone problems. Just stick to winter time (because that's the correct one) and adjust all working hours once and for all to be like summer time, for e.g. instead of business opening at 7AM and closing at 10PM they can open at 6AM and close a 9PM. There will be a period of adjustment, but we have a period of adjustment twice a year already, so nothing is lost. Why match working hours to summer time? Because I want to have more sunlight by the time I'm done with work. Especially during winter. We have it completely backwards, if we are going to adjust our clocks we should adjust them in a way that gives us more sunlight during winter, not less. I don't care if it's dawning by the time I'm going to work where I will be stuck indoors for nine hours, I want sunlight when I'm free again.
- StopDisinfo910 1y ago
  So you propose we stop shifting the clock but shift every opening and closing times instead producing exactly the same effect with basically no difference. And you want to abolish DST but the end of your comment is that you actually want even more shift including in winter. I fear I have trouble following your reasoning and understanding what you actually want.
  - butlike 1y ago
    There's more clocks than storefronts
  - xigoi 1y ago
    If I’m reading the parent correctly, they Want to switch to non-summer time and adjust opening hours to be effectively like summer time, forever.
    - hsbauauvhabzb 1y ago
      I wonder what the earliest sunrise time would be as a result, 2am?
cmiles8 1y ago
Or just use UTC and stop using timezones in code. I get it’s not that simple for some use cases but UTC exists for a reason.
- bobmcnamara 1y ago
  Use TAI. UTC has most of the same wacky timezone problems, just less often.
  - ericpauley 1y ago
    You’re signing up for a world of pain in the real world, where every other system is UTC. Just use UT1 or smear the leap second if you really can’t stand the risk.
    - bobmcnamara 1y ago
      Approaching one second per second, is that so much to ask? Unless standardization has improved, how+when the second is smeared(if at all) means timestamps from the same system aren't usefully subtractable without communicating a separate time skew calendar. That said, time is a big topic, and everything has different requirements but migrating to TAI and treating UTC as yet another human readable time zone has solved more (admittedly application specific) problems than it created.
MayeulC 1y ago
Not a crown job, but I configured my lights to turn on progressively at my scheduled wake up time, using Home Assistant. I figured that scheduling lights to turn on at <0 AM + wake up time> was a good approach... We got a surprise early wake up last year, so I ended up changing the formula to <4 AM - 4 hours + wake-up time>.
cadamsdotcom 1y ago
Great to read about something that's a solved problem now. It's been a very long time since I used a server that wasn't set to UTC.
- bobmcnamara 1y ago
  I've seen this same problem under UTC. Gotta use TAI
ckastner 1y ago
This was worked around ages ago in OpenBSD, and the workaround was already included in Debian (and by extension, Ubuntu) when I started maintaining it in 2010 (I no longer do). Here's a link to a patch [1] from a version from when the package still kept standalone patches. It's been a long long time and I honestly don't remember the details, but Debian's cron(8) still says [2]: Special considerations exist when the clock is changed by less than 3 hours, for example at the beginning and end of daylight savings time. If the time has moved forwards, those jobs which would have run in the time that was skipped will be run soon after the change. Conversely, if the time has moved backwards by less than 3 hours, those jobs that fall into the repeated time will not be re-run. Edit: According to this bug report [3], this workaround first entered Debian in 1999. [1]: https://sources.debian.org/src/cron/3.0pl1-137/debian/patches/features/Better-timeskip-handling.patch https://sources.debian.org/src/cron/3.0pl1-137/debian/patche... [2]: https://manpages.debian.org/unstable/cron/cron.8.en.html https://manpages.debian.org/unstable/cron/cron.8.en.html [3]: https://bugs.debian.org/8499 https://bugs.debian.org/8499
- silverwind 1y ago
  So we can assume the author was not running a distro that had "fixed" the issue?
  - ckastner 1y ago
    Most likely, yes. The author mentions vixie-cron, which was the name of the project before Paul Pixie joined/founded(?) ISC, and it was released as ISC from after. Debian's fork is still based on vixie-cron, but it couldn't have been the one because of the aforementioned patch.
jcynix 1y ago
> better open source job scheduler [...] one that will ensure no overlapping runs Or learn to use lock files, which would make sure that your jobs don't run when another one is already running? Unix shells offer the trap command, among other things, to cleanup, even in the case of errors, e.g.: set -e LOCK=$(ensure-lock) trap "rm $LOCK" 0 1 2 15 do-job https://en.wikipedia.org/wiki/Semaphore_%28programming%29 https://en.wikipedia.org/wiki/Semaphore_%28programming%29
- TylerE 1y ago
  Tell me how that goes after a server malfunction or power failure. Where is your trap god then?
  - jcynix 1y ago
    The topic was to prevent overlapping runs. In case of power failures etc all your server processes have this problem. That's why, for example, /var/run exists and files/directories are deleted from /var/run on reboot?
magarnicle 1y ago
The problem is not so much which timezone you pick, the problem is that cron uses 'naive' datetimes. If the timezone is explicit then you don't have duplicate hours because one is 02:00 and one is 02:00DST. And then you can translate that time to any other timezone with ease, knowing that you are referencing a specific hour in physical time that only ever happened once.
- magarnicle 1y ago
  Which is to say I wish cron had the TZ variable that Jenkins uses in it's scheduling system. And the H value to easily spread multiple jobs over the hour/day/month etc.
- LeoPanthera 1y ago
  > knowing that you are referencing a specific hour in physical time that only ever happened once "Falsehoods programmers believe about time" This assumption can break with: * Leap seconds * Calendar and time zone changes * (In exotic circumstances) Relativity
  - magarnicle 1y ago
    I'll give you leap seconds, but at least in Python historical timezone changes are recorded in the datetime library, so the timezone at any particular time in the past has the timezone offset that was active then.
atoav 1y ago
If you want to avoid overlapping runs simply simply have the task check for the existence of a lockfile, if not create a lockfile containing the start timestamp, run the task and delete the lockfile at exit or error. The timestamp gives you a chance to recognize stale/orphraned lock files in case of crashes based on their age.
loodish 1y ago
I like cron jobs that run during the work day. If something goes wrong I would prefer to handle it at 10am rather than 10pm. Why make life unpleasant? If it really needs to run during a low load period I line it up for 6am work time. Again so that if something goes wrong it leads into the work day rather than disrupts the on-call too badly. 6am also provides a daylight savings buffer either way so you don't have to care. Recent work has all been on 24/7 operations though.
dathinab 1y ago
2-3am is bad due to DST but there is another reason to also avoid 3-5am or so around 3am is the time where "in average" human attention is the worst as its where "in average" their most relevant deep sleep phases are now I guess the average over HN readers has a good chance to be somewhat later but either way if the job goes wrong and triggers alarms 2-5am it very bad for the the health and sleep rithm of whoever needs to fix it and the most likely time they make mistakes it sadly is also the time where non-international interruptions are least likely to matter if you are an international company having international teams can be an solution and if you are a very small team adopting job times to an overlap of low usage and your admins sleep habits can be an option but if you need to pick a generic time and don't provide for logistics or other "start work at 4/5am" companies probably 5-6am is a better time to run your batch jobs anyway just some random thoughts
tanin 1y ago
The details I never really thought about until I encountered timezone issues was: There are points in time that don't exist, and there are points in time that are ambiguous. If we adjust the time forward by 1 hour, then then 2:30am doesn't exist. If we adjust time backward, then 2:30am is ambiguous. 2am-3am is only safe for certain countries. America/Santiago (Chile), for example, adjusts their daylight saving on midnight. On a certain day, midnight doesn't exist because it immediately jumps to 1am. If you are building an analytics chart by day, you will encounter this issue. I wrote about several weird time zone edgecases here: https://tanin.nanakorn.com/edgecases-for-timezones/ https://tanin.nanakorn.com/edgecases-for-timezones/ To add more confusion, IIRC Java and Ruby handle these cases differently where Ruby raises exceptions (e.g. InvalidTime and AmbiguousTime) but Java doesn't. I caused 3 incidents in a row at Stripe in my first month because of this. I thought I was gonna be fired. Luckily I wasn't...
sharts 1y ago
Why run crons at that time anyway? It seems the most logical times for things are going to be midnight or beginning/end of business day.
ray_v 1y ago
Here's my radical time idea: instead of keeping the same schedule year round, we just incrementally chop parts of the day off (we just put that time in a "bucket" and pretend it doesn't exist). Then at the summer equinox we empty the time bucket into a 6 day bender of a party. /s
tsoukase 1y ago
For the most part of Europe and USA the day is either "quite" long (summer) or quite short (winter) due to the curve of the duration. Having light for long summer evenings and dark for long winter mornings is bad for many reasons, one of witch is seasonal depression, with which the Scandinavians suffer and fight. I was for the abolishment of DST until COVID cancelled it in Europe but now I am thinking it again and probably I am against.
Zanfa 1y ago
While not always possible, in general, any mission critical scheduled job should be idempotent. This avoids so much headache, especially when there's a queue involved somewhere.
nubinetwork 1y ago
Doesn't systemd timers fix this? AFAIK, a job set to run once a day shouldn't run a second time because the time rolled back.
tokyovigilante 1y ago
Whatever argument you may have against DST, “my cron jobs won’t run twice a year!” is not a strong one.

341 points by pera 1y ago

deleted 1y ago

[deleted]

I worked at a place that had set the servers to BST not UTC and used cron jobs extensively for reporting, which caused chaos twice a year and twice a year they had to explain to the business again why it caused chaos and ask for the time to fix it. Company went bust before it got fixed. Just use UTC folks unless you have a really good idea why you shouldn't.

lxgr 1y ago
"Our customers aren't in UTC" is often a compelling reason. People generally expect things like usage limits to reset overnight, not at UTC midnight, and these are often implicitly tied to (the result of) batch jobs. Financial jobs, including billing, is another big category.
- latchkey 1y ago
  Always set the servers to UTC. Convert the time to local time for display, but store in UTC. The reporting job should be run from a server on UTC and on UTC time. The report itself can convert to whatever is local time.
  - gruez 1y ago
    That doesn't fix the problem of "my quota reset (or reporting job) gets shifted by 1 hr when daylight saving time changes".
    - latchkey 1y ago
      The reporting job should be run from a server on UTC and on UTC time. The report itself can convert to whatever is local time. This is exactly what I said above.
      - lxgr 1y ago
        That works if all your consumers of the report are indifferent to it arriving at a different time half of the year. All of yours might well be, but you can't generalize that assumption to every domain and scenario.
        ipaddr 1y ago
        If you want an email of the report to send out at 9am locally. Generate before and send to everyone based on timezone.
        kjehjrktlhl 1y ago
        You don’t want a report to include 25h or 23h twice a year just to be able to send it at some distinct local time.
        lxgr 1y ago
        I absolutely want some daily business reports to cover 25 hours on one day of the year and 23 on another, since that's how long a calendar day is in local time in every place that has DST shifts.
        lenzm 1y ago
        Yes, that's exactly what I want. Everything shifts with dst - business processes, people's working hours, ach windows, ...
        gruez 1y ago
        > Generate before and send to everyone based on timezone. How are you going to do this? Clearly cron (with timezone set to UTC) doesn't work for this so you'll need another system. In that case why not have that multi-time zone scheduling system run your script?
    - ranger_danger 1y ago
      You could have the reset script check the timezone of the user before actually performing the reset, and have this script run at least once an hour to make sure you get close to accurate for most people (some timezones only differ by less than an hour though!). But also, if the user can change the timezone themselves, now you have a condition where the user could keep their quota at bay for a very long time (if not forever) if they keep changing their timezone accordingly.
    - throwaway150 1y ago
      > That doesn't fix the problem of "my quota reset (or reporting job) gets shifted by 1 hr when daylight saving time changes". How hard is to convert the UTC time to user's local time and perform quota reset? I mean you should be doing this anyway. Even if you store everything in PST or EST or CST, you never know when you'll get customers from another part of the world and then you'll have to do this anyway. So why not do this already?
      - lxgr 1y ago
        > How hard is to convert the UTC time to user's local time and perform quota reset? Probably roughly as hard as writing a timezone-aware scheduler that considers all edge cases around daylight savings time, i.e. probably possible but I'd try fairly hard to see if I can get around it. One way of getting around it is to run your batch jobs in your "primary business timezone". Sure, you might outgrow that concept eventually, but many companies never do, and in that sense "running on UTC" seems similarly aspirational in spirit (albeit at a much smaller scale) to starting with high scalability.
      - gruez 1y ago
        >Even if you store everything in PST or EST or CST, you never know when you'll get customers from another part of the world and then you'll have to do this anyway. So why not do this already? 1. allowing users to change their timezone and tying it to when their quota reset sounds like a great way to add more edge cases and complexity. For instance, does messing with your time zone mean you can get your quota reset multiple times a day? 2. Not everyone operates some sort of global b2c SaaS that's timezone agnostic. For many enterprise backoffice tasks "6am HQ time" is far more reasonable than "6am/7am, dunno depends on whether daylight saving time is on or not". In this case having a server set to the HQ's timezone makes far more sense than doing UTC and trying to work around daylight saving issues.
    - cweagans 1y ago
      Do you actually care if the requests were made on some specific calendar day? Or do you just want to make sure that heavy users are paying more and/or people aren't abusing your system? Instead of tying your quotas to calendar days in some specific time zone, tie them to a rolling 24 hour usage period. Even better: use a rolling hour.
- freedomben 1y ago
  Having tried many times to use local time instead of UTC to make life simpler for myself, I really don't think it's ever a good idea. Better to localize times for users on the client-side based on their local settings, IMHO For quotas and usage limit resets it can be a little harder, but if it's a cron job anyway, my initial approach would be to just schedule it for a time that is close to midnight for the area your customers are in.
- Kwpolska 1y ago
  That does not require setting the server to local time. Run the cron job every hour. Before starting the script, check the time and compare it to the last saved execution time. If the day has changed, do stuff and save the new date, else exit. This also happens to be more resilient to server downtime around midnight.
  - lxgr 1y ago
    > Before starting the script, check the time and compare it to the last saved execution time. If the day has changed, do stuff and save the new date, else exit. That sounds a bit like reimplementing a scheduler inside the scheduled task (which is ok if you don't trust your scheduler, but I'd avoid it if at all possible).
  - bell-cot 1y ago
    Except downtime can start during execution. So for many types of jobs... - Check if run today, exiting on "yes" - Run - Update the last-run-on date
    - latchkey 1y ago
      What is "today"?
      - bell-cot 1y ago
        Hopefully just a datetime recent enough that the job does not currently need to be (re-)run. But as soon as one job needs another to complete first, or two jobs shouldn't be run at the same time - things start getting messy.
        lxgr 1y ago
        > Hopefully just a datetime recent enough that the job does not currently need to be (re-)run. I can think of several ways to mess that seemingly trivial calculation up just from the top of my head. (Consider changing timezones, increasing the interval the job is running to within half a day of scheduling period etc.)
- skissane 1y ago
  > "Our customers aren't in UTC" is often a compelling reason. This doesn’t work as soon as you become a global business… because if you tell a customer in Asia that it resets at midnight in some US timezone, what are they going to think? Many people will accept UTC because it is the international standard. Everyone will accept “let each customer pick the timezone that works best for them”. Saying “I’m going to use a US timezone because that is easiest for our US customers” risks sending a message to your (now or future) global customers that you view them as second-class citizens.
  - lxgr 1y ago
    Really depends on your market and customers. Especially in the financial industry, local time zones (often of a given currency's central bank's main branch, a primary regulator etc.) are often very, very entrenched. > Many people will accept UTC because it is the international standard. I really wish that were true universally... And I do think that UTC is actually somewhat Eurocentric! From personal experience, it's significantly easier to mentally add/subtract one or two hours (modulo 24) than 7 or more. > Saying “I’m going to use a US timezone because that is easiest for our US customers” risks sending a message to your (now or future) global customers that you view them as second-class citizens. Definitely, but on the other hand, I do think that picking your head office's time zone as the "canonical" one for resetting quotas etc. has some merit as well, if only because it makes the concepts of "today" and "tomorrow" work a bit better. (UTC midnight might be very unintuitive to both you and your customers.)
    - skissane 1y ago
      > Definitely, but on the other hand, I do think that picking your head office's time zone as the "canonical" What happens when you move your head office to a different time zone? e.g. Oracle moved their HQ from California to Texas (and have announced a further move to Tennessee, although last I heard that still hasn’t happened) What happens when an M&A happens and you are now part of a much larger enterprise with a different HQ timezone, and systems with wildly inconsistent configured timezones due to having acquired multiple companies which had HQs in different time zones and all of which configured everything to use their HQ time? > UTC midnight might be very unintuitive to both you and your customers I live in eastern Australia, UTC+11 at the moment (UTC+10 during Southern Hemisphere winter). Right now, resetting stuff at UTC midnight means at 11am my time, resetting stuff at US Eastern midnight means at 3pm my time - roughly equal in potential inconvenience, but I’ll be much more forgiving of the first than the second. > And I do think that UTC is actually somewhat Eurocentric! From personal experience, it's significantly easier to mentally add/subtract one or two hours (modulo 24) than 7 or more. For me, add/subtract 10 is pretty easy… and then just one more during DST. Does that make UTC “Australia-centric”? Or Guam-centric or Vladivostok-centric? (both of which are also UTC+10)
latchkey 1y ago
At Marin Software, they ran each customer on their own set of servers and set the servers to the customers defined TZ. It was an endless cronjob nightmare. Now they are also bankrupt. Just found out some guy decided to try to restart the company. Good luck. https://x.com/Austen/status/1981904435539280324 https://x.com/Austen/status/1981904435539280324
ta1243 1y ago
If I want a report of what happened at a specific time I need that report at local time I get a daily status report of various things from our 24 hour operational management team which runs at 8am UK time every day. That means last week it ran at 0700UTC and this week at 0800UTC This is built around operational events, shift changes, etc. I've got another system which is in operation in Sydney from 0630 to 1630 local time, this means that maintence windows which overlap with UK shift patterns depend on the week but mean the system is operating 2130-0730 UK time at some times, 2030-0630 UK at others, and 1930-0530 at others. UTC is not "the answer". Sometimes you want things running at a UTC time, sometimes you want them running at local time. I have a regular meeting at 10am London time on Tuesday and Thursday. That can't be stored in UTC as it varies depending on the time of the year. It has to have the timezone stored and actioned.
- latchkey 1y ago
  You can tell the job when to run in UTC time and change that time depending on what the current DST rules are. But if the job is set to run in UK time and the system clock jumps around because of TZ changes, it will run twice or not at all. Having a stable zero based time that doesn't move around by external forces that you can't control is "the answer."
  - ta1243 1y ago
    So I have to manually update the job How about I use some form of library to do it. I tell it I want to run at 0800 London time, and it runs at 0800 London time If I tell it I want to run at 0130 London time (or 0330 Athens time) I still have a problem -- do I run it twice when the clocks go back, do I skip it when clocks go forward? But that's a business logic problem, and defining it as UTC and having another job to update the time twice a year doesn't actually solve the question of "what do I do at this point".
    - latchkey 1y ago
      You don't have any choice in the matter. If you control the time you want the jobs to run externally from the server, then you can have control over it. Otherwise, you're getting jobs that run 2x or not at all.
jedberg 1y ago
> Just use UTC folks unless you have a really good idea why you shouldn't. If my whole team is based in Pacific Time, I'm gonna use something close to PT so I don't have to do hard math when reading raw logs. If we're all US based, I still want something in the USA. The math is easier.
- kjehjrktlhl 1y ago
  Technical debt. Please don’t.
  - jrockway 1y ago
    Yeah. I log in json + unix timestamp nanoseconds, and just convert it to something human-readable on display (github.com/jrockway/json-logs). I am not sure why logs need to be "pretty" at time of logging when they can be made pretty at time of display. Doing it at time of display means you can just use local time, and then nobody is confused.
  - jedberg 1y ago
    [flagged]
    - throwaway150 1y ago
      > Who hurt you? Nobody hurt them. Please don't do this kind of childish rebuttals. It looks really silly on HN. If you don't want to use UTC, that's one thing. You do you. But there are reasons for sane professionals (who weren't hurt by anybody) to expect their coworkers to use UTC when possible. Because using local timezone leads to some problems including the one in TFA.
    - hobs 1y ago
      Well me, I have been burned by fixing this UTC bug at like 8 companies and overall more than four year of my life (between other things, each project took at least six months of people poking at things) because at an eventual scaling point it starts biting your ass everywhere in every service. Just because it worked once doesn't mean it's good.
yabones 1y ago
I put a little analog clock on my desk that's set to UTC time. It helps a lot with converting logs to get a quick reference.
- lozf 1y ago
  Perhaps a shell alias for: TZ="Etc/UTC" xclock -analog -update 1 -norender -hl grey -fg grey -bg black (or similar) would also suffice.
- derwiki 1y ago
  I do the same! I am in PDT and keep a UTC and EDT clock (most of my team is east coast)
- schraitle 1y ago
  I have a piece of paper on my desk, each side has the time zone, utc offset, and date when DST will change. Twice a year I flip over the paper. Right now it reads "EDT, UTC-4, until: Sun, Nov 2" Going to add a clock next to it now, that's a great idea
  - ponector 1y ago
    And then with distributed team you realize DST change dates are different around the globe.
    - pseudalopex 1y ago
      Desks around the globe can have different pieces of paper.
  - vdm 1y ago
    $ zdump -Vc 2025,2026 America/New_York America/New_York Sun Mar 9 06:59:59 2025 UT = Sun Mar 9 01:59:59 2025 EST isdst=0 gmtoff=-18000 America/New_York Sun Mar 9 07:00:00 2025 UT = Sun Mar 9 03:00:00 2025 EDT isdst=1 gmtoff=-14400 America/New_York Sun Nov 2 05:59:59 2025 UT = Sun Nov 2 01:59:59 2025 EDT isdst=1 gmtoff=-14400 America/New_York Sun Nov 2 06:00:00 2025 UT = Sun Nov 2 01:00:00 2025 EST isdst=0 gmtoff=-18000 $ zdump -Vc 2025,2026 Europe/London Europe/London Sun Mar 30 00:59:59 2025 UT = Sun Mar 30 00:59:59 2025 GMT isdst=0 gmtoff=0 Europe/London Sun Mar 30 01:00:00 2025 UT = Sun Mar 30 02:00:00 2025 BST isdst=1 gmtoff=3600 Europe/London Sun Oct 26 00:59:59 2025 UT = Sun Oct 26 01:59:59 2025 BST isdst=1 gmtoff=3600 Europe/London Sun Oct 26 01:00:00 2025 UT = Sun Oct 26 01:00:00 2025 GMT isdst=0 gmtoff=0
nodesocket 1y ago
I use UTC for all public/production servers, but for my homelab servers in my closet rack they all use local time. Makes grokking times in homelab servers much easier. The exception is database insert/update date/times which are always stored UTC.
SecondHandTofu 1y ago
Generally, yes but I there's a surprising amount of cases when it is important, which makes it difficult to generalise e.g. Huge amounts of the financial sector cares because of market times or regulatory reasons.
jacobsenscott 1y ago
Also don't use cron full stop. Build (or use an off the shelf) scheduler in your app that's more sophisticated, has access to data and client preferences, etc.

superkuh 1y ago

This isn't a problem with actual cron and crontab. It is a problem with the systemd-timers shim "crontab" which doesn't work the same in many corner cases and often has weird bugs.

lxgr 1y ago
"Not a problem" as in these only run these cron job once per day, irrespective of DST changes making a given "hour happen twice"?
Bratmon 1y ago
How does actual cron handle this situation?
LegionMammal978 1y ago
This post is literally about an issue observed in vixie-cron (as included in some distro c. 2013), not about any systemd implementation.
- nailer 1y ago
  Exactly! The article makes the point that: > until one of them achieves cron’s level of ubiquity, we have to live with cron at least some places and sometimes systemd could arguably be described as close to (maybe behind, maybe ahead of since it's the default for the most popular Linux distros) cron's level of ubiquity, and doesn't have this bug as far as we know.
ziml77 1y ago
What do you consider "actual cron"? Because the post specifically says Vixie cron, which has been the basis for most versions of cron on Linux systems.
Kwpolska 1y ago
Which cron is actual cron? There are tons of implementations out there. This post describes vixie-cron, not systemd-timers.

cuu508 1y ago

Apparently there are some timezones (Cuba, Egypt, Lebanon) where DST change happens at midnight, so, also watch out for that! https://mas.to/@mpirnat/115395859892135002 https://mas.to/@mpirnat/115395859892135002

latchkey 1y ago
There are TZ rules that do all sorts of wild things.
- noir_lord 1y ago
  Leading to my favourite web comic of all time. https://www.monkeyuser.com/2018/going-global/ https://www.monkeyuser.com/2018/going-global/ Monkey User is always entertaining.
- wat10000 1y ago
  Brazil not only had DST at midnight, but until 2008 they also had no standard rule for when DST would begin and end, setting the dates by decree often just a few weeks in advance.
  - deleted 1y ago
    [deleted]
  - 1-more 1y ago
    Kind of funny that it went from a Ramadan model to an Easter-post-computus model https://en.wikipedia.org/wiki/Ramadan#Beginning https://en.wikipedia.org/wiki/Ramadan#Beginning https://en.wikipedia.org/wiki/Date_of_Easter https://en.wikipedia.org/wiki/Date_of_Easter
- baq 1y ago
  Is there any other person other than Arthur David Olson who needed an RFC written to cover their retirement? https://www.rfc-editor.org/rfc/rfc6557.html https://www.rfc-editor.org/rfc/rfc6557.html

gadders 1y ago

Also, never set jobs to run at midnight (00:00) as nobody will be able to tell what day it is. Set it to 00:01 or something. And tbh, don't run jobs on the hour anyway. Everything kicks off then. Set stuff to run at 01:45 or 02:15 or something instead. (None of these suggestions are time change related)

tetha 1y ago
In general I try to use odd times for cronjobs. Backups starting at XX:13, analysis scripts at XX:23, data exports at XX:42 and so on. This simplifies the question of "Why does my system go whack at 23:23? Oh, wait".
- kibwen 1y ago
  With enough precision in your timestamp, you can base-10 encode arbitrary binary data and shove it in your nanos for debugging.
- ksenzee 1y ago
  This is a great tip. One of our QA engineers noticed recently that a particularly difficult-to-reproduce bug was affecting records saved at :21, which is when a particular cronjob hits. Without that extra info we’d probably still be scratching our heads over it.
- gadders 1y ago
  That's a good idea.
deleted 1y ago
[deleted]
ericpauley 1y ago
Honestly haven’t seen 00:00 cause confusion outside of “dog ate my homework” scenarios. If you’re using 12h clock then clearly it’s a bigger issue.

ttz 1y ago

> Alternatively, where possible set the server timezone to UTC so that no daylight savings changes will happen at all this is the way

bobmcnamara 1y ago
Use TAI

ldoughty 1y ago

I try to avoid crons at the top of the hour, partly because of this... but also because (in shared / serverless infrastructure) I assume a lot more people are setting their crons for 'on the hour' so there's more resource contention... I also aim for 'after 4am' where I can as well, or 'before midnight', to avoid this whole range.

sgc 1y ago
To incrementally improve that tactic, systemd has RandomizedDelaySec, which is a convenient way to reduce the possibility of scheduling conflicts.
- tetha 1y ago
  I prefer to combine this with FixedRandomDelay=true. FixedRandomDelay ensures that the randomized delay is an arbitrary number up to RandomizedDelaySec, but it is deterministic per server and timer. I find this useful because this means the timer will always run at XX:12:45 on server01, always run on XX:06:23 on server02 and so on. This combines very simple configuration, while being predictable and spreading out timers well.
layer8 1y ago
One trick for cron is to prepend the actual command in the crontab with something like sleep $(( $(od -N1 -tuC -An /dev/urandom) % 60 ))m ; which will delay it by 0 to 59 minutes at random.

throwaway106382 1y ago

bruh, just use UTC

bobmcnamara 1y ago
Ugh gross. Same non monotonic time problems just not every year. TAI4LYFE!

jedberg 1y ago

I've told this story before, but it's super relevant here. When I set up all the servers for reddit, I set them all to Arizona time. Arizona does not have DST, and is only one hour off of California (where we all were) for five months, and the same as here the rest of the year. I didn't use UTC because subtracting eight when reading raw logs is a lot harder than subtracting one. They use UTC now, which makes sense since they have a global workforce and log reading interfaces that can automatically change timezones on display.

latchkey 1y ago
Computers can do things. Build a UX to read the logs and have it automatically parse/convert the logs to show whatever is local time. https://unix.stackexchange.com/questions/274816/how-to-convert-a-timezone-in-logs-as-they-are-being-tailed https://unix.stackexchange.com/questions/274816/how-to-conve...
- jedberg 1y ago
  There are many solutions, but when you're actually running a site with millions of daily visitors and one person focused on ops, you do the easy thing, not the right thing, unless those happen to coincide.
  - latchkey 1y ago
    Sure... "I'm too busy fighting fires to come up with a solution" is always a valid answer, but that's not what I was replying to.
    - lxgr 1y ago
      Even with infinite resources (love to see that in real companies!), any such solution adds complexity, which comes at a cost.
      - latchkey 1y ago
        If this is a complexity battle, I'd rather that sysadmins have to use a tool to read logs vs. figuring out why the server is melting down because all the jobs are running 2x or answering questions about why jobs didn't run at all. In my old gig, jobs ran for many many HOURS and consumed most of the IO on the server (processing reams of data), which meant that when you got them running 2x because of overlap, it was a trainwreck.

raggles 1y ago

I have a job that deliberately runs at 2am and 3am... to update the time in a bunch of really old PLCs for DST. And check that every other device on my telemetry network has correctly updated its time.

PhilipRoman 1y ago

I read this a while ago and was looking forward to the mess that would happen with these default Alpine configurations (after setting non-UTC localtime) 0 2 * * * run-parts /etc/periodic/daily 0 3 * * 6 run-parts /etc/periodic/weekly It appears that busybox cron is smart enough to handle it and only ran the jobs once.

latchkey 1y ago
Google's AI and ChatGPT answer disagrees. https://www.google.com/search?q=busybox+cron+dst https://www.google.com/search?q=busybox+cron+dst "BusyBox’s crond is a lightweight implementation designed for embedded systems, so it doesn’t have all the timezone/DST logic of full vixie-cron or cronie. It just reads /etc/TZ or the system timezone at startup and then wakes up every minute to check if something should run."
- PhilipRoman 1y ago
  https://github.com/mirror/busybox/blob/master/miscutils/crond.c#L1093 https://github.com/mirror/busybox/blob/master/miscutils/cron...
- moderateclimate 1y ago
  Please don't do this. AI is not a source and is frequently wrong (as it is in this case).
  - latchkey 1y ago
    You're right. It was honestly just a low effort semi-sarcastic reply that I wasn't really expecting anyone to take seriously. I should have just gone and read the source code as PhilipRoman did below.

matja 1y ago

Seems to me the real problem here is not the timezone (there's legitimate business needs to run something daily at a specific localtime...) but having multiple instances of a cron job that overlap, in which case it should wait until the previous is done or not start at all. At least prefix a job with "flock -n" if it doesn't/can't handle that.

tonyhart7 1y ago

DST is such a dumb mechanism anyway just use UTC

deleted 1y ago

[deleted]

skopje 1y ago

UTC also solves plotting daily time plots in zones with time changes (twice a year when the plots make no sense). But there still isn't no good fix for leap year plots!

bobmcnamara 1y ago
Just use TAI?

meonkeys 1y ago

anacron is worth a mention here for sometimes-on computers like desktops, laptops, or personal servers. It's great for ensuring something runs at the set interval, or just whenever the machine comes back online. Minimum anacron interval is one day, so this can't simply replace cron. cron might even be the most sensible thing to use to kick off anacron on your system, and it might already be in use (see if your /etc/crontab fires off anacron). anacron won't have the OP's 2am/3am issue because it locks jobs while they run, so multiple job spawns should be no-ops. Still seems like it would be good practice to, e.g. use UTC for the host system and do whatever else might help avoid the unnecessary re-spawns.

litoE 1y ago

I run a cronjob at 2:30AM every day that uses dirvish (a perl wrapper around rsync) to perform backups of several computers. In the fall, at 2AM the clock is rolled back to 1AM. The cronjob runs at 2:30AM, which is 25 hours after the previous day's run. That's not a problem. However, in the spring, at 2AM the clock jumps to 3AM. Vixie cron seems to be smart enough to run all the cron jobs that were scheduled between 2M and 3AM at that point. However, dirvish is started but exits without running the backups! After some troubleshooting I concluded that dirvish does not make the backups because the previous day's run happened less that 24 hours ago (because that day was shortened by the change to DST) so it sees no need to run a backup yet. This problem would persist even if I moved the cronjob to a different time because it has to do with the day having only 23 hours. I solved this by adding a second entry to the crontab that runs at 3:45AM only on the 2nd Sunday in March.

deleted 1y ago
[deleted]

tomkaos 1y ago

I avoid cron job at this time for this reason and for many other random problem because 3:00 am seem to be often choose for a maintenance like automatic update, server reboot, database backup..

tclancy 1y ago
Can be tricky: an old project had a job that ran every 15 minutes to poll data from a popular smart thermostat company. I always knew when DST was starting or ending because that job would throw an error email at 2/3am on those days.

hinkley 1y ago

2 am is mid afternoon for the Pune office. If you’re running a 24/7 operation and all of your employees are in PST you’re gonna have a bad time. But the other thing that has changed since 2013 is that most of us are using autoscaling. There were times of day when there is supplemental server capacity to run periodic processes, and it made sense to schedule them during quiet daylight hours if they existed, but now the availability of bus numbers if and when a problem happens are more pressing. And they all outweigh concerns about DST. Instead of seeking the low traffic times of day you mainly need to avoid peak traffic. The batch processing system I renovated to use 7% if the CPU hours of the original design, used the same fixed parallelism trick that crawlers use so as not to knock over your sites. They keep k requests in flight at all times and if your server responds faster, they issue more requests per second. If the production services had excess capacity the job would run in ten minutes. If response times are slow it takes 15. You’re inverting Little’s Law and keeping the amount of servers fixed and varying the duration. It’s a pity that spooling up more servers to handle that job took a substantial fraction of that ten minutes or I could have gotten it to five (originally it was over 30 minutes and climbing toward 35). I tried at one point to move it entirely to a Bamboo agent but we underprovisioned them. If I’d had an agent with 2x the CPUs I could have shut down a service. I was on my way out the door before I realized that upgrading the bamboo agents would have cost us less than running that service and everyone’s builds would have been faster as a consequence. And probably let someone else do the same. False economies.

brettgriffin 1y ago

> Alternatively, where possible set the server timezone to UTC so that no daylight savings changes will happen at all. Is this what people actually want, though? I'm actually working on a scheduled process right now deployed on a cloud scheduler that does not adjust for daylight saving time (it literally says "does not take into consideration daylight saving time" next to the time input). Everybody wants the job to run four hours before the work day starts, every day of the year. I will have to change it on Sunday to ensure the job completes at the time the end users expect it. And again in March. Am I missing something?

monster_truck 1y ago
I call this "rake stomping", as long as you're running back and forth in the same places every time, you'll never notice.
waisbrot 1y ago
The article is mostly talking about using the `cron` scheduler and running things at 2am on Sunday because there's especially low traffic. These cron jobs are "I don't care when it runs, but it should have minimal chance of causing problems." Your use-case is totally different: "I want this job to run at this time" so the only lesson that applies to you is that the `cron` utility might behave weird during DST switches. No idea if it underlies your cloud provider, so it may be completely irrelevant.
evil-olive 1y ago
> Everybody wants the job to run four hours before the work day starts recurring events like this - ones that take place relative to humans in a given location - are the most common snag related to "just use UTC everywhere" advice. but they're not actually in conflict at all. they just require an extra layer of indirection. let's say this is a factory that starts early in the morning, 06:30 local time, because then "four hours before" falls into the window affected by DST. "run at 02:30 local time" is what you want to avoid, because on the "spring forward" day, 02:30 simply does not exist. the clocks jump from 01:59:59 to 03:00:00. [0] likewise, "run at 01:30 local time" is to be avoided, because on the "fall back", that occurs twice. ideally, you want to think of the recurrence as being associated with a given location, and that recurrence generates individual events, which happen in UTC time. for example, if the factory is in San Francisco, the recurrence might specify "once a day, at 02:30 SF time", which would generate a series of daily events in UTC time. during the summer those events are at 09:30Z, during the winter they're at 10:30Z. you also want the scheduler to be smart enough to understand the "once a day" constraint - so that a "once a day, at 01:30 local time" task generates only one daily event, rather than two, even though 01:30 local time happens twice in the same day. and then, because the individual events are all in UTC, and the system clock is as well, there's no ambiguity in the actual execution time of the event. the complexity of the recurrence is handled at a higher level. if you want an even harder version of this problem, imagine a team in SF and a team in London, that have a Zoom meeting every Monday at 8am (SF) / 4pm (London). it's necessary for one of those locations to "own" the recurrence, because the UK does their time changes 1-3 weeks offset from the US [1], creating weeks where it isn't a straightforward 8 hour time difference. 0: https://www.timeanddate.com/time/change/usa https://www.timeanddate.com/time/change/usa 1: https://www.timeanddate.com/time/change/uk https://www.timeanddate.com/time/change/uk

p0w3n3d 1y ago

It depends on tz. 4 am in some countries too

sedatk 1y ago

Redundantly running cron jobs are the least of our problems. Daylight savings adjustment day causes surge in deaths and ER visits[1]. Daylight savings needs to go. https://www.sciencealert.com/daylight-savings-time-change-kills-people https://www.sciencealert.com/daylight-savings-time-change-ki...

HiPhish 1y ago

DST was a mistake and it needs to die. It solves no problems and only creates them. And not even the type of problem that someone could profit from, just plain old complete waste of time for everyone problems. Just stick to winter time (because that's the correct one) and adjust all working hours once and for all to be like summer time, for e.g. instead of business opening at 7AM and closing at 10PM they can open at 6AM and close a 9PM. There will be a period of adjustment, but we have a period of adjustment twice a year already, so nothing is lost. Why match working hours to summer time? Because I want to have more sunlight by the time I'm done with work. Especially during winter. We have it completely backwards, if we are going to adjust our clocks we should adjust them in a way that gives us more sunlight during winter, not less. I don't care if it's dawning by the time I'm going to work where I will be stuck indoors for nine hours, I want sunlight when I'm free again.

StopDisinfo910 1y ago
So you propose we stop shifting the clock but shift every opening and closing times instead producing exactly the same effect with basically no difference. And you want to abolish DST but the end of your comment is that you actually want even more shift including in winter. I fear I have trouble following your reasoning and understanding what you actually want.
- butlike 1y ago
  There's more clocks than storefronts
- xigoi 1y ago
  If I’m reading the parent correctly, they Want to switch to non-summer time and adjust opening hours to be effectively like summer time, forever.
  - hsbauauvhabzb 1y ago
    I wonder what the earliest sunrise time would be as a result, 2am?

cmiles8 1y ago

Or just use UTC and stop using timezones in code. I get it’s not that simple for some use cases but UTC exists for a reason.

bobmcnamara 1y ago
Use TAI. UTC has most of the same wacky timezone problems, just less often.
- ericpauley 1y ago
  You’re signing up for a world of pain in the real world, where every other system is UTC. Just use UT1 or smear the leap second if you really can’t stand the risk.
  - bobmcnamara 1y ago
    Approaching one second per second, is that so much to ask? Unless standardization has improved, how+when the second is smeared(if at all) means timestamps from the same system aren't usefully subtractable without communicating a separate time skew calendar. That said, time is a big topic, and everything has different requirements but migrating to TAI and treating UTC as yet another human readable time zone has solved more (admittedly application specific) problems than it created.

MayeulC 1y ago

Not a crown job, but I configured my lights to turn on progressively at my scheduled wake up time, using Home Assistant. I figured that scheduling lights to turn on at <0 AM + wake up time> was a good approach... We got a surprise early wake up last year, so I ended up changing the formula to <4 AM - 4 hours + wake-up time>.

cadamsdotcom 1y ago

Great to read about something that's a solved problem now. It's been a very long time since I used a server that wasn't set to UTC.

bobmcnamara 1y ago
I've seen this same problem under UTC. Gotta use TAI

ckastner 1y ago

This was worked around ages ago in OpenBSD, and the workaround was already included in Debian (and by extension, Ubuntu) when I started maintaining it in 2010 (I no longer do). Here's a link to a patch [1] from a version from when the package still kept standalone patches. It's been a long long time and I honestly don't remember the details, but Debian's cron(8) still says [2]: Special considerations exist when the clock is changed by less than 3 hours, for example at the beginning and end of daylight savings time. If the time has moved forwards, those jobs which would have run in the time that was skipped will be run soon after the change. Conversely, if the time has moved backwards by less than 3 hours, those jobs that fall into the repeated time will not be re-run. Edit: According to this bug report [3], this workaround first entered Debian in 1999. [1]: https://sources.debian.org/src/cron/3.0pl1-137/debian/patches/features/Better-timeskip-handling.patch https://sources.debian.org/src/cron/3.0pl1-137/debian/patche... [2]: https://manpages.debian.org/unstable/cron/cron.8.en.html https://manpages.debian.org/unstable/cron/cron.8.en.html [3]: https://bugs.debian.org/8499 https://bugs.debian.org/8499

silverwind 1y ago
So we can assume the author was not running a distro that had "fixed" the issue?
- ckastner 1y ago
  Most likely, yes. The author mentions vixie-cron, which was the name of the project before Paul Pixie joined/founded(?) ISC, and it was released as ISC from after. Debian's fork is still based on vixie-cron, but it couldn't have been the one because of the aforementioned patch.

jcynix 1y ago

> better open source job scheduler [...] one that will ensure no overlapping runs Or learn to use lock files, which would make sure that your jobs don't run when another one is already running? Unix shells offer the trap command, among other things, to cleanup, even in the case of errors, e.g.: set -e LOCK=$(ensure-lock) trap "rm $LOCK" 0 1 2 15 do-job https://en.wikipedia.org/wiki/Semaphore_%28programming%29 https://en.wikipedia.org/wiki/Semaphore_%28programming%29

TylerE 1y ago
Tell me how that goes after a server malfunction or power failure. Where is your trap god then?
- jcynix 1y ago
  The topic was to prevent overlapping runs. In case of power failures etc all your server processes have this problem. That's why, for example, /var/run exists and files/directories are deleted from /var/run on reboot?

magarnicle 1y ago

The problem is not so much which timezone you pick, the problem is that cron uses 'naive' datetimes. If the timezone is explicit then you don't have duplicate hours because one is 02:00 and one is 02:00DST. And then you can translate that time to any other timezone with ease, knowing that you are referencing a specific hour in physical time that only ever happened once.

magarnicle 1y ago
Which is to say I wish cron had the TZ variable that Jenkins uses in it's scheduling system. And the H value to easily spread multiple jobs over the hour/day/month etc.
LeoPanthera 1y ago
> knowing that you are referencing a specific hour in physical time that only ever happened once "Falsehoods programmers believe about time" This assumption can break with: * Leap seconds * Calendar and time zone changes * (In exotic circumstances) Relativity
- magarnicle 1y ago
  I'll give you leap seconds, but at least in Python historical timezone changes are recorded in the datetime library, so the timezone at any particular time in the past has the timezone offset that was active then.

atoav 1y ago

If you want to avoid overlapping runs simply simply have the task check for the existence of a lockfile, if not create a lockfile containing the start timestamp, run the task and delete the lockfile at exit or error. The timestamp gives you a chance to recognize stale/orphraned lock files in case of crashes based on their age.

loodish 1y ago

I like cron jobs that run during the work day. If something goes wrong I would prefer to handle it at 10am rather than 10pm. Why make life unpleasant? If it really needs to run during a low load period I line it up for 6am work time. Again so that if something goes wrong it leads into the work day rather than disrupts the on-call too badly. 6am also provides a daylight savings buffer either way so you don't have to care. Recent work has all been on 24/7 operations though.

dathinab 1y ago

2-3am is bad due to DST but there is another reason to also avoid 3-5am or so around 3am is the time where "in average" human attention is the worst as its where "in average" their most relevant deep sleep phases are now I guess the average over HN readers has a good chance to be somewhat later but either way if the job goes wrong and triggers alarms 2-5am it very bad for the the health and sleep rithm of whoever needs to fix it and the most likely time they make mistakes it sadly is also the time where non-international interruptions are least likely to matter if you are an international company having international teams can be an solution and if you are a very small team adopting job times to an overlap of low usage and your admins sleep habits can be an option but if you need to pick a generic time and don't provide for logistics or other "start work at 4/5am" companies probably 5-6am is a better time to run your batch jobs anyway just some random thoughts

tanin 1y ago

The details I never really thought about until I encountered timezone issues was: There are points in time that don't exist, and there are points in time that are ambiguous. If we adjust the time forward by 1 hour, then then 2:30am doesn't exist. If we adjust time backward, then 2:30am is ambiguous. 2am-3am is only safe for certain countries. America/Santiago (Chile), for example, adjusts their daylight saving on midnight. On a certain day, midnight doesn't exist because it immediately jumps to 1am. If you are building an analytics chart by day, you will encounter this issue. I wrote about several weird time zone edgecases here: https://tanin.nanakorn.com/edgecases-for-timezones/ https://tanin.nanakorn.com/edgecases-for-timezones/ To add more confusion, IIRC Java and Ruby handle these cases differently where Ruby raises exceptions (e.g. InvalidTime and AmbiguousTime) but Java doesn't. I caused 3 incidents in a row at Stripe in my first month because of this. I thought I was gonna be fired. Luckily I wasn't...

sharts 1y ago

Why run crons at that time anyway? It seems the most logical times for things are going to be midnight or beginning/end of business day.

ray_v 1y ago

Here's my radical time idea: instead of keeping the same schedule year round, we just incrementally chop parts of the day off (we just put that time in a "bucket" and pretend it doesn't exist). Then at the summer equinox we empty the time bucket into a 6 day bender of a party. /s

tsoukase 1y ago

For the most part of Europe and USA the day is either "quite" long (summer) or quite short (winter) due to the curve of the duration. Having light for long summer evenings and dark for long winter mornings is bad for many reasons, one of witch is seasonal depression, with which the Scandinavians suffer and fight. I was for the abolishment of DST until COVID cancelled it in Europe but now I am thinking it again and probably I am against.

Zanfa 1y ago

While not always possible, in general, any mission critical scheduled job should be idempotent. This avoids so much headache, especially when there's a queue involved somewhere.

nubinetwork 1y ago

Doesn't systemd timers fix this? AFAIK, a job set to run once a day shouldn't run a second time because the time rolled back.

tokyovigilante 1y ago

Whatever argument you may have against DST, “my cron jobs won’t run twice a year!” is not a strong one.