5 ms·
I worked at a place that had set the servers to BST not UTC and used cron jobs extensively for reporting, which caused chaos twice a year and twice a year they
by noir_lord 1y ago
I worked at a place that had set the servers to BST not UTC and used cron jobs extensively for reporting, which caused chaos twice a year and twice a year they had to explain to the business again why it caused chaos and ask for the time to fix it.
Company went bust before it got fixed.
Just use UTC folks unless you have a really good idea why you shouldn't.
- lxgr 1y ago"Our customers aren't in UTC" is often a compelling reason. People generally expect things like usage limits to reset overnight, not at UTC midnight, and these are often implicitly tied to (the result of) batch jobs. Financial jobs, including billing, is another big category.
- latchkey 1y agoAlways set the servers to UTC. Convert the time to local time for display, but store in UTC. The reporting job should be run from a server on UTC and on UTC time. The report itself can convert to whatever is local time.
- gruez 1y agoThat doesn't fix the problem of "my quota reset (or reporting job) gets shifted by 1 hr when daylight saving time changes".
- latchkey 1y agoThe reporting job should be run from a server on UTC and on UTC time. The report itself can convert to whatever is local time. This is exactly what I said above.
- lxgr 1y agoThat works if all your consumers of the report are indifferent to it arriving at a different time half of the year. All of yours might well be, but you can't generalize that assumption to every domain and scenario.
- ipaddr 1y agoIf you want an email of the report to send out at 9am locally. Generate before and send to everyone based on timezone.
- kjehjrktlhl 1y agoYou don’t want a report to include 25h or 23h twice a year just to be able to send it at some distinct local time.
- lxgr 1y agoI absolutely want some daily business reports to cover 25 hours on one day of the year and 23 on another, since that's how long a calendar day is in local time in every place that has DST shifts.
- lenzm 1y agoYes, that's exactly what I want. Everything shifts with dst - business processes, people's working hours, ach windows, ...
- gruez 1y ago> Generate before and send to everyone based on timezone. How are you going to do this? Clearly cron (with timezone set to UTC) doesn't work for this so you'll need another system. In that case why not have that multi-time zone scheduling system run your script?
- latchkey 1y agoThe report can run at whatever time you want it to run at. But this solution solves the issue of the report running 2x or not at all.
- ranger_danger 1y agoYou could have the reset script check the timezone of the user before actually performing the reset, and have this script run at least once an hour to make sure you get close to accurate for most people (some timezones only differ by less than an hour though!). But also, if the user can change the timezone themselves, now you have a condition where the user could keep their quota at bay for a very long time (if not forever) if they keep changing their timezone accordingly.
- throwaway150 1y ago> That doesn't fix the problem of "my quota reset (or reporting job) gets shifted by 1 hr when daylight saving time changes". How hard is to convert the UTC time to user's local time and perform quota reset? I mean you should be doing this anyway. Even if you store everything in PST or EST or CST, you never know when you'll get customers from another part of the world and then you'll have to do this anyway. So why not do this already?
- lxgr 1y ago> How hard is to convert the UTC time to user's local time and perform quota reset? Probably roughly as hard as writing a timezone-aware scheduler that considers all edge cases around daylight savings time, i.e. probably possible but I'd try fairly hard to see if I can get around it. One way of getting around it is to run your batch jobs in your "primary business timezone". Sure, you might outgrow that concept eventually, but many companies never do, and in that sense "running on UTC" seems similarly aspirational in spirit (albeit at a much smaller scale) to starting with high scalability.
- gruez 1y ago>Even if you store everything in PST or EST or CST, you never know when you'll get customers from another part of the world and then you'll have to do this anyway. So why not do this already? 1. allowing users to change their timezone and tying it to when their quota reset sounds like a great way to add more edge cases and complexity. For instance, does messing with your time zone mean you can get your quota reset multiple times a day? 2. Not everyone operates some sort of global b2c SaaS that's timezone agnostic. For many enterprise backoffice tasks "6am HQ time" is far more reasonable than "6am/7am, dunno depends on whether daylight saving time is on or not". In this case having a server set to the HQ's timezone makes far more sense than doing UTC and trying to work around daylight saving issues.
- cweagans 1y agoDo you actually care if the requests were made on some specific calendar day? Or do you just want to make sure that heavy users are paying more and/or people aren't abusing your system? Instead of tying your quotas to calendar days in some specific time zone, tie them to a rolling 24 hour usage period. Even better: use a rolling hour.
- freedomben 1y agoHaving tried many times to use local time instead of UTC to make life simpler for myself, I really don't think it's ever a good idea. Better to localize times for users on the client-side based on their local settings, IMHO For quotas and usage limit resets it can be a little harder, but if it's a cron job anyway, my initial approach would be to just schedule it for a time that is close to midnight for the area your customers are in.
- Kwpolska 1y agoThat does not require setting the server to local time. Run the cron job every hour. Before starting the script, check the time and compare it to the last saved execution time. If the day has changed, do stuff and save the new date, else exit. This also happens to be more resilient to server downtime around midnight.
- lxgr 1y ago> Before starting the script, check the time and compare it to the last saved execution time. If the day has changed, do stuff and save the new date, else exit. That sounds a bit like reimplementing a scheduler inside the scheduled task (which is ok if you don't trust your scheduler, but I'd avoid it if at all possible).
- bell-cot 1y agoExcept downtime can start during execution. So for many types of jobs... - Check if run today, exiting on "yes" - Run - Update the last-run-on date
- latchkey 1y agoWhat is "today"?
- bell-cot 1y agoHopefully just a datetime recent enough that the job does not currently need to be (re-)run. But as soon as one job needs another to complete first, or two jobs shouldn't be run at the same time - things start getting messy.
- lxgr 1y ago> Hopefully just a datetime recent enough that the job does not currently need to be (re-)run. I can think of several ways to mess that seemingly trivial calculation up just from the top of my head. (Consider changing timezones, increasing the interval the job is running to within half a day of scheduling period etc.)
- skissane 1y ago> "Our customers aren't in UTC" is often a compelling reason. This doesn’t work as soon as you become a global business… because if you tell a customer in Asia that it resets at midnight in some US timezone, what are they going to think? Many people will accept UTC because it is the international standard. Everyone will accept “let each customer pick the timezone that works best for them”. Saying “I’m going to use a US timezone because that is easiest for our US customers” risks sending a message to your (now or future) global customers that you view them as second-class citizens.
- lxgr 1y agoReally depends on your market and customers. Especially in the financial industry, local time zones (often of a given currency's central bank's main branch, a primary regulator etc.) are often very, very entrenched. > Many people will accept UTC because it is the international standard. I really wish that were true universally... And I do think that UTC is actually somewhat Eurocentric! From personal experience, it's significantly easier to mentally add/subtract one or two hours (modulo 24) than 7 or more. > Saying “I’m going to use a US timezone because that is easiest for our US customers” risks sending a message to your (now or future) global customers that you view them as second-class citizens. Definitely, but on the other hand, I do think that picking your head office's time zone as the "canonical" one for resetting quotas etc. has some merit as well, if only because it makes the concepts of "today" and "tomorrow" work a bit better. (UTC midnight might be very unintuitive to both you and your customers.)
- skissane 1y ago> Definitely, but on the other hand, I do think that picking your head office's time zone as the "canonical" What happens when you move your head office to a different time zone? e.g. Oracle moved their HQ from California to Texas (and have announced a further move to Tennessee, although last I heard that still hasn’t happened) What happens when an M&A happens and you are now part of a much larger enterprise with a different HQ timezone, and systems with wildly inconsistent configured timezones due to having acquired multiple companies which had HQs in different time zones and all of which configured everything to use their HQ time? > UTC midnight might be very unintuitive to both you and your customers I live in eastern Australia, UTC+11 at the moment (UTC+10 during Southern Hemisphere winter). Right now, resetting stuff at UTC midnight means at 11am my time, resetting stuff at US Eastern midnight means at 3pm my time - roughly equal in potential inconvenience, but I’ll be much more forgiving of the first than the second. > And I do think that UTC is actually somewhat Eurocentric! From personal experience, it's significantly easier to mentally add/subtract one or two hours (modulo 24) than 7 or more. For me, add/subtract 10 is pretty easy… and then just one more during DST. Does that make UTC “Australia-centric”? Or Guam-centric or Vladivostok-centric? (both of which are also UTC+10)
- latchkey 1y agoAt Marin Software, they ran each customer on their own set of servers and set the servers to the customers defined TZ. It was an endless cronjob nightmare. Now they are also bankrupt. Just found out some guy decided to try to restart the company. Good luck. https://x.com/Austen/status/1981904435539280324 https://x.com/Austen/status/1981904435539280324
- ta1243 1y agoIf I want a report of what happened at a specific time I need that report at local time I get a daily status report of various things from our 24 hour operational management team which runs at 8am UK time every day. That means last week it ran at 0700UTC and this week at 0800UTC This is built around operational events, shift changes, etc. I've got another system which is in operation in Sydney from 0630 to 1630 local time, this means that maintence windows which overlap with UK shift patterns depend on the week but mean the system is operating 2130-0730 UK time at some times, 2030-0630 UK at others, and 1930-0530 at others. UTC is not "the answer". Sometimes you want things running at a UTC time, sometimes you want them running at local time. I have a regular meeting at 10am London time on Tuesday and Thursday. That can't be stored in UTC as it varies depending on the time of the year. It has to have the timezone stored and actioned.
- latchkey 1y agoYou can tell the job when to run in UTC time and change that time depending on what the current DST rules are. But if the job is set to run in UK time and the system clock jumps around because of TZ changes, it will run twice or not at all. Having a stable zero based time that doesn't move around by external forces that you can't control is "the answer."
- ta1243 1y agoSo I have to manually update the job How about I use some form of library to do it. I tell it I want to run at 0800 London time, and it runs at 0800 London time If I tell it I want to run at 0130 London time (or 0330 Athens time) I still have a problem -- do I run it twice when the clocks go back, do I skip it when clocks go forward? But that's a business logic problem, and defining it as UTC and having another job to update the time twice a year doesn't actually solve the question of "what do I do at this point".
- latchkey 1y agoYou don't have any choice in the matter. If you control the time you want the jobs to run externally from the server, then you can have control over it. Otherwise, you're getting jobs that run 2x or not at all.
- jedberg 1y ago> Just use UTC folks unless you have a really good idea why you shouldn't. If my whole team is based in Pacific Time, I'm gonna use something close to PT so I don't have to do hard math when reading raw logs. If we're all US based, I still want something in the USA. The math is easier.
- kjehjrktlhl 1y agoTechnical debt. Please don’t.
- jrockway 1y agoYeah. I log in json + unix timestamp nanoseconds, and just convert it to something human-readable on display (github.com/jrockway/json-logs). I am not sure why logs need to be "pretty" at time of logging when they can be made pretty at time of display. Doing it at time of display means you can just use local time, and then nobody is confused.
- jedberg 1y ago[flagged]
- throwaway150 1y ago> Who hurt you? Nobody hurt them. Please don't do this kind of childish rebuttals. It looks really silly on HN. If you don't want to use UTC, that's one thing. You do you. But there are reasons for sane professionals (who weren't hurt by anybody) to expect their coworkers to use UTC when possible. Because using local timezone leads to some problems including the one in TFA.
- hobs 1y agoWell me, I have been burned by fixing this UTC bug at like 8 companies and overall more than four year of my life (between other things, each project took at least six months of people poking at things) because at an eventual scaling point it starts biting your ass everywhere in every service. Just because it worked once doesn't mean it's good.
- yabones 1y agoI put a little analog clock on my desk that's set to UTC time. It helps a lot with converting logs to get a quick reference.
- lozf 1y agoPerhaps a shell alias for: TZ="Etc/UTC" xclock -analog -update 1 -norender -hl grey -fg grey -bg black (or similar) would also suffice.
- derwiki 1y agoI do the same! I am in PDT and keep a UTC and EDT clock (most of my team is east coast)
- schraitle 1y agoI have a piece of paper on my desk, each side has the time zone, utc offset, and date when DST will change. Twice a year I flip over the paper. Right now it reads "EDT, UTC-4, until: Sun, Nov 2" Going to add a clock next to it now, that's a great idea
- ponector 1y agoAnd then with distributed team you realize DST change dates are different around the globe.
- pseudalopex 1y agoDesks around the globe can have different pieces of paper.
- vdm 1y ago$ zdump -Vc 2025,2026 America/New_York America/New_York Sun Mar 9 06:59:59 2025 UT = Sun Mar 9 01:59:59 2025 EST isdst=0 gmtoff=-18000 America/New_York Sun Mar 9 07:00:00 2025 UT = Sun Mar 9 03:00:00 2025 EDT isdst=1 gmtoff=-14400 America/New_York Sun Nov 2 05:59:59 2025 UT = Sun Nov 2 01:59:59 2025 EDT isdst=1 gmtoff=-14400 America/New_York Sun Nov 2 06:00:00 2025 UT = Sun Nov 2 01:00:00 2025 EST isdst=0 gmtoff=-18000 $ zdump -Vc 2025,2026 Europe/London Europe/London Sun Mar 30 00:59:59 2025 UT = Sun Mar 30 00:59:59 2025 GMT isdst=0 gmtoff=0 Europe/London Sun Mar 30 01:00:00 2025 UT = Sun Mar 30 02:00:00 2025 BST isdst=1 gmtoff=3600 Europe/London Sun Oct 26 00:59:59 2025 UT = Sun Oct 26 01:59:59 2025 BST isdst=1 gmtoff=3600 Europe/London Sun Oct 26 01:00:00 2025 UT = Sun Oct 26 01:00:00 2025 GMT isdst=0 gmtoff=0
- nodesocket 1y agoI use UTC for all public/production servers, but for my homelab servers in my closet rack they all use local time. Makes grokking times in homelab servers much easier. The exception is database insert/update date/times which are always stored UTC.
- SecondHandTofu 1y agoGenerally, yes but I there's a surprising amount of cases when it is important, which makes it difficult to generalise e.g. Huge amounts of the financial sector cares because of market times or regulatory reasons.
- jacobsenscott 1y agoAlso don't use cron full stop. Build (or use an off the shelf) scheduler in your app that's more sophisticated, has access to data and client preferences, etc.