7 ms·
I've told this story before, but it's super relevant here. When I set up all the servers for reddit, I set them all to Arizona time. Arizona does not have DST
by jedberg 1y ago
I've told this story before, but it's super relevant here. When I set up all the servers for reddit, I set them all to Arizona time. Arizona does not have DST, and is only one hour off of California (where we all were) for five months, and the same as here the rest of the year.
I didn't use UTC because subtracting eight when reading raw logs is a lot harder than subtracting one.
They use UTC now, which makes sense since they have a global workforce and log reading interfaces that can automatically change timezones on display.
- latchkey 1y agoComputers can do things. Build a UX to read the logs and have it automatically parse/convert the logs to show whatever is local time. https://unix.stackexchange.com/questions/274816/how-to-convert-a-timezone-in-logs-as-they-are-being-tailed https://unix.stackexchange.com/questions/274816/how-to-conve...
- jedberg 1y agoThere are many solutions, but when you're actually running a site with millions of daily visitors and one person focused on ops, you do the easy thing, not the right thing, unless those happen to coincide.
- latchkey 1y agoSure... "I'm too busy fighting fires to come up with a solution" is always a valid answer, but that's not what I was replying to.
- lxgr 1y agoEven with infinite resources (love to see that in real companies!), any such solution adds complexity, which comes at a cost.
- latchkey 1y agoIf this is a complexity battle, I'd rather that sysadmins have to use a tool to read logs vs. figuring out why the server is melting down because all the jobs are running 2x or answering questions about why jobs didn't run at all. In my old gig, jobs ran for many many HOURS and consumed most of the IO on the server (processing reams of data), which meant that when you got them running 2x because of overlap, it was a trainwreck.
- lxgr 1y agoIn my experience, it's really not pleasant having to work with logs that require a dedicated viewer compared to regular old text files I can use Unix commands available everywhere (`tail`, `head`, `wc` etc), with, i.e. not just on the server producing the logs, but also the device I'm viewing them on. That said, I absolutely prefer UTC times in logfiles. I'd rather do some math every time than make wrong assumptions about the local timezone of the file even once. (Even correctly indicated local times are more annoying, since I have the math for the offset of my timezone to UTC memorized, but not that of every possible server time to mine.)
- latchkey 1y agoalias tail="my fancy log viewer tail wrapper" https://thecasualcoder.github.io/tztail/ https://thecasualcoder.github.io/tztail/
- 1y ago
- taftster 1y agoI mean, if this doesn't depict modern devops, I don't know what does. Unsung heroes, honestly.
- ocdtrekkie 1y agoThe problem is a lot of times when you are looking at logs you are already very far off the happy path of things you look at often and want to invest resources into displaying well.
- lxgr 1y ago> Computers can do things. Build a UX [...] That sounds like a job for a human (at least the review part), not only a computer.
- mananaysiempre 1y agoIn my experience, the hard part is getting everybody else to do that. And then also getting them to actually include the timezone in their communication with you.
- rconti 1y agoI'd rather the logs be consistent, and not rely on every single person who ever looks at the system logs make sure to use the correct tool the correct way.
- latchkey 1y agoIf you'd rather the logs be consistent, then log to UTC.
- VintageCool 1y agoNooo... we have a bunch of metrics and logs reporting systems at my company. Some of them are in UTC, some of them display my local time, some of them display China Time, and I'm trying to collaborate with colleagues in London and Australia who get data displayed in their local times as well. When I'm working to address an incident and combing through multiple systems to try to correlate events, it's a pain in the ass having to constantly double-check which time zone this data is in.
- kjehjrktlhl 1y agoThis is not smart, this is just surprising to the next person who is going to maintain your ”smart” tricks. Thank god they switched to utc, that is what everyone expect.
- adzm 1y agoSounds like it was smart at the time, and then eventually outgrew the solution.
- jedberg 1y agoActually, back then, 18 years ago, most people expected your servers to be in Pacific or Eastern time, depends on where your company was headquartered, because none of us really had global technical workforces back then. We all pretty much worked in one office and used the local time zone, because often our servers were in the building with us or in a datacenter nearby. Case in point, before reddit I was at eBay, and we kept all those servers in Pacific time, since the entire technical workforce was in Pacific time, as well as all of the servers. Making blanket statements like that without considering the context of the time is usually not a good idea. ;)
- aerostable_slug 1y ago> most people expected your servers to be in Pacific or Eastern time I was there back then, working for shops people have heard of, and I honestly don't know where you're getting this idea from. Some places did things wild and wacky when they were wee small, but most of us quickly learned that such shenanigans (like fun server naming conventions) start to fall apart and maybe we should do things differently.
- latchkey 1y agoIt only matters if the servers were running cron jobs where it mattered if they ran "not at all or 2x." Logs with weird dates on high demand production servers... less important.
- tedivm 1y agoUsing UTC for servers was standard when I entered the field in 2005.
- adzm 1y ago> I didn't use UTC because subtracting eight when reading raw logs is a lot harder than subtracting one. Also important here is that the date stays the same! Even though I've gotten used to instinctively decoding UTC, that part is still frustrating sometimes.
- aerostable_slug 1y agoI've had to try to clean up after a similar early decision and it was very painful. Please stick with utc across the board people, someone someday may have to clean up your mess.
- jedberg 1y agoNowadays I'd agree with you, UTC is probably the best bet. But back then, it wasn't.
- deleted 1y ago[deleted]
- rgbrenner 1y agousing utc on servers was very common in 2005
- tonyarkles 1y agoI’d say it was common enough but not universally, given the number of arguments I had from 2005 to 2015 about this exact issue.
- jay_kyburz 1y agoHold on, I'm not a sysadmin guy. Are you folks saying the server should not know what part of the world its in, that basically it should think it's in Greenwitch? I would have thought you configure the server to know where it is have it clock set correctly for the local time zone, and the software running on the server should operate on UTC.
- katabatic 1y ago1) Most software gets its timestamps from the system clock 2) If you have a mismatch between the system time and the application time, then you just have log timestamps that don't match up; it's a nightmare - even more so around DST/ST transitions
- devsda 1y agoJava allows setting the default timezone at jvm level and everyone in our org were setting their favourite TZ which was mostly PST somewhere in the code. We had application logs and system level logs with different timestamp and someone decided a certain db column has to be a string of the format 'yyyy-mm-dd hh:mm:ss". You can imagine the confusion and the "fun" time we had while debugging logs from multiple systems at specific times. Finally, one lead put their feet down and mandated that setting timezone to PST should be the first thing all applications do and db columns should be considered PST unless otherwise required.
- mulmen 1y agoDoes that mean you ran PST during daylight savings time and half the year your logs were “off” by an hour? Or did you actually run Pacific Time and deal with the clock changes twice a year?
- dpoloncsak 1y agoDoesn't the JVM handle this when you set the tz? Otherwise...how is it different than just setting a clock?
- array_key_first 1y agoThey're two different timezones. One is a UTC offset and one is a dynamic timezone (DST).
- skissane 1y ago> Finally, one lead put their feet down and mandated that setting timezone to PST should be the first thing all applications do and db columns should be considered PST unless otherwise required. That seems to me like a really bad mandate. If you are going to mandate something, mandate UTC. If someone forced me to change a system from UTC to something else, I’d not be very happy. It’s the kind of decision which makes you seriously question if you want to work there any more.
- lxgr 1y agoClosely related pet peeve: Log display web UIs that allow selecting display timezones including UTC absolutely insist on deriving my preferred time format (12/24 hours) from my browser language preference. If nothing else, selecting UTC should be a very strong hint to any UI that I am capable of parsing YYYY-MM-DD and 24 hour times.
- xmilka 1y ago[dead]
- webstrand 1y agoThat is the correct and only sane way to determine date format, timezone is not the same as formatting preference. But yeah it sucks. Assuming you're using the en-US locale, have you tried using the en-CA locale? It has ISO8601 formatted dates, and is otherwise pretty similar to the en-US locale. In firefox you can try it out in about:preferences by setting en-CA as the topmost option in "Choose your preferred language for displaying pages" And I just checked, firefox is capable of understanding system-wide split locale settings, if you only want LC_TIME localectl set-locale en_US.UTF-8 localectl set-locale LC_TIME=en_CA.UTF-8 and dates will format, but currencies, addresses, etc will not
- lxgr 1y ago> That is the correct and only sane way to determine date format, timezone is not the same as formatting preference. That's unfortunate, but then wouldn't the sane way for localization-aware software be to ask the user and not make any such assumptions? > Assuming you're using the en-US locale, have you tried using the en-CA locale? It has ISO8601 formatted dates, and is otherwise pretty similar to the en-US locale. Thanks, I'll give that a try tomorrow! If my log exploring UI of, well, not quite choice actually respects that, I'll be very grateful :)
- webstrand 1y agoIn a way it is asking the user. Date formatting rules are traditionally tied to locale, so the user picks their locale and their expected date formatting rules are derived from that. On Linux you can mix and match via LC_*= environmental variables, but that appears to be complexity the browser vendors didn't expose in their UI.
- creichenbach 1y agoThat's still a bit risky as Arizona might just change its time zone definition on a whim. I'm an engineer on one of the big calendaring applications, and it's mind-boggling how often stuff like this happens world-wide, sometimes short-notice (a few weeks in advance). It regularly gives us headaches.
- jaggederest 1y agoAgreed, watching the rate of changes to timezone databases will rapidly disabuse you of the notion of any constant. It's rare that a day goes by without an update to some definition somewhere, which is astounding.
- nxobject 1y agoIt might be instructive for PHBs to have a 'hasatimezonechanged.com' website counting the days since the last timezone DB change...