5 ms·

We just spent multiple hours in a team of multiple people debugging this issue. Systemd doesnt work. Checked all disks, fstab, recovery system, etc. and there w

We just spent multiple hours in a team of multiple people debugging this issue. Systemd doesnt work. Checked all disks, fstab, recovery system, etc. and there was nothing clearly wrong. Turns out something in proxmox (maybe a service?) doesn't understand daylight savings time (Dublin). The only way to know was to google "proxmox systemd 100% cpu" and find above post. Christ. Edit: The fix, of course, is `ln -sf /usr/share/zoneinfo/Etc /etc/localtime` Edit 2: Looks like that just unsets the timezone. It's too late for me to find the real fix, but you basically want to set the timezone to something else (like UTC).

gjvc 4y ago
servers should always be UTC
- LeoPanthera 4y ago
  It is entirely reasonable for a personal server to be set to the timezone it is physically located in.
  - lionkor 4y ago
    I'm no expert, but if your software can't deal with timezones, be worried. So yes, absolutely, it should be perfectly okay to use your local timezone.
    - bcrl 4y ago
      Timezones are disaster. Once upon a time it was difficult and time consuming to make changes to them. Then the world began developing more and "accurate" timekeeping devices. Atomic clocks were developed and began to be miniaturized. Then someone had the brilliant that incredibly accurate clocks could be used to determine location. GPS was born. A few decades pass and we now all have highly accurate "clocks" that we carry around in our pockets that receive GPS signals and sync their idea of time to what the satellite thinks. Sounds great, right? Except, humans don't actually use atomic clocks as the basis for their timekeeping. We look outside and expect the time on the clock to mostly match with the position of the sun in the sky. Once leap seconds became entangled in them, the whole thing basically a toxic pile of radioactive waste. Timezones are now being updated multiple times a year since the internet means we can update them at will. The whole thing is now a giant mess of toxic radioactive waste that can never be correct for any extended period of time. We, the software industry, did this to ourselves. We built this, we have to live with it, and it's never going away.
      - fsckboy 4y ago
        computers are good at keeping track of things that humans aren't. tracking timezones is in your computer's wheelhouse, don't be scared of it.
        bcrl 4y ago
        I'm not scared of it. It's unnecessary excessive complexity that we have foisted onto ourselves. It has caused countless bugs and will most likely cause countless more before I die. Show me the software that copes with timezone changes at runtime.
        nikau 4y ago
        humans having to convert times from UTC to local timezone will also result in countless hours wasted in troubleshooting.
        Spivak 4y ago
        Every phone when people travel? It really isn’t that hard. What makes you pull your hair out is dealing with 3rd party code that handles it incorrectly. Do everything internal in UTC or always attach a timezone to dates, your choice. When you cross the boundary either ingesting naive dates or displaying them read the system time zone again and convert. I mean it’s kinda annoying but “naive” times is the time equivalent of storing arrays without their length. I can’t fathom why we even allow it.
      - hnfong 4y ago
        That's what happens when you let a sect of Sun-worshipping astrologers redefine the minute to have up to 61 seconds...
  - KennyBlanken 4y ago
    It shouldn't actually matter. Hardware clocks should usually be in UTC, but the timezone should be "whatever makes sense." Webapps should be timezone aware via the browser or user preferences, internal timezone should be whatever makes sense for logging et al.
  - oceanplexian 4y ago
    Worked at a very big social media company and we set them all to Pacific since that’s where we were located. The servers themselves were physically located in different time zones. There’s no hard and fast rule what TZ’s servers need to be in IMHO.
  - computerfriend 4y ago
    Or the timezone the sole user is primarily located in.
  - sgarland 4y ago
    Generally I agree (mine are in America/New_York), but I did find [0] and fix [1] a weird bug in Debian's cron that only surfaced when I moved. I was in America/Chicago, moved to America/New_York, and then changed TZ on the servers once they were set up. Afterwards, I noticed that a daily cronjob that was supposed to fire at 0845 was instead happening at 0945. [0]: https://gist.github.com/stephanGarland/b7cdd963e0ac53ea42f8ed15e35b193d https://gist.github.com/stephanGarland/b7cdd963e0ac53ea42f8e... [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1019716 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1019716
- MrStonedOne 4y ago
  [dead]
- berkle4455 4y ago
  upvoting because i fiercely agree with you. servers should be UTC.
  - lionkor 4y ago
    Assuming servers should be UTC because timezones are hard, you could say servers should be airgapped because security is hard. Or that you should only type ASCII because Unicode is hard. I mean, yes, you can do that, but don't pretend that a program not handling unicode is the user's fault for typing unicode (s/unicode/timezones, s/typing/using). EDIT: I'm just saying that you can't blame the user for using the features that come with the system.
    - bombcar 4y ago
      It’s all true but you will have a harder life if you give all your servers Unicode names and directories and set them to some obscure time zone that is 15 minutes off.
      - throw0101b 4y ago
        > […] set them to some obscure time zone that is 15 minutes off. For anyone curious: > On the flip side, there’s one border where traveling north or south puts you ahead or behind just 15 minutes. That would be the border between India (UTC +5.5) and Nepal (UTC +5.75). * https://qz.com/357697/time-zone-deviants-part-i-the-strangest-time-zones-in-the-world https://qz.com/357697/time-zone-deviants-part-i-the-stranges... * https://www.worldtimeserver.com/learn/unusual-time-zones/ https://www.worldtimeserver.com/learn/unusual-time-zones/
        bombcar 4y ago
        The inverse of this is your test systems should be set to Nepal and India and use Unicode names for everything that has a text field. It’ll smoke out all sorts of fun.
    - bagels 4y ago
- throw0101b 4y ago
  > servers should always be UTC All of the servers I run are located in Toronto, Canada: why do I care about UTC? What would having all of them being UTC get me? A few jobs ago I took care of servers that were in: California, Ontario, Quebec, Switzerland, Singapore. UTC made sense there.
  - deleted 4y ago
    [deleted]
  - tomrod 4y ago
    Clarity as to your server time no matter what time zone nonsense a government devises.
    - throw0101b 4y ago
      Using UTC when all of my servers are in one timezones actually decreases clarity, as I live in America/Toronto with my daily existence, and which is what my laptop and (mechanical) wristwatch† are set to. If I less(1) the logs I now have to do mental math as to "when" things happen because the hour number it says in the logs does not match my human reality. If I look at something now (at Mar 25 21:09) the logs would say "Mar 26 01:09" which is not "now" in my mind. † Though I've been toying with the idea of getting a GMT watch "for fun". However I'll probably go with a chronograph—more useful feature on a day-to-day basis. Not as many choices if I want to get both complications though.
      - 8organicbits 4y ago
        I'm curious what your logs look like for 1am local time on Nov 6, 2022? I believe you set your clock back on that day, so 1am-2am would have happened twice. Maybe your logs mention the timezone / UTC offset? Otherwise it would seem possible to get confused.
        throw0101b 4y ago
        I have no idea because (a) I didn't experience any problems that needed debugging at that moment so I've never looked, and (b) I wasn't awake looking at the logs for fun at the time. But I do sometimes have to look at the logs on some random Tuesday, mid-morning, when someone says they can't log into our SSH bastion hosts (but it worked yesterday), I can see there's failed password attempts at 09:59, 10:03, and 10:04 (versus 13:59, 14:03, and 14:04 (i.e., UTC+4), which I would then have to do mental math to bring back to local time), and so it turns out they entered their password wrong a few of times so fail2ban blocked their IP Wallclock time is a human construct created for human convenience. Having UTC when all my servers are in America/Toronto is not convenient for me.
- limaoscarjuliet 4y ago
  They are UTC - the actual time on the system is measured in seconds since midnight Jan 1 1970 (https://en.wikipedia.org/wiki/Unix_time https://en.wikipedia.org/wiki/Unix_time). This time is then converted to local time per the OS timezone setting. A lot of apps (e.g. databases) will also have their own time zone conversions based on context (e.g. per database connection settings). As far as practical use of UTC vs local time: I am a developer that happens to work with customers directly. It is difficult enough to get customer to provide timestamps for issues they report in local time, I cannot imagine they would be able to convert to UTC. Love the local time!
  - anonymousiam 4y ago
    Beat me by five minutes. 100% agree.
  - j45 4y ago
    Local time is a benefit for users. As much as UTC might be beneficial for sysadmins, and technical teams. The only question is, is the implementation user focused or developer prioritized
  - IgorPartola 4y ago
    > This time is then converted to local time per the OS timezone setting. What exactly do you mean by this? As in, what part of your software stack is doing this. IIRC if you set your OS to be in UTC then anything at the OS level will only speak UTC. Databases and your own software will do what they are told to do but systemd etc. will all speak UTC only. Even running date in the shell should give you a UTC timestamp.
    - viraptor 4y ago
      Your system time is totally different from what userspace cares about. Go ahead, run "TZ=UTC date" (or a different timezone if you're using UTC already). I'm not even sure what you mean by the OS speaking a certain timezone. Basic "time()" will return seconds from the epoch UTC regardless of your system timezone. Things that care about timezones will use relevant conversions.
- anonymousiam 4y ago
  Servers (and every other Unix/Linux box) ALWAYS use UTC. The time zone is only specified for the purpose of DISPLAYING the time in some locale where the system resides. The only things that should care about the time zone are those that generate output for human consumption (such as log entries), and this is usually automated. For a system to not boot in a particular time zone is a serious bug, but you should not blame the problem on the time zone, you should blame the problem on whatever is broken in some critical piece of code that probably should not be using local time.
- skrtskrt 4y ago
  this position is getting a lot of flak, but I agree since we don't live in magic world where all software can handle time zones well. I strongly prefer everything in UTC and changing time zone is an issue for only the vary top layer of the display/user interface/frontend whatever if absolutely needed - generally libraries designed for displaying have much more robust timezone handling than just random daemons and backend software.
- akira2501 4y ago
  pid 1 shouldn't choke because of a timezone issue.
- jiggawatts 4y ago
  We share this planet with people that don't use UTC. That's just the way it is. If some computers are set to time zones other than UTC, then non-UTC time zones must be correctly handled, even "servers". There are terminal servers, for example, which are basically workstations. There are servers configured by other people for local time, and we may need to configure our own servers into its network. There is server software out there that uses the host timezone. Heck, huge vendors do this regularly. This happens in log files, in schedulers, in notification emails, and so on. People want to see their log timestamps in their local time, they want to schedule in local time, and they want their emails to show events in their local time. Yes, optimally, all times should always stored in UTC or UTC+offset formats, then displayed in local time only "at the end client". Generally, this is actually what happens. Windows, MacOS, UNIX, Linux, and Android all store the system clock as UTC and all system APIs use this. So in some sense, we are all already using UTC on our servers. The time zone setting doesn't change the system clock to something other than UTC, it just tells user-mode software how the human users expect to see time formatted for display. The problem is that server software especially is terrible at this. Just... bad. The worst offenders are text-based log files, which almost always get a formatted timestamp with an unspecified time zone. Could be UTC, could be the server's local time, could be Mars time, who knows? In summary: don't blame system administrators for correctly configuring server settings. Blame the lazy software developers who can't be bothered to use ISO timestamps that specify time zones unambiguously, irrespective of the time zone setting. Next thing you'll tell me to stop using space characters in file names...
  - cerved 4y ago
    Windows doesn't by default. Annoying when dual booting
    - joecool1029 4y ago
      Add the below as a .reg file and run it, will tell windows to use hwclock as UTC. (adding in case anyone else gets annoyed with the clock mixup situation on dual-booting) Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation] "RealTimeIsUniversal"=qword:00000001
- m-p-3 4y ago
  The server's clock should indeed be UTC, with the timezone translation done according to the user profile configuration.
CaliforniaKarl 4y ago
Don't you mean /usr/share/zoneinfo/Etc/UTC ? If I remember correctly, /usr/share/zoneinfo/Etc is a directory, and /etc/localtime should point to a zoneinfo file.
- lionkor 4y ago
  Hm, not sure, already closed the ssh. But it did work for us. The correct fix would be timedatectl, but as you can guess, that's broken together with systemd.
- starkparker 4y ago
  Thread is specific about linking to Etc. Also from the linked thread: > I've noticed systemd (PID=1) is running at 100% CPU utilization. That's not normal. I ran this command: strace -f -p 1 > The process was reporting continuously attempt to access /etc/localtime file. That's also unusual. > I managed to get the output I was receiving on the console while tracking systemd process. stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3522, ...}) = 0 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3522, ...}) = 0 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3522, ...}) = 0 Maybe if it's endlessly failing to stat the target, any valid target might help.
  - somat 4y ago
    Strace is great but it only shows syscalls. the section of code calling the stat will have to be found. stat returns 0 on successful completion.
somat 4y ago
Are you unsetting your timezone, I hesitate to sound authoritative about because I really don't know, and have not found the correct man page yet. but usually you symlink /etc/localtime to the tzfile you want as your local time. what happens when you symlink it to a directory? in your case /usr/share/zoneinfo/Etc My guess it that it effectively unset it, in your case that probably does nothing(system time is utc and dublin is in utc) what was it set to before? edit: all I can find is localtime(5) and it only says that /etc/localtime is the local time zone file.
- lionkor 4y ago
  Yes, it basically makes it useless, as far as I can tell. I don't know what the proper fix is, because timedatectl was broken.
kodah 4y ago
Sorry this happened to you, these kind of problems tend to be vague and difficult to patch up. I've advocated for over a decade as a systems engineer not to set system time to anything other than UTC. Not all languages or applications compensate for DST very well, and depending on your applications relationship with time it's a non-trivial problem to solve. What is far more trivial to do is to ship logs to a log server and let the UI of the log server translate time for the viewer of the logs. Almost every time I have argued with executives, managers, and SWEs without SE experience and lost something unexplainable and detrimental happens around DST. Edit: For the uninitiated I'll try to paint a clearer picture. All system time in Linux is tracked in seconds past Jan 1, 1970. The problem occurs in the translation between UTC and local time, which is handled by a library/package/module in your application. If your application is time sensitive and not looking for a literal time traveling event then things can get real weird, real fast. If there is a bug in that library/package/module things will also get real weird, real fast.
- lionkor 4y ago
  Apps shouldnt use anything but UCT for their timekeeping, there is no reason to do anything in the timezone's time. Setting the timezone should have no such effect.
  - kodah 4y ago
    You and I can agree on that in principle but there's a lot of reasons an application may read from system time. If the application reads the wrong system time source, then they get unexpected time travel. Even things like the RTC can experience time travel under certain conditions. My point was, using anything other than UTC is just unnecessary complexity and often enough ends in complicated problems.
  - lstamour 4y ago
    If your app interacts with humans or machines in other time zones, tracking in a different time zone can be necessary. Though often people don’t want to specify time zones with their complexities but instead specify times at locations. But I agree, generally your service doesn’t need time zones. The exception is necessary though when time zones get ambiguous or change (e.g. daylight savings going away)
cmurf 4y ago
This bug seems familiar. But I'm a little surprised the fix for this wasn't already in the version of systemd chosen for the release. March 2022 https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1966800 https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/19668... March 2021 (Fedora) https://bugzilla.redhat.com/show_bug.cgi?id=1941335 https://bugzilla.redhat.com/show_bug.cgi?id=1941335 systemd upstream fix https://github.com/systemd/systemd/pull/19075 https://github.com/systemd/systemd/pull/19075
AndyMcConachie 4y ago
Almost all software bugs come from managing state. Which is why hoarding state management in PID 0 is a dumb idea. I said this when systemd was gaining traction and I'm still saying it now. When Debian switched to systemd in the default was when I began to seriously use OpenBSD and I've never regretted the decision.
- bitwize 4y ago
  > Which is why hoarding state management in PID 0 is a dumb idea. Because the Unix process API is unreliable and unsafe ( http://catern.com/process.html http://catern.com/process.html ), managing processes is difficult to impossible in a general way under Unix systems including Linux. Linux gained some features to help with this, but for the longest time, PID 1 was the only process that could manage its child processes in a way that wasn't fraught with race conditions -- meaning systemd or something like it was the only way to have safe, effective process management because it tracked all that state in PID 1. Blame Unix being broken, not systemd, for the problems systemd solves.
- kaba0 4y ago
  Because the litany of shittily written bash scripts don’t hoard state, or what exactly are you saying? The only thing I dislike about systemd is that it is written in C, but otherwise it is a single purpose solution to the surprisingly complex problem of managing services during boot and running. I fail to see how state is relevant here — if it were written in Haskell it would also have state, almost by definition it has to handle external state. No Monad would help you there, there is just correct and not correct software.
  - kasabali 4y ago
    > single purpose solution You can't be saying this with a straight face. systemd project subsumes functionality of - device hotplug - user session manager - cron - logging - network management - time zone management - time sync - cgroup manager - deleting /tmp files - DNS resolver - UEFI boot manager and also dozen other things I've forgotten.

gjvc 4y ago

servers should always be UTC

LeoPanthera 4y ago
It is entirely reasonable for a personal server to be set to the timezone it is physically located in.
- lionkor 4y ago
  I'm no expert, but if your software can't deal with timezones, be worried. So yes, absolutely, it should be perfectly okay to use your local timezone.
  - bcrl 4y ago
    Timezones are disaster. Once upon a time it was difficult and time consuming to make changes to them. Then the world began developing more and "accurate" timekeeping devices. Atomic clocks were developed and began to be miniaturized. Then someone had the brilliant that incredibly accurate clocks could be used to determine location. GPS was born. A few decades pass and we now all have highly accurate "clocks" that we carry around in our pockets that receive GPS signals and sync their idea of time to what the satellite thinks. Sounds great, right? Except, humans don't actually use atomic clocks as the basis for their timekeeping. We look outside and expect the time on the clock to mostly match with the position of the sun in the sky. Once leap seconds became entangled in them, the whole thing basically a toxic pile of radioactive waste. Timezones are now being updated multiple times a year since the internet means we can update them at will. The whole thing is now a giant mess of toxic radioactive waste that can never be correct for any extended period of time. We, the software industry, did this to ourselves. We built this, we have to live with it, and it's never going away.
    - fsckboy 4y ago
      computers are good at keeping track of things that humans aren't. tracking timezones is in your computer's wheelhouse, don't be scared of it.
      - bcrl 4y ago
        I'm not scared of it. It's unnecessary excessive complexity that we have foisted onto ourselves. It has caused countless bugs and will most likely cause countless more before I die. Show me the software that copes with timezone changes at runtime.
        nikau 4y ago
        humans having to convert times from UTC to local timezone will also result in countless hours wasted in troubleshooting.
        Spivak 4y ago
        Every phone when people travel? It really isn’t that hard. What makes you pull your hair out is dealing with 3rd party code that handles it incorrectly. Do everything internal in UTC or always attach a timezone to dates, your choice. When you cross the boundary either ingesting naive dates or displaying them read the system time zone again and convert. I mean it’s kinda annoying but “naive” times is the time equivalent of storing arrays without their length. I can’t fathom why we even allow it.
    - hnfong 4y ago
      That's what happens when you let a sect of Sun-worshipping astrologers redefine the minute to have up to 61 seconds...
- KennyBlanken 4y ago
  It shouldn't actually matter. Hardware clocks should usually be in UTC, but the timezone should be "whatever makes sense." Webapps should be timezone aware via the browser or user preferences, internal timezone should be whatever makes sense for logging et al.
- oceanplexian 4y ago
  Worked at a very big social media company and we set them all to Pacific since that’s where we were located. The servers themselves were physically located in different time zones. There’s no hard and fast rule what TZ’s servers need to be in IMHO.
- computerfriend 4y ago
  Or the timezone the sole user is primarily located in.
- sgarland 4y ago
  Generally I agree (mine are in America/New_York), but I did find [0] and fix [1] a weird bug in Debian's cron that only surfaced when I moved. I was in America/Chicago, moved to America/New_York, and then changed TZ on the servers once they were set up. Afterwards, I noticed that a daily cronjob that was supposed to fire at 0845 was instead happening at 0945. [0]: https://gist.github.com/stephanGarland/b7cdd963e0ac53ea42f8ed15e35b193d https://gist.github.com/stephanGarland/b7cdd963e0ac53ea42f8e... [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1019716 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1019716
MrStonedOne 4y ago
[dead]
berkle4455 4y ago
upvoting because i fiercely agree with you. servers should be UTC.
- lionkor 4y ago
  Assuming servers should be UTC because timezones are hard, you could say servers should be airgapped because security is hard. Or that you should only type ASCII because Unicode is hard. I mean, yes, you can do that, but don't pretend that a program not handling unicode is the user's fault for typing unicode (s/unicode/timezones, s/typing/using). EDIT: I'm just saying that you can't blame the user for using the features that come with the system.
  - bombcar 4y ago
    It’s all true but you will have a harder life if you give all your servers Unicode names and directories and set them to some obscure time zone that is 15 minutes off.
    - throw0101b 4y ago
      > […] set them to some obscure time zone that is 15 minutes off. For anyone curious: > On the flip side, there’s one border where traveling north or south puts you ahead or behind just 15 minutes. That would be the border between India (UTC +5.5) and Nepal (UTC +5.75). * https://qz.com/357697/time-zone-deviants-part-i-the-strangest-time-zones-in-the-world https://qz.com/357697/time-zone-deviants-part-i-the-stranges... * https://www.worldtimeserver.com/learn/unusual-time-zones/ https://www.worldtimeserver.com/learn/unusual-time-zones/
      - bombcar 4y ago
        The inverse of this is your test systems should be set to Nepal and India and use Unicode names for everything that has a text field. It’ll smoke out all sorts of fun.
  - bagels 4y ago
throw0101b 4y ago
> servers should always be UTC All of the servers I run are located in Toronto, Canada: why do I care about UTC? What would having all of them being UTC get me? A few jobs ago I took care of servers that were in: California, Ontario, Quebec, Switzerland, Singapore. UTC made sense there.
- deleted 4y ago
  [deleted]
- tomrod 4y ago
  Clarity as to your server time no matter what time zone nonsense a government devises.
  - throw0101b 4y ago
    Using UTC when all of my servers are in one timezones actually decreases clarity, as I live in America/Toronto with my daily existence, and which is what my laptop and (mechanical) wristwatch† are set to. If I less(1) the logs I now have to do mental math as to "when" things happen because the hour number it says in the logs does not match my human reality. If I look at something now (at Mar 25 21:09) the logs would say "Mar 26 01:09" which is not "now" in my mind. † Though I've been toying with the idea of getting a GMT watch "for fun". However I'll probably go with a chronograph—more useful feature on a day-to-day basis. Not as many choices if I want to get both complications though.
    - 8organicbits 4y ago
      I'm curious what your logs look like for 1am local time on Nov 6, 2022? I believe you set your clock back on that day, so 1am-2am would have happened twice. Maybe your logs mention the timezone / UTC offset? Otherwise it would seem possible to get confused.
      - throw0101b 4y ago
        I have no idea because (a) I didn't experience any problems that needed debugging at that moment so I've never looked, and (b) I wasn't awake looking at the logs for fun at the time. But I do sometimes have to look at the logs on some random Tuesday, mid-morning, when someone says they can't log into our SSH bastion hosts (but it worked yesterday), I can see there's failed password attempts at 09:59, 10:03, and 10:04 (versus 13:59, 14:03, and 14:04 (i.e., UTC+4), which I would then have to do mental math to bring back to local time), and so it turns out they entered their password wrong a few of times so fail2ban blocked their IP Wallclock time is a human construct created for human convenience. Having UTC when all my servers are in America/Toronto is not convenient for me.
limaoscarjuliet 4y ago
They are UTC - the actual time on the system is measured in seconds since midnight Jan 1 1970 (https://en.wikipedia.org/wiki/Unix_time https://en.wikipedia.org/wiki/Unix_time). This time is then converted to local time per the OS timezone setting. A lot of apps (e.g. databases) will also have their own time zone conversions based on context (e.g. per database connection settings). As far as practical use of UTC vs local time: I am a developer that happens to work with customers directly. It is difficult enough to get customer to provide timestamps for issues they report in local time, I cannot imagine they would be able to convert to UTC. Love the local time!
- anonymousiam 4y ago
  Beat me by five minutes. 100% agree.
- j45 4y ago
  Local time is a benefit for users. As much as UTC might be beneficial for sysadmins, and technical teams. The only question is, is the implementation user focused or developer prioritized
- IgorPartola 4y ago
  > This time is then converted to local time per the OS timezone setting. What exactly do you mean by this? As in, what part of your software stack is doing this. IIRC if you set your OS to be in UTC then anything at the OS level will only speak UTC. Databases and your own software will do what they are told to do but systemd etc. will all speak UTC only. Even running date in the shell should give you a UTC timestamp.
  - viraptor 4y ago
    Your system time is totally different from what userspace cares about. Go ahead, run "TZ=UTC date" (or a different timezone if you're using UTC already). I'm not even sure what you mean by the OS speaking a certain timezone. Basic "time()" will return seconds from the epoch UTC regardless of your system timezone. Things that care about timezones will use relevant conversions.
anonymousiam 4y ago
Servers (and every other Unix/Linux box) ALWAYS use UTC. The time zone is only specified for the purpose of DISPLAYING the time in some locale where the system resides. The only things that should care about the time zone are those that generate output for human consumption (such as log entries), and this is usually automated. For a system to not boot in a particular time zone is a serious bug, but you should not blame the problem on the time zone, you should blame the problem on whatever is broken in some critical piece of code that probably should not be using local time.
skrtskrt 4y ago
this position is getting a lot of flak, but I agree since we don't live in magic world where all software can handle time zones well. I strongly prefer everything in UTC and changing time zone is an issue for only the vary top layer of the display/user interface/frontend whatever if absolutely needed - generally libraries designed for displaying have much more robust timezone handling than just random daemons and backend software.
akira2501 4y ago
pid 1 shouldn't choke because of a timezone issue.
jiggawatts 4y ago
We share this planet with people that don't use UTC. That's just the way it is. If some computers are set to time zones other than UTC, then non-UTC time zones must be correctly handled, even "servers". There are terminal servers, for example, which are basically workstations. There are servers configured by other people for local time, and we may need to configure our own servers into its network. There is server software out there that uses the host timezone. Heck, huge vendors do this regularly. This happens in log files, in schedulers, in notification emails, and so on. People want to see their log timestamps in their local time, they want to schedule in local time, and they want their emails to show events in their local time. Yes, optimally, all times should always stored in UTC or UTC+offset formats, then displayed in local time only "at the end client". Generally, this is actually what happens. Windows, MacOS, UNIX, Linux, and Android all store the system clock as UTC and all system APIs use this. So in some sense, we are all already using UTC on our servers. The time zone setting doesn't change the system clock to something other than UTC, it just tells user-mode software how the human users expect to see time formatted for display. The problem is that server software especially is terrible at this. Just... bad. The worst offenders are text-based log files, which almost always get a formatted timestamp with an unspecified time zone. Could be UTC, could be the server's local time, could be Mars time, who knows? In summary: don't blame system administrators for correctly configuring server settings. Blame the lazy software developers who can't be bothered to use ISO timestamps that specify time zones unambiguously, irrespective of the time zone setting. Next thing you'll tell me to stop using space characters in file names...
- cerved 4y ago
  Windows doesn't by default. Annoying when dual booting
  - joecool1029 4y ago
    Add the below as a .reg file and run it, will tell windows to use hwclock as UTC. (adding in case anyone else gets annoyed with the clock mixup situation on dual-booting) Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation] "RealTimeIsUniversal"=qword:00000001
m-p-3 4y ago
The server's clock should indeed be UTC, with the timezone translation done according to the user profile configuration.

CaliforniaKarl 4y ago

Don't you mean /usr/share/zoneinfo/Etc/UTC ? If I remember correctly, /usr/share/zoneinfo/Etc is a directory, and /etc/localtime should point to a zoneinfo file.

lionkor 4y ago
Hm, not sure, already closed the ssh. But it did work for us. The correct fix would be timedatectl, but as you can guess, that's broken together with systemd.
starkparker 4y ago
Thread is specific about linking to Etc. Also from the linked thread: > I've noticed systemd (PID=1) is running at 100% CPU utilization. That's not normal. I ran this command: strace -f -p 1 > The process was reporting continuously attempt to access /etc/localtime file. That's also unusual. > I managed to get the output I was receiving on the console while tracking systemd process. stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3522, ...}) = 0 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3522, ...}) = 0 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3522, ...}) = 0 Maybe if it's endlessly failing to stat the target, any valid target might help.
- somat 4y ago
  Strace is great but it only shows syscalls. the section of code calling the stat will have to be found. stat returns 0 on successful completion.

somat 4y ago

Are you unsetting your timezone, I hesitate to sound authoritative about because I really don't know, and have not found the correct man page yet. but usually you symlink /etc/localtime to the tzfile you want as your local time. what happens when you symlink it to a directory? in your case /usr/share/zoneinfo/Etc My guess it that it effectively unset it, in your case that probably does nothing(system time is utc and dublin is in utc) what was it set to before? edit: all I can find is localtime(5) and it only says that /etc/localtime is the local time zone file.

lionkor 4y ago
Yes, it basically makes it useless, as far as I can tell. I don't know what the proper fix is, because timedatectl was broken.

kodah 4y ago

Sorry this happened to you, these kind of problems tend to be vague and difficult to patch up. I've advocated for over a decade as a systems engineer not to set system time to anything other than UTC. Not all languages or applications compensate for DST very well, and depending on your applications relationship with time it's a non-trivial problem to solve. What is far more trivial to do is to ship logs to a log server and let the UI of the log server translate time for the viewer of the logs. Almost every time I have argued with executives, managers, and SWEs without SE experience and lost something unexplainable and detrimental happens around DST. Edit: For the uninitiated I'll try to paint a clearer picture. All system time in Linux is tracked in seconds past Jan 1, 1970. The problem occurs in the translation between UTC and local time, which is handled by a library/package/module in your application. If your application is time sensitive and not looking for a literal time traveling event then things can get real weird, real fast. If there is a bug in that library/package/module things will also get real weird, real fast.

lionkor 4y ago
Apps shouldnt use anything but UCT for their timekeeping, there is no reason to do anything in the timezone's time. Setting the timezone should have no such effect.
- kodah 4y ago
  You and I can agree on that in principle but there's a lot of reasons an application may read from system time. If the application reads the wrong system time source, then they get unexpected time travel. Even things like the RTC can experience time travel under certain conditions. My point was, using anything other than UTC is just unnecessary complexity and often enough ends in complicated problems.
- lstamour 4y ago
  If your app interacts with humans or machines in other time zones, tracking in a different time zone can be necessary. Though often people don’t want to specify time zones with their complexities but instead specify times at locations. But I agree, generally your service doesn’t need time zones. The exception is necessary though when time zones get ambiguous or change (e.g. daylight savings going away)

cmurf 4y ago

This bug seems familiar. But I'm a little surprised the fix for this wasn't already in the version of systemd chosen for the release. March 2022 https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1966800 https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/19668... March 2021 (Fedora) https://bugzilla.redhat.com/show_bug.cgi?id=1941335 https://bugzilla.redhat.com/show_bug.cgi?id=1941335 systemd upstream fix https://github.com/systemd/systemd/pull/19075 https://github.com/systemd/systemd/pull/19075

AndyMcConachie 4y ago

Almost all software bugs come from managing state. Which is why hoarding state management in PID 0 is a dumb idea. I said this when systemd was gaining traction and I'm still saying it now. When Debian switched to systemd in the default was when I began to seriously use OpenBSD and I've never regretted the decision.

bitwize 4y ago
> Which is why hoarding state management in PID 0 is a dumb idea. Because the Unix process API is unreliable and unsafe ( http://catern.com/process.html http://catern.com/process.html ), managing processes is difficult to impossible in a general way under Unix systems including Linux. Linux gained some features to help with this, but for the longest time, PID 1 was the only process that could manage its child processes in a way that wasn't fraught with race conditions -- meaning systemd or something like it was the only way to have safe, effective process management because it tracked all that state in PID 1. Blame Unix being broken, not systemd, for the problems systemd solves.
kaba0 4y ago
Because the litany of shittily written bash scripts don’t hoard state, or what exactly are you saying? The only thing I dislike about systemd is that it is written in C, but otherwise it is a single purpose solution to the surprisingly complex problem of managing services during boot and running. I fail to see how state is relevant here — if it were written in Haskell it would also have state, almost by definition it has to handle external state. No Monad would help you there, there is just correct and not correct software.
- kasabali 4y ago
  > single purpose solution You can't be saying this with a straight face. systemd project subsumes functionality of - device hotplug - user session manager - cron - logging - network management - time zone management - time sync - cgroup manager - deleting /tmp files - DNS resolver - UEFI boot manager and also dozen other things I've forgotten.