5 ms·

Setting the TZ environment variable avoids thousands of system calls (2017)

198 points by zdw 4y ago

ttyprintk 4y ago
I didn’t know that. It can be helpful to AcceptEnv TZ and SendEnv TZ over ssh. At least, ls provides relatable times.
- kondor6c 4y ago
  Good call! I put this in place on my mail server and set it on my ansible base layer. Nice username too :-)
mixmastamyk 4y ago
> Ubuntu Precise (12.04) and Ubuntu Xenial (16.04) Has glibc been updated to alleviate this since?
- boring_twenties 4y ago
  Not as of glibc 2.31
- zajio1am 4y ago
  Why it should be? The behavior is correct, /etc/localtime could change since last read so it is necessary to check whether old value can be used. The real problem is calling localtime() so often.
  - boring_twenties 4y ago
    If that behavior is correct, then the behavior with TZ=:/etc/localtime must be incorrect?
    - zajio1am 4y ago
      It is not incorrect in general, but it is kind of hack that may be incorrect in some situations (as it does not reflect change in system timezone), so it is not suitable for default behavior. Default behavior - use current system timezone Explicit TZ - use specific timezone defined by TZ variable, defined either directly (e.g. TZ="NZST-12:00:00NZDT-13:00:00,M10.1.0,M3.3.0"), or indirectly from file (e.g. TZ=":/usr/share/zoneinfo/Europe/Brussels"). As this defines specific timezone, it is not supposed to change.
      - boring_twenties 4y ago
        So basically, always set TZ=:/etc/localtime.
  - eqvinox 4y ago
    [moved my comment to top-level - https://news.ycombinator.com/item?id=34353072 https://news.ycombinator.com/item?id=34353072] TL;DR: no, not correct, put a rate limit on it.
barbazoo 4y ago
> Without setting TZ during normal operations yields approximately: 14,925 calls to stat over a 30 second period (or roughly 497 stats per second). > With TZ set during the same time period results in 8 calls to stat over a 30 second period. This is interesting but what would be even more interesting is what that means to wall time. My gut feeling is probably not that much.
- kevin_thibedeau 4y ago
  Wasted effort affects battery life.
- manv1 4y ago
  More work is more work.
- kristopolous 4y ago
  If you find yourself in the position of paying over say, $250,000/month for cloud computing, things like this can have monetary impact that your clocks ultimately don't care about.
- loeg 4y ago
  500 syscalls/sec is a huge amount of useless overhead for a high performance system.
- uberduper 4y ago
  This seriously impacted performance of Zoom on my linux laptop. It's still pretty bad, but significantly better with TZ set.
- zX41ZdbW 4y ago
  Syscalls can be heavier than expected. One example is when an application is run inside gVisor. Another example is when a lot of eBPF code is attached. A third example is when a program is run under strace. Disclaimer: I'm working on ClickHouse[1], and it is used by thousands of companies in unimaginable environments. It has to work in every possible condition... That's why we set the TZ variable at startup and also embed the timezones into the binary. And we don't use the glibc functions for timezone operations because they are astonishingly slow. https://github.com/ClickHouse/ClickHouse/ https://github.com/ClickHouse/ClickHouse/
dang 4y ago
Discussed at the time: How setting the TZ environment variable avoids thousands of system calls - https://news.ycombinator.com/item?id=13697555 https://news.ycombinator.com/item?id=13697555 - Feb 2017 (143 comments)
- paravz 4y ago
  I have started using globally exported TZ=:/etc/localtime after previous discussion, but since stopped, since at some desktop applications (slack for example) were not reading timezone correctly.
deleted 4y ago
[deleted]
imglorp 4y ago
Locale also bites people all the time with simple shell commands like sort or grep. A giant speedup can be had by setting LC_ALL=C. https://www.inmotionhosting.com/support/website/speed-up-grep-searches-with-lc-all/ https://www.inmotionhosting.com/support/website/speed-up-gre...
- moring 4y ago
  The fact that this trick exists reveals a deeper problem with the specification of how tools like sort or grep should behave: If they are supposed to act locale-specific, then this "trick" simply leads to wrong results and should be avoided. If, however, these tools are supposed to act locale-inspecific / according to the "C" locale, then the need for this trick should be tracked as a bug for them and the locale-sensitivity should be removed in code. How are these tools specified to act? What were the requirements for them? An interesting clue is how these tools are used at all. Less computer savvy users don't touch the command line anyway, so these are used by power users, developers, system operators and the like. HOW are they used? These tools cannot search through what is considered a text file by regular people, i.e. a Word document, but only what a developer would consider a "plain text" file with pure ASCII, possibly containing in-band markup. In particular, logs, configuration files, and code. There is a strong tendency to treat these with the "C" locale, making the locale-sensititivy of the tools unnecessary and even harmful. However, another aspect comes into play: These files are sometimes NOT using the ASCII encoding but UTF-8, e.g. Java source files. The next obvious problem with these tools is that they treat character encoding as part of the locale; it should be derived from the file type, with the next problem obviously being that Unix doesn't have a solid concept of file types and cannot distinguish ASCII plain-text files from UTF8 plain-text files, resorting to a crude workaround using an unrelated setting (locale) to make the user solve the problem. OTOH, maybe I'm just "afraid of the command line" /rant
  - unilynx 4y ago
    ASCII is a subset of UTF-8. It’s time for all tools and paths to just assume UTF-8 encoding. Most tasks assuming ASCII work fine on UTF-8 too (eg sorting). We do need to get rid of those byte order marks though, it breaks eg concatenation
mike503 4y ago
Seems like you can safely define it for a single run of any script or anything where you don't need to factor in a timezone change. Unless I've mistaken everything I've read. To personalize PHP execution I used to putenv("TZ=America/Los_Angeles") for example at the top of a script based on the user's desired timezone. It wound up making all the other time based calls localized which was great. date_default_timezone_set() is how I do it now, I wonder if it's as efficient (I should strace it)
vlasky 4y ago
To all programmers here, the TZ=:<zonefile> syntax is currently unsupported in the icu library (International Components for Unicode): https://unicode-org.atlassian.net/browse/ICU-13694 https://unicode-org.atlassian.net/browse/ICU-13694 https://github.com/unicode-org/icu/pull/2213 https://github.com/unicode-org/icu/pull/2213 This affects all packages that have icu as a dependency, one of them being Node.js. https://github.com/nodejs/node/issues/37271 https://github.com/nodejs/node/issues/37271 I discovered this the hard way when some code malfunctioned shortly after daylight savings time kicked in.
- tinus_hn 4y ago
  Why does the icu string library care about time zones?
layer8 4y ago
As noted in the previous thread (https://news.ycombinator.com/item?id=13701320 https://news.ycombinator.com/item?id=13701320), there is no good reason for why libc behaves differently when accessing the default file vs. one explicitly specified by TZ. Whether to stat for changes should be orthogonal to which timezone file is used (or to whether the default is used or not). There should rather be a separate way to configure the stat behavior, assuming it makes sense at all. (I don’t know how many programs can actually properly deal with intermittent changes to the timezone definitions.) One reply comment of the above link mentions user vs. admin, but that’s not a good argument, because users specifying TZ may still want a long-running background process to observe updates of the timezone file, and also TZ may have been set system-wide rather than by the user.
- kelnos 4y ago
  Sure there's a good reason: if TZ is set, glibc can assume that the timezone is going to be fixed for the duration of the process's life. If it is not set, it will consult /etc/localtime, and will then react to changes in the system timezone. (Now, when you use TZ=:/etc/localtime, I suppose you could expect that glibc would see that it's a symlink [maybe], and then decide to re-check it each time, but I think the current behavior of not doing that is consistent and sane.)
  - akira2501 4y ago
    > glibc can assume that the timezone is going to be fixed for the duration of the process's life. From what I'm seeing.. it doesn't seem to actually assume that. It does check to see if the TZ name changed between calls and will reload a new tzfile if it has. This same mechanism is what causes the file to be loaded whenever getenv("TZ") returns NULL.
    - boring_twenties 4y ago
      When I set TZ=:/etc/localtime it only stats and reads the file once. Debian 11 with glibc 2.31.
europeanguy 4y ago
Please someone tell me this isn't the case in the default installation of Debian.
vbezhenar 4y ago
It's not clear if TZ=Asia/Almaty or TZ=UTC helps to avoid those thousands. Also what about alpine musl?
- kseistrup 4y ago
  However, tzset(3) (or “man timezone”) says that if the filespec does not start with a slash, the file specification is relative to the system timezone directory, so e.g. “TZ=:Asia/Almaty” should give the desired effect.
- akira2501 4y ago
  You can also set TZ="" and UTC will be assumed.
qwertygerty 4y ago
This seems to be the same thing, question asked in 2010... https://stackoverflow.com/questions/4554271/how-to-avoid-excessive-stat-etc-localtime-calls-in-strftime-on-linux https://stackoverflow.com/questions/4554271/how-to-avoid-exc...
eqvinox 4y ago
[pulled up from some levels of reply down] No, the behavior is not correct, because changing /etc/localtime is an asynchronous operation. You're changing a file in the file system and expecting to observe a change in a running process. It's, by design, racy — and that's OK. Taking this into account, the correct behavior is to keep a timestamp of when the last stat() call was, and only stat() again if it was longer than X ago. Even a few seonds would clamp down the stat syscall rate to ambient noise. (NB: changing /etc/localtime, and then starting a new process is not asynchronous. But that's not the issue here. Two processes are doing things here — changing /etc/localtime and invoking localtime() without synchronization. You shouldn't, can't and mustn't rely on ordering of such unsynchronized events.)
- zajio1am 4y ago
  > No, the behavior is not correct, because changing /etc/localtime is an asynchronous operation Well, technically you can do both from the same process (i.e. localtime(), unlink() + symlink() to change /etc/localtime, another localtime()), in that case it would be synchronous. But in general i agree, if the timeout was small enough (say < 100 ms) i guess nobody would object.

198 points by zdw 4y ago

ttyprintk 4y ago

I didn’t know that. It can be helpful to AcceptEnv TZ and SendEnv TZ over ssh. At least, ls provides relatable times.

kondor6c 4y ago
Good call! I put this in place on my mail server and set it on my ansible base layer. Nice username too :-)

mixmastamyk 4y ago

> Ubuntu Precise (12.04) and Ubuntu Xenial (16.04) Has glibc been updated to alleviate this since?

boring_twenties 4y ago
Not as of glibc 2.31
zajio1am 4y ago
Why it should be? The behavior is correct, /etc/localtime could change since last read so it is necessary to check whether old value can be used. The real problem is calling localtime() so often.
- boring_twenties 4y ago
  If that behavior is correct, then the behavior with TZ=:/etc/localtime must be incorrect?
  - zajio1am 4y ago
    It is not incorrect in general, but it is kind of hack that may be incorrect in some situations (as it does not reflect change in system timezone), so it is not suitable for default behavior. Default behavior - use current system timezone Explicit TZ - use specific timezone defined by TZ variable, defined either directly (e.g. TZ="NZST-12:00:00NZDT-13:00:00,M10.1.0,M3.3.0"), or indirectly from file (e.g. TZ=":/usr/share/zoneinfo/Europe/Brussels"). As this defines specific timezone, it is not supposed to change.
    - boring_twenties 4y ago
      So basically, always set TZ=:/etc/localtime.
- eqvinox 4y ago
  [moved my comment to top-level - https://news.ycombinator.com/item?id=34353072 https://news.ycombinator.com/item?id=34353072] TL;DR: no, not correct, put a rate limit on it.

barbazoo 4y ago

> Without setting TZ during normal operations yields approximately: 14,925 calls to stat over a 30 second period (or roughly 497 stats per second). > With TZ set during the same time period results in 8 calls to stat over a 30 second period. This is interesting but what would be even more interesting is what that means to wall time. My gut feeling is probably not that much.

kevin_thibedeau 4y ago
Wasted effort affects battery life.
manv1 4y ago
More work is more work.
kristopolous 4y ago
If you find yourself in the position of paying over say, $250,000/month for cloud computing, things like this can have monetary impact that your clocks ultimately don't care about.
loeg 4y ago
500 syscalls/sec is a huge amount of useless overhead for a high performance system.
uberduper 4y ago
This seriously impacted performance of Zoom on my linux laptop. It's still pretty bad, but significantly better with TZ set.
zX41ZdbW 4y ago
Syscalls can be heavier than expected. One example is when an application is run inside gVisor. Another example is when a lot of eBPF code is attached. A third example is when a program is run under strace. Disclaimer: I'm working on ClickHouse[1], and it is used by thousands of companies in unimaginable environments. It has to work in every possible condition... That's why we set the TZ variable at startup and also embed the timezones into the binary. And we don't use the glibc functions for timezone operations because they are astonishingly slow. https://github.com/ClickHouse/ClickHouse/ https://github.com/ClickHouse/ClickHouse/

dang 4y ago

Discussed at the time: How setting the TZ environment variable avoids thousands of system calls - https://news.ycombinator.com/item?id=13697555 https://news.ycombinator.com/item?id=13697555 - Feb 2017 (143 comments)

paravz 4y ago
I have started using globally exported TZ=:/etc/localtime after previous discussion, but since stopped, since at some desktop applications (slack for example) were not reading timezone correctly.

deleted 4y ago

[deleted]

imglorp 4y ago

Locale also bites people all the time with simple shell commands like sort or grep. A giant speedup can be had by setting LC_ALL=C. https://www.inmotionhosting.com/support/website/speed-up-grep-searches-with-lc-all/ https://www.inmotionhosting.com/support/website/speed-up-gre...

moring 4y ago
The fact that this trick exists reveals a deeper problem with the specification of how tools like sort or grep should behave: If they are supposed to act locale-specific, then this "trick" simply leads to wrong results and should be avoided. If, however, these tools are supposed to act locale-inspecific / according to the "C" locale, then the need for this trick should be tracked as a bug for them and the locale-sensitivity should be removed in code. How are these tools specified to act? What were the requirements for them? An interesting clue is how these tools are used at all. Less computer savvy users don't touch the command line anyway, so these are used by power users, developers, system operators and the like. HOW are they used? These tools cannot search through what is considered a text file by regular people, i.e. a Word document, but only what a developer would consider a "plain text" file with pure ASCII, possibly containing in-band markup. In particular, logs, configuration files, and code. There is a strong tendency to treat these with the "C" locale, making the locale-sensititivy of the tools unnecessary and even harmful. However, another aspect comes into play: These files are sometimes NOT using the ASCII encoding but UTF-8, e.g. Java source files. The next obvious problem with these tools is that they treat character encoding as part of the locale; it should be derived from the file type, with the next problem obviously being that Unix doesn't have a solid concept of file types and cannot distinguish ASCII plain-text files from UTF8 plain-text files, resorting to a crude workaround using an unrelated setting (locale) to make the user solve the problem. OTOH, maybe I'm just "afraid of the command line" /rant
- unilynx 4y ago
  ASCII is a subset of UTF-8. It’s time for all tools and paths to just assume UTF-8 encoding. Most tasks assuming ASCII work fine on UTF-8 too (eg sorting). We do need to get rid of those byte order marks though, it breaks eg concatenation

mike503 4y ago

Seems like you can safely define it for a single run of any script or anything where you don't need to factor in a timezone change. Unless I've mistaken everything I've read. To personalize PHP execution I used to putenv("TZ=America/Los_Angeles") for example at the top of a script based on the user's desired timezone. It wound up making all the other time based calls localized which was great. date_default_timezone_set() is how I do it now, I wonder if it's as efficient (I should strace it)

vlasky 4y ago

To all programmers here, the TZ=:<zonefile> syntax is currently unsupported in the icu library (International Components for Unicode): https://unicode-org.atlassian.net/browse/ICU-13694 https://unicode-org.atlassian.net/browse/ICU-13694 https://github.com/unicode-org/icu/pull/2213 https://github.com/unicode-org/icu/pull/2213 This affects all packages that have icu as a dependency, one of them being Node.js. https://github.com/nodejs/node/issues/37271 https://github.com/nodejs/node/issues/37271 I discovered this the hard way when some code malfunctioned shortly after daylight savings time kicked in.

tinus_hn 4y ago
Why does the icu string library care about time zones?

layer8 4y ago

As noted in the previous thread (https://news.ycombinator.com/item?id=13701320 https://news.ycombinator.com/item?id=13701320), there is no good reason for why libc behaves differently when accessing the default file vs. one explicitly specified by TZ. Whether to stat for changes should be orthogonal to which timezone file is used (or to whether the default is used or not). There should rather be a separate way to configure the stat behavior, assuming it makes sense at all. (I don’t know how many programs can actually properly deal with intermittent changes to the timezone definitions.) One reply comment of the above link mentions user vs. admin, but that’s not a good argument, because users specifying TZ may still want a long-running background process to observe updates of the timezone file, and also TZ may have been set system-wide rather than by the user.

kelnos 4y ago
Sure there's a good reason: if TZ is set, glibc can assume that the timezone is going to be fixed for the duration of the process's life. If it is not set, it will consult /etc/localtime, and will then react to changes in the system timezone. (Now, when you use TZ=:/etc/localtime, I suppose you could expect that glibc would see that it's a symlink [maybe], and then decide to re-check it each time, but I think the current behavior of not doing that is consistent and sane.)
- akira2501 4y ago
  > glibc can assume that the timezone is going to be fixed for the duration of the process's life. From what I'm seeing.. it doesn't seem to actually assume that. It does check to see if the TZ name changed between calls and will reload a new tzfile if it has. This same mechanism is what causes the file to be loaded whenever getenv("TZ") returns NULL.
  - boring_twenties 4y ago
    When I set TZ=:/etc/localtime it only stats and reads the file once. Debian 11 with glibc 2.31.

europeanguy 4y ago

Please someone tell me this isn't the case in the default installation of Debian.

vbezhenar 4y ago

It's not clear if TZ=Asia/Almaty or TZ=UTC helps to avoid those thousands. Also what about alpine musl?

kseistrup 4y ago
However, tzset(3) (or “man timezone”) says that if the filespec does not start with a slash, the file specification is relative to the system timezone directory, so e.g. “TZ=:Asia/Almaty” should give the desired effect.
akira2501 4y ago
You can also set TZ="" and UTC will be assumed.

qwertygerty 4y ago

This seems to be the same thing, question asked in 2010... https://stackoverflow.com/questions/4554271/how-to-avoid-excessive-stat-etc-localtime-calls-in-strftime-on-linux https://stackoverflow.com/questions/4554271/how-to-avoid-exc...

eqvinox 4y ago

[pulled up from some levels of reply down] No, the behavior is not correct, because changing /etc/localtime is an asynchronous operation. You're changing a file in the file system and expecting to observe a change in a running process. It's, by design, racy — and that's OK. Taking this into account, the correct behavior is to keep a timestamp of when the last stat() call was, and only stat() again if it was longer than X ago. Even a few seonds would clamp down the stat syscall rate to ambient noise. (NB: changing /etc/localtime, and then starting a new process is not asynchronous. But that's not the issue here. Two processes are doing things here — changing /etc/localtime and invoking localtime() without synchronization. You shouldn't, can't and mustn't rely on ordering of such unsynchronized events.)

zajio1am 4y ago
> No, the behavior is not correct, because changing /etc/localtime is an asynchronous operation Well, technically you can do both from the same process (i.e. localtime(), unlink() + symlink() to change /etc/localtime, another localtime()), in that case it would be synchronous. But in general i agree, if the timeout was small enough (say < 100 ms) i guess nobody would object.