13 ms·

How setting the TZ environment variable avoids thousands of system calls

472 points by jcapote 10y ago

pquerna 10y ago
Good blog post explaining the behavior of glibc, I also saw this first hand when profiling Apache awhile back too: http://mail-archives.apache.org/mod_mbox/httpd-dev/201111.mbox/%3CCAMDeyhzRAZ4eyz%3D%2BstA%3DwoTibM-W6QL8TqT%2BaPio07UddCz7Tg%40mail.gmail.com%3E http://mail-archives.apache.org/mod_mbox/httpd-dev/201111.mb... https://github.com/apache/httpd/blob/trunk/server/util_time.c#L310-L327 https://github.com/apache/httpd/blob/trunk/server/util_time.... The internals of glibc can often be pretty surprising sometimes, I'd really encourage people to go spelunking into the glibc source when they are profiling applications.
sandGorgon 10y ago
does anyone know if this impacts docker images as well ?
- yeukhon 10y ago
  I would be surprised if this doesn't. There's nothing specific about docker when running this C code from any other processes running on a Linux host.
- noselasd 10y ago
  If you havn't set the TZ variable in the image, then yes.
- pquerna 10y ago
  Yes, at least the Docker images that use glibc as their libc. (eg, most Debian/Ubuntu images) It looks like musl, which is used on Alpine Linux images for example, will only read it once, and then cache it: https://github.com/esmil/musl/blob/master/src/time/__tz.c#L121 https://github.com/esmil/musl/blob/master/src/time/__tz.c#L1... It has a mutex/lock around the use of the TZ info, but avoids re-stat'ing the localtime file.
  - nathancahill 10y ago
    This is the best part of HN. Not only did you answer GP's question in 6 minutes, but you link to the exact line of the source code.
drudru11 10y ago
Side note - why do some sites completely hide information about who is behind them? I couldn't find a single thing about that on their blog or main site.
- jasonmp85 10y ago
  Maybe it's just that I use packagecloud and follow people in their circle on Twitter, but Joe Damato is the CEO and founder: https://twitter.com/joedamato https://twitter.com/joedamato I'm under the impression that he may also write a lot these himself? Not entirely certain, though.
  - jcdavis 10y ago
    Pretty sure he writes all of the linux internals posts. He also has a bunch of great ones on his blog http://timetobleed.com/ http://timetobleed.com/
- jlg23 10y ago
  All the links are in the footer of the page.
  - ryanlol 10y ago
    For what it's worth, they disappear if your browser isn't wide enough.
    - jcapote 10y ago
      Oops! I'll fix this shortly, thanks for pointing it out.
- avar 10y ago
  It's not in any way hidden, just run a whois query: $ whois packagecloud.io|grep Owner Owner Name : JOSEPH DAMATO Owner OrgName : COMPUTOLOGY, LLC Owner Addr : 359 FILLMORE ST!12 Owner Addr : SAN FRANCISCO Owner Addr : CA Owner Addr : US
jonathonf 10y ago
If this has a real-world/measurable/etc. impact why isn't this set by default? Are there potential side-effects? Is it set in some distros but not others?
- mixologic 10y ago
  Probably because while there may be tens of thousands of additional syscalls, the total amount of added latency and resources consumed are more likely to be on a scale of micro/nano/milli seconds.
  - shawnz 10y ago
    In the trace you can see that the syscall takes less than a tenth of a millisecond. I don't think this is a big penalty to check if I have changed my timezone or not, as unlikely as that is during normal operation.
- peterwwillis 10y ago
  Portability, compatibility. The system should not set environment variables if there is a reasonable default action. Env is intended to be set by the user. In general, the timezone is set during OS setup, and the system is left in a state where it's up to the applications to figure out what to do. For example, you might configure Apache (yes, I am old, leave me alone) to use a particular timezone. But if Apache senses an env var it may choose to override the configured value with what's in the env var. Or SSH might be configured to pass along all env vars, including TZ, which in all honesty it probably won't even if you tried, but it could, and then the destination server's application has the wrong timezone. Point is, it's safer not to mess with env vars unless you need to.
- noselasd 10y ago
  The default behavior is done so programs don't need to be restarted if the timezone is changed, which also has real world impact.
rootbear 10y ago
Is there a reason why the path to the timezone file is prefixed with a colon? TZ=:/etc/localtime I've set TZ sometimes without the colon and it seem to work. I did a quick online search and didn't find anything relevant.
- deleted 10y ago
  [deleted]
- avar 10y ago
  :<whatever> means "read it from the <whatever>" file. See the last part of the relevant glibc documentation: https://www.gnu.org/savannah-checkouts/gnu/libc/manual/html_node/TZ-Variable.html https://www.gnu.org/savannah-checkouts/gnu/libc/manual/html_... However the reason it works without : is that the implementation is being lazy and just ignores the : delimiter and falls back to parsing out a filename either way: https://sourceware.org/git/?p=glibc.git;a=blob;f=time/tzset.c;hb=refs/heads/master#l418 https://sourceware.org/git/?p=glibc.git;a=blob;f=time/tzset....
  - rootbear 10y ago
    You beat me to it. I was answering my own question when one of my users came in with a problem. Stupid users...
- rootbear 10y ago
  Here is the answer: https://www.gnu.org/software/libc/manual/html_node/TZ-Variable.html https://www.gnu.org/software/libc/manual/html_node/TZ-Variab... The third format looks like this: :characters Each operating system interprets this format differently; in the GNU C Library, characters is the name of a file which describes the time zone. The other formats specify the timezone directly, such as EST+5EDT. Interestingly, it seems to work okay without the colon. Perhaps the leading slash implies a filename?
  - deleted 10y ago
    [deleted]
- 10y ago
mozumder 10y ago
Does this affect FreeBSD as well?
- loeg 10y ago
  Experimentally, no. The example program calls localtime(3) 10 times but only accesses the file once, per truss: write(1,"Greetings!n",11) = 11 (0xb) access("/etc/localtime",R_OK) = 0 (0x0) open("/etc/localtime",O_RDONLY,037777777600) = 3 (0x3) fstat(3,{ mode=-r--r--r-- ,inode=11316113,size=2819,blksize=32768 }) = 0 (0x0) read(3,"TZif20000000000000"...,41448) = 2819 (0xb03) close(3) = 0 (0x0) issetugid() = 0 (0x0) open("/usr/share/zoneinfo/posixrules",O_RDONLY,00) = 3 (0x3) fstat(3,{ mode=-r--r--r-- ,inode=327579,size=3519,blksize=32768 }) = 0 (0x0) read(3,"TZif20000000000000"...,41448) = 3519 (0xdbf) close(3) = 0 (0x0) write(1,"Godspeed, dear friend!n",23) = 23 (0x17) (FreeBSD caches the database on the first call: https://svnweb.freebsd.org/base/head/contrib/tzcode/stdtime/localtime.c?view=markup#l1437 https://svnweb.freebsd.org/base/head/contrib/tzcode/stdtime/... )
- rini17 10y ago
  It is unlikely, they usually don't use GNU C library.
tytso 10y ago
System calls in Linux are really fast. So saving "thousands" of system calls when /etc/localtime is in cache doesn't actually save that much actual CPU time. I ran an experiment where I timed the runtime of the sample program provided in the OP, except I changed the number of calls to localtime() from ten times to a million. I then timed the difference with and without export TZ=:/etc/localhost. The net savings was .6 seconds. So for a single call to localtime(3), the net savings is 0.6 microseconds. That's non-zero, but it's likely in the noise compared to everything else that your program might be doing.
- dsl 10y ago
  On your base system, yes. Lots of things can hook random syscalls, or environments might have syscall monitoring. One example is the folks over at slack record every syscall for security auditing. https://slack.engineering/syscall-auditing-at-scale-e6a3ca8ac1b8#.5lj2m5nv5 https://slack.engineering/syscall-auditing-at-scale-e6a3ca8a...
  - tytso 10y ago
    Slack uses the Linux audit subsystem which is also certainly faster than you think it is. Consider how many system calls your typical application is issuing --- especially ones that are likely to be calling localtime() all the time, such as a web server. If system call auditing had that high of an overhead, everything would be horrifically slow --- but it isn't, because Linux audit sends its records out asynchronously and in batches.
    - nwmcsween 10y ago
      https://www.redhat.com/archives/linux-audit/2015-January/msg00171.html https://www.redhat.com/archives/linux-audit/2015-January/msg... of course this is RHEL 2.6.32 and it's open/close but 200000 sc/s vs 3000 sc/s shows it has some overhead. Maybe someone can rerun that test code on git and see what the overhead is.
- peterwwillis 10y ago
  Yeah, this is a perfect example of micro-optimization being unnecessary. Not only will you not see performance issues from this in the real world, it might cause problems down the road, because since it isn't set by default this way, some apps may not expect it and behave erroneously. But it's neat information to have in the back of your head.
rocky1138 10y ago
It is perhaps out of scope of the article, but it sure would have been helpful to show how to set the TZ environment variable and what to set it to.
- mmozeiko 10y ago
  Its right there in middle of article (under "Preventing extraneous system calls" section): $ TZ=:/etc/localtime strace -ttT ./test
  - rocky1138 10y ago
    I can't believe I missed that! Thanks.
kseistrup 10y ago
It would be highly inconvenient to have to set this variable if you live in a country where you change the timezone twice a year due to summertime.
- stormbrew 10y ago
  Are you talking about DST? You don't change your timezone at DST switch, the timezone data knows how to calculate the correct time based on your location (roughly) and the time of year. Mountain standard time and mountain daylight time are both part of mountain time, for example.
  - kseistrup 10y ago
    What I meant was: Currently my timezone is CET. In a month's time it will be CEST. If I have to set TZ to explicitly mirror that it will be a burden.
    - Symbiote 10y ago
      TZ may be set like this: TZ=:Europe/Copenhagen Replace Europe/Copenhagen with the appropriate entry from the list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones https://en.wikipedia.org/wiki/List_of_tz_database_time_zones Usually, /etc/localtime is a symlink, as on my laptop: /etc/localtime -> /usr/share/zoneinfo/Europe/Copenhagen so TZ=:/etc/localtime has the same result. You can demonstrate that it takes account of changes to timezones: Normal time for London: $ TZ=:Europe/London date -d '1995-12-30 12:00 UTC' -R Sat, 30 Dec 1995 12:00:00 +0000 British Summer Time: $ TZ=:Europe/London date -d '1995-06-30 12:00 UTC' -R Fri, 30 Jun 1995 13:00:00 +0100 'Double British Summer Time', used for a period during World War 2: $ TZ=:Europe/London date -d '1945-06-30 12:00 UTC' -R Sat, 30 Jun 1945 14:00:00 +0200 Before London's time was standardized to Greenwich: $ TZ=:Europe/London date -d '1845-06-30 12:00 UTC' -R Mon, 30 Jun 1845 11:58:45 -0001
      - kseistrup 10y ago
        Thanks!
Daviey 10y ago
Honestly, the primary reason I support this is to get developers out of the habbit of demanding a localized server timezone. As an infra' person, I want system time in UTC. If developers get in the habbit of setting TZ, then I can have this!
- int_19h 10y ago
  It feels like any code that needs to know the timezone of the server is inherently wrong. If timezone ever comes up in any context, it's either the timezone of the client from whom the request originates - in which case it should come as part of the request - or else the timezone somehow associated with the business process (e.g. "warehouse open 8-5 Eastern time"), in which case it should be part of the configuration for that one service.
rdtsc 10y ago
Great post. I remember when vDSOs were added we noticed a nice speedup in our code. We tuned for realtime and a few microseconds here and there add up. Most importantly, less systems calls means more predictability.
hodgesrm 10y ago
Interesting article but I can't reproduce the behavior on Ubuntu 16.01 LTS. I don't have TZ set (or anything locale-related for that matter). Here are the library dependencies: $ ldd test linux-vdso.so.1 => (0x00007ffd80baf000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8844bf7000) /lib64/ld-linux-x86-64.so.2 (0x00007f8844fbc000) Any thoughts why the behavior would be different?
- cnvogel 10y ago
  There's also a surprising difference in behavior between tm = localtime() and localtime_r(..., &tm). The former is the "traditional" function which returns a pointer to a statically allocated, global "struct_tm". The latter is the thread-safe version receiving a pointer to a use-supplied "struct tm" as it's second argument. : do { : t = time(NULL); : localtime_r(&t, &tm); : printf("The time is now %02d:%02d:%02d.\n", : tm.tm_hour, tm.tm_min, tm.tm_sec); : sleep(1); : } while(--N); with TZ set to Europe/Berlin, set to :/etc/localtime, or unset I never get a stat on anything. write(1, "The time is now 07:23:33.\n", 26The time is now 07:23:33. ) = 26 nanosleep({tv_sec=1, tv_nsec=0}, 0x7ffd9e798470) = 0 write(1, "The time is now 07:23:34.\n", 26The time is now 07:23:34. ) = 26 nanosleep({tv_sec=1, tv_nsec=0}, 0x7ffd9e798470) = 0 write(1, "The time is now 07:23:35.\n", 26The time is now 07:23:35. ) = 26 nanosleep({tv_sec=1, tv_nsec=0}, 0x7ffd9e798470) = 0 If I change it to tm = localtime()... stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2335, ...}) = 0 write(1, "The time is now 07:30:56.\n", 26The time is now 07:30:56.) = 26 nanosleep({tv_sec=1, tv_nsec=0}, 0x7ffc868c3010) = 0 One more reason to switch to the reentrant/thread-safe versions of those ugly library functions :-). Note, this is using glibc 2.24 under Arch. $ /lib/libc.so.6 GNU C Library (GNU libc) stable release version 2.24, by Roland McGrath et al. (...) Compiled by GNU CC version 6.1.1 20160802.
AceJohnny2 10y ago
> In other words: your system supports calling the time system call via the Linux kernel’s vDSO to avoid the cost of switching to the kernel. But, as soon as your program calls time, it calls localtime immediately after, which invokes a system call anyway. This reminds me of an article by Ted Unangst[1], in which he flattens the various libraries and abstractions to show how xterm (to cite one of many culprits) in one place is effectively doing: if (poll() || poll()) while (poll()) { /* ... */ } In other words, if you don't know what your library/abstraction is doing, you can end up accidentally duplicating its work. Reminds me of some aphorism, "Those who do not learn from history..." ;) [1] http://www.tedunangst.com/flak/post/accidentally-nonblocking http://www.tedunangst.com/flak/post/accidentally-nonblocking discussed https://news.ycombinator.com/item?id=11847529 https://news.ycombinator.com/item?id=11847529
- nsxwolf 10y ago
  Those who quote George Santayana are condemned to repeat him.
  - andrewbinstock 10y ago
    Those who don't know George Santayana are condemned to repeat him. FTFY.
    - koverstreet 10y ago
      You seem to have missed the joke...
      - btown 10y ago
        You seem to have missed the joke... wait a second...
      - andrewbinstock 10y ago
        Totally got it. It's a refinement of the line. As the person below understands.
leovonl 10y ago
This seems to be a simple RTFM issue to me: POSIX specifies that gmtime() uses UTC and localtime() uses current timezone. Using gmtime() would implement the desired behaviour without any need to hardcode environment variables.
- mwexler 10y ago
  ...which fixes all the code you wrote, but of course, you may have legacy binaries that you don't have access to source to change... hence a simple setting of an environment variable, hardcoded though it may be, fixes the situation for all. Though PeterWillis makes a good point akin to yours, and your (plural) point does make sense. (Edit: added mention of comment with additional background on why to avoid hardcoding the variable)
cbsmith 10y ago
This reads to me like a glibc bug. Glibc should just be watching "/etc/localtime" for changes, rather than calling out to hundreds of times a second.
- gumby 10y ago
  And how does it watch it? With the stat() syscall.
  - deleted 10y ago
    [deleted]
  - Dylan16807 10y ago
    Polling is not watching.
- noselasd 10y ago
  There's no usable mechanisms glibc can use for watching /etc/localtime for changes that does not mess up the program if it also decides to use any file watching features.
  - cbsmith 10y ago
    At least on key platforms, it's pretty easy to use an event driven model and watch for updates. Hell, if the vDSO handled TZ it'd be no problem.
    - noselasd 10y ago
      I'm not saying it's impossible, but if you actually sit down and attempt to add this to glibc, you will come to the realization that it is not easy at all. If you want to e.g. use the inotify mechanism you have a few key decision points to make * There's no place where glibc can run an event loop to watch for the changes, the application might not have an event loop. * If you need to integrate with an event loop the application runs, it'll work fine as long as the user remembers to hook up the events and deal with the corner cases. You can use this approach without any special support for glibc, and set TZ env. variable yourself when /etc/localtime changes - though at the moment that will only work for single threaded programs (setenv()/putenv() is not thread safe) * If you decide to run the event loop in a separate thread, you force every binary to be multi threaded and have quite a lot of corner cases to handle when fork()'ing. * If you don't run an event loop, you're back to polling for changes, and hardly anything is gained. * If you use the signal driven I/O notification mechanism, you interfer with the application use of signals and I/O notification, and also have a host of fork() corner cases to consider. vDSO is not a magic silver bullet that can solve this, you would at least have to have the kernel manage timezone support, or perhaps better, have the ability to transparently manage arbitarily data, and then expose that through shared memory that vDSO can use. This will not happen anytime soon.
actuator 10y ago
While trying to find the cause of slowness in Rails requests, I was running strace on an unicorn process when I encountered the same thing mentioned in the article. Rails instrumentation code calls current time before and after any instrumentation block. So, when I looked at the trace there were a lot of `stat` calls coming for `/etc/localtime` and as stat is an IO operation, I thought I discovered the cause of slowness(which I attributed to high number of IO ops) but surprisingly when I saw the strace method summary; while the call count was high, the time taken by the calls in total was not significant(<1% if I remember correctly). So I decided to set TZ with the next AMI update 15 months back but forgot about it totally. I guess I should add it to my Trello list this time. Also, I think he should have printed the aggregate summary of just CPU clock time(`-c`) as well as that is usually very low.
- caf 10y ago
  ...while the call count was high, the time taken by the calls in total was not significant(<1% if I remember correctly). Yes, on ordinary filesystems if you run stat() over and over again on the same file then it's just copying from the in-memory inode into your struct stat, there's no IO.
creeble 10y ago
Does anyone have any evidence of this actually having a resource usage impact on any common programs? I see one reference to Apache below, but not whether it actually made a measurable difference.
dpatru 10y ago
It seems to me that this is something the Linux distributions should already be doing.
kelnos 10y ago
I really enjoy when people dig into things like this and report their findings. Having said that, I question the wisdom of "bothering" with this sort of thing. Everything you do that's non-standard or works against a system's default behavior incurs a cost. It's yet another thing you have to replicate when you migrate to a new version, change provisioning systems, etc. And for what benefit? A few hundred syscalls per second? Linux syscalls are fast enough that something of that magnitude shouldn't matter much. Given that /etc/localtime will certainly be in cache with that frequency of access, a stat() should do little work in the kernel to return, so that won't be slow either. It's good that they did some benchmarking to look at the differences, but this feels like a premature optimization to me. I can't imagine that this did anything but make their application a tiny fraction of a percent faster. Was it worth the time to dig into that for this increase? Was it worth the maintenance cost I mention in my first paragraph? I wouldn't think so. I'm really trying not to take a crap on what they did; as I said, it's really cool to dig into these sorts of abstractions and find out where they're inefficient or leak (or just great as a learning exercise; we all depend on a mountain of code that most people don't understand at all). But, when looked at from a holistic systems approach, a grab bag of little "tweaks" like this can become harmful in the long run.
- lotyrin 10y ago
  You should be able to get maintenance for something like this pretty low. Add a line with a nice comment to the config template for running your apps that adds an extra env var. Any machine that runs any app now has the line... I would definitely like to see a before/after real-world metric on impact here though.
- bradfa 10y ago
  The embedded Linux system I'm working on right now takes about 17 us per stat() call due to this but time will always be kept internally as UTC, so taking advantage of this is worth considering for me. Since most embedded systems can directly translate the amount of processing needed to achieve the product goal into a real dollar cost of the hardware, any savings, even a small one, is worth investigating. Since the hardware and system are generally well understood, implementing something like this is much more reasonable. But I agree, on a general purpose OS doing general purpose thing, an optimization like what's proposed by the article may not be worth the other tradeoffs.
vesinisa 10y ago
This was a thoroughly fascinating read. Highly recommend reading the previous part in the series as well.
blunte 10y ago
This reminds me of a very similar behavior in Solaris over 20 years ago. Our C application was having odd performance problems on some client systems, and eventually we saw via truss that there were hundreds of fopen() calls every second to get the timezone. Setting the right environment variable solved the problem.
astrostl 10y ago
I do this for "not get annoyed while stracing" reasons, not perf!
jakeogh 10y ago
OpenRC users can: echo 'TZ=:/etc/localtime' > /etc/env.d/00localtime
- JensRex 10y ago
  For Debian and derivatives: echo 'TZ=:/etc/localtime' >> /etc/environment
jdamato 10y ago
Author of the post here: greetings. If you enjoyed this post, you may also enjoy our deep dive explaining exactly how system calls work on Linux[1]. [1]: https://blog.packagecloud.io/eng/2016/04/05/the-definitive-guide-to-linux-system-calls/ https://blog.packagecloud.io/eng/2016/04/05/the-definitive-g...
rargulati 10y ago
Really interesting - thanks for sharing the findings. I haven't seen it mentioned here, but for those of us using `timedatectl` via systemd, with the default setting of `UTC` are taking advantage[1] of the recommendation in the article. [1] https://github.com/systemd/systemd/blob/master/src/timedate/timedatectl.c#L85 https://github.com/systemd/systemd/blob/master/src/timedate/...
glandium 10y ago
What is missing in this post is: - Why does glibc check /etc/localtime every time localtime is called? Wild guess: so that new values of /etc/localtime are picked at runtime without restarting programs. - Corollary: why does glibc not check /etc/localtime every time localtime is called, when TZ is set to :/etc/localtime? Arguably the reason above should still apply when TZ is set to a file name, shouldn't it?
- jdamato 10y ago
  Hi, both are answered in the article: First: > What’s going on here is that the first call to localtime in glibc opens and reads the contents of /etc/localtime. All subsequent calls to localtime internally call stat, but they do this to ensure that the timezone file has not changed. and second: read the section titled "Preventing extraneous system calls" for the answer to your second question.
  - glandium 10y ago
    Those "answers" are more about how than why.
    - jdamato 10y ago
      Thanks for reading and I'm glad to hear you loved my post!
- glandium 10y ago
  For the second question: There doesn't seem to be an explicit reason for the difference of treatment. The code that does it has been there since 1996, and hasn't changed since. The only reason given is "Caching happens based on the contents of the environment variable TZ.". https://sourceware.org/git/?p=glibc.git;a=commit;h=68dbb3a69e78e24a778c6602c8cc91d715839d08 https://sourceware.org/git/?p=glibc.git;a=commit;h=68dbb3a69... I'd argue it should cache the same when both old_tz and tz are NULL (but start with an old_tz that is not NULL). I was about to file an upstream bug, but found https://sourceware.org/bugzilla/show_bug.cgi?id=5184 https://sourceware.org/bugzilla/show_bug.cgi?id=5184 and https://sourceware.org/bugzilla/show_bug.cgi?id=5186 https://sourceware.org/bugzilla/show_bug.cgi?id=5186 The latter actually implies the opposite should be happening: files given in TZ should be stat()ed just as much as /etc/localtime.
snowcrshd 10y ago
Brendan Gregg wrote about this a few years ago [1]. My favorite part: > WTF?? Why is ls(1) running stat() on /etc/localtime for every line of output? [1] http://www.brendangregg.com/blog/2014-05-11/strace-wow-much-syscall.html http://www.brendangregg.com/blog/2014-05-11/strace-wow-much-...
acscott 10y ago
This reminds me of setting noatime for disk mounts (http://askubuntu.com/questions/2099/is-it-worth-to-tune-ext4-with-noatime http://askubuntu.com/questions/2099/is-it-worth-to-tune-ext4...) Now I want to know the number of other configs to reduce the number of system calls. This all adds up to being significant the greater the number of hosts in your environment.
brendangregg 10y ago
Please quantify the speedup (I've found this before, but it's never been a significant issue). Eliminating unnecessary work is great, but what are we really talking about here? Use a CPU flamegraph, Ctrl-F and search for stat functions. It'll quantify the total on the bottom right.
- brendangregg 10y ago
  Oh, and another page that recommends strace without warning about overheads. Dangerous.
mattronson 10y ago
Linux is very fast !!
rtsisyk 10y ago
Overhead of localtime() is well-known, just RTFM. Anyway, this article provides very good explanation.
rtsisyk 10y ago
BTW, packagecloud.io is the great hosting for RPM/DEB packages. We've been using it for the last couple years. GitHub + Travis CI + PackageCloud combination allows us build and publish packages for EVERY git commit in 30+ repositories targeting 15 different Linux distributions [1]. There is no more need to hire a special devops guy for that. [1]: https://github.com/packpack/packpack#packpack https://github.com/packpack/packpack#packpack
falsedan 10y ago
Why did this post start with a tl;dr, then a Summary, and _still_ buried the important takeaway in the ultimate paragraph?
scottlamb 10y ago
There's another easy way to avoid this: use localtime_r instead of localtime. From the glibc source: /* Update internal database according to current TZ setting. POSIX.1 8.3.7.2 says that localtime_r is not required to set tzname. This is a good idea since this allows at least a bit more parallelism. */ tzset_internal (tp == &_tmbuf && use_localtime, 1); mktime also does the tzset call every time, though: time_t mktime (struct tm *tp) { #ifdef _LIBC /* POSIX.1 8.1.1 requires that whenever mktime() is called, the time zone names contained in the external variable 'tzname' shall be set as if the tzset() function had been called. */ __tzset (); #endif and I don't see any way around that other than setting TZ=: or some such.
barrystaes 10y ago
Im not an expert but the first thing that comes to mind is that 1) TFA does not quantify the performance gain in time 2) I wonder if environment variables like TZ are a security risk/vector in that these might facilitate attackers to stealthy skew/screw time within current user process... no root required.

472 points by jcapote 10y ago

Good blog post explaining the behavior of glibc, I also saw this first hand when profiling Apache awhile back too: http://mail-archives.apache.org/mod_mbox/httpd-dev/201111.mbox/%3CCAMDeyhzRAZ4eyz%3D%2BstA%3DwoTibM-W6QL8TqT%2BaPio07UddCz7Tg%40mail.gmail.com%3E http://mail-archives.apache.org/mod_mbox/httpd-dev/201111.mb... https://github.com/apache/httpd/blob/trunk/server/util_time.c#L310-L327 https://github.com/apache/httpd/blob/trunk/server/util_time.... The internals of glibc can often be pretty surprising sometimes, I'd really encourage people to go spelunking into the glibc source when they are profiling applications.

sandGorgon 10y ago

does anyone know if this impacts docker images as well ?

yeukhon 10y ago
I would be surprised if this doesn't. There's nothing specific about docker when running this C code from any other processes running on a Linux host.
noselasd 10y ago
If you havn't set the TZ variable in the image, then yes.
pquerna 10y ago
Yes, at least the Docker images that use glibc as their libc. (eg, most Debian/Ubuntu images) It looks like musl, which is used on Alpine Linux images for example, will only read it once, and then cache it: https://github.com/esmil/musl/blob/master/src/time/__tz.c#L121 https://github.com/esmil/musl/blob/master/src/time/__tz.c#L1... It has a mutex/lock around the use of the TZ info, but avoids re-stat'ing the localtime file.
- nathancahill 10y ago
  This is the best part of HN. Not only did you answer GP's question in 6 minutes, but you link to the exact line of the source code.

drudru11 10y ago

Side note - why do some sites completely hide information about who is behind them? I couldn't find a single thing about that on their blog or main site.

jasonmp85 10y ago
Maybe it's just that I use packagecloud and follow people in their circle on Twitter, but Joe Damato is the CEO and founder: https://twitter.com/joedamato https://twitter.com/joedamato I'm under the impression that he may also write a lot these himself? Not entirely certain, though.
- jcdavis 10y ago
  Pretty sure he writes all of the linux internals posts. He also has a bunch of great ones on his blog http://timetobleed.com/ http://timetobleed.com/
jlg23 10y ago
All the links are in the footer of the page.
- ryanlol 10y ago
  For what it's worth, they disappear if your browser isn't wide enough.
  - jcapote 10y ago
    Oops! I'll fix this shortly, thanks for pointing it out.
avar 10y ago
It's not in any way hidden, just run a whois query: $ whois packagecloud.io|grep Owner Owner Name : JOSEPH DAMATO Owner OrgName : COMPUTOLOGY, LLC Owner Addr : 359 FILLMORE ST!12 Owner Addr : SAN FRANCISCO Owner Addr : CA Owner Addr : US

jonathonf 10y ago

If this has a real-world/measurable/etc. impact why isn't this set by default? Are there potential side-effects? Is it set in some distros but not others?

mixologic 10y ago
Probably because while there may be tens of thousands of additional syscalls, the total amount of added latency and resources consumed are more likely to be on a scale of micro/nano/milli seconds.
- shawnz 10y ago
  In the trace you can see that the syscall takes less than a tenth of a millisecond. I don't think this is a big penalty to check if I have changed my timezone or not, as unlikely as that is during normal operation.
peterwwillis 10y ago
Portability, compatibility. The system should not set environment variables if there is a reasonable default action. Env is intended to be set by the user. In general, the timezone is set during OS setup, and the system is left in a state where it's up to the applications to figure out what to do. For example, you might configure Apache (yes, I am old, leave me alone) to use a particular timezone. But if Apache senses an env var it may choose to override the configured value with what's in the env var. Or SSH might be configured to pass along all env vars, including TZ, which in all honesty it probably won't even if you tried, but it could, and then the destination server's application has the wrong timezone. Point is, it's safer not to mess with env vars unless you need to.
noselasd 10y ago
The default behavior is done so programs don't need to be restarted if the timezone is changed, which also has real world impact.

rootbear 10y ago

Is there a reason why the path to the timezone file is prefixed with a colon? TZ=:/etc/localtime I've set TZ sometimes without the colon and it seem to work. I did a quick online search and didn't find anything relevant.

deleted 10y ago
[deleted]
avar 10y ago
:<whatever> means "read it from the <whatever>" file. See the last part of the relevant glibc documentation: https://www.gnu.org/savannah-checkouts/gnu/libc/manual/html_node/TZ-Variable.html https://www.gnu.org/savannah-checkouts/gnu/libc/manual/html_... However the reason it works without : is that the implementation is being lazy and just ignores the : delimiter and falls back to parsing out a filename either way: https://sourceware.org/git/?p=glibc.git;a=blob;f=time/tzset.c;hb=refs/heads/master#l418 https://sourceware.org/git/?p=glibc.git;a=blob;f=time/tzset....
- rootbear 10y ago
  You beat me to it. I was answering my own question when one of my users came in with a problem. Stupid users...
rootbear 10y ago
Here is the answer: https://www.gnu.org/software/libc/manual/html_node/TZ-Variable.html https://www.gnu.org/software/libc/manual/html_node/TZ-Variab... The third format looks like this: :characters Each operating system interprets this format differently; in the GNU C Library, characters is the name of a file which describes the time zone. The other formats specify the timezone directly, such as EST+5EDT. Interestingly, it seems to work okay without the colon. Perhaps the leading slash implies a filename?
- deleted 10y ago
  [deleted]
10y ago

mozumder 10y ago

Does this affect FreeBSD as well?

loeg 10y ago
Experimentally, no. The example program calls localtime(3) 10 times but only accesses the file once, per truss: write(1,"Greetings!n",11) = 11 (0xb) access("/etc/localtime",R_OK) = 0 (0x0) open("/etc/localtime",O_RDONLY,037777777600) = 3 (0x3) fstat(3,{ mode=-r--r--r-- ,inode=11316113,size=2819,blksize=32768 }) = 0 (0x0) read(3,"TZif20000000000000"...,41448) = 2819 (0xb03) close(3) = 0 (0x0) issetugid() = 0 (0x0) open("/usr/share/zoneinfo/posixrules",O_RDONLY,00) = 3 (0x3) fstat(3,{ mode=-r--r--r-- ,inode=327579,size=3519,blksize=32768 }) = 0 (0x0) read(3,"TZif20000000000000"...,41448) = 3519 (0xdbf) close(3) = 0 (0x0) write(1,"Godspeed, dear friend!n",23) = 23 (0x17) (FreeBSD caches the database on the first call: https://svnweb.freebsd.org/base/head/contrib/tzcode/stdtime/localtime.c?view=markup#l1437 https://svnweb.freebsd.org/base/head/contrib/tzcode/stdtime/... )
rini17 10y ago
It is unlikely, they usually don't use GNU C library.

tytso 10y ago

System calls in Linux are really fast. So saving "thousands" of system calls when /etc/localtime is in cache doesn't actually save that much actual CPU time. I ran an experiment where I timed the runtime of the sample program provided in the OP, except I changed the number of calls to localtime() from ten times to a million. I then timed the difference with and without export TZ=:/etc/localhost. The net savings was .6 seconds. So for a single call to localtime(3), the net savings is 0.6 microseconds. That's non-zero, but it's likely in the noise compared to everything else that your program might be doing.

dsl 10y ago
On your base system, yes. Lots of things can hook random syscalls, or environments might have syscall monitoring. One example is the folks over at slack record every syscall for security auditing. https://slack.engineering/syscall-auditing-at-scale-e6a3ca8ac1b8#.5lj2m5nv5 https://slack.engineering/syscall-auditing-at-scale-e6a3ca8a...
- tytso 10y ago
  Slack uses the Linux audit subsystem which is also certainly faster than you think it is. Consider how many system calls your typical application is issuing --- especially ones that are likely to be calling localtime() all the time, such as a web server. If system call auditing had that high of an overhead, everything would be horrifically slow --- but it isn't, because Linux audit sends its records out asynchronously and in batches.
  - nwmcsween 10y ago
    https://www.redhat.com/archives/linux-audit/2015-January/msg00171.html https://www.redhat.com/archives/linux-audit/2015-January/msg... of course this is RHEL 2.6.32 and it's open/close but 200000 sc/s vs 3000 sc/s shows it has some overhead. Maybe someone can rerun that test code on git and see what the overhead is.
peterwwillis 10y ago
Yeah, this is a perfect example of micro-optimization being unnecessary. Not only will you not see performance issues from this in the real world, it might cause problems down the road, because since it isn't set by default this way, some apps may not expect it and behave erroneously. But it's neat information to have in the back of your head.

rocky1138 10y ago

It is perhaps out of scope of the article, but it sure would have been helpful to show how to set the TZ environment variable and what to set it to.

mmozeiko 10y ago
Its right there in middle of article (under "Preventing extraneous system calls" section): $ TZ=:/etc/localtime strace -ttT ./test
- rocky1138 10y ago
  I can't believe I missed that! Thanks.

kseistrup 10y ago

It would be highly inconvenient to have to set this variable if you live in a country where you change the timezone twice a year due to summertime.

stormbrew 10y ago
Are you talking about DST? You don't change your timezone at DST switch, the timezone data knows how to calculate the correct time based on your location (roughly) and the time of year. Mountain standard time and mountain daylight time are both part of mountain time, for example.
- kseistrup 10y ago
  What I meant was: Currently my timezone is CET. In a month's time it will be CEST. If I have to set TZ to explicitly mirror that it will be a burden.
  - Symbiote 10y ago
    TZ may be set like this: TZ=:Europe/Copenhagen Replace Europe/Copenhagen with the appropriate entry from the list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones https://en.wikipedia.org/wiki/List_of_tz_database_time_zones Usually, /etc/localtime is a symlink, as on my laptop: /etc/localtime -> /usr/share/zoneinfo/Europe/Copenhagen so TZ=:/etc/localtime has the same result. You can demonstrate that it takes account of changes to timezones: Normal time for London: $ TZ=:Europe/London date -d '1995-12-30 12:00 UTC' -R Sat, 30 Dec 1995 12:00:00 +0000 British Summer Time: $ TZ=:Europe/London date -d '1995-06-30 12:00 UTC' -R Fri, 30 Jun 1995 13:00:00 +0100 'Double British Summer Time', used for a period during World War 2: $ TZ=:Europe/London date -d '1945-06-30 12:00 UTC' -R Sat, 30 Jun 1945 14:00:00 +0200 Before London's time was standardized to Greenwich: $ TZ=:Europe/London date -d '1845-06-30 12:00 UTC' -R Mon, 30 Jun 1845 11:58:45 -0001
    - kseistrup 10y ago
      Thanks!

Daviey 10y ago

Honestly, the primary reason I support this is to get developers out of the habbit of demanding a localized server timezone. As an infra' person, I want system time in UTC. If developers get in the habbit of setting TZ, then I can have this!

int_19h 10y ago
It feels like any code that needs to know the timezone of the server is inherently wrong. If timezone ever comes up in any context, it's either the timezone of the client from whom the request originates - in which case it should come as part of the request - or else the timezone somehow associated with the business process (e.g. "warehouse open 8-5 Eastern time"), in which case it should be part of the configuration for that one service.

rdtsc 10y ago

Great post. I remember when vDSOs were added we noticed a nice speedup in our code. We tuned for realtime and a few microseconds here and there add up. Most importantly, less systems calls means more predictability.

hodgesrm 10y ago

Interesting article but I can't reproduce the behavior on Ubuntu 16.01 LTS. I don't have TZ set (or anything locale-related for that matter). Here are the library dependencies: $ ldd test linux-vdso.so.1 => (0x00007ffd80baf000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8844bf7000) /lib64/ld-linux-x86-64.so.2 (0x00007f8844fbc000) Any thoughts why the behavior would be different?

cnvogel 10y ago
There's also a surprising difference in behavior between tm = localtime() and localtime_r(..., &tm). The former is the "traditional" function which returns a pointer to a statically allocated, global "struct_tm". The latter is the thread-safe version receiving a pointer to a use-supplied "struct tm" as it's second argument. : do { : t = time(NULL); : localtime_r(&t, &tm); : printf("The time is now %02d:%02d:%02d.\n", : tm.tm_hour, tm.tm_min, tm.tm_sec); : sleep(1); : } while(--N); with TZ set to Europe/Berlin, set to :/etc/localtime, or unset I never get a stat on anything. write(1, "The time is now 07:23:33.\n", 26The time is now 07:23:33. ) = 26 nanosleep({tv_sec=1, tv_nsec=0}, 0x7ffd9e798470) = 0 write(1, "The time is now 07:23:34.\n", 26The time is now 07:23:34. ) = 26 nanosleep({tv_sec=1, tv_nsec=0}, 0x7ffd9e798470) = 0 write(1, "The time is now 07:23:35.\n", 26The time is now 07:23:35. ) = 26 nanosleep({tv_sec=1, tv_nsec=0}, 0x7ffd9e798470) = 0 If I change it to tm = localtime()... stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2335, ...}) = 0 write(1, "The time is now 07:30:56.\n", 26The time is now 07:30:56.) = 26 nanosleep({tv_sec=1, tv_nsec=0}, 0x7ffc868c3010) = 0 One more reason to switch to the reentrant/thread-safe versions of those ugly library functions :-). Note, this is using glibc 2.24 under Arch. $ /lib/libc.so.6 GNU C Library (GNU libc) stable release version 2.24, by Roland McGrath et al. (...) Compiled by GNU CC version 6.1.1 20160802.

AceJohnny2 10y ago

> In other words: your system supports calling the time system call via the Linux kernel’s vDSO to avoid the cost of switching to the kernel. But, as soon as your program calls time, it calls localtime immediately after, which invokes a system call anyway. This reminds me of an article by Ted Unangst[1], in which he flattens the various libraries and abstractions to show how xterm (to cite one of many culprits) in one place is effectively doing: if (poll() || poll()) while (poll()) { /* ... */ } In other words, if you don't know what your library/abstraction is doing, you can end up accidentally duplicating its work. Reminds me of some aphorism, "Those who do not learn from history..." ;) [1] http://www.tedunangst.com/flak/post/accidentally-nonblocking http://www.tedunangst.com/flak/post/accidentally-nonblocking discussed https://news.ycombinator.com/item?id=11847529 https://news.ycombinator.com/item?id=11847529

nsxwolf 10y ago
Those who quote George Santayana are condemned to repeat him.
- andrewbinstock 10y ago
  Those who don't know George Santayana are condemned to repeat him. FTFY.
  - koverstreet 10y ago
    You seem to have missed the joke...
    - btown 10y ago
      You seem to have missed the joke... wait a second...
    - andrewbinstock 10y ago
      Totally got it. It's a refinement of the line. As the person below understands.

leovonl 10y ago

This seems to be a simple RTFM issue to me: POSIX specifies that gmtime() uses UTC and localtime() uses current timezone. Using gmtime() would implement the desired behaviour without any need to hardcode environment variables.

mwexler 10y ago
...which fixes all the code you wrote, but of course, you may have legacy binaries that you don't have access to source to change... hence a simple setting of an environment variable, hardcoded though it may be, fixes the situation for all. Though PeterWillis makes a good point akin to yours, and your (plural) point does make sense. (Edit: added mention of comment with additional background on why to avoid hardcoding the variable)

cbsmith 10y ago

This reads to me like a glibc bug. Glibc should just be watching "/etc/localtime" for changes, rather than calling out to hundreds of times a second.

gumby 10y ago
And how does it watch it? With the stat() syscall.
- deleted 10y ago
  [deleted]
- Dylan16807 10y ago
  Polling is not watching.
noselasd 10y ago
There's no usable mechanisms glibc can use for watching /etc/localtime for changes that does not mess up the program if it also decides to use any file watching features.
- cbsmith 10y ago
  At least on key platforms, it's pretty easy to use an event driven model and watch for updates. Hell, if the vDSO handled TZ it'd be no problem.
  - noselasd 10y ago
    I'm not saying it's impossible, but if you actually sit down and attempt to add this to glibc, you will come to the realization that it is not easy at all. If you want to e.g. use the inotify mechanism you have a few key decision points to make * There's no place where glibc can run an event loop to watch for the changes, the application might not have an event loop. * If you need to integrate with an event loop the application runs, it'll work fine as long as the user remembers to hook up the events and deal with the corner cases. You can use this approach without any special support for glibc, and set TZ env. variable yourself when /etc/localtime changes - though at the moment that will only work for single threaded programs (setenv()/putenv() is not thread safe) * If you decide to run the event loop in a separate thread, you force every binary to be multi threaded and have quite a lot of corner cases to handle when fork()'ing. * If you don't run an event loop, you're back to polling for changes, and hardly anything is gained. * If you use the signal driven I/O notification mechanism, you interfer with the application use of signals and I/O notification, and also have a host of fork() corner cases to consider. vDSO is not a magic silver bullet that can solve this, you would at least have to have the kernel manage timezone support, or perhaps better, have the ability to transparently manage arbitarily data, and then expose that through shared memory that vDSO can use. This will not happen anytime soon.

actuator 10y ago

While trying to find the cause of slowness in Rails requests, I was running strace on an unicorn process when I encountered the same thing mentioned in the article. Rails instrumentation code calls current time before and after any instrumentation block. So, when I looked at the trace there were a lot of `stat` calls coming for `/etc/localtime` and as stat is an IO operation, I thought I discovered the cause of slowness(which I attributed to high number of IO ops) but surprisingly when I saw the strace method summary; while the call count was high, the time taken by the calls in total was not significant(<1% if I remember correctly). So I decided to set TZ with the next AMI update 15 months back but forgot about it totally. I guess I should add it to my Trello list this time. Also, I think he should have printed the aggregate summary of just CPU clock time(`-c`) as well as that is usually very low.

caf 10y ago
...while the call count was high, the time taken by the calls in total was not significant(<1% if I remember correctly). Yes, on ordinary filesystems if you run stat() over and over again on the same file then it's just copying from the in-memory inode into your struct stat, there's no IO.

creeble 10y ago

Does anyone have any evidence of this actually having a resource usage impact on any common programs? I see one reference to Apache below, but not whether it actually made a measurable difference.

dpatru 10y ago

It seems to me that this is something the Linux distributions should already be doing.

kelnos 10y ago

I really enjoy when people dig into things like this and report their findings. Having said that, I question the wisdom of "bothering" with this sort of thing. Everything you do that's non-standard or works against a system's default behavior incurs a cost. It's yet another thing you have to replicate when you migrate to a new version, change provisioning systems, etc. And for what benefit? A few hundred syscalls per second? Linux syscalls are fast enough that something of that magnitude shouldn't matter much. Given that /etc/localtime will certainly be in cache with that frequency of access, a stat() should do little work in the kernel to return, so that won't be slow either. It's good that they did some benchmarking to look at the differences, but this feels like a premature optimization to me. I can't imagine that this did anything but make their application a tiny fraction of a percent faster. Was it worth the time to dig into that for this increase? Was it worth the maintenance cost I mention in my first paragraph? I wouldn't think so. I'm really trying not to take a crap on what they did; as I said, it's really cool to dig into these sorts of abstractions and find out where they're inefficient or leak (or just great as a learning exercise; we all depend on a mountain of code that most people don't understand at all). But, when looked at from a holistic systems approach, a grab bag of little "tweaks" like this can become harmful in the long run.

lotyrin 10y ago
You should be able to get maintenance for something like this pretty low. Add a line with a nice comment to the config template for running your apps that adds an extra env var. Any machine that runs any app now has the line... I would definitely like to see a before/after real-world metric on impact here though.
bradfa 10y ago
The embedded Linux system I'm working on right now takes about 17 us per stat() call due to this but time will always be kept internally as UTC, so taking advantage of this is worth considering for me. Since most embedded systems can directly translate the amount of processing needed to achieve the product goal into a real dollar cost of the hardware, any savings, even a small one, is worth investigating. Since the hardware and system are generally well understood, implementing something like this is much more reasonable. But I agree, on a general purpose OS doing general purpose thing, an optimization like what's proposed by the article may not be worth the other tradeoffs.

vesinisa 10y ago

This was a thoroughly fascinating read. Highly recommend reading the previous part in the series as well.

blunte 10y ago

This reminds me of a very similar behavior in Solaris over 20 years ago. Our C application was having odd performance problems on some client systems, and eventually we saw via truss that there were hundreds of fopen() calls every second to get the timezone. Setting the right environment variable solved the problem.

astrostl 10y ago

I do this for "not get annoyed while stracing" reasons, not perf!

jakeogh 10y ago

OpenRC users can: echo 'TZ=:/etc/localtime' > /etc/env.d/00localtime

JensRex 10y ago
For Debian and derivatives: echo 'TZ=:/etc/localtime' >> /etc/environment

jdamato 10y ago

Author of the post here: greetings. If you enjoyed this post, you may also enjoy our deep dive explaining exactly how system calls work on Linux[1]. [1]: https://blog.packagecloud.io/eng/2016/04/05/the-definitive-guide-to-linux-system-calls/ https://blog.packagecloud.io/eng/2016/04/05/the-definitive-g...

rargulati 10y ago

Really interesting - thanks for sharing the findings. I haven't seen it mentioned here, but for those of us using `timedatectl` via systemd, with the default setting of `UTC` are taking advantage[1] of the recommendation in the article. [1] https://github.com/systemd/systemd/blob/master/src/timedate/timedatectl.c#L85 https://github.com/systemd/systemd/blob/master/src/timedate/...

glandium 10y ago

What is missing in this post is: - Why does glibc check /etc/localtime every time localtime is called? Wild guess: so that new values of /etc/localtime are picked at runtime without restarting programs. - Corollary: why does glibc not check /etc/localtime every time localtime is called, when TZ is set to :/etc/localtime? Arguably the reason above should still apply when TZ is set to a file name, shouldn't it?

jdamato 10y ago
Hi, both are answered in the article: First: > What’s going on here is that the first call to localtime in glibc opens and reads the contents of /etc/localtime. All subsequent calls to localtime internally call stat, but they do this to ensure that the timezone file has not changed. and second: read the section titled "Preventing extraneous system calls" for the answer to your second question.
- glandium 10y ago
  Those "answers" are more about how than why.
  - jdamato 10y ago
    Thanks for reading and I'm glad to hear you loved my post!
glandium 10y ago
For the second question: There doesn't seem to be an explicit reason for the difference of treatment. The code that does it has been there since 1996, and hasn't changed since. The only reason given is "Caching happens based on the contents of the environment variable TZ.". https://sourceware.org/git/?p=glibc.git;a=commit;h=68dbb3a69e78e24a778c6602c8cc91d715839d08 https://sourceware.org/git/?p=glibc.git;a=commit;h=68dbb3a69... I'd argue it should cache the same when both old_tz and tz are NULL (but start with an old_tz that is not NULL). I was about to file an upstream bug, but found https://sourceware.org/bugzilla/show_bug.cgi?id=5184 https://sourceware.org/bugzilla/show_bug.cgi?id=5184 and https://sourceware.org/bugzilla/show_bug.cgi?id=5186 https://sourceware.org/bugzilla/show_bug.cgi?id=5186 The latter actually implies the opposite should be happening: files given in TZ should be stat()ed just as much as /etc/localtime.

snowcrshd 10y ago

Brendan Gregg wrote about this a few years ago [1]. My favorite part: > WTF?? Why is ls(1) running stat() on /etc/localtime for every line of output? [1] http://www.brendangregg.com/blog/2014-05-11/strace-wow-much-syscall.html http://www.brendangregg.com/blog/2014-05-11/strace-wow-much-...

acscott 10y ago

This reminds me of setting noatime for disk mounts (http://askubuntu.com/questions/2099/is-it-worth-to-tune-ext4-with-noatime http://askubuntu.com/questions/2099/is-it-worth-to-tune-ext4...) Now I want to know the number of other configs to reduce the number of system calls. This all adds up to being significant the greater the number of hosts in your environment.

brendangregg 10y ago

Please quantify the speedup (I've found this before, but it's never been a significant issue). Eliminating unnecessary work is great, but what are we really talking about here? Use a CPU flamegraph, Ctrl-F and search for stat functions. It'll quantify the total on the bottom right.

brendangregg 10y ago
Oh, and another page that recommends strace without warning about overheads. Dangerous.

mattronson 10y ago

Linux is very fast !!

rtsisyk 10y ago

Overhead of localtime() is well-known, just RTFM. Anyway, this article provides very good explanation.

rtsisyk 10y ago

BTW, packagecloud.io is the great hosting for RPM/DEB packages. We've been using it for the last couple years. GitHub + Travis CI + PackageCloud combination allows us build and publish packages for EVERY git commit in 30+ repositories targeting 15 different Linux distributions [1]. There is no more need to hire a special devops guy for that. [1]: https://github.com/packpack/packpack#packpack https://github.com/packpack/packpack#packpack

falsedan 10y ago

Why did this post start with a tl;dr, then a Summary, and _still_ buried the important takeaway in the ultimate paragraph?

scottlamb 10y ago

There's another easy way to avoid this: use localtime_r instead of localtime. From the glibc source: /* Update internal database according to current TZ setting. POSIX.1 8.3.7.2 says that localtime_r is not required to set tzname. This is a good idea since this allows at least a bit more parallelism. */ tzset_internal (tp == &_tmbuf && use_localtime, 1); mktime also does the tzset call every time, though: time_t mktime (struct tm *tp) { #ifdef _LIBC /* POSIX.1 8.1.1 requires that whenever mktime() is called, the time zone names contained in the external variable 'tzname' shall be set as if the tzset() function had been called. */ __tzset (); #endif and I don't see any way around that other than setting TZ=: or some such.

barrystaes 10y ago

Im not an expert but the first thing that comes to mind is that 1) TFA does not quantify the performance gain in time 2) I wonder if environment variables like TZ are a security risk/vector in that these might facilitate attackers to stealthy skew/screw time within current user process... no root required.