8 ms·

System calls in Linux are really fast. So saving "thousands" of system calls when /etc/localtime is in cache doesn't actually save that much actual CPU time.

by tytso 10y ago

System calls in Linux are really fast. So saving "thousands" of system calls when /etc/localtime is in cache doesn't actually save that much actual CPU time. I ran an experiment where I timed the runtime of the sample program provided in the OP, except I changed the number of calls to localtime() from ten times to a million. I then timed the difference with and without export TZ=:/etc/localhost. The net savings was .6 seconds. So for a single call to localtime(3), the net savings is 0.6 microseconds. That's non-zero, but it's likely in the noise compared to everything else that your program might be doing.

dsl 10y ago
On your base system, yes. Lots of things can hook random syscalls, or environments might have syscall monitoring. One example is the folks over at slack record every syscall for security auditing. https://slack.engineering/syscall-auditing-at-scale-e6a3ca8ac1b8#.5lj2m5nv5 https://slack.engineering/syscall-auditing-at-scale-e6a3ca8a...
- tytso 10y ago
  Slack uses the Linux audit subsystem which is also certainly faster than you think it is. Consider how many system calls your typical application is issuing --- especially ones that are likely to be calling localtime() all the time, such as a web server. If system call auditing had that high of an overhead, everything would be horrifically slow --- but it isn't, because Linux audit sends its records out asynchronously and in batches.
  - nwmcsween 10y ago
    https://www.redhat.com/archives/linux-audit/2015-January/msg00171.html https://www.redhat.com/archives/linux-audit/2015-January/msg... of course this is RHEL 2.6.32 and it's open/close but 200000 sc/s vs 3000 sc/s shows it has some overhead. Maybe someone can rerun that test code on git and see what the overhead is.
peterwwillis 10y ago
Yeah, this is a perfect example of micro-optimization being unnecessary. Not only will you not see performance issues from this in the real world, it might cause problems down the road, because since it isn't set by default this way, some apps may not expect it and behave erroneously. But it's neat information to have in the back of your head.
- ishtu 10y ago
  Unnecessary? I had a really bad experience with ancient skype version on modern Ubuntu desktop, and the fix for this was to set TZ environment variable to speedup first login/history fetch. Skype process was spending so much time doing useless work it was noticeable.
- __jal 10y ago
  That's just not possible to authoritatively state. The best you can do is "this shouldn't normally cause a noticeable impact on most systems". As just one example, what you're stat()ing over NFS with a busy, flaky and/or distant server? A bit of thought and you'll come up with a bunch of other times it suddenly starts to matter.
rpcope1 10y ago
This might be true for your system and libc, where the system calls make use of things like vDSO for gettimeofday go fast, but in general this isn't guaranteed at all. Even on x64, for certain libc implementations, like musl, if I recall correctly, syscalls are made the old fashioned way by trapping 0x80, which would mean you would see a much bigger effect by reducing the number of syscalls.
- tytso 10y ago
  There is no vDSO for calls to stat(2). The claim in the article was that by setting the TZ environment variable to ":/etc/localtime", one could save "thousands" of stat system calls. Even for old-fashioned system calls where you use trap 0x80, Linux is still amazingly fast. This can actually be a problem, since there are applications like git which assume stat is fast, and so it aggressively stat's all of the working files in the repository to check the mod times to see if anything has changed. That's fine on Linux, but it's a disaster on Windows, where the stat system call is dog-slow. Still, I'd call that a Windows bug, not a git bug.
  - kingosticks 10y ago
    It's also a disaster on NFS.
  - codedokode 10y ago
    Does Windows has stat() call? It is probably a function from some POSIX emulation layer and maybe that is why it is not fast.
- amluto 10y ago
  Not quite. On x86_32, for complicated and ultimately ridiculous but nevertheless valid reasons, lots of syscalls on musl use int $0x80. I have a patch to make this fixable but Linus shot it down. Maybe I should try again. On x86_64, syscalls only use SYSCALL. It's very fast if audit and such are off and reasonably fast otherwise. (I extensively rewrote this code recently. Older teardowns of the syscall path are dated.)
- nwmcsween 10y ago
raverbashing 10y ago
System calls in x86 are fast. Other archs behave differently. And the syscall time is not the only thing that matters, but potentially yielding execution
- philsnow 10y ago
  I thought they were fast because x86 has multiple register files, enough for kernel space and user space to have their own, so that entry/exit to system calls doesn't require flushing registers to L1 (in the common case). If that's true, then one test where you have a single process spinning into and out of a single syscall will have very different performance characteristics than a test where you have more processes than processor cores, because context switches flush the TLB. Somebody who knows actual things about x86 and so forth please tell me if I'm spouting 90s-era comp sci architecture textbook stuff that no longer applies.
  - jdamato 10y ago
    Check out the post linked from the article: https://blog.packagecloud.io/eng/2016/04/05/the-definitive-guide-to-linux-system-calls/ https://blog.packagecloud.io/eng/2016/04/05/the-definitive-g... to learn more about how system calls work on x86 Linux.
  - amluto 10y ago
    They're fast because x86 has a decently fast privilege change mechanism for system calls and Linux works fairly hard to avoid doing unnecessary work to handle them. In the simplest case, registers are saved, a function is called, regs are restored, and the kernel switches back to user mode. The asm code is fairly straightforward in Linux these days. I'm proud of it. :)
cbsmith 10y ago
> System calls in Linux are really fast. So saving "thousands" of system calls when /etc/localtime is in cache doesn't actually save that much actual CPU time. "fast" is a relative term, and is somewhat orthogonal to "efficient". There's a reason why certain functions use a vDSO. If you're just going to use a syscall anyway, there's kind of no point.
- deathanatos 10y ago
  You're assuming that all cases where the vDSO call is made gets paired with a real syscall; that's simply not the case. There are plenty of calls in a server that won't need localtime (basically, anything that just needs the current time in UTC: best-practice code should not be looking at the machine's TZ setting¹). Look at the examples the article's author offers: > formatting dates and times This shouldn't require a call to localtime; more explanation on the part of the article is required here. Breaking a seconds-since-epoch out into year/mo/day/etc. is "simple" math, and shouldn't require a filesystem access. Something else is amiss here. > for everything from log messages You're about to hit disk; a cache'd stat() isn't going to matter. > to SQL queries. You're about to hit the network; a cache'd stat() isn't going to matter. (Now, I'm not saying you shouldn't set TZ; if it saves some syscalls, fine, and it might be the only sane value anyways.) ¹one of my old teams had an informal rule that any invocation of datetime.datetime.now() was a bug.
  - lfowles 10y ago
    >Breaking a seconds-since-epoch out into year/mo/day/etc. is "simple" math, and shouldn't require a filesystem access. To do it simply yes, but not correctly. See the "Falsehoods programmers believe about time" series. http://infiniteundo.com/post/25326999628/falsehoods-programmers-believe-about-time http://infiniteundo.com/post/25326999628/falsehoods-programm... http://infiniteundo.com/post/25509354022/more-falsehoods-programmers-believe-about-time http://infiniteundo.com/post/25509354022/more-falsehoods-pro...
    - deathanatos 10y ago
      > To do it simply yes, but not correctly. No, it do it correctly doesn't require filesystem access either. I've read both articles in the past: neither refutes the point I made above. If I were incorrect, linking to an article that enumerates tens of things (some of them arguably incorrect) isn't useful. If you're trying to imply that you need to take timezones into account, yes, you do. Yes, typically those definitions are stored on disk, but the context here is requiring filesystem access each and every time; most libraries (including glibc) will load the timezone definitions once, and keep them in memory. Thus, you can break a seconds-since-epoch out into year/mo/day/etc. with "simple" math, and it doesn't require a filesystem access. (Beyond the amortized one time load, but given the point and purposes of the article, I'm not considering that.)
vesinisa 10y ago
> TZ=:/etc/localhost Hope this is just a typo in your comment, not the actual test ;)
- deathanatos 10y ago
  This isn't a typo, but is part of the syntax used by the TZ variable. (The same format appears in the article itself.) See `man timezone` on a Linux system[1]. Specifically, see the passage that I've quoted below. Note that this is the third of three different formats that the man page describes that you can use in TZ: > The second format specifies that the timezone information should be read from a file: :[filespec] > *If the file specification filespec is omitted, or its value cannot be interpreted, then Coordinated Universal Time (UTC) is used. If filespec is given, it specifies another tzfile(5)-format file to read the timezone information from. If filespec does not begin with a '/', the file specification is relative to the system timezone directory. If the colon is omitted each of the above TZ formats will be tried. [1]: https://linux.die.net/man/3/timezone https://linux.die.net/man/3/timezone
  - vesinisa 10y ago
    Sure, but at least on none of my Linux systems there is no such file /etc/localhost. I think the parent was referring to /etc/localtime. Not sure what is the behaviour if non-existent file is specified - perhaps the "value cannot be interpreted" case applies, but it's not pefectly clear, since it could be argued that the value is valid, just refers to a non-existent file.
    - deathanatos 10y ago
      Ah, correct you are! :-) I had missed that myself, and the : syntax is so rarely seen I naturally assumed that was what was intended.
mfukar 10y ago
System calls in Linux are not faster than not doing them.
andrelaszlo 10y ago
I did the same, but with 10M iterations: $ time ./tz ./tz 2,24s user 6,28s system 98% cpu 8,612 total $ export TZ=:/etc/localtime $ time ./tz ./tz 1,35s user 0,00s system 98% cpu 1,364 total So 0.7 microseconds on my machine.
kingosticks 10y ago
I did the same experiment on a Raspberry Pi 2. The net saving was 5.803 seconds, so 5.803 microseconds per call. Obviously if you care about performance then you wouldn't be running your program on a Raspberry Pi in the first place. But for everything else there's this free speed up.
- mfukar 10y ago
  I build a bunch of home automation stuff (as a hobby) using Pis and other microcontrollers. Performance in those things translates almost directly to power savings, and is very desirable. OTOH, I've never encountered an issue like this on those systems.. (yet)

by tytso 10y ago

dsl 10y ago

On your base system, yes. Lots of things can hook random syscalls, or environments might have syscall monitoring. One example is the folks over at slack record every syscall for security auditing. https://slack.engineering/syscall-auditing-at-scale-e6a3ca8ac1b8#.5lj2m5nv5 https://slack.engineering/syscall-auditing-at-scale-e6a3ca8a...

tytso 10y ago
Slack uses the Linux audit subsystem which is also certainly faster than you think it is. Consider how many system calls your typical application is issuing --- especially ones that are likely to be calling localtime() all the time, such as a web server. If system call auditing had that high of an overhead, everything would be horrifically slow --- but it isn't, because Linux audit sends its records out asynchronously and in batches.
- nwmcsween 10y ago
  https://www.redhat.com/archives/linux-audit/2015-January/msg00171.html https://www.redhat.com/archives/linux-audit/2015-January/msg... of course this is RHEL 2.6.32 and it's open/close but 200000 sc/s vs 3000 sc/s shows it has some overhead. Maybe someone can rerun that test code on git and see what the overhead is.

peterwwillis 10y ago

Yeah, this is a perfect example of micro-optimization being unnecessary. Not only will you not see performance issues from this in the real world, it might cause problems down the road, because since it isn't set by default this way, some apps may not expect it and behave erroneously. But it's neat information to have in the back of your head.

ishtu 10y ago
Unnecessary? I had a really bad experience with ancient skype version on modern Ubuntu desktop, and the fix for this was to set TZ environment variable to speedup first login/history fetch. Skype process was spending so much time doing useless work it was noticeable.
__jal 10y ago
That's just not possible to authoritatively state. The best you can do is "this shouldn't normally cause a noticeable impact on most systems". As just one example, what you're stat()ing over NFS with a busy, flaky and/or distant server? A bit of thought and you'll come up with a bunch of other times it suddenly starts to matter.

rpcope1 10y ago

This might be true for your system and libc, where the system calls make use of things like vDSO for gettimeofday go fast, but in general this isn't guaranteed at all. Even on x64, for certain libc implementations, like musl, if I recall correctly, syscalls are made the old fashioned way by trapping 0x80, which would mean you would see a much bigger effect by reducing the number of syscalls.

tytso 10y ago
There is no vDSO for calls to stat(2). The claim in the article was that by setting the TZ environment variable to ":/etc/localtime", one could save "thousands" of stat system calls. Even for old-fashioned system calls where you use trap 0x80, Linux is still amazingly fast. This can actually be a problem, since there are applications like git which assume stat is fast, and so it aggressively stat's all of the working files in the repository to check the mod times to see if anything has changed. That's fine on Linux, but it's a disaster on Windows, where the stat system call is dog-slow. Still, I'd call that a Windows bug, not a git bug.
- kingosticks 10y ago
  It's also a disaster on NFS.
- codedokode 10y ago
  Does Windows has stat() call? It is probably a function from some POSIX emulation layer and maybe that is why it is not fast.
amluto 10y ago
Not quite. On x86_32, for complicated and ultimately ridiculous but nevertheless valid reasons, lots of syscalls on musl use int $0x80. I have a patch to make this fixable but Linus shot it down. Maybe I should try again. On x86_64, syscalls only use SYSCALL. It's very fast if audit and such are off and reasonably fast otherwise. (I extensively rewrote this code recently. Older teardowns of the syscall path are dated.)
nwmcsween 10y ago

raverbashing 10y ago

System calls in x86 are fast. Other archs behave differently. And the syscall time is not the only thing that matters, but potentially yielding execution

philsnow 10y ago
I thought they were fast because x86 has multiple register files, enough for kernel space and user space to have their own, so that entry/exit to system calls doesn't require flushing registers to L1 (in the common case). If that's true, then one test where you have a single process spinning into and out of a single syscall will have very different performance characteristics than a test where you have more processes than processor cores, because context switches flush the TLB. Somebody who knows actual things about x86 and so forth please tell me if I'm spouting 90s-era comp sci architecture textbook stuff that no longer applies.
- jdamato 10y ago
  Check out the post linked from the article: https://blog.packagecloud.io/eng/2016/04/05/the-definitive-guide-to-linux-system-calls/ https://blog.packagecloud.io/eng/2016/04/05/the-definitive-g... to learn more about how system calls work on x86 Linux.
- amluto 10y ago
  They're fast because x86 has a decently fast privilege change mechanism for system calls and Linux works fairly hard to avoid doing unnecessary work to handle them. In the simplest case, registers are saved, a function is called, regs are restored, and the kernel switches back to user mode. The asm code is fairly straightforward in Linux these days. I'm proud of it. :)

cbsmith 10y ago

> System calls in Linux are really fast. So saving "thousands" of system calls when /etc/localtime is in cache doesn't actually save that much actual CPU time. "fast" is a relative term, and is somewhat orthogonal to "efficient". There's a reason why certain functions use a vDSO. If you're just going to use a syscall anyway, there's kind of no point.

deathanatos 10y ago
You're assuming that all cases where the vDSO call is made gets paired with a real syscall; that's simply not the case. There are plenty of calls in a server that won't need localtime (basically, anything that just needs the current time in UTC: best-practice code should not be looking at the machine's TZ setting¹). Look at the examples the article's author offers: > formatting dates and times This shouldn't require a call to localtime; more explanation on the part of the article is required here. Breaking a seconds-since-epoch out into year/mo/day/etc. is "simple" math, and shouldn't require a filesystem access. Something else is amiss here. > for everything from log messages You're about to hit disk; a cache'd stat() isn't going to matter. > to SQL queries. You're about to hit the network; a cache'd stat() isn't going to matter. (Now, I'm not saying you shouldn't set TZ; if it saves some syscalls, fine, and it might be the only sane value anyways.) ¹one of my old teams had an informal rule that any invocation of datetime.datetime.now() was a bug.
- lfowles 10y ago
  >Breaking a seconds-since-epoch out into year/mo/day/etc. is "simple" math, and shouldn't require a filesystem access. To do it simply yes, but not correctly. See the "Falsehoods programmers believe about time" series. http://infiniteundo.com/post/25326999628/falsehoods-programmers-believe-about-time http://infiniteundo.com/post/25326999628/falsehoods-programm... http://infiniteundo.com/post/25509354022/more-falsehoods-programmers-believe-about-time http://infiniteundo.com/post/25509354022/more-falsehoods-pro...
  - deathanatos 10y ago
    > To do it simply yes, but not correctly. No, it do it correctly doesn't require filesystem access either. I've read both articles in the past: neither refutes the point I made above. If I were incorrect, linking to an article that enumerates tens of things (some of them arguably incorrect) isn't useful. If you're trying to imply that you need to take timezones into account, yes, you do. Yes, typically those definitions are stored on disk, but the context here is requiring filesystem access each and every time; most libraries (including glibc) will load the timezone definitions once, and keep them in memory. Thus, you can break a seconds-since-epoch out into year/mo/day/etc. with "simple" math, and it doesn't require a filesystem access. (Beyond the amortized one time load, but given the point and purposes of the article, I'm not considering that.)

vesinisa 10y ago

> TZ=:/etc/localhost Hope this is just a typo in your comment, not the actual test ;)

deathanatos 10y ago
This isn't a typo, but is part of the syntax used by the TZ variable. (The same format appears in the article itself.) See `man timezone` on a Linux system[1]. Specifically, see the passage that I've quoted below. Note that this is the third of three different formats that the man page describes that you can use in TZ: > The second format specifies that the timezone information should be read from a file: :[filespec] > *If the file specification filespec is omitted, or its value cannot be interpreted, then Coordinated Universal Time (UTC) is used. If filespec is given, it specifies another tzfile(5)-format file to read the timezone information from. If filespec does not begin with a '/', the file specification is relative to the system timezone directory. If the colon is omitted each of the above TZ formats will be tried. [1]: https://linux.die.net/man/3/timezone https://linux.die.net/man/3/timezone
- vesinisa 10y ago
  Sure, but at least on none of my Linux systems there is no such file /etc/localhost. I think the parent was referring to /etc/localtime. Not sure what is the behaviour if non-existent file is specified - perhaps the "value cannot be interpreted" case applies, but it's not pefectly clear, since it could be argued that the value is valid, just refers to a non-existent file.
  - deathanatos 10y ago
    Ah, correct you are! :-) I had missed that myself, and the : syntax is so rarely seen I naturally assumed that was what was intended.

mfukar 10y ago

System calls in Linux are not faster than not doing them.

andrelaszlo 10y ago

I did the same, but with 10M iterations: $ time ./tz ./tz 2,24s user 6,28s system 98% cpu 8,612 total $ export TZ=:/etc/localtime $ time ./tz ./tz 1,35s user 0,00s system 98% cpu 1,364 total So 0.7 microseconds on my machine.

kingosticks 10y ago

I did the same experiment on a Raspberry Pi 2. The net saving was 5.803 seconds, so 5.803 microseconds per call. Obviously if you care about performance then you wouldn't be running your program on a Raspberry Pi in the first place. But for everything else there's this free speed up.

mfukar 10y ago
I build a bunch of home automation stuff (as a hobby) using Pis and other microcontrollers. Performance in those things translates almost directly to power savings, and is very desirable. OTOH, I've never encountered an issue like this on those systems.. (yet)