4 ms·

> I happened to run strace -tt against my solution (which provides microsecond-accurate timing information for syscalls) Weirdly I always found the timing resu

by josefx 5y ago

> I happened to run strace -tt against my solution (which provides microsecond-accurate timing information for syscalls) Weirdly I always found the timing results of strace at under one millisecond generally unreliable, it just seemed to add too much overhead itself. Also making system calls from your own code varies between prohibited and badly supported on most OSes. Some see calls that didn't pass through the system libc as a security issue and will intercept them, while Linux may just silently corrupt your process memory if you try something fancy as the Go team had to find out.

badsectoracula 5y ago
> Also making system calls from your own code varies between prohibited and badly supported on most OSes. This is not the case on Linux though, the official API of the kernel is its system calls, not any specific C library.
- dwattttt 5y ago
  While this is true, the go vs vDSO stack clash corruption highlights that a C ABI that doesn't specify stack usage is under defined. A platforms libc can be written such that it's aware of these limitations.
  - badsectoracula 5y ago
    This can be said about any wrapper to an underdefined interface, it doesn't make that interface any less official.
mgaunard 5y ago
That's the difference between precision and accuracy. Having more precision doesn't magically make the data accurate.
commandlinefan 5y ago
> making system calls from your own code varies between prohibited and badly supported Which is probably at least part of the reason that libc itself ends up being slower than a specific, targeted solution.
mananaysiempre 5y ago
Umm, what’s the story with Go on Linux? My understanding is that Linux explicitly supports making syscalls from your own code, it’s just that on platforms where the syscall story is a giant mess (32-bit x86) the prescription is to either use the slow historical interface (INT 80h) or to jump to the vDSO (which will do SYSENTER or SYSCALL or whatever—SYSENTER in particular has the lovely property of not saving the return address, so a common stub is pretty much architecturally required[1]). If I’m guessing correctly about what you’re referring to, the Go people did something in between, got broken in a Linux kernel release, complained at the kernel people, the kernel got a patch to unbreak them. The story on MacOS or OpenBSD, where the kernel developers will cheerfully tell you to take a hike if you make a syscall yourself, seems much worse to me. (And yes, I’d say there is a meaningful difference between a couple of instructions in the vDSO and Glibc’s endless wrappers.) [1]: https://lore.kernel.org/lkml/Pine.LNX.4.44.0212172225410.1368-100000@home.transmeta.com/ https://lore.kernel.org/lkml/Pine.LNX.4.44.0212172225410.136... ETA: Wait, no, I was thinking about the Android breakage[2]. What’s the Go story then? [2]: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=30bfa7b3488bfb1bb75c9f50a5fcac1832970c60 https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/...
- aw1621107 5y ago
  This gets rather over my head, but my best guess is this bug having to do with the Go runtime underestimating how much stack space vDSO code may need (blog post at [0], fix at [1]). [0]: https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ [1]: https://github.com/golang/go/commit/a158382b1c9c0b95a7d41865a405736be6bc585f https://github.com/golang/go/commit/a158382b1c9c0b95a7d41865...
  - mananaysiempre 5y ago
    > I built 32 kernels, one for each bit of the SHA-1 prefix, which only took 29 minutes. Oh, that is evil, thank you. I even encountered a link to this article a couple of months ago[1], but wasn’t hooked enough to go and read it. Though the conclusion sounds somewhat dubious to me: the kernel doesn’t document or control stack usage limits for the vDSO, they happen to blow way up on a system built with obscure (if arguably well-motivated) compiler options, a language runtime that tries to minimize stack usage crashes as a result, and somehow the fault is with the runtime in question for not going via libc (which happens to virtually always run with a large stack and a guard page, thus turning this insidious concurrent memory corruption bug into a mere extremely unlikely crash)? More like we’re collectively garbage at accounting for our stack usage. To be fair to the kernel developers, I would also never guess, looking at this implementation of clock_gettime() [2], that you could compile it in such a way that it ends up requiring 4K of stack space on pain of memory corruption in concurrent programs only (it originating in the kernel source tree has little to do with the bug, it’s just weirdly-compiled userspace C code executing on a small unguarded stack). [1] in <https://utcc.utoronto.ca/~cks/space/blog/unix/UnixAPIAndCRuntime https://utcc.utoronto.ca/~cks/space/blog/unix/UnixAPIAndCRun...> via <https://utcc.utoronto.ca/~cks/space/blog/programming/CStackSizeInvisible https://utcc.utoronto.ca/~cks/space/blog/programming/CStackS...>, <https://utcc.utoronto.ca/~cks/space/blog/unix/StackSizeLimitWhen https://utcc.utoronto.ca/~cks/space/blog/unix/StackSizeLimit...>, and <https://utcc.utoronto.ca/~cks/space/blog/tech/StackSizeLimitWhy https://utcc.utoronto.ca/~cks/space/blog/tech/StackSizeLimit...>. [2] https://elixir.bootlin.com/linux/latest/C/ident/__cvdso_clock_gettime https://elixir.bootlin.com/linux/latest/C/ident/__cvdso_cloc...
scottlamb 5y ago
josefx> Also making system calls from your own code varies between prohibited and badly supported on most OSes. Some see calls that didn't pass through the system libc as a security issue and will intercept them, while Linux may just silently corrupt your process memory if you try something fancy as the Go team had to find out. On Linux, making syscalls directly is fine. Good point about other platforms, but many people only care about Linux, for better or worse. And the author's last paragraph (quoted below) suggests using an alternate/static-linking-friendly libc, not making direct syscalls yourself. Presumably on platforms where those alternate libcs aren't available, you continue using the standard libc. ryanhileman> If you're running into process startup time issues in a real world scenario and ever actually need to do this, it might be worth your time to profile and try one of the alternative libc implementations (like musl libc or diet libc). IMHO, the Go vDSO problem [1] wasn't due to making direct syscalls but basically calling a userspace library without meeting its assumptions. I'd describe Linux's vDSO as a userspace library for making certain syscalls with less overhead. (If you don't care about the overhead, you can call them as you'd call any other syscall instead.) It assumed the standard ABI, in which there's a guard page as well as typically a generous amount of stack to begin with. Golang called into it from a thread that used Go's non-standard ABI with less stack space available (Go's own functions check the stack size on entry and copy the stack if necessary) and no guard page. On some Linux builds (with -fstack-check, apparently used by Gentoo Hardened ), it actually used enough stack to overflow. Without a guard page, that caused memory corruption. [1] https://github.com/golang/go/issues/20427 https://github.com/golang/go/issues/20427

by josefx 5y ago

badsectoracula 5y ago

> Also making system calls from your own code varies between prohibited and badly supported on most OSes. This is not the case on Linux though, the official API of the kernel is its system calls, not any specific C library.

dwattttt 5y ago
While this is true, the go vs vDSO stack clash corruption highlights that a C ABI that doesn't specify stack usage is under defined. A platforms libc can be written such that it's aware of these limitations.
- badsectoracula 5y ago
  This can be said about any wrapper to an underdefined interface, it doesn't make that interface any less official.

mgaunard 5y ago

That's the difference between precision and accuracy. Having more precision doesn't magically make the data accurate.

commandlinefan 5y ago

> making system calls from your own code varies between prohibited and badly supported Which is probably at least part of the reason that libc itself ends up being slower than a specific, targeted solution.

mananaysiempre 5y ago

Umm, what’s the story with Go on Linux? My understanding is that Linux explicitly supports making syscalls from your own code, it’s just that on platforms where the syscall story is a giant mess (32-bit x86) the prescription is to either use the slow historical interface (INT 80h) or to jump to the vDSO (which will do SYSENTER or SYSCALL or whatever—SYSENTER in particular has the lovely property of not saving the return address, so a common stub is pretty much architecturally required[1]). If I’m guessing correctly about what you’re referring to, the Go people did something in between, got broken in a Linux kernel release, complained at the kernel people, the kernel got a patch to unbreak them. The story on MacOS or OpenBSD, where the kernel developers will cheerfully tell you to take a hike if you make a syscall yourself, seems much worse to me. (And yes, I’d say there is a meaningful difference between a couple of instructions in the vDSO and Glibc’s endless wrappers.) [1]: https://lore.kernel.org/lkml/Pine.LNX.4.44.0212172225410.1368-100000@home.transmeta.com/ https://lore.kernel.org/lkml/Pine.LNX.4.44.0212172225410.136... ETA: Wait, no, I was thinking about the Android breakage[2]. What’s the Go story then? [2]: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=30bfa7b3488bfb1bb75c9f50a5fcac1832970c60 https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/...

aw1621107 5y ago
This gets rather over my head, but my best guess is this bug having to do with the Go runtime underestimating how much stack space vDSO code may need (blog post at [0], fix at [1]). [0]: https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ [1]: https://github.com/golang/go/commit/a158382b1c9c0b95a7d41865a405736be6bc585f https://github.com/golang/go/commit/a158382b1c9c0b95a7d41865...
- mananaysiempre 5y ago
  > I built 32 kernels, one for each bit of the SHA-1 prefix, which only took 29 minutes. Oh, that is evil, thank you. I even encountered a link to this article a couple of months ago[1], but wasn’t hooked enough to go and read it. Though the conclusion sounds somewhat dubious to me: the kernel doesn’t document or control stack usage limits for the vDSO, they happen to blow way up on a system built with obscure (if arguably well-motivated) compiler options, a language runtime that tries to minimize stack usage crashes as a result, and somehow the fault is with the runtime in question for not going via libc (which happens to virtually always run with a large stack and a guard page, thus turning this insidious concurrent memory corruption bug into a mere extremely unlikely crash)? More like we’re collectively garbage at accounting for our stack usage. To be fair to the kernel developers, I would also never guess, looking at this implementation of clock_gettime() [2], that you could compile it in such a way that it ends up requiring 4K of stack space on pain of memory corruption in concurrent programs only (it originating in the kernel source tree has little to do with the bug, it’s just weirdly-compiled userspace C code executing on a small unguarded stack). [1] in <https://utcc.utoronto.ca/~cks/space/blog/unix/UnixAPIAndCRuntime https://utcc.utoronto.ca/~cks/space/blog/unix/UnixAPIAndCRun...> via <https://utcc.utoronto.ca/~cks/space/blog/programming/CStackSizeInvisible https://utcc.utoronto.ca/~cks/space/blog/programming/CStackS...>, <https://utcc.utoronto.ca/~cks/space/blog/unix/StackSizeLimitWhen https://utcc.utoronto.ca/~cks/space/blog/unix/StackSizeLimit...>, and <https://utcc.utoronto.ca/~cks/space/blog/tech/StackSizeLimitWhy https://utcc.utoronto.ca/~cks/space/blog/tech/StackSizeLimit...>. [2] https://elixir.bootlin.com/linux/latest/C/ident/__cvdso_clock_gettime https://elixir.bootlin.com/linux/latest/C/ident/__cvdso_cloc...

scottlamb 5y ago

josefx> Also making system calls from your own code varies between prohibited and badly supported on most OSes. Some see calls that didn't pass through the system libc as a security issue and will intercept them, while Linux may just silently corrupt your process memory if you try something fancy as the Go team had to find out. On Linux, making syscalls directly is fine. Good point about other platforms, but many people only care about Linux, for better or worse. And the author's last paragraph (quoted below) suggests using an alternate/static-linking-friendly libc, not making direct syscalls yourself. Presumably on platforms where those alternate libcs aren't available, you continue using the standard libc. ryanhileman> If you're running into process startup time issues in a real world scenario and ever actually need to do this, it might be worth your time to profile and try one of the alternative libc implementations (like musl libc or diet libc). IMHO, the Go vDSO problem [1] wasn't due to making direct syscalls but basically calling a userspace library without meeting its assumptions. I'd describe Linux's vDSO as a userspace library for making certain syscalls with less overhead. (If you don't care about the overhead, you can call them as you'd call any other syscall instead.) It assumed the standard ABI, in which there's a guard page as well as typically a generous amount of stack to begin with. Golang called into it from a thread that used Go's non-standard ABI with less stack space available (Go's own functions check the stack size on entry and copy the stack if necessary) and no guard page. On some Linux builds (with -fstack-check, apparently used by Gentoo Hardened ), it actually used enough stack to overflow. Without a guard page, that caused memory corruption. [1] https://github.com/golang/go/issues/20427 https://github.com/golang/go/issues/20427