17 ms·

Go 1.16 will make system calls through Libc on OpenBSD

298 points by lladnar 6y ago

tptacek 6y ago
It was very relevant to me a couple days ago, so I think it's not the case that Go programs that look up names link by default dynamically to libc, just for whatever that's worth to anyone. (Our company, Fly.io, runs container images for customers on Firecracker microVMs around the world, and I had to build a DNS-dependent service, in Go, that runs directly from our (Rust) init and can't assume a libc exists).
- CameronNemo 6y ago
  Is your Rust init private? What does it do?
  - tptacek 6y ago
    Mostly just init stuff. We're transforming Docker containers (mostly) into standalone VMs, so it's doing all the scut work of taking a completely stripped-bare booted kernel and getting it to a state where you can run an arbitrary Linux program on it. I wouldn't want to take the thread off on a huge tangent, it's just funny that this was just recently super relevant to me (it would have been problematic if DNS-dependent Go programs depended on libc, because right now I can't assume there's a libc binary to be dynamically linked to). Bringing it back to Go and its (sometimes libc-dependent) DNS libraries: it is very annoying how fiddly it is to get a Go program to use an alternative DNS server.
    - JoshTriplett 6y ago
      I'm currently in the process, right now, of writing a minimal init in Rust to do exactly the same thing. Is yours something you'd consider sharing? The sum total of what I want to do: bring up loopback, bring up the one and only Ethernet interface, set up its IP and basic routing, run another program, and do some basic log reporting.
      - tptacek 6y ago
        I hack on our init, but I didn't write it. So it's not my place to share it. And there's some us-specific stuff in it that wouldn't be super helpful to you. But you could definitely ask Jerome on our team; he's a super helpful guy, even if he's super quiet here (he's like the anti-me). One way or the other I'm sure we can help you get where you're going!
        JoshTriplett 6y ago
        I reached out to Jerome. Thanks!
        tptacek 6y ago
        Feel free to follow up with Kurt, me or Michael if you don't get a quick response; Jerome is in Montreal and he may be buried under a mountain of snow and brown gravy.
    - alaties 6y ago
      Had a similar problem a couple years ago where I needed to use alternative DNS libraries to troubleshoot issues in a company's infrastructure. Golang's rules for what implementation to use are found here: https://golang.org/pkg/net/#hdr-Name_Resolution https://golang.org/pkg/net/#hdr-Name_Resolution A really solid alternative DNS client implementation can be found here: https://github.com/miekg/dns https://github.com/miekg/dns. Real easy to read and vet compared to a few other libraries I ran into when working on this problem.
    - vidarh 6y ago
      If you can can control which DNS servers it talks to, implementing the DNS spec directly is a pretty trivial exercise to get full control. (If you can't, the main complexity is dealing with implementation quirks)
- X-Istence 6y ago
  The downside to Go shipping it's own implementation of DNS resolution is that on systems that support far more rich set of DNS options (such as split DNS based upon hostnames in macOS) is that it doesn't work if the binary is built with a pure Go implementation. That leaves users annoyed, because sending all DNS into a VPN tunnel is not always an option.
  - yencabulator 6y ago
    If cgo is enabled (the usual case), Go uses libc for DNS if the configuration looks exotic enough that the pure-Go code wouldn't give the same results. https://golang.org/pkg/net/#hdr-Name_Resolution https://golang.org/pkg/net/#hdr-Name_Resolution
    - ArchOversight 6y ago
      This is a long standing issue in various open source projects that use Go, where they want to ship static binaries. Most recently, this happened with Concourse and it's Fly binary: https://github.com/concourse/concourse/issues/3691 https://github.com/concourse/concourse/issues/3691 The developers don't want to use CGO, or when they do use CGO they disable the net part... and now stuff doesn't work. Projects have to specifically build with cgo enabled on macOS, or else it fails.
saagarjha 6y ago
Guess they finally saw sense and realized that the when platforms say the syscall ABI is unstable, it really is unstable…
- chungy 6y ago
  OpenBSD has never really cared about backwards compatibility with binaries. It often works, but then suddenly they break. The OpenBSD developers really prefer fixing bugs instead of working around them.
  - segfaultbuserr 6y ago
    It's also the advantage you have when you officially maintain a complete operating system, not just a kernel.
    - anthk 6y ago
      As long as you can (cross) compile go source easily, I don't see it as a problem.
- jart 6y ago
  There's no good reason for a Unix system to have an unstable system call ABI. I'm sorry but companies like Apple (who broke every Go binary a few years ago) don't get to copy syscall definitions from Bell System V and then declare it their own internal API. Usually the only time the SYSCALL ABI breaks, it's because kernel authors intentionally chose to do it for no apparent reason. For example, OpenBSD at some point changed how mmap() was defined so that it takes seven arguments: void *sys_mmap(void *addr, size_t len, int prot, int flags, int fd, long pad, off_t pos); The sixth argument doesn't do anything. It just breaks binary compatibility. It's also noncompliant with the System V ABI specification, which says system calls have six arguments max. I work on a project called Cosmopolitan Libc which lets you create static binaries that just work on Linux + Mac + Windows + FreeBSD + OpenBSD. It was only possible to do this because Unix systems generally agree on definitions. I worked really hard to support OpenBSD since I believe in the project. I just hope they keep the ABI stable going forward. For example, I'm really happy that this restriction only applies to dynamic binaries. It seems perfectly reasonable that they'd want to make the assumption that if a program chooses to link OpenBSD's Libc that it intends to use it. That's fine just so long as we continue having the ability to build static binaries with an alternative cross-platform Libc like Cosmopolitan. Speaking of which, I think I might actually implement some of OpenBSD's ideas in Cosmopolitan. I could probably track down all the functions that need raw SYSCALL and use __section__ so they're all linked to the same part of the binary and then call the msyscall() function to limit it just to that page. That way as a guest libc author I'm upholding the spirit of the intent. When in Rome do as the Romans.
bithavoc 6y ago
Same for macOS and iOS since Go 1.11, it all goes through libSystem now.
tedunangst 6y ago
The driving force for this was the syscall origin check, but syscall ABIs change too. There's a speculative execution bug in ARM CPUs which requires barriers. This also requires patching. https://marc.info/?l=openbsd-ports-cvs&m=158083696719245&w=2 https://marc.info/?l=openbsd-ports-cvs&m=158083696719245&w=2
- Flow 6y ago
  > All except some of the most recent arm64 processors have a speculative execution flaw that occurs across a syscall boundary, which cannot be mitigated in the kernel. Big Ouch. I wonder how the Linux developers will approach this bug since they don't enforce syscalls to be done from glibc.
  - spijdar 6y ago
    It's not enforced, but I'd dare say by and large almost everything will just use glibc. I'd assume if you're playing with fire enough to be calling syscalls yourself, you can mitigate the bugs yourself. I don't think it'll be a bit problem, anyway. In my experience not very much calls syscalls directly. Go is a big exception, though...
    - arghwhat 6y ago
      > It's not enforced, but I'd dare say by and large almost everything will just use glibc. What about musl? uClibc? Linux is well known for the fact that it guarantees its syscall interface as primary contract to userspace.
      - spijdar 6y ago
        Okay, so I'm referring mainly to the typical "GNU/Linux" desktop/server OS vs embedded or "container Linux" which is where the majority of alternative libc use will happen. In either case though there is a libc that most software will use, and the mitigations can be applied there. Even though direct use of syscalls is legal on Linux, the fact that it's stable is primarily relevant and interesting to said libc developers. The fact that syscalls aren't guaranteed on other systems is usually of little consequence since the libc is developed in tandem with the kernels of those systems. Linux's situation as a fully decoupled kernel means it does things differently in that sense. The developers are fully separated, so there needs to be a strong "contract" that syscalls will be stable. Doesn't mean (IMO) it's a good idea for end-users e.g. software developers to use syscalls except in exceptional circumstances. Which is usually the case!
ashishmax31 6y ago
So does that mean no statically linked binaries since glibc can't be statically linked?
- deleted 6y ago
  [deleted]
- tephra 6y ago
  OpenBSD does not use glibc.
- dilyevsky 6y ago
  Not glibc in mac os/openbsd case but yeah can’t have fully static go binaries on those.
- pjmlp 6y ago
  glibc is Linux only.
  - sanxiyn 6y ago
    glibc also supports Hurd.
    - pjmlp 6y ago
      Yep, it is already at 1.0 finally?
    - Blikkentrekker 6y ago
      Did Debian not also port most of glibc to support FreeBSD's kernel?
      - sanxiyn 6y ago
        Yes they did. See http://www.nongnu.org/glibc-bsd/ http://www.nongnu.org/glibc-bsd/.
- makapuf 6y ago
kisamoto 6y ago
Can someone please ELI5 the significance of this for me? Thanks
- Someone 6y ago
  In Linux, the system call interface is stable (on any given architecture, but not across them. See https://stackoverflow.com/questions/10281567/why-are-the-system-call-numbers-different-in-amd64-linux https://stackoverflow.com/questions/10281567/why-are-the-sys...): the way you call any kernel function is guaranteed to stay the same forever. That means that programs that directly make system calls will keep working on newer OSes. On many (¿most? https://unix.stackexchange.com/questions/473137/do-other-unix-like-kernels-have-stable-syscall-abis https://unix.stackexchange.com/questions/473137/do-other-uni...) other operating systems, that’s not the case; the OS ships with a library that provides a stable interface, and system call numbers, arguments, or calling conventions can change (in theory, the interface could even change across reboots or process runs). On openBSD, that library is Libc (and, unfortunately, is a lot larger than just the OS interface. IMO, in an ideal world, it should be split in two parts, the OS interface and a C library) Go wants to produce statically linked executables. It can’t do both that and link with LibC. It now changed to dynamically link with LibC, guaranteeing that what you compile today will run as well on next year’s OpenBSD as it does on today’s one. On top of that, openBSD has a security feature where it verifies that system calls are made via LibC. That feature wasn’t implemented as thoroughly as possible because go made direct system calls. This change allows OpenBSD to tighten that feauture.
  - kisamoto 6y ago
    Thank you! That was really informative and I will check out those links to learn more. I've been developing with Go for a few years now but never strayed too low level so was unsure how this change in Go 1.16 would affect me. I doubt my code will run on OpenBSD in the near future however I'm happy to know that if it does it will be supported in the future.
    - ben_bai 6y ago
      go apps are and will be easy to run on OpenBSD. It's one of the languages which are really as easy as "go get githbu.com/..." Edit: It's just the go mantra of compile once run forever doesn't really work on OpenBSD. You do have to recompile under certain conditions.
lifeplusplus 6y ago
does this have performance implication
- IcePic 6y ago
  It will cost a cycle or three extra, but given the great cost to make syscalls in the first place (especially if you have to do flushes and mitigations for the 'recent' security bugs on x86) it probably won't matter much. 1000 vs 1003 or 10000 vs 10003 would be hard to measure in a real world program. Also, many things libc does (which go programs need to do themselves) is to wrap a lot of the calls to things like malloc/calloc/realloc to use internal buckets and as seldom as possible call out to the kernel to get one or ten pages of ram in one call, then hand out suballocations from them for each "obj = malloc(8);" so that you minimize the amount of syscalls made, regardless of if you do them directly or via libc. Since syscalls always were expensive (you need to save registers, check userid permissions, flip to kernel mode, do the work asked for with or without SMP locking protections, give permissions to uid for the resource returned, perhaps check if its time to deliver signals or switch to another process and if not, flip out of kernel mode, restore registers and return to the process again) a lot of the calls done by a C program is kept in libc if possible. For example, gettimeofday() springs to mind, where a lot of trickery is done to give all programs a readonly page with the current time mapped into your program space so the call doesn't have to go via the kernel but instead becomes a memory read, since programs tend to call this thousands of times.
  - saagarjha 6y ago
    Of course, on Linux the kernel API extends to the vDSO, so Go can certainly rely on its existence. But reimplementing a libc for no good reason is typically a fool's errand.
    - masklinn 6y ago
      > Of course, on Linux the kernel API extends to the vDSO The kernel ABI, however, does not. vDSO are shared objects exposing a C ABI, and oddball compilation settings have broken Go's vDSO calls in the past: https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/
tus88 6y ago
Err....what about the stacks man?
bitcharmer 6y ago
I always thought GO makes syscalls via libc. If it doesn't am I correct assuming no LD_PRELOAD magic will work with GO? As in no custom memory allocators, no kernel bypass for TCP stack, etc.
- rfoo 6y ago
  Yes, you are correct. That also means no proxychains.
Hydraulix989 6y ago
I agree with OpenBSD's philosophy of introducing breaking ABI changes between versions, if security can be improved.
- Hydraulix989 6y ago
  This got downvoted?! If you want thirty years of backwards compatibility, go use Windows (you saw how well that worked for security...). OpenBSD is an OS that people choose to use when they want security prioritized as a trade-off against other things such as performance and binary compatibility (there's always trade-offs). Many firewalls use it, for example. If you value other things more than sheer data security, then there's other (beautiful) choices. (Analogously, there's no single best vehicle for everybody.) Anyone with a modicum of actual OS kernel development experience would acknowledge these mere facts as universal truths, instead of "unpopular opinions."
floatboth 6y ago
Now do this on FreeBSD too.
- EdSchouten 6y ago
  Why? As far as I know, FreeBSD's system call ABI is supposed to be (relatively) stable. It's a requirement anyway if you want to run jails of a different version.
  - floatboth 6y ago
    Personally: because it makes porting to new CPU architectures hell. (The Go assembler is the worst!) I've abandoned the FreeBSD/arm64 port and two other people had to pick it up to finish it. But also: while it's stable, it's not public. IIRC, Go developers were told about this, but decided to ignore. And of course: stop breaking LD_PRELOAD hooks :P
sivizius 6y ago
https://lwn.net/Articles/806776/ https://lwn.net/Articles/806776/ says, system-call-origin verification is for mitigation of ROP. ROP is (always?) a result of stack buffer overflows, which are a result of bad programming in asm/C/C++, but usually not a thing at all with modern languages like Go, Rust, Swift, Haskell, etc. while C/C++-programs usually use the libc already. This enforces programs written in such languages to use another layer, written in C, that is not really necessary but might introduce new vulnerabilities. Do I miss something?
- roblabla 6y ago
  The system call interface is usually written in an unsafe language such as asm anyways. Going through the libc is very unlikely to actually introduce vulnerabilities, especially if going through the lowest level function that directly wrap the syscall. Not using the libc was always a risky proposition on BSDs anyways. They don't have a stable kernel ABI the same way the Linux kernel does. From OpenBSD's perspective, the stable ABI is the libc, and anything using the kernel ABI directly is liable for breakage with each update.
  - loosescrews 6y ago
    > Going through the libc is very unlikely to actually introduce vulnerabilities Citation needed. glibc has a long history of security bugs. https://www.cvedetails.com/vulnerability-list/vendor_id-72/product_id-767/GNU-Glibc.html https://www.cvedetails.com/vulnerability-list/vendor_id-72/p...
    - lxgr 6y ago
      To quote the grandparent post: > especially if going through the lowest level function that directly wrap the syscall. I think the argument is that security bugs are unlikely to occur in those low-level wrappers.
    - roblabla 6y ago
      Geez man, way to take things out of context. Let me fix your quote: > Going through the libc is very unlikely to actually introduce vulnerabilities, especially if going through the lowest level function that directly wrap the syscall. Sure, glibc has a bunch of bugs. But the lowest level of functions, that just wrap the syscalls, are very unlikely to have bugs. Here's the `read` implementation, for instance: https://github.com/bminor/glibc/blob/21c3f4b5368686ade28d90d8c7d79c4c95c72c1b/sysdeps/unix/sysv/linux/read.c#L24 https://github.com/bminor/glibc/blob/21c3f4b5368686ade28d90d... All it does is delegate to the low-level syscall, with some extra handling around to handle async calls (which can be removed when compiling glibc yourself, but you're probably not doing this). Here's clone: https://github.com/bminor/glibc/blob/21c3f4b5368686ade28d90d8c7d79c4c95c72c1b/sysdeps/unix/sysv/linux/x86_64/clone.S#L50 https://github.com/bminor/glibc/blob/21c3f4b5368686ade28d90d... This one's written in asm, and you can't really simplify it all that much more. All the functions that wrap the low-level syscalls are very hard to get wrong, really. Where the glibc bugs come from are the high-level functions, like pthread. But those can trivially be bypassed if necessary.
simias 6y ago
Whenever the topic of Go bypassing the libc was brought up before the concerns were usually dismissed by the Go devs, saying that it was technically impractical due to some of Go's requirements not matching the libc's semantics. Does this mean that the Go developers managed to work around this problem or that it was just a flimsy post-hoc justification for chronic NIH syndrome? TFA itself links to another blog post discussing "Some reasons for Go to not make system calls through the standard C library"[1] but as far as I can tell it doesn't explain why it suddenly stopped being a problem on OpenBSD. [1] https://utcc.utoronto.ca/~cks/space/blog/programming/GoCLibraryAPIIssues https://utcc.utoronto.ca/~cks/space/blog/programming/GoCLibr...
- killingtime74 6y ago
  The NIH one, as always with Go
  - knorker 6y ago
    It's a clash of two NIH worlds: Go and OpenBSD. :-)
    - masklinn 6y ago
      I mean OpenBSD is hardly the first let alone only system which mandates going through libc (or equivalent) to interact with the OS.
      - knorker 6y ago
        Don't get me wrong, it makes perfect sense. And Go seems to aim more towards pragmaticism than OpenBSD's elegance and correctness. OpenBSD is deciding to invent their own paintbrush without even looking at what other people are painting with, while Go took the closest bucket of paint and threw it on the floor, not realizing they were standing in a corner while doing so. And I say that as a happy user of both.
- AnIdiotOnTheNet 6y ago
DominoTree 6y ago
At one point, building Go and running its test suite on OpenBSD would replace /dev/null with a standard flat file. After some time, the disk would fill up because everything piped to /dev/null was now being stored on disk. Not to mention the additional I/O happening. Go already uses libc by default on many platforms. But there are issues - sometimes the libc behaves differently than Go's documented APIs, but this is primarily a documentation issue. Contrariwise, sometimes, Go's native APIs don't behave on systems due to platform-specific implementation bugs. I think this is a good move overall.

298 points by lladnar 6y ago

It was very relevant to me a couple days ago, so I think it's not the case that Go programs that look up names link by default dynamically to libc, just for whatever that's worth to anyone. (Our company, Fly.io, runs container images for customers on Firecracker microVMs around the world, and I had to build a DNS-dependent service, in Go, that runs directly from our (Rust) init and can't assume a libc exists).

CameronNemo 6y ago
Is your Rust init private? What does it do?
- tptacek 6y ago
  Mostly just init stuff. We're transforming Docker containers (mostly) into standalone VMs, so it's doing all the scut work of taking a completely stripped-bare booted kernel and getting it to a state where you can run an arbitrary Linux program on it. I wouldn't want to take the thread off on a huge tangent, it's just funny that this was just recently super relevant to me (it would have been problematic if DNS-dependent Go programs depended on libc, because right now I can't assume there's a libc binary to be dynamically linked to). Bringing it back to Go and its (sometimes libc-dependent) DNS libraries: it is very annoying how fiddly it is to get a Go program to use an alternative DNS server.
  - JoshTriplett 6y ago
    I'm currently in the process, right now, of writing a minimal init in Rust to do exactly the same thing. Is yours something you'd consider sharing? The sum total of what I want to do: bring up loopback, bring up the one and only Ethernet interface, set up its IP and basic routing, run another program, and do some basic log reporting.
    - tptacek 6y ago
      I hack on our init, but I didn't write it. So it's not my place to share it. And there's some us-specific stuff in it that wouldn't be super helpful to you. But you could definitely ask Jerome on our team; he's a super helpful guy, even if he's super quiet here (he's like the anti-me). One way or the other I'm sure we can help you get where you're going!
      - JoshTriplett 6y ago
        I reached out to Jerome. Thanks!
        tptacek 6y ago
        Feel free to follow up with Kurt, me or Michael if you don't get a quick response; Jerome is in Montreal and he may be buried under a mountain of snow and brown gravy.
  - alaties 6y ago
    Had a similar problem a couple years ago where I needed to use alternative DNS libraries to troubleshoot issues in a company's infrastructure. Golang's rules for what implementation to use are found here: https://golang.org/pkg/net/#hdr-Name_Resolution https://golang.org/pkg/net/#hdr-Name_Resolution A really solid alternative DNS client implementation can be found here: https://github.com/miekg/dns https://github.com/miekg/dns. Real easy to read and vet compared to a few other libraries I ran into when working on this problem.
  - vidarh 6y ago
    If you can can control which DNS servers it talks to, implementing the DNS spec directly is a pretty trivial exercise to get full control. (If you can't, the main complexity is dealing with implementation quirks)
X-Istence 6y ago
The downside to Go shipping it's own implementation of DNS resolution is that on systems that support far more rich set of DNS options (such as split DNS based upon hostnames in macOS) is that it doesn't work if the binary is built with a pure Go implementation. That leaves users annoyed, because sending all DNS into a VPN tunnel is not always an option.
- yencabulator 6y ago
  If cgo is enabled (the usual case), Go uses libc for DNS if the configuration looks exotic enough that the pure-Go code wouldn't give the same results. https://golang.org/pkg/net/#hdr-Name_Resolution https://golang.org/pkg/net/#hdr-Name_Resolution
  - ArchOversight 6y ago
    This is a long standing issue in various open source projects that use Go, where they want to ship static binaries. Most recently, this happened with Concourse and it's Fly binary: https://github.com/concourse/concourse/issues/3691 https://github.com/concourse/concourse/issues/3691 The developers don't want to use CGO, or when they do use CGO they disable the net part... and now stuff doesn't work. Projects have to specifically build with cgo enabled on macOS, or else it fails.

saagarjha 6y ago

Guess they finally saw sense and realized that the when platforms say the syscall ABI is unstable, it really is unstable…

chungy 6y ago
OpenBSD has never really cared about backwards compatibility with binaries. It often works, but then suddenly they break. The OpenBSD developers really prefer fixing bugs instead of working around them.
- segfaultbuserr 6y ago
  It's also the advantage you have when you officially maintain a complete operating system, not just a kernel.
  - anthk 6y ago
    As long as you can (cross) compile go source easily, I don't see it as a problem.
jart 6y ago
There's no good reason for a Unix system to have an unstable system call ABI. I'm sorry but companies like Apple (who broke every Go binary a few years ago) don't get to copy syscall definitions from Bell System V and then declare it their own internal API. Usually the only time the SYSCALL ABI breaks, it's because kernel authors intentionally chose to do it for no apparent reason. For example, OpenBSD at some point changed how mmap() was defined so that it takes seven arguments: void *sys_mmap(void *addr, size_t len, int prot, int flags, int fd, long pad, off_t pos); The sixth argument doesn't do anything. It just breaks binary compatibility. It's also noncompliant with the System V ABI specification, which says system calls have six arguments max. I work on a project called Cosmopolitan Libc which lets you create static binaries that just work on Linux + Mac + Windows + FreeBSD + OpenBSD. It was only possible to do this because Unix systems generally agree on definitions. I worked really hard to support OpenBSD since I believe in the project. I just hope they keep the ABI stable going forward. For example, I'm really happy that this restriction only applies to dynamic binaries. It seems perfectly reasonable that they'd want to make the assumption that if a program chooses to link OpenBSD's Libc that it intends to use it. That's fine just so long as we continue having the ability to build static binaries with an alternative cross-platform Libc like Cosmopolitan. Speaking of which, I think I might actually implement some of OpenBSD's ideas in Cosmopolitan. I could probably track down all the functions that need raw SYSCALL and use __section__ so they're all linked to the same part of the binary and then call the msyscall() function to limit it just to that page. That way as a guest libc author I'm upholding the spirit of the intent. When in Rome do as the Romans.

bithavoc 6y ago

Same for macOS and iOS since Go 1.11, it all goes through libSystem now.

tedunangst 6y ago

The driving force for this was the syscall origin check, but syscall ABIs change too. There's a speculative execution bug in ARM CPUs which requires barriers. This also requires patching. https://marc.info/?l=openbsd-ports-cvs&m=158083696719245&w=2 https://marc.info/?l=openbsd-ports-cvs&m=158083696719245&w=2

Flow 6y ago
> All except some of the most recent arm64 processors have a speculative execution flaw that occurs across a syscall boundary, which cannot be mitigated in the kernel. Big Ouch. I wonder how the Linux developers will approach this bug since they don't enforce syscalls to be done from glibc.
- spijdar 6y ago
  It's not enforced, but I'd dare say by and large almost everything will just use glibc. I'd assume if you're playing with fire enough to be calling syscalls yourself, you can mitigate the bugs yourself. I don't think it'll be a bit problem, anyway. In my experience not very much calls syscalls directly. Go is a big exception, though...
  - arghwhat 6y ago
    > It's not enforced, but I'd dare say by and large almost everything will just use glibc. What about musl? uClibc? Linux is well known for the fact that it guarantees its syscall interface as primary contract to userspace.
    - spijdar 6y ago
      Okay, so I'm referring mainly to the typical "GNU/Linux" desktop/server OS vs embedded or "container Linux" which is where the majority of alternative libc use will happen. In either case though there is a libc that most software will use, and the mitigations can be applied there. Even though direct use of syscalls is legal on Linux, the fact that it's stable is primarily relevant and interesting to said libc developers. The fact that syscalls aren't guaranteed on other systems is usually of little consequence since the libc is developed in tandem with the kernels of those systems. Linux's situation as a fully decoupled kernel means it does things differently in that sense. The developers are fully separated, so there needs to be a strong "contract" that syscalls will be stable. Doesn't mean (IMO) it's a good idea for end-users e.g. software developers to use syscalls except in exceptional circumstances. Which is usually the case!

ashishmax31 6y ago

So does that mean no statically linked binaries since glibc can't be statically linked?

deleted 6y ago
[deleted]
tephra 6y ago
OpenBSD does not use glibc.
dilyevsky 6y ago
Not glibc in mac os/openbsd case but yeah can’t have fully static go binaries on those.
pjmlp 6y ago
glibc is Linux only.
- sanxiyn 6y ago
  glibc also supports Hurd.
  - pjmlp 6y ago
    Yep, it is already at 1.0 finally?
  - Blikkentrekker 6y ago
    Did Debian not also port most of glibc to support FreeBSD's kernel?
    - sanxiyn 6y ago
      Yes they did. See http://www.nongnu.org/glibc-bsd/ http://www.nongnu.org/glibc-bsd/.
makapuf 6y ago

kisamoto 6y ago

Can someone please ELI5 the significance of this for me? Thanks

Someone 6y ago
In Linux, the system call interface is stable (on any given architecture, but not across them. See https://stackoverflow.com/questions/10281567/why-are-the-system-call-numbers-different-in-amd64-linux https://stackoverflow.com/questions/10281567/why-are-the-sys...): the way you call any kernel function is guaranteed to stay the same forever. That means that programs that directly make system calls will keep working on newer OSes. On many (¿most? https://unix.stackexchange.com/questions/473137/do-other-unix-like-kernels-have-stable-syscall-abis https://unix.stackexchange.com/questions/473137/do-other-uni...) other operating systems, that’s not the case; the OS ships with a library that provides a stable interface, and system call numbers, arguments, or calling conventions can change (in theory, the interface could even change across reboots or process runs). On openBSD, that library is Libc (and, unfortunately, is a lot larger than just the OS interface. IMO, in an ideal world, it should be split in two parts, the OS interface and a C library) Go wants to produce statically linked executables. It can’t do both that and link with LibC. It now changed to dynamically link with LibC, guaranteeing that what you compile today will run as well on next year’s OpenBSD as it does on today’s one. On top of that, openBSD has a security feature where it verifies that system calls are made via LibC. That feature wasn’t implemented as thoroughly as possible because go made direct system calls. This change allows OpenBSD to tighten that feauture.
- kisamoto 6y ago
  Thank you! That was really informative and I will check out those links to learn more. I've been developing with Go for a few years now but never strayed too low level so was unsure how this change in Go 1.16 would affect me. I doubt my code will run on OpenBSD in the near future however I'm happy to know that if it does it will be supported in the future.
  - ben_bai 6y ago
    go apps are and will be easy to run on OpenBSD. It's one of the languages which are really as easy as "go get githbu.com/..." Edit: It's just the go mantra of compile once run forever doesn't really work on OpenBSD. You do have to recompile under certain conditions.

lifeplusplus 6y ago

does this have performance implication

IcePic 6y ago
It will cost a cycle or three extra, but given the great cost to make syscalls in the first place (especially if you have to do flushes and mitigations for the 'recent' security bugs on x86) it probably won't matter much. 1000 vs 1003 or 10000 vs 10003 would be hard to measure in a real world program. Also, many things libc does (which go programs need to do themselves) is to wrap a lot of the calls to things like malloc/calloc/realloc to use internal buckets and as seldom as possible call out to the kernel to get one or ten pages of ram in one call, then hand out suballocations from them for each "obj = malloc(8);" so that you minimize the amount of syscalls made, regardless of if you do them directly or via libc. Since syscalls always were expensive (you need to save registers, check userid permissions, flip to kernel mode, do the work asked for with or without SMP locking protections, give permissions to uid for the resource returned, perhaps check if its time to deliver signals or switch to another process and if not, flip out of kernel mode, restore registers and return to the process again) a lot of the calls done by a C program is kept in libc if possible. For example, gettimeofday() springs to mind, where a lot of trickery is done to give all programs a readonly page with the current time mapped into your program space so the call doesn't have to go via the kernel but instead becomes a memory read, since programs tend to call this thousands of times.
- saagarjha 6y ago
  Of course, on Linux the kernel API extends to the vDSO, so Go can certainly rely on its existence. But reimplementing a libc for no good reason is typically a fool's errand.
  - masklinn 6y ago
    > Of course, on Linux the kernel API extends to the vDSO The kernel ABI, however, does not. vDSO are shared objects exposing a C ABI, and oddball compilation settings have broken Go's vDSO calls in the past: https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/

tus88 6y ago

Err....what about the stacks man?

bitcharmer 6y ago

I always thought GO makes syscalls via libc. If it doesn't am I correct assuming no LD_PRELOAD magic will work with GO? As in no custom memory allocators, no kernel bypass for TCP stack, etc.

rfoo 6y ago
Yes, you are correct. That also means no proxychains.

Hydraulix989 6y ago

I agree with OpenBSD's philosophy of introducing breaking ABI changes between versions, if security can be improved.

Hydraulix989 6y ago
This got downvoted?! If you want thirty years of backwards compatibility, go use Windows (you saw how well that worked for security...). OpenBSD is an OS that people choose to use when they want security prioritized as a trade-off against other things such as performance and binary compatibility (there's always trade-offs). Many firewalls use it, for example. If you value other things more than sheer data security, then there's other (beautiful) choices. (Analogously, there's no single best vehicle for everybody.) Anyone with a modicum of actual OS kernel development experience would acknowledge these mere facts as universal truths, instead of "unpopular opinions."

floatboth 6y ago

Now do this on FreeBSD too.

EdSchouten 6y ago
Why? As far as I know, FreeBSD's system call ABI is supposed to be (relatively) stable. It's a requirement anyway if you want to run jails of a different version.
- floatboth 6y ago
  Personally: because it makes porting to new CPU architectures hell. (The Go assembler is the worst!) I've abandoned the FreeBSD/arm64 port and two other people had to pick it up to finish it. But also: while it's stable, it's not public. IIRC, Go developers were told about this, but decided to ignore. And of course: stop breaking LD_PRELOAD hooks :P

sivizius 6y ago

https://lwn.net/Articles/806776/ https://lwn.net/Articles/806776/ says, system-call-origin verification is for mitigation of ROP. ROP is (always?) a result of stack buffer overflows, which are a result of bad programming in asm/C/C++, but usually not a thing at all with modern languages like Go, Rust, Swift, Haskell, etc. while C/C++-programs usually use the libc already. This enforces programs written in such languages to use another layer, written in C, that is not really necessary but might introduce new vulnerabilities. Do I miss something?

roblabla 6y ago
The system call interface is usually written in an unsafe language such as asm anyways. Going through the libc is very unlikely to actually introduce vulnerabilities, especially if going through the lowest level function that directly wrap the syscall. Not using the libc was always a risky proposition on BSDs anyways. They don't have a stable kernel ABI the same way the Linux kernel does. From OpenBSD's perspective, the stable ABI is the libc, and anything using the kernel ABI directly is liable for breakage with each update.
- loosescrews 6y ago
  > Going through the libc is very unlikely to actually introduce vulnerabilities Citation needed. glibc has a long history of security bugs. https://www.cvedetails.com/vulnerability-list/vendor_id-72/product_id-767/GNU-Glibc.html https://www.cvedetails.com/vulnerability-list/vendor_id-72/p...
  - lxgr 6y ago
    To quote the grandparent post: > especially if going through the lowest level function that directly wrap the syscall. I think the argument is that security bugs are unlikely to occur in those low-level wrappers.
  - roblabla 6y ago
    Geez man, way to take things out of context. Let me fix your quote: > Going through the libc is very unlikely to actually introduce vulnerabilities, especially if going through the lowest level function that directly wrap the syscall. Sure, glibc has a bunch of bugs. But the lowest level of functions, that just wrap the syscalls, are very unlikely to have bugs. Here's the `read` implementation, for instance: https://github.com/bminor/glibc/blob/21c3f4b5368686ade28d90d8c7d79c4c95c72c1b/sysdeps/unix/sysv/linux/read.c#L24 https://github.com/bminor/glibc/blob/21c3f4b5368686ade28d90d... All it does is delegate to the low-level syscall, with some extra handling around to handle async calls (which can be removed when compiling glibc yourself, but you're probably not doing this). Here's clone: https://github.com/bminor/glibc/blob/21c3f4b5368686ade28d90d8c7d79c4c95c72c1b/sysdeps/unix/sysv/linux/x86_64/clone.S#L50 https://github.com/bminor/glibc/blob/21c3f4b5368686ade28d90d... This one's written in asm, and you can't really simplify it all that much more. All the functions that wrap the low-level syscalls are very hard to get wrong, really. Where the glibc bugs come from are the high-level functions, like pthread. But those can trivially be bypassed if necessary.

simias 6y ago

Whenever the topic of Go bypassing the libc was brought up before the concerns were usually dismissed by the Go devs, saying that it was technically impractical due to some of Go's requirements not matching the libc's semantics. Does this mean that the Go developers managed to work around this problem or that it was just a flimsy post-hoc justification for chronic NIH syndrome? TFA itself links to another blog post discussing "Some reasons for Go to not make system calls through the standard C library"[1] but as far as I can tell it doesn't explain why it suddenly stopped being a problem on OpenBSD. [1] https://utcc.utoronto.ca/~cks/space/blog/programming/GoCLibraryAPIIssues https://utcc.utoronto.ca/~cks/space/blog/programming/GoCLibr...

killingtime74 6y ago
The NIH one, as always with Go
- knorker 6y ago
  It's a clash of two NIH worlds: Go and OpenBSD. :-)
  - masklinn 6y ago
    I mean OpenBSD is hardly the first let alone only system which mandates going through libc (or equivalent) to interact with the OS.
    - knorker 6y ago
      Don't get me wrong, it makes perfect sense. And Go seems to aim more towards pragmaticism than OpenBSD's elegance and correctness. OpenBSD is deciding to invent their own paintbrush without even looking at what other people are painting with, while Go took the closest bucket of paint and threw it on the floor, not realizing they were standing in a corner while doing so. And I say that as a happy user of both.
AnIdiotOnTheNet 6y ago

DominoTree 6y ago

At one point, building Go and running its test suite on OpenBSD would replace /dev/null with a standard flat file. After some time, the disk would fill up because everything piped to /dev/null was now being stored on disk. Not to mention the additional I/O happening. Go already uses libc by default on many platforms. But there are issues - sometimes the libc behaves differently than Go's documented APIs, but this is primarily a documentation issue. Contrariwise, sometimes, Go's native APIs don't behave on systems due to platform-specific implementation bugs. I think this is a good move overall.