5 ms·

Sorry, that generalization doesn’t hold water. It makes sense for systems where libc is tightly coupled and coversioned with the kernel, e.g. BSDs, but Linux

Sorry, that generalization doesn’t hold water. It makes sense for systems where libc is tightly coupled and coversioned with the kernel, e.g. BSDs, but Linux always relied on third-party C libraries and supported static binaries, etc. You could argue that BSD made the mistake of intending to have a Windows-style C library compat guarantee but not enforcing it, but that was not in scope for Linux. The philosophy has always been syscall-level compat (and there are lots of famous threads with Linus re-enforcing this to others who would presume that things should be “fixed in user space”). So it’s hardly reasonable to generalize based on some BSD concerns; Linux is WAI and represents the most common Unix-like system people use today by far. There’s a pretty good argument that this level of compat, while the source of some problems, has also made other things much easier: consider container images that are bundled with their own system libraries. (You could certainly invent schemes to inject these libraries, but dealing with link and library level compatibility seems even more complex to deal with than system call-level compatibility.)

pcwalton 7y ago
Darwin/macOS has the same rules as Windows and the BSDs--syscalls are private API--and it's extremely popular due to iOS. Linux is in fact the odd one out here.
- giovannibajo1 7y ago
  There is a difference though: libSystem on Darwin is a very thin wrapper over the kernel syscalls; on the contrary, libc is a library that was designed for C, then standardized in POSIX, and has several layer of abstraction over kernel syscalls including many bad defaults that are universally recognized as wrong today (eg: libc’s created file descriptors will all inherit by default).
  - cesarb 7y ago
    > (eg: libc’s created file descriptors will all inherit by default) Isn't that the kernel default? Even if you use system calls directly, file descriptors still inherit by default.
    - Gaelan 7y ago
      I mean, theoretically (i.e. I have no idea if anything does this), you could have underlying system file descriptors, which did not inherit, then a mapping from "libc" file descriptors onto OS ones and some code in fork() to copy over any OS file descriptors that are exposed to libc-using code.
      - loeg 7y ago
        The mapping is the identity function and whether a descriptor is inherit or not is just a function of O_CLOEXEC / fcntl(FD_CLOEXEC) / some special fd types are always cloexec, such as kqueues on FreeBSD. Libc fds aren't special to the operating system in any way.
    - loeg 7y ago
      Yeah, libc's syscall wrappers just do what you tell them. If you don't pass O_CLOEXEC to the kernel syscalls, you get the inherit behavior. Libc's syscall wrappers don't change this in any way. To the extent that Go's default for file descriptors today is !inherit (I'm unfamiliar, but if so, it's a good choice), the Go runtime must already add O_CLOEXEC to bare syscalls. There's no reason to believe it incapable of adding the flag to libc syscalls instead.
      - giovannibajo1 7y ago
        You can't do that atomically with libc. There's a short window in which the file descriptor will potentially be inherited, if another thread forks.
        jlokier 7y ago
        That's incorrect. You are thinking of the older way, where fcntl(fd, F_SETFD, FD_CLOEXEC) must be used after open(), leaving a short window in which the file descriptor may be inherited. The newer way passes the O_CLOEXEC to open() and there is no fcntl() call. This is atomic with respect to inheritability: The kernel returns a non-inheritable file descriptor to libc, and libc returns it to the application. Other syscalls that return a file descriptor have similar flags, so they are atomic too. These flags and behaviours are exactly the same, whether done by calling through libc as most programs do, or direct kernel syscalls bypassing libc, as Go and a few other programs do.
  - loeg 7y ago
    Go isn't obligated to use any libc APIs or abstractions other than those providing syscalls. You're incorrect or maybe just misleading about libc created file descriptors inheriting, as stated. Either way, it is unrelated to using libc for syscalls vs bare machine traps.
    - giovannibajo1 7y ago
      I think it's mandated by POSIX standard; but even if I'm wrong, there's still the problem that libc doesn't allow to do that atomically, for instance. In general, it's an old interface that doesn't fully expose the full power of all modern syscalls.
      - tedunangst 7y ago
        Most libc you'd care to use are going to include functions like dup3. If they don't, it's probably because the system call isn't available either.
      - loeg 7y ago
        You're mistaken. https://news.ycombinator.com/item?id=21740035 https://news.ycombinator.com/item?id=21740035 https://news.ycombinator.com/item?id=21740498 https://news.ycombinator.com/item?id=21740498
- barrkel 7y ago
  libc is a C runtime library, while kernel32 is not.
- typical182 7y ago
  For Go on Darwin: libSystem is now used when making syscalls on Darwin, ensuring forward-compatibility with future versions of macOS and iOS. From https://golang.org/doc/go1.12#darwin https://golang.org/doc/go1.12#darwin
  - loeg 7y ago
    This is mentioned obliquely in the top-level comment for this thread: > solaris/macos approach … of calling libc stubs
justincormack 7y ago
The plan for containers on Solaris, after rejecting injecting libc from the host, was to have users rebuild all containers after OS upgrade. Windows has to virtualise containers with an incompatible OS version. It is definitely less convenient than Linux. OpenBSD does have somewhat different constraints and they seem to think this will work for them.
- xenadu02 7y ago
  The iOS Simulator does something along these lines for macOS. When macOS is updated CoreSimulator rebuilds the dyld_sim shared caches because simulator runtimes pull in libsystem_kernel, libsystem_pthread, and libsystem_platform (which cover the core of the kernel’s ABI).
- binarycrusader 7y ago
  The plan for containers on Solaris, after rejecting injecting libc from the host, was to have users rebuild all containers after OS upgrade. What containers are you referring to? Because this is definitely not how Zones work on Solaris. There are two types of Zones in Solaris 11; "Kernel Zones" which run their own independent version of Solaris and "non-global Zones" which are automatically kept at the same version as the host. Windows has to virtualise containers with an incompatible OS version. Not as far as I'm aware. Windows Sandboxes don't work that way nor do other technologies I'm aware of. What are you referring to?
  - justincormack 7y ago
    This was Docker containers, which were under development, and I think one version shipped. Windows containers have to run under hyper-v for incompatible kernel versions.
masklinn 7y ago
> It makes sense for systems where libc is tightly coupled and coversioned with the kernel, e.g. BSDs, but Linux always relied on third-party C libraries and supported static binaries, etc. Linux is not a system. Linux is a kernel, linux has distributions, it is not a single coherent system where the kernel and standard library are co-developed. That's the entire point I'm making. > So it’s hardly reasonable to generalize based on some BSD concerns It's not "some BSD concerns", it's pretty much every non-linux unix. What's not reasonable is generalising "syscalls are a perfectly fine interface" which is almost exclusively a Linux exclusivity. Or don't claim compatibility with anything other than linux, that's also a perfectly fine choice. > There’s a pretty good argument that this level of compat, while the source of some problems, has also made other things much easier: consider container images that are bundled with their own system libraries. Last time I checked, Go did not run exclusively on linux. If it did, raw syscalls would indeed not be a concern (though even then they try to have their cake and eat it, as e.g. they want to do raw syscalls yet benefit from vDSO, which has been an issue in the past because their assumptions did not hold: https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/)

pcwalton 7y ago

Darwin/macOS has the same rules as Windows and the BSDs--syscalls are private API--and it's extremely popular due to iOS. Linux is in fact the odd one out here.

giovannibajo1 7y ago
There is a difference though: libSystem on Darwin is a very thin wrapper over the kernel syscalls; on the contrary, libc is a library that was designed for C, then standardized in POSIX, and has several layer of abstraction over kernel syscalls including many bad defaults that are universally recognized as wrong today (eg: libc’s created file descriptors will all inherit by default).
- cesarb 7y ago
  > (eg: libc’s created file descriptors will all inherit by default) Isn't that the kernel default? Even if you use system calls directly, file descriptors still inherit by default.
  - Gaelan 7y ago
    I mean, theoretically (i.e. I have no idea if anything does this), you could have underlying system file descriptors, which did not inherit, then a mapping from "libc" file descriptors onto OS ones and some code in fork() to copy over any OS file descriptors that are exposed to libc-using code.
    - loeg 7y ago
      The mapping is the identity function and whether a descriptor is inherit or not is just a function of O_CLOEXEC / fcntl(FD_CLOEXEC) / some special fd types are always cloexec, such as kqueues on FreeBSD. Libc fds aren't special to the operating system in any way.
  - loeg 7y ago
    Yeah, libc's syscall wrappers just do what you tell them. If you don't pass O_CLOEXEC to the kernel syscalls, you get the inherit behavior. Libc's syscall wrappers don't change this in any way. To the extent that Go's default for file descriptors today is !inherit (I'm unfamiliar, but if so, it's a good choice), the Go runtime must already add O_CLOEXEC to bare syscalls. There's no reason to believe it incapable of adding the flag to libc syscalls instead.
    - giovannibajo1 7y ago
      You can't do that atomically with libc. There's a short window in which the file descriptor will potentially be inherited, if another thread forks.
      - jlokier 7y ago
        That's incorrect. You are thinking of the older way, where fcntl(fd, F_SETFD, FD_CLOEXEC) must be used after open(), leaving a short window in which the file descriptor may be inherited. The newer way passes the O_CLOEXEC to open() and there is no fcntl() call. This is atomic with respect to inheritability: The kernel returns a non-inheritable file descriptor to libc, and libc returns it to the application. Other syscalls that return a file descriptor have similar flags, so they are atomic too. These flags and behaviours are exactly the same, whether done by calling through libc as most programs do, or direct kernel syscalls bypassing libc, as Go and a few other programs do.
- loeg 7y ago
  Go isn't obligated to use any libc APIs or abstractions other than those providing syscalls. You're incorrect or maybe just misleading about libc created file descriptors inheriting, as stated. Either way, it is unrelated to using libc for syscalls vs bare machine traps.
  - giovannibajo1 7y ago
    I think it's mandated by POSIX standard; but even if I'm wrong, there's still the problem that libc doesn't allow to do that atomically, for instance. In general, it's an old interface that doesn't fully expose the full power of all modern syscalls.
    - tedunangst 7y ago
      Most libc you'd care to use are going to include functions like dup3. If they don't, it's probably because the system call isn't available either.
    - loeg 7y ago
      You're mistaken. https://news.ycombinator.com/item?id=21740035 https://news.ycombinator.com/item?id=21740035 https://news.ycombinator.com/item?id=21740498 https://news.ycombinator.com/item?id=21740498
barrkel 7y ago
libc is a C runtime library, while kernel32 is not.
typical182 7y ago
For Go on Darwin: libSystem is now used when making syscalls on Darwin, ensuring forward-compatibility with future versions of macOS and iOS. From https://golang.org/doc/go1.12#darwin https://golang.org/doc/go1.12#darwin
- loeg 7y ago
  This is mentioned obliquely in the top-level comment for this thread: > solaris/macos approach … of calling libc stubs

justincormack 7y ago

The plan for containers on Solaris, after rejecting injecting libc from the host, was to have users rebuild all containers after OS upgrade. Windows has to virtualise containers with an incompatible OS version. It is definitely less convenient than Linux. OpenBSD does have somewhat different constraints and they seem to think this will work for them.

xenadu02 7y ago
The iOS Simulator does something along these lines for macOS. When macOS is updated CoreSimulator rebuilds the dyld_sim shared caches because simulator runtimes pull in libsystem_kernel, libsystem_pthread, and libsystem_platform (which cover the core of the kernel’s ABI).
binarycrusader 7y ago
The plan for containers on Solaris, after rejecting injecting libc from the host, was to have users rebuild all containers after OS upgrade. What containers are you referring to? Because this is definitely not how Zones work on Solaris. There are two types of Zones in Solaris 11; "Kernel Zones" which run their own independent version of Solaris and "non-global Zones" which are automatically kept at the same version as the host. Windows has to virtualise containers with an incompatible OS version. Not as far as I'm aware. Windows Sandboxes don't work that way nor do other technologies I'm aware of. What are you referring to?
- justincormack 7y ago
  This was Docker containers, which were under development, and I think one version shipped. Windows containers have to run under hyper-v for incompatible kernel versions.

masklinn 7y ago

> It makes sense for systems where libc is tightly coupled and coversioned with the kernel, e.g. BSDs, but Linux always relied on third-party C libraries and supported static binaries, etc. Linux is not a system. Linux is a kernel, linux has distributions, it is not a single coherent system where the kernel and standard library are co-developed. That's the entire point I'm making. > So it’s hardly reasonable to generalize based on some BSD concerns It's not "some BSD concerns", it's pretty much every non-linux unix. What's not reasonable is generalising "syscalls are a perfectly fine interface" which is almost exclusively a Linux exclusivity. Or don't claim compatibility with anything other than linux, that's also a perfectly fine choice. > There’s a pretty good argument that this level of compat, while the source of some problems, has also made other things much easier: consider container images that are bundled with their own system libraries. Last time I checked, Go did not run exclusively on linux. If it did, raw syscalls would indeed not be a concern (though even then they try to have their cake and eat it, as e.g. they want to do raw syscalls yet benefit from vDSO, which has been an issue in the past because their assumptions did not hold: https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/)