5 ms·

Why is it that that every time the topic of system calls comes up, all these people come out of the woodwork to say we're not allowed to not link a platform's l

by jart 4y ago

Why is it that that every time the topic of system calls comes up, all these people come out of the woodwork to say we're not allowed to not link a platform's libc? That's an impossible to compromise with position, because Cosmopolitan Libc is a libc, so it can't very well depend on six other c libraries. It would destroy the project. Also where is this even written? The maintainers of systems like FreeBSD and NetBSD have done SO MUCH to help our project. I don't think they're anything like the anti-competitive userspace control freaks people on forums suggest they are. I think the folks making kernels are great people and outstanding engineers who want to cultivate an environment where programs can do what they want.

nibbleshifter 4y ago
Many developers seem to have a cultish belief that the systems libc is the Only True Libc, missing the point (or being incapable of understanding) that alternatives are possible.
- iainmerrick 4y ago
  Is that really any more cultish than the idea that the kernel interface is the only true interface, and everything else should be replaceable? It’s just a different approach. What’s so crazy about an OS vendor wanting to exercise control over that userspace shim layer, rather than assuming that user programs will be poking directly at the kernel? You can argue that having multiple libc implementations is good, sure, but it’s not the only way of doing things and it’s not without tradeoffs.
  - mananaysiempre 4y ago
    The argument in the first paragraph doesn’t really work, I think: the kernel-user boundary is a security one, while the libc-app one is not (OpenBSD is something of an outlier in this respect, and even its case is a bit murky). So it is a tad more natural to draw the ABI boundary at the former point than at the latter, though of course this is not at all the only design consideration. Looking at this through a KERNEL32 lens, I can actually somewhat see the alternative point of view: in NT < 4 it was mostly an in-process RPC proxy for the subsystem server in CSRSS, and one can argue that libc on Unix-likes is in the same position (cf the vDSO on Linux). You can build RPC to a trusted server either way: define a stable wire protocol; or require the client to load proxy code into its address space. COM on Windows and hardware-accelerated graphics on Linux both take the second approach. But then why the hell does the proxy also have opinions about memory allocation, assignment of TLS slots, or floating-point formatting? (KERNEL32 has them on the first two points as well, mind you.) History[1] aside, does this really look like good engineering? (KERNEL32 makes a bit of a point there, but given how much HeapAlloc sucks as an allocator compared to the interoperability benefits it brings, I’m not sure it should be taken too seriously.) [1] https://utcc.utoronto.ca/~cks/space/blog/unix/UnixAPIAndCRuntime https://utcc.utoronto.ca/~cks/space/blog/unix/UnixAPIAndCRun...
    - comex 4y ago
      It's a bit of a paradox, isn't it? In theory it seems quite elegant to avoid tightly coupling the memory allocator to the kernel. But in practice, the fact that every pointer has to be freed by the same DLL that allocated it is famous for causing problems for Windows developers. As is, more generally, the inability to pass C and C++ standard library types across DLL boundaries. I think my ideal system would look like Linux in some ways. Any given Linux system has a 'standard' shared library ecosystem, with a single shared libc, and usually a shared C++ standard library as well. Executables and libraries that participate in that ecosystem can assume they'll use the same standard libraries and so can pass those libraries' types across module boundaries. Alongside them are 'hermit' executables, like Go programs or C programs statically linking to musl, that avoid the standard libc. But those executables typically also avoid all other system shared libraries (or greatly constrain their usage), so the difficulty in using their APIs (due to mismatched standard library types) isn't a big deal. Now, Linux sort of forces this approach by having the dynamic linker itself be part of libc. But that has some downsides, such as forcing the vDSO to work differently from every other library. I think I'd prefer to have a universal dynamic linker, like on Windows, but to still have 'ecosystem shares a libc, while hermits stick to themselves' as a convention.
- twic 4y ago
  This is incorrect. On most platforms, the libc is part of the operating system, and the libc API is the supported system call interface; user programs aren't allowed to make system calls directly to the kernel, any more than they're allowed to jump directly into the middle of kernel routines. As described elsewhere in this thread, that is the policy of MacOS X and Windows. It is also a policy which OpenBSD is moving towards: https://lwn.net/Articles/806863/ https://lwn.net/Articles/806863/ According to the Go devs, this is also the policy on Solaris-family OSs (Go uses libc there): https://github.com/golang/go/issues/24357#issuecomment-373007875 https://github.com/golang/go/issues/24357#issuecomment-37300... AFAIK, it is only Linux which explicitly supports the kernel system call interface, and so where alternative libcs are possible.
  - nibbleshifter 4y ago
    > AFAIK, it is only Linux which explicitly supports the kernel system call interface, and so where alternative libcs are possible. You can do this on Windows. The syscall numbers change sometimes. I use it to evade EDR hooks :)
  - bogomipz 4y ago
    >"AFAIK, it is only Linux which explicitly supports the kernel system call interface.: Could you elaborate on this? Specifically I'm not understanding what exactly is "explicit" with regards system calls on Linux compared to other OSes.
    - nibbleshifter 4y ago
      I think by this they mean the policy of the Linux kernel "never breaking user land". Libc is seen as user land by the Linux kernel, it provides an API for programs to abstract them away from syscalls. Libc dies syscalls for programs. By "never breaking userland", Linux promises to never break libc by never changing syscall numbers/args/types - new syscalls are added in ways that don't break older syscalls. Other platforms don't make this explicit promise and reserve the right to shuffle syscalls around, change them, remove them, etc as desired - making maintaining a libc for them a pain in the ass, and programmers are told to only use the libc as syscalls don't come with guarantees. You can (and I often do) program on windows using only syscalls, but your program tends to need rebuilding across major versions as the syscall interface isn't stable. Fuck me typing that on a phone was painful.
catiopatio 4y ago
> ... all these people come out of the woodwork to say we're not allowed to not link a platform's libc? You're allowed, of course, it's just not supported. We pop out of the woodwork to note that it's not supported and will break. Linux's approach of treating the syscall ABI as a long-term supported, stable interface is somewhat unusual. > Cosmopolitan Libc is a libc, so it can't very well depend on six other c libraries At least on macOS, trapping to the kernel without going through libSystem first is simply not a supported use-case.
brynet 4y ago
OpenBSD removed the mincore(2) syscall your table lists as "best" 3 years ago. https://marc.info/?l=openbsd-cvs&m=154723400730318&w=2 https://marc.info/?l=openbsd-cvs&m=154723400730318&w=2
- deleted 4y ago
  [deleted]
kjeetgill 4y ago
I get you're carrying the weight of a thousand such conversations, but this poster didn't really come out railing against your choice or your libc. They just explained a consideration around this topic that others made. Didn't mention that you or anyone else shouldn't have and used fairly gentle language like "I believe" at that.
lgg 4y ago
Because you asked: https://developer.apple.com/library/archive/qa/qa1118/_index.html https://developer.apple.com/library/archive/qa/qa1118/_index... "Apple does not support statically linked binaries on Mac OS X. A statically linked binary assumes binary compatibility at the kernel system call interface, and we do not make any guarantees on that front. Rather, we strive to ensure binary compatibility in each dynamically linked system library and framework." More to the point, platform vendors decide what their ABI boundaries are. Historically Unix vendors (back in the late 80s and early 90s) had no ABI boundaries... the expectation was every new OS release would require a recompile of all software. Obviously Linus has very different ideas about stability than the other Unix like systems of the time (which was a good thing), and focused on syscall stability. That made a lot of sense since he only wanted to maintain a kernel, not a full OS distribution. When modern macOS and Windows developed their ABIs they both were relatively mature OS distributions including a dynamic linker and default runtime libraries, and the ABI boundary chosen was well above the kernel as that is an easier place to define and maintain it.
- loup-vaillant 4y ago
  > the ABI boundary chosen was well above the kernel as that is an easier place to define and maintain it. How do we actually know that placing the boundary above the kernel makes it easier to define and maintain?
  - dataflow 4y ago
    One reason (may not be their main one) could be that system calls have more overhead than function calls, and some things (like futexes) make more sense to implement on the userspace side. As another example, imagine a function for measuring time. You don't want that to be a system call if it's meant to be efficient, and when a better mechanism comes along, changing the implementation on the user space side is a lot easier and potentially more efficient if the kernel interface can be changed without breaking compatibility.
  - bell-cot 4y ago
    Quip: The higher up the rug, the more crap that there's room to sweep under it. Or: Life can be easier for the kernel team if they're allowed to make breaking changes. Then task the lib team with writing shims / wrappers / etc. to fix all the problems which that causes. Then the manager of the lib team may have a perfect reason to boost his headcount. Then...
hyperman1 4y ago
Let me crawl out of the woodwork to ask a question about the consequences: I understood Windows changed its syscall API a few times. How do you deal with that? What happens if they renumber again?
- Quekid5 4y ago
  Just because I'm a bit curious and don't know the history here: Did they need to renumber, and if so... why? (I'm not arguing for nor against syscalls-as-kernel-API, I'm honestly just curious. Without knowing too much about it, it seems pretty sensible to support statically linked executables, but I might be missing something. I guess linking against an "as old as you need to support" libc dynamically isn't the end of the world, but it does constrain the build environment somewhat.)
  - catiopatio 4y ago
    > I guess linking against an "as old as you need to support" libc dynamically isn't the end of the world, but it does constrain the build environment somewhat. I can't speak for Windows, but on macOS, you build for older targets by passing `-(mmacos|ios)-version-min=` to the compiler to specify your deployment target version; this is used to determine symbol visibility, API visibility, toggled #ifdefs, etc. The provided version is also stored in the Mach-O load commands of your executable, and will used to select compatible symbols (and enable compatibility shims) when loading your binary's image at runtime. No need for static linking — or jumping through hoops to build against an "as old as you need to support" set of installed libraries.
  - wongarsu 4y ago
    I'm pretty sure Windows just automatically generates syscall numbers in their build process. Since the kernel and the dynamic libraries that are the only blessed way to interact with the kernel are shipped together, that's not a problem from Microsoft's point of view. For example, look at [1] and search for e.g. 0x01aa, to see how the meaning of that syscall number changes in a pretty systematic way over releases. 1: https://github.com/j00ru/windows-syscalls/blob/master/x86/csv/nt.csv https://github.com/j00ru/windows-syscalls/blob/master/x86/cs...
iainmerrick 4y ago
Cosmopolitan Libc is a libc, so it can't very well depend on six other c libraries Is that really the case? Given it’s specifically intended as a highly portable libc, in some cases maybe it could be implemented by calling through to the blessed system libc, rather than kernel calls. I mean, that likely wouldn’t fit with your goals for the project, and it would likely need some horrible linker hackery, but in principle it seems technically possible. (I’ll believe you if you say it’s completely impossible, though!)
throwaway5920 4y ago
You’re clearly an extremely talented dev, but this is an immature attitude to have. It has nothing to do with being “anti-competitive”. It’s about how a particular system is designed. Linux decided for many reasons, including organizational and political reasons, to have the stable surface be the kernel ABI. Other systems have different considerations and design philosophies and chose differently. These are all valid choices (and even if you disagree, you’re not going to convince them to change). It’s the job of a developer targeting a platform to deal with that reality.
- anthk 4y ago
  That's why some people prefer OpenBSD (security, correctness, balance between functionality and minimalism), others NetBSD (portability for small/legacy/older systems, a barebones BSD) or FreeBSD (desktop/server performance focused, SMP and network/BUS I/O takes preference over security).

by jart 4y ago

nibbleshifter 4y ago

Many developers seem to have a cultish belief that the systems libc is the Only True Libc, missing the point (or being incapable of understanding) that alternatives are possible.

iainmerrick 4y ago
Is that really any more cultish than the idea that the kernel interface is the only true interface, and everything else should be replaceable? It’s just a different approach. What’s so crazy about an OS vendor wanting to exercise control over that userspace shim layer, rather than assuming that user programs will be poking directly at the kernel? You can argue that having multiple libc implementations is good, sure, but it’s not the only way of doing things and it’s not without tradeoffs.
- mananaysiempre 4y ago
  The argument in the first paragraph doesn’t really work, I think: the kernel-user boundary is a security one, while the libc-app one is not (OpenBSD is something of an outlier in this respect, and even its case is a bit murky). So it is a tad more natural to draw the ABI boundary at the former point than at the latter, though of course this is not at all the only design consideration. Looking at this through a KERNEL32 lens, I can actually somewhat see the alternative point of view: in NT < 4 it was mostly an in-process RPC proxy for the subsystem server in CSRSS, and one can argue that libc on Unix-likes is in the same position (cf the vDSO on Linux). You can build RPC to a trusted server either way: define a stable wire protocol; or require the client to load proxy code into its address space. COM on Windows and hardware-accelerated graphics on Linux both take the second approach. But then why the hell does the proxy also have opinions about memory allocation, assignment of TLS slots, or floating-point formatting? (KERNEL32 has them on the first two points as well, mind you.) History[1] aside, does this really look like good engineering? (KERNEL32 makes a bit of a point there, but given how much HeapAlloc sucks as an allocator compared to the interoperability benefits it brings, I’m not sure it should be taken too seriously.) [1] https://utcc.utoronto.ca/~cks/space/blog/unix/UnixAPIAndCRuntime https://utcc.utoronto.ca/~cks/space/blog/unix/UnixAPIAndCRun...
  - comex 4y ago
    It's a bit of a paradox, isn't it? In theory it seems quite elegant to avoid tightly coupling the memory allocator to the kernel. But in practice, the fact that every pointer has to be freed by the same DLL that allocated it is famous for causing problems for Windows developers. As is, more generally, the inability to pass C and C++ standard library types across DLL boundaries. I think my ideal system would look like Linux in some ways. Any given Linux system has a 'standard' shared library ecosystem, with a single shared libc, and usually a shared C++ standard library as well. Executables and libraries that participate in that ecosystem can assume they'll use the same standard libraries and so can pass those libraries' types across module boundaries. Alongside them are 'hermit' executables, like Go programs or C programs statically linking to musl, that avoid the standard libc. But those executables typically also avoid all other system shared libraries (or greatly constrain their usage), so the difficulty in using their APIs (due to mismatched standard library types) isn't a big deal. Now, Linux sort of forces this approach by having the dynamic linker itself be part of libc. But that has some downsides, such as forcing the vDSO to work differently from every other library. I think I'd prefer to have a universal dynamic linker, like on Windows, but to still have 'ecosystem shares a libc, while hermits stick to themselves' as a convention.
twic 4y ago
This is incorrect. On most platforms, the libc is part of the operating system, and the libc API is the supported system call interface; user programs aren't allowed to make system calls directly to the kernel, any more than they're allowed to jump directly into the middle of kernel routines. As described elsewhere in this thread, that is the policy of MacOS X and Windows. It is also a policy which OpenBSD is moving towards: https://lwn.net/Articles/806863/ https://lwn.net/Articles/806863/ According to the Go devs, this is also the policy on Solaris-family OSs (Go uses libc there): https://github.com/golang/go/issues/24357#issuecomment-373007875 https://github.com/golang/go/issues/24357#issuecomment-37300... AFAIK, it is only Linux which explicitly supports the kernel system call interface, and so where alternative libcs are possible.
- nibbleshifter 4y ago
  > AFAIK, it is only Linux which explicitly supports the kernel system call interface, and so where alternative libcs are possible. You can do this on Windows. The syscall numbers change sometimes. I use it to evade EDR hooks :)
- bogomipz 4y ago
  >"AFAIK, it is only Linux which explicitly supports the kernel system call interface.: Could you elaborate on this? Specifically I'm not understanding what exactly is "explicit" with regards system calls on Linux compared to other OSes.
  - nibbleshifter 4y ago
    I think by this they mean the policy of the Linux kernel "never breaking user land". Libc is seen as user land by the Linux kernel, it provides an API for programs to abstract them away from syscalls. Libc dies syscalls for programs. By "never breaking userland", Linux promises to never break libc by never changing syscall numbers/args/types - new syscalls are added in ways that don't break older syscalls. Other platforms don't make this explicit promise and reserve the right to shuffle syscalls around, change them, remove them, etc as desired - making maintaining a libc for them a pain in the ass, and programmers are told to only use the libc as syscalls don't come with guarantees. You can (and I often do) program on windows using only syscalls, but your program tends to need rebuilding across major versions as the syscall interface isn't stable. Fuck me typing that on a phone was painful.

catiopatio 4y ago

> ... all these people come out of the woodwork to say we're not allowed to not link a platform's libc? You're allowed, of course, it's just not supported. We pop out of the woodwork to note that it's not supported and will break. Linux's approach of treating the syscall ABI as a long-term supported, stable interface is somewhat unusual. > Cosmopolitan Libc is a libc, so it can't very well depend on six other c libraries At least on macOS, trapping to the kernel without going through libSystem first is simply not a supported use-case.

brynet 4y ago

OpenBSD removed the mincore(2) syscall your table lists as "best" 3 years ago. https://marc.info/?l=openbsd-cvs&m=154723400730318&w=2 https://marc.info/?l=openbsd-cvs&m=154723400730318&w=2

deleted 4y ago
[deleted]

kjeetgill 4y ago

I get you're carrying the weight of a thousand such conversations, but this poster didn't really come out railing against your choice or your libc. They just explained a consideration around this topic that others made. Didn't mention that you or anyone else shouldn't have and used fairly gentle language like "I believe" at that.

lgg 4y ago

Because you asked: https://developer.apple.com/library/archive/qa/qa1118/_index.html https://developer.apple.com/library/archive/qa/qa1118/_index... "Apple does not support statically linked binaries on Mac OS X. A statically linked binary assumes binary compatibility at the kernel system call interface, and we do not make any guarantees on that front. Rather, we strive to ensure binary compatibility in each dynamically linked system library and framework." More to the point, platform vendors decide what their ABI boundaries are. Historically Unix vendors (back in the late 80s and early 90s) had no ABI boundaries... the expectation was every new OS release would require a recompile of all software. Obviously Linus has very different ideas about stability than the other Unix like systems of the time (which was a good thing), and focused on syscall stability. That made a lot of sense since he only wanted to maintain a kernel, not a full OS distribution. When modern macOS and Windows developed their ABIs they both were relatively mature OS distributions including a dynamic linker and default runtime libraries, and the ABI boundary chosen was well above the kernel as that is an easier place to define and maintain it.

loup-vaillant 4y ago
> the ABI boundary chosen was well above the kernel as that is an easier place to define and maintain it. How do we actually know that placing the boundary above the kernel makes it easier to define and maintain?
- dataflow 4y ago
  One reason (may not be their main one) could be that system calls have more overhead than function calls, and some things (like futexes) make more sense to implement on the userspace side. As another example, imagine a function for measuring time. You don't want that to be a system call if it's meant to be efficient, and when a better mechanism comes along, changing the implementation on the user space side is a lot easier and potentially more efficient if the kernel interface can be changed without breaking compatibility.
- bell-cot 4y ago
  Quip: The higher up the rug, the more crap that there's room to sweep under it. Or: Life can be easier for the kernel team if they're allowed to make breaking changes. Then task the lib team with writing shims / wrappers / etc. to fix all the problems which that causes. Then the manager of the lib team may have a perfect reason to boost his headcount. Then...

hyperman1 4y ago

Let me crawl out of the woodwork to ask a question about the consequences: I understood Windows changed its syscall API a few times. How do you deal with that? What happens if they renumber again?

Quekid5 4y ago
Just because I'm a bit curious and don't know the history here: Did they need to renumber, and if so... why? (I'm not arguing for nor against syscalls-as-kernel-API, I'm honestly just curious. Without knowing too much about it, it seems pretty sensible to support statically linked executables, but I might be missing something. I guess linking against an "as old as you need to support" libc dynamically isn't the end of the world, but it does constrain the build environment somewhat.)
- catiopatio 4y ago
  > I guess linking against an "as old as you need to support" libc dynamically isn't the end of the world, but it does constrain the build environment somewhat. I can't speak for Windows, but on macOS, you build for older targets by passing `-(mmacos|ios)-version-min=` to the compiler to specify your deployment target version; this is used to determine symbol visibility, API visibility, toggled #ifdefs, etc. The provided version is also stored in the Mach-O load commands of your executable, and will used to select compatible symbols (and enable compatibility shims) when loading your binary's image at runtime. No need for static linking — or jumping through hoops to build against an "as old as you need to support" set of installed libraries.
- wongarsu 4y ago
  I'm pretty sure Windows just automatically generates syscall numbers in their build process. Since the kernel and the dynamic libraries that are the only blessed way to interact with the kernel are shipped together, that's not a problem from Microsoft's point of view. For example, look at [1] and search for e.g. 0x01aa, to see how the meaning of that syscall number changes in a pretty systematic way over releases. 1: https://github.com/j00ru/windows-syscalls/blob/master/x86/csv/nt.csv https://github.com/j00ru/windows-syscalls/blob/master/x86/cs...

iainmerrick 4y ago

Cosmopolitan Libc is a libc, so it can't very well depend on six other c libraries Is that really the case? Given it’s specifically intended as a highly portable libc, in some cases maybe it could be implemented by calling through to the blessed system libc, rather than kernel calls. I mean, that likely wouldn’t fit with your goals for the project, and it would likely need some horrible linker hackery, but in principle it seems technically possible. (I’ll believe you if you say it’s completely impossible, though!)

throwaway5920 4y ago

You’re clearly an extremely talented dev, but this is an immature attitude to have. It has nothing to do with being “anti-competitive”. It’s about how a particular system is designed. Linux decided for many reasons, including organizational and political reasons, to have the stable surface be the kernel ABI. Other systems have different considerations and design philosophies and chose differently. These are all valid choices (and even if you disagree, you’re not going to convince them to change). It’s the job of a developer targeting a platform to deal with that reality.

anthk 4y ago
That's why some people prefer OpenBSD (security, correctness, balance between functionality and minimalism), others NetBSD (portability for small/legacy/older systems, a barebones BSD) or FreeBSD (desktop/server performance focused, SMP and network/BUS I/O takes preference over security).