4 ms·

> The direct-syscalls-inside-the-binary model used by go (and only go, noone else in the history of the unix software does this) provided the biggest resistance

by yobert 3y ago

> The direct-syscalls-inside-the-binary model used by go (and only go, noone else in the history of the unix software does this) provided the biggest resistance against this effort". I know this annoys unix people. But I have to say I actually really like that Go shakes this up. I believe the C function monopoly just isn't healthy for things. You should be able to make a new completely unrelated language. The Go developers were the first in a long time to do this, not because they are stupid, but because they were ambitious.

steveklabnik 3y ago
My hot take here is that this split exists because of the way that Linux development works differently than other Unices and other operating systems. Essentially, it's an example of Conway's Law. As always, things are about API boundaries: what is internal to you, and what is exposed externally to your users. With Linux, well, we all know the rms copypasta: "I'd just like to interject for a moment. What you're referring to as Linux, is in fact, GNU/Linux, or as I've recently taken to calling it, GNU plus Linux." This is a joke, but it also points to something real and serious: Linux, being just a kernel, means that they provide a stable kernel API. glibc, being written by a different organization, builds on top of that API and adds its own stable API. By contrast, many other operating systems are developed as a full operating system. They don't produce a kernel as a standalone component. As such, they choose some other sort of boundary as the API for the OS. On unices, that's often the libc. On Windows, that's the Windows API, of which win32 is an example. There are good reasons to not make the kernel your API boundary. Different systems make different choices, and that's a good thing, not a bad thing.
- akira2501 3y ago
  > There are good reasons to not make the kernel your API boundary. Good reasons for the developers, not so much for the end users.
  - steveklabnik 3y ago
    I don't agree that those two things are at odds.
  - SSLy 3y ago
    WinNT seems to do just fine with that. And it's only because of users insisting on using abandonware
    - j16sdiz 3y ago
      Developers works on win32api, not the kernel side NT* whatever stuffs.
      - SSLy 3y ago
        Yes, that's my point. App developers (actual users of the win32 ABI) seem just fine with not calling kernel directly. App users also enjoy best in world backcompat.
anonacct37 3y ago
I do like this about go. And on Linux it arguably makes sense (I say arguable because DNS without CGO is still a common cause of issues and incompatibility). But Linux has, to the best of my understanding said "yes, we are ok with users using syscalls". Linux doesn't think that glibc is the only project allowed to interface with the kernel. But for other platforms like OpenBSD and windows they are quite simply relying on implementation details that the vendors consider to be a private and unsupported interface. This whole thing is also separate from "is making libc the only caller of the syscalls instruction" a good and meaningful security improvement.
- p_l 3y ago
  Windows also provides language-independent ways of calling low-level (and pretty high-level) OS interfaces. The library you have to link to access system services is not going to pollute your language environment with bad runtime.
- akira2501 3y ago
  > I say arguable because DNS without CGO is still a common cause of issues and incompatibility DNS without CGO works perfectly. The vendor specific ad hoc mechanisms for extending DNS in a site local context are not well supported. If they were implemented more sensibly, then Go, or any other language, would have no problem taking advantage of them even without the "C Library Resolver." Speaking of which, that "C Library Resolver," in my opinion, has one of the worst library interfaces in all of unix. It's not at all a hill worth new projects dying on.
  - anonacct37 3y ago
    > DNS without CGO works perfectly It does not. I know this because it impacts my daily work and the work of others. Honestly if you could make my day and go figure out exactly what's going wrong with the pure go DNS implementation it would make my life alot simpler and I wouldn't have to maintain shell scripts that update etc/hosts to hard code in ipv4 addresses for the APIs I access with terraform. https://github.com/hashicorp/terraform-provider-google/issues/6782 https://github.com/hashicorp/terraform-provider-google/issue...
cryptonector 3y ago
This isn't about Go vs C. This is about about indirection of system calls through stubs and also in a way about static linking vs. dynamic linking. Something like this could be done for C and others, but the kernel would have to walk the C stack to ensure that a) the callers of the system calls are run-time library stubs listed in the ELF, and b) that the callers of the stubs are listed in the ELFs (plural). Clearly this is easier to do if there's neither stubs nor dynamic linking, but that's not a reason to think Go is somehow superior to the rest.
devit 3y ago
Anything statically linked to the C library will also have direct syscalls in the binary.
Rusky 3y ago
I don't think syscall-vs-C-function is at all meaningful for someone making a completely unrelated language. The system has to pick some ABI for communicating with the kernel, and syscalls don't change that. The ISA generally doesn't define how arguments are passed, it's really just in the business of providing a fancy jump instruction. Arguably, using the same ABI for userspace C functions and for communicating with the kernel reduces the amount of work required of completely new languages, because they are likely to need C interop support anyway.
- touisteur 3y ago
  If you're designing a whole new language today, you might want to build something to fit better with the new io_uring-like interfaces to the kernel and make 'old school' syscalls a fallback or special case. Would be interesting to have this (queueing, batching, completion) as a basic layer.
mike_hock 3y ago
Yes, I'd like the OpenBSD approach a lot more if the specially privileged syscall library was a language-agnostic library just for syscalls instead of libc. Also, if you're gonna add new ELF sections anyway, why not do syscall "relocation" directly (with a similar randomization like ASLR) instead of going through stubs? This "relocation" doesn't actually need to change any memory or use any offset tables since the syscall numbers are a farce under the new system anyway. Just make it a map of the location of syscall instructions to implied syscall numbers. Once you've phased out the old syscall model, you can even repurpose RAX as an additional syscall parameter.
- Someone 3y ago
  > I'd like the OpenBSD approach a lot more if the specially privileged syscall library was a language-agnostic library just for syscalls instead of libc. I think they should split that part of, too. Such a split would better reflect the split of responsibilities between kernel maintainers and libc maintainers. At the same the effective difference for people using the code is minute. Any decent linker will strip out unused functions in libraries you link with, so if you currently only use the part of libc that would become “libsyscall”, the end effect would be the same as when “libsyscall” existed.

by yobert 3y ago

steveklabnik 3y ago

My hot take here is that this split exists because of the way that Linux development works differently than other Unices and other operating systems. Essentially, it's an example of Conway's Law. As always, things are about API boundaries: what is internal to you, and what is exposed externally to your users. With Linux, well, we all know the rms copypasta: "I'd just like to interject for a moment. What you're referring to as Linux, is in fact, GNU/Linux, or as I've recently taken to calling it, GNU plus Linux." This is a joke, but it also points to something real and serious: Linux, being just a kernel, means that they provide a stable kernel API. glibc, being written by a different organization, builds on top of that API and adds its own stable API. By contrast, many other operating systems are developed as a full operating system. They don't produce a kernel as a standalone component. As such, they choose some other sort of boundary as the API for the OS. On unices, that's often the libc. On Windows, that's the Windows API, of which win32 is an example. There are good reasons to not make the kernel your API boundary. Different systems make different choices, and that's a good thing, not a bad thing.

akira2501 3y ago
> There are good reasons to not make the kernel your API boundary. Good reasons for the developers, not so much for the end users.
- steveklabnik 3y ago
  I don't agree that those two things are at odds.
- SSLy 3y ago
  WinNT seems to do just fine with that. And it's only because of users insisting on using abandonware
  - j16sdiz 3y ago
    Developers works on win32api, not the kernel side NT* whatever stuffs.
    - SSLy 3y ago
      Yes, that's my point. App developers (actual users of the win32 ABI) seem just fine with not calling kernel directly. App users also enjoy best in world backcompat.

anonacct37 3y ago

I do like this about go. And on Linux it arguably makes sense (I say arguable because DNS without CGO is still a common cause of issues and incompatibility). But Linux has, to the best of my understanding said "yes, we are ok with users using syscalls". Linux doesn't think that glibc is the only project allowed to interface with the kernel. But for other platforms like OpenBSD and windows they are quite simply relying on implementation details that the vendors consider to be a private and unsupported interface. This whole thing is also separate from "is making libc the only caller of the syscalls instruction" a good and meaningful security improvement.

p_l 3y ago
Windows also provides language-independent ways of calling low-level (and pretty high-level) OS interfaces. The library you have to link to access system services is not going to pollute your language environment with bad runtime.
akira2501 3y ago
> I say arguable because DNS without CGO is still a common cause of issues and incompatibility DNS without CGO works perfectly. The vendor specific ad hoc mechanisms for extending DNS in a site local context are not well supported. If they were implemented more sensibly, then Go, or any other language, would have no problem taking advantage of them even without the "C Library Resolver." Speaking of which, that "C Library Resolver," in my opinion, has one of the worst library interfaces in all of unix. It's not at all a hill worth new projects dying on.
- anonacct37 3y ago
  > DNS without CGO works perfectly It does not. I know this because it impacts my daily work and the work of others. Honestly if you could make my day and go figure out exactly what's going wrong with the pure go DNS implementation it would make my life alot simpler and I wouldn't have to maintain shell scripts that update etc/hosts to hard code in ipv4 addresses for the APIs I access with terraform. https://github.com/hashicorp/terraform-provider-google/issues/6782 https://github.com/hashicorp/terraform-provider-google/issue...

cryptonector 3y ago

This isn't about Go vs C. This is about about indirection of system calls through stubs and also in a way about static linking vs. dynamic linking. Something like this could be done for C and others, but the kernel would have to walk the C stack to ensure that a) the callers of the system calls are run-time library stubs listed in the ELF, and b) that the callers of the stubs are listed in the ELFs (plural). Clearly this is easier to do if there's neither stubs nor dynamic linking, but that's not a reason to think Go is somehow superior to the rest.

devit 3y ago

Anything statically linked to the C library will also have direct syscalls in the binary.

Rusky 3y ago

I don't think syscall-vs-C-function is at all meaningful for someone making a completely unrelated language. The system has to pick some ABI for communicating with the kernel, and syscalls don't change that. The ISA generally doesn't define how arguments are passed, it's really just in the business of providing a fancy jump instruction. Arguably, using the same ABI for userspace C functions and for communicating with the kernel reduces the amount of work required of completely new languages, because they are likely to need C interop support anyway.

touisteur 3y ago
If you're designing a whole new language today, you might want to build something to fit better with the new io_uring-like interfaces to the kernel and make 'old school' syscalls a fallback or special case. Would be interesting to have this (queueing, batching, completion) as a basic layer.

mike_hock 3y ago

Yes, I'd like the OpenBSD approach a lot more if the specially privileged syscall library was a language-agnostic library just for syscalls instead of libc. Also, if you're gonna add new ELF sections anyway, why not do syscall "relocation" directly (with a similar randomization like ASLR) instead of going through stubs? This "relocation" doesn't actually need to change any memory or use any offset tables since the syscall numbers are a farce under the new system anyway. Just make it a map of the location of syscall instructions to implied syscall numbers. Once you've phased out the old syscall model, you can even repurpose RAX as an additional syscall parameter.

Someone 3y ago
> I'd like the OpenBSD approach a lot more if the specially privileged syscall library was a language-agnostic library just for syscalls instead of libc. I think they should split that part of, too. Such a split would better reflect the split of responsibilities between kernel maintainers and libc maintainers. At the same the effective difference for people using the code is minute. Any decent linker will strip out unused functions in libraries you link with, so if you currently only use the part of libc that would become “libsyscall”, the end effect would be the same as when “libsyscall” existed.