5 ms·

So... it's basically kernel32.dll? That is, the shared library with a stable API across all supported platforms, and stable ABI inside any one specific platform

So... it's basically kernel32.dll? That is, the shared library with a stable API across all supported platforms, and stable ABI inside any one specific platform which hides all the gritty details of switching to and back from the kernel? One wonders why it has not appeared much sooner but better late than never, I guess.

guerrilla 5y ago
The explanation for why it's useful is at the bottom of the page, but it wasn't necessary before glibc got behind and before Linux started adding things that glibc would probably never implement*. Generally, on UNIX-like systems, the C library isn't just the C library but also exposes the UNIX/POSIX API which is just a set of wrappers around calling into the kernel (done with software interrupt 0x80 on x86 Linux.) * Remember that glibc is part of the GNU project and is meant to be portable beyond Linux, especially to Hurd. It might not make sense for them to implement interfaces that wouldn't work or would require significant work to emulate on Hurd, the BSDs and Solaris.
- Joker_vD 5y ago
  Well, yeah, merging kernel calls into the runtime of one specifically favoured programming language is a silly idea, why not have the kernel calls in a separate, cleanly defined language-agnostic OS-specific library? "It just the way the things has come to work historically"?
  - guerrilla 5y ago
    Well, first of all the UNIX/POSIX API is a C API, so that question doesn't make much sense. On top of that, there have been no competitive systems programming languages until recently (and certainly none that weren't written in C), so that wouldn't have been worth it. Also, it wasn't really an idea per se: C is just the native language and lingua franca of UNIX and that's just how UNIX developed organically.
    - GoblinSlayer 5y ago
      Why would it matter what system programming languages are there? Unix can't run other languages?
      - koprulusector 5y ago
        The kernel is written in C. Ok, why does that matter? Because its interfaces and architecture are tightly intertwined with C. Remember that C was partially developed to use in building Unix. So, kernel was written in C and therefore uses C data structures, pointers, and follows C conventions for using them (for example, kernel might return a struct of pointers). This applies to interfaces and structures, making syscalls (depending on which syscall and what is needed), checking return values or error codes. You don’t NEED this stuff exactly to run on Unix. What and how much you need, or even just convenience, are how it boils down. If you want to use eBPF to trace things from raw network packets, syscalls, file systems, you can do some in Python but you’ll need some degree of C. You want to run a Django or flask app in just Python? Good to go. [This article on Linux ABI](https://en.wikipedia.org/wiki/Application_binary_interface https://en.wikipedia.org/wiki/Application_binary_interface) might help explain better/in more detail. Sorry if this isn’t the best explanation, I’m on mobile and up too late.
        GoblinSlayer 5y ago
        Design of unix was influenced more by PDP-7 than by C. What unix does, it does PDP-7 stuff, not C stuff. C itself isn't limited to POSIX.
      - johannes1234321 5y ago
        Unix can run other languages. But for answering your question deeper, we have to find an understanding what boundary we're talking about. The kernel syscall interface? Things like Linux's vdso? POSIX library (libc)? The more we go on in that sequence the more C we get into the interfaces and many of those APIs are essential for writing well-behaving non-trivial programs. The closer we stay to the kernel's syscall interface the less C you get, but still the syscall interface and C are related. Strings are often NULL-terminated. Arguemtns are passed (via registers) from left to right, not like in Pascal convention right to left. (First arg always in EBX, second in EXC etc.) A different language will need some interoperability with C conventions for being a good citizen on Unix. A Pascal operating system for instance would use Pascal strings (historically what tis now called ShortString) where length is encoded in the first byte.
  - hnlmorg 5y ago
    Because every programming language will have their own opinions on how to link to external libraries and what those external libraries should look like (eg Java libraries are ZIP archives with a .jar extension, JIT languages will have their libraries as raw source code). But since C is the de facto standard for OS libraries, most languages end up supporting calls to C libraries. Thus C libraries do become the language-agnostic library.
    - GoblinSlayer 5y ago
      Most languages can call C libraries because they can call any libraries, not because C libraries are language agnostic.
      - couchand 5y ago
        That doesn't ring true to me. My FFI experience is somewhat limited (JS to C, JS to C++ via C wrappers, Rust to C, Rust to C++ via C wrappers) but from what I've seen you have to speak C to make FFI.
        GoblinSlayer 5y ago
        Few languages truly speak full C (e.g. through clang plugin), they support ABI, but there are libraries like libpng and gtk with API disconnected from ABI, for them FFI doesn't resemble heavily macro encumbered C API.
      - matheusmoreira 5y ago
        Higher level languages such as C++ have extremely complex binary interfaces. Only C++ code dares to link against C++ code. Even then there are problems between different compilers and even different versions of the same compiler. New standard version comes out, it requires that standard classes have certain performance characteristics... Suddenly compilers are breaking binary compatibility in order to restablish conformance. So in order to have usable libraries people will wrap their C++ implementation in C interfaces. Rust faces the same problem. All those nice abstractions result in a complex ABI that will never touched by anything outside the Rust ecosystem. There is no escaping the extern "C". Some languages are virtualized. Java, C#, Ruby, Python, Javascript all have virtual machines. There's no way to interface directly with code running in those languages. We must interface with the virtual machine instead: import it into our process and ask it to call the functions we want to use. C binary interfaces are not language agnostic but they are quite simple. The result of compiling a C library is symbols pointing to functions with simple calling conventions. As a result, virtually any language can look up these symbols and call them.
  - deleted 5y ago
    [deleted]
GoblinSlayer 5y ago
kernel32.dll is a little more language agnostic: it provides heap, threads, mutexes, environment variables and whatnot.
matheusmoreira 5y ago
Yes! The difference is on Linux a shared library is not actually required for kernel system calls. The binary interface is stable and will not change. https://github.com/torvalds/linux/blob/master/Documentation/ABI/stable/syscalls https://github.com/torvalds/linux/blob/master/Documentation/... On Windows and other operating systems, things such as system call numbers can and do change. There's no way to interface with the kernel without linking to vendor libraries. So on Linux you can ditch libc and write your own code. A Rust liblinux is definitely possible, I tested it. You can even have a compiler generate the system calls for you! It just needs to support the simple calling convention. No libraries necessary. I've seen a project that does just that: https://github.com/titzer/virgil https://github.com/titzer/virgil
- Joker_vD 5y ago
  I mean, I also can call functions from kernel32.dll on Windows without need of any libc, and the ABI is stable and will not change; and if/when MS decides to change the actual mechanism of switching into the kernel mode, or decide to move some stuff from the kernel into the user space ― they can, and I don't have to worry about that (and recompile my apps) because it's all abstracted away! "Fix it once in the shared library, and all its clients will benefit automatically". On Linux, however, we have, for example, glibc on one hand which claims that it doesn't have a goal of forwarding all of the Linux's syscall, and several failed initiatives on the other hand to allow the user's code to perform syscalls only from inside of glibc (which spells doom to anything that doesn't use libc, e.g. Golang) because of security reasons. And the history of still having to support INT 80h together with SYSCALL on x64 on the third hand.
  - matheusmoreira 5y ago
    > several failed initiatives on the other hand to allow the user's code to perform syscalls only from inside of glibc (which spells doom to anything that doesn't use libc, e.g. Golang) because of security reasons I didn't know about these efforts. Please elaborate.
    - 5y ago

guerrilla 5y ago

The explanation for why it's useful is at the bottom of the page, but it wasn't necessary before glibc got behind and before Linux started adding things that glibc would probably never implement*. Generally, on UNIX-like systems, the C library isn't just the C library but also exposes the UNIX/POSIX API which is just a set of wrappers around calling into the kernel (done with software interrupt 0x80 on x86 Linux.) * Remember that glibc is part of the GNU project and is meant to be portable beyond Linux, especially to Hurd. It might not make sense for them to implement interfaces that wouldn't work or would require significant work to emulate on Hurd, the BSDs and Solaris.

Joker_vD 5y ago
Well, yeah, merging kernel calls into the runtime of one specifically favoured programming language is a silly idea, why not have the kernel calls in a separate, cleanly defined language-agnostic OS-specific library? "It just the way the things has come to work historically"?
- guerrilla 5y ago
  Well, first of all the UNIX/POSIX API is a C API, so that question doesn't make much sense. On top of that, there have been no competitive systems programming languages until recently (and certainly none that weren't written in C), so that wouldn't have been worth it. Also, it wasn't really an idea per se: C is just the native language and lingua franca of UNIX and that's just how UNIX developed organically.
  - GoblinSlayer 5y ago
    Why would it matter what system programming languages are there? Unix can't run other languages?
    - koprulusector 5y ago
      The kernel is written in C. Ok, why does that matter? Because its interfaces and architecture are tightly intertwined with C. Remember that C was partially developed to use in building Unix. So, kernel was written in C and therefore uses C data structures, pointers, and follows C conventions for using them (for example, kernel might return a struct of pointers). This applies to interfaces and structures, making syscalls (depending on which syscall and what is needed), checking return values or error codes. You don’t NEED this stuff exactly to run on Unix. What and how much you need, or even just convenience, are how it boils down. If you want to use eBPF to trace things from raw network packets, syscalls, file systems, you can do some in Python but you’ll need some degree of C. You want to run a Django or flask app in just Python? Good to go. [This article on Linux ABI](https://en.wikipedia.org/wiki/Application_binary_interface https://en.wikipedia.org/wiki/Application_binary_interface) might help explain better/in more detail. Sorry if this isn’t the best explanation, I’m on mobile and up too late.
      - GoblinSlayer 5y ago
        Design of unix was influenced more by PDP-7 than by C. What unix does, it does PDP-7 stuff, not C stuff. C itself isn't limited to POSIX.
    - johannes1234321 5y ago
      Unix can run other languages. But for answering your question deeper, we have to find an understanding what boundary we're talking about. The kernel syscall interface? Things like Linux's vdso? POSIX library (libc)? The more we go on in that sequence the more C we get into the interfaces and many of those APIs are essential for writing well-behaving non-trivial programs. The closer we stay to the kernel's syscall interface the less C you get, but still the syscall interface and C are related. Strings are often NULL-terminated. Arguemtns are passed (via registers) from left to right, not like in Pascal convention right to left. (First arg always in EBX, second in EXC etc.) A different language will need some interoperability with C conventions for being a good citizen on Unix. A Pascal operating system for instance would use Pascal strings (historically what tis now called ShortString) where length is encoded in the first byte.
- hnlmorg 5y ago
  Because every programming language will have their own opinions on how to link to external libraries and what those external libraries should look like (eg Java libraries are ZIP archives with a .jar extension, JIT languages will have their libraries as raw source code). But since C is the de facto standard for OS libraries, most languages end up supporting calls to C libraries. Thus C libraries do become the language-agnostic library.
  - GoblinSlayer 5y ago
    Most languages can call C libraries because they can call any libraries, not because C libraries are language agnostic.
    - couchand 5y ago
      That doesn't ring true to me. My FFI experience is somewhat limited (JS to C, JS to C++ via C wrappers, Rust to C, Rust to C++ via C wrappers) but from what I've seen you have to speak C to make FFI.
      - GoblinSlayer 5y ago
        Few languages truly speak full C (e.g. through clang plugin), they support ABI, but there are libraries like libpng and gtk with API disconnected from ABI, for them FFI doesn't resemble heavily macro encumbered C API.
    - matheusmoreira 5y ago
      Higher level languages such as C++ have extremely complex binary interfaces. Only C++ code dares to link against C++ code. Even then there are problems between different compilers and even different versions of the same compiler. New standard version comes out, it requires that standard classes have certain performance characteristics... Suddenly compilers are breaking binary compatibility in order to restablish conformance. So in order to have usable libraries people will wrap their C++ implementation in C interfaces. Rust faces the same problem. All those nice abstractions result in a complex ABI that will never touched by anything outside the Rust ecosystem. There is no escaping the extern "C". Some languages are virtualized. Java, C#, Ruby, Python, Javascript all have virtual machines. There's no way to interface directly with code running in those languages. We must interface with the virtual machine instead: import it into our process and ask it to call the functions we want to use. C binary interfaces are not language agnostic but they are quite simple. The result of compiling a C library is symbols pointing to functions with simple calling conventions. As a result, virtually any language can look up these symbols and call them.
- deleted 5y ago
  [deleted]

GoblinSlayer 5y ago

kernel32.dll is a little more language agnostic: it provides heap, threads, mutexes, environment variables and whatnot.

matheusmoreira 5y ago

Yes! The difference is on Linux a shared library is not actually required for kernel system calls. The binary interface is stable and will not change. https://github.com/torvalds/linux/blob/master/Documentation/ABI/stable/syscalls https://github.com/torvalds/linux/blob/master/Documentation/... On Windows and other operating systems, things such as system call numbers can and do change. There's no way to interface with the kernel without linking to vendor libraries. So on Linux you can ditch libc and write your own code. A Rust liblinux is definitely possible, I tested it. You can even have a compiler generate the system calls for you! It just needs to support the simple calling convention. No libraries necessary. I've seen a project that does just that: https://github.com/titzer/virgil https://github.com/titzer/virgil

Joker_vD 5y ago
I mean, I also can call functions from kernel32.dll on Windows without need of any libc, and the ABI is stable and will not change; and if/when MS decides to change the actual mechanism of switching into the kernel mode, or decide to move some stuff from the kernel into the user space ― they can, and I don't have to worry about that (and recompile my apps) because it's all abstracted away! "Fix it once in the shared library, and all its clients will benefit automatically". On Linux, however, we have, for example, glibc on one hand which claims that it doesn't have a goal of forwarding all of the Linux's syscall, and several failed initiatives on the other hand to allow the user's code to perform syscalls only from inside of glibc (which spells doom to anything that doesn't use libc, e.g. Golang) because of security reasons. And the history of still having to support INT 80h together with SYSCALL on x64 on the third hand.
- matheusmoreira 5y ago
  > several failed initiatives on the other hand to allow the user's code to perform syscalls only from inside of glibc (which spells doom to anything that doesn't use libc, e.g. Golang) because of security reasons I didn't know about these efforts. Please elaborate.
  - 5y ago