3 ms·

Modularizing (and versioning each part independently) libc would go a long way. There is also a need to separate the stuff needed for system integration with wh

by yxhuvud 2mo ago

Modularizing (and versioning each part independently) libc would go a long way. There is also a need to separate the stuff needed for system integration with what is necessary for users of the C language to actually do stuff.

matheusmoreira 2mo ago
> Modularizing Can't be done. The libc is legacy, it can't be changed without breaking everything. It's also mandatory on every operating system other than Linux. A change in paradigm is necessary. Freestanding C, not hosted C. This completely gets rid of the libc and is a surprisingly clean language. Linux only, because it's the only kernel with a stable binary interface. Every other OS forces a C runtime. I once worked on a liblinux project that embodied this... Stopped because Linux itself has a nolibc thing in the kernel tree and I didn't want to compete with it. Now I'm working on the Rust version. > what is necessary for users of the C language to actually do stuff Surprisingly little. I wrote an entire lisp interpreter in freestanding C with Linux system calls. It managed to survive for a rather long time without any memory allocation at all. The system layer is refreshingly tiny. It consists of a memory allocator and extremely basic functions like memmove and strlen. I successfully got rid of total nonsense like thread local errno, locales, implicit buffering, cached global state, possibly more. All that stuff is gone! Exactly one global survived: the stack canary generated by GCC and clang. Every other symbol in the ELF is controlled by me. Wasn't able to get rid of the NUL terminator. Linux itself needs it. To get rid of that little billion dollar mistake requires an entirely new kernel with zero UNIX/POSIX influence. I had to make my peace with that one. All my buffers maintain an extra NUL byte at the end.
- pg83 2mo ago
  > A change in paradigm is necessary. Freestanding C, not hosted C. This completely gets rid of the libc and is a surprisingly clean language. Linux only, because it's the only kernel with a stable binary interface. Every other OS forces a C runtime. Great choice for small programs, but what if I want hardware accelerated 3d?
  - Splizard 2mo ago
    exactly
  - matheusmoreira 2mo ago
    Yeah, that's the annoying part. Been wondering about this for years, and graphics support was among the first issues raised on the lone lisp GitHub repository. At this point I've even started exploring the mesa codebase, made some patches but didn't submit them yet due to the AI stigma. With Linux system calls alone it should be possible to set up kernel mode setting without depending on any toolkit at all. This should be enough to get a framebuffer for software rendering. For hardware acceleration though, one must give this graphics context to an OpenGL ES implementation. That's where it gets ugly. There is no way to divorce that from the libc short of literally rewriting it. Maybe Vulkan will enable it? I can't say for sure at my current knowledge level.
    - pjmlp 2mo ago
      Vulkan is designed to only load core statically and the whole extension spaghetti dynamically, using driver entry points. https://vulkan.lunarg.com/doc/view/latest/mac/LoaderInterfaceArchitecture.html https://vulkan.lunarg.com/doc/view/latest/mac/LoaderInterfac...
  - inigyou 2mo ago
    Modern 3D APIs are built around passing around a few buffers per frame and long command queues. Should be doable with IPC.
- dwattttt 2mo ago
  > Can't be done. The libc is legacy, it can't be changed without breaking everything. It's also mandatory on every operating system other than Linux. Windows explicitly does not want you to link the system libc. You are expected to bring your own, and doing so means your process has multiple libc's loaded into its address space. And if you choose to build a binary that doesn't need a libc, you won't be bringing one.
  - matheusmoreira 2mo ago
    > You are expected to bring your own, and doing so means your process has multiple libc's loaded into its address space. That only massively compounds the problem. > And if you choose to build a binary that doesn't need a libc, you won't be bringing one. NT system calls are not stable. You still need to link against ntdll.dll at the very least, like a forced Linux vDSO.
    - dwattttt 2mo ago
      > That only massively compounds the problem. The Windows ecosystem, that manages to deliver built binaries easily & widely, regardless of whether the author has a 1 year old OS or a 15 year old OS, suggests that it's not as big a problem as you believe.
      - matheusmoreira 2mo ago
        It's not a problem in the same way that things like snap or flatpak aren't problems. It works but it bloats things up considerably and makes you wonder where it all went so wrong. I mean, dozens of slightly incompatible runtimes inside a single process?
        dwattttt 2mo ago
        Those incompatible runtimes are separated by a linker that doesn't resolve all symbols globally, but rather scoped to the shared object they're expected from. They all coexist happily, and if you're so inclined you could resolve the same symbol from each, if you had reason to do so.
- torginus 2mo ago
  >A change in paradigm is necessary. Freestanding C, not hosted C. This completely gets rid of the libc and is a surprisingly clean language. Linux only, because it's the only kernel with a stable binary interface. Every other OS forces a C runtime. I'm sure those OSes make efforts to make said runtime binary compatible between executables.
  - matheusmoreira 2mo ago
    Yes, but the whole idea is to delete the runtime binary and talk to the kernel directly. That's when you run into the Darth Vader of binary interfaces. > I have altered the ABI. Pray I do not alter it further. -- De Raadt
    - torginus 2mo ago
      I am not sure having a stable kernel ABI/API is really a goal in practice or intent in all cases. GPU drivers are a perfect example - GPUs are hellishly complex, and their interfaces are super high-level - OpenGL/Vulkan is not something that should be implemented by the driver. I think a general practice that is followed with GPU drivers is to move as much of the complex and non-privileged stuff out of the kernel, and only keep modesetting/power management/memory and control stuff in. That reduces massively the amount of code that needs to run in Ring 0, but in exchange, the interface between the user and kernel blobs is non-stable. Programs talk to the userspace blob and there's no guarantee of compatibility between different kernel/userspace driver versions. This is something even Linus accepts, with the only contention that the userspace blob needs to be open source as well. They're not maintaining something in tree, that they have no way of testing without proprietary external deps.
- nextaccountic 2mo ago
  > A change in paradigm is necessary. Freestanding C, not hosted C. Or.. any other programming language. A lot of systems programming is done in Rust nowadays.
  - matheusmoreira 2mo ago
    Yeah. I made an entire lisp for this, and I've recently revived my Rust liblinux crate too. I've been documenting the system calls as I add them. Hopefully one day it'll rival man7 as the Linux system call reference.
- okanat 2mo ago
  > Can't be done. The libc is legacy, it can't be changed without breaking everything... Every other OS forces a C runtime. Windows never had this problem. If you really want it, you can actually skip CRT completely on Windows. Only thing you need is the initialization code that's statically linked anyway: https://nullprogram.com/blog/2023/02/15/ https://nullprogram.com/blog/2023/02/15/ . You can still make Win32 calls with it. CRT is there as a convenience. And since Linux distros are compiling everything from source anyway, this makes Linux a prime candidate for complete libc and complete ABI replacement. Thanks to stable kernel ABI, we can also use the same kernel even with Docker etc! Musl did it. Why can a completely modularized libc not? Will it break a lot of programs in the userspace? YES! Hyrum's Law exists. Will it be rejected by a significant part of user base? YES! They rejected systemd as well. Will it take one or more decades to have something working? HELL YES! This is Linux we're talking about. Everybody has seen how Wayland played out. However, maybe if Valve has enough motivation to fix Linux binary problem, we maybe can get finally a binary-compatible distro in a decade or half a decade.

by yxhuvud 2mo ago

matheusmoreira 2mo ago

> Modularizing Can't be done. The libc is legacy, it can't be changed without breaking everything. It's also mandatory on every operating system other than Linux. A change in paradigm is necessary. Freestanding C, not hosted C. This completely gets rid of the libc and is a surprisingly clean language. Linux only, because it's the only kernel with a stable binary interface. Every other OS forces a C runtime. I once worked on a liblinux project that embodied this... Stopped because Linux itself has a nolibc thing in the kernel tree and I didn't want to compete with it. Now I'm working on the Rust version. > what is necessary for users of the C language to actually do stuff Surprisingly little. I wrote an entire lisp interpreter in freestanding C with Linux system calls. It managed to survive for a rather long time without any memory allocation at all. The system layer is refreshingly tiny. It consists of a memory allocator and extremely basic functions like memmove and strlen. I successfully got rid of total nonsense like thread local errno, locales, implicit buffering, cached global state, possibly more. All that stuff is gone! Exactly one global survived: the stack canary generated by GCC and clang. Every other symbol in the ELF is controlled by me. Wasn't able to get rid of the NUL terminator. Linux itself needs it. To get rid of that little billion dollar mistake requires an entirely new kernel with zero UNIX/POSIX influence. I had to make my peace with that one. All my buffers maintain an extra NUL byte at the end.

pg83 2mo ago
> A change in paradigm is necessary. Freestanding C, not hosted C. This completely gets rid of the libc and is a surprisingly clean language. Linux only, because it's the only kernel with a stable binary interface. Every other OS forces a C runtime. Great choice for small programs, but what if I want hardware accelerated 3d?
- Splizard 2mo ago
  exactly
- matheusmoreira 2mo ago
  Yeah, that's the annoying part. Been wondering about this for years, and graphics support was among the first issues raised on the lone lisp GitHub repository. At this point I've even started exploring the mesa codebase, made some patches but didn't submit them yet due to the AI stigma. With Linux system calls alone it should be possible to set up kernel mode setting without depending on any toolkit at all. This should be enough to get a framebuffer for software rendering. For hardware acceleration though, one must give this graphics context to an OpenGL ES implementation. That's where it gets ugly. There is no way to divorce that from the libc short of literally rewriting it. Maybe Vulkan will enable it? I can't say for sure at my current knowledge level.
  - pjmlp 2mo ago
    Vulkan is designed to only load core statically and the whole extension spaghetti dynamically, using driver entry points. https://vulkan.lunarg.com/doc/view/latest/mac/LoaderInterfaceArchitecture.html https://vulkan.lunarg.com/doc/view/latest/mac/LoaderInterfac...
- inigyou 2mo ago
  Modern 3D APIs are built around passing around a few buffers per frame and long command queues. Should be doable with IPC.
dwattttt 2mo ago
> Can't be done. The libc is legacy, it can't be changed without breaking everything. It's also mandatory on every operating system other than Linux. Windows explicitly does not want you to link the system libc. You are expected to bring your own, and doing so means your process has multiple libc's loaded into its address space. And if you choose to build a binary that doesn't need a libc, you won't be bringing one.
- matheusmoreira 2mo ago
  > You are expected to bring your own, and doing so means your process has multiple libc's loaded into its address space. That only massively compounds the problem. > And if you choose to build a binary that doesn't need a libc, you won't be bringing one. NT system calls are not stable. You still need to link against ntdll.dll at the very least, like a forced Linux vDSO.
  - dwattttt 2mo ago
    > That only massively compounds the problem. The Windows ecosystem, that manages to deliver built binaries easily & widely, regardless of whether the author has a 1 year old OS or a 15 year old OS, suggests that it's not as big a problem as you believe.
    - matheusmoreira 2mo ago
      It's not a problem in the same way that things like snap or flatpak aren't problems. It works but it bloats things up considerably and makes you wonder where it all went so wrong. I mean, dozens of slightly incompatible runtimes inside a single process?
      - dwattttt 2mo ago
        Those incompatible runtimes are separated by a linker that doesn't resolve all symbols globally, but rather scoped to the shared object they're expected from. They all coexist happily, and if you're so inclined you could resolve the same symbol from each, if you had reason to do so.
torginus 2mo ago
>A change in paradigm is necessary. Freestanding C, not hosted C. This completely gets rid of the libc and is a surprisingly clean language. Linux only, because it's the only kernel with a stable binary interface. Every other OS forces a C runtime. I'm sure those OSes make efforts to make said runtime binary compatible between executables.
- matheusmoreira 2mo ago
  Yes, but the whole idea is to delete the runtime binary and talk to the kernel directly. That's when you run into the Darth Vader of binary interfaces. > I have altered the ABI. Pray I do not alter it further. -- De Raadt
  - torginus 2mo ago
    I am not sure having a stable kernel ABI/API is really a goal in practice or intent in all cases. GPU drivers are a perfect example - GPUs are hellishly complex, and their interfaces are super high-level - OpenGL/Vulkan is not something that should be implemented by the driver. I think a general practice that is followed with GPU drivers is to move as much of the complex and non-privileged stuff out of the kernel, and only keep modesetting/power management/memory and control stuff in. That reduces massively the amount of code that needs to run in Ring 0, but in exchange, the interface between the user and kernel blobs is non-stable. Programs talk to the userspace blob and there's no guarantee of compatibility between different kernel/userspace driver versions. This is something even Linus accepts, with the only contention that the userspace blob needs to be open source as well. They're not maintaining something in tree, that they have no way of testing without proprietary external deps.
nextaccountic 2mo ago
> A change in paradigm is necessary. Freestanding C, not hosted C. Or.. any other programming language. A lot of systems programming is done in Rust nowadays.
- matheusmoreira 2mo ago
  Yeah. I made an entire lisp for this, and I've recently revived my Rust liblinux crate too. I've been documenting the system calls as I add them. Hopefully one day it'll rival man7 as the Linux system call reference.
okanat 2mo ago
> Can't be done. The libc is legacy, it can't be changed without breaking everything... Every other OS forces a C runtime. Windows never had this problem. If you really want it, you can actually skip CRT completely on Windows. Only thing you need is the initialization code that's statically linked anyway: https://nullprogram.com/blog/2023/02/15/ https://nullprogram.com/blog/2023/02/15/ . You can still make Win32 calls with it. CRT is there as a convenience. And since Linux distros are compiling everything from source anyway, this makes Linux a prime candidate for complete libc and complete ABI replacement. Thanks to stable kernel ABI, we can also use the same kernel even with Docker etc! Musl did it. Why can a completely modularized libc not? Will it break a lot of programs in the userspace? YES! Hyrum's Law exists. Will it be rejected by a significant part of user base? YES! They rejected systemd as well. Will it take one or more decades to have something working? HELL YES! This is Linux we're talking about. Everybody has seen how Wayland played out. However, maybe if Valve has enough motivation to fix Linux binary problem, we maybe can get finally a binary-compatible distro in a decade or half a decade.