3 ms·

That's good! But I wonder why it wasn't then enabled for that configuration or why the override wasn't "global" enough and the default allocator was still partl

• tialaramex 2mo ago
  Rust doesn't get to override musl internals. Ripgrep uses opendir, a POSIX library feature implemented in musl to look at er, directories. The stack trace suggests we blew up when Rust's std::sys::fs::unix::readdir internal detail called opendir, and it in turn allocated. On Linux it would be possible for ripgrep to talk directly to the kernel via documented system calls without libc, but that wouldn't work on any other popular OS.
  • inigyou 2mo ago
    I assume it already doesn't work on Windows. At some point you have to define your compatibility boundary. And high performance often coincides with mediocre compatibility.
    • galangalalgol 2mo ago
      A good example is go. On linux you can use a from scratch image fairly easily because it only uses syscalls. But for windows or mac the moving target wasn't maintainable so they link against shared objects. Linus enforcing the don't break userspace rule is what made that possible. That definitely has tradeoffs. At some point relibc or something similar will allow the same (stably) for rust. But using posix as that compatibility boundary gets you a much larger set of OS and only occasionally has a performance penalty. Often, a posix api tuned to the kernel is more performant. Musl is an exception precisely because it makes an openbsd-esque trade of performance for simple small attack surface.
      • inigyou 2mo ago
        The Linux kernel defines syscalls as its stable ABI (note: there are also non-kernel ABIs on Linux, such as Wayland) while Windows defines the DLL calls as its stable ABI. One isn't better than the other. Linux's approach allows binaries to be fully statically linked, which is a more predictable environment for binaries, but Windows's approach composes better, as every process loads DLLs and this allows for things like graphics drivers and COM to work more reliably. As things stand on Linux you can't use the GPU in a portable statically linked app, because the kernel doesn't define the semantics of dynamic linking.
        • fc417fc802 2mo ago
          TBF the GPU thing makes a fair amount of sense if you dig into it. It arguably falls entirely outside the kernel's domain of responsibility. However given how fundamental GPU acceleration is (as well as various other pieces of dedicated hardware in various scenarios) it would be nice if the kernel defined some basic portable semantics for linking with important drivers. These could exist independently of libc and the rest of userspace purely as an optional fallback. ... or we could all just include a glibc compatible dlopen routine in our statically linked binaries instead of worrying about pedantic hypotheticals.
          • inigyou 2mo ago
            Yeah we could all just use glibc. You see the problem with that though right?
            • fc417fc802 2mo ago
              That isn't at all what I said. I'm pointing out that in reality there aren't 20 competing libc implementations each with a unique dynamic linking scheme used by the userspace of 20 corresponding linux distributions. Instead we have ... glibc. So if the kernel actually specified a common portable linking scheme to fall back on as a means to interface with critical drivers it would presumably adopt the glibc way of doing things for that purpose. Also I did not say "just use glibc" but rather suggested to just include a secondary glibc compatible dlopen routine. Exactly as you would have to do if the kernel bothered to specify.
      • dataflow 2mo ago
        FYI, Windows's approach is better in some respects, like letting you avoid syscall overhead in some cases.
        • inigyou 2mo ago
          Notably, Linux has this too, but it's a special mechanism (vDSO) while on Windows it's just a normal function call with no difference from any other function.
          • dataflow 2mo ago
            Or in other words, they agreed the Windows approach was better, and tried to retrofit it (;
    • ChrisSD 2mo ago
      You can call into the kernel just fine in Windows. The fact you use a function call wrapper instead of a raw syscall is not really relevant. Most Windows NT kernel functions take a buffer to use rather than allocating their own memory. Even many Win32 functions don't allocate (although there you have to be much more careful).
  • VorpalWay 2mo ago
    > Rust doesn't get to override musl internals. Is malloc not a weak symbol in musl? I would expect it to be overridable like it is with glibc. Or does ripgrep only override Rust's global allocator?
    • masklinn 2mo ago
      > Or does ripgrep only override Rust's global allocator? That one, as you can see in the code linked above, it only does rust-level overriding as that was where musl's allocator was found to impact ripgrep (https://github.com/BurntSushi/ripgrep/commit/03bf37ff4a29361c47843369f7d3dc5689b8fdac https://github.com/BurntSushi/ripgrep/commit/03bf37ff4a29361...)
    • Orphis 2mo ago
      Last time I checked (a while ago), it wasn't. You needed to compile musl and exclude a few files from the archive to then provide your own allocator. Depending on the toolchain, I'm assuming you could use some tricks or hacks to make it work better, but the tooling I used (Bazel) did that automatically and I haven't bothered to look at the internals yet .