3 ms·

You'd expect that but, no. That's why you cannot load glibc-linked binaries in a Musl distro. Edit: that's why the hacks like the original post is needed, as we

by okanat 2mo ago

You'd expect that but, no. That's why you cannot load glibc-linked binaries in a Musl distro. Edit: that's why the hacks like the original post is needed, as well. The ABI is strongly dependent on explicit libc implementation in current Linux systems. There is no libc independent ABI on Linux.

uecker 2mo ago
Sorry, can you be more specific. I do not understand what the problem is. If Musl does not implement support for the ABI, this would be a musl problem?
- okanat 2mo ago
  There is no libc independent ABI. ABI doesn't purely mean just calling conventions. When you compile libc, you also get a binary loader ld-linux.so with it. They are not two independent components of a system. Basically all .so files compiled with glibc require the ld-linux.so that's also generated by that glibc (or a later version, if they didn't break the binary compatibility). There are a lot of stuff that's executed by ld-linux.so and glibc that are not explicitly documented but they are absolutely necessary for your program to start and correctly initialize things like global variables or signal handling or loading other dynamic libraries. Some of that functionality sits in ld-linux.so and some of that in glibc. They have circular dependencies to each other. glibc expects ld-linux.so to put things in certain order but ld-linux.so also must load glibc first to have access to certain APIs. They are not part of System V ABI. They are not documented. Musl maybe can implement this but it is simply reverse engineering what glibc did and then playing a game of cat and mouse. There is no independent ABI standard.
  - uecker 1mo ago
    Sorry, again this too vague for me. What is the exact problem with atomics and thread_local in C that would make the ABI dependent on a specific version of glibc? I know the ABI is not just calling convention and e.g. for atomics may involve calling a function from libatomic. But from my understanding, this is all part of a standardized ABI that does not change and can be provided by different implementations. Then, what is the exact reason a library compiled against glibc must be loaded by a specific ld-linux? I could see that this is true for C++ perhaps, or when you use very special features, but I do not see this for C. I often compiled programs against one version of glibc and run it against a different version, so I know there is not a tight coupling. So please be specific in explaining in what scenarios this would break.
    - okanat 1mo ago
      Here read it from the lion's mouth: https://wiki.musl-libc.org/design-concepts https://wiki.musl-libc.org/design-concepts > Then, what is the exact reason a library compiled against glibc must be loaded by a specific ld-linux? I could see that this is true for C++ perhaps, or when you use very special features, but I do not see this for C. You still have functionality like `dlopen` with C or `pthreads` with C. ld-linux.so is the thing that prepares the stack frame, or the segment pointers (FS_BASE on x86_64). When you have your main loaded __libc_start_main_impl is called from glibc and if you disassemble it you'll see this line: call 0x7ffff7c28790 <_dl_audit_preinit@plt> Guess where _dl_audit_preinit lives? (gdb) info symbol _dl_audit_preinit _dl_audit_preinit in section .text of /lib64/ld-linux-x86-64.so.2 Moreover glibc has "magic" sections like .init_first. Only ld-linux.so that is compiled from glibc source knows how to handle that. https://elixir.bootlin.com/glibc/glibc-2.44.9000/source/csu/libc-start.c#L42 https://elixir.bootlin.com/glibc/glibc-2.44.9000/source/csu/... > I often compiled programs against one version of glibc and run it against a different version, so I know there is not a tight coupling. So please be specific in explaining in what scenarios this would break. It didn't break, since glibc hasn't broken backwards compatibility of ld-linux.so and glibc combination lately. Last time they broke it was in 2024 for a really specific subset: https://sourceware.org/pipermail/libc-alpha/2024-December/163146.html https://sourceware.org/pipermail/libc-alpha/2024-December/16... A breakage hasn't been observed doesn't mean that there is no tight-coupling between ld-linux.so and glibc that is compiled from glibc source. If you want a really specific scenario: - Write a very simple C file that contains a global (volatile if you want to stop the compiler optimizations) thread_local variable with C23 syntax - Compile a simple .so file on a GNU distro targetting glibc ABI with GCC (pass -std=c23) - start up a Musl distro (e.g. Alpine Docker image), copy that .so file in - Write a C program that dlopens the GNU .so file - Try accessing the thread_local variable you defined - Watch the world burn
      - uecker 1mo ago
        Thanks, I need to dig into this more. But I still not convinced there is a real issue here. Of course different infrastructure needs to exist for different feature. What exactly is the underlying issue with thread_local in your specific scenario? It seems Musl does not setup the the infrastructure needed for ABI-compliant thread-locaL data access?

by okanat 2mo ago

uecker 2mo ago

Sorry, can you be more specific. I do not understand what the problem is. If Musl does not implement support for the ABI, this would be a musl problem?

okanat 2mo ago
There is no libc independent ABI. ABI doesn't purely mean just calling conventions. When you compile libc, you also get a binary loader ld-linux.so with it. They are not two independent components of a system. Basically all .so files compiled with glibc require the ld-linux.so that's also generated by that glibc (or a later version, if they didn't break the binary compatibility). There are a lot of stuff that's executed by ld-linux.so and glibc that are not explicitly documented but they are absolutely necessary for your program to start and correctly initialize things like global variables or signal handling or loading other dynamic libraries. Some of that functionality sits in ld-linux.so and some of that in glibc. They have circular dependencies to each other. glibc expects ld-linux.so to put things in certain order but ld-linux.so also must load glibc first to have access to certain APIs. They are not part of System V ABI. They are not documented. Musl maybe can implement this but it is simply reverse engineering what glibc did and then playing a game of cat and mouse. There is no independent ABI standard.
- uecker 1mo ago
  Sorry, again this too vague for me. What is the exact problem with atomics and thread_local in C that would make the ABI dependent on a specific version of glibc? I know the ABI is not just calling convention and e.g. for atomics may involve calling a function from libatomic. But from my understanding, this is all part of a standardized ABI that does not change and can be provided by different implementations. Then, what is the exact reason a library compiled against glibc must be loaded by a specific ld-linux? I could see that this is true for C++ perhaps, or when you use very special features, but I do not see this for C. I often compiled programs against one version of glibc and run it against a different version, so I know there is not a tight coupling. So please be specific in explaining in what scenarios this would break.
  - okanat 1mo ago
    Here read it from the lion's mouth: https://wiki.musl-libc.org/design-concepts https://wiki.musl-libc.org/design-concepts > Then, what is the exact reason a library compiled against glibc must be loaded by a specific ld-linux? I could see that this is true for C++ perhaps, or when you use very special features, but I do not see this for C. You still have functionality like `dlopen` with C or `pthreads` with C. ld-linux.so is the thing that prepares the stack frame, or the segment pointers (FS_BASE on x86_64). When you have your main loaded __libc_start_main_impl is called from glibc and if you disassemble it you'll see this line: call 0x7ffff7c28790 <_dl_audit_preinit@plt> Guess where _dl_audit_preinit lives? (gdb) info symbol _dl_audit_preinit _dl_audit_preinit in section .text of /lib64/ld-linux-x86-64.so.2 Moreover glibc has "magic" sections like .init_first. Only ld-linux.so that is compiled from glibc source knows how to handle that. https://elixir.bootlin.com/glibc/glibc-2.44.9000/source/csu/libc-start.c#L42 https://elixir.bootlin.com/glibc/glibc-2.44.9000/source/csu/... > I often compiled programs against one version of glibc and run it against a different version, so I know there is not a tight coupling. So please be specific in explaining in what scenarios this would break. It didn't break, since glibc hasn't broken backwards compatibility of ld-linux.so and glibc combination lately. Last time they broke it was in 2024 for a really specific subset: https://sourceware.org/pipermail/libc-alpha/2024-December/163146.html https://sourceware.org/pipermail/libc-alpha/2024-December/16... A breakage hasn't been observed doesn't mean that there is no tight-coupling between ld-linux.so and glibc that is compiled from glibc source. If you want a really specific scenario: - Write a very simple C file that contains a global (volatile if you want to stop the compiler optimizations) thread_local variable with C23 syntax - Compile a simple .so file on a GNU distro targetting glibc ABI with GCC (pass -std=c23) - start up a Musl distro (e.g. Alpine Docker image), copy that .so file in - Write a C program that dlopens the GNU .so file - Try accessing the thread_local variable you defined - Watch the world burn
    - uecker 1mo ago
      Thanks, I need to dig into this more. But I still not convinced there is a real issue here. Of course different infrastructure needs to exist for different feature. What exactly is the underlying issue with thread_local in your specific scenario? It seems Musl does not setup the the infrastructure needed for ABI-compliant thread-locaL data access?