5 ms·

Something that has been bugging me for a while: Why is cross compiling so hard? After all, compilers are just C programs (or C++/Go/Rust/… programs). It's not l

Something that has been bugging me for a while: Why is cross compiling so hard? After all, compilers are just C programs (or C++/Go/Rust/… programs). It's not like a target implementation is using some magic instructions from the host to generate the assembly, or that it is harder to calculate some 32bit offsets for the compiled program, when the compiler is running on a 64bit host environment. Or is it about the helper programs / scripts around the core compilation process that are too often hard coded to read out the configuration from the host system? So basically there is no technical hurdle, just the social norm that target arch == host arch, and thus make procedures aren't sufficiently tested for cross compilation from the get go?

marmaduke 8y ago
the ease of using golang's cross compilation suggests it is historically the norm that target == host
ceidsness 8y ago
Yes, that's exactly it. A lot of source packages expect to be build on the same system they will run on.
TickleSteve 8y ago
the compilers themselves (e.g gcc) aren't the problem typically. It tends to be the dependencies (of the compiler and the libraries) that assume target==host.
- IshKebab 8y ago
  Erm, GCC is a huge problem for cross-compiling, because it's compilation target is determined at compile time. I mean at GCC compile time. You have to recompile GCC to target a different machine. Clang isn't broken like that, and for clang you are right that the issue is normally dependencies. Again GNU's libc is kind of broken. Cross compiling with Clang and Musl is not too bad.
  - phkahler 8y ago
    This IMHO is a huge problem for embedded development. I want to be able to just select what target I want and recompile a project. Even in small embedded systems you still need target specific headers and .bss code, but that's more of an IDE problem on top of the compiler not supporting all architectures out of the box. A C library that uses no hardware specific features should be able to recompile trivially for any supported target for example. If my chip vendor decides to replace the ARM core on a SoC with a RISC-V core but leave all peripherals and memory mappings the same, I want to simply change the target and rebuild.
    - microcolonel 8y ago
      > If my chip vendor decides to replace the ARM core on a SoC with a RISC-V core but leave all peripherals and memory mappings the same, I want to simply change the target and rebuild. I mean, I don't know what distro you're working on (and whether they package these toolchains conveniently), but on Arch you just change your configured target in your conf/build system, and make sure you have the other one installed. If you're using autotools or cmake, you may not even need to change the configure scripts. If you're doing embedded with memory mapped peripherals, surely you're building all your deps statically anyway.
    - pvarangot 8y ago
      If you are memory mapping stuff like that you most likely already have your own linker script or something from the vendor of arch1 and you can, with not much pain, adapt the linker script for arch2 to use the same base addresses and magic numbers. I mean I haven't personally done it but worked on projects were we were doing that and the headers for both archs were almost the same. If you don't have hardcoded addresses and are fully abstracted by a modern HAL and VM like on Linux then it's mostly a problem of the Linux distro or the OS vendor.
      - dbuder 8y ago
        You end up relying on the linker developer.
    - 8y ago
de_watcher 8y ago
The non-native languages like to keep their build systems in a form of a big mess. Python is probably still not cross-compilable, npm that depends on an old ssl, things like that.
ajross 8y ago
Cross compiling isn't hard, not really. But software building is hard. There are a lot of libraries and linkage magic needed to turn source files into something executable. And for a host system, all that work has been done for you. So if you want to have a cross toolchain for, say, RISC-V on x86, it's not enough to have the compiler and assembler, you need a RISC-V linker script (for your particular RISC-V target), a RISC-V build of the C library (again target-specific), the libgcc helper library, headers for all the relevant platform code (they seem simple, but in my experience headers are routinely the biggest mess). And unlike the host toolchain where there is an unambiguous "correct" answer for all that stuff, a cross environment might be asked to support all kinds of crazy alternatives. Your ARM gcc might be used to produce a binary to run on Fedora, or for a busybox/uclibc build on an ancient kernel, or to target Zephyr (my world)... It gets messy very fast. But it's not "hard". Putting it together for one system isn't any harder than it is for any other.
steveklabnik 8y ago
There's a few reasons, in my experience: Different toolchains support it differently. So for example, if you want to cross-compile with gcc and ld, you need to first compile gcc and ld for the target architecture. This is because you pick the host and target at compile time. So you get aarch64-elf-gcc instead of just plain gcc. This means that you either hope that someone else has already done this for you, or you have to build it yourself. It's not impossible but it is a pain. Contrast this with llvm; clang and lldb support multiple hosts and targets in one binary. You pass --target (or whatever) and you're good to go. This removes that setup step. The outside world. If you want to target an OS that's not your OS... you need the API for that OS. On Linuxes, that's the actual syscall interface, but on many other OSes, for example, Windows, the syscall interface is not stable. You must use the library the OS provides. This means that, if you're on Linux and you want to cross-compile to Windows, you need to get a copy of all the stuff that's in C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.10.24728\lib or whatever, because your program is going to need it. Is your project all in one language? If not, you'll need to do the above in multiple ways, maybe. For example, a pure Ruby program is easy to "cross compile", you just cross-compile the interpreter. (I don't know how hard that actually is these days, but it's an analogy, work with me here.) But if you use C extensions, you also need to have them cross-compiled. Use both Rust and C? You'll need the toolchains for both, and to coordinate them. (We've tried to generally make this very easy for Rust but there's still details where it's not.) Then, you need to make sure that the software understands that host != target. So for example, in Rust, we now have compile time function evaluation. The way that this works is, we include a full interpreter into the compiler, and it runs CTFE stuff with the interpreter set to the properties of the target, not the host. Not all languages do this kind of thing. There's tons of other ways this can possibly go wrong too, as it's not the case that many people expect. For example, maybe the software uses linux-specific stuff. You can't just automatically cross-compile it then. But it's not like anything tells you you're using os-specific stuff, and so if you never attempt to cross-compile, then you may not even realize.
- skissane 8y ago
  > Contrast this with llvm; clang and lldb support multiple hosts and targets in one binary. I wonder, could the GNU toolchain devs be convinced to add support for multiple hosts/targets in one binary? Do they have a philosophical objection to it? Or is it simply that it is a lot of work and nobody has done it? It would make cross-compiling with the GNU toolchain a lot easier.
rkeene2 8y ago
I cross-compile an entire Linux distribution (even if you're compiling on Linux/x86_64 FOR Linux/x86_64, it's still going to get cross-compiled since it's a different platform). This is around 300 external/upstream packages and 100 local packages. The vast majority of software that uses GNU autoconf has little to no problems. Some software that uses CMake has some small problems. Building the cross-compiling toolchain is relatively easy, compared to dealing with the problems in software I ran into a bug in bash [0], for example, that you would really only hit if you were cross-compiling it (there are other ways, because it wasn't related to being cross-compiled just features that were disabled automatically when cross-compiling). It took a while to convince people that this bug existed because it wasn't obvious that it was caused by being cross-compiled. The real problems come from "modern" package managers as well as scripting languages in general. Scripting languages and their extension mechanism are, in general, absolutely ignorant of cross-compiling and the idea that you might be building an extension to the language for a different platform than the running interpreter is completely foreign to most. Ruby fails especially hard here -- it looks ONLY at the running interpreter to determine how to build the extension. Python is a bit better because, but still rigid in versions. Perl's cross-compilation story requires having access to a system to SSH into (though there are alternative autoconf-based build systems that make this sane -- I don't know if they work for extensions, I just abandoned the idea of including Perl based on how poor their build system was). Tcl was the best, since it's extension system (TEA) is autoconf-based. LuaJIT can't be cross-compiled from a 32-bit system for a 64-bit system [1] since it, at build-time, tries to do some stuff and fails at it. [0] https://lists.gnu.org/archive/html/bug-bash/2015-06/msg00042.html https://lists.gnu.org/archive/html/bug-bash/2015-06/msg00042... [1] http://luajit.org/install.html#cross http://luajit.org/install.html#cross
Thaxll 8y ago
Go compiler is built in Go, big difference ( all of Go is built in Go). Cross-compiling in Go is the easiest of any languages as far as I know. Ex: GOOS=windows go build . GOOS=linux GOARCH=arm go build . And you can run that from any os / arch, I can build a single Windows binary from my Raspberry Pi with the above example. https://golang.org/doc/install/source#environment https://golang.org/doc/install/source#environment
- krylon 8y ago
  As opposed to C/C++ compilers being built in C/C++? EDIT: Try to build NetBSD for some esoteric platform from the comfort of your desktop machine. Remember to pick up your jaw from the floow when you're done. ;-) My point is, it does not have to be hard.
  - steveklabnik 8y ago
    In my understanding, Go re-implements the system's interface, rather than using libc (or the equivalent), even on platforms where the syscall interface isn't considered stable. This means cross-compiling is really easy, but it can also mean breakage when the unstable interface changes. A tradeoff like anything else.
  - Thaxll 8y ago
    Rust compiler is LLVM afaik so C++, same for the JVM / C#. Having your entire runtime / tools in the same language make it easier.
    - steveklabnik 8y ago
      This is true for Rust, however, given that LLVM is the project doing the codegen, it's a little bit special; cross-compiling it to the platforms it supports is a bit easier than a random C++ library, since it's the one with the support in the first place. Rust projects make much more use of C libraries than Go projects do, though, so for things other than the Rust compiler, this is still a good point.
- jhawk28 8y ago

marmaduke 8y ago

the ease of using golang's cross compilation suggests it is historically the norm that target == host

ceidsness 8y ago

Yes, that's exactly it. A lot of source packages expect to be build on the same system they will run on.

TickleSteve 8y ago

the compilers themselves (e.g gcc) aren't the problem typically. It tends to be the dependencies (of the compiler and the libraries) that assume target==host.

IshKebab 8y ago
Erm, GCC is a huge problem for cross-compiling, because it's compilation target is determined at compile time. I mean at GCC compile time. You have to recompile GCC to target a different machine. Clang isn't broken like that, and for clang you are right that the issue is normally dependencies. Again GNU's libc is kind of broken. Cross compiling with Clang and Musl is not too bad.
- phkahler 8y ago
  This IMHO is a huge problem for embedded development. I want to be able to just select what target I want and recompile a project. Even in small embedded systems you still need target specific headers and .bss code, but that's more of an IDE problem on top of the compiler not supporting all architectures out of the box. A C library that uses no hardware specific features should be able to recompile trivially for any supported target for example. If my chip vendor decides to replace the ARM core on a SoC with a RISC-V core but leave all peripherals and memory mappings the same, I want to simply change the target and rebuild.
  - microcolonel 8y ago
    > If my chip vendor decides to replace the ARM core on a SoC with a RISC-V core but leave all peripherals and memory mappings the same, I want to simply change the target and rebuild. I mean, I don't know what distro you're working on (and whether they package these toolchains conveniently), but on Arch you just change your configured target in your conf/build system, and make sure you have the other one installed. If you're using autotools or cmake, you may not even need to change the configure scripts. If you're doing embedded with memory mapped peripherals, surely you're building all your deps statically anyway.
  - pvarangot 8y ago
    If you are memory mapping stuff like that you most likely already have your own linker script or something from the vendor of arch1 and you can, with not much pain, adapt the linker script for arch2 to use the same base addresses and magic numbers. I mean I haven't personally done it but worked on projects were we were doing that and the headers for both archs were almost the same. If you don't have hardcoded addresses and are fully abstracted by a modern HAL and VM like on Linux then it's mostly a problem of the Linux distro or the OS vendor.
    - dbuder 8y ago
      You end up relying on the linker developer.
  - 8y ago

de_watcher 8y ago

The non-native languages like to keep their build systems in a form of a big mess. Python is probably still not cross-compilable, npm that depends on an old ssl, things like that.

ajross 8y ago

Cross compiling isn't hard, not really. But software building is hard. There are a lot of libraries and linkage magic needed to turn source files into something executable. And for a host system, all that work has been done for you. So if you want to have a cross toolchain for, say, RISC-V on x86, it's not enough to have the compiler and assembler, you need a RISC-V linker script (for your particular RISC-V target), a RISC-V build of the C library (again target-specific), the libgcc helper library, headers for all the relevant platform code (they seem simple, but in my experience headers are routinely the biggest mess). And unlike the host toolchain where there is an unambiguous "correct" answer for all that stuff, a cross environment might be asked to support all kinds of crazy alternatives. Your ARM gcc might be used to produce a binary to run on Fedora, or for a busybox/uclibc build on an ancient kernel, or to target Zephyr (my world)... It gets messy very fast. But it's not "hard". Putting it together for one system isn't any harder than it is for any other.

steveklabnik 8y ago

There's a few reasons, in my experience: Different toolchains support it differently. So for example, if you want to cross-compile with gcc and ld, you need to first compile gcc and ld for the target architecture. This is because you pick the host and target at compile time. So you get aarch64-elf-gcc instead of just plain gcc. This means that you either hope that someone else has already done this for you, or you have to build it yourself. It's not impossible but it is a pain. Contrast this with llvm; clang and lldb support multiple hosts and targets in one binary. You pass --target (or whatever) and you're good to go. This removes that setup step. The outside world. If you want to target an OS that's not your OS... you need the API for that OS. On Linuxes, that's the actual syscall interface, but on many other OSes, for example, Windows, the syscall interface is not stable. You must use the library the OS provides. This means that, if you're on Linux and you want to cross-compile to Windows, you need to get a copy of all the stuff that's in C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.10.24728\lib or whatever, because your program is going to need it. Is your project all in one language? If not, you'll need to do the above in multiple ways, maybe. For example, a pure Ruby program is easy to "cross compile", you just cross-compile the interpreter. (I don't know how hard that actually is these days, but it's an analogy, work with me here.) But if you use C extensions, you also need to have them cross-compiled. Use both Rust and C? You'll need the toolchains for both, and to coordinate them. (We've tried to generally make this very easy for Rust but there's still details where it's not.) Then, you need to make sure that the software understands that host != target. So for example, in Rust, we now have compile time function evaluation. The way that this works is, we include a full interpreter into the compiler, and it runs CTFE stuff with the interpreter set to the properties of the target, not the host. Not all languages do this kind of thing. There's tons of other ways this can possibly go wrong too, as it's not the case that many people expect. For example, maybe the software uses linux-specific stuff. You can't just automatically cross-compile it then. But it's not like anything tells you you're using os-specific stuff, and so if you never attempt to cross-compile, then you may not even realize.

skissane 8y ago
> Contrast this with llvm; clang and lldb support multiple hosts and targets in one binary. I wonder, could the GNU toolchain devs be convinced to add support for multiple hosts/targets in one binary? Do they have a philosophical objection to it? Or is it simply that it is a lot of work and nobody has done it? It would make cross-compiling with the GNU toolchain a lot easier.

rkeene2 8y ago

I cross-compile an entire Linux distribution (even if you're compiling on Linux/x86_64 FOR Linux/x86_64, it's still going to get cross-compiled since it's a different platform). This is around 300 external/upstream packages and 100 local packages. The vast majority of software that uses GNU autoconf has little to no problems. Some software that uses CMake has some small problems. Building the cross-compiling toolchain is relatively easy, compared to dealing with the problems in software I ran into a bug in bash [0], for example, that you would really only hit if you were cross-compiling it (there are other ways, because it wasn't related to being cross-compiled just features that were disabled automatically when cross-compiling). It took a while to convince people that this bug existed because it wasn't obvious that it was caused by being cross-compiled. The real problems come from "modern" package managers as well as scripting languages in general. Scripting languages and their extension mechanism are, in general, absolutely ignorant of cross-compiling and the idea that you might be building an extension to the language for a different platform than the running interpreter is completely foreign to most. Ruby fails especially hard here -- it looks ONLY at the running interpreter to determine how to build the extension. Python is a bit better because, but still rigid in versions. Perl's cross-compilation story requires having access to a system to SSH into (though there are alternative autoconf-based build systems that make this sane -- I don't know if they work for extensions, I just abandoned the idea of including Perl based on how poor their build system was). Tcl was the best, since it's extension system (TEA) is autoconf-based. LuaJIT can't be cross-compiled from a 32-bit system for a 64-bit system [1] since it, at build-time, tries to do some stuff and fails at it. [0] https://lists.gnu.org/archive/html/bug-bash/2015-06/msg00042.html https://lists.gnu.org/archive/html/bug-bash/2015-06/msg00042... [1] http://luajit.org/install.html#cross http://luajit.org/install.html#cross

Thaxll 8y ago

Go compiler is built in Go, big difference ( all of Go is built in Go). Cross-compiling in Go is the easiest of any languages as far as I know. Ex: GOOS=windows go build . GOOS=linux GOARCH=arm go build . And you can run that from any os / arch, I can build a single Windows binary from my Raspberry Pi with the above example. https://golang.org/doc/install/source#environment https://golang.org/doc/install/source#environment

krylon 8y ago
As opposed to C/C++ compilers being built in C/C++? EDIT: Try to build NetBSD for some esoteric platform from the comfort of your desktop machine. Remember to pick up your jaw from the floow when you're done. ;-) My point is, it does not have to be hard.
- steveklabnik 8y ago
  In my understanding, Go re-implements the system's interface, rather than using libc (or the equivalent), even on platforms where the syscall interface isn't considered stable. This means cross-compiling is really easy, but it can also mean breakage when the unstable interface changes. A tradeoff like anything else.
- Thaxll 8y ago
  Rust compiler is LLVM afaik so C++, same for the JVM / C#. Having your entire runtime / tools in the same language make it easier.
  - steveklabnik 8y ago
    This is true for Rust, however, given that LLVM is the project doing the codegen, it's a little bit special; cross-compiling it to the platforms it supports is a bit easier than a random C++ library, since it's the one with the support in the first place. Rust projects make much more use of C libraries than Go projects do, though, so for things other than the Rust compiler, this is still a good point.
jhawk28 8y ago