3 ms·

> I wish the barriers to using new architectures were lower. > For instance, suppose binaries were typically distributed in a platform-agnostic format, like LL

> I wish the barriers to using new architectures were lower. > For instance, suppose binaries were typically distributed in a platform-agnostic format, like LLVM intermediate representation or something equivalent. We're doing a pretty good job on portability these days already. Well-written Unix applications in C/C++ will compile happily for any old ISA and run just the same. Safe high-level languages like JavaScript, Java, and Safe Rust are pretty much ISA-independent by definition, it's 'just' a matter of getting the compilers and runtimes ported across. Adopting LLVM IR for portable distribution, probably isn't the way forward. I don't see that it adds much compared to compiling from source, and it's not what it's intended for. (LLVM may wish to change the representation in a subsequent major version, for instance.) For programs which are architecture-sensitive by nature, such as certain parts of kernels, there are no shortcuts. Or, rather, I'm confident the major OSs already use all the practical shortcuts they can think up. > When you run your program the first time, it's compiled to native code for your architecture and cached for later use. Source-based package management systems already give us something a lot like this. There are operating systems that take this approach, such as Inferno. [0] I like this HN comment on Inferno [1]: kernels are the wrong place for 'grand abstractions' of this sort. > I realize I've sort of just re-invented Javascript Don't be too harsh on yourself, JavaScript would be a terrible choice as a universal IR! > [[ to the bulk of your second paragraph ]] In the Free and Open Source world, we're already free to recompile the whole universe. The major distros do so as compiler technology improves. > Processors would be able to compete directly with each other without regard to vendor-lock-in. For most application-level code, we're already there. For example, your Java code will most likely run just as happily on one of Amazon's AArch64 instances as on an AMD64 machine. In the unlikely case you encounter a bug, well, that's pretty much always a risk, no matter which abstractions we use. [0] https://en.wikipedia.org/wiki/Inferno_(operating_system) https://en.wikipedia.org/wiki/Inferno_(operating_system) [1] https://news.ycombinator.com/item?id=9807777 https://news.ycombinator.com/item?id=9807777

elihu 6y ago
> Well-written Unix applications in C/C++ will compile happily for any old ISA and run just the same. Safe high-level languages like JavaScript, Java, and Safe Rust are pretty much ISA-independent by definition, it's 'just' a matter of getting the compilers and runtimes ported across. That sort of works, but duplicating the proper development environment in the end-user's computer would take a lot of space and would be complicated by the enormous variety of programming languages and environments. Linux distros manage this with a lot of effort. I'm imagining something like a universal intermediate representation that can be compiled quickly (because a lot of the early language-specific part of compilation will have already been done by whoever you get your packages from) and in a uniform way because there's a common intermediate representation format that all the compiled languages use. Universal binaries might also be acceptable for commercial, closed-source applications where source distribution would not.
- MaxBarraclough 6y ago
  > duplicating the proper development environment in the end-user's computer would take a lot of space Most distros offer precompiled binaries, there are relatively few that use source-based distribution and expect the user to have all the necessary compilers installed. > would be complicated by the enormous variety of programming languages and environments That problem isn't effectively addressed by a universal IR. You can't have a single IR that works well for all languages, precisely because of the variety of languages. > Linux distros manage this with a lot of effort. Hopefully that should improve if the trend toward languages like Safe Rust continues. C and C++ are infamously full of footguns. > I'm imagining something like a universal intermediate representation that can be compiled quickly (because a lot of the early language-specific part of compilation will have already been done by whoever you get your packages from) and in a uniform way because there's a common intermediate representation format that all the compiled languages use. Again this can't be done effectively. There are good technical reasons why Java, Haskell, and JavaScript, don't generally use LLVM as their backend. The differences between languages aren't just skin deep, they extend right through the compiler stack. To be more precise: it could be done, but there would be an unacceptable performance cost. After all, you could start distributing binaries for the SuperH SH-4, and just use emulation everywhere. The question is whether it could be done effectively. I mentioned before that LLVM IR is not intended to be used this way, although the Google Native Client project took LLVM and turned it into what you're suggesting. C and C++ are quite different from Java. The size of the int type varies between platforms, for instance. They also have a preprocessor which allows the programmer to conditionally compile platform-specific code, e.g. intrinsics, fragments of assembly code, or workarounds. The program might use system-specific macros that expand before compilation. Languages like Haskell are very different from the sorts of languages that LLVM is built for. Even Java prefers to use its own backend, with tight integration with its GC. There's also a package-management question, although this issue wouldn't be as significant. The C/C++ way is to have the build system (autotoools or CMake or whatever) detect what libraries are available on the system. If an optional library is missing, the C/C++ code is automatically adjusted by the build system, prior to compilation. It would be unusual to detect availability of libraries at runtime. This approach doesn't play nicely with a universal IR. This might not be an issue if the IR is treated as a surrogate for the native-code binary, but the IR wouldn't be a good surrogate for the source. The C/C++ philosophy is to accommodate platform variations, in contrast with the JVM approach of mandating compliance to a virtual machine. With the JVM approach you forbid the sorts of variations that C and C++ permit (everything from int_fast32_t varying between platforms, to hand-written SIMD assembly). Others have already mentioned WASM and Google Native Client, both of which are stable, but neither of which are going to become mainstream ways of distributing Unix application code. This topic has turned up on HN before, but frustratingly I wasn't able to find the thread. > Universal binaries might also be acceptable for commercial, closed-source applications where source distribution would not. True, but I think modern Unix OSs do a pretty good job on ABI stability. If they want portability without releasing source (something I don't think GNU/Linux should aim to accommodate, incidentally) they already have other options, like Java. JavaFX doesn't get much attention but it pretty much 'just works' for portable GUI applications. edit skissane has an interesting comment on ANDF, a solution I hadn't heard of before.
karteum 6y ago
> "Adopting LLVM IR for portable distribution, probably isn't the way forward. I don't see that it adds much compared to compiling from source, and it's not what it's intended for. (LLVM may wish to change the representation in a subsequent major version, for instance.)" Maybe, but PNaCl (unfortunately deprecated by Google) "defines a low-level stable portable intermediate representation (based on the IR used by the open-source LLVM compiler project) which is used as the wire format instead of x86 or ARM machine code" https://www.chromium.org/nativeclient/pnacl/introduction-to-portable-native-client https://www.chromium.org/nativeclient/pnacl/introduction-to-...
- MaxBarraclough 6y ago
  Sure. I'm not saying it's impossible to construct such an IR and get it to work, I'm saying I doubt it's the best way forward. See my other comment [0] where I mention Google Native Client. It would be a poor fit for certain languages, there may be performance penalties depending on target platform, it would preclude legitimate platform-specific code such as SIMD assembly, it would preclude platform-specific build-time customization, etc. The way toward painless portability is to move away from unsafe languages like C and C++, where you're never more than an expression away from undefined behaviour, and where programmers may be tempted to make silly mistakes like writing code sensitive to the endianness of the target architecture. [1] With C and C++, disciplined developers working carefully, can write portable code. With Safe Rust, code can be pretty close to 'portable by construction', like Java. If you feed Windows-style path strings to Linux, or vice versa, then things might go wrong, but for the most part you'll be on solid ground. [0] https://news.ycombinator.com/item?id=26398199 https://news.ycombinator.com/item?id=26398199 [1] https://commandcenter.blogspot.com/2012/04/byte-order-fallacy.html https://commandcenter.blogspot.com/2012/04/byte-order-fallac...

elihu 6y ago

> Well-written Unix applications in C/C++ will compile happily for any old ISA and run just the same. Safe high-level languages like JavaScript, Java, and Safe Rust are pretty much ISA-independent by definition, it's 'just' a matter of getting the compilers and runtimes ported across. That sort of works, but duplicating the proper development environment in the end-user's computer would take a lot of space and would be complicated by the enormous variety of programming languages and environments. Linux distros manage this with a lot of effort. I'm imagining something like a universal intermediate representation that can be compiled quickly (because a lot of the early language-specific part of compilation will have already been done by whoever you get your packages from) and in a uniform way because there's a common intermediate representation format that all the compiled languages use. Universal binaries might also be acceptable for commercial, closed-source applications where source distribution would not.

MaxBarraclough 6y ago
> duplicating the proper development environment in the end-user's computer would take a lot of space Most distros offer precompiled binaries, there are relatively few that use source-based distribution and expect the user to have all the necessary compilers installed. > would be complicated by the enormous variety of programming languages and environments That problem isn't effectively addressed by a universal IR. You can't have a single IR that works well for all languages, precisely because of the variety of languages. > Linux distros manage this with a lot of effort. Hopefully that should improve if the trend toward languages like Safe Rust continues. C and C++ are infamously full of footguns. > I'm imagining something like a universal intermediate representation that can be compiled quickly (because a lot of the early language-specific part of compilation will have already been done by whoever you get your packages from) and in a uniform way because there's a common intermediate representation format that all the compiled languages use. Again this can't be done effectively. There are good technical reasons why Java, Haskell, and JavaScript, don't generally use LLVM as their backend. The differences between languages aren't just skin deep, they extend right through the compiler stack. To be more precise: it could be done, but there would be an unacceptable performance cost. After all, you could start distributing binaries for the SuperH SH-4, and just use emulation everywhere. The question is whether it could be done effectively. I mentioned before that LLVM IR is not intended to be used this way, although the Google Native Client project took LLVM and turned it into what you're suggesting. C and C++ are quite different from Java. The size of the int type varies between platforms, for instance. They also have a preprocessor which allows the programmer to conditionally compile platform-specific code, e.g. intrinsics, fragments of assembly code, or workarounds. The program might use system-specific macros that expand before compilation. Languages like Haskell are very different from the sorts of languages that LLVM is built for. Even Java prefers to use its own backend, with tight integration with its GC. There's also a package-management question, although this issue wouldn't be as significant. The C/C++ way is to have the build system (autotoools or CMake or whatever) detect what libraries are available on the system. If an optional library is missing, the C/C++ code is automatically adjusted by the build system, prior to compilation. It would be unusual to detect availability of libraries at runtime. This approach doesn't play nicely with a universal IR. This might not be an issue if the IR is treated as a surrogate for the native-code binary, but the IR wouldn't be a good surrogate for the source. The C/C++ philosophy is to accommodate platform variations, in contrast with the JVM approach of mandating compliance to a virtual machine. With the JVM approach you forbid the sorts of variations that C and C++ permit (everything from int_fast32_t varying between platforms, to hand-written SIMD assembly). Others have already mentioned WASM and Google Native Client, both of which are stable, but neither of which are going to become mainstream ways of distributing Unix application code. This topic has turned up on HN before, but frustratingly I wasn't able to find the thread. > Universal binaries might also be acceptable for commercial, closed-source applications where source distribution would not. True, but I think modern Unix OSs do a pretty good job on ABI stability. If they want portability without releasing source (something I don't think GNU/Linux should aim to accommodate, incidentally) they already have other options, like Java. JavaFX doesn't get much attention but it pretty much 'just works' for portable GUI applications. edit skissane has an interesting comment on ANDF, a solution I hadn't heard of before.

karteum 6y ago

> "Adopting LLVM IR for portable distribution, probably isn't the way forward. I don't see that it adds much compared to compiling from source, and it's not what it's intended for. (LLVM may wish to change the representation in a subsequent major version, for instance.)" Maybe, but PNaCl (unfortunately deprecated by Google) "defines a low-level stable portable intermediate representation (based on the IR used by the open-source LLVM compiler project) which is used as the wire format instead of x86 or ARM machine code" https://www.chromium.org/nativeclient/pnacl/introduction-to-portable-native-client https://www.chromium.org/nativeclient/pnacl/introduction-to-...

MaxBarraclough 6y ago
Sure. I'm not saying it's impossible to construct such an IR and get it to work, I'm saying I doubt it's the best way forward. See my other comment [0] where I mention Google Native Client. It would be a poor fit for certain languages, there may be performance penalties depending on target platform, it would preclude legitimate platform-specific code such as SIMD assembly, it would preclude platform-specific build-time customization, etc. The way toward painless portability is to move away from unsafe languages like C and C++, where you're never more than an expression away from undefined behaviour, and where programmers may be tempted to make silly mistakes like writing code sensitive to the endianness of the target architecture. [1] With C and C++, disciplined developers working carefully, can write portable code. With Safe Rust, code can be pretty close to 'portable by construction', like Java. If you feed Windows-style path strings to Linux, or vice versa, then things might go wrong, but for the most part you'll be on solid ground. [0] https://news.ycombinator.com/item?id=26398199 https://news.ycombinator.com/item?id=26398199 [1] https://commandcenter.blogspot.com/2012/04/byte-order-fallacy.html https://commandcenter.blogspot.com/2012/04/byte-order-fallac...