11 ms·
Patching GCC to build Actually Portable Executables
- 1vuio0pswjnm7 3y ago"But I've spent a good chunk of time removing obvious counterexamples, and a lot of popular software builds seamlessly." Publishing a list of software that will successfully compile might be helpful. Perhaps this already exists.
- ahgamut 3y agoYou can build software like vim, emacs, ninja, bash, git, gcc etc -- here's a list of software I got to build with this technique: https://github.com/ahgamut/superconfigure https://github.com/ahgamut/superconfigure The superconfigure script is just a wrapper around the usual configure script used to build your software, supplying flags like --enable-static.
- KerrAvon 3y agoThe switch-to-if thing is pretty inelegant. It seems like you could fix this by mapping to a platform-neutral set of constants, so you could dereference the runtime symbol with a function -- `switch(CosmoMappErrno(EINVAL))` and `case COSMO_EINVAL:` and so avoid turning the code into goto mush.
- outworlder 3y agoDo we even care if the replacement is done automatically?
- jsmith45 3y agoBut that won't help with compiling unmodified code. Nobody should worry about gotos being created inside the compiler. It is doing that all over the place anyway. Unless of course you mean that the compiler should be recognizing switches like this, and instead of rewriting them to if trees, it should be rewriting them to switches, changing the labels to use special cosmo specific constants for each of the values, and wraping the input value to the switch with a call to a function that maps the current runtimes platform's values for these over to corresponding cosmo specific constants (and letting other values pass though unchanged). That... actually might be a simpler compiler transformation to implement. There would be complexity in needing to recognize which of the multiple sets of of constants are being used, and applying the right call or calls to the switch input to map them. It also require complexity on the library side though (creating these ). Not sure if the author would want to implement it, given they have a working implementation of the other transformation. Lastly such a mapping approach would have risks of inappropriately mapping, or trickiness like having to invert the input before mapping for code that does the negative errno value thing.
- ahgamut 3y agoDereferencing the runtime symbol with a function sounds interesting! If you can show me an example of where it works, I'd be happy to try it out. I like the if-else-goto arrangement because it fit in perfectly with the other parts of gcc -- if you look at my patch[1], you will find that I had to change very little of gcc's existing code to add this capability. [1]: https://github.com/ahgamut/gcc/tree/portcosmo-11.2 https://github.com/ahgamut/gcc/tree/portcosmo-11.2
- charonn0 3y agoN.B.: "Portable Executable" or "PE" is also the name of the Windows executable format.
- epcoa 3y agoLooks like they changed the story headline to Actually Portable Executables (which is the original “correct” name). Other than an audio format relatively few use I suppose APE isn’t the most clashy file format name.
- 1vuio0pswjnm7 3y agoIt seems that the term "portable" in PE refers to portability to different hardware whereas the same term used in APE refers to portability to different operating systems.
- throwaway8356 3y agoPortable is a generic term. The PE format is the same for x86 and ARM64 and other architectures. https://learn.microsoft.com/en-us/windows/win32/debug/pe-format https://learn.microsoft.com/en-us/windows/win32/debug/pe-for... > The name "Portable Executable" refers to the fact that the format is not architecture specific https://en.m.wikipedia.org/wiki/Portable_Executable https://en.m.wikipedia.org/wiki/Portable_Executable > On Windows NT operating systems, PE currently supports the x86-32, x86-64 (AMD64/Intel 64), IA-64, ARM and ARM64 instruction set architectures (ISAs). Prior to Windows 2000, Windows NT (and thus PE) supported the MIPS, Alpha, and PowerPC ISAs. Because PE is used on Windows CE, it continues to support several variants of the MIPS, ARM (including Thumb), and SuperH ISAs. In my opinion it qualifies as portable
- skissane 3y agoIt is portable in comparison to its main predecessor NE which was x86 only It also is used by more than just Windows - EFI uses it too. There is no reason in principle why some non-Windows OS couldn’t use it as the native executable format.
- appleflaxen 3y agoThis is incredible cool.
- yjftsjthsd-h 3y agoIn case the author sees this: > Of course, my patch isn’t perfect. It can’t handle some anonymous structs, enums, const ints, or amazing things like using SIGILL as an array index [...] Is "SIGILL" a typo?
- hkgjjgjfjfjfjf 3y ago[dead]
- inglor_cz 3y agoThe SIGILL signal is raised when an attempt is made to execute an invalid, privileged, or ill-formed instruction. SIGILL is usually caused by a program error that overlays code with data or by a call to a function that is not linked into the program load module. [0] https://support.sas.com/documentation/onlinedoc/ccompiler/doc700/html/lr1/z2005346.htm https://support.sas.com/documentation/onlinedoc/ccompiler/do...
- yjftsjthsd-h 3y agoThanks, didn't know that was an actual signal!
- spacechild1 3y agoAnother common cause is trying to execute instructions that the CPU doesn't support, e.g. trying to run AVX2 code on a machine that only supports SSE4.
- inglor_cz 3y agoYeah, the only situation when I saw SIGILL "live" was my attempt to run some AES instructions on an emulator that didn't support them.
- ahgamut 3y agoSIGILL is for "illegal instruction", according to https://man7.org/linux/man-pages/man7/signal.7.html https://man7.org/linux/man-pages/man7/signal.7.html I too thought it was a typo when I caused it to occur while building BLIS, but then I found out it was because I had added some AVX thing to the config that my local computer did not have.
- dataangel 3y agoIs there any practical purpose for APE? Like who is shipping single executables that need to run on N OSes?
- Timon3 3y agoI'm trying to release future side projects that way. I even build a Redbean executable for any client-only SPAs. The idea of this stuff just working on so many systems is amazing.
- nicoburns 3y agoProblem with this is that self-modifying code will likely trip security blocks on most platforms, which is probably more annoying for users than having to download separate executables.
- ElectricalUnion 3y agoIf the person needs to run with "security blocks" up, it's statistically unlikely that the same person can randomly download the specific and potentially outdated version of a dependency you need for the code to run in the first place.
- jjnoakes 3y agoStoring executables on file systems shared across operating systems or architectures gets a lot easier if you don't have to have special paths for each os+arch combination.
- fit2rule 3y ago[dead]
- ahgamut 3y agoI wrote this post: the title should be "Patching GCC to build Actually Portable Executables", because it refers to Cosmopolitan Libc and jart's Actually Portable Executable format. With my gcc patch, you can now build software like vim, emacs, ninja, bash, git, gcc etc with Cosmopolitan Libc, via their usual autotools/cmake-style build system. The built executables should run on Linux, FreeBSD, MacOS, OpenBSD, NetBSD, and also Windows (although I haven't tested Windows yet.) Here's a list of software I got to build with this technique: https://github.com/ahgamut/superconfigure https://github.com/ahgamut/superconfigure The superconfigure script is just a wrapper around the usual configure script used to build your software, supplying flags like --enable-static. If you want to build gcc using Cosmopolitan Libc -- try out this repo: https://github.com/ahgamut/musl-cross-make/tree/gccbuild https://github.com/ahgamut/musl-cross-make/tree/gccbuild
- foresto 3y agoNice! Do you foresee any problems with using this behind a transpiled language like Nim?
- ahgamut 3y agoI haven't used Nim much, but I remember a repo on Github had setup a build script for compiling Nim with Cosmopolitan Libc. https://github.com/Yardanico/cosmonim https://github.com/Yardanico/cosmonim This gcc patch makes such build scripts simpler, because you will need to change less of your code -- let me know how it works!
- ChickeNES 3y agoHave you attempted to build busybox?
- ahgamut 3y agoYes, I have gotten busybox to build successfully, but at present it's not as nice as bash or vim. I expect someone will write a better build script soon enough. Maybe it will be you! The default build parameters in busybox make a lot of OS-related assumptions, so I'd recommend you use make menuconfig instead of make depconfig. in the TUI, you can disable all console and networking utilities, linux modprobe utilities, some linux system utilities, free, uptime, etc. Start with a minimal config and then slowly add things. In terms of source code changes: modify `u_signal_names.c` to use a switch statement instead of the amazing "use SIGHUP as an array index" method it follows, use int32_t instead of the "smallint" type that busybox seems to prefer, and IIRC there's an enum in there somewhere that should be rewritten as a #define. That should get you pretty close to a successful build.
- develatio 3y agoThere is a ticket in Go's repo about this: https://github.com/golang/go/issues/51900 https://github.com/golang/go/issues/51900
- cratermoon 3y agoIn case anyone is confused. "Actually Portable Executables" https://justine.lol/ape.html https://justine.lol/ape.html
- ahgamut 3y agoI added a comment mentioning this, but it hasn't changed yet. I expect it will change soon. Perhaps I can ping @dang from here?
- jwilk 3y agohttps://news.ycombinator.com/item?id=36526450 https://news.ycombinator.com/item?id=36526450 > The only way to get reliable message delivery is to email hn@ycombinator.com.
- cratermoon 3y agoJustine's post did a great job teaching me the details of leveraging the various executable formats to make it possible.
- ylyn 3y agoI don't see why you are so surprised by code that indexes using SIGxxx. The signal numbers are well-defined and known to be mostly consecutive and start from 1.
- ahgamut 3y agoI had tried to build a bunch of different codebases (git, bash, gcc, curl etc.) before trying busybox, and I'd never seen code like that before. Plus that error came alongside a compiler error in my patch that took a long time to debug, so it was a memorable surprise :) Would you consider a code pattern like that to be common? What other codebases apart from busybox have it? If there are many examples, I might spend some time trying to update my patch to handle patterns of that kind.
- mgaunard 3y agoIf errno is such a big problem, what's the hope in even running the C++ standard library and its std::error_code... No way to use magic macros there.
- foota 3y agoThe issue _was_ the magic macros, I believe. Normally, the macros expand to a constant, but now they don't, which breaks some things. std::error_code doesn't have the same issue, since it's not a constant?
- mgaunard 3y agoWell if they need to replace those integer constants with function calls, clearly that breaks everything. std::error_code is just a wrapper for several different enums. I should have probably just given the posix one, std::errc. You can't just make the various enumerations of an enum be function calls. The right implementation would have been to convert error codes from/to posix, rather than just give native error codes and make looking up the reference value a runtime operation.
- Gordonjcp 3y ago"If EINVAL is not a compile-time constant, four of the initializations above are not valid" Wait, why isn't EINVAL a compile-time constant, shouldn't it just be an enum?
- londons_explore 3y agoIt has a different constant value on different OS's (Linux, Mac, Windows, etc). Therefore the resulting code, which must run on any OS, cannot treat it as a constant.
- mike_hock 3y agoThus proving that APEs are not viable, the end. The assumption that constants are constants is a sound assumption, and I highly doubt many would be willing to rewrite the world to make some "clever hack" work.
- ahgamut 3y ago> I highly doubt many would be willing to rewrite the world to make some "clever hack" work. I specifically wanted to avoid rewriting the world in order to port established codebases like git/curl to Cosmopolitan Libc. My goal with this gcc patch was to answer the question: "what is the minimum amount of source code I would have to manually change in order to port software to Cosmopolitan Libc?". As we find out, sometimes we don't have to change anything to compile code for an Actually Portable Executable -- just do the usual ./configure && make, and a small `objcopy` command at the end to create the APE.
- Gordonjcp 3y agoAh, right, so while you and I don't care what numerical value in the enum EINVAL has, if I take that numerical value and stuff the binary into another OS then it means something totally different. Right, it's not that it *isn't* a constant, it's that its not the *same* constant everywhere?
- 3y ago
- f33d5173 3y agoI suppose this is all quite neat, but I wonder about the theory behind it. If you're building a new OS target, the logical thing to do is to define new contants. Rather than passing in the OSes constants on every underlying OS, you would translate the OSes constants to be the same everywhere. Then, use of constants as array indexes and case statements would work normally. This might incur more overhead than does the approach described in the article, but it seems a great deal easier, and simpler to use for whoever decides to compile a new program against cosmopolitan libc.
- ahgamut 3y agoNew constants do make compilation a lot easier, but my personal opinion is that the overhead to convert to/from the old constants during runtime is too much. Look at the list of constants here: https://github.com/jart/cosmopolitan/blob/master/libc/sysv/consts.sh https://github.com/jart/cosmopolitan/blob/master/libc/sysv/c... Every time I used any of these constants, I'd have to load a whole bunch of them into my binary as a large lookup table, and go through that table every time I needed a check in my program. It might not be that slow, but I believe it would definitely be noticeable. My goal was to make porting easier without changing a lot of source code in either the libc or in the software I was trying to port, and still produce binaries that are close or better in performance. Under those constraints, this gcc patch seemed like the best way to simplify the process. If I run into enough codebases where SIGHUP is used as an array index initializer, I will probably attempt your suggestion just to measure the tradeoffs. Or you could try it out and let me know if a separate set of constants is better.