11 ms·

Can somebody explain in layman’s terms what this is all about? I tried my best reading the other thread, but it went well above me.

Gaelan 6y ago
Windows, macOS, Linux, and BSD all (usually) run on Intel processors, so in theory it should be possible to write a program that runs on all three. However, despite ultimately using the same instruction sets, the four OSes use different formats to store metadata about the programs, and have different ways for programs to communicate with the operating system. Justine Tunney created two things: 1) Actually Portable Executables, a clever way of formatting a program so that all four OSes interpret it a valid program in their own format. 2) Cosmopolitan libc, a library for communicating with the OS that handles each OS's interface, allowing programs to work on all four.
- abhinav22 6y ago
  Thank you (and thanks to everybody else for their helpful answers). Your note on intel processors meaning in theory programs should run on all operating systems was most helpful and got me across the line - now I get it!
  - jart 6y ago
    That's pretty much it. Cosmopolitan Libc is the simplest project that could have happened decades ago. Anyone could have built it. There's never been a technical reason why an x86 binary can't run on all x86 operating systems. It's just that traditionally only operating systems publish C libraries, and they have no incentive to spend money supporting their competitors. So it's the kind of project that could only happen if it was done by an indie developer just trying to have fun.
    - fctorial 6y ago
      Another reason is that it requires fairly thorough knowledge of 7 different programming environments. Most people don't understand 1.
      - jart 6y ago
        The only programming environment I used to build this was Emacs on Linux. The rest was simply researching the magic numbers needed to use the binary interface of each system. I figured life's too short to be pulling my hair out with MSVC/Xcode warnings.
        fctorial 6y ago
        s/programming environments/systems/
    - thysultan 6y ago
      How does one unzip the .com file? Is that even possible or have i misunderstood how it works?
      - jart 6y ago
        I recommend using InfoZIP on the Linux or Mac command line. For example, to list the files that are in your .com executables you can run a command like: $ wget https://justine.lol/redbean/redbean-2021-02-27.com $ unzip -vl redbean-2021-02-27.com Archive: redbean-2021-02-27.com Length Method Size Cmpr Date Time CRC-32 Name -------- ------ ------- ---- ---------- ----- -------- ---- 426 Defl:N 215 50% 02-09-2021 09:24 f9cc9464 tool/net/redbean.css 896 Defl:N 509 43% 02-09-2021 09:24 eb74f8a0 tool/net/redbean.html 16958 Defl:N 6093 64% 02-09-2021 09:24 6a76bd39 tool/net/redbean.ico 5073 Stored 5073 0% 02-09-2021 09:24 86c2afe6 tool/net/redbean.png 554 Defl:N 285 49% 02-09-2021 09:24 0aa6870a usr/share/zoneinfo/Beijing 2335 Defl:N 1093 53% 02-09-2021 09:24 6ccd4450 usr/share/zoneinfo/Berlin 2453 Defl:N 1170 52% 02-09-2021 09:24 7fed6d80 usr/share/zoneinfo/Boulder ... Another ZIP tool that works well with APE is Windows 10 file explorer. But you have to change the extension from .com to .zip temporarily.
    - lambda_obrien 6y ago
      Does cosmolibc allow you to run on bare metal? If so, I've got an old project in mind that you may have just made happen, and I'm very very excited to know!
      - collinmanderson 6y ago
        "Actually Portable Executable"s can boot from BIOS if that's what you're wondering. https://justine.lol/ape.html https://justine.lol/ape.html
        lambda_obrien 6y ago
        Cool, thanks! I missed that before. It makes a project idea I had way more realistic for me to implement.
- shrx 6y ago
  They also run on AMD ;)
- galfarragem 6y ago
  Will this rush an optimized version of Electron or even deprecate it?
  - Nextgrid 6y ago
    Technically, I guess you could use this to compile a universal browser binary that would run on all platforms. However, this wouldn't solve any of the problems caused by Electron.
    - darkfirefly 6y ago
      Or really any of the problems Electron attempts to solve; the GUI toolkits are still not (generally) not cross-platform in the way Electron is.
    - dj_mc_merlin 6y ago
      I think his point was that with Cosmopolitan native binaries one would no longer need Electron to provide cross-platform binaries. There is a lack of good GUI frameworks to build this native Electron replacement with though, nor is it certain the effort would not be too high.
  - simias 6y ago
    Note that packaged electron apps are not portable across OS, so it's orthogonal. What this could do is make the same electron package work on different OS without rebuild by using a portable browser binary. Meanwhile Cosmopolitan doesn't provide a portable GUI toolkit so it can't replace electron on its own. Maybe if you could package Cosmopolitan + Qt or something like that you could end up with something very interesting though.
    - cxr 6y ago
      > What this could do is make the same electron package work on different OS without rebuild by using a portable browser binary. Likewise for non-GUI apps, like command-line utilities that are implemented with NodeJS. As prior art, Fabrice Bellard's QuickJS is able to compile JS programs to spit out a native binary that requires no external qjs runtime. If you modified it to output Actual Portable Executable binaries instead, it would be even more portable than the programs installed with `npm install`. I've done some experiments and progressed to writing tooling that approaches that I-want-ridiculous-portability problem differently. It doesn't use APE, so it still requires a separate interpreter (such as NodeJS) capable of executing the compiled output, but the object files it outputs are such that even if you don't have a copy of NodeJS installed on your system (or the right copy of NodeJS installed), then you can piggy back off the interpreter built in to your browser. You get a similar double-click-to-run experience, but when you do double-click to run, it launches in the browser, and you can go from object file to modifiable source trivially, since the object file and the Git repo from which it is built are automorphisms. The downside, if you consider it one, is that it doesn't work with arbitrary NPM modules. You have to really buy in to the scripting language dialect and the development philosophy to get anything out of it. It would be interesting the use the QuickJS+APE strategy above so that there is _no_ reliance on an external interpreter, at the cost of marrying yourself to x86 and introducing some small hurdles in the path to modification and also giving up the safety of the browser sandbox, since you're back to running arbitrary executables.
- A12-B 6y ago
  What happens when intel goes bust and we all have to switch to new processors
  - OOPMan 6y ago
    A little optimistic about Apples SoCs aren't you? Also...given that AMD exists and the x86 instruction set is not some magic potion ingredients...
- munawwar 6y ago
  So it's like a compatibly layer like wine except built into the binary. Does this bloat the binary as well?
  - sgtnoodle 6y ago
    The only code that really needs to be duplicated would be the lowest level of userspace code that actually calls into the various operating systems. Things like the guts of "open", "read", "write", etc. Regardless of any particular OS's flavor, they're all going to roughly take in the same arguments and return the same results via some flavor of system call. Each individual function is probably on the order of a few dozen machine instructions. Let's say there's 30 such functions that are implemented and 7 OSes to support. 36307 is about 7500 instructions. The average instruction length on x86-64 is 2-3 bytes. So maybe it ends up around 20KB of code space for a program that uses every possible operating system feature that's implemented? I haven't looked at the implementation. I wonder how it selects between OS implementations. Does it switch on every system call, use a function pointer table, or do some sort of clever in-memory code rewriting?
gostsamo 6y ago
In short, there is a "framework" that allows to compile a single binary that can run under both Linux, Windows, bare metal, and many others. Like the Apple universal binaries, but actually universal, not just covering the m1's arm64 and x86 64.
- Gys 6y ago
  It is x86 only, arm is not supported. But would be a very nice addition :)
- TickleSteve 6y ago
  non-x86 is emulated and bare-metal limits the portability.
snarfy 6y ago
It makes binaries that can run anywhere by shimming(?) libc functions and values with a custom loader/linker.
max_ 6y ago
You can compile a program on MacOS. The result, an executable binary, can run on Linux, Mac, Windows, FreeBSD, OpenBSD, NetBSD, BIOS .... the same EXACT binary!
matheusmoreira 6y ago
The author somehow came up with an executable file format that's compatible with everything. It's simultaneously a valid shell script, Unix ELF, Windows PE file, boot sector code and zip file. Then she wrote a standard C library that detects which OS you're using at runtime and branches to the appropriate implementation, allowing the same x86_64 code to run on any supported system. if (IsWindows()) DoWindowsThing(); else DoUnixThing(); Traditionally, these system differences are resolved at build time: only the code for the target platform is included. Why would anyone want Windows-specific code in a Linux build? It's dead code... right? The new run-anywhere executable format makes that code useful.
- IshKebab 6y ago
  Are you sure? Other answers are suggesting it isn't a valid ELF file and I don't get why it would need to be a valid boot sector or zip file. You might be right though. The author might be very good at executable hacks but they're a terrible writer. Their explanation explains nothing.
  - reikonomusha 6y ago
    The author seems to be an excellent writer. They just may not have taken the enormous amount of time to answer tons of questions or write tons of documentation people are seeking. One person did all this!
    - IshKebab 6y ago
      You really think so? Here's their description of how it works: > Here's how it works: > > MZqFpD=' > BIOS BOOT SECTOR' > exec 7<> $(command -v $0) > printf '\177ELF...LINKER-ENCODED-FREEBSD-HEADER' >&7 > exec "$0" "$@" > exec qemu-x86_64 "$0" "$@" > exit 1 > REAL MODE... > ELF SEGMENTS... > OPENBSD NOTE... > NETBSD NOTE... > MACHO HEADERS... > CODE AND DATA... > ZIP DIRECTORY... Ok well that clears it up. MZqFpD='. Of course!
  - matheusmoreira 6y ago
    Actually I'm not 100% sure. I can barely make sense of it. All I know is it somehow works as if it was a valid ELF. Some dark magic going on here...
baq 6y ago
No worries. You’re in good company. This actually portable executable thing is batshit insane (in the best possible way).

Gaelan 6y ago

Windows, macOS, Linux, and BSD all (usually) run on Intel processors, so in theory it should be possible to write a program that runs on all three. However, despite ultimately using the same instruction sets, the four OSes use different formats to store metadata about the programs, and have different ways for programs to communicate with the operating system. Justine Tunney created two things: 1) Actually Portable Executables, a clever way of formatting a program so that all four OSes interpret it a valid program in their own format. 2) Cosmopolitan libc, a library for communicating with the OS that handles each OS's interface, allowing programs to work on all four.

abhinav22 6y ago
Thank you (and thanks to everybody else for their helpful answers). Your note on intel processors meaning in theory programs should run on all operating systems was most helpful and got me across the line - now I get it!
- jart 6y ago
  That's pretty much it. Cosmopolitan Libc is the simplest project that could have happened decades ago. Anyone could have built it. There's never been a technical reason why an x86 binary can't run on all x86 operating systems. It's just that traditionally only operating systems publish C libraries, and they have no incentive to spend money supporting their competitors. So it's the kind of project that could only happen if it was done by an indie developer just trying to have fun.
  - fctorial 6y ago
    Another reason is that it requires fairly thorough knowledge of 7 different programming environments. Most people don't understand 1.
    - jart 6y ago
      The only programming environment I used to build this was Emacs on Linux. The rest was simply researching the magic numbers needed to use the binary interface of each system. I figured life's too short to be pulling my hair out with MSVC/Xcode warnings.
      - fctorial 6y ago
        s/programming environments/systems/
  - thysultan 6y ago
    How does one unzip the .com file? Is that even possible or have i misunderstood how it works?
    - jart 6y ago
      I recommend using InfoZIP on the Linux or Mac command line. For example, to list the files that are in your .com executables you can run a command like: $ wget https://justine.lol/redbean/redbean-2021-02-27.com $ unzip -vl redbean-2021-02-27.com Archive: redbean-2021-02-27.com Length Method Size Cmpr Date Time CRC-32 Name -------- ------ ------- ---- ---------- ----- -------- ---- 426 Defl:N 215 50% 02-09-2021 09:24 f9cc9464 tool/net/redbean.css 896 Defl:N 509 43% 02-09-2021 09:24 eb74f8a0 tool/net/redbean.html 16958 Defl:N 6093 64% 02-09-2021 09:24 6a76bd39 tool/net/redbean.ico 5073 Stored 5073 0% 02-09-2021 09:24 86c2afe6 tool/net/redbean.png 554 Defl:N 285 49% 02-09-2021 09:24 0aa6870a usr/share/zoneinfo/Beijing 2335 Defl:N 1093 53% 02-09-2021 09:24 6ccd4450 usr/share/zoneinfo/Berlin 2453 Defl:N 1170 52% 02-09-2021 09:24 7fed6d80 usr/share/zoneinfo/Boulder ... Another ZIP tool that works well with APE is Windows 10 file explorer. But you have to change the extension from .com to .zip temporarily.
  - lambda_obrien 6y ago
    Does cosmolibc allow you to run on bare metal? If so, I've got an old project in mind that you may have just made happen, and I'm very very excited to know!
    - collinmanderson 6y ago
      "Actually Portable Executable"s can boot from BIOS if that's what you're wondering. https://justine.lol/ape.html https://justine.lol/ape.html
      - lambda_obrien 6y ago
        Cool, thanks! I missed that before. It makes a project idea I had way more realistic for me to implement.
shrx 6y ago
They also run on AMD ;)
galfarragem 6y ago
Will this rush an optimized version of Electron or even deprecate it?
- Nextgrid 6y ago
  Technically, I guess you could use this to compile a universal browser binary that would run on all platforms. However, this wouldn't solve any of the problems caused by Electron.
  - darkfirefly 6y ago
    Or really any of the problems Electron attempts to solve; the GUI toolkits are still not (generally) not cross-platform in the way Electron is.
  - dj_mc_merlin 6y ago
    I think his point was that with Cosmopolitan native binaries one would no longer need Electron to provide cross-platform binaries. There is a lack of good GUI frameworks to build this native Electron replacement with though, nor is it certain the effort would not be too high.
- simias 6y ago
  Note that packaged electron apps are not portable across OS, so it's orthogonal. What this could do is make the same electron package work on different OS without rebuild by using a portable browser binary. Meanwhile Cosmopolitan doesn't provide a portable GUI toolkit so it can't replace electron on its own. Maybe if you could package Cosmopolitan + Qt or something like that you could end up with something very interesting though.
  - cxr 6y ago
    > What this could do is make the same electron package work on different OS without rebuild by using a portable browser binary. Likewise for non-GUI apps, like command-line utilities that are implemented with NodeJS. As prior art, Fabrice Bellard's QuickJS is able to compile JS programs to spit out a native binary that requires no external qjs runtime. If you modified it to output Actual Portable Executable binaries instead, it would be even more portable than the programs installed with `npm install`. I've done some experiments and progressed to writing tooling that approaches that I-want-ridiculous-portability problem differently. It doesn't use APE, so it still requires a separate interpreter (such as NodeJS) capable of executing the compiled output, but the object files it outputs are such that even if you don't have a copy of NodeJS installed on your system (or the right copy of NodeJS installed), then you can piggy back off the interpreter built in to your browser. You get a similar double-click-to-run experience, but when you do double-click to run, it launches in the browser, and you can go from object file to modifiable source trivially, since the object file and the Git repo from which it is built are automorphisms. The downside, if you consider it one, is that it doesn't work with arbitrary NPM modules. You have to really buy in to the scripting language dialect and the development philosophy to get anything out of it. It would be interesting the use the QuickJS+APE strategy above so that there is _no_ reliance on an external interpreter, at the cost of marrying yourself to x86 and introducing some small hurdles in the path to modification and also giving up the safety of the browser sandbox, since you're back to running arbitrary executables.
A12-B 6y ago
What happens when intel goes bust and we all have to switch to new processors
- OOPMan 6y ago
  A little optimistic about Apples SoCs aren't you? Also...given that AMD exists and the x86 instruction set is not some magic potion ingredients...
munawwar 6y ago
So it's like a compatibly layer like wine except built into the binary. Does this bloat the binary as well?
- sgtnoodle 6y ago
  The only code that really needs to be duplicated would be the lowest level of userspace code that actually calls into the various operating systems. Things like the guts of "open", "read", "write", etc. Regardless of any particular OS's flavor, they're all going to roughly take in the same arguments and return the same results via some flavor of system call. Each individual function is probably on the order of a few dozen machine instructions. Let's say there's 30 such functions that are implemented and 7 OSes to support. 36307 is about 7500 instructions. The average instruction length on x86-64 is 2-3 bytes. So maybe it ends up around 20KB of code space for a program that uses every possible operating system feature that's implemented? I haven't looked at the implementation. I wonder how it selects between OS implementations. Does it switch on every system call, use a function pointer table, or do some sort of clever in-memory code rewriting?

gostsamo 6y ago

In short, there is a "framework" that allows to compile a single binary that can run under both Linux, Windows, bare metal, and many others. Like the Apple universal binaries, but actually universal, not just covering the m1's arm64 and x86 64.

Gys 6y ago
It is x86 only, arm is not supported. But would be a very nice addition :)
TickleSteve 6y ago
non-x86 is emulated and bare-metal limits the portability.

snarfy 6y ago

It makes binaries that can run anywhere by shimming(?) libc functions and values with a custom loader/linker.

max_ 6y ago

You can compile a program on MacOS. The result, an executable binary, can run on Linux, Mac, Windows, FreeBSD, OpenBSD, NetBSD, BIOS .... the same EXACT binary!

matheusmoreira 6y ago

The author somehow came up with an executable file format that's compatible with everything. It's simultaneously a valid shell script, Unix ELF, Windows PE file, boot sector code and zip file. Then she wrote a standard C library that detects which OS you're using at runtime and branches to the appropriate implementation, allowing the same x86_64 code to run on any supported system. if (IsWindows()) DoWindowsThing(); else DoUnixThing(); Traditionally, these system differences are resolved at build time: only the code for the target platform is included. Why would anyone want Windows-specific code in a Linux build? It's dead code... right? The new run-anywhere executable format makes that code useful.

IshKebab 6y ago
Are you sure? Other answers are suggesting it isn't a valid ELF file and I don't get why it would need to be a valid boot sector or zip file. You might be right though. The author might be very good at executable hacks but they're a terrible writer. Their explanation explains nothing.
- reikonomusha 6y ago
  The author seems to be an excellent writer. They just may not have taken the enormous amount of time to answer tons of questions or write tons of documentation people are seeking. One person did all this!
  - IshKebab 6y ago
    You really think so? Here's their description of how it works: > Here's how it works: > > MZqFpD=' > BIOS BOOT SECTOR' > exec 7<> $(command -v $0) > printf '\177ELF...LINKER-ENCODED-FREEBSD-HEADER' >&7 > exec "$0" "$@" > exec qemu-x86_64 "$0" "$@" > exit 1 > REAL MODE... > ELF SEGMENTS... > OPENBSD NOTE... > NETBSD NOTE... > MACHO HEADERS... > CODE AND DATA... > ZIP DIRECTORY... Ok well that clears it up. MZqFpD='. Of course!
- matheusmoreira 6y ago
  Actually I'm not 100% sure. I can barely make sense of it. All I know is it somehow works as if it was a valid ELF. Some dark magic going on here...

baq 6y ago

No worries. You’re in good company. This actually portable executable thing is batshit insane (in the best possible way).