5 ms·

Windows, macOS, Linux, and BSD all (usually) run on Intel processors, so in theory it should be possible to write a program that runs on all three. However, des

by Gaelan 6y ago

Windows, macOS, Linux, and BSD all (usually) run on Intel processors, so in theory it should be possible to write a program that runs on all three. However, despite ultimately using the same instruction sets, the four OSes use different formats to store metadata about the programs, and have different ways for programs to communicate with the operating system. Justine Tunney created two things: 1) Actually Portable Executables, a clever way of formatting a program so that all four OSes interpret it a valid program in their own format. 2) Cosmopolitan libc, a library for communicating with the OS that handles each OS's interface, allowing programs to work on all four.

abhinav22 6y ago
Thank you (and thanks to everybody else for their helpful answers). Your note on intel processors meaning in theory programs should run on all operating systems was most helpful and got me across the line - now I get it!
- jart 6y ago
  That's pretty much it. Cosmopolitan Libc is the simplest project that could have happened decades ago. Anyone could have built it. There's never been a technical reason why an x86 binary can't run on all x86 operating systems. It's just that traditionally only operating systems publish C libraries, and they have no incentive to spend money supporting their competitors. So it's the kind of project that could only happen if it was done by an indie developer just trying to have fun.
  - fctorial 6y ago
    Another reason is that it requires fairly thorough knowledge of 7 different programming environments. Most people don't understand 1.
    - jart 6y ago
      The only programming environment I used to build this was Emacs on Linux. The rest was simply researching the magic numbers needed to use the binary interface of each system. I figured life's too short to be pulling my hair out with MSVC/Xcode warnings.
      - fctorial 6y ago
        s/programming environments/systems/
  - thysultan 6y ago
    How does one unzip the .com file? Is that even possible or have i misunderstood how it works?
    - jart 6y ago
      I recommend using InfoZIP on the Linux or Mac command line. For example, to list the files that are in your .com executables you can run a command like: $ wget https://justine.lol/redbean/redbean-2021-02-27.com $ unzip -vl redbean-2021-02-27.com Archive: redbean-2021-02-27.com Length Method Size Cmpr Date Time CRC-32 Name -------- ------ ------- ---- ---------- ----- -------- ---- 426 Defl:N 215 50% 02-09-2021 09:24 f9cc9464 tool/net/redbean.css 896 Defl:N 509 43% 02-09-2021 09:24 eb74f8a0 tool/net/redbean.html 16958 Defl:N 6093 64% 02-09-2021 09:24 6a76bd39 tool/net/redbean.ico 5073 Stored 5073 0% 02-09-2021 09:24 86c2afe6 tool/net/redbean.png 554 Defl:N 285 49% 02-09-2021 09:24 0aa6870a usr/share/zoneinfo/Beijing 2335 Defl:N 1093 53% 02-09-2021 09:24 6ccd4450 usr/share/zoneinfo/Berlin 2453 Defl:N 1170 52% 02-09-2021 09:24 7fed6d80 usr/share/zoneinfo/Boulder ... Another ZIP tool that works well with APE is Windows 10 file explorer. But you have to change the extension from .com to .zip temporarily.
  - lambda_obrien 6y ago
    Does cosmolibc allow you to run on bare metal? If so, I've got an old project in mind that you may have just made happen, and I'm very very excited to know!
    - collinmanderson 6y ago
      "Actually Portable Executable"s can boot from BIOS if that's what you're wondering. https://justine.lol/ape.html https://justine.lol/ape.html
      - lambda_obrien 6y ago
        Cool, thanks! I missed that before. It makes a project idea I had way more realistic for me to implement.
shrx 6y ago
They also run on AMD ;)
galfarragem 6y ago
Will this rush an optimized version of Electron or even deprecate it?
- Nextgrid 6y ago
  Technically, I guess you could use this to compile a universal browser binary that would run on all platforms. However, this wouldn't solve any of the problems caused by Electron.
  - darkfirefly 6y ago
    Or really any of the problems Electron attempts to solve; the GUI toolkits are still not (generally) not cross-platform in the way Electron is.
  - dj_mc_merlin 6y ago
    I think his point was that with Cosmopolitan native binaries one would no longer need Electron to provide cross-platform binaries. There is a lack of good GUI frameworks to build this native Electron replacement with though, nor is it certain the effort would not be too high.
- simias 6y ago
  Note that packaged electron apps are not portable across OS, so it's orthogonal. What this could do is make the same electron package work on different OS without rebuild by using a portable browser binary. Meanwhile Cosmopolitan doesn't provide a portable GUI toolkit so it can't replace electron on its own. Maybe if you could package Cosmopolitan + Qt or something like that you could end up with something very interesting though.
  - cxr 6y ago
    > What this could do is make the same electron package work on different OS without rebuild by using a portable browser binary. Likewise for non-GUI apps, like command-line utilities that are implemented with NodeJS. As prior art, Fabrice Bellard's QuickJS is able to compile JS programs to spit out a native binary that requires no external qjs runtime. If you modified it to output Actual Portable Executable binaries instead, it would be even more portable than the programs installed with `npm install`. I've done some experiments and progressed to writing tooling that approaches that I-want-ridiculous-portability problem differently. It doesn't use APE, so it still requires a separate interpreter (such as NodeJS) capable of executing the compiled output, but the object files it outputs are such that even if you don't have a copy of NodeJS installed on your system (or the right copy of NodeJS installed), then you can piggy back off the interpreter built in to your browser. You get a similar double-click-to-run experience, but when you do double-click to run, it launches in the browser, and you can go from object file to modifiable source trivially, since the object file and the Git repo from which it is built are automorphisms. The downside, if you consider it one, is that it doesn't work with arbitrary NPM modules. You have to really buy in to the scripting language dialect and the development philosophy to get anything out of it. It would be interesting the use the QuickJS+APE strategy above so that there is _no_ reliance on an external interpreter, at the cost of marrying yourself to x86 and introducing some small hurdles in the path to modification and also giving up the safety of the browser sandbox, since you're back to running arbitrary executables.
A12-B 6y ago
What happens when intel goes bust and we all have to switch to new processors
- OOPMan 6y ago
  A little optimistic about Apples SoCs aren't you? Also...given that AMD exists and the x86 instruction set is not some magic potion ingredients...
munawwar 6y ago
So it's like a compatibly layer like wine except built into the binary. Does this bloat the binary as well?
- sgtnoodle 6y ago
  The only code that really needs to be duplicated would be the lowest level of userspace code that actually calls into the various operating systems. Things like the guts of "open", "read", "write", etc. Regardless of any particular OS's flavor, they're all going to roughly take in the same arguments and return the same results via some flavor of system call. Each individual function is probably on the order of a few dozen machine instructions. Let's say there's 30 such functions that are implemented and 7 OSes to support. 36307 is about 7500 instructions. The average instruction length on x86-64 is 2-3 bytes. So maybe it ends up around 20KB of code space for a program that uses every possible operating system feature that's implemented? I haven't looked at the implementation. I wonder how it selects between OS implementations. Does it switch on every system call, use a function pointer table, or do some sort of clever in-memory code rewriting?

by Gaelan 6y ago

abhinav22 6y ago

Thank you (and thanks to everybody else for their helpful answers). Your note on intel processors meaning in theory programs should run on all operating systems was most helpful and got me across the line - now I get it!

jart 6y ago
That's pretty much it. Cosmopolitan Libc is the simplest project that could have happened decades ago. Anyone could have built it. There's never been a technical reason why an x86 binary can't run on all x86 operating systems. It's just that traditionally only operating systems publish C libraries, and they have no incentive to spend money supporting their competitors. So it's the kind of project that could only happen if it was done by an indie developer just trying to have fun.
- fctorial 6y ago
  Another reason is that it requires fairly thorough knowledge of 7 different programming environments. Most people don't understand 1.
  - jart 6y ago
    The only programming environment I used to build this was Emacs on Linux. The rest was simply researching the magic numbers needed to use the binary interface of each system. I figured life's too short to be pulling my hair out with MSVC/Xcode warnings.
    - fctorial 6y ago
      s/programming environments/systems/
- thysultan 6y ago
  How does one unzip the .com file? Is that even possible or have i misunderstood how it works?
  - jart 6y ago
    I recommend using InfoZIP on the Linux or Mac command line. For example, to list the files that are in your .com executables you can run a command like: $ wget https://justine.lol/redbean/redbean-2021-02-27.com $ unzip -vl redbean-2021-02-27.com Archive: redbean-2021-02-27.com Length Method Size Cmpr Date Time CRC-32 Name -------- ------ ------- ---- ---------- ----- -------- ---- 426 Defl:N 215 50% 02-09-2021 09:24 f9cc9464 tool/net/redbean.css 896 Defl:N 509 43% 02-09-2021 09:24 eb74f8a0 tool/net/redbean.html 16958 Defl:N 6093 64% 02-09-2021 09:24 6a76bd39 tool/net/redbean.ico 5073 Stored 5073 0% 02-09-2021 09:24 86c2afe6 tool/net/redbean.png 554 Defl:N 285 49% 02-09-2021 09:24 0aa6870a usr/share/zoneinfo/Beijing 2335 Defl:N 1093 53% 02-09-2021 09:24 6ccd4450 usr/share/zoneinfo/Berlin 2453 Defl:N 1170 52% 02-09-2021 09:24 7fed6d80 usr/share/zoneinfo/Boulder ... Another ZIP tool that works well with APE is Windows 10 file explorer. But you have to change the extension from .com to .zip temporarily.
- lambda_obrien 6y ago
  Does cosmolibc allow you to run on bare metal? If so, I've got an old project in mind that you may have just made happen, and I'm very very excited to know!
  - collinmanderson 6y ago
    "Actually Portable Executable"s can boot from BIOS if that's what you're wondering. https://justine.lol/ape.html https://justine.lol/ape.html
    - lambda_obrien 6y ago
      Cool, thanks! I missed that before. It makes a project idea I had way more realistic for me to implement.

shrx 6y ago

They also run on AMD ;)

galfarragem 6y ago

Will this rush an optimized version of Electron or even deprecate it?

Nextgrid 6y ago
Technically, I guess you could use this to compile a universal browser binary that would run on all platforms. However, this wouldn't solve any of the problems caused by Electron.
- darkfirefly 6y ago
  Or really any of the problems Electron attempts to solve; the GUI toolkits are still not (generally) not cross-platform in the way Electron is.
- dj_mc_merlin 6y ago
  I think his point was that with Cosmopolitan native binaries one would no longer need Electron to provide cross-platform binaries. There is a lack of good GUI frameworks to build this native Electron replacement with though, nor is it certain the effort would not be too high.
simias 6y ago
Note that packaged electron apps are not portable across OS, so it's orthogonal. What this could do is make the same electron package work on different OS without rebuild by using a portable browser binary. Meanwhile Cosmopolitan doesn't provide a portable GUI toolkit so it can't replace electron on its own. Maybe if you could package Cosmopolitan + Qt or something like that you could end up with something very interesting though.
- cxr 6y ago
  > What this could do is make the same electron package work on different OS without rebuild by using a portable browser binary. Likewise for non-GUI apps, like command-line utilities that are implemented with NodeJS. As prior art, Fabrice Bellard's QuickJS is able to compile JS programs to spit out a native binary that requires no external qjs runtime. If you modified it to output Actual Portable Executable binaries instead, it would be even more portable than the programs installed with `npm install`. I've done some experiments and progressed to writing tooling that approaches that I-want-ridiculous-portability problem differently. It doesn't use APE, so it still requires a separate interpreter (such as NodeJS) capable of executing the compiled output, but the object files it outputs are such that even if you don't have a copy of NodeJS installed on your system (or the right copy of NodeJS installed), then you can piggy back off the interpreter built in to your browser. You get a similar double-click-to-run experience, but when you do double-click to run, it launches in the browser, and you can go from object file to modifiable source trivially, since the object file and the Git repo from which it is built are automorphisms. The downside, if you consider it one, is that it doesn't work with arbitrary NPM modules. You have to really buy in to the scripting language dialect and the development philosophy to get anything out of it. It would be interesting the use the QuickJS+APE strategy above so that there is _no_ reliance on an external interpreter, at the cost of marrying yourself to x86 and introducing some small hurdles in the path to modification and also giving up the safety of the browser sandbox, since you're back to running arbitrary executables.

A12-B 6y ago

What happens when intel goes bust and we all have to switch to new processors

OOPMan 6y ago
A little optimistic about Apples SoCs aren't you? Also...given that AMD exists and the x86 instruction set is not some magic potion ingredients...

munawwar 6y ago

So it's like a compatibly layer like wine except built into the binary. Does this bloat the binary as well?

sgtnoodle 6y ago
The only code that really needs to be duplicated would be the lowest level of userspace code that actually calls into the various operating systems. Things like the guts of "open", "read", "write", etc. Regardless of any particular OS's flavor, they're all going to roughly take in the same arguments and return the same results via some flavor of system call. Each individual function is probably on the order of a few dozen machine instructions. Let's say there's 30 such functions that are implemented and 7 OSes to support. 36307 is about 7500 instructions. The average instruction length on x86-64 is 2-3 bytes. So maybe it ends up around 20KB of code space for a program that uses every possible operating system feature that's implemented? I haven't looked at the implementation. I wonder how it selects between OS implementations. Does it switch on every system call, use a function pointer table, or do some sort of clever in-memory code rewriting?