8 ms·

How Fat Does a Fat Binary Need to Be?

171 points by jart 6y ago

als0 6y ago
There will be other architectures after x86 has died. Mark my words.
- brazzy 6y ago
  Probably developed and run by intelligent rabbits to whom humans are fairy tales.
  - KMag 6y ago
    It wasn't that long ago that programmers had to be reminded that there were non-VAX processors.
deleted 6y ago
[deleted]
cjohnson318 6y ago
I wanted to see if I could use this with Python, so I Googled "python cosmopolitan", and this was the first result: Passion Python Sex Position - Cosmopolitan [0]. I had completely forgotten about that magazine. [0](https://www.cosmopolitan.com/sex-love/positions/a26874/passion-python-sex-position/ https://www.cosmopolitan.com/sex-love/positions/a26874/passi...)
mshockwave 6y ago
Shout out to the cosmopolitan project. It looks super cool. Though I can imagine there might be more ABI issues when you move to other languages like C++
marcodiego 6y ago
Now that there is a possibility that arm machines may become more common, it would be interesting to be able to build multi-arch AppImages. Maybe Ryan Gordon FatELF could be part of it.
ape4 6y ago
Seems impossible on the face of it. Linux binaries have to start with 0x7F ELF and Windows binaries have to start with MZ or ZM so I don't get how it could work with both.
- icedchai 6y ago
  Based on the generated code, you may need to broaden the definition of "binary" to include anything that is executable (shell script.)
  - chungy 6y ago
    It's not really a shell script though, it just looks like one enough to the point that a bourne shell can modify it to be directly executable. I'd probably be most worried about the bourne shell part. It seems like it'll be easy for this to break on any "non-standard" shell that Linux might execute.
    - icedchai 6y ago
      This is true, but the shell is executing it "as if" it were a script, right? Until the binary part takes over...
    - jart 6y ago
      The behavior that Cosmopolitan needs is required by POSIX because UNIX Sixth Edition didn't have shebang lines. It's normally an implementation detail of execve() in C libraries and the behavior is implemented by nearly all shells too. So far the only issue has been that some shells, like zsh, have a binary safety check where it refuses to run scripts that have embedded NUL characters. I worked with the FreeBSD team to get POSIX revised to clarify the correct behavior w.r.t. these restrictions. FreeBSD's /bin/sh has already been patched and zsh is pending upstream. https://github.com/jart/zsh/commit/94a4bc14bb2e415ec3d10cf716512bd3e0d99f48 https://github.com/jart/zsh/commit/94a4bc14bb2e415ec3d10cf71...
      - ashkankiani 6y ago
comex 6y ago
> That's how big a statically-linked executable needs to be to run natively without dependencies on Linux, Mac, Windows, FreeBSD, OpenBSD, NetBSD, and BIOS. * on x86. There was a time period when it was reasonable to just assume everything that mattered was x86. It started sometime in the early 2000s, as non-x86 servers died out and Macs switched to x86. And it ended last November, when Apple announced Macs would switch to ARM. The author has suggested sticking with x86 and handling other architectures using emulation. But x86 is the worst possible architecture to emulate, because its strong memory ordering model means that emulators on other architectures essentially must limit the emulated process to a single core. (Unless you have special hardware support like M1 Macs do… for now.) Also, any hardware CPU architecture is going to be slower to emulate than a purpose-built bytecode – and we already have a good enough bytecode, WebAssembly. Emulating x86 might make sense anyway if you assume that x86 is and will be the norm, and non-x86 an edge case that can afford to take a performance hit. But that sure doesn't seem to be the way the wind is blowing. [1] https://justine.lol/ape.html https://justine.lol/ape.html
- spijdar 6y ago
  I'll admit to being biased up front, as I run a lot of ARM and POWER stuff, so I have a vested interest, but I have to agree that this feels short sighted. ARM at the very least has a big potential to take a much bigger slice of the market, with ARM servers and ARM Macs "finally" coming out in force, while ARM has been dominating smartphone and mobile devices for at least the last, say, 15 years, and has gotten a lot of mindshare through SBCs like the Raspberry Pi. It's a fun project, but I don't think the assumption that everything is x86 is as safe as the author proposes.
- jart 6y ago
  Author here. These binaries will re-exec themselves under Qemu if you run them on different architectures. So they truly are build-once run-anywhere. I think improving performance on other architectures is great. There's nothing that inherently limits the Actually Portable Executable format from baking in native support for other architectures. It's simply a question of focus. What Cosmopolitan does, it does really well. No other C library is able to target multiple operating systems and all the x86 microarchitectures. You can now easily distribute software that can be easily run on the vast majority of PCs and servers. Want it to do more? Consider becoming a sponsor. The impact the project is already having should be all the proof we need that it could be doing much more. In order for that to happen it needs your support.
lostmsu 6y ago
Or you can ship Java or .NET binary (obviously)
refactor_master 6y ago
Using Docker and Python I can only say: very, very fat. The Tensorflow image is something like 5 GB. On the other hand, Docker images is literally the only thing taking up space on my hard drive by now.
trizic 6y ago
This looks great! Combining this with Nim would be extremely powerful. Is it a matter of changing compilers like musl or would some tweaks need to be done for it to work with Nim?
zxspectrum1982 6y ago
Is it possible to build real applications with Cosmopolitan? I'm thinking stuff that requires other dependencies, e .g. build a universal Python interpreter by using Cosmopolitan. Would that be possible?
- jart 6y ago
  It's been formally verified that Cosmopolitan lets you compute anything that's computable. It's able to build real apps too! For example, JavaScript is the most popular language and Cosmopolitan has an embedded JavaScript interpreter. See https://github.com/jart/cosmopolitan/blob/master/examples/hellojs.c https://github.com/jart/cosmopolitan/blob/master/examples/he... where we built a 300kb Actually Portable Executable + ZIP file polyglot that reflectively loads interprted sources from within its own ZIP file structure. You can download the compiled executable here: https://justine.lol/hellojs.com https://justine.lol/hellojs.com It would be trivial to support Python too, along with any other language.
  - sgtnoodle 6y ago
    First of all, cool project! I've done some less ambitious stuff in regards to mashing together scripts and zip archives to create what I'd describe as "dynamically linked python executables", and what you've managed to achieve with compiled code is awe inspiring. Pure computation and file/socket based IO is enough to solve many problems, but I'm curious what the plan is down the road to support more exotic forms of IO like high performance graphics, audio, human input devices, etc. in a portable way? It seems like the fully-statically-linked approach can only get as far as the common subset of all supported platforms, which is probably a tautology, but means that the cosmopolitan library will have to not only provide APIs for any possible type of IO someone might want to do, but also bake in all the userspace code to implement it on every platform. For a lot of things that seems currently very difficult, as many hardware vendors only provide dynamic libraries as their interface. I suppose statically linking isn't mutually exclusive with dynamically loading, though? The other route seems to be targeting a virtual machine of some sort, whether it's running the code and providing a consistent ABI like qemu, or communicating over files/sockets like rendering graphics and sound in a web browser. I get the impression that's the less interesting route to go down though. I appreciate the holistic nature of the project, and it seems like you've approached it with a pragmatic mindset. Myself thinking pragmatically, it just seems like you'll eventually want to at least support opengl or a GUI library or something, to appeal to the types of programs that really benefit most from easy portability.
perl4ever 6y ago
What is the history of the term "fat binary"? I first heard of it as a term for Mac applications that combined 68k and PowerPC code, and wasn't really aware of any wider use. Can you make a program these days that runs natively on M1, x86, PowerPC, and 68k Macs?
- danaliv 6y ago
  Not sure about the origin, but before Mac fat binaries (which were formally called “universal binaries”), NeXTstep had quad-fat binaries with m68k, i386, SPARC, and PA-RISC. Edit: Just realized this all might have been happening at the same time. The ‘90s are a blur!
- saagarjha 6y ago
  I believe so, yes. I've gotten three fourths of the way there, if anyone has the last one (though, it would be a stretch to call them "Macs" at that point) feel free to post it here: https://github.com/saagarjha/dummy_thicc https://github.com/saagarjha/dummy_thicc
nextaccountic 6y ago
How is it possible to build once and run on both Windows and Linux if Windows uses the PE executable format and Linux uses ELF? Is there some kind of trickery to make a file both PE and ELF valid? Doesn't magic numbers conflict?
- wernercd 6y ago
  IL code. .net compiles to intermediate language which runs on a local interpeter. https://en.wikipedia.org/wiki/Common_Intermediate_Language https://en.wikipedia.org/wiki/Common_Intermediate_Language https://www.i-programmer.info/programming/other-languages/933-getting-started-with-il.html https://www.i-programmer.info/programming/other-languages/93...
- jart 6y ago
  APE executables are PE executables that embed a shell script in the MS-DOS stub. The GNU or LLVM linker is configured to generate a printf '\177ELF...' >$0 statement which inserts the ELF or Mach-O header into the first 64-bytes depending on where the executable is run. See https://justine.lol/ape.html https://justine.lol/ape.html and https://raw.githubusercontent.com/jart/cosmopolitan/master/ape/ape.S https://raw.githubusercontent.com/jart/cosmopolitan/master/a...
  - nextaccountic 6y ago
    This is so freaking cool!
  - nextaccountic 6y ago
    .. actually, wait, are you the author? Thanks for open sourcing this! Can I use it to make a Rust executable? Maybe it can post-process the executable built by Rust somehow? (I guess Rust needs to link against the cosmopolitan libc and otherwise not make any syscalls) Maybe it's possible to make a cosmopolitan Rust target, like, x86-unknown-cosmopolitan? Also, does it support x86_64? (and maybe a fat x86/x86_64 binary as well?)
QuadrupleA 6y ago
On the subject of bloated executables, git on windows is epic - it's broken into something like 17 separate executables, 3.1mb each, over 500mb in total.
- glandium 6y ago
  That's why more and more things are builtins now.
- saagarjha 6y ago
  Why?
enriquto 6y ago
I'm in awe at the existence of this thing. Cannot say anything. EDIT: Alright, I found something to say. Everything is stunningly perfect about this project except for one thing: the disturbing "mock greek" (using d as o and m as u, etc). Is it a sort of modern utf8 version of 1337-speek? Maybe it is too ironical to understand? EDIT2: OK, now I get it. The "actually portable executables" use an amazing hack where the PE header itself gets executed as machine code, it turns out to be mostly harmless, and execution is captured from that forward, using other data parts of the header as code. This is the same thing your brain does when reading letters in mixed alphabets. It is confusing when you know both alfabets and try to read on each, but if you squint your eyes, then only the actual shape of the characters matters, and you can read it for its intended meaning. Thus it is a sort of self-referential title which is perfectly appropriate here.
waynesonfire 6y ago
In a world of accidental complexity explosion when you think all hope is lost, a gem like this appears to demonstrate what technological progress looks like.
haps 6y ago
Reminds me of .kkrieger, a full 3D shooter with sounds, textures, enemies, levels, etc. packed in ~96kb. https://en.wikipedia.org/wiki/.kkrieger https://en.wikipedia.org/wiki/.kkrieger
- jart 6y ago
  It's amazing what compression and Perlin's noise generator can do.
- GuB-42 6y ago
  Sizecoding is a major part of the demoscene. .kkrieger may be the most notorious in the game category but 96k is actually on the heavy side for such productions. One other example of a really small game is BootChess, a chess engine in less than 512 bytes to fit in a BIOS boot sector. And you should look what people are fitting into 4k intros nowadays.

171 points by jart 6y ago

als0 6y ago

There will be other architectures after x86 has died. Mark my words.

brazzy 6y ago
Probably developed and run by intelligent rabbits to whom humans are fairy tales.
- KMag 6y ago
  It wasn't that long ago that programmers had to be reminded that there were non-VAX processors.

deleted 6y ago

[deleted]

cjohnson318 6y ago

I wanted to see if I could use this with Python, so I Googled "python cosmopolitan", and this was the first result: Passion Python Sex Position - Cosmopolitan [0]. I had completely forgotten about that magazine. [0](https://www.cosmopolitan.com/sex-love/positions/a26874/passion-python-sex-position/ https://www.cosmopolitan.com/sex-love/positions/a26874/passi...)

mshockwave 6y ago

Shout out to the cosmopolitan project. It looks super cool. Though I can imagine there might be more ABI issues when you move to other languages like C++

marcodiego 6y ago

Now that there is a possibility that arm machines may become more common, it would be interesting to be able to build multi-arch AppImages. Maybe Ryan Gordon FatELF could be part of it.

ape4 6y ago

Seems impossible on the face of it. Linux binaries have to start with 0x7F ELF and Windows binaries have to start with MZ or ZM so I don't get how it could work with both.

icedchai 6y ago
Based on the generated code, you may need to broaden the definition of "binary" to include anything that is executable (shell script.)
- chungy 6y ago
  It's not really a shell script though, it just looks like one enough to the point that a bourne shell can modify it to be directly executable. I'd probably be most worried about the bourne shell part. It seems like it'll be easy for this to break on any "non-standard" shell that Linux might execute.
  - icedchai 6y ago
    This is true, but the shell is executing it "as if" it were a script, right? Until the binary part takes over...
  - jart 6y ago
    The behavior that Cosmopolitan needs is required by POSIX because UNIX Sixth Edition didn't have shebang lines. It's normally an implementation detail of execve() in C libraries and the behavior is implemented by nearly all shells too. So far the only issue has been that some shells, like zsh, have a binary safety check where it refuses to run scripts that have embedded NUL characters. I worked with the FreeBSD team to get POSIX revised to clarify the correct behavior w.r.t. these restrictions. FreeBSD's /bin/sh has already been patched and zsh is pending upstream. https://github.com/jart/zsh/commit/94a4bc14bb2e415ec3d10cf716512bd3e0d99f48 https://github.com/jart/zsh/commit/94a4bc14bb2e415ec3d10cf71...
    - ashkankiani 6y ago

comex 6y ago

> That's how big a statically-linked executable needs to be to run natively without dependencies on Linux, Mac, Windows, FreeBSD, OpenBSD, NetBSD, and BIOS. * on x86. There was a time period when it was reasonable to just assume everything that mattered was x86. It started sometime in the early 2000s, as non-x86 servers died out and Macs switched to x86. And it ended last November, when Apple announced Macs would switch to ARM. The author has suggested sticking with x86 and handling other architectures using emulation. But x86 is the worst possible architecture to emulate, because its strong memory ordering model means that emulators on other architectures essentially must limit the emulated process to a single core. (Unless you have special hardware support like M1 Macs do… for now.) Also, any hardware CPU architecture is going to be slower to emulate than a purpose-built bytecode – and we already have a good enough bytecode, WebAssembly. Emulating x86 might make sense anyway if you assume that x86 is and will be the norm, and non-x86 an edge case that can afford to take a performance hit. But that sure doesn't seem to be the way the wind is blowing. [1] https://justine.lol/ape.html https://justine.lol/ape.html

spijdar 6y ago
I'll admit to being biased up front, as I run a lot of ARM and POWER stuff, so I have a vested interest, but I have to agree that this feels short sighted. ARM at the very least has a big potential to take a much bigger slice of the market, with ARM servers and ARM Macs "finally" coming out in force, while ARM has been dominating smartphone and mobile devices for at least the last, say, 15 years, and has gotten a lot of mindshare through SBCs like the Raspberry Pi. It's a fun project, but I don't think the assumption that everything is x86 is as safe as the author proposes.
jart 6y ago
Author here. These binaries will re-exec themselves under Qemu if you run them on different architectures. So they truly are build-once run-anywhere. I think improving performance on other architectures is great. There's nothing that inherently limits the Actually Portable Executable format from baking in native support for other architectures. It's simply a question of focus. What Cosmopolitan does, it does really well. No other C library is able to target multiple operating systems and all the x86 microarchitectures. You can now easily distribute software that can be easily run on the vast majority of PCs and servers. Want it to do more? Consider becoming a sponsor. The impact the project is already having should be all the proof we need that it could be doing much more. In order for that to happen it needs your support.

lostmsu 6y ago

Or you can ship Java or .NET binary (obviously)

refactor_master 6y ago

Using Docker and Python I can only say: very, very fat. The Tensorflow image is something like 5 GB. On the other hand, Docker images is literally the only thing taking up space on my hard drive by now.

trizic 6y ago

This looks great! Combining this with Nim would be extremely powerful. Is it a matter of changing compilers like musl or would some tweaks need to be done for it to work with Nim?

zxspectrum1982 6y ago

Is it possible to build real applications with Cosmopolitan? I'm thinking stuff that requires other dependencies, e .g. build a universal Python interpreter by using Cosmopolitan. Would that be possible?

jart 6y ago
It's been formally verified that Cosmopolitan lets you compute anything that's computable. It's able to build real apps too! For example, JavaScript is the most popular language and Cosmopolitan has an embedded JavaScript interpreter. See https://github.com/jart/cosmopolitan/blob/master/examples/hellojs.c https://github.com/jart/cosmopolitan/blob/master/examples/he... where we built a 300kb Actually Portable Executable + ZIP file polyglot that reflectively loads interprted sources from within its own ZIP file structure. You can download the compiled executable here: https://justine.lol/hellojs.com https://justine.lol/hellojs.com It would be trivial to support Python too, along with any other language.
- sgtnoodle 6y ago
  First of all, cool project! I've done some less ambitious stuff in regards to mashing together scripts and zip archives to create what I'd describe as "dynamically linked python executables", and what you've managed to achieve with compiled code is awe inspiring. Pure computation and file/socket based IO is enough to solve many problems, but I'm curious what the plan is down the road to support more exotic forms of IO like high performance graphics, audio, human input devices, etc. in a portable way? It seems like the fully-statically-linked approach can only get as far as the common subset of all supported platforms, which is probably a tautology, but means that the cosmopolitan library will have to not only provide APIs for any possible type of IO someone might want to do, but also bake in all the userspace code to implement it on every platform. For a lot of things that seems currently very difficult, as many hardware vendors only provide dynamic libraries as their interface. I suppose statically linking isn't mutually exclusive with dynamically loading, though? The other route seems to be targeting a virtual machine of some sort, whether it's running the code and providing a consistent ABI like qemu, or communicating over files/sockets like rendering graphics and sound in a web browser. I get the impression that's the less interesting route to go down though. I appreciate the holistic nature of the project, and it seems like you've approached it with a pragmatic mindset. Myself thinking pragmatically, it just seems like you'll eventually want to at least support opengl or a GUI library or something, to appeal to the types of programs that really benefit most from easy portability.

perl4ever 6y ago

What is the history of the term "fat binary"? I first heard of it as a term for Mac applications that combined 68k and PowerPC code, and wasn't really aware of any wider use. Can you make a program these days that runs natively on M1, x86, PowerPC, and 68k Macs?

danaliv 6y ago
Not sure about the origin, but before Mac fat binaries (which were formally called “universal binaries”), NeXTstep had quad-fat binaries with m68k, i386, SPARC, and PA-RISC. Edit: Just realized this all might have been happening at the same time. The ‘90s are a blur!
saagarjha 6y ago
I believe so, yes. I've gotten three fourths of the way there, if anyone has the last one (though, it would be a stretch to call them "Macs" at that point) feel free to post it here: https://github.com/saagarjha/dummy_thicc https://github.com/saagarjha/dummy_thicc

nextaccountic 6y ago

How is it possible to build once and run on both Windows and Linux if Windows uses the PE executable format and Linux uses ELF? Is there some kind of trickery to make a file both PE and ELF valid? Doesn't magic numbers conflict?

wernercd 6y ago
IL code. .net compiles to intermediate language which runs on a local interpeter. https://en.wikipedia.org/wiki/Common_Intermediate_Language https://en.wikipedia.org/wiki/Common_Intermediate_Language https://www.i-programmer.info/programming/other-languages/933-getting-started-with-il.html https://www.i-programmer.info/programming/other-languages/93...
jart 6y ago
APE executables are PE executables that embed a shell script in the MS-DOS stub. The GNU or LLVM linker is configured to generate a printf '\177ELF...' >$0 statement which inserts the ELF or Mach-O header into the first 64-bytes depending on where the executable is run. See https://justine.lol/ape.html https://justine.lol/ape.html and https://raw.githubusercontent.com/jart/cosmopolitan/master/ape/ape.S https://raw.githubusercontent.com/jart/cosmopolitan/master/a...
- nextaccountic 6y ago
  This is so freaking cool!
- nextaccountic 6y ago
  .. actually, wait, are you the author? Thanks for open sourcing this! Can I use it to make a Rust executable? Maybe it can post-process the executable built by Rust somehow? (I guess Rust needs to link against the cosmopolitan libc and otherwise not make any syscalls) Maybe it's possible to make a cosmopolitan Rust target, like, x86-unknown-cosmopolitan? Also, does it support x86_64? (and maybe a fat x86/x86_64 binary as well?)

QuadrupleA 6y ago

On the subject of bloated executables, git on windows is epic - it's broken into something like 17 separate executables, 3.1mb each, over 500mb in total.

glandium 6y ago
That's why more and more things are builtins now.
saagarjha 6y ago
Why?

enriquto 6y ago

I'm in awe at the existence of this thing. Cannot say anything. EDIT: Alright, I found something to say. Everything is stunningly perfect about this project except for one thing: the disturbing "mock greek" (using d as o and m as u, etc). Is it a sort of modern utf8 version of 1337-speek? Maybe it is too ironical to understand? EDIT2: OK, now I get it. The "actually portable executables" use an amazing hack where the PE header itself gets executed as machine code, it turns out to be mostly harmless, and execution is captured from that forward, using other data parts of the header as code. This is the same thing your brain does when reading letters in mixed alphabets. It is confusing when you know both alfabets and try to read on each, but if you squint your eyes, then only the actual shape of the characters matters, and you can read it for its intended meaning. Thus it is a sort of self-referential title which is perfectly appropriate here.

waynesonfire 6y ago

In a world of accidental complexity explosion when you think all hope is lost, a gem like this appears to demonstrate what technological progress looks like.

haps 6y ago

Reminds me of .kkrieger, a full 3D shooter with sounds, textures, enemies, levels, etc. packed in ~96kb. https://en.wikipedia.org/wiki/.kkrieger https://en.wikipedia.org/wiki/.kkrieger

jart 6y ago
It's amazing what compression and Perlin's noise generator can do.
GuB-42 6y ago
Sizecoding is a major part of the demoscene. .kkrieger may be the most notorious in the game category but 96k is actually on the heavy side for such productions. One other example of a really small game is BootChess, a chess engine in less than 512 bytes to fit in a BIOS boot sector. And you should look what people are fitting into 4k intros nowadays.