16 ms·
How fat does a fat binary need to be? (2021)
- j_leboulanger 4y agoI tried on MacOS and it does not work (exec format error)
- diego_moita 4y agoI suspect your CPU is Silicon (ARM based), not Intel. The binary code is for Intel processors. Silicon chips have a completely different instruction set. For this same reason Android, iOS and RTos (for embeded systems) are not included among the targets.
- BeefWellington 4y agoThis highlights the ongoing problem of article titles needing to be eye-grabbing but usually inaccurate/misleading or more complex than a simple title allows for.
- sph 4y agoIt's a title, it cannot be a paragraph long to be pedantically clear and unambiguous in what to expect. "Website that lets you download a 1KB long executable that can be run on 7 _kernels_ and x86_64 architectures, but actually it's only been tested on Intel and will probably not work on Android even if technically it's still a Linux kernel, also Linux < 2.6 is unsupported so YMMV"
- BeefWellington 4y agoMy comment (which I could have made more clear) was intended to be responding to the idea of quoting the title and then claiming it's misleading. I agree fully with what you're saying.
- Folcon 4y agoI had this also happen because macos now uses zsh as the default shell and this approach is bash compatible. So you needed additional steps for macos. Not sure if that has been recently resolved?
- alganet 4y agoThere is a fix: https://github.com/jart/zsh/commit/94a4bc14bb2e415ec3d10cf716512bd3e0d99f48 https://github.com/jart/zsh/commit/94a4bc14bb2e415ec3d10cf71... Not sure if it was merged. Her approach is not bash-only, zsh is the odd one out on this one.
- jart 4y agoIt was merged: https://github.com/zsh-users/zsh/commit/326d9c203b3980c0f841bc62b06e37134c6e51ea https://github.com/zsh-users/zsh/commit/326d9c203b3980c0f841... We patched fish too! Plus NetBSD sh and FreeBSD sh.
- ok_dad 4y agoIf it's a zsh problem, you need to do something like: bash -c './redbean.com' or whatever your executable is called. ZSH is doing something incorrectly that prevents running these binaries.
- theta_d 4y agoPedantry incoming: Intel processors are also made of silicon. "Apple Silicon" is just a way to say chips designed by Apple vs a 3rd party like Intel. The instructions are Arm vs x86. This has nothing to do with the underlying substrate of silicon.
- kelnos 4y agoI know that I have a general bias against Apple, but it has generally annoyed me that Apple has decided to take a generic industry term like "$FOO Silicon" and turn it into a marketing title.
- andsoitis 4y agoTo add more color. There are two kinds of ARM licenses. One is for chipset designs (e.g. Cortex - https://en.wikipedia.org/wiki/ARM_Cortex-A https://en.wikipedia.org/wiki/ARM_Cortex-A) that you get fabricated to get your CPUs. The other ARM license is an ISA license. You get no chip design. All you get is the instruction set architecture but you have to design the chip yourself. That's what Apple has been doing.
- _joel 4y agoWill it run via rosetta?
- paulclinger 4y agoYes, there have been several recent changes to make Cosmopolitan executables run under Rosetta.
- j_leboulanger 4y agoNo I'm using an intel MBP.
- edualm 4y agoWorks for me both under rosetta and native arm64 (macOS 12.2.1, M1 Max) - both by double-clicking the executable and using zsh. Only thing I had to do was rename the file to open it in Finder first because of gatekeeper requirements. arch -x86_64 ./hello2 hello world ./hello2 hello world
- jeroenhd 4y agoTechnically the executable only runs on 6 operating systems as the 7th option basically turns the application into an operating system running on bare metal. Justine's impressive work on this subject makes me wonder if it's possible to build truly native applications using more complete toolsets and more modern languages. Multiplatform toy programs have existed for ages, but this SDK of sorts makes it possible to write executables that don't require manually tricking the compiler or adjusting your code for portability with inline assembly. As long as the program is statically compiled and contains the necessary drivers for things like syscalls, this should be perfectly possible, though the current implementation isn't exactly user friendly, relying on giant headers like https://justine.lol/cosmopolitan/cosmopolitan.h https://justine.lol/cosmopolitan/cosmopolitan.h, statically linked components like https://justine.lol/cosmopolitan/crt.o https://justine.lol/cosmopolitan/crt.o and https://justine.lol/cosmopolitan/ape.o https://justine.lol/cosmopolitan/ape.o, and other such tweaks. I'd love a Go/Rust compile target that just works regardless of operating system. As an added bonus, the bare metal nature would be excellent for platforms like Firecracker to run services without even needing to boot a kernel!
- ElectricalUnion 4y agoA target like Rust x86_64-linux-unknown-cosmo ? https://news.ycombinator.com/item?id=32260619 https://news.ycombinator.com/item?id=32260619 https://ahgamut.github.io/2022/07/27/ape-rust-example/ https://ahgamut.github.io/2022/07/27/ape-rust-example/
- jeroenhd 4y agoThis is great, I must've missed it! Thanks for the link!
- jart 4y agoThere's so much great stuff going on in the cosmoverse that it's hard to keep track of it all! We recently started https://github.com/shmup/awesome-cosmopolitan https://github.com/shmup/awesome-cosmopolitan for that reason.
- 4y ago
- rocqua 4y agoTitle seems wrong. I think this was intended to be "12 Kb executable runs natively on 7 operating systems". The HN suggestion of copying the article title would be: "How Fat Does a Fat Binary Need To Be?"
- unwind 4y agoI believe HN filters out numbers at the start of the title.
- chrismorgan 4y agoWhen HN mangles your title, you can restore it to sanity by editing the item: it only changes it on creation, not edition.
- moffkalast 4y agoI wonder what's the reasoning behind that, it sounds pretty braindead.
- capableweb 4y agoFiltering "listicles", articles with titles like "6 Secrets to Achieving Work-Life Balance", "23 Amazing Moments From The Harry Potter Movies", "5 Steps to Booking a Cheap Flight Online" and similar.
- moffkalast 4y agoBut what if number 4 shocks you? /s
- dang 4y agoYes, but the original title shouldn't have been rewritten in the first place since it was neither misleading nor linkbait (from https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html: "Please use the original title, unless it is misleading or linkbait; don't editorialize."). I've fixed it now.
- deleted 4y ago[deleted]
- fsiefken 4y agoIt would be an interesting exercise to use executable compression on the fat binary to make it less fat that also transparently runs on the same 7 operating systems.
- chrismorgan 4y agoPrevious discussion a year and a half ago: https://news.ycombinator.com/item?id=26103769 https://news.ycombinator.com/item?id=26103769
- dang 4y agoThanks! Macroexpanded: How Fat Does a Fat Binary Need to Be? - https://news.ycombinator.com/item?id=26103769 https://news.ycombinator.com/item?id=26103769 - Feb 2021 (67 comments)
- swills 4y agoGo has it's own run time, right? Why doesn't it do this?
- georgia_peach 4y agohttps://github.com/jart/cosmopolitan/issues/140 https://github.com/jart/cosmopolitan/issues/140 https://github.com/golang/go/issues/51900 https://github.com/golang/go/issues/51900 & from what I gather, APE is tightly-bound to x86_64, & depends on an embedded emulator for other architectures: > All we have to do embed an ARM build of the emulator above within our x86 executables, and have them morph and re-exec appropriately, similar to how Cosmopolitan is already doing doing with qemu-x86_64, except that this wouldn't need to be installed beforehand. The tradeoff is that, if we do this, binaries will only be 10x smaller than Go's Hello World, instead of 100x smaller. could still be useful as an optional build target though... (i.e. GOOS=cosmopolitan)
- jart 4y agoPlease support our runtime proposal https://github.com/golang/go/issues/51900 https://github.com/golang/go/issues/51900 We're trying to add Cosmopolitan to Go's list of supported runtimes!
- georgia_peach 4y agoThis thing of yours is quite an accomplishment. Even so, I hope it goes no further. If it does (& it probably will, because it is convenient), most of the computing world will have stacked yet another sub-optimal low-level mono-culture on top of the one that was already there. From the robustness/survivability perspective, the mono-cultures we have are bad enough.
- jart 4y agoYes we're already stuck with all these mono-cultures, so why not just define the union of all these mono-cultures into a single build target?
- stiray 4y agoIf I remember correctly from the DOS times, the limit for com files was 64KB. I haven't tried to compile one for ages, is this any different on Windows NT+ (I would try but I dont have a single windows machine at home)?
- petercooper 4y agoReady to be corrected, as I haven't worked with DOS or Windows for decades, but I seem to recall ".com" is merely used for familiarity/compatibility and hasn't meant anything about the underlying format for a very long time now. The Wikipedia page for .com suggests that 64 bit Windows NT+ systems can't run the original style .com files anymore due to lacking the MS-DOS emulation subsystem, so the point may be moot.
- TazeTSchnitzel 4y agoAlready thirty years ago there were DOS executables with a .COM extension that weren't actually in the COM format. COMMAND.COM in later versions of DOS is an example.
- jhoechtl 4y agoI think you are right. Totally cheating would be to integrate the DOS application functionally into the Windows DOS stub. Instead of printing "This application requires Microsoft Windows" it could even use overlays to circumvent the 64kB com restriction.
- tripa 4y agoThe limit for a .COM's useful payload is 64KiB, but that wouldn't necessarily prevent the actual file from being bigger, with either truncate or wrapover semantics.
- deleted 4y ago[deleted]
- yakubin 4y agoThat's a quasi-dupe of <https://news.ycombinator.com/item?id=32642874 https://news.ycombinator.com/item?id=32642874>.
- capableweb 4y agoDon't think so, although they are complementary. Submission you link is more like a reference/article while the submission we're commenting on, is more like an application/demonstration of said technique.
- tromp 4y agoMinor nitpick: I'd prefer to see size denoted as 12kB (or 12 KB) rather than 12kb, where I have to wonder whether it means kilobit or kilobyte. Note that Justine has also worked on languages whose programs consists of bits [1]. [1] https://justine.lol/lambda/ https://justine.lol/lambda/
- remram 4y agoAlso if I read "kb executable", I assume one kb. Just like when I read "backpack computer", I assume it fits in one backpack. It's 12 kB, aka 96 kb. The HN submission's title is not even the article's title, which is "How Fat Does a Fat Binary Need To Be?"
- ThePadawan 4y agoAs pointed out in https://news.ycombinator.com/item?id=32649569 https://news.ycombinator.com/item?id=32649569 , this is probably HN autoremoving numbers from the start of titles to combat blog spam titles like "12 reasons to switch to XYZ".
- titzer 4y agoThat bit-encoded lambda calculus is nuts!
- shrimp_emoji 4y agoIf you get a scientific education, the importance of clearly noting your units is drilled into you. Programmers with their non-STEM laxity smh :p
- deleted 4y ago[deleted]
- jart 4y agoAs you wish. kB for kibibytes.
- laumars 4y agoThat’s more likely HN reformatting titles. HN auto-title cases submissions titles and removes any numbers prefixing the title string. So “7 KB” would be changed to “Kb”.
- mysterydip 4y agoCan this be used for something that displays graphics? Or does the runtime size balloon with supporting different functionality on each system?
- nudpiedo 4y agoI am not an expert on the topic but red language achieves that with a 1MB self contained compiler, so I guess with some additional bloat should be possible (and a lot of work)
- ok_dad 4y agoYou can run a web app locally using https://redbean.dev/ https://redbean.dev/ and open it in a browser. That's the graphics support so far.
- gwbas1c 4y agoIt says Windows, but is it x86 or x64? Can it make a binary that's both x86 and x64 for Windows? Is that even possible?
- poizan42 4y ago> It says Windows, but is it x86 or x64? x64 > Can it make a binary that's both x86 and x64 for Windows? Seems to be a PE32+, so it won't run on 32-bit Windows. > Is that even possible? Yes, it should be. A 32-bit process on 64-bit Windows can run 64-bit code by changing the segment selector - make a far call or jump to 0x33:<address>. See [0] for more details. There are some caveats though. kernel32.dll refuses to be loaded as both the 32-bit and 64-bit versions in the same executable. So the executable can probably only import ntdll. One thing one might do is just compile both a 32-bit and 64-bit version of the actual program as dll's and embed them inside the launcher and then extract and load them using only api's in ntdll. In the case of APE it would probably just embed an x64-on-x86 emulator and run the executable in that in case it's being run on x86. Another solution might be to create a .NET AnyCPU executable which will run as 64-bit on 64-bit versions of Windows. [0]: https://www.malwaretech.com/2014/02/the-0x33-segment-selector-heavens-gate.html https://www.malwaretech.com/2014/02/the-0x33-segment-selecto...
- gwbas1c 4y ago> Another solution might be to create a .NET AnyCPU executable which will run as 64-bit on 64-bit versions of Windows. But will that run on all the other OSes?
- poizan42 4y agoIf you just turn the PE part into a .NET executable and keep the rest as it is, yes. Windows will see a .NET executable and the other OSes will see a shell script just like it works now (and a BIOS will still see the boot sector)
- TillE 4y agoI don't think so, the PE headers are different. If you really wanted to, you could hack your own "fat binary" format by packing the 64-bit EXE as a resource within an x86 binary that detects the current architecture, but that's not very exciting.
- nudpiedo 4y agoI actually like a lot the idea, and it’s impressive, however nowadays there is also the challenge of many consumer architectures on the commodity desktops, not just x86 like it used to be. I guess that, and shared libraries ABI, is the biggest challenge now.
- TruthWillHurt 4y agoMate, you gotta relax a bit. You're making the rest of us look bad.
- cauefcr 4y agoJust don't compare against tour-de-forces from these geniuses, much easier (and what I do).
- civilized 4y agoWas the 12 removed by that horrible bookmarklet that automatically removes words it doesn't like?
- avg_dev 4y ago> Is the Cosmopolitan Runtime lean and mean like Go's 2mb Hello World executables? I laughed at this. I am a Go programmer, FWIW :) https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=%22strip%20hello%22&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
- svnpenn 4y ago> strip hello package main func main() { println("hello world") } then: go build -ldflags -s -gcflags all=-l hello.go ls hello result: Mode LastWriteTime Length Name ---- ------------- ------ ---- -a---- 2022-08-30 7:47 PM 747520 hello.exe
- rep_lodsb 4y agoObviously Bill Gates was wrong: 640K isn't even enough to print hello world /s
- encryptluks2 4y agoI don't understand what the issue is with cross-compiling software, especially for different architectures. This seems more geared for running a binary on the same architecture across multiple different operating systems. This is great and all until you use some API that isn't the same on those operating systems. I imagine once you see something like this attempting to use comparable features and APIs as Go or something else, that you'll see the bloat increase to be equal in size if not more.
- diffxx 4y ago> I don't understand what the issue is with cross-compiling software, especially for different architectures. Agreed. The cosmopolitan project is incredible hacking and I appreciate the author's ingenuity, but this has always felt a little bit to me like a solution in search of a problem. How is cosmopolitan software intended to be distributed? If the answer is on the internet, I don't see how a single fat binary helps the end user dramatically more than separate download links. For the developer, cross compilation can be automated as part of deployment. The platform specific binaries should both be smaller than the universal ape binaries and won't resort to hackery to work properly.
- jart 4y agoBecause end users use multiple operating systems these days. Also, no. The operating system integration code is tiny in the grand scheme of things. Consider Fabrice Bellard's JavaScript interpreter. If I build it for six operating systems, then it's 782kB: master jart@nightmare:~/cosmo2$ ls -hal o/tiny/third_party/quickjs/qjs.com -rwxr-xr-x 1 jart jart 782K Aug 30 10:53 o/tiny/third_party/quickjs/qjs.com If I build it for just Linux, then it's 710kB: master jart@nightmare:~/cosmo2$ ls -hal o/tinylinux/third_party/quickjs/qjs.com -rwxr-xr-x 1 jart jart 710K Aug 30 10:52 o/tinylinux/third_party/quickjs/qjs.com That's a 72kB difference, which includes all those Windows NT hacks and bare metal support. Why not just stuff it in there and be done with it? That way you only have to list one file on your download site.
- ghoward 4y ago
- kubb 4y agoThis is really cool! I have a question for people who know this stuff: what's the "f.▼ä" pattern that appears at the end of these binaries? It looks intriguing...
- jart 4y agoThat's a fat nop, or cs nopw 0x0(%rax,%rax,1) to be precise. .byte 0x66,0x2e,0x0f,0x1f,0x84,0x00,0x00,0x00 # f.▼ä .byte 0x00,0x00 ld.bfd uses them to fill empty spaces. See also `o//tool/decode/x86opinfo.com 66 2e 0f 1f 84 00 00 00 00 00` and https://justine.lol/dox/modrm.txt https://justine.lol/dox/modrm.txt and https://justine.lol/dox/geek.txt https://justine.lol/dox/geek.txt
- dietr1ch 4y agoWhy is so much padding needed? Once I remove support for Windows and Bare Metal it seems that it might be possible to halve the binary size if you can remove the legal section (or at least change it for a link + checksum?) and the padding. BTW, once I enable windows I think there's padding at the end of the binary section that's hard to notice, it might be good to replace it with something visible. I think it's coming from `ape_idata_iatend`.
- rcgorton 4y ago
- pdimitar 4y agoJustine's work on all this has been hugely inspiring and a big breath of fresh air in an area that always kicks the can of reducing final binary sizes down the road. That's one of my very few gripes with Rust: compared to what Zig / C / C++ can produce, Rust's release binaries are quite huge. There's likely a lot of possible work to be done in the area of tree shaking and generally only leaving in what's actually used in the final binary. I hope this becomes a priority soon-ish.
- jart 4y agoRust adopted Node's quadratic dependency model, so Rust binaries are going to be O(n^2) larger and tree shaking isn't going to help unfortunately.
- pdimitar 4y agoWhy wouldn't it help? Surely this (and others) method do static analysis on what's actually used in the final program, no? Example: you use a crate with 13 functions. You only use 1 but it depends on 2 others in the same crate that don't depend on anything else. Why shouldn't the end result only compile 3 of the 13 functions? Or maybe I am misunderstanding or I am badly informed (not a compiler / linker engineer so I'm likely with naive assumptions) -- apologies if so. I understand it's not easy, I am just a bit surprised that Rust's team is not chasing this a bit more aggressively, let's say. It puts a slight stain on an otherwise excellent project.
- jart 4y agoWell it helps, but it'd be like spooning water out of a bucket when a faucet is pouring more into it. The NPM dependency model solves the diamond dependency problem by schlepping in multiple versions of the same package. C / C++ / Java (and I assume Zig too) require a single version of any given package. The way Rust and NPM do things adds a whole dimension of complexity to what needs to be shaken. It's sort of like a devil's bargain because it helps their ecosystems grow really fast, but it creates a lot of bloat over the long run. Rust is still relatively new so binary sizes are still pretty good compared to what we might witness in the future. So enjoy the golden age of Rust while it's golden!
- kweingar 4y agoLots of nits being picked here, but this project is extremely cool! Great work jart! I hope this catches on.