15 ms·

Actually Portable Executables

683 points by krab 6y ago

krab 6y ago
I just saw this tweet - https://twitter.com/JustineTunney/status/1365805503561957376 https://twitter.com/JustineTunney/status/1365805503561957376 Cosmopolitan is going to be a big thing.
- andai 6y ago
  >We've just created a 336kb Lua interpreter binary that runs on seven operating systems, thanks to Cosmopolitan Libc. https://ahgamut.github.io/c/2021/02/27/ape-cosmo/ https://ahgamut.github.io/c/2021/02/27/ape-cosmo/
  - svenfaw 6y ago
    Awesome! But why no download link?
    - pfortuny 6y ago
      Because as OP says it is not totally complete and lacks some parts.
    - ahgamut 6y ago
      You can compile it yourself if you like. I created a fork of the Lua Github mirror and made the necessary changes to compile with Cosmopolitan: https://github.com/ahgamut/lua/tree/cosmopolitan https://github.com/ahgamut/lua/tree/cosmopolitan
- poc43 6y ago
  Likely only if it’s forked away from anything to do with its current maintainer, for reasons only a Google search away: http://valleywag.gawker.com/why-does-google-employ-a-pro-slavery-lunatic-1612868507 http://valleywag.gawker.com/why-does-google-employ-a-pro-sla... That’s the problem with trolling like that: once you do something cool that’s unrelated and try to move on it’s not clear if you’re genuine about anything any more. The history there will absolutely make larger interests hesitant to associate, contribute or integrate in something like, say, LLVM (regardless of the merit or lack thereof), particularly in today’s less forgiving culture. Note that I’m not sharing an opinion on the content. It’s a metaobservation about where we are, for better or worse. That’s on page 1 of my Google search for “wow, who’s this amazing coder?”, so my path is the same anyone else would take doing due diligence on getting involved.
  - anthk 6y ago
    >Rather than inclusionary leftism. Traditional left didn't give a shit on gender, race or whatever bullshit you think you should declare yourself "inferior" because of an excuse. Just work and equality. I hate both poshy techno-aristocrats (I would kick them out to Somalia or a Mexico narco shithole, to see what they can do without social support or a state backed police), and SWJs with far more points in common with fascism (Fuck that so-called cultural appropiation) than the common worker.
    - poc43 6y ago
      I mean, you’re engaging with the espoused opinion of someone who simultaneously advocated class warfare and drew Google payroll, so you may as well be arguing with a figment of our collective imagination. If that’s not obvious by nature of how incongruous it is I fear for the Internet.
  - 1996 6y ago
    > Likely only if it’s forked away from anything to do with its current maintainer, And exactly why should I care about the current maintainer? Except maybe to say "thank you"? Let's put the question upside down: what have you (or the gawker journalist) brought to me? I mean, besides your outraged opinions (which anyone is entitled to have!) On one hand, you have a wonderful technical creation by someone asking nothing in return. On the other, you are providing... hatred towards the author? On a 2 hours old account? I'm sorry if she breaks the kumbaya illusion, but people are not equal. Some can't escape the working class, or the welfare class - not due to any personal failure or limits, but due to societal indoctrination. Some other people break free, and then bring gifts of the gods to mankind. They are called innovators, entrepreneurs - the name vary. But you know one when you see one. There is nothing is cosmopolitan that was "impossible" to achieve by anyone dedicated, even years before. There's nothing magic- except in the mind of the author, that associated the right pieces together. > The history there will absolutely make larger interests hesitant to associate, contribute or integrate If anything, your comment make me less likely to associate with you! > particularly in today’s less forgiving culture. I couldn't care less. The less forgiving culture is a problem for those who need to work ("will code for food" as they said during the dotcom crisis) or integrate with the rest of society. Unfortunately, that means most of the people here, especially those working for the FANGs. But there are some of us who just don't care - except to lament that, when shown the way to escape, most geeks double down on their own mistakes, and remain chained to their FANGs masters. So please go on attacking the author if that makes you happy, while the rest of us admire the creativity and new things made possible by cosmopolitan (and make money out of that too!)
ers35 6y ago
Maybe Cosmopolitan could be used with luastatic[1] to compile a Lua program to an "Actually Portable Executable": CC="" luastatic main.lua # Compile main.luastatic.c with Cosmopolitan Libc [1] https://github.com/ers35/luastatic https://github.com/ers35/luastatic
zbendefy 6y ago
Could webassembly be also used for this?
- AstralStorm 6y ago
  That's not as portable as you need an interpreter.
- iFreilicht 6y ago
  Depends on what you mean by “using”. It is conceivably possible to compile a webassembly interpreter like wac[1] with cosmopolitan, which would then run on all OSes and bare metal, yes. [1]: https://github.com/kanaka/wac https://github.com/kanaka/wac
  - deleted 6y ago
    [deleted]
- _roqj 6y ago
  Yes it can! Wasm3 is the fastest WebAssembly interpreter. Since they recently adopted Cosmopolitan it became the most universal one too. Check out the 150kb APE binary they've got on the release page. https://github.com/wasm3/wasm3/releases/tag/v0.4.8 https://github.com/wasm3/wasm3/releases/tag/v0.4.8
abhinav22 6y ago
Can somebody explain in layman’s terms what this is all about? I tried my best reading the other thread, but it went well above me.
- Gaelan 6y ago
  Windows, macOS, Linux, and BSD all (usually) run on Intel processors, so in theory it should be possible to write a program that runs on all three. However, despite ultimately using the same instruction sets, the four OSes use different formats to store metadata about the programs, and have different ways for programs to communicate with the operating system. Justine Tunney created two things: 1) Actually Portable Executables, a clever way of formatting a program so that all four OSes interpret it a valid program in their own format. 2) Cosmopolitan libc, a library for communicating with the OS that handles each OS's interface, allowing programs to work on all four.
  - abhinav22 6y ago
    Thank you (and thanks to everybody else for their helpful answers). Your note on intel processors meaning in theory programs should run on all operating systems was most helpful and got me across the line - now I get it!
    - jart 6y ago
      That's pretty much it. Cosmopolitan Libc is the simplest project that could have happened decades ago. Anyone could have built it. There's never been a technical reason why an x86 binary can't run on all x86 operating systems. It's just that traditionally only operating systems publish C libraries, and they have no incentive to spend money supporting their competitors. So it's the kind of project that could only happen if it was done by an indie developer just trying to have fun.
      - fctorial 6y ago
        Another reason is that it requires fairly thorough knowledge of 7 different programming environments. Most people don't understand 1.
jart 6y ago
Cosmopolitan Libc author here. When I saw this article earlier today, it put a big smile on my face, because I would have thought there'd be so many more issues than there turned out to be! For additional context, we've got a GitHub issue that's tracking progress on the changes that need to be made based on what we learned: https://github.com/jart/cosmopolitan/issues/61 https://github.com/jart/cosmopolitan/issues/61
- ducktective 6y ago
  Any recommendations to somebody who wants to learn C "properly" in $CURRENT_YEAR?
  - unkulunkulu 6y ago
    $CURRENT_BEST_BOOK Sorry :)
  - jtvjan 6y ago
    I really liked "The C Programming Language (second edition)" by Kernighan & Ritchie. It doesn't teach C99 and later features, but if you want to write portable code you're not using those anyway.
    - sillysaurusx 6y ago
      K&R is a horrible way to learn C. It's the best way to make someone run in the other direction, though. It's like telling someone that SICP is the best way to learn Scheme. Maybe, but lots of people learn differently. Unfortunately I don't have good suggestions myself, but at least they won't feel bad if K&R isn't their cup of tea.
    - e12e 6y ago
      From various discussions on Hn, I think the consensus is most people really like k&r as a technical book (it's concise, clear and a fun read) - but everyone working with C says its a terrible book for learning modern C. I tried to refresh my memory from previous threads, and I think the general trend has been to recommend (as seen in sibling comments): https://modernc.gforge.inria.fr/ https://modernc.gforge.inria.fr/ https://nostarch.com/Effective_C https://nostarch.com/Effective_C I've also seen the older: https://www.oreilly.com/library/view/21st-century-c/9781449344382/ https://www.oreilly.com/library/view/21st-century-c/97814493... Mentioned. There's also Architecture of Open Source Applications, which can help with starting to read some larger code bases, some of which are in C: http://aosabook.org/en/index.html http://aosabook.org/en/index.html And there's a general recommendation to go read the source code of the Redis cache/db. Finally i came across a mention of this short article on gdb (nb: mention of TUI text ui should probably have been in the top, not a foot note): https://www.recurse.com/blog/5-learning-c-with-gdb https://www.recurse.com/blog/5-learning-c-with-gdb I feel I'm missing a book that has come up often, but can't think of which one. I did like zed Shaws learn c the hard way, but I'm afraid it's getting a bit long in the tooth: https://learncodethehardway.org/c/ https://learncodethehardway.org/c/
antman 6y ago
Portable Lus is a great idea. By using Cosmopilitan does this also run on baremetal machines?
antman 6y ago
Portable Lua is a great idea. By using Cosmopilitan does this also run on baremetal machines? Portable micropython might also be a low hanging fruit target.
ngcc_hk 6y ago
This actually come as a surprise. Just to ask as a testing case - can we have an actually portable (common) lisp/python interpreter with at least terminal interface? Can the terminal have something like VT-100 or other basic interface? and Can we even run javascript to do screen related items ... like https://github.com/d3/d3 https://github.com/d3/d3
- matheusmoreira 6y ago
  Yes. She built a javascript interpreter with cosmopolitan. Since the executables are also zip files, the interpreter can even load its own sources from the binary itself. https://github.com/jart/cosmopolitan/blob/master/examples/hellojs.c https://github.com/jart/cosmopolitan/blob/master/examples/he...
temp00345 6y ago
Amazing accomplishment ! Not to detract from the technical breakthrough, but won't this be a major gift to virus writers ?
- jxf 6y ago
  Just because an executable is cross-platform doesn't mean it can exploit issues on other platforms. Those issues have to actually exist there, and most vulnerabilities don't look like that.
- Nextgrid 6y ago
  The hard part when making a virus is to exploit the target system to run said virus. And if you have such an exploit, then supplying a compatible binary depending on which OS the target system runs is easy. Exploit attempts for embedded systems (routers, cameras, etc) typically start with attempting to execute code in a portable language such as Shell scripts (which would be the only thing this new project would replace), those scripts try to detect the architecture of the system and then download the appropriate binary (the server hosting the malicious binaries provides many variants for different architectures). In the end I don't see this solving a real problem when it comes to malware - this problem has already been solved through other means.
- jart 6y ago
  Cosmopolitan Libc is designed to put power in your hands, and power can be used for good or bad. The best way for us to all keep that power, is to start using Cosmopolitan to do as many good deeds as possible. This project is going to benefit developers on all platforms, because it supports everyone without bias. Indie developers are going to have more opportunities to be successful writing native apps, since Cosmo helps them reach a broader audience. Before Cosmopolitan only big companies could introduce new projects (e.g. TensorFlow) that effectively solve the portability problem, since the only way to do it before was brute force cash. Lastly, Cosmopolitan is going to relieve language authors of many of the portability burdens they've each needed to carry on their own, which means they're going to have more time to focus on their visions. If we don't use Actually Portable Executable to accomplish grand acts of public service, then operating systems are just going to block it. For example, UPX is a project that does creative things with executable formats. If you read the XNU source code you'll notice that they have explicit source code for blocking those executables and they call out the project by name.
deleted 6y ago
[deleted]
dennis2 6y ago
Nice
csmpltn 6y ago
I'm not sure what problem Cosmopolitan is trying to solve. I think it's an interesting research project, but from a practical perspective - there are significantly better (and more mature) ways to achieve cross-platform compatibility that don't involve many of the compile-time and run-time hacks required to get Cosmopolitan to work. "Actually Portable Executables" are not any more "portable" than a statically compiled .exe running natively on Windows, or using Wine[0] on a Linux, as an example. [0] https://en.wikipedia.org/wiki/Wine_(software) https://en.wikipedia.org/wiki/Wine_(software)
- jart 6y ago
  I'm sorry but WINE is the kind of interpreter that gives me enough time to take a coffee break as it loads solitaire.exe. I needed to use it a few weeks ago because I'm supporting someone who needs it for their CI environment. I discovered that APIs for things as fundamental as things like memory that Microsoft introduced back in 2006 hadn't been implemented in the WINE release Debian installed on my machine that was made in 2019. As far as I can tell the project wasn't funded for the longest time, which is unfortunate, because a once great tool then succumbed to tragedy of the commons. That's why I hope Cosmopolitan can have a positive impact. Because interpreters are great but we shouldn't need to rely on them as much as we currently do. The only thing your Actually Portable Executables need is the canonical stock preferred kernel interfaces that have indefinite stability promises backed by the most powerful technology corporations on earth. No one got fired for depending on the IBM's of the world. Your software is going to have a long life with little maintenance and best of all it's going to be fast.
  - VMG 6y ago
    I'm surprised by this. I'm using WINE to play League of Legends on Linux and it works surprisingly well.
    - jonathanstrange 6y ago
      Once an application is running it's reasonably fast but Wine has always taken endlessly time to start up, at least on my machine. It feels similar to Java VMs in that respect.
pbronez 6y ago
Something like this could be great for open source projects that target a lot of platforms. For example, Conda-Forge [0] has automated build pipelines for Linux, Mac and Windows. Perhaps something like this could let you reduce that to an x86 build and an Arm build... removing an entire dimension from the build matrix. Gonna need to solve that 1/2=g problem before Numpy is happy though! [0] https://conda-forge.org/ https://conda-forge.org/
TazeTSchnitzel 6y ago
Previous discussion of Cosmpolitan: https://news.ycombinator.com/item?id=25556286 https://news.ycombinator.com/item?id=25556286
EamonnMR 6y ago
I wonder if you could somehow get this working with libraries that are widely ported across platforms already like SDL.
- phire 6y ago
  It might take a bit of effort, but should be possible. One problem for libraries like SDL is they disable all their per platform code with #ifdefs. To build a static, cross platform version you would have to convert them all those #ifdefs to runtime branching, dynamically load libraries and provide headers files for all platforms.
barneygale 6y ago
It would be fantastic to see a byte-by-byte breakdown of the universal header.
benatkin 6y ago
I'm not any more or less excited about this from a utility point of view than I am about golang's support for embedding data in an executable. It is pretty damn impressive, though. I'll give it that. It's like that cat pushing a watermelon meme. This is a binary that runs on seven operating systems. Your argument is invalid.
dazhbog 6y ago
Does anyone know if there is any way to run another exe/binary that its bundled in a cross platform app, while its all enclosed in 1 binary file? Because Cosmo could work as a wrapper of windows/mac/linux binary executables and then the cosmo app can decide which one to invoke. Based on my understanding if program A is to run program B, program B needs to be addressable in the filesystem. So based on my testing, the bundled executables need to be written to disk first and then the main program could run them. This also might work with macos .app folder formats but not sure how to do it on linux to avoid chmod +x issues and how to have 1 binary file on windows. Any ideas?
- pwdisswordfish0 6y ago
  Huh? > Cosmo could work as a wrapper of windows/mac/linux binary executables and then the cosmo app can decide which one to invoke. It sounds like you misunderstand the reason for the project and that you're trying to add another layer of indirection to the thing that is already able to achieve the effect that you you seek to accomplish with that layer of indirection. No need for the indirection, just use the thing directly. If you want to run $APP on platforms X, Y, and Z, then you'd use Cosmopolitan to build the $APP binary as an Actual Portable Executable, and you'd run that executable on any of those platforms.
  - dazhbog 6y ago
    When the source is available, then of course everything can be compiled in one nice executable. ;) I am referring to cases where the source is not available or its hard to set up some projects together. Some projects (usually hardware related) require different compilers, compile options, etc. If the project already has the binaries, then for distribution maybe a wrapper could be a good option.
ericb 6y ago
Would these binaries work on Apple Silicon? It seems like no, since it isn't x86? What are the limits to the portability, and is there a path past those limits?
- momothereal 6y ago
  That's right, it only supports x86. However the binary will detect other architectures and try to run itself with QEMU in that case.
- samatman 6y ago
  I would love for someone to do this experiment. In principle, Rosetta should detect that it's x86, perform the translation, and everything should "just work". In practice, it's possible that the weird things APE is doing to get cross-platform compatibility will undermine one or more of Rosetta's assumptions. I don't have an M1 Mac, so I'm not in a position to try it. But someone should!
momothereal 6y ago
I wonder if there's a compatibility/sanity test suite for Lua? It would be nice to run this interpreter with real scripts to see how it compares.
rahimiali 6y ago
The author mentions the portable binary is smaller than the original. It seems Cosmopolitan's output is actually an executable pkzip file: https://justine.lol/ape.html https://justine.lol/ape.html
- comex 6y ago
  That's how I read it the first time through, but on a closer look, they're not comparing a normal native binary to an APE binary. They're comparing an intermediate step of the APE build process to the final output. `objcopy -S -O binary` does two things. One is to strip all the ELF headers and just dump the section contents into an output file. It looks like the input to this command is an executable within an executable, with the BIOS boot sector, ELF segments, Mach-O headers, etc. all embedded as section contents within an outer ELF file; objcopy then removes the outer shell. Clearly a necessary step, but it won't make much difference in binary size. However, the `-S` tells objcopy to strip debug information, which is included in the original since `-g` was passed to `gcc`. This is presumably responsible for almost all of the difference; debug info is notoriously huge. You can get a similar size reduction in a normal compilation by running `strip`. Edit: But objcopy does not do anything fancy like compress the input. It looks like the pkzip part only contains associated files the binary might need (e.g. "hellojs.com" contains a JavaScript file and time zone files), not the executable code itself.
al2o3cr 6y ago
The dream of "Write Once, Debug Everywhere" will never die
qyi 6y ago
Cool hack but extremely fragile. It's actually pretty sad that people will consider using something like this in real projects.
TJSomething 6y ago
I was trying to get something from Nim to build like this, but I probably need to understand linking a bit better.
amai 6y ago
Actually portable malware coming in 3,2,1,...
utbabya 6y ago
Learned a lot from this and related pages, thanks a lot. It is one of the best hack I've ever seen, ever. Although as others have pointed out I wouldn't consider it exactly elegant (IIUC basically shell script to morph it on first run on nix); Nor exactly ground breaking with the fat-binary-like concept, if you don't use platform specific libraries or calls anyway, cross compile works just fine? What's the use case here? I mean for platforms where performance or size overhead of vm-based languages is a problem, you probably don't want it to include extra code for other platforms anyway? OTOH Blinkenlights looks really nice tho. I've see a dude live hand editing code page 417 with notepad.exe to change a jmp so it doesn't look for a CD with my own eyes ;) Always admire hackers of this rank. I think Blinkenlights could help me achieve something similar someday if I allocate bandwidth to it, would spend lots of time to play with it.

683 points by krab 6y ago

krab 6y ago

I just saw this tweet - https://twitter.com/JustineTunney/status/1365805503561957376 https://twitter.com/JustineTunney/status/1365805503561957376 Cosmopolitan is going to be a big thing.

andai 6y ago
>We've just created a 336kb Lua interpreter binary that runs on seven operating systems, thanks to Cosmopolitan Libc. https://ahgamut.github.io/c/2021/02/27/ape-cosmo/ https://ahgamut.github.io/c/2021/02/27/ape-cosmo/
- svenfaw 6y ago
  Awesome! But why no download link?
  - pfortuny 6y ago
    Because as OP says it is not totally complete and lacks some parts.
  - ahgamut 6y ago
    You can compile it yourself if you like. I created a fork of the Lua Github mirror and made the necessary changes to compile with Cosmopolitan: https://github.com/ahgamut/lua/tree/cosmopolitan https://github.com/ahgamut/lua/tree/cosmopolitan
poc43 6y ago
Likely only if it’s forked away from anything to do with its current maintainer, for reasons only a Google search away: http://valleywag.gawker.com/why-does-google-employ-a-pro-slavery-lunatic-1612868507 http://valleywag.gawker.com/why-does-google-employ-a-pro-sla... That’s the problem with trolling like that: once you do something cool that’s unrelated and try to move on it’s not clear if you’re genuine about anything any more. The history there will absolutely make larger interests hesitant to associate, contribute or integrate in something like, say, LLVM (regardless of the merit or lack thereof), particularly in today’s less forgiving culture. Note that I’m not sharing an opinion on the content. It’s a metaobservation about where we are, for better or worse. That’s on page 1 of my Google search for “wow, who’s this amazing coder?”, so my path is the same anyone else would take doing due diligence on getting involved.
- anthk 6y ago
  >Rather than inclusionary leftism. Traditional left didn't give a shit on gender, race or whatever bullshit you think you should declare yourself "inferior" because of an excuse. Just work and equality. I hate both poshy techno-aristocrats (I would kick them out to Somalia or a Mexico narco shithole, to see what they can do without social support or a state backed police), and SWJs with far more points in common with fascism (Fuck that so-called cultural appropiation) than the common worker.
  - poc43 6y ago
    I mean, you’re engaging with the espoused opinion of someone who simultaneously advocated class warfare and drew Google payroll, so you may as well be arguing with a figment of our collective imagination. If that’s not obvious by nature of how incongruous it is I fear for the Internet.
- 1996 6y ago
  > Likely only if it’s forked away from anything to do with its current maintainer, And exactly why should I care about the current maintainer? Except maybe to say "thank you"? Let's put the question upside down: what have you (or the gawker journalist) brought to me? I mean, besides your outraged opinions (which anyone is entitled to have!) On one hand, you have a wonderful technical creation by someone asking nothing in return. On the other, you are providing... hatred towards the author? On a 2 hours old account? I'm sorry if she breaks the kumbaya illusion, but people are not equal. Some can't escape the working class, or the welfare class - not due to any personal failure or limits, but due to societal indoctrination. Some other people break free, and then bring gifts of the gods to mankind. They are called innovators, entrepreneurs - the name vary. But you know one when you see one. There is nothing is cosmopolitan that was "impossible" to achieve by anyone dedicated, even years before. There's nothing magic- except in the mind of the author, that associated the right pieces together. > The history there will absolutely make larger interests hesitant to associate, contribute or integrate If anything, your comment make me less likely to associate with you! > particularly in today’s less forgiving culture. I couldn't care less. The less forgiving culture is a problem for those who need to work ("will code for food" as they said during the dotcom crisis) or integrate with the rest of society. Unfortunately, that means most of the people here, especially those working for the FANGs. But there are some of us who just don't care - except to lament that, when shown the way to escape, most geeks double down on their own mistakes, and remain chained to their FANGs masters. So please go on attacking the author if that makes you happy, while the rest of us admire the creativity and new things made possible by cosmopolitan (and make money out of that too!)

ers35 6y ago

Maybe Cosmopolitan could be used with luastatic[1] to compile a Lua program to an "Actually Portable Executable": CC="" luastatic main.lua # Compile main.luastatic.c with Cosmopolitan Libc [1] https://github.com/ers35/luastatic https://github.com/ers35/luastatic

zbendefy 6y ago

Could webassembly be also used for this?

AstralStorm 6y ago
That's not as portable as you need an interpreter.
iFreilicht 6y ago
Depends on what you mean by “using”. It is conceivably possible to compile a webassembly interpreter like wac[1] with cosmopolitan, which would then run on all OSes and bare metal, yes. [1]: https://github.com/kanaka/wac https://github.com/kanaka/wac
- deleted 6y ago
  [deleted]
_roqj 6y ago
Yes it can! Wasm3 is the fastest WebAssembly interpreter. Since they recently adopted Cosmopolitan it became the most universal one too. Check out the 150kb APE binary they've got on the release page. https://github.com/wasm3/wasm3/releases/tag/v0.4.8 https://github.com/wasm3/wasm3/releases/tag/v0.4.8

abhinav22 6y ago

Can somebody explain in layman’s terms what this is all about? I tried my best reading the other thread, but it went well above me.

Gaelan 6y ago
Windows, macOS, Linux, and BSD all (usually) run on Intel processors, so in theory it should be possible to write a program that runs on all three. However, despite ultimately using the same instruction sets, the four OSes use different formats to store metadata about the programs, and have different ways for programs to communicate with the operating system. Justine Tunney created two things: 1) Actually Portable Executables, a clever way of formatting a program so that all four OSes interpret it a valid program in their own format. 2) Cosmopolitan libc, a library for communicating with the OS that handles each OS's interface, allowing programs to work on all four.
- abhinav22 6y ago
  Thank you (and thanks to everybody else for their helpful answers). Your note on intel processors meaning in theory programs should run on all operating systems was most helpful and got me across the line - now I get it!
  - jart 6y ago
    That's pretty much it. Cosmopolitan Libc is the simplest project that could have happened decades ago. Anyone could have built it. There's never been a technical reason why an x86 binary can't run on all x86 operating systems. It's just that traditionally only operating systems publish C libraries, and they have no incentive to spend money supporting their competitors. So it's the kind of project that could only happen if it was done by an indie developer just trying to have fun.
    - fctorial 6y ago
      Another reason is that it requires fairly thorough knowledge of 7 different programming environments. Most people don't understand 1.

jart 6y ago

Cosmopolitan Libc author here. When I saw this article earlier today, it put a big smile on my face, because I would have thought there'd be so many more issues than there turned out to be! For additional context, we've got a GitHub issue that's tracking progress on the changes that need to be made based on what we learned: https://github.com/jart/cosmopolitan/issues/61 https://github.com/jart/cosmopolitan/issues/61

ducktective 6y ago
Any recommendations to somebody who wants to learn C "properly" in $CURRENT_YEAR?
- unkulunkulu 6y ago
  $CURRENT_BEST_BOOK Sorry :)
- jtvjan 6y ago
  I really liked "The C Programming Language (second edition)" by Kernighan & Ritchie. It doesn't teach C99 and later features, but if you want to write portable code you're not using those anyway.
  - sillysaurusx 6y ago
    K&R is a horrible way to learn C. It's the best way to make someone run in the other direction, though. It's like telling someone that SICP is the best way to learn Scheme. Maybe, but lots of people learn differently. Unfortunately I don't have good suggestions myself, but at least they won't feel bad if K&R isn't their cup of tea.
  - e12e 6y ago
    From various discussions on Hn, I think the consensus is most people really like k&r as a technical book (it's concise, clear and a fun read) - but everyone working with C says its a terrible book for learning modern C. I tried to refresh my memory from previous threads, and I think the general trend has been to recommend (as seen in sibling comments): https://modernc.gforge.inria.fr/ https://modernc.gforge.inria.fr/ https://nostarch.com/Effective_C https://nostarch.com/Effective_C I've also seen the older: https://www.oreilly.com/library/view/21st-century-c/9781449344382/ https://www.oreilly.com/library/view/21st-century-c/97814493... Mentioned. There's also Architecture of Open Source Applications, which can help with starting to read some larger code bases, some of which are in C: http://aosabook.org/en/index.html http://aosabook.org/en/index.html And there's a general recommendation to go read the source code of the Redis cache/db. Finally i came across a mention of this short article on gdb (nb: mention of TUI text ui should probably have been in the top, not a foot note): https://www.recurse.com/blog/5-learning-c-with-gdb https://www.recurse.com/blog/5-learning-c-with-gdb I feel I'm missing a book that has come up often, but can't think of which one. I did like zed Shaws learn c the hard way, but I'm afraid it's getting a bit long in the tooth: https://learncodethehardway.org/c/ https://learncodethehardway.org/c/

antman 6y ago

Portable Lus is a great idea. By using Cosmopilitan does this also run on baremetal machines?

antman 6y ago

Portable Lua is a great idea. By using Cosmopilitan does this also run on baremetal machines? Portable micropython might also be a low hanging fruit target.

ngcc_hk 6y ago

This actually come as a surprise. Just to ask as a testing case - can we have an actually portable (common) lisp/python interpreter with at least terminal interface? Can the terminal have something like VT-100 or other basic interface? and Can we even run javascript to do screen related items ... like https://github.com/d3/d3 https://github.com/d3/d3

matheusmoreira 6y ago
Yes. She built a javascript interpreter with cosmopolitan. Since the executables are also zip files, the interpreter can even load its own sources from the binary itself. https://github.com/jart/cosmopolitan/blob/master/examples/hellojs.c https://github.com/jart/cosmopolitan/blob/master/examples/he...

temp00345 6y ago

Amazing accomplishment ! Not to detract from the technical breakthrough, but won't this be a major gift to virus writers ?

jxf 6y ago
Just because an executable is cross-platform doesn't mean it can exploit issues on other platforms. Those issues have to actually exist there, and most vulnerabilities don't look like that.
Nextgrid 6y ago
The hard part when making a virus is to exploit the target system to run said virus. And if you have such an exploit, then supplying a compatible binary depending on which OS the target system runs is easy. Exploit attempts for embedded systems (routers, cameras, etc) typically start with attempting to execute code in a portable language such as Shell scripts (which would be the only thing this new project would replace), those scripts try to detect the architecture of the system and then download the appropriate binary (the server hosting the malicious binaries provides many variants for different architectures). In the end I don't see this solving a real problem when it comes to malware - this problem has already been solved through other means.
jart 6y ago
Cosmopolitan Libc is designed to put power in your hands, and power can be used for good or bad. The best way for us to all keep that power, is to start using Cosmopolitan to do as many good deeds as possible. This project is going to benefit developers on all platforms, because it supports everyone without bias. Indie developers are going to have more opportunities to be successful writing native apps, since Cosmo helps them reach a broader audience. Before Cosmopolitan only big companies could introduce new projects (e.g. TensorFlow) that effectively solve the portability problem, since the only way to do it before was brute force cash. Lastly, Cosmopolitan is going to relieve language authors of many of the portability burdens they've each needed to carry on their own, which means they're going to have more time to focus on their visions. If we don't use Actually Portable Executable to accomplish grand acts of public service, then operating systems are just going to block it. For example, UPX is a project that does creative things with executable formats. If you read the XNU source code you'll notice that they have explicit source code for blocking those executables and they call out the project by name.

deleted 6y ago

[deleted]

dennis2 6y ago

Nice

csmpltn 6y ago

I'm not sure what problem Cosmopolitan is trying to solve. I think it's an interesting research project, but from a practical perspective - there are significantly better (and more mature) ways to achieve cross-platform compatibility that don't involve many of the compile-time and run-time hacks required to get Cosmopolitan to work. "Actually Portable Executables" are not any more "portable" than a statically compiled .exe running natively on Windows, or using Wine[0] on a Linux, as an example. [0] https://en.wikipedia.org/wiki/Wine_(software) https://en.wikipedia.org/wiki/Wine_(software)

jart 6y ago
I'm sorry but WINE is the kind of interpreter that gives me enough time to take a coffee break as it loads solitaire.exe. I needed to use it a few weeks ago because I'm supporting someone who needs it for their CI environment. I discovered that APIs for things as fundamental as things like memory that Microsoft introduced back in 2006 hadn't been implemented in the WINE release Debian installed on my machine that was made in 2019. As far as I can tell the project wasn't funded for the longest time, which is unfortunate, because a once great tool then succumbed to tragedy of the commons. That's why I hope Cosmopolitan can have a positive impact. Because interpreters are great but we shouldn't need to rely on them as much as we currently do. The only thing your Actually Portable Executables need is the canonical stock preferred kernel interfaces that have indefinite stability promises backed by the most powerful technology corporations on earth. No one got fired for depending on the IBM's of the world. Your software is going to have a long life with little maintenance and best of all it's going to be fast.
- VMG 6y ago
  I'm surprised by this. I'm using WINE to play League of Legends on Linux and it works surprisingly well.
  - jonathanstrange 6y ago
    Once an application is running it's reasonably fast but Wine has always taken endlessly time to start up, at least on my machine. It feels similar to Java VMs in that respect.

pbronez 6y ago

Something like this could be great for open source projects that target a lot of platforms. For example, Conda-Forge [0] has automated build pipelines for Linux, Mac and Windows. Perhaps something like this could let you reduce that to an x86 build and an Arm build... removing an entire dimension from the build matrix. Gonna need to solve that 1/2=g problem before Numpy is happy though! [0] https://conda-forge.org/ https://conda-forge.org/

TazeTSchnitzel 6y ago

Previous discussion of Cosmpolitan: https://news.ycombinator.com/item?id=25556286 https://news.ycombinator.com/item?id=25556286

EamonnMR 6y ago

I wonder if you could somehow get this working with libraries that are widely ported across platforms already like SDL.

phire 6y ago
It might take a bit of effort, but should be possible. One problem for libraries like SDL is they disable all their per platform code with #ifdefs. To build a static, cross platform version you would have to convert them all those #ifdefs to runtime branching, dynamically load libraries and provide headers files for all platforms.

barneygale 6y ago

It would be fantastic to see a byte-by-byte breakdown of the universal header.

benatkin 6y ago

I'm not any more or less excited about this from a utility point of view than I am about golang's support for embedding data in an executable. It is pretty damn impressive, though. I'll give it that. It's like that cat pushing a watermelon meme. This is a binary that runs on seven operating systems. Your argument is invalid.

dazhbog 6y ago

Does anyone know if there is any way to run another exe/binary that its bundled in a cross platform app, while its all enclosed in 1 binary file? Because Cosmo could work as a wrapper of windows/mac/linux binary executables and then the cosmo app can decide which one to invoke. Based on my understanding if program A is to run program B, program B needs to be addressable in the filesystem. So based on my testing, the bundled executables need to be written to disk first and then the main program could run them. This also might work with macos .app folder formats but not sure how to do it on linux to avoid chmod +x issues and how to have 1 binary file on windows. Any ideas?

pwdisswordfish0 6y ago
Huh? > Cosmo could work as a wrapper of windows/mac/linux binary executables and then the cosmo app can decide which one to invoke. It sounds like you misunderstand the reason for the project and that you're trying to add another layer of indirection to the thing that is already able to achieve the effect that you you seek to accomplish with that layer of indirection. No need for the indirection, just use the thing directly. If you want to run $APP on platforms X, Y, and Z, then you'd use Cosmopolitan to build the $APP binary as an Actual Portable Executable, and you'd run that executable on any of those platforms.
- dazhbog 6y ago
  When the source is available, then of course everything can be compiled in one nice executable. ;) I am referring to cases where the source is not available or its hard to set up some projects together. Some projects (usually hardware related) require different compilers, compile options, etc. If the project already has the binaries, then for distribution maybe a wrapper could be a good option.

ericb 6y ago

Would these binaries work on Apple Silicon? It seems like no, since it isn't x86? What are the limits to the portability, and is there a path past those limits?

momothereal 6y ago
That's right, it only supports x86. However the binary will detect other architectures and try to run itself with QEMU in that case.
samatman 6y ago
I would love for someone to do this experiment. In principle, Rosetta should detect that it's x86, perform the translation, and everything should "just work". In practice, it's possible that the weird things APE is doing to get cross-platform compatibility will undermine one or more of Rosetta's assumptions. I don't have an M1 Mac, so I'm not in a position to try it. But someone should!

momothereal 6y ago

I wonder if there's a compatibility/sanity test suite for Lua? It would be nice to run this interpreter with real scripts to see how it compares.

rahimiali 6y ago

The author mentions the portable binary is smaller than the original. It seems Cosmopolitan's output is actually an executable pkzip file: https://justine.lol/ape.html https://justine.lol/ape.html

comex 6y ago
That's how I read it the first time through, but on a closer look, they're not comparing a normal native binary to an APE binary. They're comparing an intermediate step of the APE build process to the final output. `objcopy -S -O binary` does two things. One is to strip all the ELF headers and just dump the section contents into an output file. It looks like the input to this command is an executable within an executable, with the BIOS boot sector, ELF segments, Mach-O headers, etc. all embedded as section contents within an outer ELF file; objcopy then removes the outer shell. Clearly a necessary step, but it won't make much difference in binary size. However, the `-S` tells objcopy to strip debug information, which is included in the original since `-g` was passed to `gcc`. This is presumably responsible for almost all of the difference; debug info is notoriously huge. You can get a similar size reduction in a normal compilation by running `strip`. Edit: But objcopy does not do anything fancy like compress the input. It looks like the pkzip part only contains associated files the binary might need (e.g. "hellojs.com" contains a JavaScript file and time zone files), not the executable code itself.

al2o3cr 6y ago

The dream of "Write Once, Debug Everywhere" will never die

qyi 6y ago

Cool hack but extremely fragile. It's actually pretty sad that people will consider using something like this in real projects.

TJSomething 6y ago

I was trying to get something from Nim to build like this, but I probably need to understand linking a bit better.

amai 6y ago

Actually portable malware coming in 3,2,1,...

utbabya 6y ago

Learned a lot from this and related pages, thanks a lot. It is one of the best hack I've ever seen, ever. Although as others have pointed out I wouldn't consider it exactly elegant (IIUC basically shell script to morph it on first run on nix); Nor exactly ground breaking with the fat-binary-like concept, if you don't use platform specific libraries or calls anyway, cross compile works just fine? What's the use case here? I mean for platforms where performance or size overhead of vm-based languages is a problem, you probably don't want it to include extra code for other platforms anyway? OTOH Blinkenlights looks really nice tho. I've see a dude live hand editing code page 417 with notepad.exe to change a jmp so it doesn't look for a CD with my own eyes ;) Always admire hackers of this rank. I think Blinkenlights could help me achieve something similar someday if I allocate bandwidth to it, would spend lots of time to play with it.