6 ms·

I'm always somewhat scared - not sure if rationally or irrationally - about cosmopolitan. It's a cool hack but I somehow feel like it should not work.

hiAndrewQuinn 2y ago
I'm somewhat of the same mind, but I'm fairly sure a study of Operating Systems: Three Easy Steps would get me over the hump. There's no actual reason to suspect this thing would like, sidestep normal process management or memory virtualization or something and run amok... I think.
alganet 2y ago
It should work. We should have portable executables across many operating systems and architectures. The other expectation is scary.
- cpach 2y ago
  There’s also Java, Python, Ruby, NodeJS etc etc.
- 3836293648 2y ago
  Working across unices is reasonable. Adding Windows is less so
  - cpach 2y ago
    It’s quite rare to ship multi-Unix binaries, isn’t it? Cosmopolitan is a very cool hack indeed, but IMHO it’s not likely to gain widespread usage.
  - flohofwoe 2y ago
    A thin POSIX layer and ELF loader shouldn't be too much of a problem for Microsoft to implement if they wanted to do so (WinNT actually did have a POSIX personality at some point, but I don't think that's still supported). I'd also like to see a builtin WASM runtime in all operating systems.
    - boricj 2y ago
      Microsoft did implement a not-so-thin POSIX layer and an ELF loader atop the NT kernel, it's WSL1 (Windows Subsystem for Linux). It was obsoleted by WSL2, which uses a specifically-tuned Linux VM instead for performance and completeness reasons. I haven't played with it, but I think the classic Windows POSIX subsystem used the COFF/PE file formats instead of ELF.
    - vidarh 2y ago
      Indeed, back in '95 or so there was a library called CrossELF that'd let you compile ELF so files and use a tiny loader linked to CrossELF for each platform you cared about to load the main .so file, and you could build platform-independent code with it. I remember writing some simple networking code where the loader just had a tiny set of shims for a few calls and the rest of the networking code was a single binary for both Linux and Win32. The problem is wrapping the relevant APIs - as you can see w/e.g. Wine. For some functionality - like networking - the surface is pretty small, for others its a nightmare.
  - TimTheTinker 2y ago
    Creating polyglot source code is a well-studied type of hack. If that's possible, why not polyglot executables? After all, an executable file format is essentially another type of source code. Pair that with a polyglot ABI and polyglot system calls... and you get APE binaries :)
- duped 2y ago
  Why? Hardware and software architectures often have radically different needs from their loader, it doesn't make sense to have one format to rule them all. And any format flexible enough to support all use cases would just be a container for other formats, and in practice, loaders would ignore the variants that they don't/can't support so developers would need to care about porting things anyway. ELF is pretty stable across many architectures and platforms but it's proven to be inflexible enough for plenty of applications where people develop their own, or alter it in varying ways.
mgaunard 2y ago
It doesn't really work, the libc is widely non-conforming. I'd rather have a proper interface designed for portability rather than a hacked-together POSIX-but-not-really.
- Conscat 2y ago
  Programmers already deal with POSIX-but-not-really any time they deliver software for a Linux. At least this has the same quirks everywhere.
  - jart 2y ago
    > At least this has the same quirks everywhere. That would make a great slogan for the project.
asveikau 2y ago
Years ago I read some of the author's posts (they're active on HN too iirc) about it, and it seemed to me like they were relying heavily on the internals of the loader or dynamic linker on each platform, such that a new release for any given OS could conceivably break your binary. It's probably a very fun project to hack on but I would advise against distributing the binaries and expecting them to work over the long term.
- TimTheTinker 2y ago
  To my recollection, at least on *nix operating systems, they got some changes made to the POSIX standard to formalize behavior the binaries rely on. So going forward, mere POSIX compliance and ongoing ABI compatibility guarantee the binaries will continue to work on *nix operating systems. On Windows, backwards ABI and executable compatibility has always been an extremely high priority, so I think the danger of future breakage is low. Neither of those speak to macOS, but maybe someone who knows more can help clarify.
  - asveikau 2y ago
    I disagree with your assessment in all cases. You simply do not know what you were talking about. Standardization efforts and backward compatibility assumes a well-behaved application. If you explicitly depend on really weird hacky stuff like abusing corner cases in the object file format, you risk breakage and you will break. In the Unix world, firstly, platforms that aren't Linux typically say that if you don't do syscalls through system libc, all bets are off. Second, if they standardized a few things here and there, they are likely standardizing stuff that will formed applications linked with typical libraries will exercise. Standardization does not imply explicitly listing all possible corner cases of your object file format. On Windows, I happen to be a former Microsoft dev who worked on Windows in 2008-2011. If an app were trying to push the limits of the PE format, I don't think it would get fixed on the platform side... I've seen popular applications do much less bad things and get broken.
    - yjftsjthsd-h 2y ago
      > I disagree with your assessment in all cases. You simply do not know what you were talking about. No need to be rude. > Standardization efforts and backward compatibility assumes a well-behaved application. If you explicitly depend on really weird hacky stuff like abusing corner cases in the object file format, you risk breakage and you will break. Sure, but if you get the standard to include your use then it's not a hacky edge case anymore. Ex. https://justine.lol/cosmo3/ https://justine.lol/cosmo3/ "POSIX even changed their rules about binary in shell scripts specifically to let us do it" > https://austingroupbugs.net/view.php?id=1250 https://austingroupbugs.net/view.php?id=1250
boricj 2y ago
You can commit a lot of sins with ABIs if you throw the academic books into the fire. For example, on my side I'm developing tooling that allows one to delink programs back into object files. This allows me to commit a whole bunch of heresy according to CS101, such as making a native port of a Linux program to Windows without having access to its source code or taking binary code from a PlayStation game and stuffing it into a Linux MIPS program. When you're doing that kind of dark magic, it's one of those "know the rules so well you can break them" situation. Instead of following the theory that tells you what you should do, you follow the real-life world that constrains what you can do.
Aerbil313 2y ago
> It's a cool hack but I somehow feel like it should not work. It's rather obvious. It's bandaid that lets us continue on with the current outdated paradigms. Not a real innovation.

hiAndrewQuinn 2y ago

I'm somewhat of the same mind, but I'm fairly sure a study of Operating Systems: Three Easy Steps would get me over the hump. There's no actual reason to suspect this thing would like, sidestep normal process management or memory virtualization or something and run amok... I think.

alganet 2y ago

It should work. We should have portable executables across many operating systems and architectures. The other expectation is scary.

cpach 2y ago
There’s also Java, Python, Ruby, NodeJS etc etc.
3836293648 2y ago
Working across unices is reasonable. Adding Windows is less so
- cpach 2y ago
  It’s quite rare to ship multi-Unix binaries, isn’t it? Cosmopolitan is a very cool hack indeed, but IMHO it’s not likely to gain widespread usage.
- flohofwoe 2y ago
  A thin POSIX layer and ELF loader shouldn't be too much of a problem for Microsoft to implement if they wanted to do so (WinNT actually did have a POSIX personality at some point, but I don't think that's still supported). I'd also like to see a builtin WASM runtime in all operating systems.
  - boricj 2y ago
    Microsoft did implement a not-so-thin POSIX layer and an ELF loader atop the NT kernel, it's WSL1 (Windows Subsystem for Linux). It was obsoleted by WSL2, which uses a specifically-tuned Linux VM instead for performance and completeness reasons. I haven't played with it, but I think the classic Windows POSIX subsystem used the COFF/PE file formats instead of ELF.
  - vidarh 2y ago
    Indeed, back in '95 or so there was a library called CrossELF that'd let you compile ELF so files and use a tiny loader linked to CrossELF for each platform you cared about to load the main .so file, and you could build platform-independent code with it. I remember writing some simple networking code where the loader just had a tiny set of shims for a few calls and the rest of the networking code was a single binary for both Linux and Win32. The problem is wrapping the relevant APIs - as you can see w/e.g. Wine. For some functionality - like networking - the surface is pretty small, for others its a nightmare.
- TimTheTinker 2y ago
  Creating polyglot source code is a well-studied type of hack. If that's possible, why not polyglot executables? After all, an executable file format is essentially another type of source code. Pair that with a polyglot ABI and polyglot system calls... and you get APE binaries :)
duped 2y ago
Why? Hardware and software architectures often have radically different needs from their loader, it doesn't make sense to have one format to rule them all. And any format flexible enough to support all use cases would just be a container for other formats, and in practice, loaders would ignore the variants that they don't/can't support so developers would need to care about porting things anyway. ELF is pretty stable across many architectures and platforms but it's proven to be inflexible enough for plenty of applications where people develop their own, or alter it in varying ways.

mgaunard 2y ago

It doesn't really work, the libc is widely non-conforming. I'd rather have a proper interface designed for portability rather than a hacked-together POSIX-but-not-really.

Conscat 2y ago
Programmers already deal with POSIX-but-not-really any time they deliver software for a Linux. At least this has the same quirks everywhere.
- jart 2y ago
  > At least this has the same quirks everywhere. That would make a great slogan for the project.

asveikau 2y ago

Years ago I read some of the author's posts (they're active on HN too iirc) about it, and it seemed to me like they were relying heavily on the internals of the loader or dynamic linker on each platform, such that a new release for any given OS could conceivably break your binary. It's probably a very fun project to hack on but I would advise against distributing the binaries and expecting them to work over the long term.

TimTheTinker 2y ago
To my recollection, at least on *nix operating systems, they got some changes made to the POSIX standard to formalize behavior the binaries rely on. So going forward, mere POSIX compliance and ongoing ABI compatibility guarantee the binaries will continue to work on *nix operating systems. On Windows, backwards ABI and executable compatibility has always been an extremely high priority, so I think the danger of future breakage is low. Neither of those speak to macOS, but maybe someone who knows more can help clarify.
- asveikau 2y ago
  I disagree with your assessment in all cases. You simply do not know what you were talking about. Standardization efforts and backward compatibility assumes a well-behaved application. If you explicitly depend on really weird hacky stuff like abusing corner cases in the object file format, you risk breakage and you will break. In the Unix world, firstly, platforms that aren't Linux typically say that if you don't do syscalls through system libc, all bets are off. Second, if they standardized a few things here and there, they are likely standardizing stuff that will formed applications linked with typical libraries will exercise. Standardization does not imply explicitly listing all possible corner cases of your object file format. On Windows, I happen to be a former Microsoft dev who worked on Windows in 2008-2011. If an app were trying to push the limits of the PE format, I don't think it would get fixed on the platform side... I've seen popular applications do much less bad things and get broken.
  - yjftsjthsd-h 2y ago
    > I disagree with your assessment in all cases. You simply do not know what you were talking about. No need to be rude. > Standardization efforts and backward compatibility assumes a well-behaved application. If you explicitly depend on really weird hacky stuff like abusing corner cases in the object file format, you risk breakage and you will break. Sure, but if you get the standard to include your use then it's not a hacky edge case anymore. Ex. https://justine.lol/cosmo3/ https://justine.lol/cosmo3/ "POSIX even changed their rules about binary in shell scripts specifically to let us do it" > https://austingroupbugs.net/view.php?id=1250 https://austingroupbugs.net/view.php?id=1250

boricj 2y ago

You can commit a lot of sins with ABIs if you throw the academic books into the fire. For example, on my side I'm developing tooling that allows one to delink programs back into object files. This allows me to commit a whole bunch of heresy according to CS101, such as making a native port of a Linux program to Windows without having access to its source code or taking binary code from a PlayStation game and stuffing it into a Linux MIPS program. When you're doing that kind of dark magic, it's one of those "know the rules so well you can break them" situation. Instead of following the theory that tells you what you should do, you follow the real-life world that constrains what you can do.

Aerbil313 2y ago

> It's a cool hack but I somehow feel like it should not work. It's rather obvious. It's bandaid that lets us continue on with the current outdated paradigms. Not a real innovation.