6 ms·

This is an interesting, and potentially compelling argument that WebAssembly binaries and runtime actually aren’t the future of compile-once-run-anywhere system

This is an interesting, and potentially compelling argument that WebAssembly binaries and runtime actually aren’t the future of compile-once-run-anywhere systems software: these support syscalls, sockets, and stdin/stdout —- all of which requires platform-specific glue code to achieve using WebAssembly and standard libc (or JS).

jart 6y ago
Author here. WebAssembly has played an important role protecting CI on the Internet and it's cool that it enables us to run LLVM in the browser. I was very surprised when I learned that some people were using it offline outside of browsers, since doing so requires a JVM-like runtime such as wasmtime. Cosmopolitan proves that it's possible to just fix C instead, which might fix all the stuff that's built on top of C too. We can in fact have first-class portable native binaries, which will only require JIT compilation if they're launched on a non-x86 machine. This is probably how Java would have been designed, if Intel's instruction set hadn't been encumbered by patents at the time. That's changed. The x86-64 patents just expired this year. So I say why not use it as a canonical software encoding.
- moonchild 6y ago
  Because x86 instruction encoding is a mess. Take a look at this[1] flowchart from the amd manual (vol. 3). That doesn't even tell the whole story; there's a ton of path dependence, and the operand format is quite complicated and has tons of special cases. 1. https://0x0.st/-rce.png https://0x0.st/-rce.png
  - jart 6y ago
    I thought x86 was pretty straightforward once I understood the octal encoding. Cosmopolitan provides a 3kb x86 instruction length decoder that supports all past, present, and future ISAs: https://github.com/jart/cosmopolitan/blob/master/third_party/xed/x86ild.greg.c https://github.com/jart/cosmopolitan/blob/master/third_party... The Cosmopolitan codebase also provides a debugger / emulator for i8086 + x86-64 that's similar to GDB TUI and Bochs. See https://justine.lol/blinkenlights/index.html https://justine.lol/blinkenlights/index.html For example, here's the ALU code: https://github.com/jart/cosmopolitan/blob/master/tool/build/lib/alu.c https://github.com/jart/cosmopolitan/blob/master/tool/build/...
    - Someone 6y ago
      I wouldn’t call 3kb “pretty straightforward”, certainly not relative to the competition. The instruction length decoder for quite a few CPUs is return INSTRUCTION_SIZE; I don’t know any other that has more than a few cases for immediate constants and jump addresses.
  - dnautics 6y ago
    xoreaxeaxeax's videos about how to systematically parse the asm space of x86 really hit home to me how bad the encoding is. https://www.youtube.com/watch?v=KrksBdWcZgQ https://www.youtube.com/watch?v=KrksBdWcZgQ
- Someone 6y ago
  If Sun would have picked a ‘real’ CPU, I doubt it would have been x86 (did they even sell any of them at the time?) Also, picking a real CPU is only worth it for running on that CPU, and that particular one. If they had picked x86 in 1996, they wouldn’t even have supported MMX. In a platform-independent design, using n registers in the virtual CPU is only a good idea for n = 0 (i.e. a stack machine, as in the JVM, .NET CLR and Web Assembly) and n = ∞ (as in LLVM). If you pick any other number your platform-independent code needs to handle both the case where the number of op real registers is smaller (so you need to have a register allocator) and the case where it is larger (there, you could just ignore the additional registers, but that means giving up a lot of performance, so you have to somehow figure out which data would have been in registers if the CPU had more of them). Why write and optimize both of these complex problems, if you can pick a number of registers at either end of the scale, and spend all your resources on one of them? And that’s even more true for x86-64, which doesn’t have a fully orthogonal instruction set, has I don’t know how many ways to do vector instructions, none of which allow you to clearly express how long the vectors you’re iterating over are (making it hard to optimally map them to other CPUs or newer vector instructions), has 80-bit floats, etc. There’s a reason Google stopped work on https://en.wikipedia.org/wiki/Google_Native_Client https://en.wikipedia.org/wiki/Google_Native_Client in favor of Web Assembly.
  - MaxBarraclough 6y ago
    Also, the high-level nature of Java bytecode enables/simplifies many optimisations in the JIT. For example, dynamic dispatch is handled by the JIT, it's not encoded into low-level Java bytecode instructions, so if the JIT can see there's only one class loaded that implements a certain interface, it can generate code that directly invokes methods of that implementing class, without going through a vtable. It can do this even across library boundaries. That wouldn't be possible (or at least, would be greatly complicated) if Java bytecode provided pre-baked machine code. Modern JVMs also have pretty deep integration of the GC and the JIT, if I understand correctly. The Java bytecode format is high level so the JIT is quite free to implement memory-management however it likes. If the JVM took a truly low-level approach to its IR, we'd presumably be stuck with 90's GC technology. I imagine it would also have implications for the way the JVM handles concurrency. It seems right that it defines its own memory model with its own safety guarantees, rather than defining its model as whatever x86 does. It's telling that .Net took the same high-level approach that Java bytecode did.
- aledthemathguy 6y ago
  excuse my ignorance on this, but this sounds amazing. game changing. yet it's being "served" here on ice. casually. what am I missing?
  - morelisp 6y ago
    WORA's been promised for decades. Even if it's true no one will believe it (including me). And portability is not the only reason people choose Rust or Go over C. That's before you get into whether you can aesthetically convince everyone that x86-64 is good, actually. Still very interesting though, especially compared to the usual Show HN crap.
    - the-dude 6y ago
      Your comment would be better without the last sentence.
  - ddalex 6y ago
    Portability was a long desired attribute of computer software up until it turns out that it doesn't matter at all. When software was deployed on floppy disks and there were 10k architectures going around, portability was critical to expand the market reach of any distributed software. But in the modern age when the software runs "in the cloud" and there are dominant platforms out there, turns out that portability doesn't matter - nobody is changing platforms when they don't need to. So this makes this achievement a pretty cool demo without any industry impact, since people don't distribute C binaries anymore. The idea behind it and the technology developed is pretty cool :)
- pjmlp 6y ago
  > We can in fact have first-class portable native binaries, which will only require JIT compilation if they're launched on a non-x86 machine. This is nothing new, IBM and Unisys mainframes have been doing it since the 60's.
  - jart 6y ago
    IBM System/360 and NexGen i586 were big inspirations for the design of Cosmopolitan.
    - thequux 6y ago
      Are you familiar with IBM i (née AS/400)? There's a similar idea there, where programs are compiled to a virtual instruction set called TIMI then statically recompiled to whatever the machine's native instruction set is at the first load on a given machine. As a result, you can just drop a System/36 binary on a modern PowerPC system and not even notice that you've just crossed two changes of instruction set.
- roca 6y ago
  "Only require JIT compilation if they're launched on a non-x86 machine" is a heck of a caveat. The x86-64 instruction set is not a good portable IR. Some well-known problems: * It's much more irregular to decode than other instruction sets/IRs * There is a truly vast set of instructions, very many of which are almost never used. So in practice you need to define the subset of x86-64 you're using. E.g. does your subset contain AVX-512? AVX2? AVX? BMI? X87? MMX? XGETBV? etc etc etc. These decisions will impact the performance of your code on actual x86 CPUs as well as non-x86 CPUs. * x86-64 assumes the TSO memory model which is horribly expensive to support on CPUs whose native model is weaker (e.g. ARM) (which is why Apple added a TSO mode for M1; no other ARM chips have this) Honestly, declaring x86-64 your portable IR and then claiming that as a technological breakthrough sounds like a trick to me. I'd agree it's a breakthrough if you define your x86-64 subset, show your x86-to-ARM compiler, and show that it produces code competitive with your competitors (e.g. WASM compilers).
  - ssokolow 6y ago
    > * It's much more irregular to decode than other instruction sets/IRs According to this blog post, it's not just "much more irregular to decode", instruction deciding is the Achilles Heel of x86 which allows M1 to be so fast by comparison. https://debugger.medium.com/why-is-apples-m1-chip-so-fast-3262b158cba2 https://debugger.medium.com/why-is-apples-m1-chip-so-fast-32...
- MaxBarraclough 6y ago
  > doing so requires a JVM-like runtime such as wasmtime. Cosmopolitan proves that it's possible to just fix C instead This doesn't sound like a fair comparison. WASM isn't just providing portability, it's also providing security and runtime safety. Cosmopolitan doesn't, it presumably requires you to trust the codebase, and doesn't protect you from C's undefined behaviour. Of course, WASM also imposes a considerable performance penalty, I presume the Cosmopolitan approach easily outperforms WASM. > This is probably how Java would have been designed, if Intel's instruction set hadn't been encumbered by patents at the time. To mirror the comment by Someone, I sincerely doubt this. Java bytecode is a very high level stack-based IR, nothing like a CPU ISA. They could easily have made it resemble a CPU ISA if they'd wanted to. The lesser-known GNU Lightning JIT engine takes this approach, for instance.
  - rini17 6y ago
    WASM only provides sandboxing. That is not the same as security nor it means runtime safety nor protection from undefined behavior.
    - MaxBarraclough 6y ago
      > WASM only provides sandboxing. That is not the same as security The relevant Wikipedia article is named Sandbox (computer security). > nor it means runtime safety nor protection from undefined behavior It puts stronger constraints on what mischief undefined behaviour can lead to, and guarantees that various runtime errors are handled with traps. [0] This isn't the same as a hard guarantee that execution will terminate whenever undefined behaviour is invoked, but it's still a step up. [0] https://webassembly.org/docs/security/ https://webassembly.org/docs/security/
      - ssokolow 6y ago
        Speaking as someone who would love to see WebAssembly succeed as a cross-platform, cross-language way for me to write sandboxed plugins and CLI utilities, I do have to point out that, for out-of-browser use, WASM's MVP does regress various things: * https://00f.net/2018/11/25/webassembly-doesnt-make-unsafe-languages-safe/ https://00f.net/2018/11/25/webassembly-doesnt-make-unsafe-la... * https://www.usenix.org/system/files/sec20-lehmann.pdf https://www.usenix.org/system/files/sec20-lehmann.pdf (eg. Under WebAssembly's memory model, dereferencing NULL=0 won't lead to a segfault.)
nn3 6y ago
WebAsm is not performance competitive: "We find that the mean slowdown of WebAssembly vs. native across SPEC benchmarks is 1.55× for Chrome and 1.45× for Firefox, with peak slowdowns of 2.5× in Chrome and 2.08× in Firefox." https://arxiv.org/pdf/1901.09056.pdf https://arxiv.org/pdf/1901.09056.pdf
- comex 6y ago
  x86 isn’t performance competitive either if you’re targeting a CPU that doesn’t run it natively. In fact, it’s much worse than WebAssembly. Even Apple Rosetta takes roughly a 1.25x-1.50x hit on SPEC compared to native [1], and that’s with ahead-of-time compilation and no software sandboxing, whereas WebAssembly VMs typically use JIT compilation and do provide software sandboxing, both of which come with overhead. In contrast, inNative [2] is a WebAssembly compiler that has ahead-of-time compilation and no software sandboxing, and it can supposedly reach 95% of native performance (no idea if the benchmarks are cherry-picked though). This makes sense: WebAssembly is higher-level, closer to a compiler IR, whereas x86 assembly has already decided on low-level details (like register allocation and calling conventions) that are suboptimal to emulate on another architecture. And other x86 emulators will do worse than Rosetta. For one thing, Rosetta can take advantage of the Total Store Ordering mode added to M1 chips for the specific purpose of emulating x86 faster. An emulator running on pretty much any other CPU has to emulate a stronger memory model in software on top of a weaker one, which comes with a massive unavoidable penalty for multithreaded workloads. Also, most x86 emulators are just not as efficient as Rosetta, adding an additional penalty. [1] https://www.anandtech.com/show/16252/mac-mini-apple-m1-tested/6 https://www.anandtech.com/show/16252/mac-mini-apple-m1-teste... [2] https://innative.dev/news/ https://innative.dev/news/
timClicks 6y ago
These binaries wouldn't provide the isolation that wasm provides... Although it's looks like they're essentially unikernels, which means that they could potentially be run per-invocation by a hypervisor.

jart 6y ago

Author here. WebAssembly has played an important role protecting CI on the Internet and it's cool that it enables us to run LLVM in the browser. I was very surprised when I learned that some people were using it offline outside of browsers, since doing so requires a JVM-like runtime such as wasmtime. Cosmopolitan proves that it's possible to just fix C instead, which might fix all the stuff that's built on top of C too. We can in fact have first-class portable native binaries, which will only require JIT compilation if they're launched on a non-x86 machine. This is probably how Java would have been designed, if Intel's instruction set hadn't been encumbered by patents at the time. That's changed. The x86-64 patents just expired this year. So I say why not use it as a canonical software encoding.

moonchild 6y ago
Because x86 instruction encoding is a mess. Take a look at this[1] flowchart from the amd manual (vol. 3). That doesn't even tell the whole story; there's a ton of path dependence, and the operand format is quite complicated and has tons of special cases. 1. https://0x0.st/-rce.png https://0x0.st/-rce.png
- jart 6y ago
  I thought x86 was pretty straightforward once I understood the octal encoding. Cosmopolitan provides a 3kb x86 instruction length decoder that supports all past, present, and future ISAs: https://github.com/jart/cosmopolitan/blob/master/third_party/xed/x86ild.greg.c https://github.com/jart/cosmopolitan/blob/master/third_party... The Cosmopolitan codebase also provides a debugger / emulator for i8086 + x86-64 that's similar to GDB TUI and Bochs. See https://justine.lol/blinkenlights/index.html https://justine.lol/blinkenlights/index.html For example, here's the ALU code: https://github.com/jart/cosmopolitan/blob/master/tool/build/lib/alu.c https://github.com/jart/cosmopolitan/blob/master/tool/build/...
  - Someone 6y ago
    I wouldn’t call 3kb “pretty straightforward”, certainly not relative to the competition. The instruction length decoder for quite a few CPUs is return INSTRUCTION_SIZE; I don’t know any other that has more than a few cases for immediate constants and jump addresses.
- dnautics 6y ago
  xoreaxeaxeax's videos about how to systematically parse the asm space of x86 really hit home to me how bad the encoding is. https://www.youtube.com/watch?v=KrksBdWcZgQ https://www.youtube.com/watch?v=KrksBdWcZgQ
Someone 6y ago
If Sun would have picked a ‘real’ CPU, I doubt it would have been x86 (did they even sell any of them at the time?) Also, picking a real CPU is only worth it for running on that CPU, and that particular one. If they had picked x86 in 1996, they wouldn’t even have supported MMX. In a platform-independent design, using n registers in the virtual CPU is only a good idea for n = 0 (i.e. a stack machine, as in the JVM, .NET CLR and Web Assembly) and n = ∞ (as in LLVM). If you pick any other number your platform-independent code needs to handle both the case where the number of op real registers is smaller (so you need to have a register allocator) and the case where it is larger (there, you could just ignore the additional registers, but that means giving up a lot of performance, so you have to somehow figure out which data would have been in registers if the CPU had more of them). Why write and optimize both of these complex problems, if you can pick a number of registers at either end of the scale, and spend all your resources on one of them? And that’s even more true for x86-64, which doesn’t have a fully orthogonal instruction set, has I don’t know how many ways to do vector instructions, none of which allow you to clearly express how long the vectors you’re iterating over are (making it hard to optimally map them to other CPUs or newer vector instructions), has 80-bit floats, etc. There’s a reason Google stopped work on https://en.wikipedia.org/wiki/Google_Native_Client https://en.wikipedia.org/wiki/Google_Native_Client in favor of Web Assembly.
- MaxBarraclough 6y ago
  Also, the high-level nature of Java bytecode enables/simplifies many optimisations in the JIT. For example, dynamic dispatch is handled by the JIT, it's not encoded into low-level Java bytecode instructions, so if the JIT can see there's only one class loaded that implements a certain interface, it can generate code that directly invokes methods of that implementing class, without going through a vtable. It can do this even across library boundaries. That wouldn't be possible (or at least, would be greatly complicated) if Java bytecode provided pre-baked machine code. Modern JVMs also have pretty deep integration of the GC and the JIT, if I understand correctly. The Java bytecode format is high level so the JIT is quite free to implement memory-management however it likes. If the JVM took a truly low-level approach to its IR, we'd presumably be stuck with 90's GC technology. I imagine it would also have implications for the way the JVM handles concurrency. It seems right that it defines its own memory model with its own safety guarantees, rather than defining its model as whatever x86 does. It's telling that .Net took the same high-level approach that Java bytecode did.
aledthemathguy 6y ago
excuse my ignorance on this, but this sounds amazing. game changing. yet it's being "served" here on ice. casually. what am I missing?
- morelisp 6y ago
  WORA's been promised for decades. Even if it's true no one will believe it (including me). And portability is not the only reason people choose Rust or Go over C. That's before you get into whether you can aesthetically convince everyone that x86-64 is good, actually. Still very interesting though, especially compared to the usual Show HN crap.
  - the-dude 6y ago
    Your comment would be better without the last sentence.
- ddalex 6y ago
  Portability was a long desired attribute of computer software up until it turns out that it doesn't matter at all. When software was deployed on floppy disks and there were 10k architectures going around, portability was critical to expand the market reach of any distributed software. But in the modern age when the software runs "in the cloud" and there are dominant platforms out there, turns out that portability doesn't matter - nobody is changing platforms when they don't need to. So this makes this achievement a pretty cool demo without any industry impact, since people don't distribute C binaries anymore. The idea behind it and the technology developed is pretty cool :)
pjmlp 6y ago
> We can in fact have first-class portable native binaries, which will only require JIT compilation if they're launched on a non-x86 machine. This is nothing new, IBM and Unisys mainframes have been doing it since the 60's.
- jart 6y ago
  IBM System/360 and NexGen i586 were big inspirations for the design of Cosmopolitan.
  - thequux 6y ago
    Are you familiar with IBM i (née AS/400)? There's a similar idea there, where programs are compiled to a virtual instruction set called TIMI then statically recompiled to whatever the machine's native instruction set is at the first load on a given machine. As a result, you can just drop a System/36 binary on a modern PowerPC system and not even notice that you've just crossed two changes of instruction set.
roca 6y ago
"Only require JIT compilation if they're launched on a non-x86 machine" is a heck of a caveat. The x86-64 instruction set is not a good portable IR. Some well-known problems: * It's much more irregular to decode than other instruction sets/IRs * There is a truly vast set of instructions, very many of which are almost never used. So in practice you need to define the subset of x86-64 you're using. E.g. does your subset contain AVX-512? AVX2? AVX? BMI? X87? MMX? XGETBV? etc etc etc. These decisions will impact the performance of your code on actual x86 CPUs as well as non-x86 CPUs. * x86-64 assumes the TSO memory model which is horribly expensive to support on CPUs whose native model is weaker (e.g. ARM) (which is why Apple added a TSO mode for M1; no other ARM chips have this) Honestly, declaring x86-64 your portable IR and then claiming that as a technological breakthrough sounds like a trick to me. I'd agree it's a breakthrough if you define your x86-64 subset, show your x86-to-ARM compiler, and show that it produces code competitive with your competitors (e.g. WASM compilers).
- ssokolow 6y ago
  > * It's much more irregular to decode than other instruction sets/IRs According to this blog post, it's not just "much more irregular to decode", instruction deciding is the Achilles Heel of x86 which allows M1 to be so fast by comparison. https://debugger.medium.com/why-is-apples-m1-chip-so-fast-3262b158cba2 https://debugger.medium.com/why-is-apples-m1-chip-so-fast-32...
MaxBarraclough 6y ago
> doing so requires a JVM-like runtime such as wasmtime. Cosmopolitan proves that it's possible to just fix C instead This doesn't sound like a fair comparison. WASM isn't just providing portability, it's also providing security and runtime safety. Cosmopolitan doesn't, it presumably requires you to trust the codebase, and doesn't protect you from C's undefined behaviour. Of course, WASM also imposes a considerable performance penalty, I presume the Cosmopolitan approach easily outperforms WASM. > This is probably how Java would have been designed, if Intel's instruction set hadn't been encumbered by patents at the time. To mirror the comment by Someone, I sincerely doubt this. Java bytecode is a very high level stack-based IR, nothing like a CPU ISA. They could easily have made it resemble a CPU ISA if they'd wanted to. The lesser-known GNU Lightning JIT engine takes this approach, for instance.
- rini17 6y ago
  WASM only provides sandboxing. That is not the same as security nor it means runtime safety nor protection from undefined behavior.
  - MaxBarraclough 6y ago
    > WASM only provides sandboxing. That is not the same as security The relevant Wikipedia article is named Sandbox (computer security). > nor it means runtime safety nor protection from undefined behavior It puts stronger constraints on what mischief undefined behaviour can lead to, and guarantees that various runtime errors are handled with traps. [0] This isn't the same as a hard guarantee that execution will terminate whenever undefined behaviour is invoked, but it's still a step up. [0] https://webassembly.org/docs/security/ https://webassembly.org/docs/security/
    - ssokolow 6y ago
      Speaking as someone who would love to see WebAssembly succeed as a cross-platform, cross-language way for me to write sandboxed plugins and CLI utilities, I do have to point out that, for out-of-browser use, WASM's MVP does regress various things: * https://00f.net/2018/11/25/webassembly-doesnt-make-unsafe-languages-safe/ https://00f.net/2018/11/25/webassembly-doesnt-make-unsafe-la... * https://www.usenix.org/system/files/sec20-lehmann.pdf https://www.usenix.org/system/files/sec20-lehmann.pdf (eg. Under WebAssembly's memory model, dereferencing NULL=0 won't lead to a segfault.)

nn3 6y ago

WebAsm is not performance competitive: "We find that the mean slowdown of WebAssembly vs. native across SPEC benchmarks is 1.55× for Chrome and 1.45× for Firefox, with peak slowdowns of 2.5× in Chrome and 2.08× in Firefox." https://arxiv.org/pdf/1901.09056.pdf https://arxiv.org/pdf/1901.09056.pdf

comex 6y ago
x86 isn’t performance competitive either if you’re targeting a CPU that doesn’t run it natively. In fact, it’s much worse than WebAssembly. Even Apple Rosetta takes roughly a 1.25x-1.50x hit on SPEC compared to native [1], and that’s with ahead-of-time compilation and no software sandboxing, whereas WebAssembly VMs typically use JIT compilation and do provide software sandboxing, both of which come with overhead. In contrast, inNative [2] is a WebAssembly compiler that has ahead-of-time compilation and no software sandboxing, and it can supposedly reach 95% of native performance (no idea if the benchmarks are cherry-picked though). This makes sense: WebAssembly is higher-level, closer to a compiler IR, whereas x86 assembly has already decided on low-level details (like register allocation and calling conventions) that are suboptimal to emulate on another architecture. And other x86 emulators will do worse than Rosetta. For one thing, Rosetta can take advantage of the Total Store Ordering mode added to M1 chips for the specific purpose of emulating x86 faster. An emulator running on pretty much any other CPU has to emulate a stronger memory model in software on top of a weaker one, which comes with a massive unavoidable penalty for multithreaded workloads. Also, most x86 emulators are just not as efficient as Rosetta, adding an additional penalty. [1] https://www.anandtech.com/show/16252/mac-mini-apple-m1-tested/6 https://www.anandtech.com/show/16252/mac-mini-apple-m1-teste... [2] https://innative.dev/news/ https://innative.dev/news/

timClicks 6y ago

These binaries wouldn't provide the isolation that wasm provides... Although it's looks like they're essentially unikernels, which means that they could potentially be run per-invocation by a hypervisor.