7 ms·

Stop Hiding the Sharp Knives: The WebAssembly Linux Interface

98 points by yurivish 3y ago

flohofwoe 3y ago
I would really love being able to take any POSIX command line tool, compile that to WASI, and run it on (at least) Linux, Windows and macOS like a regular executable without having to install a separate WASI runtime or any scare-popups showing up like on macOS. I'm a 'WASI convert' since I was able to take an ancient 8-bit assembler written in C89 in the mid-90's (http://xi6.com/projects/asmx/ http://xi6.com/projects/asmx/), compile that as-is with the WASI SDK (https://github.com/WebAssembly/wasi-sdk https://github.com/WebAssembly/wasi-sdk), and then integrate it into a VSCode extension (https://marketplace.visualstudio.com/items?itemName=floooh.vscode-kcide https://marketplace.visualstudio.com/items?itemName=floooh.v...). A similar problem I'd like to solve better is a shader cross-compiler (https://github.com/floooh/sokol-tools https://github.com/floooh/sokol-tools) which needs to run on Linux, macOS and Windows and takes too long to build locally, thus I currently need to distribute that as pre-built binaries. Compiling this to WASI works, but the filesystem access restrictions built into current WASM runtimes are a bit of a hassle to manage, and a WASM runtime needs to be installed, which is also an additional step which shouldn't be needed.
goombacloud 3y ago
I really think we should have means of spawning wasm components from wasm components. How the runtime runs them should be up to the runtime - it could be directly backed by kernel primitives but it could also be in a browser. Leaking posix things into wasm… is something I'd rather never want to see. Let's come up with something better as wasm did aim for from the start.
evujumenuk 3y ago
This completely kills all of my excitement for WASM, because it is clear that WALI, or something conceptually close to it, will definitely prevail over the capability-based alternatives, simply because people love compiling their existing programs for new platforms so much, throwing out the advantages brought about by their differences in exchange for sweet, sweet convenience. It doesn't matter that entire layers of indirection, accumulated over literal decades, could be torn away and consigned to the grave, as long as Joe Sixpack can compile ffmpeg on WASM without lifting a finger. I'm not buying that WALI is just going to be a stepping stone for a more restrictive platform. The mere existence of an anything-goes-screw-safety runtime environment will precipitate a worse-is-better avalanche of ported software, completely drowning out any interest in an improved development target, and there's simply nothing anyone could fathomably do about it.
superkuh 3y ago
Websites are at least supposedly are sandboxed so they are not as much of a risk as running native binaries. But this is getting worse and worse though as browsers expose more and more of their host operating system's functionality. The benefits of using a website instead of a native app are quickly disappearing, while the drawbacks have only been somewhat mitigated. We're getting to the point where browsers are worthy of the decades old criticism Emacs has received. They have eventually become an OS with many fine features - simply lacking a good web browser.
- nulld3v 3y ago
  But this isn't for the browser though. The idea here is to use WASM kinda like Java. You can run the same binary on different systems.
  - titzer 3y ago
    Right, we weren't envisioning that browsers would be required to implement WALI or expose it. That would probably be a bad idea. :)
  - ElijahLynn 3y ago
    Ahh, thanks for explaining this. At first I was like "what, WASM is already portable, why have a Linux specific thing" and then you answered my question.
- esrauch 3y ago
  Essentially no new sandbox capabilities are being added for wasm in the browser compared to JavaScript and that's not changing anytime soon. This shiny new stuff is about wasm for security reasons on desktop, embedded/IOT, Cloud server stuff instead of native binaries.
rockwotj 3y ago
I feel like we’re having our xkcd competing standards moment for WebAssembly outside the browser. There is WASI (preview1 and preview2 with components), WASIX, and now WALI proposed here.
- rockwotj 3y ago
  I sort of wish that these could be new additions to proposals/extensions to WASI instead of separate standards
- striking 3y ago
  There's a paragraph with the heading "The Need for a Complete Wasm OS Interface" in the introduction that breaks down the differences.
- titzer 3y ago
  This is not a competing standard, it's a complementary, lower-level interface that can be used to implement (and sandbox) the WASI implementation. It's defense-in-depth and layering: WASI moves up a layer, and not every engine needs to implement it independently, as it can be a Wasm module that runs on any engine.
SuperNinKenDo 3y ago
Everything feels like it's going the way of modern USB. One universal standard that you're never sure is going to be compatible with anything you try and use it with or for.
mikewarot 3y ago
> Unlike WASI, WALI does not prescribe capability-based security for filesystem and sockets. Instead, the relaxed security model of WALI allows existing APIs (e.g. WASI) to be implemented over it (Fig. 6). Oh no.... I was worried someone was going to do this. You might as well skip WASM altogether if you start down that road. If you give away file system access by default, what's the difference between WASM and running native code? The key advantage of WASM is the capability based security model. This throws that away.
- api 3y ago
  Capability based security would be nice, but nobody is going to use it because it requires all software to be at least partly rewritten. If we could introduce these kinds of fundamental improvements at the base layer of OSes we would not need containers, orchestrators, or virtual machines. We could just fix the OS to have all the necessary controls, fault tolerance, and other abstractions and “do it right.” There was a serious effort to do this in the 90s with things like Plan 9, the JVM, and Mosix among other things. Its all dead. Containers, VMs, overlays, orchestrators, etc are all hacks because we are permanently stuck with 1970s OS paradigms due to sunk cost. Edit: forgot to mention async programming and/or green threads like goroutines. Those are a massively complex and costly hack to get around the fact that both OS concurrency APIs and OS I/O APIs are 1970s relics. You should just be able to spawn threads and you should have native zero copy event driven I/O, but again we can’t fix the foundation.
  - mikewarot 3y ago
    >Capability based security would be nice, but nobody is going to use it because it requires all software to be at least partly rewritten. I can just imagine someone in 1910 saying "Circuit Breakers in Houses would be nice, but nobody is going to use them because they require all houses to be at least partly rewired". Capability based security is the main feature of WASM. Going around it is like putting pennies in a fuse box, or wiring around circuit breakers. The circuit breaker prevents a load from consuming all available electrical resources (burning down the house as a side effect). Capability based security does the same thing for ensuring code doesn't have unlimited side effects. If you need to run native code, just do so. Don't sabotage the only decent chance at finally fixing computer security since Multics to do so. Almost every consumer installable electrical load out there can plug into an outlet. The big heavy loads require an electrician to install them. If your code is the equivalent of an industrial load, you shouldn't be trying to cram it into WASM, the equivalent of a 15 amp 110 GFI circuit.
syrusakbary 3y ago
I love seeing this. The more work we do on top of WASI the more I realize their design flaws when trying to run already existing programs on top of it. WASIX aims to fix many of these flaws, but the underlying problems from WASI always arises (such as mandatory relative paths). I welcome WALI with open hands and excitement. If anyone wants to add support for it in Wasmer I’d be more than happy to integrate it there! For those interested, as is not trivial to find it, the source code of WALI is here: https://github.com/arjunr2/WALI https://github.com/arjunr2/WALI
panic 3y ago
One nice thing about WebAssembly is that it can run in a browser. How would you implement things like fork() in that context, given the inability to manipulate the stack?
- abetusk 3y ago
  Web workers? [0] [0] https://emscripten.org/docs/api_reference/wasm_workers.html https://emscripten.org/docs/api_reference/wasm_workers.html
- syrusakbary 3y ago
  In WASIX we fully solved that, so is definitely doable to run fork on the browser. Although I'm extremely curious the WALI fork strategy would work in the browser. Maybe the creators can chime in here?
  - kg 3y ago
    fork in the browser is functionally impossible unless you mean some sort of 'damaged fork' where only the native wasm heap is mirrored and not everything else, which would be effectively useless.
    - deleted 3y ago
      [deleted]
  - yuri91 3y ago
    Could you elaborate on how you solved the issue? A link to the relevant code would be nice.
    - syrusakbary 3y ago
      Sure thing: https://github.com/wasix-org/wasix-libc/blob/main/libc-top-half/musl/src/process/fork.c https://github.com/wasix-org/wasix-libc/blob/main/libc-top-h...
      - 3y ago
pxeger1 3y ago
It’s not clear to me how this is especially novel. Why is it a paper and not just like a GitHub repo or something?
- sheepshear 3y ago
  Papers are for resumes and funding. Novelty is something the journal or investor could define and require, if they wanted it.

98 points by yurivish 3y ago

I would really love being able to take any POSIX command line tool, compile that to WASI, and run it on (at least) Linux, Windows and macOS like a regular executable without having to install a separate WASI runtime or any scare-popups showing up like on macOS. I'm a 'WASI convert' since I was able to take an ancient 8-bit assembler written in C89 in the mid-90's (http://xi6.com/projects/asmx/ http://xi6.com/projects/asmx/), compile that as-is with the WASI SDK (https://github.com/WebAssembly/wasi-sdk https://github.com/WebAssembly/wasi-sdk), and then integrate it into a VSCode extension (https://marketplace.visualstudio.com/items?itemName=floooh.vscode-kcide https://marketplace.visualstudio.com/items?itemName=floooh.v...). A similar problem I'd like to solve better is a shader cross-compiler (https://github.com/floooh/sokol-tools https://github.com/floooh/sokol-tools) which needs to run on Linux, macOS and Windows and takes too long to build locally, thus I currently need to distribute that as pre-built binaries. Compiling this to WASI works, but the filesystem access restrictions built into current WASM runtimes are a bit of a hassle to manage, and a WASM runtime needs to be installed, which is also an additional step which shouldn't be needed.

goombacloud 3y ago

I really think we should have means of spawning wasm components from wasm components. How the runtime runs them should be up to the runtime - it could be directly backed by kernel primitives but it could also be in a browser. Leaking posix things into wasm… is something I'd rather never want to see. Let's come up with something better as wasm did aim for from the start.

evujumenuk 3y ago

This completely kills all of my excitement for WASM, because it is clear that WALI, or something conceptually close to it, will definitely prevail over the capability-based alternatives, simply because people love compiling their existing programs for new platforms so much, throwing out the advantages brought about by their differences in exchange for sweet, sweet convenience. It doesn't matter that entire layers of indirection, accumulated over literal decades, could be torn away and consigned to the grave, as long as Joe Sixpack can compile ffmpeg on WASM without lifting a finger. I'm not buying that WALI is just going to be a stepping stone for a more restrictive platform. The mere existence of an anything-goes-screw-safety runtime environment will precipitate a worse-is-better avalanche of ported software, completely drowning out any interest in an improved development target, and there's simply nothing anyone could fathomably do about it.

superkuh 3y ago

Websites are at least supposedly are sandboxed so they are not as much of a risk as running native binaries. But this is getting worse and worse though as browsers expose more and more of their host operating system's functionality. The benefits of using a website instead of a native app are quickly disappearing, while the drawbacks have only been somewhat mitigated. We're getting to the point where browsers are worthy of the decades old criticism Emacs has received. They have eventually become an OS with many fine features - simply lacking a good web browser.

nulld3v 3y ago
But this isn't for the browser though. The idea here is to use WASM kinda like Java. You can run the same binary on different systems.
- titzer 3y ago
  Right, we weren't envisioning that browsers would be required to implement WALI or expose it. That would probably be a bad idea. :)
- ElijahLynn 3y ago
  Ahh, thanks for explaining this. At first I was like "what, WASM is already portable, why have a Linux specific thing" and then you answered my question.
esrauch 3y ago
Essentially no new sandbox capabilities are being added for wasm in the browser compared to JavaScript and that's not changing anytime soon. This shiny new stuff is about wasm for security reasons on desktop, embedded/IOT, Cloud server stuff instead of native binaries.

rockwotj 3y ago

I feel like we’re having our xkcd competing standards moment for WebAssembly outside the browser. There is WASI (preview1 and preview2 with components), WASIX, and now WALI proposed here.

rockwotj 3y ago
I sort of wish that these could be new additions to proposals/extensions to WASI instead of separate standards
striking 3y ago
There's a paragraph with the heading "The Need for a Complete Wasm OS Interface" in the introduction that breaks down the differences.
titzer 3y ago
This is not a competing standard, it's a complementary, lower-level interface that can be used to implement (and sandbox) the WASI implementation. It's defense-in-depth and layering: WASI moves up a layer, and not every engine needs to implement it independently, as it can be a Wasm module that runs on any engine.

SuperNinKenDo 3y ago

Everything feels like it's going the way of modern USB. One universal standard that you're never sure is going to be compatible with anything you try and use it with or for.

mikewarot 3y ago

> Unlike WASI, WALI does not prescribe capability-based security for filesystem and sockets. Instead, the relaxed security model of WALI allows existing APIs (e.g. WASI) to be implemented over it (Fig. 6). Oh no.... I was worried someone was going to do this. You might as well skip WASM altogether if you start down that road. If you give away file system access by default, what's the difference between WASM and running native code? The key advantage of WASM is the capability based security model. This throws that away.

api 3y ago
Capability based security would be nice, but nobody is going to use it because it requires all software to be at least partly rewritten. If we could introduce these kinds of fundamental improvements at the base layer of OSes we would not need containers, orchestrators, or virtual machines. We could just fix the OS to have all the necessary controls, fault tolerance, and other abstractions and “do it right.” There was a serious effort to do this in the 90s with things like Plan 9, the JVM, and Mosix among other things. Its all dead. Containers, VMs, overlays, orchestrators, etc are all hacks because we are permanently stuck with 1970s OS paradigms due to sunk cost. Edit: forgot to mention async programming and/or green threads like goroutines. Those are a massively complex and costly hack to get around the fact that both OS concurrency APIs and OS I/O APIs are 1970s relics. You should just be able to spawn threads and you should have native zero copy event driven I/O, but again we can’t fix the foundation.
- mikewarot 3y ago
  >Capability based security would be nice, but nobody is going to use it because it requires all software to be at least partly rewritten. I can just imagine someone in 1910 saying "Circuit Breakers in Houses would be nice, but nobody is going to use them because they require all houses to be at least partly rewired". Capability based security is the main feature of WASM. Going around it is like putting pennies in a fuse box, or wiring around circuit breakers. The circuit breaker prevents a load from consuming all available electrical resources (burning down the house as a side effect). Capability based security does the same thing for ensuring code doesn't have unlimited side effects. If you need to run native code, just do so. Don't sabotage the only decent chance at finally fixing computer security since Multics to do so. Almost every consumer installable electrical load out there can plug into an outlet. The big heavy loads require an electrician to install them. If your code is the equivalent of an industrial load, you shouldn't be trying to cram it into WASM, the equivalent of a 15 amp 110 GFI circuit.

syrusakbary 3y ago

I love seeing this. The more work we do on top of WASI the more I realize their design flaws when trying to run already existing programs on top of it. WASIX aims to fix many of these flaws, but the underlying problems from WASI always arises (such as mandatory relative paths). I welcome WALI with open hands and excitement. If anyone wants to add support for it in Wasmer I’d be more than happy to integrate it there! For those interested, as is not trivial to find it, the source code of WALI is here: https://github.com/arjunr2/WALI https://github.com/arjunr2/WALI

panic 3y ago

One nice thing about WebAssembly is that it can run in a browser. How would you implement things like fork() in that context, given the inability to manipulate the stack?

abetusk 3y ago
Web workers? [0] [0] https://emscripten.org/docs/api_reference/wasm_workers.html https://emscripten.org/docs/api_reference/wasm_workers.html
syrusakbary 3y ago
In WASIX we fully solved that, so is definitely doable to run fork on the browser. Although I'm extremely curious the WALI fork strategy would work in the browser. Maybe the creators can chime in here?
- kg 3y ago
  fork in the browser is functionally impossible unless you mean some sort of 'damaged fork' where only the native wasm heap is mirrored and not everything else, which would be effectively useless.
  - deleted 3y ago
    [deleted]
- yuri91 3y ago
  Could you elaborate on how you solved the issue? A link to the relevant code would be nice.
  - syrusakbary 3y ago
    Sure thing: https://github.com/wasix-org/wasix-libc/blob/main/libc-top-half/musl/src/process/fork.c https://github.com/wasix-org/wasix-libc/blob/main/libc-top-h...
    - 3y ago

pxeger1 3y ago

It’s not clear to me how this is especially novel. Why is it a paper and not just like a GitHub repo or something?

sheepshear 3y ago
Papers are for resumes and funding. Novelty is something the journal or investor could define and require, if they wanted it.