10 ms·

VSCode-WASM: Implement a first version of a WebShell

90 points by tech234a 3y ago

rektide 3y ago
Getting a FS going on the web has been such a brutal slog for me. There's some pretty decent existing projects lole lightning-fs but it feels like the web APIs have been changing & changing & I keep wanting newer. Very cool to see this nicely tackled. Still poking around but seems integrative with wasi, which hopefully grows closer & closer with good fast web APIs. There's been a ton of criticism I think justly so that neither the wasi nor the web fs teams have spent much time trying to figure out how to work together, & so there's real peril possible too, but I think wasi is a much better much less hacked out model & is probably the "right" real target. Having build in object capability sandbox security puts it lightyears beyond what the web spec folk have been up to, and the rest of their fs model is also much much better built out. Alas it just means that running on the web the target so going to be primarily in-memory for a long time because the web spec folk seem so uninterested in doing a good actual filesystems impl. Origin Private File System seems to be the one place there's interest in being fast at all, and they seem overwhelmingly concerned about making sqlite in emscripted fast & not a ton else. What we would do for real wasi on the web! Already a very solid start building out coreutils. There's a bunch of similar ish efforts out there already, building JS impls of core utils. For example, posix-tools, https://github.com/shama/posix-tools https://github.com/shama/posix-tools
- jakear 3y ago
  > running on the web the target so going to be primarily in-memory for a long time What do you mean by this? FS implementations I've seen (including lightning-fs) use IndexedDB for the backing store, which is not "in-memory". I authored VS Code's web file system driver and it also uses IndexedDB, which I didn't see evidence of being memory-constrained. Furthermore, there's the File System Access API for web which is also not in-memory (I also authored VS Code's driver for that...)
  - rektide 3y ago
    Color me impressed. I have been aware of indexeddb drivers, and I didn't investigate Vs code's driver (looked briefly through the PR but was short on time). I don't know why I don't credit that route enough. File System Access API just feels way to limited. There's so much missing stuff. No symlinks, no hard links, no xattr. I worry about what the user experience is going to be when the web api has so many limitations.
    - jakear 3y ago
      > There's so much missing stuff. No symlinks, no hard links, no xattr. I worry about what the user experience is going to be Personally, I've never met a "normal" user who cares about any of those. You can try vscode.dev => explorer => "Open Folder" to see what the experience is like for yourself. I'd be interested to hear of the pain points, though I moved on from the project a while back.
- cxr 3y ago
  > Alas it just means that running on the web the target so going to be primarily in-memory for a long time because the web spec folk seem so uninterested in doing a good actual filesystems impl. Origin Private File System seems to be the one place there's interest in being fast at all, and they seem overwhelmingly concerned about making sqlite in emscripted fast & not a ton else. What we would do for real wasi on the web! Despite the constraints of the browser sandbox and the performance penalties, when you consider all the environments where people have actually used coreutils to great effect, it's undeniable that people were getting real work done in the 90s and early 2000s on strictly worse systems; if you tried to simulate many of those experiences entirely in-browser today, then you still wind up at an advantage (even with modest hardware, like, say, a 5-year old Chromebook) compared to what people were actually getting by with at the time. The biggest stopping force is really the premature fatalism that follows from the comparison to programs that run with the system linker/loader, have unmitigated access on the host, and the freedom ("pleasure"?) of running memory-unsafe code, or so on. I'll further stipulate that programs that are written in such a way that they require any one of these things are fundamentally broken (at least for most values of x).
syrusakbary 3y ago
The goal of the extension is great, as it allows developers to use WASI easily when creating new extensions. Mad props to the team that worked on it. But is unfortunate that they used literally the name of the most famous extension for WebAssembly in VSCode (vscode-wasm), which has been around already for a few years, as it might confuse people on what is what: https://marketplace.visualstudio.com/items?itemName=dtsvet.vscode-wasm https://marketplace.visualstudio.com/items?itemName=dtsvet.v... (113k installations) Repo here: https://github.com/wasmerio/vscode-wasm https://github.com/wasmerio/vscode-wasm
- nl 3y ago
  > allows developers to use WASI easily when creating new extensions What is WASI?
  - re-thc 3y ago
    API for Webassembly. Common interface.
  - nindalf 3y ago
    > WASI stands for WebAssembly System Interface. It's an API designed by the Wasmtime project that provides access to several operating-system-like features, including files and filesystems, Berkeley sockets, clocks, and random numbers, that we'll be proposing for standardization. Read more here - https://github.com/bytecodealliance/wasmtime/blob/main/docs/WASI-intro.md https://github.com/bytecodealliance/wasmtime/blob/main/docs/...
    - throwaway525142 3y ago
      Permalink: https://github.com/bytecodealliance/wasmtime/blob/8eccc63c46503adbc6ba91034324963b69a7523b/docs/WASI-intro.md https://github.com/bytecodealliance/wasmtime/blob/8eccc63c46....
  - cjbprime 3y ago
    Am not expert, but here's the understanding I've picked up: When you use JavaScript through Node, your JS gets the ability to open files and make network sockets and so on, because there are a set of APIs implemented by Node and exposed to your JS code, like so: JS <-> native APIs <-> Node implementation of platform-specific operations WASI applies to WebAssembly instead of JS. So again if we're on Node: wasm <-> native APIs (WASI) <-> Node implementation of platform-specific operations But WASI is much more general than Node's APIs, and meant to be used by different engines in different environments (browsers, native bindings, etc).
yewenjie 3y ago
It is kind of sad that Monaco (VS Code's editor as a JS web component) still doesn't support tree-sitter.
- debugnik 3y ago
  Tree-sitter grammars take more effort than textmate ones for the most basic highlighting, but provide less accurate parse trees than a compiler could use for more advanced features and highlighting. I understand what they were going for but I don't "get" the trade-off. At least as long as the parse trees are inherently worse: Even ignoring context-dependent grammars, last time I checked tree-sitter parsers had many edge cases in which they'd accept invalid syntax but not mark the parse trees as bad, which makes them unsuitable for compiler use.
  - DannyBee 3y ago
    Using compilers for highlighting basically doesn't happen because they are too slow. They are used for progressive improvement as a result (depending how fast, it can take at least 5-10 seconds post-update to get a result. For slower languages, minutes). I'm not sure what "more effort" means - for lots of languages, it's less effort than textmate because people spend lots of hacks trying to work around textmate to get it to highlight even slightly complex languages right. It's nesting mechanism is complicated, etc. Tree-sitter is faster than textmate on initial parse in lots of cases, and update a lot faster than textmate (since they can be incrementally updated and textmate can't). They are not meant as compiler replacements. Again, compilers are slow, and rarely incremental. Using tree sitter for folding/highlighting/etc, and compilers for additional progressive type info, is the right path. Textmate is worse than useless for anything but super simple languages. I say this as a person who has written about 15 vscode language servers for all sorts of varied languages, from CNC g-code to you name it. I even implemented incremental parsing/lexing for ANTLR's typescript port (and then java port), as a comparison to tree-sitter ;) It can be made as fast, but tree-sitter is clearly winning the current popularity contest ;) I'd say for those languages, textmate work was the worst part of all of them, and getting textmate to highlight properly required more work than writing proper lexers/parsers. I still fix more bugs in textmate grammars than anywhere else. Part of the reason is that it requires fairly complex regexps. Part of it is getting it to stop trigger on things it shouldn't, because you have to emulate a lot of state. Again, you start to spend a lot of time working around it, rather than working with it. Meanwhile, yes, the upfront work of a lexer in tree-sitter is sometimes more. But it is much easier to get it to work, to rely on it working, etc. In the end, all my language servers were easier to implement in tree-sitter, and some don't even have compilers (like g-code), but have control flow/blocks/etc. tree-sitter handled all language duties there. Hopefully this helps you "get" the tradeoff. Anyone who has written enough vscode language servers would tell you the same thing. I have yet to meet someone in the community who writes language servers who is excited or happy with textmate vs tree-sitter. Where do you think the push came from?
JoelMcCracken 3y ago
Can anyone explain what this is? all of the comments seem to be talking about different things, and when I look at code though it looks to me like they're implementing a JS based terminal?
pipes 3y ago
Can anyone explain what this is?
csense 3y ago
So... what is a WebShell, and how does it differ from, say, what you get if you compile a "lightweight" shell like Ash to a WASM / WASI target?

90 points by tech234a 3y ago

Getting a FS going on the web has been such a brutal slog for me. There's some pretty decent existing projects lole lightning-fs but it feels like the web APIs have been changing & changing & I keep wanting newer. Very cool to see this nicely tackled. Still poking around but seems integrative with wasi, which hopefully grows closer & closer with good fast web APIs. There's been a ton of criticism I think justly so that neither the wasi nor the web fs teams have spent much time trying to figure out how to work together, & so there's real peril possible too, but I think wasi is a much better much less hacked out model & is probably the "right" real target. Having build in object capability sandbox security puts it lightyears beyond what the web spec folk have been up to, and the rest of their fs model is also much much better built out. Alas it just means that running on the web the target so going to be primarily in-memory for a long time because the web spec folk seem so uninterested in doing a good actual filesystems impl. Origin Private File System seems to be the one place there's interest in being fast at all, and they seem overwhelmingly concerned about making sqlite in emscripted fast & not a ton else. What we would do for real wasi on the web! Already a very solid start building out coreutils. There's a bunch of similar ish efforts out there already, building JS impls of core utils. For example, posix-tools, https://github.com/shama/posix-tools https://github.com/shama/posix-tools

jakear 3y ago
> running on the web the target so going to be primarily in-memory for a long time What do you mean by this? FS implementations I've seen (including lightning-fs) use IndexedDB for the backing store, which is not "in-memory". I authored VS Code's web file system driver and it also uses IndexedDB, which I didn't see evidence of being memory-constrained. Furthermore, there's the File System Access API for web which is also not in-memory (I also authored VS Code's driver for that...)
- rektide 3y ago
  Color me impressed. I have been aware of indexeddb drivers, and I didn't investigate Vs code's driver (looked briefly through the PR but was short on time). I don't know why I don't credit that route enough. File System Access API just feels way to limited. There's so much missing stuff. No symlinks, no hard links, no xattr. I worry about what the user experience is going to be when the web api has so many limitations.
  - jakear 3y ago
    > There's so much missing stuff. No symlinks, no hard links, no xattr. I worry about what the user experience is going to be Personally, I've never met a "normal" user who cares about any of those. You can try vscode.dev => explorer => "Open Folder" to see what the experience is like for yourself. I'd be interested to hear of the pain points, though I moved on from the project a while back.
cxr 3y ago
> Alas it just means that running on the web the target so going to be primarily in-memory for a long time because the web spec folk seem so uninterested in doing a good actual filesystems impl. Origin Private File System seems to be the one place there's interest in being fast at all, and they seem overwhelmingly concerned about making sqlite in emscripted fast & not a ton else. What we would do for real wasi on the web! Despite the constraints of the browser sandbox and the performance penalties, when you consider all the environments where people have actually used coreutils to great effect, it's undeniable that people were getting real work done in the 90s and early 2000s on strictly worse systems; if you tried to simulate many of those experiences entirely in-browser today, then you still wind up at an advantage (even with modest hardware, like, say, a 5-year old Chromebook) compared to what people were actually getting by with at the time. The biggest stopping force is really the premature fatalism that follows from the comparison to programs that run with the system linker/loader, have unmitigated access on the host, and the freedom ("pleasure"?) of running memory-unsafe code, or so on. I'll further stipulate that programs that are written in such a way that they require any one of these things are fundamentally broken (at least for most values of x).

syrusakbary 3y ago

The goal of the extension is great, as it allows developers to use WASI easily when creating new extensions. Mad props to the team that worked on it. But is unfortunate that they used literally the name of the most famous extension for WebAssembly in VSCode (vscode-wasm), which has been around already for a few years, as it might confuse people on what is what: https://marketplace.visualstudio.com/items?itemName=dtsvet.vscode-wasm https://marketplace.visualstudio.com/items?itemName=dtsvet.v... (113k installations) Repo here: https://github.com/wasmerio/vscode-wasm https://github.com/wasmerio/vscode-wasm

nl 3y ago
> allows developers to use WASI easily when creating new extensions What is WASI?
- re-thc 3y ago
  API for Webassembly. Common interface.
- nindalf 3y ago
  > WASI stands for WebAssembly System Interface. It's an API designed by the Wasmtime project that provides access to several operating-system-like features, including files and filesystems, Berkeley sockets, clocks, and random numbers, that we'll be proposing for standardization. Read more here - https://github.com/bytecodealliance/wasmtime/blob/main/docs/WASI-intro.md https://github.com/bytecodealliance/wasmtime/blob/main/docs/...
  - throwaway525142 3y ago
    Permalink: https://github.com/bytecodealliance/wasmtime/blob/8eccc63c46503adbc6ba91034324963b69a7523b/docs/WASI-intro.md https://github.com/bytecodealliance/wasmtime/blob/8eccc63c46....
- cjbprime 3y ago
  Am not expert, but here's the understanding I've picked up: When you use JavaScript through Node, your JS gets the ability to open files and make network sockets and so on, because there are a set of APIs implemented by Node and exposed to your JS code, like so: JS <-> native APIs <-> Node implementation of platform-specific operations WASI applies to WebAssembly instead of JS. So again if we're on Node: wasm <-> native APIs (WASI) <-> Node implementation of platform-specific operations But WASI is much more general than Node's APIs, and meant to be used by different engines in different environments (browsers, native bindings, etc).

yewenjie 3y ago

It is kind of sad that Monaco (VS Code's editor as a JS web component) still doesn't support tree-sitter.

debugnik 3y ago
Tree-sitter grammars take more effort than textmate ones for the most basic highlighting, but provide less accurate parse trees than a compiler could use for more advanced features and highlighting. I understand what they were going for but I don't "get" the trade-off. At least as long as the parse trees are inherently worse: Even ignoring context-dependent grammars, last time I checked tree-sitter parsers had many edge cases in which they'd accept invalid syntax but not mark the parse trees as bad, which makes them unsuitable for compiler use.
- DannyBee 3y ago
  Using compilers for highlighting basically doesn't happen because they are too slow. They are used for progressive improvement as a result (depending how fast, it can take at least 5-10 seconds post-update to get a result. For slower languages, minutes). I'm not sure what "more effort" means - for lots of languages, it's less effort than textmate because people spend lots of hacks trying to work around textmate to get it to highlight even slightly complex languages right. It's nesting mechanism is complicated, etc. Tree-sitter is faster than textmate on initial parse in lots of cases, and update a lot faster than textmate (since they can be incrementally updated and textmate can't). They are not meant as compiler replacements. Again, compilers are slow, and rarely incremental. Using tree sitter for folding/highlighting/etc, and compilers for additional progressive type info, is the right path. Textmate is worse than useless for anything but super simple languages. I say this as a person who has written about 15 vscode language servers for all sorts of varied languages, from CNC g-code to you name it. I even implemented incremental parsing/lexing for ANTLR's typescript port (and then java port), as a comparison to tree-sitter ;) It can be made as fast, but tree-sitter is clearly winning the current popularity contest ;) I'd say for those languages, textmate work was the worst part of all of them, and getting textmate to highlight properly required more work than writing proper lexers/parsers. I still fix more bugs in textmate grammars than anywhere else. Part of the reason is that it requires fairly complex regexps. Part of it is getting it to stop trigger on things it shouldn't, because you have to emulate a lot of state. Again, you start to spend a lot of time working around it, rather than working with it. Meanwhile, yes, the upfront work of a lexer in tree-sitter is sometimes more. But it is much easier to get it to work, to rely on it working, etc. In the end, all my language servers were easier to implement in tree-sitter, and some don't even have compilers (like g-code), but have control flow/blocks/etc. tree-sitter handled all language duties there. Hopefully this helps you "get" the tradeoff. Anyone who has written enough vscode language servers would tell you the same thing. I have yet to meet someone in the community who writes language servers who is excited or happy with textmate vs tree-sitter. Where do you think the push came from?

JoelMcCracken 3y ago

Can anyone explain what this is? all of the comments seem to be talking about different things, and when I look at code though it looks to me like they're implementing a JS based terminal?

pipes 3y ago

Can anyone explain what this is?

csense 3y ago

So... what is a WebShell, and how does it differ from, say, what you get if you compile a "lightweight" shell like Ash to a WASM / WASI target?