3 ms·

Getting a FS going on the web has been such a brutal slog for me. There's some pretty decent existing projects lole lightning-fs but it feels like the web APIs

Getting a FS going on the web has been such a brutal slog for me. There's some pretty decent existing projects lole lightning-fs but it feels like the web APIs have been changing & changing & I keep wanting newer. Very cool to see this nicely tackled. Still poking around but seems integrative with wasi, which hopefully grows closer & closer with good fast web APIs. There's been a ton of criticism I think justly so that neither the wasi nor the web fs teams have spent much time trying to figure out how to work together, & so there's real peril possible too, but I think wasi is a much better much less hacked out model & is probably the "right" real target. Having build in object capability sandbox security puts it lightyears beyond what the web spec folk have been up to, and the rest of their fs model is also much much better built out. Alas it just means that running on the web the target so going to be primarily in-memory for a long time because the web spec folk seem so uninterested in doing a good actual filesystems impl. Origin Private File System seems to be the one place there's interest in being fast at all, and they seem overwhelmingly concerned about making sqlite in emscripted fast & not a ton else. What we would do for real wasi on the web! Already a very solid start building out coreutils. There's a bunch of similar ish efforts out there already, building JS impls of core utils. For example, posix-tools, https://github.com/shama/posix-tools https://github.com/shama/posix-tools

jakear 4y ago
> running on the web the target so going to be primarily in-memory for a long time What do you mean by this? FS implementations I've seen (including lightning-fs) use IndexedDB for the backing store, which is not "in-memory". I authored VS Code's web file system driver and it also uses IndexedDB, which I didn't see evidence of being memory-constrained. Furthermore, there's the File System Access API for web which is also not in-memory (I also authored VS Code's driver for that...)
- rektide 4y ago
  Color me impressed. I have been aware of indexeddb drivers, and I didn't investigate Vs code's driver (looked briefly through the PR but was short on time). I don't know why I don't credit that route enough. File System Access API just feels way to limited. There's so much missing stuff. No symlinks, no hard links, no xattr. I worry about what the user experience is going to be when the web api has so many limitations.
  - jakear 4y ago
    > There's so much missing stuff. No symlinks, no hard links, no xattr. I worry about what the user experience is going to be Personally, I've never met a "normal" user who cares about any of those. You can try vscode.dev => explorer => "Open Folder" to see what the experience is like for yourself. I'd be interested to hear of the pain points, though I moved on from the project a while back.
cxr 4y ago
> Alas it just means that running on the web the target so going to be primarily in-memory for a long time because the web spec folk seem so uninterested in doing a good actual filesystems impl. Origin Private File System seems to be the one place there's interest in being fast at all, and they seem overwhelmingly concerned about making sqlite in emscripted fast & not a ton else. What we would do for real wasi on the web! Despite the constraints of the browser sandbox and the performance penalties, when you consider all the environments where people have actually used coreutils to great effect, it's undeniable that people were getting real work done in the 90s and early 2000s on strictly worse systems; if you tried to simulate many of those experiences entirely in-browser today, then you still wind up at an advantage (even with modest hardware, like, say, a 5-year old Chromebook) compared to what people were actually getting by with at the time. The biggest stopping force is really the premature fatalism that follows from the comparison to programs that run with the system linker/loader, have unmitigated access on the host, and the freedom ("pleasure"?) of running memory-unsafe code, or so on. I'll further stipulate that programs that are written in such a way that they require any one of these things are fundamentally broken (at least for most values of x).

jakear 4y ago

> running on the web the target so going to be primarily in-memory for a long time What do you mean by this? FS implementations I've seen (including lightning-fs) use IndexedDB for the backing store, which is not "in-memory". I authored VS Code's web file system driver and it also uses IndexedDB, which I didn't see evidence of being memory-constrained. Furthermore, there's the File System Access API for web which is also not in-memory (I also authored VS Code's driver for that...)

rektide 4y ago
Color me impressed. I have been aware of indexeddb drivers, and I didn't investigate Vs code's driver (looked briefly through the PR but was short on time). I don't know why I don't credit that route enough. File System Access API just feels way to limited. There's so much missing stuff. No symlinks, no hard links, no xattr. I worry about what the user experience is going to be when the web api has so many limitations.
- jakear 4y ago
  > There's so much missing stuff. No symlinks, no hard links, no xattr. I worry about what the user experience is going to be Personally, I've never met a "normal" user who cares about any of those. You can try vscode.dev => explorer => "Open Folder" to see what the experience is like for yourself. I'd be interested to hear of the pain points, though I moved on from the project a while back.

cxr 4y ago

> Alas it just means that running on the web the target so going to be primarily in-memory for a long time because the web spec folk seem so uninterested in doing a good actual filesystems impl. Origin Private File System seems to be the one place there's interest in being fast at all, and they seem overwhelmingly concerned about making sqlite in emscripted fast & not a ton else. What we would do for real wasi on the web! Despite the constraints of the browser sandbox and the performance penalties, when you consider all the environments where people have actually used coreutils to great effect, it's undeniable that people were getting real work done in the 90s and early 2000s on strictly worse systems; if you tried to simulate many of those experiences entirely in-browser today, then you still wind up at an advantage (even with modest hardware, like, say, a 5-year old Chromebook) compared to what people were actually getting by with at the time. The biggest stopping force is really the premature fatalism that follows from the comparison to programs that run with the system linker/loader, have unmitigated access on the host, and the freedom ("pleasure"?) of running memory-unsafe code, or so on. I'll further stipulate that programs that are written in such a way that they require any one of these things are fundamentally broken (at least for most values of x).