9 ms·

BrowserFS: An in-browser filesystem supports various back ends

146 points by ngaut 7y ago

McTossOut 7y ago
You got your data abstraction in my web app! You got your web app in my data abstraction!
Etheryte 7y ago
In general, projects like these are great — they unify a lot of underlying architecture and, with time, can make them ubiquitous to use. LocalForage[1] is a great example of this, and I would recommend using that over regular local storage for every serious web project. This project raises my eyebrows at two technical points, though: There is no mention of "Promise" or "async/await" in the readme. I know the project tries to mirror the current Node filesystem, but that also offers Promises[2] — something I would call the industry standard for writing readable async code these days. Despite being in beta, I would argue that this project should also aim to support Promises over nested callback pyramids as the default pattern. Secondly, maybe I'm just not familiar with the idea, but I'm very sceptical of the AsyncMirror backend — when you use an asynchronous service synchronously, how do you guarantee all writes are flushed before the user leaves the page? Other than that, a project of this scale is no small feat, so godspeed to all contributors. [1] https://github.com/localForage/localForage https://github.com/localForage/localForage [2] https://nodejs.org/dist/latest-v10.x/docs/api/fs.html#fs_fs_promises_api https://nodejs.org/dist/latest-v10.x/docs/api/fs.html#fs_fs_...
- drenvuk 7y ago
  Does es6-promisify solve your first concern? The callbacks are definitely a pain.
- jbob2000 7y ago
  It’s straightforward and quick to write a promise wrapper on callbacks. If one of the goals of the project is interoperability, then it makes sense to leave async/promise implementations to the user.
drenvuk 7y ago
I posted this earlier but it failed to gain enough upvotes, thx. BrowserFS is a pretty darned nice library that I'm using to make an extension based file manager. I'm hoping IndexedDB is fast enough for my use case. I haven't been able to benchmark it yet.
- yodon 7y ago
  Are you rolling your own UI or is there a similarly nice file system UI package out there?
  - drenvuk 7y ago
    There're quite a few a file system UIs out there but they're usually coupled to php backends. I found one that's fairly agnostic though. https://github.com/Studio-42/elFinder https://github.com/Studio-42/elFinder Right now I'm modifying the node connector to use BrowserFS. It's a bit of work but should be worth it.
- coribuci 7y ago
  Am I the only one who thinks this is a rootkit ? What's next? A graphics driver in browser ? Seriously the browser looks more and more like malware.
  - badrabbit 7y ago
    Unusual like nalware but lacking the malicious intent(!)
  - drenvuk 7y ago
    To be perfectly honest, yes it does. But what am I to do if I want maximum interoperability and ease of installation? The difference in penetration my project will get via web extension installation vs binary distribution is huge. I can extend it later with native application calls. Not to mention all of the interactive content that we instantly have access to in the browser vs only on a single person's machine. In addition, the extension also works on mobile Firefox for both ios and android when I use Fenec and extend it. There are use cases that you'll see in the future that couldn't grow or exist otherwise. Hyperbolic tldr; Don't be old and crotchety. Join us, lead us, or die.
edu 7y ago
Really nice, but this is not a "File System" but an I/O library.
- MaxBarraclough 7y ago
  It seems to be part of a larger framework which looks something like an in-browser OS. Here's their paper, mentioned in the GitHub page (PDF format): https://people.cs.umass.edu/~emery/pubs/doppio.pdf https://people.cs.umass.edu/~emery/pubs/doppio.pdf
jadbox 7y ago
I haven't used this yet, but I thought I'd also mention RxDB and GUN that also fit in this similar space of cross client<>server db engines. https://github.com/pubkey/rxdb https://github.com/pubkey/rxdb https://gun.eco/ https://gun.eco/ EDIT: I'd love to see a discussion comparing localForage, BrowserFS, GUN, and RxDB :D
- marknadal 7y ago
  Huh, this would be pretty cool. BrowserFS seems to even have Dropbox etc. backends, and GUN already has a NodeJS `fs` adapter, so could theoretical could seamlessly swap BrowserFS in for fs and get GUN backed up to Dropbox!!!
bhl 7y ago
In general, do people recommend using libraries like this for in-browser filesystem or building it from scratch? I’m looking for something to sync files between Google Drive, IndexedDB, and which has a treeview component to display and modify files.
greatjack613 7y ago
The really cool thing about this is it highlights how complex something as simple as a filesystem can be in a browser. Seriously, over 20 ways to backend a filesystem!!! Really really incredible!
CMCDragonkai 7y ago
We wrote something similar a long time ago before knowing about browserfs. js-virtualfs https://github.com/MatrixAI/js-virtualfs/blob/master/README.md https://github.com/MatrixAI/js-virtualfs/blob/master/README.... it has a list of unique features on it that makes it a better simulation of real posix filesystems.
musicale 7y ago
> If you decide to use BrowserFS in a project that leads to a publication, please cite the academic papers on Doppio and Browsix I empathize with their desire for more citations, but this rubs me the wrong way. Either they don't trust authors to cite sources appropriately, or they are asking authors to create inappropriate citations. I suggest trusting authors to cite appropriately. Moreover, using a system isn't the same as building on work from a paper. If one needs to add a reference for a system, it may be more useful to cite a more up-to-date reference for the system rather than the original paper where it was proposed.

146 points by ngaut 7y ago

You got your data abstraction in my web app! You got your web app in my data abstraction!

In general, projects like these are great — they unify a lot of underlying architecture and, with time, can make them ubiquitous to use. LocalForage[1] is a great example of this, and I would recommend using that over regular local storage for every serious web project. This project raises my eyebrows at two technical points, though: There is no mention of "Promise" or "async/await" in the readme. I know the project tries to mirror the current Node filesystem, but that also offers Promises[2] — something I would call the industry standard for writing readable async code these days. Despite being in beta, I would argue that this project should also aim to support Promises over nested callback pyramids as the default pattern. Secondly, maybe I'm just not familiar with the idea, but I'm very sceptical of the AsyncMirror backend — when you use an asynchronous service synchronously, how do you guarantee all writes are flushed before the user leaves the page? Other than that, a project of this scale is no small feat, so godspeed to all contributors. [1] https://github.com/localForage/localForage https://github.com/localForage/localForage [2] https://nodejs.org/dist/latest-v10.x/docs/api/fs.html#fs_fs_promises_api https://nodejs.org/dist/latest-v10.x/docs/api/fs.html#fs_fs_...

drenvuk 7y ago
Does es6-promisify solve your first concern? The callbacks are definitely a pain.
jbob2000 7y ago
It’s straightforward and quick to write a promise wrapper on callbacks. If one of the goals of the project is interoperability, then it makes sense to leave async/promise implementations to the user.

drenvuk 7y ago

I posted this earlier but it failed to gain enough upvotes, thx. BrowserFS is a pretty darned nice library that I'm using to make an extension based file manager. I'm hoping IndexedDB is fast enough for my use case. I haven't been able to benchmark it yet.

yodon 7y ago
Are you rolling your own UI or is there a similarly nice file system UI package out there?
- drenvuk 7y ago
  There're quite a few a file system UIs out there but they're usually coupled to php backends. I found one that's fairly agnostic though. https://github.com/Studio-42/elFinder https://github.com/Studio-42/elFinder Right now I'm modifying the node connector to use BrowserFS. It's a bit of work but should be worth it.
coribuci 7y ago
Am I the only one who thinks this is a rootkit ? What's next? A graphics driver in browser ? Seriously the browser looks more and more like malware.
- badrabbit 7y ago
  Unusual like nalware but lacking the malicious intent(!)
- drenvuk 7y ago
  To be perfectly honest, yes it does. But what am I to do if I want maximum interoperability and ease of installation? The difference in penetration my project will get via web extension installation vs binary distribution is huge. I can extend it later with native application calls. Not to mention all of the interactive content that we instantly have access to in the browser vs only on a single person's machine. In addition, the extension also works on mobile Firefox for both ios and android when I use Fenec and extend it. There are use cases that you'll see in the future that couldn't grow or exist otherwise. Hyperbolic tldr; Don't be old and crotchety. Join us, lead us, or die.

edu 7y ago

Really nice, but this is not a "File System" but an I/O library.

MaxBarraclough 7y ago
It seems to be part of a larger framework which looks something like an in-browser OS. Here's their paper, mentioned in the GitHub page (PDF format): https://people.cs.umass.edu/~emery/pubs/doppio.pdf https://people.cs.umass.edu/~emery/pubs/doppio.pdf

jadbox 7y ago

I haven't used this yet, but I thought I'd also mention RxDB and GUN that also fit in this similar space of cross client<>server db engines. https://github.com/pubkey/rxdb https://github.com/pubkey/rxdb https://gun.eco/ https://gun.eco/ EDIT: I'd love to see a discussion comparing localForage, BrowserFS, GUN, and RxDB :D

marknadal 7y ago
Huh, this would be pretty cool. BrowserFS seems to even have Dropbox etc. backends, and GUN already has a NodeJS `fs` adapter, so could theoretical could seamlessly swap BrowserFS in for fs and get GUN backed up to Dropbox!!!

bhl 7y ago

In general, do people recommend using libraries like this for in-browser filesystem or building it from scratch? I’m looking for something to sync files between Google Drive, IndexedDB, and which has a treeview component to display and modify files.

greatjack613 7y ago

The really cool thing about this is it highlights how complex something as simple as a filesystem can be in a browser. Seriously, over 20 ways to backend a filesystem!!! Really really incredible!

CMCDragonkai 7y ago

We wrote something similar a long time ago before knowing about browserfs. js-virtualfs https://github.com/MatrixAI/js-virtualfs/blob/master/README.md https://github.com/MatrixAI/js-virtualfs/blob/master/README.... it has a list of unique features on it that makes it a better simulation of real posix filesystems.

musicale 7y ago

> If you decide to use BrowserFS in a project that leads to a publication, please cite the academic papers on Doppio and Browsix I empathize with their desire for more citations, but this rubs me the wrong way. Either they don't trust authors to cite sources appropriately, or they are asking authors to create inappropriate citations. I suggest trusting authors to cite appropriately. Moreover, using a system isn't the same as building on work from a paper. If one needs to add a reference for a system, it may be more useful to cite a more up-to-date reference for the system rather than the original paper where it was proposed.