4 ms·

This is a good writeup. It's worth noting that this loads the entire db into memory, and only saves to the filesystem when you tell it to (not on each transact

This is a good writeup. It's worth noting that this loads the entire db into memory, and only saves to the filesystem when you tell it to (not on each transaction) so you could loose changes on a crash. There is a brilliant project to add true transactional flushing and a block like storage backend to WASM Sqlite.js called "Absurd SQL", worth checking out. It's currently built on top of IndexedDB but they are working with the WG designing the FileSystemAccess APIs to ensure it has suitable block level support and locking for this type of tool. https://github.com/jlongster/absurd-sql https://github.com/jlongster/absurd-sql https://jlongster.com/future-sql-web https://jlongster.com/future-sql-web

wngr 5y ago
Combining these two approaches would let you get rid of the IndexedDB abomination.
- samwillis 5y ago
  I believe FileSystemAccessAPI, as implemented in browsers, does not yet support block level access and locking. Both required to make this work, the developer behind Absurd SQL is working with the team designing the API to ensure it will have support. IndexedDB is the only way of doing this on browsers currently. https://github.com/WICG/file-system-access/issues/323 https://github.com/WICG/file-system-access/issues/323
  - dmitriid 5y ago
    Unless I'm mistaken, FileSystemAccessAPI is only implemented in Chrome. Both Safari and Firefox are not implementing until four different file proposals (all from Chrome) can be reduced to one.
    - tehbeard 5y ago
      I thought Firefox flat out didn't want to implement FileSystem APIs on grounds of "security" and protecting their users?
      - ildon 5y ago
        Yes, many are refusing to implement it due to security concerns. A notable one is Brave, which is based on Chromium, and yet refuses to enable it (https://github.com/brave/brave-browser/issues/11407 https://github.com/brave/brave-browser/issues/11407). I'm skeptical that reducing access to the FS actually protects users. Those that would be fooled by scams based on FileSystemAccess APIs would very likely be fooled also with other less intricate tactics. So I doubt that the overall security of users is in practice affected. At least, browsers that refuse to implement the FileSystemAccess APIs could implement them, but leave them disabled by default, and require some non-trivial action to enable them. So users with a very basic understanding of how things work in the browser would not be able to enable them.
        dmitriid 5y ago
        > So I doubt that the overall security of users is in practice affected. The problem is the sheer number of APIs that Chrome ships and wants other browsers to ship, and what browsers already ship that require access via prompts: camera, location, notifications, file access, bluetooth, usb, motion sensors, serial ports, midi devices, clipboard... Just prompting user to allow stuff is no longer enough, and adding more prompts leads to worse security.
        slaymaker1907 5y ago
        It's sort of a double prompt scenario since the FileSystemAccessAPI requires users to actually select the file/directory for the web app to use. You can save the file handles into IndexedDB in which case the user will only be prompted if they refresh the page to authorize the web app to continue using that file/directory. From my use of it, the only real vulnerability I see is that Chrome still considers anything from file:// to be the same origin. This means as a developer you absolutely should not be saving file handles to IndexedDB if you are loading your app via file:// instead of https:// https://. This is a pretty niche use case, but I do think static html "apps" are an underappreciated form for distributing software and this new API makes such apps a lot more powerful. I wrote a plugin for TiddlyWiki that lets it operate really smoothly using this API https://github.com/slaymaker1907/TW5-browser-nativesaver https://github.com/slaymaker1907/TW5-browser-nativesaver that demonstrates why this API is worth the trouble.
- ildon 5y ago
  If you need to use IndexedDB and you do not want to have to handle the absurdities of IndexedDB, give Dexie a try (https://dexie.org/ https://dexie.org/). It really makes using IndexedDB a breeze.
ildon 5y ago
Glad you liked it! And thanks for mentioning absurd-sql, I did not know the project. You are absolutely right on memory and crashes. It is a very inefficient way of storing data in a browser from a memory perspective (and data loss risks). I did mention it in my conclusions :-) If FileSystemAccessAPIs added block level support and locking for at least certain files that would be a game changer for things like this. In the meanwhile IndexedDB is surely a better option.
- typingmonkey 5y ago
  I analyzed [1] that problem and tried to reproduce data loss with LokiJS, which also has everything in memory and only partially writes to disc. It is very hard to reproduce a loss. You can call the saveDatabase() when the window 'unload' event fires and also after each write when the database is idle again. This works quite reliable. The only way to lose data is when I shutdown power directly after a write. It is also to mention that having everything in memory will not support multi-tab usage. [1] https://rxdb.info/slow-indexeddb.html https://rxdb.info/slow-indexeddb.html

wngr 5y ago

Combining these two approaches would let you get rid of the IndexedDB abomination.

samwillis 5y ago
I believe FileSystemAccessAPI, as implemented in browsers, does not yet support block level access and locking. Both required to make this work, the developer behind Absurd SQL is working with the team designing the API to ensure it will have support. IndexedDB is the only way of doing this on browsers currently. https://github.com/WICG/file-system-access/issues/323 https://github.com/WICG/file-system-access/issues/323
- dmitriid 5y ago
  Unless I'm mistaken, FileSystemAccessAPI is only implemented in Chrome. Both Safari and Firefox are not implementing until four different file proposals (all from Chrome) can be reduced to one.
  - tehbeard 5y ago
    I thought Firefox flat out didn't want to implement FileSystem APIs on grounds of "security" and protecting their users?
    - ildon 5y ago
      Yes, many are refusing to implement it due to security concerns. A notable one is Brave, which is based on Chromium, and yet refuses to enable it (https://github.com/brave/brave-browser/issues/11407 https://github.com/brave/brave-browser/issues/11407). I'm skeptical that reducing access to the FS actually protects users. Those that would be fooled by scams based on FileSystemAccess APIs would very likely be fooled also with other less intricate tactics. So I doubt that the overall security of users is in practice affected. At least, browsers that refuse to implement the FileSystemAccess APIs could implement them, but leave them disabled by default, and require some non-trivial action to enable them. So users with a very basic understanding of how things work in the browser would not be able to enable them.
      - dmitriid 5y ago
        > So I doubt that the overall security of users is in practice affected. The problem is the sheer number of APIs that Chrome ships and wants other browsers to ship, and what browsers already ship that require access via prompts: camera, location, notifications, file access, bluetooth, usb, motion sensors, serial ports, midi devices, clipboard... Just prompting user to allow stuff is no longer enough, and adding more prompts leads to worse security.
        slaymaker1907 5y ago
        It's sort of a double prompt scenario since the FileSystemAccessAPI requires users to actually select the file/directory for the web app to use. You can save the file handles into IndexedDB in which case the user will only be prompted if they refresh the page to authorize the web app to continue using that file/directory. From my use of it, the only real vulnerability I see is that Chrome still considers anything from file:// to be the same origin. This means as a developer you absolutely should not be saving file handles to IndexedDB if you are loading your app via file:// instead of https:// https://. This is a pretty niche use case, but I do think static html "apps" are an underappreciated form for distributing software and this new API makes such apps a lot more powerful. I wrote a plugin for TiddlyWiki that lets it operate really smoothly using this API https://github.com/slaymaker1907/TW5-browser-nativesaver https://github.com/slaymaker1907/TW5-browser-nativesaver that demonstrates why this API is worth the trouble.
ildon 5y ago
If you need to use IndexedDB and you do not want to have to handle the absurdities of IndexedDB, give Dexie a try (https://dexie.org/ https://dexie.org/). It really makes using IndexedDB a breeze.

ildon 5y ago

Glad you liked it! And thanks for mentioning absurd-sql, I did not know the project. You are absolutely right on memory and crashes. It is a very inefficient way of storing data in a browser from a memory perspective (and data loss risks). I did mention it in my conclusions :-) If FileSystemAccessAPIs added block level support and locking for at least certain files that would be a game changer for things like this. In the meanwhile IndexedDB is surely a better option.

typingmonkey 5y ago
I analyzed [1] that problem and tried to reproduce data loss with LokiJS, which also has everything in memory and only partially writes to disc. It is very hard to reproduce a loss. You can call the saveDatabase() when the window 'unload' event fires and also after each write when the database is idle again. This works quite reliable. The only way to lose data is when I shutdown power directly after a write. It is also to mention that having everything in memory will not support multi-tab usage. [1] https://rxdb.info/slow-indexeddb.html https://rxdb.info/slow-indexeddb.html