11 ms·

> … support persistent client-side storage using available JS APIs. As of this writing, that includes the Origin-Private FileSystem (OPFS) [0] This is really g

> … support persistent client-side storage using available JS APIs. As of this writing, that includes the Origin-Private FileSystem (OPFS) [0] This is really good news, and exactly what the OPFS was designed for. You may have seen “Absurd SQL” [1] which was a proof of concept for building a SQLite Virtual FS backend using IndexedDB. It provided full ACID compliment transactions. Incredible work but a hack at best. The OPFS supersedes all that and makes it possible to have proper consistent and resilient transactions. WASM SQLite with the OPFS is the future of offline first web app development. The concept of a single codebase web/mobile/desktop app with proper offline storage is here. What I really want to see next is an eventually constant sync system between browser and server (or truly distributed with WebRTC). The SQLite Session Extension [2] potentially has the building blocks needed for such a system. 0: https://webkit.org/blog/12257/the-file-system-access-api-with-origin-private-file-system/ https://webkit.org/blog/12257/the-file-system-access-api-wit... 1: https://github.com/jlongster/absurd-sql https://github.com/jlongster/absurd-sql 2: https://www.sqlite.org/sessionintro.html https://www.sqlite.org/sessionintro.html

Existenceblinks 4y ago
> persistent Question: is it really persisted indefinitely? And will this SQLite file be portable (import and export to somewhere else) ?
- srcreigh 4y ago
  It’s not persistent everywhere. On iOS the data is deleted after 7 days of inactivity on a website. It’s treated like a cache. https://webkit.org/blog/12257/the-file-system-access-api-with-origin-private-file-system/ https://webkit.org/blog/12257/the-file-system-access-api-wit... https://web.dev/storage-for-the-web/#:~:text=Starting%20in%20iOS%20and%20iPadOS,not%20interact%20with%20the%20site https://web.dev/storage-for-the-web/#:~:text=Starting%20in%2....
  - tehbeard 4y ago
    Missed one part from web.dev > ...This eviction policy does not apply to installed PWAs that have been added to the home screen.
- sgbeal 4y ago
  > And will this SQLite file be portable (import and export to somewhere else) ? sqlite's storage format is independent of the underlying device. OPFS is "just another backend" and the only tricky part of implementing it was that OPFS's API is largely asynchronous and sqlite3 requires synchronous I/O. The db that's stored in OPFS is exactly what would be stored on your hard drive if you were operating outside of the browser.
harryvederci 4y ago
I can already imagine the youtube video titles: "Mozilla and Microsoft TRIED to block it, but COULDN'T stop it!" Seriously though, I can't wait to use this for my website. This is going to be ridiculously awesome combined with htmx.
- dyml 4y ago
  What features will this unlock for you?
  - harryvederci 4y ago
    Serving my htmx app as a WASM app, so interaction by users will be done on their computer. Then only the DB changes will be have to be synced with the backend, saving a lot of traffic / server capacity. This was already possible, but then the user's changes would be lost if they don't sync in time. If I understand things correctly this is the missing piece: persistence. So if the user goes offline and makes changes, those will persist so they can be synced when the user goes online again. (All of this is as far as I understand things, I could be completely wrong.)
    - k__ 4y ago
      Isn't that model (without the wasm part) alread possible with pouchdb/couchdb?
      - harryvederci 4y ago
        I haven't looked into those, but I'm happily using SQLite already, and this would mean I won't have to swap it for something else to achieve the same thing.
    - ris58h 4y ago
      > If I understand things correctly this is the missing piece: persistence. So if the user goes offline and makes changes, those will persist so they can be synced when the user goes online again. 1. You can persist data with localStorage/IndexDB, can't you? 2. There is nothing in the article about syncing. There is no out-of-the-box solution, as I can see.
thruflo 4y ago
> What I really want to see next is an eventually consistent sync system between browser and server We’ve made one: https://electric-sql.com https://electric-sql.com Active-active SQLite to Postgres with transactional causal+ consistency based on CRDTs. Disclaimer: founder.
- samwillis 4y ago
  That is awesome! I’m going to have to play with that. I previously experimented with combining Yjs (CRDT toolkit) with Pouch/CouchDB to create an eventually consistent db, but decided that CouchDB was the wrong backend. It looks like you have built exactly what I wanted to do if I had had time.
  - thruflo 4y ago
    Very cool! Would love to swap ideas if you're up for a chat :)
- nrser 4y ago
  I clicked through to the technical intro, and it says you’re using AntidoteDB for the top-level replication https://electric-sql.com/docs/overview/technical-intro https://electric-sql.com/docs/overview/technical-intro I happen across AntidoteDB just the other day, and my take-away was that it was still under development but not really ready for production use. Curious your opinion / experience with it?
  - thruflo 4y ago
    Yup, it’s a very rigorous system developed over ~10 years with solid testing, benchmarking and formal proofs. However it has some known issues, such as behaviour under high load and not currently handling node failure within a DC. It’s not production ready and neither are we (we’re in developer preview mode, which is like a public alpha [0]). There are also other aspects on the Antidote roadmap such as efficiently materialising consistent secondary indexes that are ongoing challenges but aren’t so relevant to how we’re using it as a replication layer. We are working, alongside others, with the Antidote developers to help fix these issues and generally improve reliability / engineer correct behaviour under load. (Professor Annette Bieniusa, who leads the development, is our Chief Architect). We have a fork at electric-sql/vaxine [1]. We are also taking advantage of some simplifications which mitigate the known issues. We have an #antidote channel in the ElectricSQL discord [2] if you’d be interested in chatting more. [0]: https://electric-sql.com/docs/overview/faqs https://electric-sql.com/docs/overview/faqs [1]: https://github.com/electric-sql/vaxine https://github.com/electric-sql/vaxine [2]: https://discord.gg/B7kHGwDcbj https://discord.gg/B7kHGwDcbj
chasd00 4y ago
> WASM SQLite with the OPFS is the future of offline first web app development. is that basically a desktop app that doesn't need to be installed? You still download it using a browser but then you don't install it on your operating system you just run it in your browser?
- misschresser 4y ago
  yea exactly. seems pretty sweet
tantaman 4y ago
> What I really want to see next is an eventually consistent sync system between browser and server Another CRDT + SQLite project as a native extension: https://github.com/aphrodite-sh/cr-sqlite https://github.com/aphrodite-sh/cr-sqlite Should have a release out here shortly but feel free to browse the code, tests and readme till then to see how it works.
ilyt 4y ago
I find it kinda sad that browsers get to remove Web SQL on shaky grounds (which was just sqlite available to JS) yet here we are, back to that same spot, but with less performance and more complexity
- saurik 4y ago
  This is somewhat more complex for the developers of websites but dramatically less complex for the developers of web browsers, and that is where we really should be caring: the complexity in the browser leads to both increased attack surface and nigh unto guarantees there will only ever be a single good implementation of the browser. Both standards and security critical systems should be easy to build and easy to reason about.
- sgbeal 4y ago
  > I find it kinda sad that browsers get to remove Web SQL on shaky grounds (which was just sqlite available to JS) i recently had to work with WebSQL in the context of comparing it to sqlite's new WASM support (of which i'm the developer). WebSQL, quite frankly, is a toy. Its execution model is far too limited and excludes all sorts of functionality, not the least of which is that it's impossible to delete a WebSQL db. > yet here we are, back to that same spot, but with less performance and more complexity Wrong. We have benchmarked the two in apples-to-apples comparisons, taking into account WebSQL's limitations. The two approaches are roughly equivalent, with both winning out under certain loads, despite WebSQL being implemented in native code.
  - ilyt 4y ago
    Who is "we" and where I can see those benchmarks ? Also even at similar performance you still need to download a bunch of extra stuff to even run the WASM version. My whole webpage takes less than it...
    - sgbeal 4y ago
      > Who is "we" We is the sqlite team. i'm the "JS/WASM Guy" for the project. > and where I can see those benchmarks ? You can't currently because we don't have them in a user-consumable form. We've done a tremendous amount of benchmarking during the development because All The Speed was one of our design goals. However, all such records were in transient spreadsheets intended for one-shot note-taking use, not publication. Once our documentation effort settles down, and responding to user feedback from the initial announcement slows down, i hope to implement a benchmarking application similar to: <https://rhashimoto.github.io/wa-sqlite/demo/benchmarks.html https://rhashimoto.github.io/wa-sqlite/demo/benchmarks.html> Until then, however, you'll simply have to (A) take my word for it, (B) try it out yourself, or (C) none of the above, as you wish. Edit: or (D): we have a WASM port of sqlite's standard benchmarking tool, known as "speedtest1", in the sqlite source tree, but getting it up and running requires reading a good deal of documentation: <https://sqlite.org/src/dir/ext/wasm&ci=trunk https://sqlite.org/src/dir/ext/wasm&ci=trunk> That tool is how we've benchmarked it so far, with the exception of comparing it to WebSQL, which required a custom application which is also in that directory (batch-runner.*). batch-runner, however, is in no way user friendly.
Asraelite 4y ago
I'm a bit confused as to why it supports OPFS, but not the much more widely supported regular FileSystem API. As far as I understand, this is also capable of reading and writing to files directly, and also allows seeking. All the user needs to do is select a file in a file picker first. Is there a required feature that only OPFS provides, other than file locking (which seems like a pretty non-essential feature tbh)?

Existenceblinks 4y ago

> persistent Question: is it really persisted indefinitely? And will this SQLite file be portable (import and export to somewhere else) ?

srcreigh 4y ago
It’s not persistent everywhere. On iOS the data is deleted after 7 days of inactivity on a website. It’s treated like a cache. https://webkit.org/blog/12257/the-file-system-access-api-with-origin-private-file-system/ https://webkit.org/blog/12257/the-file-system-access-api-wit... https://web.dev/storage-for-the-web/#:~:text=Starting%20in%20iOS%20and%20iPadOS,not%20interact%20with%20the%20site https://web.dev/storage-for-the-web/#:~:text=Starting%20in%2....
- tehbeard 4y ago
  Missed one part from web.dev > ...This eviction policy does not apply to installed PWAs that have been added to the home screen.
sgbeal 4y ago
> And will this SQLite file be portable (import and export to somewhere else) ? sqlite's storage format is independent of the underlying device. OPFS is "just another backend" and the only tricky part of implementing it was that OPFS's API is largely asynchronous and sqlite3 requires synchronous I/O. The db that's stored in OPFS is exactly what would be stored on your hard drive if you were operating outside of the browser.

harryvederci 4y ago

I can already imagine the youtube video titles: "Mozilla and Microsoft TRIED to block it, but COULDN'T stop it!" Seriously though, I can't wait to use this for my website. This is going to be ridiculously awesome combined with htmx.

dyml 4y ago
What features will this unlock for you?
- harryvederci 4y ago
  Serving my htmx app as a WASM app, so interaction by users will be done on their computer. Then only the DB changes will be have to be synced with the backend, saving a lot of traffic / server capacity. This was already possible, but then the user's changes would be lost if they don't sync in time. If I understand things correctly this is the missing piece: persistence. So if the user goes offline and makes changes, those will persist so they can be synced when the user goes online again. (All of this is as far as I understand things, I could be completely wrong.)
  - k__ 4y ago
    Isn't that model (without the wasm part) alread possible with pouchdb/couchdb?
    - harryvederci 4y ago
      I haven't looked into those, but I'm happily using SQLite already, and this would mean I won't have to swap it for something else to achieve the same thing.
  - ris58h 4y ago
    > If I understand things correctly this is the missing piece: persistence. So if the user goes offline and makes changes, those will persist so they can be synced when the user goes online again. 1. You can persist data with localStorage/IndexDB, can't you? 2. There is nothing in the article about syncing. There is no out-of-the-box solution, as I can see.

thruflo 4y ago

> What I really want to see next is an eventually consistent sync system between browser and server We’ve made one: https://electric-sql.com https://electric-sql.com Active-active SQLite to Postgres with transactional causal+ consistency based on CRDTs. Disclaimer: founder.

samwillis 4y ago
That is awesome! I’m going to have to play with that. I previously experimented with combining Yjs (CRDT toolkit) with Pouch/CouchDB to create an eventually consistent db, but decided that CouchDB was the wrong backend. It looks like you have built exactly what I wanted to do if I had had time.
- thruflo 4y ago
  Very cool! Would love to swap ideas if you're up for a chat :)
nrser 4y ago
I clicked through to the technical intro, and it says you’re using AntidoteDB for the top-level replication https://electric-sql.com/docs/overview/technical-intro https://electric-sql.com/docs/overview/technical-intro I happen across AntidoteDB just the other day, and my take-away was that it was still under development but not really ready for production use. Curious your opinion / experience with it?
- thruflo 4y ago
  Yup, it’s a very rigorous system developed over ~10 years with solid testing, benchmarking and formal proofs. However it has some known issues, such as behaviour under high load and not currently handling node failure within a DC. It’s not production ready and neither are we (we’re in developer preview mode, which is like a public alpha [0]). There are also other aspects on the Antidote roadmap such as efficiently materialising consistent secondary indexes that are ongoing challenges but aren’t so relevant to how we’re using it as a replication layer. We are working, alongside others, with the Antidote developers to help fix these issues and generally improve reliability / engineer correct behaviour under load. (Professor Annette Bieniusa, who leads the development, is our Chief Architect). We have a fork at electric-sql/vaxine [1]. We are also taking advantage of some simplifications which mitigate the known issues. We have an #antidote channel in the ElectricSQL discord [2] if you’d be interested in chatting more. [0]: https://electric-sql.com/docs/overview/faqs https://electric-sql.com/docs/overview/faqs [1]: https://github.com/electric-sql/vaxine https://github.com/electric-sql/vaxine [2]: https://discord.gg/B7kHGwDcbj https://discord.gg/B7kHGwDcbj

chasd00 4y ago

> WASM SQLite with the OPFS is the future of offline first web app development. is that basically a desktop app that doesn't need to be installed? You still download it using a browser but then you don't install it on your operating system you just run it in your browser?

misschresser 4y ago
yea exactly. seems pretty sweet

tantaman 4y ago

> What I really want to see next is an eventually consistent sync system between browser and server Another CRDT + SQLite project as a native extension: https://github.com/aphrodite-sh/cr-sqlite https://github.com/aphrodite-sh/cr-sqlite Should have a release out here shortly but feel free to browse the code, tests and readme till then to see how it works.

ilyt 4y ago

I find it kinda sad that browsers get to remove Web SQL on shaky grounds (which was just sqlite available to JS) yet here we are, back to that same spot, but with less performance and more complexity

saurik 4y ago
This is somewhat more complex for the developers of websites but dramatically less complex for the developers of web browsers, and that is where we really should be caring: the complexity in the browser leads to both increased attack surface and nigh unto guarantees there will only ever be a single good implementation of the browser. Both standards and security critical systems should be easy to build and easy to reason about.
sgbeal 4y ago
> I find it kinda sad that browsers get to remove Web SQL on shaky grounds (which was just sqlite available to JS) i recently had to work with WebSQL in the context of comparing it to sqlite's new WASM support (of which i'm the developer). WebSQL, quite frankly, is a toy. Its execution model is far too limited and excludes all sorts of functionality, not the least of which is that it's impossible to delete a WebSQL db. > yet here we are, back to that same spot, but with less performance and more complexity Wrong. We have benchmarked the two in apples-to-apples comparisons, taking into account WebSQL's limitations. The two approaches are roughly equivalent, with both winning out under certain loads, despite WebSQL being implemented in native code.
- ilyt 4y ago
  Who is "we" and where I can see those benchmarks ? Also even at similar performance you still need to download a bunch of extra stuff to even run the WASM version. My whole webpage takes less than it...
  - sgbeal 4y ago
    > Who is "we" We is the sqlite team. i'm the "JS/WASM Guy" for the project. > and where I can see those benchmarks ? You can't currently because we don't have them in a user-consumable form. We've done a tremendous amount of benchmarking during the development because All The Speed was one of our design goals. However, all such records were in transient spreadsheets intended for one-shot note-taking use, not publication. Once our documentation effort settles down, and responding to user feedback from the initial announcement slows down, i hope to implement a benchmarking application similar to: <https://rhashimoto.github.io/wa-sqlite/demo/benchmarks.html https://rhashimoto.github.io/wa-sqlite/demo/benchmarks.html> Until then, however, you'll simply have to (A) take my word for it, (B) try it out yourself, or (C) none of the above, as you wish. Edit: or (D): we have a WASM port of sqlite's standard benchmarking tool, known as "speedtest1", in the sqlite source tree, but getting it up and running requires reading a good deal of documentation: <https://sqlite.org/src/dir/ext/wasm&ci=trunk https://sqlite.org/src/dir/ext/wasm&ci=trunk> That tool is how we've benchmarked it so far, with the exception of comparing it to WebSQL, which required a custom application which is also in that directory (batch-runner.*). batch-runner, however, is in no way user friendly.

Asraelite 4y ago

I'm a bit confused as to why it supports OPFS, but not the much more widely supported regular FileSystem API. As far as I understand, this is also capable of reading and writing to files directly, and also allows seeking. All the user needs to do is select a file in a file picker first. Is there a required feature that only OPFS provides, other than file locking (which seems like a pretty non-essential feature tbh)?