16 ms·

SQLite in the browser with WASM/JS

547 points by hochmartinez 4y ago

thesuperbigfrog 4y ago
"Folks have been building sqlite3 for WASM since at least as far back as 2019, but this subproject is the first effort "officially" associated with the SQLite project, created with the goal of making WASM builds of the library first-class members of the family of supported SQLite deliverables." Awesome news. SQLite is already a defacto cross-platform file format. An official WASM release will be widely welcomed and extremely useful.
- lovasoa 4y ago
  SQLite was one of the first large C code bases to be ported to the web. Alon Zakai (@kripken, the creator of emscripten), made the first commit to sql.js at the start of 2012 [0]. I got involved two years later, and have been maintaining sql.js since. A lot was added since 2012, but the initial core api with 3 functions 'open', 'exec', and 'close' still works today. https://github.com/kripken/sql.js/commit/cebd80648dbd369b34804c5a00b4d0bddc1cbf05 https://github.com/kripken/sql.js/commit/cebd80648dbd369b348...
  - sgbeal 4y ago
    > SQLite was one of the first large C code bases to be ported to the web. Thank you very much for that info. i'll update the sqlite wasm docs with that. Edit: updated, thank you! <https://sqlite.org/wasm/info/cacc33abc5d9013f https://sqlite.org/wasm/info/cacc33abc5d9013f>
    - lovasoa 4y ago
      > Folks have been building sqlite3 for WASM since as far back as 2012 for *the web*, not for WASM. SQLite was initially compiled to simple javascript, then to asm.js when it appeared, then to WASM when it replaced asm.js :) The sql.js project is older than the idea of WASM itself. And I like to think it contributed to showing the potential of compiling native code to the browser, and in the creation of the WASM standard.
      - sgbeal 4y ago
        > for the web, not for WASM. Docs have been updated. Thank you for the feedback! Edit: for future feedback (from anyone reading this) i can be reached via stephan at sqlite org. i'm unlikely to catch most doc feedback posted to HN.
hochmartinez 4y ago
Demos you can inspect... https://sqlite.org/wasm/doc/tip/demo-123.html https://sqlite.org/wasm/doc/tip/demo-123.html https://sqlite.org/wasm/doc/tip/demo-123-worker.html https://sqlite.org/wasm/doc/tip/demo-123-worker.html
hn2017 4y ago
Excited about this news, this the kind of change the modern web needs.
- boppo1 4y ago
  Can you help me understand how/why this is a big change the web needs? I'm not seeing the value proposition on a database-in-a-webpage
  - Scarbutt 4y ago
    For the same reason many complex desktop applications use sqlite or some other database.
  - exelib 4y ago
    My use case is an offline-capable app. There was already WebSQL, but it got deprectaed.
    - chasd00 4y ago
      > My use case is an offline-capable app. not trying to be snarky but do you mean a desktop app?
      - EvanAnderson 4y ago
        Offline-capable apps on phones are "a thing". There are plenty of times when I have poor (or nonexistent) cellular coverage and would love web-based apps to work.
      - hot_gril 4y ago
        A desktop app that's cross-platform.
      - exelib 4y ago
        Currently, it is a browser app intended to run on mobile devices. Not exactly PWA, but very similar. Later it will be, probably, packaged with capacitor.
adamrezich 4y ago
awesome! it's also great to see a project as important as sqlite provide first-class wasm support like this.
coder543 4y ago
As a little pro-tip, the demos appear to be serving a gzip compressed WASM file, but offering a brotli compressed version for clients that say they accept it would be even better: 665K sqlite3.wasm 306K sqlite3.wasm.gz (-54% compared to uncompressed) 266K sqlite3.wasm.br (-13% compared to gz, -60% compared to uncompressed)
- dstaley 4y ago
  Oooff, 266 KB is a hefty price to pay. I really wish WebSQL would have won out over IndexedDB, because I would have loved to have been able to build libraries on top of that instead of the headache IndexedDB is.
  - coder543 4y ago
    "Hefty"... for an asset that can easily be cached long term, so it is effectively a one time penalty for a given website, and many web apps will load images larger than this. Certainly, just about any marketing/product page will load more bytes than this in the form of images, let alone videos. So, "hefty" seems like a bit of an exaggeration to me. All other things equal, lighter is always better, but 266KB for all the functionality SQLite offers isn't that bad, IMO. On the topic of caching, since SQLite is (intentionally or not) going to be setting an example with their docs and demos[0], I would suggest that SQLite should demonstrate the industry best practices with regards to caching as well. Static assets like WASM and JavaScript should be served with a very high cache duration, and the name of the asset should include a hash of the asset. This way, the site maintainer can update the HTML to reference the new SQLite bundles by the new hash whenever they upload new versions, and browsers will immediately request the new versions, but they will otherwise instantly load the cached version after the first visit whenever there isn't a new version. [0]: https://sqlite.org/wasm/doc/trunk/demo-123.md https://sqlite.org/wasm/doc/trunk/demo-123.md
    - fooey 4y ago
      cross-origin caching on the web has been dead for years doesn't seem possible to do without leaking data
ramesh31 4y ago
Feels like WASM is hitting an inflection point this year. After nearly a decade of hearing "someday", that day is finally here.
- MuffinFlavored 4y ago
  I wonder what the catalyst finally is/was to cause such an acceleration
  - ilyt 4y ago
    Compound interest. Tools that make making other tools easier. Once programming languages started having WASM compile target a great many things became easier
  - sgbeal 4y ago
    > I wonder what the catalyst finally is/was to cause such an acceleration IMO, all that was missing was a truly compelling use case. The combination of the ubiquitous sqlite with non-trivially-sized persistent storage gives us that use case.
- sgbeal 4y ago
  > Feels like WASM is hitting an inflection point this year. After nearly a decade of hearing "someday", that day is finally here. At the start of this development effort (April, IIRC (2022)) we (in the sqlite project) had only ever heard of wasm but hadn't paid any attention to it. Within just a few days of starting this project, we were fully convinced that the combination of sqlite and OPFS will be one of the Next Big Things for web app development. As the project's "JS/WASM Guy" i'm exceedingly excited to see what people do with this and what improvements we'll make based on user feedback. (It's long been my experience that the most interesting feature suggestions come from users.)
exelib 4y ago
Amazing! 313kb gzipped. I hope the author(s) of IndexedDB would finally understand how stupid the idea of IDB was. Are there any numbers on performance?
- sgbeal 4y ago
  > Are there any numbers on performance? Tucked away in spreadsheets there are but we don't currently have any benchmarks in a publishable form. In broad strokes, i can assure you (as the one who performed the benchmarks) that the new sqlite wasm is competitive with WebSQL in terms of performance, with either one winning out in certain benchmarks. Given that WebSQL is implemented in native code and sqlite in wasmified C, we're quite happy with those results. Note, however, that benchmarks are very browser-dependent. Firefox's wasm engine, for example, is significantly slower than Chrome's, but it's also more consistent. If you run a given test 10 times in FF, the difference in runtimes across them will be small (maybe 10%), whereas there will be a +/-50% difference in runtimes for the same tests in Chrome. In Chrome, if the dev tools are open when wasm is running, wasm's performance can (for unknown reasons) drop by as much as half or more even if the wasm code produces no output.
  - exelib 4y ago
    Thank you very much, that sounds great!
- tehbeard 4y ago
  > I hope the author(s) of IndexedDB would finally understand how stupid the idea of IDB was While not one of said authors... You mind giving me an actual rundown on issues you've got with it instead of just slinging mud? I've used IDB in a few projects, and apart from: - The god awful callback crap born of being pre-promises. - The lack of partial indexes (e.g. indexing a subset of documents based on some parameters) - The iOS Webkit teams continued habit adding weird, app-breaking bugs every other release to an api/system that should be stable? It's been fairly useful. Hell, it supports storing JS types like CryptoKey or Blob/File with no real issues[1] [1] Sans iOS WebKit, which a version or 2 back would just eat Blobs at random.
nilslice 4y ago
very cool to look at the unpacked module, in all its wasm glory https://whatsinyourwasm.com/module/6 https://whatsinyourwasm.com/module/6 (warning: not mobile friendly...)
- sgbeal 4y ago
  > very cool to look at the unpacked module, in all its wasm glory Very nice :). Be aware that the copy of sqlite3.wasm on the sqlite.org/wasm site gets updated fairly frequently, so may differ from that at any moment. That particular site is only for documentation purposes, not for hosting the canonical wasm file release (which is pending along with the release of sqlite 3.40), and its sqlite3.wasm/js copies get updated hand in hand with development of the canonical copies.
  - nilslice 4y ago
    Good to know! FWIW, this is a very beta demo version of a product I'm working on (Modsurfer). It's made to capture snapshot/moment-in-time details about a module (maybe you're running untrusted code in your system and want to track what you're actually running). However, in the future, it will likely be able to automatically import and track modules from cloud storage or URLs.
ibash 4y ago
If you want to use sqlite in the browser today (chrome only) take a look at websql.
- samwillis 4y ago
  WebSQL is dead, long live WASM SQLite. And in all seriousness, WASM is exactly the way this should be done. It doesn’t tie a db engine to a specific standardised version. It’s far more important to develop low level block storage apis like the OPFS that WASM SQLite is using.
  - ilyt 4y ago
    I'd still prefer some version of SQLite to be bundled, just to save on transfer for apps that are tiny
- resoluteteeth 4y ago
  chrome is presumably eventually going to remove websql. I guess they are trying to do it slowly so they don't break stuff, but they're already doing things like removing FTS support.
  - sgbeal 4y ago
    > I guess they are trying to do it slowly so they don't break stuff, but they're already doing things like removing FTS support. FWIW, i can say with some authority that they have been waiting on this new sqlite/wasm stuff so that they can offer a replacement to WebSQL for their folks who still use it. Despite Google's long history of pulling the plug on products (G+, how you are missed!), they're not willing to outright drop WebSQL without a viable replacement. It would not be difficult to write a drop-in workalike WebSQL wrapper on top of the new wasm/js APIs, and we (in the sqlite project) may even get around to doing so as time allows for. The difficulty would be making it "quirk for quirk compatible," as such quirks are not documented anywhere. OTOH, as far as we're aware, the only extant WebSQL implementation is the one in Chrome, so that's the only one which counts for purposes of quirks.
samwillis 4y ago
> … support persistent client-side storage using available JS APIs. As of this writing, that includes the Origin-Private FileSystem (OPFS) [0] This is really good news, and exactly what the OPFS was designed for. You may have seen “Absurd SQL” [1] which was a proof of concept for building a SQLite Virtual FS backend using IndexedDB. It provided full ACID compliment transactions. Incredible work but a hack at best. The OPFS supersedes all that and makes it possible to have proper consistent and resilient transactions. WASM SQLite with the OPFS is the future of offline first web app development. The concept of a single codebase web/mobile/desktop app with proper offline storage is here. What I really want to see next is an eventually constant sync system between browser and server (or truly distributed with WebRTC). The SQLite Session Extension [2] potentially has the building blocks needed for such a system. 0: https://webkit.org/blog/12257/the-file-system-access-api-with-origin-private-file-system/ https://webkit.org/blog/12257/the-file-system-access-api-wit... 1: https://github.com/jlongster/absurd-sql https://github.com/jlongster/absurd-sql 2: https://www.sqlite.org/sessionintro.html https://www.sqlite.org/sessionintro.html
- Existenceblinks 4y ago
  > persistent Question: is it really persisted indefinitely? And will this SQLite file be portable (import and export to somewhere else) ?
  - srcreigh 4y ago
    It’s not persistent everywhere. On iOS the data is deleted after 7 days of inactivity on a website. It’s treated like a cache. https://webkit.org/blog/12257/the-file-system-access-api-with-origin-private-file-system/ https://webkit.org/blog/12257/the-file-system-access-api-wit... https://web.dev/storage-for-the-web/#:~:text=Starting%20in%20iOS%20and%20iPadOS,not%20interact%20with%20the%20site https://web.dev/storage-for-the-web/#:~:text=Starting%20in%2....
    - tehbeard 4y ago
      Missed one part from web.dev > ...This eviction policy does not apply to installed PWAs that have been added to the home screen.
hinkley 4y ago
I entirely missed the era where SQLite was a candidate for W3C inclusion. It was always a mental bookmark in case I found a need to have an app with any substantial offline mode, but by the time I found a perfect candidate (cataloging plants in situ), it was already deprecated, and a key-value store is no replacement for SQLite.
- exelib 4y ago
  Key-value store would be OK, we (developers) can handle that. And in fact, we do it already. But IndexedDB is broken by design. But I agree, handle relational data is what we want. And you can easily emulate non-relation data with it, if you need.
  - sgbeal 4y ago
    > Key-value store would be OK, we (developers) can handle that. And in fact, we do it already. ... But I agree, handle relational data is what we want. You can now have both: a relational database in your key-value localStorage: <https://sqlite.org/wasm/doc/trunk/persistence.md#kvvfs https://sqlite.org/wasm/doc/trunk/persistence.md#kvvfs> :-D
    - exelib 4y ago
      localStorage suffers from issues like size limits and performance (which, probably, leads to higher battery consumption).
      - sgbeal 4y ago
        > localStorage suffers from issues like size limits and performance Absolutely, but _it works_ and nothing trumps working code ;). A slow and space-limited database is better than none at all! We implemented that functionality primarily to make peoples' eyes bug out ;), but also so that folks who don't yet have OPFS can have some form of persistent sqlite databases.
      - hinkley 4y ago
rcarmo 4y ago
Very nice. I built sqlite3 for WASI/a-Shell to use on my iPad (https://github.com/rcarmo/wasi-sqlite https://github.com/rcarmo/wasi-sqlite) and it still has a few issues, I hope this will help (although right now the biggest issue seems to be that the REPL has some sort of memory leak when run inside WASI).
lovasoa 4y ago
Hi! I'm the main maintainer of sql.js. Is there somewhere I can get in touch with you? Would you be interested in publishing this as a new major version of sql.js itself ?
- remram 4y ago
  Their forum: https://sqlite.org/forum/forum https://sqlite.org/forum/forum
- erikpukinskis 4y ago
  Thank you for sql.js by the way. It is revolutionary!
harryvederci 4y ago
I couldn't find a link to the source code on that page, I think this is it: https://sqlite.org/wasm/tree https://sqlite.org/wasm/tree As it's a fossil repo, here's the steps to clone it and check it out locally: sudo apt install fossil # Install fossil (on Debian) fossil clone https://sqlite.org/wasm # Clone the SQLite WASM repo cd wasm # Navigate to the cloned repo fossil ui # Launch fossil in UI-mode
thruflo 4y ago
The Google Chrome team is working with the SQLite team to standardise a WASM based SQLite in the browser to replace WebSQL. https://developer.chrome.com/blog/deprecating-web-sql/ https://developer.chrome.com/blog/deprecating-web-sql/ https://twitter.com/chromiumdev/status/1565105522092695553 https://twitter.com/chromiumdev/status/1565105522092695553 Which would be pretty cool. Projects like SQL.js and absurd-sql are awesome but naturally a bit rough around the edges and not highly maintained.
- modeless 4y ago
  This is good but it can never be as good as Web SQL could have been, because it can't be a truly shared library. Like all WASM modules, It has to be downloaded and JIT compiled separately for every site that uses it. Not for technical reasons, but privacy reasons, so it's basically unfixable: https://developer.chrome.com/en/blog/http-cache-partitioning/ https://developer.chrome.com/en/blog/http-cache-partitioning... Maybe if half the sites on the web start using it, browsers can finally be convinced that it would be OK to add SQLite to the base platform.
  - thruflo 4y ago
    Interesting — so basically an overhead until it’s fully native? I was reading the Chrome initiative as a pathway to native. Would that mean it can’t be WASM?
    - modeless 4y ago
      That tweet is not implying that it will ever be native. Quite the opposite; Web SQL is native today and they are going to remove it since it was rejected by other browsers.
  - samwillis 4y ago
    I disagree, this is much better than WebSQL. WebSQL would have been tied to one specific version of SQLite, and developers would have to work to the lowest supported version. There would have been no extension mechanism. WASM SQLite is the correct solution. It’s extendable by the developer using it, they can uses SQLite extension modules, and build their own. But almost more so it proves the idea of a WASM db engine backed by a low level block FS api. We will see other db engines uses this architecture. DuckDB have already done it. I’m sure MongoDBs Realm and CouchBase Mobile will do the same soon too. We are in for an exciting time in the next few years.
alberth 4y ago
Will this help Cloudflare, Fly.io and others who started offering SQLite-like of offerings in the cloud?
TheRealPomax 4y ago
The thing I don't get here is that literally every browser already uses Sqlite3 for a million things, why are we not just finally acknowledging that, yes, this very specific library is a universal good, just like gzip compression, or the jpeg image format, and here's the JS API for directly working with it? Why do we need a separate WASM version of something that's already built right into every single browser? Why is there no all-browser-vendor-blessed `Sqlite3` global?
- easton 4y ago
  They tried that at one point, I think it was called WebSQL? But it was decided that they didn’t want to make the quirks of sqlite3 a web standard since it’d be hard to do a clean room implementation.
  - TheRealPomax 4y ago
    Except I don't want WebSQL, I literally want Sqlite3 as a JS API, backed by the unadulterated authoritative Sqlite3 library. I don't wonder why there's no good in-browser database solution that uses SQL, I literally don't get why sqlite3, only, and specifically, still isn't usable by JS in 2022.
    - deleted 4y ago
      [deleted]
    - sgbeal 4y ago
      > I literally don't get why sqlite3, only, and specifically, still isn't usable by JS in 2022. It is now :). We provide 4 separate APIs, from the lowest-level 1-to-1 C-via-WASM bindings to one quite similar to sql.js, plus all of the low-level pieces necessary to create your own. Historical note: those of us within the sqlite project had never paid any attention to the unfortunately-named WebAssembly (which has been dubbed "neither web nor assembly") because it didn't seem to hold any relevance for us. It wasn't until April-ish 2022 that we took a look at it, and have been working on "officially" bringing it to the browser world ever since. Even so, folks have been producing WASM bindings of it for a number of years now, and the relative ease of doing so (sqlite3.c requires no changes whatsoever to compile to WASM) is possibly (i opine) why we didn't get requests from users to do this sooner. Sidebar: someone is going to ask "if it's so easy, why did you need 6 months to get it out the door?" Fair question: we had some very specific technical goals which i'm not at liberty to elaborate on, and only a single developer to put on it. Plus, WASM was a completely new tech for us, so there was much learning and experimentation involved.
mch82 4y ago
Browser support is currently limited to Chrome and Edge. Firefox and Safari don't support this yet. Source: https://web.dev/file-system-access/#browser-support https://web.dev/file-system-access/#browser-support Here's a better overview of the sqlite3 WASM project: https://sqlite.org/wasm/doc/trunk/index.md https://sqlite.org/wasm/doc/trunk/index.md. Very excited to try this once support is added to Firefox and Safari!
- sgbeal 4y ago
  > Browser support is currently limited to Chrome and Edge. Firefox and Safari don't support this yet. Within the sqlite project we're fairly convinced that FF and Safari will catch up as soon as their larger customers start targeting Chromium-based browsers simply for the OPFS support. My estimate is mid- to late- 2023 at the latest. My (mis?)understanding is that Safari has most of this support but not the latest changes from Google (namely "sync handles"), and sqlite needs those latest features in order to use OPFS. > Very excited to try this once support is added to Firefox and Safari! That's up to the browser vendors, but it seems very likely that they'll jump on board once large apps start making use of it OPFS (independently of whether or not those apps use sqlite). If their impls are API-compatible with Chrome's (which Google is certainly pushing for), sqlite will "just work". It is likely that the OPFS APIs will be tweaked somewhat in the mean time (e.g. changes in the locking-related support are under discussion), and sqlite's support will/would need to be adjusted accordingly, but "one of these days" it will "just work" across the 3 major browsers.
  - panzerboiler 4y ago
    I love SQLite but this is a terrible take, in my opinion. The Web is not meant to be a property of Google. Browsers don't have to implement the Chrome's API in a compatible way. There is a standardization process in place that should govern the progress of the Web Platform. The "do whatever Chrome does, then everybody else will be forced to follow suit" is doing it wrong.
    - sgbeal 4y ago
      > The Web is not meant to be a property of Google. i'm not sure where you read in that that Google is owning this whole thing. They're just the first out the gate with working OPFS, so that's the implementation we worked against to get this up and running with sqlite. Google has worked with the other browser vendors from the start on the API.
crazysim 4y ago
I think there may be a space for super-large multi-GB files served from static storage being accessible from SQlite as well. Another one would be this full-text search of a 43GB SQLite database of Wikipedia's full text search: http://static.wiki/ http://static.wiki/ . Hearing there's official support for this is awesome and I hope they also might add some nice stuff for those sticking with POSIX/Emscripten as well; maybe some optimizations to access patterns or other stuff like indexing or split DBs? There are some amazing things for SQLite in the browser especially if you're looking for ways to host queryable data for cheap. The example I have below costs $0.42 cents a month to host. For 28GB. Insane. I have a hacked up POC experimental version of the datasette-lite Python UI which runs in Emscripten to be able to look at multi-GB databases at https://github.com/simonw/datasette-lite/pull/49 https://github.com/simonw/datasette-lite/pull/49. It uses a hacked up chunk'd lazyFile implementation from emscripten and others to grab pages from Cloudflare R2. Here's a test/demo with california's unclaimed property records (https://www.sco.ca.gov/upd_download_property_records.html https://www.sco.ca.gov/upd_download_property_records.html) of a 28GB searching up that guy who owns Twitter: https://datasette-lite-lab.mindflakes.com/index.html?url=https://datasette-lite-lab.mindflakes.com/sdb/2022-10-02_93eff57de3573985_ca_unclaimed_property.sqlite#/2022-10-02_93eff57de3573985_ca_unclaimed_property?sql=SELECT+*+FROM+records+WHERE+records.owner_name+MATCH+%22Elon+Musk%22+ORDER+BY+CAST%28CURRENT_CASH_BALANCE+AS+FLOAT%29++DESC%3B https://datasette-lite-lab.mindflakes.com/index.html?url=htt...
- simonw 4y ago
  That demo link is so impressive: I just watched my browser DevTools and it loaded 23MB of data in order to run that query against that 28GB database. I really need to dig in and figure out how Datasette (and Datasette Lite) can work better for this. I think this issue might help - the ability to turn off row counts entirely: https://github.com/simonw/datasette/issues/1818 https://github.com/simonw/datasette/issues/1818 (I noticed that trying to access the table directly seems to suck in a LOT of data, presumably because it's trying to calculate a count across the whole table?)
  - deleted 4y ago
    [deleted]
llimllib 4y ago
I wrote up how to compile SQLite to WASM and also include an extension: https://github.com/llimllib/wasm_sqlite_with_stats https://github.com/llimllib/wasm_sqlite_with_stats You can play with the wasm-compiled extension at the version of the sandbox I compiled here: https://llimllib.github.io/wasm_sqlite_with_stats/ https://llimllib.github.io/wasm_sqlite_with_stats/ I tried to be thorough with the writeup, so hopefully it helps somebody if that's something they need.
quolpr 4y ago
I am preparing for this! Building Kikko[0] that widely supports all the of platforms that have SQLite on it, and allows building reactive interface on top of React/Vue/AngularJS(WIP)/whatever. Here is the code example for react: https://kikko-doc.netlify.app/react-integration/installation https://kikko-doc.netlify.app/react-integration/installation. Some technical details: with special sql`INSERT INTO ${sql.table`some-table`}` syntax, it tracks in which tables changes are happened, and notify other tabs to make refetch to the subscribed tables. Project is in alpha, and already has support of absurd-sql, wa-sqlite for web; expo, tauri, electron, ionic, React Native. I am super excited to add support of official SQLite wasm implementation 0: https://github.com/kikko-land/kikko https://github.com/kikko-land/kikko
- quolpr 4y ago
  Btw, I still see one problem of official SQLite solution — it requires COOP to use ( https://web.dev/i18n/en/coop-coep/ https://web.dev/i18n/en/coop-coep/ ). It is a pretty annoying restriction. For example, you will not be able to use iframe to other resources(like embed youtube video). COOP is needed due to SharedArrayBuffer usage. wa-sqlite can work without COOP, but it is under GPL, unfortunately. It don't allow using it in private codebase without open-sourcing the project
  - sgbeal 4y ago
    > Btw, I still see one problem of official SQLite solution — it requires COOP to use Only for the OPFS support, because any solution involving hiding an asynchronous API (OPFS) behind a synchronous one (sqlite) requires it because the "await" keyword in JS is "viral": it can only be used from global-scope code or from functions which are themselves flagged as "async", and flagging a function as "async" changes its return semantics in ways which are fundamentally incompatible with C code. Any solution to that problem in JS requires SharedArrayBuffer and the Atomics APIs. WASMFS's OPFS implementation has the same limitation, despite being implemented in native code. (That said: there is some talk among those who know better than i of modifying WASM to be able to accept Promises as return values, and returning the resolved promise value to C. i don't think it's possible because promise _rejection_ cannot pass through C code, but folks who know better than i seem to think it can be done.) If you don't need OPFS support you don't need COOP/COEP. (Citation: i'm the sqlite js/wasm developer and have had this discussion with Google's OPFS folks.)
jokoon 4y ago
It depends if mozilla will support OPFS.
zubairq 4y ago
Sqlite is amazing! At yazz.com we have built a tool to build UI apps which use Sqlite as a back end, and then the apps compile to a HTML page which includes the Sqlite engine and the data, which works offline too. +100 for sqlite which makes this possible!
steida 4y ago
React Hooks library for local-first software with end-to-end encrypted backup and sync using SQLite and CRDT https://github.com/evoluhq/evolu https://github.com/evoluhq/evolu

547 points by hochmartinez 4y ago

"Folks have been building sqlite3 for WASM since at least as far back as 2019, but this subproject is the first effort "officially" associated with the SQLite project, created with the goal of making WASM builds of the library first-class members of the family of supported SQLite deliverables." Awesome news. SQLite is already a defacto cross-platform file format. An official WASM release will be widely welcomed and extremely useful.

lovasoa 4y ago
SQLite was one of the first large C code bases to be ported to the web. Alon Zakai (@kripken, the creator of emscripten), made the first commit to sql.js at the start of 2012 [0]. I got involved two years later, and have been maintaining sql.js since. A lot was added since 2012, but the initial core api with 3 functions 'open', 'exec', and 'close' still works today. https://github.com/kripken/sql.js/commit/cebd80648dbd369b34804c5a00b4d0bddc1cbf05 https://github.com/kripken/sql.js/commit/cebd80648dbd369b348...
- sgbeal 4y ago
  > SQLite was one of the first large C code bases to be ported to the web. Thank you very much for that info. i'll update the sqlite wasm docs with that. Edit: updated, thank you! <https://sqlite.org/wasm/info/cacc33abc5d9013f https://sqlite.org/wasm/info/cacc33abc5d9013f>
  - lovasoa 4y ago
    > Folks have been building sqlite3 for WASM since as far back as 2012 for *the web*, not for WASM. SQLite was initially compiled to simple javascript, then to asm.js when it appeared, then to WASM when it replaced asm.js :) The sql.js project is older than the idea of WASM itself. And I like to think it contributed to showing the potential of compiling native code to the browser, and in the creation of the WASM standard.
    - sgbeal 4y ago
      > for the web, not for WASM. Docs have been updated. Thank you for the feedback! Edit: for future feedback (from anyone reading this) i can be reached via stephan at sqlite org. i'm unlikely to catch most doc feedback posted to HN.

hochmartinez 4y ago

Demos you can inspect... https://sqlite.org/wasm/doc/tip/demo-123.html https://sqlite.org/wasm/doc/tip/demo-123.html https://sqlite.org/wasm/doc/tip/demo-123-worker.html https://sqlite.org/wasm/doc/tip/demo-123-worker.html

hn2017 4y ago

Excited about this news, this the kind of change the modern web needs.

boppo1 4y ago
Can you help me understand how/why this is a big change the web needs? I'm not seeing the value proposition on a database-in-a-webpage
- Scarbutt 4y ago
  For the same reason many complex desktop applications use sqlite or some other database.
- exelib 4y ago
  My use case is an offline-capable app. There was already WebSQL, but it got deprectaed.
  - chasd00 4y ago
    > My use case is an offline-capable app. not trying to be snarky but do you mean a desktop app?
    - EvanAnderson 4y ago
      Offline-capable apps on phones are "a thing". There are plenty of times when I have poor (or nonexistent) cellular coverage and would love web-based apps to work.
    - hot_gril 4y ago
      A desktop app that's cross-platform.
    - exelib 4y ago
      Currently, it is a browser app intended to run on mobile devices. Not exactly PWA, but very similar. Later it will be, probably, packaged with capacitor.

adamrezich 4y ago

awesome! it's also great to see a project as important as sqlite provide first-class wasm support like this.

coder543 4y ago

As a little pro-tip, the demos appear to be serving a gzip compressed WASM file, but offering a brotli compressed version for clients that say they accept it would be even better: 665K sqlite3.wasm 306K sqlite3.wasm.gz (-54% compared to uncompressed) 266K sqlite3.wasm.br (-13% compared to gz, -60% compared to uncompressed)

dstaley 4y ago
Oooff, 266 KB is a hefty price to pay. I really wish WebSQL would have won out over IndexedDB, because I would have loved to have been able to build libraries on top of that instead of the headache IndexedDB is.
- coder543 4y ago
  "Hefty"... for an asset that can easily be cached long term, so it is effectively a one time penalty for a given website, and many web apps will load images larger than this. Certainly, just about any marketing/product page will load more bytes than this in the form of images, let alone videos. So, "hefty" seems like a bit of an exaggeration to me. All other things equal, lighter is always better, but 266KB for all the functionality SQLite offers isn't that bad, IMO. On the topic of caching, since SQLite is (intentionally or not) going to be setting an example with their docs and demos[0], I would suggest that SQLite should demonstrate the industry best practices with regards to caching as well. Static assets like WASM and JavaScript should be served with a very high cache duration, and the name of the asset should include a hash of the asset. This way, the site maintainer can update the HTML to reference the new SQLite bundles by the new hash whenever they upload new versions, and browsers will immediately request the new versions, but they will otherwise instantly load the cached version after the first visit whenever there isn't a new version. [0]: https://sqlite.org/wasm/doc/trunk/demo-123.md https://sqlite.org/wasm/doc/trunk/demo-123.md
  - fooey 4y ago
    cross-origin caching on the web has been dead for years doesn't seem possible to do without leaking data

ramesh31 4y ago

Feels like WASM is hitting an inflection point this year. After nearly a decade of hearing "someday", that day is finally here.

MuffinFlavored 4y ago
I wonder what the catalyst finally is/was to cause such an acceleration
- ilyt 4y ago
  Compound interest. Tools that make making other tools easier. Once programming languages started having WASM compile target a great many things became easier
- sgbeal 4y ago
  > I wonder what the catalyst finally is/was to cause such an acceleration IMO, all that was missing was a truly compelling use case. The combination of the ubiquitous sqlite with non-trivially-sized persistent storage gives us that use case.
sgbeal 4y ago
> Feels like WASM is hitting an inflection point this year. After nearly a decade of hearing "someday", that day is finally here. At the start of this development effort (April, IIRC (2022)) we (in the sqlite project) had only ever heard of wasm but hadn't paid any attention to it. Within just a few days of starting this project, we were fully convinced that the combination of sqlite and OPFS will be one of the Next Big Things for web app development. As the project's "JS/WASM Guy" i'm exceedingly excited to see what people do with this and what improvements we'll make based on user feedback. (It's long been my experience that the most interesting feature suggestions come from users.)

exelib 4y ago

Amazing! 313kb gzipped. I hope the author(s) of IndexedDB would finally understand how stupid the idea of IDB was. Are there any numbers on performance?

sgbeal 4y ago
> Are there any numbers on performance? Tucked away in spreadsheets there are but we don't currently have any benchmarks in a publishable form. In broad strokes, i can assure you (as the one who performed the benchmarks) that the new sqlite wasm is competitive with WebSQL in terms of performance, with either one winning out in certain benchmarks. Given that WebSQL is implemented in native code and sqlite in wasmified C, we're quite happy with those results. Note, however, that benchmarks are very browser-dependent. Firefox's wasm engine, for example, is significantly slower than Chrome's, but it's also more consistent. If you run a given test 10 times in FF, the difference in runtimes across them will be small (maybe 10%), whereas there will be a +/-50% difference in runtimes for the same tests in Chrome. In Chrome, if the dev tools are open when wasm is running, wasm's performance can (for unknown reasons) drop by as much as half or more even if the wasm code produces no output.
- exelib 4y ago
  Thank you very much, that sounds great!
tehbeard 4y ago
> I hope the author(s) of IndexedDB would finally understand how stupid the idea of IDB was While not one of said authors... You mind giving me an actual rundown on issues you've got with it instead of just slinging mud? I've used IDB in a few projects, and apart from: - The god awful callback crap born of being pre-promises. - The lack of partial indexes (e.g. indexing a subset of documents based on some parameters) - The iOS Webkit teams continued habit adding weird, app-breaking bugs every other release to an api/system that should be stable? It's been fairly useful. Hell, it supports storing JS types like CryptoKey or Blob/File with no real issues[1] [1] Sans iOS WebKit, which a version or 2 back would just eat Blobs at random.

nilslice 4y ago

very cool to look at the unpacked module, in all its wasm glory https://whatsinyourwasm.com/module/6 https://whatsinyourwasm.com/module/6 (warning: not mobile friendly...)

sgbeal 4y ago
> very cool to look at the unpacked module, in all its wasm glory Very nice :). Be aware that the copy of sqlite3.wasm on the sqlite.org/wasm site gets updated fairly frequently, so may differ from that at any moment. That particular site is only for documentation purposes, not for hosting the canonical wasm file release (which is pending along with the release of sqlite 3.40), and its sqlite3.wasm/js copies get updated hand in hand with development of the canonical copies.
- nilslice 4y ago
  Good to know! FWIW, this is a very beta demo version of a product I'm working on (Modsurfer). It's made to capture snapshot/moment-in-time details about a module (maybe you're running untrusted code in your system and want to track what you're actually running). However, in the future, it will likely be able to automatically import and track modules from cloud storage or URLs.

ibash 4y ago

If you want to use sqlite in the browser today (chrome only) take a look at websql.

samwillis 4y ago
WebSQL is dead, long live WASM SQLite. And in all seriousness, WASM is exactly the way this should be done. It doesn’t tie a db engine to a specific standardised version. It’s far more important to develop low level block storage apis like the OPFS that WASM SQLite is using.
- ilyt 4y ago
  I'd still prefer some version of SQLite to be bundled, just to save on transfer for apps that are tiny
resoluteteeth 4y ago
chrome is presumably eventually going to remove websql. I guess they are trying to do it slowly so they don't break stuff, but they're already doing things like removing FTS support.
- sgbeal 4y ago
  > I guess they are trying to do it slowly so they don't break stuff, but they're already doing things like removing FTS support. FWIW, i can say with some authority that they have been waiting on this new sqlite/wasm stuff so that they can offer a replacement to WebSQL for their folks who still use it. Despite Google's long history of pulling the plug on products (G+, how you are missed!), they're not willing to outright drop WebSQL without a viable replacement. It would not be difficult to write a drop-in workalike WebSQL wrapper on top of the new wasm/js APIs, and we (in the sqlite project) may even get around to doing so as time allows for. The difficulty would be making it "quirk for quirk compatible," as such quirks are not documented anywhere. OTOH, as far as we're aware, the only extant WebSQL implementation is the one in Chrome, so that's the only one which counts for purposes of quirks.

samwillis 4y ago

> … support persistent client-side storage using available JS APIs. As of this writing, that includes the Origin-Private FileSystem (OPFS) [0] This is really good news, and exactly what the OPFS was designed for. You may have seen “Absurd SQL” [1] which was a proof of concept for building a SQLite Virtual FS backend using IndexedDB. It provided full ACID compliment transactions. Incredible work but a hack at best. The OPFS supersedes all that and makes it possible to have proper consistent and resilient transactions. WASM SQLite with the OPFS is the future of offline first web app development. The concept of a single codebase web/mobile/desktop app with proper offline storage is here. What I really want to see next is an eventually constant sync system between browser and server (or truly distributed with WebRTC). The SQLite Session Extension [2] potentially has the building blocks needed for such a system. 0: https://webkit.org/blog/12257/the-file-system-access-api-with-origin-private-file-system/ https://webkit.org/blog/12257/the-file-system-access-api-wit... 1: https://github.com/jlongster/absurd-sql https://github.com/jlongster/absurd-sql 2: https://www.sqlite.org/sessionintro.html https://www.sqlite.org/sessionintro.html

Existenceblinks 4y ago
> persistent Question: is it really persisted indefinitely? And will this SQLite file be portable (import and export to somewhere else) ?
- srcreigh 4y ago
  It’s not persistent everywhere. On iOS the data is deleted after 7 days of inactivity on a website. It’s treated like a cache. https://webkit.org/blog/12257/the-file-system-access-api-with-origin-private-file-system/ https://webkit.org/blog/12257/the-file-system-access-api-wit... https://web.dev/storage-for-the-web/#:~:text=Starting%20in%20iOS%20and%20iPadOS,not%20interact%20with%20the%20site https://web.dev/storage-for-the-web/#:~:text=Starting%20in%2....
  - tehbeard 4y ago
    Missed one part from web.dev > ...This eviction policy does not apply to installed PWAs that have been added to the home screen.

hinkley 4y ago

I entirely missed the era where SQLite was a candidate for W3C inclusion. It was always a mental bookmark in case I found a need to have an app with any substantial offline mode, but by the time I found a perfect candidate (cataloging plants in situ), it was already deprecated, and a key-value store is no replacement for SQLite.

exelib 4y ago
Key-value store would be OK, we (developers) can handle that. And in fact, we do it already. But IndexedDB is broken by design. But I agree, handle relational data is what we want. And you can easily emulate non-relation data with it, if you need.
- sgbeal 4y ago
  > Key-value store would be OK, we (developers) can handle that. And in fact, we do it already. ... But I agree, handle relational data is what we want. You can now have both: a relational database in your key-value localStorage: <https://sqlite.org/wasm/doc/trunk/persistence.md#kvvfs https://sqlite.org/wasm/doc/trunk/persistence.md#kvvfs> :-D
  - exelib 4y ago
    localStorage suffers from issues like size limits and performance (which, probably, leads to higher battery consumption).
    - sgbeal 4y ago
      > localStorage suffers from issues like size limits and performance Absolutely, but _it works_ and nothing trumps working code ;). A slow and space-limited database is better than none at all! We implemented that functionality primarily to make peoples' eyes bug out ;), but also so that folks who don't yet have OPFS can have some form of persistent sqlite databases.
    - hinkley 4y ago

rcarmo 4y ago

Very nice. I built sqlite3 for WASI/a-Shell to use on my iPad (https://github.com/rcarmo/wasi-sqlite https://github.com/rcarmo/wasi-sqlite) and it still has a few issues, I hope this will help (although right now the biggest issue seems to be that the REPL has some sort of memory leak when run inside WASI).

lovasoa 4y ago

Hi! I'm the main maintainer of sql.js. Is there somewhere I can get in touch with you? Would you be interested in publishing this as a new major version of sql.js itself ?

remram 4y ago
Their forum: https://sqlite.org/forum/forum https://sqlite.org/forum/forum
erikpukinskis 4y ago
Thank you for sql.js by the way. It is revolutionary!

harryvederci 4y ago

I couldn't find a link to the source code on that page, I think this is it: https://sqlite.org/wasm/tree https://sqlite.org/wasm/tree As it's a fossil repo, here's the steps to clone it and check it out locally: sudo apt install fossil # Install fossil (on Debian) fossil clone https://sqlite.org/wasm # Clone the SQLite WASM repo cd wasm # Navigate to the cloned repo fossil ui # Launch fossil in UI-mode

thruflo 4y ago

The Google Chrome team is working with the SQLite team to standardise a WASM based SQLite in the browser to replace WebSQL. https://developer.chrome.com/blog/deprecating-web-sql/ https://developer.chrome.com/blog/deprecating-web-sql/ https://twitter.com/chromiumdev/status/1565105522092695553 https://twitter.com/chromiumdev/status/1565105522092695553 Which would be pretty cool. Projects like SQL.js and absurd-sql are awesome but naturally a bit rough around the edges and not highly maintained.

modeless 4y ago
This is good but it can never be as good as Web SQL could have been, because it can't be a truly shared library. Like all WASM modules, It has to be downloaded and JIT compiled separately for every site that uses it. Not for technical reasons, but privacy reasons, so it's basically unfixable: https://developer.chrome.com/en/blog/http-cache-partitioning/ https://developer.chrome.com/en/blog/http-cache-partitioning... Maybe if half the sites on the web start using it, browsers can finally be convinced that it would be OK to add SQLite to the base platform.
- thruflo 4y ago
  Interesting — so basically an overhead until it’s fully native? I was reading the Chrome initiative as a pathway to native. Would that mean it can’t be WASM?
  - modeless 4y ago
    That tweet is not implying that it will ever be native. Quite the opposite; Web SQL is native today and they are going to remove it since it was rejected by other browsers.
- samwillis 4y ago
  I disagree, this is much better than WebSQL. WebSQL would have been tied to one specific version of SQLite, and developers would have to work to the lowest supported version. There would have been no extension mechanism. WASM SQLite is the correct solution. It’s extendable by the developer using it, they can uses SQLite extension modules, and build their own. But almost more so it proves the idea of a WASM db engine backed by a low level block FS api. We will see other db engines uses this architecture. DuckDB have already done it. I’m sure MongoDBs Realm and CouchBase Mobile will do the same soon too. We are in for an exciting time in the next few years.

alberth 4y ago

Will this help Cloudflare, Fly.io and others who started offering SQLite-like of offerings in the cloud?

TheRealPomax 4y ago

The thing I don't get here is that literally every browser already uses Sqlite3 for a million things, why are we not just finally acknowledging that, yes, this very specific library is a universal good, just like gzip compression, or the jpeg image format, and here's the JS API for directly working with it? Why do we need a separate WASM version of something that's already built right into every single browser? Why is there no all-browser-vendor-blessed `Sqlite3` global?

easton 4y ago
They tried that at one point, I think it was called WebSQL? But it was decided that they didn’t want to make the quirks of sqlite3 a web standard since it’d be hard to do a clean room implementation.
- TheRealPomax 4y ago
  Except I don't want WebSQL, I literally want Sqlite3 as a JS API, backed by the unadulterated authoritative Sqlite3 library. I don't wonder why there's no good in-browser database solution that uses SQL, I literally don't get why sqlite3, only, and specifically, still isn't usable by JS in 2022.
  - deleted 4y ago
    [deleted]
  - sgbeal 4y ago
    > I literally don't get why sqlite3, only, and specifically, still isn't usable by JS in 2022. It is now :). We provide 4 separate APIs, from the lowest-level 1-to-1 C-via-WASM bindings to one quite similar to sql.js, plus all of the low-level pieces necessary to create your own. Historical note: those of us within the sqlite project had never paid any attention to the unfortunately-named WebAssembly (which has been dubbed "neither web nor assembly") because it didn't seem to hold any relevance for us. It wasn't until April-ish 2022 that we took a look at it, and have been working on "officially" bringing it to the browser world ever since. Even so, folks have been producing WASM bindings of it for a number of years now, and the relative ease of doing so (sqlite3.c requires no changes whatsoever to compile to WASM) is possibly (i opine) why we didn't get requests from users to do this sooner. Sidebar: someone is going to ask "if it's so easy, why did you need 6 months to get it out the door?" Fair question: we had some very specific technical goals which i'm not at liberty to elaborate on, and only a single developer to put on it. Plus, WASM was a completely new tech for us, so there was much learning and experimentation involved.

mch82 4y ago

Browser support is currently limited to Chrome and Edge. Firefox and Safari don't support this yet. Source: https://web.dev/file-system-access/#browser-support https://web.dev/file-system-access/#browser-support Here's a better overview of the sqlite3 WASM project: https://sqlite.org/wasm/doc/trunk/index.md https://sqlite.org/wasm/doc/trunk/index.md. Very excited to try this once support is added to Firefox and Safari!

sgbeal 4y ago
> Browser support is currently limited to Chrome and Edge. Firefox and Safari don't support this yet. Within the sqlite project we're fairly convinced that FF and Safari will catch up as soon as their larger customers start targeting Chromium-based browsers simply for the OPFS support. My estimate is mid- to late- 2023 at the latest. My (mis?)understanding is that Safari has most of this support but not the latest changes from Google (namely "sync handles"), and sqlite needs those latest features in order to use OPFS. > Very excited to try this once support is added to Firefox and Safari! That's up to the browser vendors, but it seems very likely that they'll jump on board once large apps start making use of it OPFS (independently of whether or not those apps use sqlite). If their impls are API-compatible with Chrome's (which Google is certainly pushing for), sqlite will "just work". It is likely that the OPFS APIs will be tweaked somewhat in the mean time (e.g. changes in the locking-related support are under discussion), and sqlite's support will/would need to be adjusted accordingly, but "one of these days" it will "just work" across the 3 major browsers.
- panzerboiler 4y ago
  I love SQLite but this is a terrible take, in my opinion. The Web is not meant to be a property of Google. Browsers don't have to implement the Chrome's API in a compatible way. There is a standardization process in place that should govern the progress of the Web Platform. The "do whatever Chrome does, then everybody else will be forced to follow suit" is doing it wrong.
  - sgbeal 4y ago
    > The Web is not meant to be a property of Google. i'm not sure where you read in that that Google is owning this whole thing. They're just the first out the gate with working OPFS, so that's the implementation we worked against to get this up and running with sqlite. Google has worked with the other browser vendors from the start on the API.

crazysim 4y ago

I think there may be a space for super-large multi-GB files served from static storage being accessible from SQlite as well. Another one would be this full-text search of a 43GB SQLite database of Wikipedia's full text search: http://static.wiki/ http://static.wiki/ . Hearing there's official support for this is awesome and I hope they also might add some nice stuff for those sticking with POSIX/Emscripten as well; maybe some optimizations to access patterns or other stuff like indexing or split DBs? There are some amazing things for SQLite in the browser especially if you're looking for ways to host queryable data for cheap. The example I have below costs $0.42 cents a month to host. For 28GB. Insane. I have a hacked up POC experimental version of the datasette-lite Python UI which runs in Emscripten to be able to look at multi-GB databases at https://github.com/simonw/datasette-lite/pull/49 https://github.com/simonw/datasette-lite/pull/49. It uses a hacked up chunk'd lazyFile implementation from emscripten and others to grab pages from Cloudflare R2. Here's a test/demo with california's unclaimed property records (https://www.sco.ca.gov/upd_download_property_records.html https://www.sco.ca.gov/upd_download_property_records.html) of a 28GB searching up that guy who owns Twitter: https://datasette-lite-lab.mindflakes.com/index.html?url=https://datasette-lite-lab.mindflakes.com/sdb/2022-10-02_93eff57de3573985_ca_unclaimed_property.sqlite#/2022-10-02_93eff57de3573985_ca_unclaimed_property?sql=SELECT+*+FROM+records+WHERE+records.owner_name+MATCH+%22Elon+Musk%22+ORDER+BY+CAST%28CURRENT_CASH_BALANCE+AS+FLOAT%29++DESC%3B https://datasette-lite-lab.mindflakes.com/index.html?url=htt...

simonw 4y ago
That demo link is so impressive: I just watched my browser DevTools and it loaded 23MB of data in order to run that query against that 28GB database. I really need to dig in and figure out how Datasette (and Datasette Lite) can work better for this. I think this issue might help - the ability to turn off row counts entirely: https://github.com/simonw/datasette/issues/1818 https://github.com/simonw/datasette/issues/1818 (I noticed that trying to access the table directly seems to suck in a LOT of data, presumably because it's trying to calculate a count across the whole table?)
- deleted 4y ago
  [deleted]

llimllib 4y ago

I wrote up how to compile SQLite to WASM and also include an extension: https://github.com/llimllib/wasm_sqlite_with_stats https://github.com/llimllib/wasm_sqlite_with_stats You can play with the wasm-compiled extension at the version of the sandbox I compiled here: https://llimllib.github.io/wasm_sqlite_with_stats/ https://llimllib.github.io/wasm_sqlite_with_stats/ I tried to be thorough with the writeup, so hopefully it helps somebody if that's something they need.

quolpr 4y ago

I am preparing for this! Building Kikko[0] that widely supports all the of platforms that have SQLite on it, and allows building reactive interface on top of React/Vue/AngularJS(WIP)/whatever. Here is the code example for react: https://kikko-doc.netlify.app/react-integration/installation https://kikko-doc.netlify.app/react-integration/installation. Some technical details: with special sql`INSERT INTO ${sql.table`some-table`}` syntax, it tracks in which tables changes are happened, and notify other tabs to make refetch to the subscribed tables. Project is in alpha, and already has support of absurd-sql, wa-sqlite for web; expo, tauri, electron, ionic, React Native. I am super excited to add support of official SQLite wasm implementation 0: https://github.com/kikko-land/kikko https://github.com/kikko-land/kikko

quolpr 4y ago
Btw, I still see one problem of official SQLite solution — it requires COOP to use ( https://web.dev/i18n/en/coop-coep/ https://web.dev/i18n/en/coop-coep/ ). It is a pretty annoying restriction. For example, you will not be able to use iframe to other resources(like embed youtube video). COOP is needed due to SharedArrayBuffer usage. wa-sqlite can work without COOP, but it is under GPL, unfortunately. It don't allow using it in private codebase without open-sourcing the project
- sgbeal 4y ago
  > Btw, I still see one problem of official SQLite solution — it requires COOP to use Only for the OPFS support, because any solution involving hiding an asynchronous API (OPFS) behind a synchronous one (sqlite) requires it because the "await" keyword in JS is "viral": it can only be used from global-scope code or from functions which are themselves flagged as "async", and flagging a function as "async" changes its return semantics in ways which are fundamentally incompatible with C code. Any solution to that problem in JS requires SharedArrayBuffer and the Atomics APIs. WASMFS's OPFS implementation has the same limitation, despite being implemented in native code. (That said: there is some talk among those who know better than i of modifying WASM to be able to accept Promises as return values, and returning the resolved promise value to C. i don't think it's possible because promise _rejection_ cannot pass through C code, but folks who know better than i seem to think it can be done.) If you don't need OPFS support you don't need COOP/COEP. (Citation: i'm the sqlite js/wasm developer and have had this discussion with Google's OPFS folks.)

jokoon 4y ago

It depends if mozilla will support OPFS.

zubairq 4y ago

Sqlite is amazing! At yazz.com we have built a tool to build UI apps which use Sqlite as a back end, and then the apps compile to a HTML page which includes the Sqlite engine and the data, which works offline too. +100 for sqlite which makes this possible!

steida 4y ago

React Hooks library for local-first software with end-to-end encrypted backup and sync using SQLite and CRDT https://github.com/evoluhq/evolu https://github.com/evoluhq/evolu