4 ms·

As a little pro-tip, the demos appear to be serving a gzip compressed WASM file, but offering a brotli compressed version for clients that say they accept it wo

As a little pro-tip, the demos appear to be serving a gzip compressed WASM file, but offering a brotli compressed version for clients that say they accept it would be even better: 665K sqlite3.wasm 306K sqlite3.wasm.gz (-54% compared to uncompressed) 266K sqlite3.wasm.br (-13% compared to gz, -60% compared to uncompressed)

dstaley 4y ago
Oooff, 266 KB is a hefty price to pay. I really wish WebSQL would have won out over IndexedDB, because I would have loved to have been able to build libraries on top of that instead of the headache IndexedDB is.
- coder543 4y ago
  "Hefty"... for an asset that can easily be cached long term, so it is effectively a one time penalty for a given website, and many web apps will load images larger than this. Certainly, just about any marketing/product page will load more bytes than this in the form of images, let alone videos. So, "hefty" seems like a bit of an exaggeration to me. All other things equal, lighter is always better, but 266KB for all the functionality SQLite offers isn't that bad, IMO. On the topic of caching, since SQLite is (intentionally or not) going to be setting an example with their docs and demos[0], I would suggest that SQLite should demonstrate the industry best practices with regards to caching as well. Static assets like WASM and JavaScript should be served with a very high cache duration, and the name of the asset should include a hash of the asset. This way, the site maintainer can update the HTML to reference the new SQLite bundles by the new hash whenever they upload new versions, and browsers will immediately request the new versions, but they will otherwise instantly load the cached version after the first visit whenever there isn't a new version. [0]: https://sqlite.org/wasm/doc/trunk/demo-123.md https://sqlite.org/wasm/doc/trunk/demo-123.md
  - fooey 4y ago
    cross-origin caching on the web has been dead for years doesn't seem possible to do without leaking data
    - coder543 4y ago
      I agree. I said: > so it is effectively a one time penalty for a given website which shows that I already acknowledged the demise of cross-origin caching. Each website the user visits that uses this library has to pay a 266KB penalty once, which isn't that bad. (Well, once, unless the site developer decides to upgrade the library, then the penalty applies once more, but that's expected behavior.)
  - sgbeal 4y ago
    > On the topic of caching, since SQLite is (intentionally or not) going to be setting an example with their docs and demos[0], I would suggest that SQLite should demonstrate the industry best practices with regards to caching as well. Static assets like WASM and JavaScript should be served with a very high cache duration That can't work on the documentation site because the wasm file is being served from the Fossil SCM and fossil cannot cache resources which are fetched by name because a new version may be checked in at any given moment (they're currently updated very often: https://sqlite.org/wasm/finfo/jswasm/sqlite3.wasm https://sqlite.org/wasm/finfo/jswasm/sqlite3.wasm). Fossil can hypothetically cache resources which are fetched by hash, but that's not a feasible way for us to maintain the documentation on that site.
    - coder543 4y ago
      That makes sense, although these topics could still potentially be mentioned in the documentation.
      - sgbeal 4y ago
        > ... although these topics could still potentially be mentioned in the documentation. They're not relevant to the wasm/js deliverables. They're an implementation detail of the server which happens to be serving the related documentation. If you have specific verbiage which you feel would improve the docs in this regard, please feel free to send it. i'm likely to miss most responses in HN, so please use email (stephan at sqlite dot org) or the sqlite forum: https://sqlite.org/forum https://sqlite.org/forum.
- sgbeal 4y ago
  > Oooff, 266 KB is a hefty price to pay. i think you'll find that many high-end websites often download more than a megabyte of CSS and JS code. imdb.com home page: 2.13mb transferred for 6.odd mb data. drive.google.com: 4.75mb transferred for nearly 15mb of data(!!!). 266kb doesn't even register nowadays for app-centric pages.
hochmartinez 4y ago
sqlite3.wasm should be included in Chrome, Firefox, Safary, Brave, etc. And updated automatically. Sqlite updates are solid and as far as I know, do not break your code.
- Beltalowda 4y ago
  > Sqlite updates are solid and as far as I know, do not break your code. Almost every major SQLite release has a few minor releases after it which fix bug and regressions, including things like queries returning the wrong result.
  - modeless 4y ago
    Browser engines have regressions too. SQLite has a better test suite than any browser. It's not going to be a major source of regressions compared to the rest of the platform.
    - hochmartinez 4y ago
      That's correct. SQLite has an impressive test suite that makes it very stable. That's why it's used in millions of Android devices or will be used in Cloudflare's edge with D1, for example.
      - sgbeal 4y ago
        > That's why it's used in millions of Android devices s/millions/billions/g. It is widely believed to be either the single most widely-deployed piece of software in the world, or maybe second behind zlib (we have no way of being sure).
- josephg 4y ago
  The downside of that is that application developers would have to use the minimum supported version of SQLite. If every site embeds their own version, the application developers can choose their own version that runs the same across every browser. Or even compile it themselves if they want, so they can use extensions. The download size is unfortunate but remember - unlike javascript, wasm bytecode loads almost instantly. Having a 250k wasm module is more like having a 250k image on your site than 250k of javascript. (Though it might still affect time-to-interactive depending on how the site is built)
sgbeal 4y ago
> As a little pro-tip, the demos appear to be serving a gzip compressed WASM file, but offering a brotli compressed version for clients that say they accept it would be even better: That is interesting, but the repository you're looking at is a Fossil SCM repo and the wasm file is being served directly from it. Fossil does the compression transparently and doesn't support brotli. (Edit: i'll investigate whether brotli compression would be interesting for us to add to fossil.) However, the sqlite project will not be hosting shared copies of the wasm/js files for use by arbitrary 3rd-party sites. It's up to each site to host their own (and even build their own if they need to customize the build), so they're free to use whatever compression they like.

dstaley 4y ago

Oooff, 266 KB is a hefty price to pay. I really wish WebSQL would have won out over IndexedDB, because I would have loved to have been able to build libraries on top of that instead of the headache IndexedDB is.

coder543 4y ago
"Hefty"... for an asset that can easily be cached long term, so it is effectively a one time penalty for a given website, and many web apps will load images larger than this. Certainly, just about any marketing/product page will load more bytes than this in the form of images, let alone videos. So, "hefty" seems like a bit of an exaggeration to me. All other things equal, lighter is always better, but 266KB for all the functionality SQLite offers isn't that bad, IMO. On the topic of caching, since SQLite is (intentionally or not) going to be setting an example with their docs and demos[0], I would suggest that SQLite should demonstrate the industry best practices with regards to caching as well. Static assets like WASM and JavaScript should be served with a very high cache duration, and the name of the asset should include a hash of the asset. This way, the site maintainer can update the HTML to reference the new SQLite bundles by the new hash whenever they upload new versions, and browsers will immediately request the new versions, but they will otherwise instantly load the cached version after the first visit whenever there isn't a new version. [0]: https://sqlite.org/wasm/doc/trunk/demo-123.md https://sqlite.org/wasm/doc/trunk/demo-123.md
- fooey 4y ago
  cross-origin caching on the web has been dead for years doesn't seem possible to do without leaking data
  - coder543 4y ago
    I agree. I said: > so it is effectively a one time penalty for a given website which shows that I already acknowledged the demise of cross-origin caching. Each website the user visits that uses this library has to pay a 266KB penalty once, which isn't that bad. (Well, once, unless the site developer decides to upgrade the library, then the penalty applies once more, but that's expected behavior.)
- sgbeal 4y ago
  > On the topic of caching, since SQLite is (intentionally or not) going to be setting an example with their docs and demos[0], I would suggest that SQLite should demonstrate the industry best practices with regards to caching as well. Static assets like WASM and JavaScript should be served with a very high cache duration That can't work on the documentation site because the wasm file is being served from the Fossil SCM and fossil cannot cache resources which are fetched by name because a new version may be checked in at any given moment (they're currently updated very often: https://sqlite.org/wasm/finfo/jswasm/sqlite3.wasm https://sqlite.org/wasm/finfo/jswasm/sqlite3.wasm). Fossil can hypothetically cache resources which are fetched by hash, but that's not a feasible way for us to maintain the documentation on that site.
  - coder543 4y ago
    That makes sense, although these topics could still potentially be mentioned in the documentation.
    - sgbeal 4y ago
      > ... although these topics could still potentially be mentioned in the documentation. They're not relevant to the wasm/js deliverables. They're an implementation detail of the server which happens to be serving the related documentation. If you have specific verbiage which you feel would improve the docs in this regard, please feel free to send it. i'm likely to miss most responses in HN, so please use email (stephan at sqlite dot org) or the sqlite forum: https://sqlite.org/forum https://sqlite.org/forum.
sgbeal 4y ago
> Oooff, 266 KB is a hefty price to pay. i think you'll find that many high-end websites often download more than a megabyte of CSS and JS code. imdb.com home page: 2.13mb transferred for 6.odd mb data. drive.google.com: 4.75mb transferred for nearly 15mb of data(!!!). 266kb doesn't even register nowadays for app-centric pages.

hochmartinez 4y ago

sqlite3.wasm should be included in Chrome, Firefox, Safary, Brave, etc. And updated automatically. Sqlite updates are solid and as far as I know, do not break your code.

Beltalowda 4y ago
> Sqlite updates are solid and as far as I know, do not break your code. Almost every major SQLite release has a few minor releases after it which fix bug and regressions, including things like queries returning the wrong result.
- modeless 4y ago
  Browser engines have regressions too. SQLite has a better test suite than any browser. It's not going to be a major source of regressions compared to the rest of the platform.
  - hochmartinez 4y ago
    That's correct. SQLite has an impressive test suite that makes it very stable. That's why it's used in millions of Android devices or will be used in Cloudflare's edge with D1, for example.
    - sgbeal 4y ago
      > That's why it's used in millions of Android devices s/millions/billions/g. It is widely believed to be either the single most widely-deployed piece of software in the world, or maybe second behind zlib (we have no way of being sure).
josephg 4y ago
The downside of that is that application developers would have to use the minimum supported version of SQLite. If every site embeds their own version, the application developers can choose their own version that runs the same across every browser. Or even compile it themselves if they want, so they can use extensions. The download size is unfortunate but remember - unlike javascript, wasm bytecode loads almost instantly. Having a 250k wasm module is more like having a 250k image on your site than 250k of javascript. (Though it might still affect time-to-interactive depending on how the site is built)

sgbeal 4y ago

> As a little pro-tip, the demos appear to be serving a gzip compressed WASM file, but offering a brotli compressed version for clients that say they accept it would be even better: That is interesting, but the repository you're looking at is a Fossil SCM repo and the wasm file is being served directly from it. Fossil does the compression transparently and doesn't support brotli. (Edit: i'll investigate whether brotli compression would be interesting for us to add to fossil.) However, the sqlite project will not be hosting shared copies of the wasm/js files for use by arbitrary 3rd-party sites. It's up to each site to host their own (and even build their own if they need to customize the build), so they're free to use whatever compression they like.