5 ms·

This is good but it can never be as good as Web SQL could have been, because it can't be a truly shared library. Like all WASM modules, It has to be downloaded

This is good but it can never be as good as Web SQL could have been, because it can't be a truly shared library. Like all WASM modules, It has to be downloaded and JIT compiled separately for every site that uses it. Not for technical reasons, but privacy reasons, so it's basically unfixable: https://developer.chrome.com/en/blog/http-cache-partitioning/ https://developer.chrome.com/en/blog/http-cache-partitioning... Maybe if half the sites on the web start using it, browsers can finally be convinced that it would be OK to add SQLite to the base platform.

thruflo 4y ago
Interesting — so basically an overhead until it’s fully native? I was reading the Chrome initiative as a pathway to native. Would that mean it can’t be WASM?
- modeless 4y ago
  That tweet is not implying that it will ever be native. Quite the opposite; Web SQL is native today and they are going to remove it since it was rejected by other browsers.
samwillis 4y ago
I disagree, this is much better than WebSQL. WebSQL would have been tied to one specific version of SQLite, and developers would have to work to the lowest supported version. There would have been no extension mechanism. WASM SQLite is the correct solution. It’s extendable by the developer using it, they can uses SQLite extension modules, and build their own. But almost more so it proves the idea of a WASM db engine backed by a low level block FS api. We will see other db engines uses this architecture. DuckDB have already done it. I’m sure MongoDBs Realm and CouchBase Mobile will do the same soon too. We are in for an exciting time in the next few years.
- modeless 4y ago
  > WebSQL would have been tied to one specific version of SQLite, and developers would have to work to the lowest supported version. There would have been no extension mechanism. Why not? Web SQL would not be different from WebGL/GLSL or even JavaScript itself in that respect. Developers use feature detection and and work to the lowest supported version. APIs and languages can evolve, but in (mostly) backwards compatible ways. Extension and versioning mechanisms can be made. That's how the web works and Web SQL could have worked that way too. More broadly, you could apply arguments like this against everything in the entire web platform. Maybe everything should be a WASM module that developers could choose themselves! Image loading, video decoding, font rendering, DOM, JS engine, why not? It's actually a beautiful vision and I'd be all for it if not for cache partitioning. Every site would have to re-download and re-JIT an entire browser engine before it could do anything. The base platform needs to include a diverse set of commonly used features so that apps don't have to download the world, and on a list of ubiquitous libraries SQLite is right up there with other libraries backing the web platform, like zlib.
  - samwillis 4y ago
    WebGL/GLSL is a low level API the equivalent of the OPFS api. Standardising WebSQL would be like browsers standardising on Three.JS rather than WebGL/GLSL. WebGL/GLSL give the developer a low level api to the graphics hardware. Video decoding apis talk to the hardware video decoding hardware. OPFS gives developers a low level API to the persistent file system / HDD / SSD.
    - modeless 4y ago
      The web platform has tons of very high level stuff in it, much higher level than SQLite, and more is being added. Even specifically on the topic of 3D, they're trying to add a <model> tag and standard 3D model file format to HTML right now. It doesn't make sense to reject Web SQL, which is much more foundational, on the grounds of being too high level. Not now, but even less so back when the decision was made in 2010, when the web itself was all higher level and the lower level APIs you mentioned didn't even exist.
- ilyt 4y ago
  I'd still rather have basic SQL (that is enough for most) to just be there, for the apps where including SQL engine would be vastly more code than app itself. If we had block API and SQL (it could be just a concrete version running from WASM itself, to not bloat browser itself) it could save a lot on app size. App then could look at browser version and decide to use that or to download newest one to run.
- sgbeal 4y ago
  > WebSQL would have been tied to one specific version of SQLite, and developers would have to work to the lowest supported version. There would have been no extension mechanism. Not only that, but WebSQL disables a lot of mundane functionality, like the instr() SQL function and it's impossible to VACUUM because WebSQL requires explicit transactions and VACUUM cannot run in a transaction. i had the "pleasure" of having to work with WebSQL over the past couple of months for purposes of comparing its performance to the new sqlite features, and IMO, WebSQL is, as the kids say today, _weak sauce_. Its API is far too limited.
- croes 4y ago
  Unless frameworks and libraries start depending on it and each one of them uses different extensions. All of a sudden you have multiple slightly different instances running just because there isn't a standard.
yarg 4y ago
Dependency resolution for the web would fix this. It can be done, and needs to be - and more securely than the half-assed efforts of the likes of NPM and Maven. All dependencies signed - let's encrypt has made this a viable option. I don't see how there's anything approximating an unresolvable privacy concern here.
- coder543 4y ago
  Signing dependencies is related to integrity, which is orthogonal to privacy. Sharing a cache between websites has proven to be a privacy issue. Read more here: https://developer.chrome.com/en/blog/http-cache-partitioning/ https://developer.chrome.com/en/blog/http-cache-partitioning... or here: https://www.peakhour.io/blog/cache-partitioning-firefox-chrome/ https://www.peakhour.io/blog/cache-partitioning-firefox-chro...
  - yarg 4y ago
    As per your second link: > This means if you visited a website and it loaded the resource: > https://www.somesite.com/foo.js https://www.somesite.com/foo.js > and you then visited a second website, and it also included the same resource, then the resource would be loaded from the shared cache rather than being downloaded from the internet a second time. Cookies set by these resources would also be shared. The privacy problem is not a result of the shared dependency, it's a result of the shared cookies. Yes, if you share the execution space between multiple programs running the same lib, there's a privacy concern. No shit - don't fucking do that.
    - coder543 4y ago
      > The privacy problem is not a result of the shared dependency, it's a result of the shared cookies. No, it's not the result of the shared cookies. You just ignored all of the timing attacks and fingerprinting which a shared cache allows, as those articles discuss. The cookie thing is honestly completely irrelevant to the topic of privacy, if you understand how the shared cache used to work. If you loaded the same library from separate CDNs on different websites, the shared cache didn't come into play at all. The library was loaded twice anyways. There was no chance for cookies from different CDNs to accidentally cross the streams. Browsers weren't attempting to heuristically determine if you were trying to load the same asset from different hosts. The shared cache only came into play if you loaded the same asset from the same third-party CDN on multiple websites. The host serving an asset is the one responsible for setting the cookies, and sharing the cookies is helpful in that case, since the same CDN is the host serving the asset to the browser for both websites. These aren't cookies controlled by separate websites, they're cookies supplied by the CDN, and they should be the same for both requests anyways, outside of maybe some remote possibility of a theoretical attack involving a malicious CDN intentionally setting some weird request-dependent cookies, but I can't see how that would even do anything harmful anyways. So, the cookies get shared because the browser is serving a cached response for the same asset from the same host to both sites, which makes sense. So, cookies aren't the problem here, and they're not the reason the shared cache was partitioned. If you want browsers to undo that in any form, you would have to solve the actual privacy problems here.
Beltalowda 4y ago
> It has to be downloaded and JIT compiled separately for every site that uses it I don't really see the problem with that. Looking at the sql.js demo[1] the WASM binary is 610K (305K compressed/transferred), and it seems to run pretty fast even on my slow laptop. > Maybe if half the sites on the web start using it Most websites have no reason to use it; simple key/value localStorage is enough for many sites or apps that need some sort of storage. It's kind of a niche thing. Many regular desktop applications have no need SQLite, either.
- wruza 4y ago
  It’s an ubiquitous need for any app that has a dynamic collection view. You can’t take all sites, see that 99% of them are static documents developed as degenerate apps and then conclude that “most apps” have no reason to use it. It’s like saying that most cats are on jpegs, so cat food is kind of a niche thing for a cat owner.
  - Beltalowda 4y ago
    > “most apps” Please do not put words in quotes as if I said them when, in fact, I did not. Thank you.
    - wruza 4y ago
      What would that trivial correction change, in your opinion?
- ilyt 4y ago
  > I don't really see the problem with that. Looking at the sql.js demo[1] the WASM binary is 610K (305K compressed/transferred), and it seems to run pretty fast even on my slow laptop. ...and that is small by your standards ?
  - Beltalowda 4y ago
    It's not small; it's also not huge. For the type of website that has use for this it's not really all that much.

thruflo 4y ago

Interesting — so basically an overhead until it’s fully native? I was reading the Chrome initiative as a pathway to native. Would that mean it can’t be WASM?

modeless 4y ago
That tweet is not implying that it will ever be native. Quite the opposite; Web SQL is native today and they are going to remove it since it was rejected by other browsers.

samwillis 4y ago

I disagree, this is much better than WebSQL. WebSQL would have been tied to one specific version of SQLite, and developers would have to work to the lowest supported version. There would have been no extension mechanism. WASM SQLite is the correct solution. It’s extendable by the developer using it, they can uses SQLite extension modules, and build their own. But almost more so it proves the idea of a WASM db engine backed by a low level block FS api. We will see other db engines uses this architecture. DuckDB have already done it. I’m sure MongoDBs Realm and CouchBase Mobile will do the same soon too. We are in for an exciting time in the next few years.

modeless 4y ago
> WebSQL would have been tied to one specific version of SQLite, and developers would have to work to the lowest supported version. There would have been no extension mechanism. Why not? Web SQL would not be different from WebGL/GLSL or even JavaScript itself in that respect. Developers use feature detection and and work to the lowest supported version. APIs and languages can evolve, but in (mostly) backwards compatible ways. Extension and versioning mechanisms can be made. That's how the web works and Web SQL could have worked that way too. More broadly, you could apply arguments like this against everything in the entire web platform. Maybe everything should be a WASM module that developers could choose themselves! Image loading, video decoding, font rendering, DOM, JS engine, why not? It's actually a beautiful vision and I'd be all for it if not for cache partitioning. Every site would have to re-download and re-JIT an entire browser engine before it could do anything. The base platform needs to include a diverse set of commonly used features so that apps don't have to download the world, and on a list of ubiquitous libraries SQLite is right up there with other libraries backing the web platform, like zlib.
- samwillis 4y ago
  WebGL/GLSL is a low level API the equivalent of the OPFS api. Standardising WebSQL would be like browsers standardising on Three.JS rather than WebGL/GLSL. WebGL/GLSL give the developer a low level api to the graphics hardware. Video decoding apis talk to the hardware video decoding hardware. OPFS gives developers a low level API to the persistent file system / HDD / SSD.
  - modeless 4y ago
    The web platform has tons of very high level stuff in it, much higher level than SQLite, and more is being added. Even specifically on the topic of 3D, they're trying to add a <model> tag and standard 3D model file format to HTML right now. It doesn't make sense to reject Web SQL, which is much more foundational, on the grounds of being too high level. Not now, but even less so back when the decision was made in 2010, when the web itself was all higher level and the lower level APIs you mentioned didn't even exist.
ilyt 4y ago
I'd still rather have basic SQL (that is enough for most) to just be there, for the apps where including SQL engine would be vastly more code than app itself. If we had block API and SQL (it could be just a concrete version running from WASM itself, to not bloat browser itself) it could save a lot on app size. App then could look at browser version and decide to use that or to download newest one to run.
sgbeal 4y ago
> WebSQL would have been tied to one specific version of SQLite, and developers would have to work to the lowest supported version. There would have been no extension mechanism. Not only that, but WebSQL disables a lot of mundane functionality, like the instr() SQL function and it's impossible to VACUUM because WebSQL requires explicit transactions and VACUUM cannot run in a transaction. i had the "pleasure" of having to work with WebSQL over the past couple of months for purposes of comparing its performance to the new sqlite features, and IMO, WebSQL is, as the kids say today, _weak sauce_. Its API is far too limited.
croes 4y ago
Unless frameworks and libraries start depending on it and each one of them uses different extensions. All of a sudden you have multiple slightly different instances running just because there isn't a standard.

yarg 4y ago

Dependency resolution for the web would fix this. It can be done, and needs to be - and more securely than the half-assed efforts of the likes of NPM and Maven. All dependencies signed - let's encrypt has made this a viable option. I don't see how there's anything approximating an unresolvable privacy concern here.

coder543 4y ago
Signing dependencies is related to integrity, which is orthogonal to privacy. Sharing a cache between websites has proven to be a privacy issue. Read more here: https://developer.chrome.com/en/blog/http-cache-partitioning/ https://developer.chrome.com/en/blog/http-cache-partitioning... or here: https://www.peakhour.io/blog/cache-partitioning-firefox-chrome/ https://www.peakhour.io/blog/cache-partitioning-firefox-chro...
- yarg 4y ago
  As per your second link: > This means if you visited a website and it loaded the resource: > https://www.somesite.com/foo.js https://www.somesite.com/foo.js > and you then visited a second website, and it also included the same resource, then the resource would be loaded from the shared cache rather than being downloaded from the internet a second time. Cookies set by these resources would also be shared. The privacy problem is not a result of the shared dependency, it's a result of the shared cookies. Yes, if you share the execution space between multiple programs running the same lib, there's a privacy concern. No shit - don't fucking do that.
  - coder543 4y ago
    > The privacy problem is not a result of the shared dependency, it's a result of the shared cookies. No, it's not the result of the shared cookies. You just ignored all of the timing attacks and fingerprinting which a shared cache allows, as those articles discuss. The cookie thing is honestly completely irrelevant to the topic of privacy, if you understand how the shared cache used to work. If you loaded the same library from separate CDNs on different websites, the shared cache didn't come into play at all. The library was loaded twice anyways. There was no chance for cookies from different CDNs to accidentally cross the streams. Browsers weren't attempting to heuristically determine if you were trying to load the same asset from different hosts. The shared cache only came into play if you loaded the same asset from the same third-party CDN on multiple websites. The host serving an asset is the one responsible for setting the cookies, and sharing the cookies is helpful in that case, since the same CDN is the host serving the asset to the browser for both websites. These aren't cookies controlled by separate websites, they're cookies supplied by the CDN, and they should be the same for both requests anyways, outside of maybe some remote possibility of a theoretical attack involving a malicious CDN intentionally setting some weird request-dependent cookies, but I can't see how that would even do anything harmful anyways. So, the cookies get shared because the browser is serving a cached response for the same asset from the same host to both sites, which makes sense. So, cookies aren't the problem here, and they're not the reason the shared cache was partitioned. If you want browsers to undo that in any form, you would have to solve the actual privacy problems here.

Beltalowda 4y ago

> It has to be downloaded and JIT compiled separately for every site that uses it I don't really see the problem with that. Looking at the sql.js demo[1] the WASM binary is 610K (305K compressed/transferred), and it seems to run pretty fast even on my slow laptop. > Maybe if half the sites on the web start using it Most websites have no reason to use it; simple key/value localStorage is enough for many sites or apps that need some sort of storage. It's kind of a niche thing. Many regular desktop applications have no need SQLite, either.

wruza 4y ago
It’s an ubiquitous need for any app that has a dynamic collection view. You can’t take all sites, see that 99% of them are static documents developed as degenerate apps and then conclude that “most apps” have no reason to use it. It’s like saying that most cats are on jpegs, so cat food is kind of a niche thing for a cat owner.
- Beltalowda 4y ago
  > “most apps” Please do not put words in quotes as if I said them when, in fact, I did not. Thank you.
  - wruza 4y ago
    What would that trivial correction change, in your opinion?
ilyt 4y ago
> I don't really see the problem with that. Looking at the sql.js demo[1] the WASM binary is 610K (305K compressed/transferred), and it seems to run pretty fast even on my slow laptop. ...and that is small by your standards ?
- Beltalowda 4y ago
  It's not small; it's also not huge. For the type of website that has use for this it's not really all that much.