7 ms·

The thing I don't get here is that literally every browser already uses Sqlite3 for a million things, why are we not just finally acknowledging that, yes, this

The thing I don't get here is that literally every browser already uses Sqlite3 for a million things, why are we not just finally acknowledging that, yes, this very specific library is a universal good, just like gzip compression, or the jpeg image format, and here's the JS API for directly working with it? Why do we need a separate WASM version of something that's already built right into every single browser? Why is there no all-browser-vendor-blessed `Sqlite3` global?

easton 4y ago
They tried that at one point, I think it was called WebSQL? But it was decided that they didn’t want to make the quirks of sqlite3 a web standard since it’d be hard to do a clean room implementation.
- TheRealPomax 4y ago
  Except I don't want WebSQL, I literally want Sqlite3 as a JS API, backed by the unadulterated authoritative Sqlite3 library. I don't wonder why there's no good in-browser database solution that uses SQL, I literally don't get why sqlite3, only, and specifically, still isn't usable by JS in 2022.
  - deleted 4y ago
    [deleted]
  - sgbeal 4y ago
    > I literally don't get why sqlite3, only, and specifically, still isn't usable by JS in 2022. It is now :). We provide 4 separate APIs, from the lowest-level 1-to-1 C-via-WASM bindings to one quite similar to sql.js, plus all of the low-level pieces necessary to create your own. Historical note: those of us within the sqlite project had never paid any attention to the unfortunately-named WebAssembly (which has been dubbed "neither web nor assembly") because it didn't seem to hold any relevance for us. It wasn't until April-ish 2022 that we took a look at it, and have been working on "officially" bringing it to the browser world ever since. Even so, folks have been producing WASM bindings of it for a number of years now, and the relative ease of doing so (sqlite3.c requires no changes whatsoever to compile to WASM) is possibly (i opine) why we didn't get requests from users to do this sooner. Sidebar: someone is going to ask "if it's so easy, why did you need 6 months to get it out the door?" Fair question: we had some very specific technical goals which i'm not at liberty to elaborate on, and only a single developer to put on it. Plus, WASM was a completely new tech for us, so there was much learning and experimentation involved.
    - TheRealPomax 4y ago
      No, WASM applications aren't a built-in universal web API, they are a huge payload that needs to be sent, anew (thanks to per-site caching in modern browsers), that solves the problem I don't understand exists in the first place. It's great that you went "what if we compile to WASM?" but this is something that's not on you: this is something that should have been on browser vendors to just _expose_ because every browser ships with sqlite baked in already. Every user already has it, they just can't use it.
      - sgbeal 4y ago
        > ... but this is something that's not on you: this is something that should have been on browser vendors to just _expose_ because every browser ships with sqlite baked in already. Richard (the sqlite lead) has often described his definition of "freedom" as "being able to take care of yourself," a philosophy he lives and breathes with his software. The sqlite project providing wasm builds for folks, and the materials they need to make their own custom builds, is directly in line with that. Depending on every browser vendor to play along in sync is, quite frankly, a lost cause. > ... they are a huge payload ... If you truly believe that 500kb of content is "huge" nowadays, i challenge you to go watch (via the browser dev tools) how much stuff your favorite websites are serving. (HN, of course, is a spartan exception to the rule.) Hit any given news or social media site and you'll get at least a meg of content, most of which is constantly replaced (so caching it is of little use). Hit IMDB and you'll get 2+mb (compressed). Hit GDrive and you'll get 4.75mb compressed (nearly 15mb uncompressed!). i just hit www.google.com, which has a long history of minimalism, and it transferred 881kb (2.21mb uncompressed). By comparison, 500kb-1mb (uncompressed) isn't even worthy of honorable mention.
        TheRealPomax 4y ago
        > Depending on every browser vendor to play along in sync is, quite frankly, a lost cause. Very true, but that is not mutually exclusive with not understanding why it's a lost cause. There's literally three browser vendors, and of those, really only one of them needs to go "fuck it, sqlite now has an API" and the other two kind of don't have a lot of choice but to follow suit. It's a Chrome world right now. We could have easily done this, we chose not to. Why? (and you probably don't have the answer to that. I don't know if anyone does)
- wruza 4y ago
  And that “clean room” implementation would be done by whom? By someone who makes stable, compatible, 100% battle-tested embedded SQL engines every saturday morning? There are plenty of teams to choose from: End of list.
sedatk 4y ago
W3C tried this but it failed due to lack of independent implementations[1]: > This document was on the W3C Recommendation track but specification work has stopped. The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path. [1] https://www.w3.org/TR/webdatabase/ https://www.w3.org/TR/webdatabase/
- TheRealPomax 4y ago
  See, that's the insanity: why would we need an independent implementation? sqlite has authoritative libraries, that everyone uses, and sqlite3 has been stable for literally over a decade. For all that time, the world could have benefited from "just bloody expose it" as opposed to "but there's no API-equivalent alternative implementations!".
  - googlryas 4y ago
    I assume due to Hyrum's Law. Any quirks/oddities in sqlite would become the de facto standard.
    - TillE 4y ago
      Sure but the idea is that the standard is "just embed SQLite". Nobody has to pretend it's a generic database interface or anything.
      - masklinn 4y ago
        “The standard is a third party project we have no view or control over” is not a standard.
    - wruza 4y ago
      We all know that 99.99% sites that need a client-side database will use $subj or its derivative regardless of these kicks and screams. Because there is no decent 300kb RDBMS which could be ported into a browser. All that was achieved by that decision was to put the whole industry on hold for more than ten years.

easton 4y ago

They tried that at one point, I think it was called WebSQL? But it was decided that they didn’t want to make the quirks of sqlite3 a web standard since it’d be hard to do a clean room implementation.

TheRealPomax 4y ago
Except I don't want WebSQL, I literally want Sqlite3 as a JS API, backed by the unadulterated authoritative Sqlite3 library. I don't wonder why there's no good in-browser database solution that uses SQL, I literally don't get why sqlite3, only, and specifically, still isn't usable by JS in 2022.
- deleted 4y ago
  [deleted]
- sgbeal 4y ago
  > I literally don't get why sqlite3, only, and specifically, still isn't usable by JS in 2022. It is now :). We provide 4 separate APIs, from the lowest-level 1-to-1 C-via-WASM bindings to one quite similar to sql.js, plus all of the low-level pieces necessary to create your own. Historical note: those of us within the sqlite project had never paid any attention to the unfortunately-named WebAssembly (which has been dubbed "neither web nor assembly") because it didn't seem to hold any relevance for us. It wasn't until April-ish 2022 that we took a look at it, and have been working on "officially" bringing it to the browser world ever since. Even so, folks have been producing WASM bindings of it for a number of years now, and the relative ease of doing so (sqlite3.c requires no changes whatsoever to compile to WASM) is possibly (i opine) why we didn't get requests from users to do this sooner. Sidebar: someone is going to ask "if it's so easy, why did you need 6 months to get it out the door?" Fair question: we had some very specific technical goals which i'm not at liberty to elaborate on, and only a single developer to put on it. Plus, WASM was a completely new tech for us, so there was much learning and experimentation involved.
  - TheRealPomax 4y ago
    No, WASM applications aren't a built-in universal web API, they are a huge payload that needs to be sent, anew (thanks to per-site caching in modern browsers), that solves the problem I don't understand exists in the first place. It's great that you went "what if we compile to WASM?" but this is something that's not on you: this is something that should have been on browser vendors to just _expose_ because every browser ships with sqlite baked in already. Every user already has it, they just can't use it.
    - sgbeal 4y ago
      > ... but this is something that's not on you: this is something that should have been on browser vendors to just _expose_ because every browser ships with sqlite baked in already. Richard (the sqlite lead) has often described his definition of "freedom" as "being able to take care of yourself," a philosophy he lives and breathes with his software. The sqlite project providing wasm builds for folks, and the materials they need to make their own custom builds, is directly in line with that. Depending on every browser vendor to play along in sync is, quite frankly, a lost cause. > ... they are a huge payload ... If you truly believe that 500kb of content is "huge" nowadays, i challenge you to go watch (via the browser dev tools) how much stuff your favorite websites are serving. (HN, of course, is a spartan exception to the rule.) Hit any given news or social media site and you'll get at least a meg of content, most of which is constantly replaced (so caching it is of little use). Hit IMDB and you'll get 2+mb (compressed). Hit GDrive and you'll get 4.75mb compressed (nearly 15mb uncompressed!). i just hit www.google.com, which has a long history of minimalism, and it transferred 881kb (2.21mb uncompressed). By comparison, 500kb-1mb (uncompressed) isn't even worthy of honorable mention.
      - TheRealPomax 4y ago
        > Depending on every browser vendor to play along in sync is, quite frankly, a lost cause. Very true, but that is not mutually exclusive with not understanding why it's a lost cause. There's literally three browser vendors, and of those, really only one of them needs to go "fuck it, sqlite now has an API" and the other two kind of don't have a lot of choice but to follow suit. It's a Chrome world right now. We could have easily done this, we chose not to. Why? (and you probably don't have the answer to that. I don't know if anyone does)
wruza 4y ago
And that “clean room” implementation would be done by whom? By someone who makes stable, compatible, 100% battle-tested embedded SQL engines every saturday morning? There are plenty of teams to choose from: End of list.

sedatk 4y ago

W3C tried this but it failed due to lack of independent implementations[1]: > This document was on the W3C Recommendation track but specification work has stopped. The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path. [1] https://www.w3.org/TR/webdatabase/ https://www.w3.org/TR/webdatabase/

TheRealPomax 4y ago
See, that's the insanity: why would we need an independent implementation? sqlite has authoritative libraries, that everyone uses, and sqlite3 has been stable for literally over a decade. For all that time, the world could have benefited from "just bloody expose it" as opposed to "but there's no API-equivalent alternative implementations!".
- googlryas 4y ago
  I assume due to Hyrum's Law. Any quirks/oddities in sqlite would become the de facto standard.
  - TillE 4y ago
    Sure but the idea is that the standard is "just embed SQLite". Nobody has to pretend it's a generic database interface or anything.
    - masklinn 4y ago
      “The standard is a third party project we have no view or control over” is not a standard.
  - wruza 4y ago
    We all know that 99.99% sites that need a client-side database will use $subj or its derivative regardless of these kicks and screams. Because there is no decent 300kb RDBMS which could be ported into a browser. All that was achieved by that decision was to put the whole industry on hold for more than ten years.