20 ms·

SQLite builds for WASI since 3.41.0

111 points by rapnie 3y ago

aargh_aargh 3y ago
What would you use this for? I don't get the whole picture and the article doesn't talk about it.
- samwillis 3y ago
  So far the official WASM support from the SQLite project have only targeted WASM in the browser with a JS API. If I follow correctly this is about working with the SQLite team to upstream WASI support, which will eventually enable official support for it, and hence WASM on the server/cloud/edge and other WASI complement runners.
  - ridruejo 3y ago
    Correct!
  - sgbeal 3y ago
    > If I follow correctly this is about working with the SQLite team to upstream WASI support, which will eventually enable official support for it, and hence WASM on the server/cloud/edge and other WASI complement runners. (sqlite team's Wasm/JS guy here) That's the long and the short if it. We currently _actively_ target browsers only because our team has exceedingly limited bandwidth and has to be choosy with regards to where our dev time goes. Long-term, we intend to target all wasm platforms, but getting there is an ongoing process.
- cfn 3y ago
  The file of my desktop app is a SQLite db in disguise, I could use this to open these files in a web version of my app, I suppose. I know there is a javascript implementation of SQLite out there but maybe this has better performance.
- jraph 3y ago
  Apparently they use it for their tests. > We wanted to contribute upstream our patches for SQLite, as we were heavily relying on libsqlite for our tests of PHP.wasm with WordPress and Drupal.
- rcarmo 3y ago
  Well, you can use it to access SQLite files in WASM runners, of course. So you get a portable, platform-independent piece of code that can run anywhere. Although it was a bit of a surprise to see that they started out with the toy CLI tool that my self and Nicholas hacked on for a weekend (I wanted to be able to run SQLite commands in a-Shell, which is an excellent iPad CLI environment).
- fastball 3y ago
  The goal of WASI is to add a standardized system API to WebAssembly, allowing you to access underlying systems (filesystem, networking, etc) in a homomorphic and secure way. In practice, this means that you can compile your Rust/C++/etc code to WASM + WASI, and that code will run anywhere you have a WebAssembly VM. So the same code that accesses the "filesystem" in a browser context can also access the filesystem in a desktop/mobile/edge/etc environment. You could also think of it as an attempt to turn WebAssembly into a fully-fledged JVM alternative. Compile once, run anywhere (including browser). For this specifically, it means that SQLite can integrate with WASI APIs (instead of only Web APIs as before) so that an SQLite Wasm build can run in other contexts besides browsers.
  - aargh_aargh 3y ago
    What I don't get specifically is what's being implemented here with SQLite. Assuming WASI is an interface an application can call into for things like filesystem access, sockets, does this add SQLite as interface (analogously to sqlite.h) next to these as part of WASI? So SQLite will be "under" the interface, i.e. part of a standard runtime? How much "batteries included" is this runtime (planned to be)? If I understood this correctly, then there would be a tradeoff with having more batteries included which would imply more functionality for applications but make porting to new targets more work.
    - angelmm 3y ago
      The main work behind this patch is to ensure that SQLite can compile to Wasm32-wasi. WASI doesn't offer all syscalls that SQLite may require, so it may require some conditional definitions and code to ensure it works properly. It doesn't mean that SQLite is part of WASI, but you can embed SQLite in a Wasm32-wasi module. In the future, the WASI standard will include more and more features. This will allow SQLite to enable more internal features that are currently skipped due to the limitations on WASI.
      - aargh_aargh 3y ago
        Ah, so it's SQLite running on top of WASI, as a user (application, or rather library). Thanks for clearing that up for me.
- angelmm 3y ago
  (Wasm Labs team member here) SQLite is a pretty popular database and it's a critical dependency for many different applications. By compiling it to Wasm32-wasi, you can add it to any WebAssembly module. This enables a new set of possibilities for Wasm and SQLite. For example, now you can run a full WordPress application in the browser [1][2] / server [3] using the same Wasm module. Note that for the browser these projects use Emscripten [4], but in the future the same Wasm32-wasi module will work. A teammate gave a lightning talk about SQLite and WASI at KubeCon EU [5]. In general, any environment that includes a wasm runtime can potentially run applications that uses SQLite under the hood. Before, it wasn't possible. - [1] https://wordpress.wasmlabs.dev/ https://wordpress.wasmlabs.dev/ - [2] https://developer.wordpress.org/playground/demo/ https://developer.wordpress.org/playground/demo/ - [3] https://wasmlabs.dev/articles/running-wordpress-with-mod-wasm/ https://wasmlabs.dev/articles/running-wordpress-with-mod-was... - [4] https://emscripten.org/ https://emscripten.org/ - [5] https://www.youtube.com/watch?v=E7tWtgf9V2s https://www.youtube.com/watch?v=E7tWtgf9V2s
  - i2cmaster 3y ago
    I guess that's one way to prevent malicious plugins from destroying your site.
    - gzurl 3y ago
      Yes, exactly. We wrote an article here regarding PHP on Wasm that covers that: https://wasmlabs.dev/articles/mitigating-php-vulnerabilities-with-webassembly/ https://wasmlabs.dev/articles/mitigating-php-vulnerabilities...
    - candiodari 3y ago
      It does what WASM is meant to do: it allows extra functionality, such as a database, without exposing more surface area for exploits.
    - remram 3y ago
      They can still destroy your site, they just can't destroy anything else on the same server.
raybb 3y ago
> For legal reasons, we could not contribute directly (their patches) What were the legal reasons?
- foota 3y ago
  Looks like this: https://www.sqlite.org/copyright.html https://www.sqlite.org/copyright.html Kind of weird, couldn't they use a CLA?
  - jraph 3y ago
    It's not enough to guarantee the code has not been copied from somewhere / can be released in the public domain. The only way to have this guarantee is to write it yourself.
    - foota 3y ago
      I suppose that's true, but that's no different from any other project, no? If I write a contribution to React that wasn't actually mine, it doesn't matter that it's just open source and not being released into the public domain, it's still a violation of the real author's copyright. So if they've chosen a different risk model than essentially every other company doing open source software then that's up to them, but I'm not sure the need for it follows from it being public domain?
      - jraph 3y ago
        > I'm not sure the need for it follows from it being public domain? There's something specific about the public domain that in many places, you can't just decide to release something in the public domain. I guess that's why they actually accept contribution but require an affidavit dedicating the contribution into the public domain. IIUC an affidavit specifically needs to be witnessed by taker of oaths, which probably ensures that the work can indeed be put in public domain. Not sure it's the whole story, or even the main reason, or the correct analyze, though.
jraph 3y ago
> For legal reasons, we could not contribute directly, but we could discuss the required changes with someone from the SQLite team and then iterate with them by testing or commenting. Ah, here's the catch! I knew SQLite didn't accept code contributions, I was suprised they managed to contribute anyway, and then this sentence clarified the situation. So they indeed did not manage to contribute their code itself, but the SQLite team agreed to write this code themselves. I wonder what was the SQLite team's motivation behind this cooperative work.
- jillesvangurp 3y ago
  https://www.sqlite.org/copyright.html https://www.sqlite.org/copyright.html To summarize, instead of using one of the OSS licenses, the copyright holders simply declare the source to be in the public domain. In order to preserve that status they don't accept patches unless you submit some signed document that you agree with that. To make things more complicated, they also use their a relatively niche version management system instead of git. Which would complicate making contributions (if they accepted them). There's a popular fork that fixes all of these issues: https://github.com/libsql/libsql https://github.com/libsql/libsql It is MIT licensed, on Github, and open for contributions. Kind of a weird legal situation for a popular project like this that so many people depend on to have. Not judging; but it is odd. Seems like a lot of wasted efforts between users, would be contributors, and the people that forked this thing to address all that.
  - jraph 3y ago
    > To summarize, instead of using one of the OSS licenses, the copyright holders simply declare the source to be in the public domain. In order to preserve that status they don't accept patches unless you submit some signed document that you agree with that. Yep, see also this discussion about it in this thread https://news.ycombinator.com/item?id=36054521#36055014 https://news.ycombinator.com/item?id=36054521#36055014
  - mehdix 3y ago
    Aren't all those potential limitations, in fact their strengths? They are eating their own dogfood, which works for them. That sounds great to me.
sureglymop 3y ago
This is awesome. What I would really love to see is pandoc compiled to wasm and usable in a browser context. I tried before but since I'm unfamiliar with Haskell i didn't get very far.
- gzurl 3y ago
  I didn't know about pandoc. Really useful!
ncruces 3y ago
I think this should come with the disclaimer that this completely eschews synchronization mechanisms used by SQLite. Using this to read from a DB concurrently with other writers will produce garbage results. Using this to write to a DB concurrently with other writers will corrupt your DB. WASI does not expose file locking primitives, and the unix VFS simply stubs them out in WASI builds: https://github.com/sqlite/sqlite/blob/4e8e33ba84e253878797d7f4443810387cb87469/src/os_unix.c#L200 https://github.com/sqlite/sqlite/blob/4e8e33ba84e253878797d7...

111 points by rapnie 3y ago

aargh_aargh 3y ago

What would you use this for? I don't get the whole picture and the article doesn't talk about it.

samwillis 3y ago
So far the official WASM support from the SQLite project have only targeted WASM in the browser with a JS API. If I follow correctly this is about working with the SQLite team to upstream WASI support, which will eventually enable official support for it, and hence WASM on the server/cloud/edge and other WASI complement runners.
- ridruejo 3y ago
  Correct!
- sgbeal 3y ago
  > If I follow correctly this is about working with the SQLite team to upstream WASI support, which will eventually enable official support for it, and hence WASM on the server/cloud/edge and other WASI complement runners. (sqlite team's Wasm/JS guy here) That's the long and the short if it. We currently _actively_ target browsers only because our team has exceedingly limited bandwidth and has to be choosy with regards to where our dev time goes. Long-term, we intend to target all wasm platforms, but getting there is an ongoing process.
cfn 3y ago
The file of my desktop app is a SQLite db in disguise, I could use this to open these files in a web version of my app, I suppose. I know there is a javascript implementation of SQLite out there but maybe this has better performance.
jraph 3y ago
Apparently they use it for their tests. > We wanted to contribute upstream our patches for SQLite, as we were heavily relying on libsqlite for our tests of PHP.wasm with WordPress and Drupal.
rcarmo 3y ago
Well, you can use it to access SQLite files in WASM runners, of course. So you get a portable, platform-independent piece of code that can run anywhere. Although it was a bit of a surprise to see that they started out with the toy CLI tool that my self and Nicholas hacked on for a weekend (I wanted to be able to run SQLite commands in a-Shell, which is an excellent iPad CLI environment).
fastball 3y ago
The goal of WASI is to add a standardized system API to WebAssembly, allowing you to access underlying systems (filesystem, networking, etc) in a homomorphic and secure way. In practice, this means that you can compile your Rust/C++/etc code to WASM + WASI, and that code will run anywhere you have a WebAssembly VM. So the same code that accesses the "filesystem" in a browser context can also access the filesystem in a desktop/mobile/edge/etc environment. You could also think of it as an attempt to turn WebAssembly into a fully-fledged JVM alternative. Compile once, run anywhere (including browser). For this specifically, it means that SQLite can integrate with WASI APIs (instead of only Web APIs as before) so that an SQLite Wasm build can run in other contexts besides browsers.
- aargh_aargh 3y ago
  What I don't get specifically is what's being implemented here with SQLite. Assuming WASI is an interface an application can call into for things like filesystem access, sockets, does this add SQLite as interface (analogously to sqlite.h) next to these as part of WASI? So SQLite will be "under" the interface, i.e. part of a standard runtime? How much "batteries included" is this runtime (planned to be)? If I understood this correctly, then there would be a tradeoff with having more batteries included which would imply more functionality for applications but make porting to new targets more work.
  - angelmm 3y ago
    The main work behind this patch is to ensure that SQLite can compile to Wasm32-wasi. WASI doesn't offer all syscalls that SQLite may require, so it may require some conditional definitions and code to ensure it works properly. It doesn't mean that SQLite is part of WASI, but you can embed SQLite in a Wasm32-wasi module. In the future, the WASI standard will include more and more features. This will allow SQLite to enable more internal features that are currently skipped due to the limitations on WASI.
    - aargh_aargh 3y ago
      Ah, so it's SQLite running on top of WASI, as a user (application, or rather library). Thanks for clearing that up for me.
angelmm 3y ago
(Wasm Labs team member here) SQLite is a pretty popular database and it's a critical dependency for many different applications. By compiling it to Wasm32-wasi, you can add it to any WebAssembly module. This enables a new set of possibilities for Wasm and SQLite. For example, now you can run a full WordPress application in the browser [1][2] / server [3] using the same Wasm module. Note that for the browser these projects use Emscripten [4], but in the future the same Wasm32-wasi module will work. A teammate gave a lightning talk about SQLite and WASI at KubeCon EU [5]. In general, any environment that includes a wasm runtime can potentially run applications that uses SQLite under the hood. Before, it wasn't possible. - [1] https://wordpress.wasmlabs.dev/ https://wordpress.wasmlabs.dev/ - [2] https://developer.wordpress.org/playground/demo/ https://developer.wordpress.org/playground/demo/ - [3] https://wasmlabs.dev/articles/running-wordpress-with-mod-wasm/ https://wasmlabs.dev/articles/running-wordpress-with-mod-was... - [4] https://emscripten.org/ https://emscripten.org/ - [5] https://www.youtube.com/watch?v=E7tWtgf9V2s https://www.youtube.com/watch?v=E7tWtgf9V2s
- i2cmaster 3y ago
  I guess that's one way to prevent malicious plugins from destroying your site.
  - gzurl 3y ago
    Yes, exactly. We wrote an article here regarding PHP on Wasm that covers that: https://wasmlabs.dev/articles/mitigating-php-vulnerabilities-with-webassembly/ https://wasmlabs.dev/articles/mitigating-php-vulnerabilities...
  - candiodari 3y ago
    It does what WASM is meant to do: it allows extra functionality, such as a database, without exposing more surface area for exploits.
  - remram 3y ago
    They can still destroy your site, they just can't destroy anything else on the same server.

raybb 3y ago

> For legal reasons, we could not contribute directly (their patches) What were the legal reasons?

foota 3y ago
Looks like this: https://www.sqlite.org/copyright.html https://www.sqlite.org/copyright.html Kind of weird, couldn't they use a CLA?
- jraph 3y ago
  It's not enough to guarantee the code has not been copied from somewhere / can be released in the public domain. The only way to have this guarantee is to write it yourself.
  - foota 3y ago
    I suppose that's true, but that's no different from any other project, no? If I write a contribution to React that wasn't actually mine, it doesn't matter that it's just open source and not being released into the public domain, it's still a violation of the real author's copyright. So if they've chosen a different risk model than essentially every other company doing open source software then that's up to them, but I'm not sure the need for it follows from it being public domain?
    - jraph 3y ago
      > I'm not sure the need for it follows from it being public domain? There's something specific about the public domain that in many places, you can't just decide to release something in the public domain. I guess that's why they actually accept contribution but require an affidavit dedicating the contribution into the public domain. IIUC an affidavit specifically needs to be witnessed by taker of oaths, which probably ensures that the work can indeed be put in public domain. Not sure it's the whole story, or even the main reason, or the correct analyze, though.

jraph 3y ago

> For legal reasons, we could not contribute directly, but we could discuss the required changes with someone from the SQLite team and then iterate with them by testing or commenting. Ah, here's the catch! I knew SQLite didn't accept code contributions, I was suprised they managed to contribute anyway, and then this sentence clarified the situation. So they indeed did not manage to contribute their code itself, but the SQLite team agreed to write this code themselves. I wonder what was the SQLite team's motivation behind this cooperative work.

jillesvangurp 3y ago
https://www.sqlite.org/copyright.html https://www.sqlite.org/copyright.html To summarize, instead of using one of the OSS licenses, the copyright holders simply declare the source to be in the public domain. In order to preserve that status they don't accept patches unless you submit some signed document that you agree with that. To make things more complicated, they also use their a relatively niche version management system instead of git. Which would complicate making contributions (if they accepted them). There's a popular fork that fixes all of these issues: https://github.com/libsql/libsql https://github.com/libsql/libsql It is MIT licensed, on Github, and open for contributions. Kind of a weird legal situation for a popular project like this that so many people depend on to have. Not judging; but it is odd. Seems like a lot of wasted efforts between users, would be contributors, and the people that forked this thing to address all that.
- jraph 3y ago
  > To summarize, instead of using one of the OSS licenses, the copyright holders simply declare the source to be in the public domain. In order to preserve that status they don't accept patches unless you submit some signed document that you agree with that. Yep, see also this discussion about it in this thread https://news.ycombinator.com/item?id=36054521#36055014 https://news.ycombinator.com/item?id=36054521#36055014
- mehdix 3y ago
  Aren't all those potential limitations, in fact their strengths? They are eating their own dogfood, which works for them. That sounds great to me.

sureglymop 3y ago

This is awesome. What I would really love to see is pandoc compiled to wasm and usable in a browser context. I tried before but since I'm unfamiliar with Haskell i didn't get very far.

gzurl 3y ago
I didn't know about pandoc. Really useful!

ncruces 3y ago

I think this should come with the disclaimer that this completely eschews synchronization mechanisms used by SQLite. Using this to read from a DB concurrently with other writers will produce garbage results. Using this to write to a DB concurrently with other writers will corrupt your DB. WASI does not expose file locking primitives, and the unix VFS simply stubs them out in WASI builds: https://github.com/sqlite/sqlite/blob/4e8e33ba84e253878797d7f4443810387cb87469/src/os_unix.c#L200 https://github.com/sqlite/sqlite/blob/4e8e33ba84e253878797d7...