7 ms·

I'm not very familiar with Deno but I wonder how deno compile compares to pkg when it comes to native modules. One of my gripes with PKG (and all other node.js

I'm not very familiar with Deno but I wonder how deno compile compares to pkg when it comes to native modules. One of my gripes with PKG (and all other node.js packaging tools) has been that it's a pain in the ass to package when your dependency includes a native module, for example SQLite. Since Deno has a different architecture and works in different ways, thought I would ask just in case there's a solution for this on Deno, that would be great.

j1elo 6y ago
Yes, packaging a binary dependency can be problematic. I guess it depends on how easy the library puts it to import the binary in a compatible way (with the mechanisms that are expected by pkg)
miohtama 6y ago
WebAssembly fixes this.
- zubairq 6y ago
  Not really as the WASM version of Sqlite is memory only and cannot write to disk. Please feel free to correct me if this has changed though
  - tracker1 6y ago
    Nope... the wasm fs adapters don't allow the flexibility needed to really operate correctly. It might be best to try alternatives like wrappers around leveldb as a baseline, which of course isn't sql and has its' own drawbacks.
    - kevin_thibedeau 6y ago
      SQLite still has a workaround for locking on Win95. That could be adapted for wasm. https://github.com/sqlite/sqlite/blob/610f11de25993960ff616edde0f5dc47d1c0b56e/src/os.h#L94 https://github.com/sqlite/sqlite/blob/610f11de25993960ff616e...
      - tracker1 6y ago
        In another comment I mention, that I often use a desktop sqlite client to actively connect to the operation sqlite database an application may be running against. Wasm will not support this, even with the workaround above.
  - greggman3 6y ago
    I don't know all the issues but AFAICT wasm is just executing code. The libraries you provide for it are up to you. So if you so, for example, thinking of node, if I expose `fs.openSync` `fs.readSync`, `fs.writeSync` and `fs.closeSync` what's missing that wasm can't write to disk?
  - miohtama 6y ago
    WASI fixes this https://wasi.dev/ https://wasi.dev/
- tracker1 6y ago
  webassembly doesn't support file locking for sqlite databases that would allow for other processes to interact with the same database your application uses. It's actually one of my biggest gripes. You can use it in-memory and save the output or use a different database approach, and lose the features of sqlite. Of course, a different kind of database, similar to say leveldb could work in single-process mode.
  - Sephr 6y ago
    WebAssembly supports threads and shared memory, which should make it possible to implement software-based locks in the meantime. I definitely see the issue that you're bringing up, especially wrt external interoperability.
    - tracker1 6y ago
      For me, I often have a sqlite client connected to my active database while my application runs to run adhoc queries to check application's state/structure... with a wasm implementation, that is simply not possible.
    - seg_lol 6y ago
      We need a file system abstraction implemented in a webworker. Do you know if it or something similar exists?

j1elo 6y ago

Yes, packaging a binary dependency can be problematic. I guess it depends on how easy the library puts it to import the binary in a compatible way (with the mechanisms that are expected by pkg)

miohtama 6y ago

WebAssembly fixes this.

zubairq 6y ago
Not really as the WASM version of Sqlite is memory only and cannot write to disk. Please feel free to correct me if this has changed though
- tracker1 6y ago
  Nope... the wasm fs adapters don't allow the flexibility needed to really operate correctly. It might be best to try alternatives like wrappers around leveldb as a baseline, which of course isn't sql and has its' own drawbacks.
  - kevin_thibedeau 6y ago
    SQLite still has a workaround for locking on Win95. That could be adapted for wasm. https://github.com/sqlite/sqlite/blob/610f11de25993960ff616edde0f5dc47d1c0b56e/src/os.h#L94 https://github.com/sqlite/sqlite/blob/610f11de25993960ff616e...
    - tracker1 6y ago
      In another comment I mention, that I often use a desktop sqlite client to actively connect to the operation sqlite database an application may be running against. Wasm will not support this, even with the workaround above.
- greggman3 6y ago
  I don't know all the issues but AFAICT wasm is just executing code. The libraries you provide for it are up to you. So if you so, for example, thinking of node, if I expose `fs.openSync` `fs.readSync`, `fs.writeSync` and `fs.closeSync` what's missing that wasm can't write to disk?
- miohtama 6y ago
  WASI fixes this https://wasi.dev/ https://wasi.dev/
tracker1 6y ago
webassembly doesn't support file locking for sqlite databases that would allow for other processes to interact with the same database your application uses. It's actually one of my biggest gripes. You can use it in-memory and save the output or use a different database approach, and lose the features of sqlite. Of course, a different kind of database, similar to say leveldb could work in single-process mode.
- Sephr 6y ago
  WebAssembly supports threads and shared memory, which should make it possible to implement software-based locks in the meantime. I definitely see the issue that you're bringing up, especially wrt external interoperability.
  - tracker1 6y ago
    For me, I often have a sqlite client connected to my active database while my application runs to run adhoc queries to check application's state/structure... with a wasm implementation, that is simply not possible.
  - seg_lol 6y ago
    We need a file system abstraction implemented in a webworker. Do you know if it or something similar exists?