4 ms·

The VM cannot have full TCP/IP stack, so any data research tasks are likely to need a special code paths and support for downloads. No SQL databases, etc.

Klasiaster 5y ago
The VM could have its own TCP/IP stack, possibly with a SLIRP layer for translation of connections to the outside. Internet connectivity can be done by limiting it to AJAX, or forwarding the packets to a proxy (something like http://artemyankov.com/tcp-client-for-browsers/ http://artemyankov.com/tcp-client-for-browsers/), or including a Tor client that connects to a Tor bridge, etc.
- westurner 5y ago
  Is all of that necessary to LD_PRELOAD sockets and tunnel them over WebSockets, WebRTC, etc? So e.g. curl doesn't work without (File System Access API,) local storage && translation of e.g. at least normal curl syscalls to just HTTP/3?
westurner 5y ago
From "Hosting SQLite Databases on GitHub Pages" https://news.ycombinator.com/item?id=28021766 https://news.ycombinator.com/item?id=28021766 https://westurner.github.io/hnlog/#comment-28021766 https://westurner.github.io/hnlog/#comment-28021766 : DuckDB can query [and page] Parquet from GitHub, sql.js-httpvfs, sqltorrent, File System Access API (Chrome only so far; IDK about resource quotas and multi-GB datasets), serverless search with WASM workers https://github.com/phiresky/sql.js-httpvfs https://github.com/phiresky/sql.js-httpvfs : > sql.js is a light wrapper around SQLite compiled with EMScripten for use in the browser (client-side). > This [sql.js-httpvfs] repo is a fork of and wrapper around sql.js to provide a read-only HTTP-Range-request based virtual file system for SQLite. It allows hosting an SQLite database on a static file hoster and querying that database from the browser without fully downloading it. > The virtual file system is an emscripten filesystem with some "smart" logic to accelerate fetching with virtual read heads that speed up when sequential data is fetched. It could also be useful to other applications, the code is in lazyFile.ts. It might also be useful to implement this lazy fetching as an SQLite VFS [*] since then SQLite could be compiled with e.g. WASI SDK without relying on all the emscripten OS emulation.
- westurner 5y ago
  Also, I'm not sure if jupyterlab/jupyterlab-google-drive works in JupyterLite yet? Is it yet possible to save notebooks and other files from JupyterLite running in WASM in the browser to one or more cloud storage providers? https://github.com/jupyterlab/jupyterlab-google-drive/issues/203 https://github.com/jupyterlab/jupyterlab-google-drive/issues... https://github.com/jupyterlite/jupyterlite/issues/464 https://github.com/jupyterlite/jupyterlite/issues/464
marwis 5y ago
StackBlitz' WebContainers have in-browser TCP/IP stack, I think from MirageOS.
- dboreham 5y ago
  How does it get raw ip packets out from inside the browser?
  - westurner 5y ago
    Does it need to reframe packet structs and then fix fragmentation issues, or can it set the initial MSS (because MTU discovery likely won't work quite right) so that it doesn't have to ~shrink and re-fragment tunneled {TCP,} packets? https://en.wikipedia.org/wiki/Maximum_segment_size#MSS_and_MTU https://en.wikipedia.org/wiki/Maximum_segment_size#MSS_and_M...
  - marwis 5y ago
    I think it's only externally exposed as local-only HTTP server, presumably via service worker, so you can open the website you are developing. The IP network is entirely virtual.

Klasiaster 5y ago

The VM could have its own TCP/IP stack, possibly with a SLIRP layer for translation of connections to the outside. Internet connectivity can be done by limiting it to AJAX, or forwarding the packets to a proxy (something like http://artemyankov.com/tcp-client-for-browsers/ http://artemyankov.com/tcp-client-for-browsers/), or including a Tor client that connects to a Tor bridge, etc.

westurner 5y ago
Is all of that necessary to LD_PRELOAD sockets and tunnel them over WebSockets, WebRTC, etc? So e.g. curl doesn't work without (File System Access API,) local storage && translation of e.g. at least normal curl syscalls to just HTTP/3?

westurner 5y ago

From "Hosting SQLite Databases on GitHub Pages" https://news.ycombinator.com/item?id=28021766 https://news.ycombinator.com/item?id=28021766 https://westurner.github.io/hnlog/#comment-28021766 https://westurner.github.io/hnlog/#comment-28021766 : DuckDB can query [and page] Parquet from GitHub, sql.js-httpvfs, sqltorrent, File System Access API (Chrome only so far; IDK about resource quotas and multi-GB datasets), serverless search with WASM workers https://github.com/phiresky/sql.js-httpvfs https://github.com/phiresky/sql.js-httpvfs : > sql.js is a light wrapper around SQLite compiled with EMScripten for use in the browser (client-side). > This [sql.js-httpvfs] repo is a fork of and wrapper around sql.js to provide a read-only HTTP-Range-request based virtual file system for SQLite. It allows hosting an SQLite database on a static file hoster and querying that database from the browser without fully downloading it. > The virtual file system is an emscripten filesystem with some "smart" logic to accelerate fetching with virtual read heads that speed up when sequential data is fetched. It could also be useful to other applications, the code is in lazyFile.ts. It might also be useful to implement this lazy fetching as an SQLite VFS [*] since then SQLite could be compiled with e.g. WASI SDK without relying on all the emscripten OS emulation.

westurner 5y ago
Also, I'm not sure if jupyterlab/jupyterlab-google-drive works in JupyterLite yet? Is it yet possible to save notebooks and other files from JupyterLite running in WASM in the browser to one or more cloud storage providers? https://github.com/jupyterlab/jupyterlab-google-drive/issues/203 https://github.com/jupyterlab/jupyterlab-google-drive/issues... https://github.com/jupyterlite/jupyterlite/issues/464 https://github.com/jupyterlite/jupyterlite/issues/464

marwis 5y ago

StackBlitz' WebContainers have in-browser TCP/IP stack, I think from MirageOS.

dboreham 5y ago
How does it get raw ip packets out from inside the browser?
- westurner 5y ago
  Does it need to reframe packet structs and then fix fragmentation issues, or can it set the initial MSS (because MTU discovery likely won't work quite right) so that it doesn't have to ~shrink and re-fragment tunneled {TCP,} packets? https://en.wikipedia.org/wiki/Maximum_segment_size#MSS_and_MTU https://en.wikipedia.org/wiki/Maximum_segment_size#MSS_and_M...
- marwis 5y ago
  I think it's only externally exposed as local-only HTTP server, presumably via service worker, so you can open the website you are developing. The IP network is entirely virtual.