3 ms·

Enhance it with the fact you can lazy load sqlite query using HTTP Range as demonstrated in: https://news.ycombinator.com/item?id=27016630 https://news.ycombina

by yonixw 5y ago

Enhance it with the fact you can lazy load sqlite query using HTTP Range as demonstrated in: https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630

Closi 5y ago
Exactly the thought I had reading this! Combining these projects would be awesome, proper semantic web! Plus it would mean near-zero loading times as the db size scaled (although ultimately some peers would still need the full database).
lovasoa 5y ago
Well, you could reimplement "absurd sql" over ipfs. Possible, but far from trivial.
samwillis 5y ago
As the SQLite file is being downloaded vie IPFS it wouldn’t be a HTTP Range request. Does IPFS have the equivalent? (My guess would be yes as I believe it breaks up the file for distribution, but have no idea if it’s exposed in the IPFS.js api) EDIT: After a quick scan of the docs, I think you can do this (but I certainly don't know enough). With at least the "The Mutable Files API" which "is a virtual file system on top of IPFS that exposes a Unix like API", you can provide an `offset` and `length` to `ipfs.files.read(path, [options])`. I don't know if that then translates to only downloading that part of the file from IPFS or not. https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FILES.md#ipfsfilesreadpath-options https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FI... EDIT2: In fact the `ipfs.cat` api that the OP is using takes `offset` and `length` parameters too. https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FILES.md#ipfscatipfspath-options https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FI...
- SahAssar 5y ago
  It doesn't have to be range requests, you could split the file instead of depending on the range requests. Essentially it's the same as the chunking instructions for hosters who have a maximum file size as is demonstrated here: https://github.com/phiresky/sql.js-httpvfs/blob/master/create_db.sh https://github.com/phiresky/sql.js-httpvfs/blob/master/creat...
- samwillis 5y ago
  Continuing my looking into this, the SQLite range requests trick was implemented in this [0] project. It turns out that project was inspired by a few others [1], two of which [2][3] implement a vfs for SQLite on top of bittorrent doing exactly this suggestion, but with a bittorrent hosted file rather than IPFS. They only download the parts of the SQLite files needed when querying. 0: https://github.com/phiresky/sql.js-httpvfs https://github.com/phiresky/sql.js-httpvfs 1: https://github.com/phiresky/sql.js-httpvfs#inspiration https://github.com/phiresky/sql.js-httpvfs#inspiration 2: https://github.com/lmatteis/torrent-net https://github.com/lmatteis/torrent-net 3: https://github.com/bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent

by yonixw 5y ago

Enhance it with the fact you can lazy load sqlite query using HTTP Range as demonstrated in: https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630

Closi 5y ago

Exactly the thought I had reading this! Combining these projects would be awesome, proper semantic web! Plus it would mean near-zero loading times as the db size scaled (although ultimately some peers would still need the full database).

lovasoa 5y ago

Well, you could reimplement "absurd sql" over ipfs. Possible, but far from trivial.

samwillis 5y ago

As the SQLite file is being downloaded vie IPFS it wouldn’t be a HTTP Range request. Does IPFS have the equivalent? (My guess would be yes as I believe it breaks up the file for distribution, but have no idea if it’s exposed in the IPFS.js api) EDIT: After a quick scan of the docs, I think you can do this (but I certainly don't know enough). With at least the "The Mutable Files API" which "is a virtual file system on top of IPFS that exposes a Unix like API", you can provide an `offset` and `length` to `ipfs.files.read(path, [options])`. I don't know if that then translates to only downloading that part of the file from IPFS or not. https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FILES.md#ipfsfilesreadpath-options https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FI... EDIT2: In fact the `ipfs.cat` api that the OP is using takes `offset` and `length` parameters too. https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FILES.md#ipfscatipfspath-options https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FI...

SahAssar 5y ago
It doesn't have to be range requests, you could split the file instead of depending on the range requests. Essentially it's the same as the chunking instructions for hosters who have a maximum file size as is demonstrated here: https://github.com/phiresky/sql.js-httpvfs/blob/master/create_db.sh https://github.com/phiresky/sql.js-httpvfs/blob/master/creat...
samwillis 5y ago
Continuing my looking into this, the SQLite range requests trick was implemented in this [0] project. It turns out that project was inspired by a few others [1], two of which [2][3] implement a vfs for SQLite on top of bittorrent doing exactly this suggestion, but with a bittorrent hosted file rather than IPFS. They only download the parts of the SQLite files needed when querying. 0: https://github.com/phiresky/sql.js-httpvfs https://github.com/phiresky/sql.js-httpvfs 1: https://github.com/phiresky/sql.js-httpvfs#inspiration https://github.com/phiresky/sql.js-httpvfs#inspiration 2: https://github.com/lmatteis/torrent-net https://github.com/lmatteis/torrent-net 3: https://github.com/bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent