4 ms·
Interesting question. I looked at the source code to understand that. SQLite knows where to look for when you open a SQLite database and you run a query, right
by frafra 6y ago
Interesting question. I looked at the source code to understand that.
SQLite knows where to look for when you open a SQLite database and you run a query, right? It just asks the underlying filesystem to provide N bytes starting from an offset using a C function, then it repeats the same operation on different portions of the file, it does its computation and everybody is happy.
The software relies on sqltorrent, which is a custom VFS for SQLite. That means that SQLite function to read data from a file stored in the filesystem is replaced by a custom function. Such custom code computes which Torrent block(s) should have the highest priority, by dividing the offset and the number of bytes that SQLite wants to read by the size of the torrent blocks. It is just a division.
See: https://github.com/bittorrent/sqltorrent/blob/master/sqltorrent.cpp#L109-L158 https://github.com/bittorrent/sqltorrent/blob/master/sqltorr...
- banana_giraffe 6y agoYep. I've used the same basic technique to write a S3 VFS wrapper for APSW. It's not quick, by any means, but it's dang useful in cases where I have very large SQLite database son S3 and only need to get a small bit of information out of them.
- frafra 6y agoI was not aware of APSW. Thank you. For future reference: "APSW has the following enhancements/differences over pysqlite 2 (wrapping SQLite 3): - APSW stays up to date with SQLite. As features are added and functionality changed in SQLite, APSW tracks them. - APSW gives all functionality of SQLite including virtual tables, Virtual File System (VFS), BLOB I/O, backups and file control. [...]" -- https://rogerbinns.github.io/apsw/pysqlite.html#pysqlitediffs https://rogerbinns.github.io/apsw/pysqlite.html#pysqlitediff...
- banana_giraffe 6y agoOne thing I'd warn about with APSW: https://rogerbinns.github.io/apsw/download.html#easy-install-pip-pypi https://rogerbinns.github.io/apsw/download.html#easy-install... Basically, if you install it from pip (without following the directions to install from git directly), then you end up with some ancient version, which rather defeats the point of APSW.