3 ms·
Why don't they put their data dumps in an SQLite database and indexed for full-text-search https://sqlite.org/fts5.html https://sqlite.org/fts5.html Then put t
by sktrdie 6y ago
Why don't they put their data dumps in an SQLite database and indexed for full-text-search https://sqlite.org/fts5.html https://sqlite.org/fts5.html
Then put the file in a torrent. Let the users seed it.
Users can use sqltorrent (https://github.com/bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent) to query the db without downloading the entire torrent - essentially it knows to download only the pieces of the torrent to satisfy the query.
Every time a new dump is published by internet archive, the peers can change to the new torrent and reuse the pieces they already have - since SQLite is indexed in an optimal way to reduce file changes (and hence piece changes) when the data is updated.
I talk a bit about it here: https://medium.com/@lmatteis/torrentnet-bd4f6dab15e4 https://medium.com/@lmatteis/torrentnet-bd4f6dab15e4
Would save Internet Archive lots of bandwidth and hassle
- contravariant 6y agoSeems that you're being downvoted at the moment because it looks like you're making fun of archive.org, but that article is actually rather interesting. How does this compare to something like IPFS?
- Aeolun 6y agoA 50PB sqlite database? I don’t think any filesystem would be happy with that.
- sktrdie 6y agoIndeed. But nothing nothing stops them from storing torrents of torrents. The search index would be just that, the text index. Which would point to another torrent storing the actual content. Assets would point to yet other torrents. Would be interesting to learn how it's currently partitioned. I would mimic the same portioning system but use torrent instead so users can help with hosting. And use sqltorrent to serve queries efficiently.