4 ms·

There was that project some guy posted a while back that used a combination of sqlite and partial downloads to enable searches on a database before it was downl

There was that project some guy posted a while back that used a combination of sqlite and partial downloads to enable searches on a database before it was downloaded all the way. If you can fit PDFs somewhere into that you'd be golden. Or just use IPFS I suppose.

hkt 5y ago
Isn't that essentially mapreduce? Either way, interesting and I'd love to see the link.
- vorticalbox 5y ago
  I believe this is the project mentioned https://github.com/lmatteis/torrent-net https://github.com/lmatteis/torrent-net
  - tmkadamcz 5y ago
    This looks like it could be a good approach.
  - contravariant 5y ago
    That one looks familiar. Though apparently the same thing has been tried in several different ways going by the replies I got.
- ric2b 5y ago
  It's this one: https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ https://phiresky.github.io/blog/2021/hosting-sqlite-database...
o8r3oFTZPE 5y ago
"There was that project some guy posted a while back that used a combination of sqlite and partial downloads to enable searches on a database before it was downloaded all the way." https://github.com/bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent
- jagged-chisel 5y ago
  this is the one: https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ https://phiresky.github.io/blog/2021/hosting-sqlite-database... HN submission: https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630
- o8r3oFTZPE 5y ago
  This is the original. Then came https://github.com/lmatteis/net-torrent https://github.com/lmatteis/net-torrent and later one written in Javascript, inspired by net-torrent.
divbzero 5y ago
IPFS would face a similar challenge as the “keep torrents seeded” problem mentioned by GP. Wouldn’t there be risk to peers who host the PDFs?
- Natsu 5y ago
  I sort of feel like there should be some way to use some kind of construct to get people to seed things so that others seed things for them, but I haven't seen that invented yet.
  - zolland 5y ago
    I think seed ratios and seed time (mostly used by private trackers) attempt to solve this problem.
  - miloignis 5y ago
    Been a while since I've looked at them, but IPFS with FileCoin and Ethereum Swarm had that kind of goal. It might be beneficial to create something like what you describe without any cryptocurrency association though, and I've been mulling over possibilities for distributed systems that are inherently currency-less to avoid all of the scams that cryptocurrency attracts.
    - Taek 5y ago
      The leader in that space is Skynet, which basically is like IPFS + Filecoin but also has dynamic elements to it, and a lot better performance + reliability. Cryptocurrency is helpful because it allows you to incentivize people to hold the data. If you don't have cryptocurrency, you're basically dependent on altruism to keep data alive (like bittorrent, or ipfs without filecoin). Popular files do okay for a while, but even popular files struggle to provide good uptime and reliability after a few years. On an incentivized network like Sia or Filecoin, you can have high performance and reliability without ever needing any users to keep their machines online and seeding.
      - 5y ago

hkt 5y ago

Isn't that essentially mapreduce? Either way, interesting and I'd love to see the link.

vorticalbox 5y ago
I believe this is the project mentioned https://github.com/lmatteis/torrent-net https://github.com/lmatteis/torrent-net
- tmkadamcz 5y ago
  This looks like it could be a good approach.
- contravariant 5y ago
  That one looks familiar. Though apparently the same thing has been tried in several different ways going by the replies I got.
ric2b 5y ago
It's this one: https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ https://phiresky.github.io/blog/2021/hosting-sqlite-database...

o8r3oFTZPE 5y ago

"There was that project some guy posted a while back that used a combination of sqlite and partial downloads to enable searches on a database before it was downloaded all the way." https://github.com/bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent

jagged-chisel 5y ago
this is the one: https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ https://phiresky.github.io/blog/2021/hosting-sqlite-database... HN submission: https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630
o8r3oFTZPE 5y ago
This is the original. Then came https://github.com/lmatteis/net-torrent https://github.com/lmatteis/net-torrent and later one written in Javascript, inspired by net-torrent.

divbzero 5y ago

IPFS would face a similar challenge as the “keep torrents seeded” problem mentioned by GP. Wouldn’t there be risk to peers who host the PDFs?

Natsu 5y ago
I sort of feel like there should be some way to use some kind of construct to get people to seed things so that others seed things for them, but I haven't seen that invented yet.
- zolland 5y ago
  I think seed ratios and seed time (mostly used by private trackers) attempt to solve this problem.
- miloignis 5y ago
  Been a while since I've looked at them, but IPFS with FileCoin and Ethereum Swarm had that kind of goal. It might be beneficial to create something like what you describe without any cryptocurrency association though, and I've been mulling over possibilities for distributed systems that are inherently currency-less to avoid all of the scams that cryptocurrency attracts.
  - Taek 5y ago
    The leader in that space is Skynet, which basically is like IPFS + Filecoin but also has dynamic elements to it, and a lot better performance + reliability. Cryptocurrency is helpful because it allows you to incentivize people to hold the data. If you don't have cryptocurrency, you're basically dependent on altruism to keep data alive (like bittorrent, or ipfs without filecoin). Popular files do okay for a while, but even popular files struggle to provide good uptime and reliability after a few years. On an incentivized network like Sia or Filecoin, you can have high performance and reliability without ever needing any users to keep their machines online and seeding.
    - 5y ago