4 ms·
I thought about creating a search engine using https://github.com/phiresky/sql.js-httpvfs https://github.com/phiresky/sql.js-httpvfs, commoncrawl and cloudflare
by manx 2y ago
I thought about creating a search engine using https://github.com/phiresky/sql.js-httpvfs https://github.com/phiresky/sql.js-httpvfs, commoncrawl and cloudflare R2. But never found the time to start...
- mallets 2y agoMany things seem feasible with competitive object storage pricing. Still needs a little a bit of local caching to reduce read requests and origin abuse. I think rclone mount can do the same thing with its chunked reads + cache, wonder what's the memory overhead for the process.
- oersted 2y agoYou will like this then, that was the main demo from the Quickwit team. https://common-crawl.quickwit.io/ https://common-crawl.quickwit.io/