2 ms·
Do you have thoughts on implementing the distributed search? I'm thinking about playing around with this in my spare time, but that part seems the hardest to d
by Shared404 6y ago
Do you have thoughts on implementing the distributed search?
I'm thinking about playing around with this in my spare time, but that part seems the hardest to do.
- edgyquant 6y agoI would think everyone would run their own “crawler” but maybe you could use a ledger and delegate sites to different workers. If you’re whitelisting sites you could maybe only crawl a link or two deep. It’d take a lot of cycles while small but if you get a network growing you could even have sub-networks with their own whitelist additions (and every user has a blacklist.)
- rakoo 6y agoDon't start from scratch, take a look at yacy (https://yacy.net/ https://yacy.net/) that already does most of what is discussed here