4 ms·
Aside from this a big reason to build this is it seems a lot simpler than writing a giant web crawler ala google and thus is a good target for an open source so
by edgyquant 6y ago
Aside from this a big reason to build this is it seems a lot simpler than writing a giant web crawler ala google and thus is a good target for an open source solution. Which is the biggest problem with duck duck go.
- Shared404 6y agoDo you have thoughts on implementing the distributed search? I'm thinking about playing around with this in my spare time, but that part seems the hardest to do.
- edgyquant 6y agoI would think everyone would run their own “crawler” but maybe you could use a ledger and delegate sites to different workers. If you’re whitelisting sites you could maybe only crawl a link or two deep. It’d take a lot of cycles while small but if you get a network growing you could even have sub-networks with their own whitelist additions (and every user has a blacklist.)
- rakoo 6y agoDon't start from scratch, take a look at yacy (https://yacy.net/ https://yacy.net/) that already does most of what is discussed here