3 ms·
> why build a search engine at all in this case? On a public scale, you could make an argument for tighter integration/better privacy with the lists. For examp
by Shared404 6y ago
> why build a search engine at all in this case?
On a public scale, you could make an argument for tighter integration/better privacy with the lists. For example:
Browser -----Request-to-SE-----> Search Engine
^ |
| Unfiltered Results (In YAML/JSON)
| |
| V
|--Desired Results------ Local Filtering/Rendering
On a private scale, if you are only crawling sites on the allow list than you have the possibility of being able to better maintain a local database of sites to show up on the search.
Edit: Possibly this could be easier to use to set up distributed search as well, as each node could index a given list, and then distribute that list similarly to DNS. Don't really know how well that would work though, just an idea.
- edgyquant 6y agoAside from this a big reason to build this is it seems a lot simpler than writing a giant web crawler ala google and thus is a good target for an open source solution. Which is the biggest problem with duck duck go.
- Shared404 6y agoDo you have thoughts on implementing the distributed search? I'm thinking about playing around with this in my spare time, but that part seems the hardest to do.
- edgyquant 6y agoI would think everyone would run their own “crawler” but maybe you could use a ledger and delegate sites to different workers. If you’re whitelisting sites you could maybe only crawl a link or two deep. It’d take a lot of cycles while small but if you get a network growing you could even have sub-networks with their own whitelist additions (and every user has a blacklist.)
- rakoo 6y agoDon't start from scratch, take a look at yacy (https://yacy.net/ https://yacy.net/) that already does most of what is discussed here