4 ms·
Fair point. How is it to be funded?
by dotancohen 5y ago
Fair point.
How is it to be funded?
- daoudc 5y agoThe plan is to fund it through donations
- smt88 5y agoCan you make it a contributory database? I wouldn't mind "donating" my browsing history and page downloads to build the index and train the algorithm. You'd have to find a way to verify reputation to make sure no bad actors could contribute.
- aspenmayer 5y agoHow does Internet Archive verify dumps submitted by Archive Team and other groups? This may already be a solved problem. Not knowing their implementation details, I’m guessing it could be doable without reinventing much. An oracle could dispatch a P2P archive job to a pool of clients randomly assigned tasks, with both the first to archive and the first to validate being recognized by the swarm somehow, with periodic re-archiving and re-verification, rate adjusted by popularity of site and of search keywords.
- daoudc 5y agoYes, I'm planning to do something like this.
- hirako2000 5y agois the size of the common Web already way too large to play catch up against google/bing at this point? my dream, is a distributed/p2p index. each browser contribute to storing part of the overall index, and handle queries coming from other users so that how to fund huge data centers never become a question.
- daoudc 5y agoCheck out YaCy. It's great if you're happy with slow search!
- jesprenj 5y agoI had problems with YaCy. Slow search and slow crawling. Regarding search I think it could be improved if instead of HTTP requests to other peers a more efficient protocol (UDP) could be used. Regarding crawling it's quite possible that doing this in Java and running it on a Rock64 may not be the best combination (: It started OOMing after some days.
- charcircuit 5y agoYaCy's results weren't great and there was the raking was very bad letting sides game it by just spamming a ton of keywords.
- dotancohen 5y ago> is the size of the common Web already way too large to play catch up against google/bing at this point? Probably. But I would prefer a search engine that didn't search the whole web. I would prefer a search engine that searched the sites related to fields that I'm interested in. So I would pay for or donate to a search engine that provided me good results in e.g. software development. They could add additional fields as demand warrants, so long as quality as maintained. I would even like to see a faceting feature, so I could search for e.g. Matrix and get results on the mathematical concept when need be without having to wade through movie review or fiddle with magic search keywords.
- hirako2000 5y agonot searching the whole Web makes sense, except that it isn't clear what belongs to your field of interest and what doesn't. should indexing skip a blog page because a the author usually write about algorithm but here goes on and on about business while sporadically mentioning algorithmic technical aspects? and what if you want to search about pottery that Sunday morning, turn to Google? I think segmenting search results by topics is a useful consumer query feature, not sure segmenting what gets indexed would provide a useful service other than covering niches hence not really fulfilling a web search engine. I find the idea less ambitious so maybe that's how an open search engine should approach the problem, federation of hosts could, cover the whole Web eventually.