3 ms·

It just requires a group of people to start a new bunch of niche indexing projects. My idea, which may be a bit off the wall, is really to go down a path where

It just requires a group of people to start a new bunch of niche indexing projects. My idea, which may be a bit off the wall, is really to go down a path whereby you only index a subject. So node.js guys just give a search engine that indexes node.js material, Python people index Python articles, short-haired cat people index only short-haired cats... They key is to make the subject you are indexing manageable in scope. So you keep it small. The key things that Google have IMO is how good they are at locating documents, and ensuring that they have mapped out their search to what is relevant. But I think that for those who know what they are looking for, specific indexing of a particular domain of knowledge would make searching more relevant. Now the next steps are to find a way to discover this content. But in this case, submitting to the smaller indexers would be helpful. Take HN for instance - in the grand scheme of things it's a big index of stuff that interests hackers, now if you indexed HN that would be a search engine based on user submitted links. Then you could figure out a way for indexes to collaborate with each other. What it would in essence do would be to democratize search. And it would penalise information that wasn't in the open, though someone could make it work by paying for links to good content and charging a monthly fee to use their index. The index would be a broker, but the more reputable it is the more content it can index. And the market would stabilise on a price - if you can't afford it, you use a competitor, of you could make your own. It would need a framework for decentralised collaboration, and perhaps a good mechanism for the indexing and displaying search results. Just an idea. Might not be practical!

awinter-py 11y ago
If you know how to do the topic detection you don't need separate indexes
idrios 11y ago
I really like your enthusiasm and I can tell you put a lot of thought into this. In regards to the parent comment's concerns about Google, though, I don't think he is saying that their indexing algorithm needs to be improved upon, or even that we need an alternative to it. I read it as "Google, in its ability to select sites for indexing, can control people's access to information and can therefore manipulate thought". The problem is that Google has so much power and inherent ulterior motives as a company that make it potentially threatening. I think that's why you're pitching this project to democratize search, but I think what you just pitched is a Google/Reddit hybrid--or basically a search engine that only gives results from a selected subreddit. Which could be cool, and I don't know if that exists or not. However, I think the solution to displacing Google from being the internet itself is to make an open source search engine that is equally good as a search tool, but can sell itself on the basis that it doesn't track you or record your information (or if it does, there is no way for a human person to access your information). I think the solution is DuckDuckGo.

awinter-py 11y ago

If you know how to do the topic detection you don't need separate indexes

idrios 11y ago

I really like your enthusiasm and I can tell you put a lot of thought into this. In regards to the parent comment's concerns about Google, though, I don't think he is saying that their indexing algorithm needs to be improved upon, or even that we need an alternative to it. I read it as "Google, in its ability to select sites for indexing, can control people's access to information and can therefore manipulate thought". The problem is that Google has so much power and inherent ulterior motives as a company that make it potentially threatening. I think that's why you're pitching this project to democratize search, but I think what you just pitched is a Google/Reddit hybrid--or basically a search engine that only gives results from a selected subreddit. Which could be cool, and I don't know if that exists or not. However, I think the solution to displacing Google from being the internet itself is to make an open source search engine that is equally good as a search tool, but can sell itself on the basis that it doesn't track you or record your information (or if it does, there is no way for a human person to access your information). I think the solution is DuckDuckGo.