3 ms·

Someone needs to build a search engine that does'nt depend on page rank. Page rank is the thing that filters out 99.9% of the internet: it's the millions of pa

Someone needs to build a search engine that does'nt depend on page rank. Page rank is the thing that filters out 99.9% of the internet: it's the millions of pages you don't see that could potentially have better answers in it. Now, surely, there's a lot of junk in that 99%, but I'd be willing to bet there's a lot of gold nuggets in there too. Google's fundamental assumption and it's biggest weakness is that: Excellent content will always have an excellent source to upvote it with links. This isn't always true. If ever another search engine were to supplant google, it would probably take advantage of this weakness. This weakness is even bigger in niche areas that google AI is not able to recognize.

cookiecaper 8y ago
I don't know how much Google realizes it, but "walled gardens" are a fundamental threat to the quality of their search. Since most "normal user" internet activity is now locked up behind login pages on Facebook, the value of PageRank grows more dubious by the day. This leaves institutional and commercial websites as the corpus of "open content" available for Google to index/test, and while they are sometimes useful, the incentives are much less straightforward when you bring in money and organizational politics. Depending on the organic voice of the users who were publishing personal web pages and just wanted to discuss their interests and share the best information available got Google where it was, but now that lifeblood is locked down and thoroughly controlled by Google's competitors. Nowadays, many hobbyists and small businesses even forgo the web site entirely and just operate out of a Facebook page or group. Worse still, open forums typically either completely disallow links with accurate anchor text or explicitly tell search engines to ignore them with rel=nofollow. This makes PageRank even less valuable, because the shared URL is not associated with link text. Some forums will transparently place anything containing a URL in a moderation queue, leaving no information at all for a search crawler to extract (ostensibly to prevent spam, but I've found in practice it's more often used by niche forum operators to block out competitors and get higher prices for ads; you can even see this on reddit, as many subreddit mods configure AutoModerator to filter any URLs pointing to unapproved domains). Google was designed for the web as it stood in 1999. It's not 1999 anymore. Sadly, hyper-aggressive legal restrictions like the CFAA and the RAM Copy Doctrine mean that a useful search for the 2018 web is unlikely to materialize (at least in the U.S. or countries that impose similar restrictions).

cookiecaper 8y ago

I don't know how much Google realizes it, but "walled gardens" are a fundamental threat to the quality of their search. Since most "normal user" internet activity is now locked up behind login pages on Facebook, the value of PageRank grows more dubious by the day. This leaves institutional and commercial websites as the corpus of "open content" available for Google to index/test, and while they are sometimes useful, the incentives are much less straightforward when you bring in money and organizational politics. Depending on the organic voice of the users who were publishing personal web pages and just wanted to discuss their interests and share the best information available got Google where it was, but now that lifeblood is locked down and thoroughly controlled by Google's competitors. Nowadays, many hobbyists and small businesses even forgo the web site entirely and just operate out of a Facebook page or group. Worse still, open forums typically either completely disallow links with accurate anchor text or explicitly tell search engines to ignore them with rel=nofollow. This makes PageRank even less valuable, because the shared URL is not associated with link text. Some forums will transparently place anything containing a URL in a moderation queue, leaving no information at all for a search crawler to extract (ostensibly to prevent spam, but I've found in practice it's more often used by niche forum operators to block out competitors and get higher prices for ads; you can even see this on reddit, as many subreddit mods configure AutoModerator to filter any URLs pointing to unapproved domains). Google was designed for the web as it stood in 1999. It's not 1999 anymore. Sadly, hyper-aggressive legal restrictions like the CFAA and the RAM Copy Doctrine mean that a useful search for the 2018 web is unlikely to materialize (at least in the U.S. or countries that impose similar restrictions).