8 ms·

Everyone has missed the most important aspect of search engines, from the point of view of their core function of information retrieval: they're the internet eq

• arpa 7y ago
  maybe crowdsourcing would be a solution - something similar to "@home" project, only for web indexes/cache - maybe even leverage the browsers via plugin for web scraping. It already kind of works for getpocket.
• ScottFree 7y ago
  > or you find a novel way to make a neutral search engine Just because nobody's said it in this thread yet: blockchain? I never bought into the whole bitcoin buzz, but using a blockchain as an internet index could be interesting.
  • arpa 7y ago
    even better, have something like git for the web - effectively working as an archive.
    • ScottFree 7y ago
      The problem with git is countering nefarious forces. The blockchain is better in that regard because the consensus algorithm can be used to verify that the listings are legitimate.
      • arpa 7y ago
        content change signed by creators private key, otherwise merge is rejected? or, wiki approach...
        • fehrnstr 7y ago
          Just signing with a private key isn't a guarantor of anything other than that if you trust that the person with the key is who they say they are, then the actual content is from them. But that would require a massively large web of trust in itself: that all the private keys would be trusted. And if you only let in private keys that you explicitly trusted, then it's very likely you could end up with an echo chamber
          • arpa 7y ago
            good point, but we already have the PKI in place, and use it for SSL.
      • deleted 7y ago
        [deleted]
  • KirinDave 7y ago
    How would Merkle DAGs be relevant?
  • deleted 7y ago
    [deleted]
• narag 7y ago
  A few years ago there was that blogs thing, with rss... all things that favoured federation, independent content generation, etc. Now it's all about platforms. I understand that "regular people" are more comfortable with Facebook but, other than that, why are blogs and forums less popular now?
  • arpa 7y ago
    the problem is multiple actually: a) most internet-connected devices these days favor content consumption vs content creation (blogs vs instagram), b) mainstream culture > closely-knit communities (facebook > forums) c) big-player takeovers (facebook for groups, google for search) over previously somewhat niche areas and, actually, internet infrastructure d) if you're not a big player, you don't exist... and back to c)
    • asark 7y ago
      > a) most internet-connected devices these days favor content consumption vs content creation (blogs vs instagram), You chose Instagram as your example, to make the point that phones favor consumption over creation?
      • arpa 7y ago
        Yes! Instagram has the appearance of a OC/creation platform, but, typically of such platforms (such as twitter/fb) the "content" is low-effort "convenient" opportunistic trivia, and the product consumed is likes, followers, etc.
  • JaumeGreen 7y ago
    The problem with forums is that you end visiting 5~10 different forums, each with their own login, and some of them might be restricted at work (not that you should visit them often). So it's easier to have 2~4 aggregators in where all the information you desire resides, even if in each of them there are different forums. A unified entry point helps adoption.
    • 7y ago
• basch 7y ago
  They are also a spam filter. It's not just an index of whats relevant, but removal of what maliciously appears to be relevant at first glance.
  • izendejas 7y ago
    This. Everyone's missing the point of a search engine. We're talking about billions of pages and if not ranked (authority is a good hueristic), filtered (de-ranked), etc then good luck finding valuable information because everyone is gaming the systems to improve their ranking. I think this is part of the reason you get a lot of fake news on social media. It's a constant stream of information (a new dimension of time has been added to the ranking, basically) that needs to be ranked and with humans in the loop, there's no way to do this very easily without filtering for noise and outright malicious content.
    • luxuryballs 7y ago
      The reverse is a problem too, Google filtering things out based on their political leanings in an attempt to shape public opinion.
      • Nasrudith 7y ago
        I haven't seen any examples which were anything other than runaway persecution complexes of those who found their world view was less popular than they believed - which were greeted with exasperation by testifying engineers who had to explain how absurdly unscaleable it would be to do it manually.
        • aslaan 7y ago
          https://gohmert.house.gov/news/documentsingle.aspx?DocumentID=399712 https://gohmert.house.gov/news/documentsingle.aspx?DocumentI...
    • basch 7y ago
      i disagree that there isnt a way, just that nobodies tried a good one yet. take reddit for example. it should be very easy to establish a few voters who make "good" decisions, and then extrapolate their good decisions based on people with similar voting patterns. it would combine a million monkeys with typewriters with expert meritocracy. you want different sorting, sort by different experts until you get the results you want. it seems every platform is too busy fighting noise to focus on amplifying signal, or are focused on teaching machines to do the entire task, instead of using machines to multiply the efficiency of people with taste who can make a good judgement call with regard to whether something is novel or or pseudo-intellectual. Not to pick on them, but I would suspect an expert to be better at deranking aeon/brainpickings type clickbait than an eruditelike ai, if only because humans can still more easily determine if someone is making an actual worthwhile point, vs repeating a platitude, conventional wisdom, or something hollow.
• davemp 7y ago
  The problem is that current search engines are indexing what is essentially a stack of random books thrown together by anonymous library goers. Before being able to guide readers to books, librarians have to the following non-trivial tasks over the entire collection: - identify the book's theme - measure the quality of the information - determine authenticity / malicious content - remember the position of the book in the colossal stacks Then the librarian can start to refer people to books. This problem was actually present in libraries before the revolutionary Dewy Decimal System [1]. Libraries found that the disorganization caused too much reliance on librarians and made it hard to train replacements if anything happened. The Internet just solved the problem by building a better librarian rather than building a better library. Personally I welcome any attempts to build a more organized internet. I don't think the communal book pile approach is scaling very well. [1]: https://en.wikipedia.org/wiki/Dewey_Decimal_Classification https://en.wikipedia.org/wiki/Dewey_Decimal_Classification
  • jasode 7y ago
    >I welcome any attempts to build a more organized internet. I don't think the communal book pile approach is scaling very well. Let me know if I misunderstand your comment but to me, this has already been tried. Yahoo's founders originally tried to "organize" the internet like a good librarian. Yahoo in 1994 was originally called, "Jerry and David's Guide to the World Wide Web"[0] with hierarchical directories to curated links. However, Jerry & David noticed that Google's search results were more useful to web surfers and Yahoo was losing traffic. Therefore, in 2000 they licensed Google's search engine. Google's approach was more scaleable than Yahoo's. I often see several suggestions that the alternative to Google is curated directories but I can't tell if people are unaware of the early internet's history and don't know that such an idea was already tried and how it ultimately failed. [0] http://static3.businessinsider.com/image/57977a3188e4a714088bac98/jerry-yang-and-david-filo-created-yahoo-back-in-january-1994-while-studying-at-stanford-university-they-renamed-it-yahoo-in-march.jpg http://static3.businessinsider.com/image/57977a3188e4a714088...
    • organsnyder 7y ago
      I remember trying to get one of my company's sites listed on Yahoo! back in the late 1990s. Despite us being an established company (founded in 1985) with a good domain name (cardgames.com) and a bunch of good, free content (rules for various card games, links to various places to play those games online, etc.), it took months.
• IanSanders 7y ago
  I think heavy reliance on human language (and its ambiguity) is one of the main problems. Maybe personal whitelist/blacklist for domains and authors could improve things. Sort of "Web of trust" but done properly. Not completely without search engines, but for example, if every website was responsible for maintaining it's own index, we could effectively run our own search engines after initialising "base" trusted website lists. Let's say I'm new to this "new internet", I ask around what are some good websites for information I'm interested in. My friend tells me wikipedia is good for general information, webmd for health queries, stackoverflow for programming questions, and so on. I add wikipedia.org/searchindex, webdm.com/searchindex and stackoverflow.com/searchindex to my personal search engine instance, and every time I search something, these three are queried. This could be improved with local cache, synonyms, etc. As you carry on using it, you expand your "library". Of course it would increase workload of individual resources, but has potential to give feel of that web 1.0 once again.
  • TheOtherHobbes 7y ago
    This would be ludicrously easy to game. Crowdsourcing would also be ludicrously easy to game. The problem isn't solvable without a good AI content scraper. The scraper/indexer either has to be centralised - an international resource run independently of countries, corporations, and paid interest groups - or it has be an impossible-to-game distributed resource. The former is hugely challenging politically, because the org would effectively have editorial control over online content, and there would be huge fights over neutrality and censorship. (This is more or less where are now with Google. Ironically, given the cognitive distortions built into corporate capitalism, users today are more likely to trust a giant corporation with an agenda than a not-for-profit trying to run independently and operate as objectively as possible.) Distributed content analysis and indexing - let's call it a kind of auto-DNS-for-content - is even harder, because you have to create an un-hackable un-gameable network protocol to handle it. If it isn't un-gameable it become a battle of cycles, with interests with access to more cycles being able to out-index those with fewer - which will be another way to editorialise and control the results. Short answer - yes, it's possible, but probably not with current technology, and certainly not with current politics.
• Nasrudith 7y ago
  The only assured way to have a "neutral" search engine is to run your own spiders and indexers which you understand completely. Neutral in that sense is only "not serving the agenda or judgement of another" at the obvious cost of labor and not just as a one off thing as the searched content often attempts to optimize for views. It isn't like a library of passive books to sort through but a Harry Potter wizard portrait gallery full of jealous media vying for attention. And pendantically it isn't true neutral - but serves your agenda to the best of your ability. A "true neutral" would serve all to the best of their ability. Besides neutrality in a search engine on a literal level is oxymoronic and self defeating - its whole function is to prioritize content in the first place.
• greglindahl 7y ago
  The library index only indexes the information that fits on a card catalog card. That's extremely unlike a web search engine. If you'd like to see an experimental discovery interface for a library that goes deeper into book contents, check out https://books.archivelab.org/dateviz/ https://books.archivelab.org/dateviz/ -- sorry, not very mobile friendly. Not surprisingly, this book thingie is a big centralized service, like a web search engine.
• ehnto 7y ago
  So long as there is a mechanism for categorizing information and ranking the results, people will try to game the mechanism to get the top spot regardless of your own incentives. Despite their incentives to make money, Google have actually been trying for years to stop people from gaming the system. It's impressive how far they've been able to come, but their efforts are thrwarted at every turn thanks to the big budgets employed to get traffic to commercial websites.
• tracker1 7y ago
  I don't think it would be an issue if Google wasn't creating "special" rules for specific winners and losers (overall). Hell, I really wish they'd make it easy to individually exclude certain domains from results. The canonical example to me of something to exclude would be the expertsexchange site. After stack overflow, ee was more than useless, and even before it was just annoying. There are lots of sites with paywalls, and other obfuscations to content and imho these sites are the ones that should be dropped/low-ranked. But the fact that there's no autocomplete for "Hillary Clinton is|has" (though "Donald Trump is" is also filtered). Yes, it's been heavily gamed. It's also had active meddling. And their control over YouTube seems to be even worse, with disclosed documents/video that indicate they're willing to go so far as outright election manipulation. With all indications that Facebook, Pinterest and others are going the same route.
• z3t4 7y ago
  A search engine is more like putting the books in a paper schredder and writing the book title on every piece, then ordering the pices by whatever words you can find on it, putting all pieces that has the word "hacker" on it in the same box. Where as the problem becomes how you sort the pieces. Want to find a book about "hacking"? This box has all the shreds that has the word "hacker" on it, you can find the book title on the back of the piece. Second problem becomes how relevant the word is to the book.