5 ms·

The problem is that current search engines are indexing what is essentially a stack of random books thrown together by anonymous library goers. Before being abl

by davemp 7y ago

The problem is that current search engines are indexing what is essentially a stack of random books thrown together by anonymous library goers. Before being able to guide readers to books, librarians have to the following non-trivial tasks over the entire collection: - identify the book's theme - measure the quality of the information - determine authenticity / malicious content - remember the position of the book in the colossal stacks Then the librarian can start to refer people to books. This problem was actually present in libraries before the revolutionary Dewy Decimal System [1]. Libraries found that the disorganization caused too much reliance on librarians and made it hard to train replacements if anything happened. The Internet just solved the problem by building a better librarian rather than building a better library. Personally I welcome any attempts to build a more organized internet. I don't think the communal book pile approach is scaling very well. [1]: https://en.wikipedia.org/wiki/Dewey_Decimal_Classification https://en.wikipedia.org/wiki/Dewey_Decimal_Classification

jasode 7y ago
>I welcome any attempts to build a more organized internet. I don't think the communal book pile approach is scaling very well. Let me know if I misunderstand your comment but to me, this has already been tried. Yahoo's founders originally tried to "organize" the internet like a good librarian. Yahoo in 1994 was originally called, "Jerry and David's Guide to the World Wide Web"[0] with hierarchical directories to curated links. However, Jerry & David noticed that Google's search results were more useful to web surfers and Yahoo was losing traffic. Therefore, in 2000 they licensed Google's search engine. Google's approach was more scaleable than Yahoo's. I often see several suggestions that the alternative to Google is curated directories but I can't tell if people are unaware of the early internet's history and don't know that such an idea was already tried and how it ultimately failed. [0] http://static3.businessinsider.com/image/57977a3188e4a714088bac98/jerry-yang-and-david-filo-created-yahoo-back-in-january-1994-while-studying-at-stanford-university-they-renamed-it-yahoo-in-march.jpg http://static3.businessinsider.com/image/57977a3188e4a714088...
- organsnyder 7y ago
  I remember trying to get one of my company's sites listed on Yahoo! back in the late 1990s. Despite us being an established company (founded in 1985) with a good domain name (cardgames.com) and a bunch of good, free content (rules for various card games, links to various places to play those games online, etc.), it took months.
  - Stronico 7y ago
    That was my experience as well. For old companies and new. Yahoo was just really slow.
  - dsparkman 7y ago
    That was not a bad thing. It was curated. Most of the crap never made it in the directory precisely because humans made decisions about what got in. If you wanted in the directory faster, you could pay a fee to get to the front of the queue. The result is that Yahoo could hire people to process the queue and make money without ads.
    - organsnyder 7y ago
      Isn't paying money to jump to the front of the queue just another form of advertising?
- davemp 7y ago
  > I often see several suggestions that the alternative to Google is curated directories but I can't tell if people are unaware of the early internet's history and don't know that such an idea was already tried and how it ultimately failed. ¿Por qué no los dos? 1) The idea is that a more organized structure is easier for a librarian to index. Today, libraries still have librarians. The book pile just wouldn't take decades to build familiarity. 2) Times change. New technology exists, people use the internet differently, and there's more at stake. Just because an approach didn't work before doesn't mean that it won't work now. There are real problems with an organizational approach, but I don't see why the idea isn't worth a revisit.
PeterisP 7y ago
The current search engines are also indexing books maliciously inserted in the library in a way to maximize their exposure e.g. a million "different" pamphlets advertising Bob's Bible Auto Repair Service inserted in the Bible category. A "better library" can't be permissionless and unfiltered; Dewey Decimal System relies on the metadata being truthful, and the internet is anything but. You can't rely on information provided by content creators; Manual curation is an option but doesn't scale (see the other answer re: early Yahoo and Google).
- davemp 7y ago
  Perhaps there exists a happy medium between: manual curation -- unfiltered PageRank is kind of a pseudo manual curation. The manual effort is just farmed out to the greater internet and analyzed.
zaphar 7y ago
The really hard part of this to scale is the quality metric. Google was the first to really scale quality measurement by outsourcing it to the web content creators themselves. Any attempt to create a decentralized index will need to tackle the quality metric problem.
agumonkey 7y ago
Also, there's an massive economic market on top on what is on the closest shelves. Libraries are less sensitive to these forces.

by davemp 7y ago

jasode 7y ago

>I welcome any attempts to build a more organized internet. I don't think the communal book pile approach is scaling very well. Let me know if I misunderstand your comment but to me, this has already been tried. Yahoo's founders originally tried to "organize" the internet like a good librarian. Yahoo in 1994 was originally called, "Jerry and David's Guide to the World Wide Web"[0] with hierarchical directories to curated links. However, Jerry & David noticed that Google's search results were more useful to web surfers and Yahoo was losing traffic. Therefore, in 2000 they licensed Google's search engine. Google's approach was more scaleable than Yahoo's. I often see several suggestions that the alternative to Google is curated directories but I can't tell if people are unaware of the early internet's history and don't know that such an idea was already tried and how it ultimately failed. [0] http://static3.businessinsider.com/image/57977a3188e4a714088bac98/jerry-yang-and-david-filo-created-yahoo-back-in-january-1994-while-studying-at-stanford-university-they-renamed-it-yahoo-in-march.jpg http://static3.businessinsider.com/image/57977a3188e4a714088...

organsnyder 7y ago
I remember trying to get one of my company's sites listed on Yahoo! back in the late 1990s. Despite us being an established company (founded in 1985) with a good domain name (cardgames.com) and a bunch of good, free content (rules for various card games, links to various places to play those games online, etc.), it took months.
- Stronico 7y ago
  That was my experience as well. For old companies and new. Yahoo was just really slow.
- dsparkman 7y ago
  That was not a bad thing. It was curated. Most of the crap never made it in the directory precisely because humans made decisions about what got in. If you wanted in the directory faster, you could pay a fee to get to the front of the queue. The result is that Yahoo could hire people to process the queue and make money without ads.
  - organsnyder 7y ago
    Isn't paying money to jump to the front of the queue just another form of advertising?
davemp 7y ago
> I often see several suggestions that the alternative to Google is curated directories but I can't tell if people are unaware of the early internet's history and don't know that such an idea was already tried and how it ultimately failed. ¿Por qué no los dos? 1) The idea is that a more organized structure is easier for a librarian to index. Today, libraries still have librarians. The book pile just wouldn't take decades to build familiarity. 2) Times change. New technology exists, people use the internet differently, and there's more at stake. Just because an approach didn't work before doesn't mean that it won't work now. There are real problems with an organizational approach, but I don't see why the idea isn't worth a revisit.

PeterisP 7y ago

The current search engines are also indexing books maliciously inserted in the library in a way to maximize their exposure e.g. a million "different" pamphlets advertising Bob's Bible Auto Repair Service inserted in the Bible category. A "better library" can't be permissionless and unfiltered; Dewey Decimal System relies on the metadata being truthful, and the internet is anything but. You can't rely on information provided by content creators; Manual curation is an option but doesn't scale (see the other answer re: early Yahoo and Google).

davemp 7y ago
Perhaps there exists a happy medium between: manual curation -- unfiltered PageRank is kind of a pseudo manual curation. The manual effort is just farmed out to the greater internet and analyzed.

zaphar 7y ago

The really hard part of this to scale is the quality metric. Google was the first to really scale quality measurement by outsourcing it to the web content creators themselves. Any attempt to create a decentralized index will need to tackle the quality metric problem.

agumonkey 7y ago

Also, there's an massive economic market on top on what is on the closest shelves. Libraries are less sensitive to these forces.