4 ms·

Yes, it was called Yahoo and it did a good job of cataloging the internet when hundreds of sites were added per week: https://web.archive.org/web/19961227005023

Yes, it was called Yahoo and it did a good job of cataloging the internet when hundreds of sites were added per week: https://web.archive.org/web/19961227005023/http://www2.yahoo.com/ https://web.archive.org/web/19961227005023/http://www2.yahoo... I'm old enough to remember sorting sites by new to see what new URLs were being created, and getting to that bottom of that list within a few minutes. Google and search was a natural response to solving that problem as the number of sites added to the internet grew exponentially...meaning we need search.

kickscondor 7y ago
Directories are still useful - Archive of Our Own (https://archiveofourown.org/ https://archiveofourown.org/) is a large example for fan fiction, Wikipedia has a full directory (https://en.wikipedia.org/wiki/Category:Main_topic_classifications https://en.wikipedia.org/wiki/Category:Main_topic_classifica...), Reddit wikis perform this function, Awesome directories (https://github.com/sindresorhus/awesome https://github.com/sindresorhus/awesome) or personal directories like mine at href.cool. The Web is too big for a single large directory - but a network of small directories seems promising. (Supported by link-sharing sites like Pinboard and HN.)
- ninju 7y ago
  How about this https://en.wikipedia.org/wiki/List_of_lists_of_lists https://en.wikipedia.org/wiki/List_of_lists_of_lists
  - kickscondor 7y ago
    Yes! But, of course, for directories outside of Wikipedia. This is very interesting for its classification structure. It's so typical of Wikipedia that a 'master list of lists' (by my count, there are 589 list links on this page) contains lists such as "Lists of Melrose Place episodes" and "Lists of Middle-earth articles" alongside lists such as "lists of wars" or "lists of banks".
- brokensegue 7y ago
  Ao3 isn't really a directory since they do the actual hosting
  - kickscondor 7y ago
    Yes, thank you - I only mean in terms of organization.
adrianmonk 7y ago
I used Yahoo back in those days, and it literally proved the point that hand-cataloging the internet wasn't tractable, at least not the way Yahoo tried to do it. There was just too much volume. It was wonderful to have things so carefully organized, but it took months for them to add sites. Their backlog was enormous. Their failure to keep up is basically what pushed people to an automated approach, i.e. the search engine.
- bitwize 7y ago
  I found myself briefly wondering if it were possible to have a decentralized open source repository of curated sites that anyone could fork, add to, or modify. Then I remembered dmoz, which wasn't really decentralized -- and realized that "awesome lists" on GitHub may be a critical step in the direction I had envisioned.
  - insulanus 7y ago
    I think this could work for small, specific areas of interest. For example, there are only so many people writing about, and interested in reading about, programming language design. Those small communities could stand ready with their community-curated index when an "outsider" wants to research something they know well.
stakhanov 7y ago
You don't have to go all the way back into Yahoo-era when it comes to manually curated directories: DMOZ was actively maintained until quite recently, but ultimately given up for what seems like good reasons.
- gerbilly 7y ago
  Earlier than that there was a list of ftp sites giving a summary of what was available on each.
- iblaine 7y ago
  This is true, and DMOZ was used heavily by Google's earlier search algorithms to rank sites within Google. Early moderators of DMOZ had god like powers to influence search results.
alangibson 7y ago
I wonder if you could build a Yahoo/Google hybrid where you start with many trusted catalogs run by special interest groups then index only those sites for search. Doesn't fully solve the centralization problem, but interesting none the less.
deleted 7y ago
[deleted]

kickscondor 7y ago

Directories are still useful - Archive of Our Own (https://archiveofourown.org/ https://archiveofourown.org/) is a large example for fan fiction, Wikipedia has a full directory (https://en.wikipedia.org/wiki/Category:Main_topic_classifications https://en.wikipedia.org/wiki/Category:Main_topic_classifica...), Reddit wikis perform this function, Awesome directories (https://github.com/sindresorhus/awesome https://github.com/sindresorhus/awesome) or personal directories like mine at href.cool. The Web is too big for a single large directory - but a network of small directories seems promising. (Supported by link-sharing sites like Pinboard and HN.)

ninju 7y ago
How about this https://en.wikipedia.org/wiki/List_of_lists_of_lists https://en.wikipedia.org/wiki/List_of_lists_of_lists
- kickscondor 7y ago
  Yes! But, of course, for directories outside of Wikipedia. This is very interesting for its classification structure. It's so typical of Wikipedia that a 'master list of lists' (by my count, there are 589 list links on this page) contains lists such as "Lists of Melrose Place episodes" and "Lists of Middle-earth articles" alongside lists such as "lists of wars" or "lists of banks".
brokensegue 7y ago
Ao3 isn't really a directory since they do the actual hosting
- kickscondor 7y ago
  Yes, thank you - I only mean in terms of organization.

adrianmonk 7y ago

I used Yahoo back in those days, and it literally proved the point that hand-cataloging the internet wasn't tractable, at least not the way Yahoo tried to do it. There was just too much volume. It was wonderful to have things so carefully organized, but it took months for them to add sites. Their backlog was enormous. Their failure to keep up is basically what pushed people to an automated approach, i.e. the search engine.

bitwize 7y ago
I found myself briefly wondering if it were possible to have a decentralized open source repository of curated sites that anyone could fork, add to, or modify. Then I remembered dmoz, which wasn't really decentralized -- and realized that "awesome lists" on GitHub may be a critical step in the direction I had envisioned.
- insulanus 7y ago
  I think this could work for small, specific areas of interest. For example, there are only so many people writing about, and interested in reading about, programming language design. Those small communities could stand ready with their community-curated index when an "outsider" wants to research something they know well.

stakhanov 7y ago

You don't have to go all the way back into Yahoo-era when it comes to manually curated directories: DMOZ was actively maintained until quite recently, but ultimately given up for what seems like good reasons.

gerbilly 7y ago
Earlier than that there was a list of ftp sites giving a summary of what was available on each.
iblaine 7y ago
This is true, and DMOZ was used heavily by Google's earlier search algorithms to rank sites within Google. Early moderators of DMOZ had god like powers to influence search results.

alangibson 7y ago

I wonder if you could build a Yahoo/Google hybrid where you start with many trusted catalogs run by special interest groups then index only those sites for search. Doesn't fully solve the centralization problem, but interesting none the less.

deleted 7y ago

[deleted]