4 ms·

Google didn’t gaslight the web: Web directories died out because they go stale, fast, and require far more human curation than is economically viable. I remembe

Google didn’t gaslight the web: Web directories died out because they go stale, fast, and require far more human curation than is economically viable. I remember Yahoo effectively deprecating their directory pages long before I heard of Google, back when I was using Dogpile and CompuServe.

PaulHoule 4y ago
I never saw a web directory that took a serious approach to using automation to curate the directory. It's probably more feasible than people think because it's a matter of classifying links up or down (relevant or not) which is a much more ontologically and mathematically tractable problem than learning a ranking function or trying to classify things into one of N>2 categories. I think the problem may more have been the lack of a sustainable revenue model. Getting volunteers to curate the directory is particularly destructive because the people who most want to volunteer either (1) want to promote something or (2) want to get paid so somebody else can promote something.
- davemp 4y ago
  I consider Reddit to be a form of web directory in a way.
  - PaulHoule 4y ago
    I can't use it though because I've got this disability that memes cause me extreme distress.
    - naasking 4y ago
      That sounds like a meme.
      - Shorel 4y ago
        Every idea is a meme in the classical sense.
    - 314 4y ago
      Panik!! Kalm. Panik!!
- danans 4y ago
  > I never saw a web directory that took a serious approach to using automation to curate the directory. That's what search indexes are. Automatically curated directories. It's just that the UI to the index is natural language queries instead of clicks through a graph, because the latter isn't scalable to large corpuses.
  - PaulHoule 4y ago
    No, I think a modern directory would be a set of topics in an ontology with links. If I had to seed one I would suck all the external links out of Wikipedia, impose some organizing structures (probably overlapping trees or dags) and then build a set of classifiers for nested topic relevance, spaminess, etc. There are certain sites which have a landing page for every topic in some set of topics, you could add a lot of links quickly if you built rules for importing links from particular sites. Adding 10 sites a day with 1000 links each would be very possible, in 100 days you could build out a million links.
    - candiodari 4y ago
      Exactly how do you think Google (and Bing, and ...) work? They do start from known indexes, in particular wikipedia. Hell, Google even has internal papers where they claim they improved search quality by "wikipedia's" as a unit.
      - PaulHoule 4y ago
        Actually Microsoft bought a company called Powerset which did information from Wikipedia to build a "semantic as in semantic web" index, that technology became the heart of Bing. Google was caught flat footed and wound up buying and killing Freebase in order to catch up. They lied and said they rejected "semantic web" approaches despite hiring one of the leaders of the Cyc project as their head of research. Still there is a difference between exposing that kind of database through a full text index vs exposing it through a browsing interface.
ctippett 4y ago
It's not quite the same as the web directories of old, but I find the links and content contained in the "Awesome [topic]" lists on GitHub to be a useful resource. The curation aspect is managed via pull requests just as any other open source project, although this doesn't make it immune from going stale.
- PaulHoule 4y ago
  That's a good example and they aren't overrun with spam.

PaulHoule 4y ago

I never saw a web directory that took a serious approach to using automation to curate the directory. It's probably more feasible than people think because it's a matter of classifying links up or down (relevant or not) which is a much more ontologically and mathematically tractable problem than learning a ranking function or trying to classify things into one of N>2 categories. I think the problem may more have been the lack of a sustainable revenue model. Getting volunteers to curate the directory is particularly destructive because the people who most want to volunteer either (1) want to promote something or (2) want to get paid so somebody else can promote something.

davemp 4y ago
I consider Reddit to be a form of web directory in a way.
- PaulHoule 4y ago
  I can't use it though because I've got this disability that memes cause me extreme distress.
  - naasking 4y ago
    That sounds like a meme.
    - Shorel 4y ago
      Every idea is a meme in the classical sense.
  - 314 4y ago
    Panik!! Kalm. Panik!!
danans 4y ago
> I never saw a web directory that took a serious approach to using automation to curate the directory. That's what search indexes are. Automatically curated directories. It's just that the UI to the index is natural language queries instead of clicks through a graph, because the latter isn't scalable to large corpuses.
- PaulHoule 4y ago
  No, I think a modern directory would be a set of topics in an ontology with links. If I had to seed one I would suck all the external links out of Wikipedia, impose some organizing structures (probably overlapping trees or dags) and then build a set of classifiers for nested topic relevance, spaminess, etc. There are certain sites which have a landing page for every topic in some set of topics, you could add a lot of links quickly if you built rules for importing links from particular sites. Adding 10 sites a day with 1000 links each would be very possible, in 100 days you could build out a million links.
  - candiodari 4y ago
    Exactly how do you think Google (and Bing, and ...) work? They do start from known indexes, in particular wikipedia. Hell, Google even has internal papers where they claim they improved search quality by "wikipedia's" as a unit.
    - PaulHoule 4y ago
      Actually Microsoft bought a company called Powerset which did information from Wikipedia to build a "semantic as in semantic web" index, that technology became the heart of Bing. Google was caught flat footed and wound up buying and killing Freebase in order to catch up. They lied and said they rejected "semantic web" approaches despite hiring one of the leaders of the Cyc project as their head of research. Still there is a difference between exposing that kind of database through a full text index vs exposing it through a browsing interface.

ctippett 4y ago

It's not quite the same as the web directories of old, but I find the links and content contained in the "Awesome [topic]" lists on GitHub to be a useful resource. The curation aspect is managed via pull requests just as any other open source project, although this doesn't make it immune from going stale.

PaulHoule 4y ago
That's a good example and they aren't overrun with spam.