4 ms·

Can you substantiate #1? The content on the sites I visit is created by humans. Until automation genuinely overtakes us, I'm not ready to accept at face value

Can you substantiate #1? The content on the sites I visit is created by humans. Until automation genuinely overtakes us, I'm not ready to accept at face value the scale of the internet has grown so large that humans couldn't tackle the problem.

WalterBright 6y ago
Google says there are 2 billion web sites. Each one may consist of a huge number of web pages (see wikipedia.org). All I can say is, good luck with your human curation startup.
- rkagerer 6y ago
  Sure, it also estimates more than twice that many people in the world with access to the internet. And Wikipedia is already well-curated by humans. If there were orders magnitude more pages than humans, I'd agree. But I'd also ask: Who created them all?
  - WalterBright 6y ago
    Wikipedia is one website. There are 2 billion. I don't believe wikipedia's methods scale to the internet. Also, any setup that allows everybody to be a moderator will be promptly gamed. What you want is what yahoo used to do - a hand-curated search engine. It worked when the internet was small, but got buried under the eventual avalanche of web sites.
  - rkagerer 6y ago
    Granted it's silly, but I'm fascinated by this thought experiment. It's not easy to quantify the amount of useful content on the internet. The 2bn figure above seems to stem from registered domains, and depending who you ask [1][2] around three quarters of them are "inactive" (e.g. landing page for a parked domain). At the other end of the spectrum, Google's index surpassed 130 trillion pages four years ago [3]. Point in favour of my opponent! If everyone connected to the internet indexed one page a day over the course of their lifetime we might just about do it. And anyone creating a new page would need to [arrange to] index it themselves. [1] https://www.internetlivestats.com/total-number-of-websites/ https://www.internetlivestats.com/total-number-of-websites/ [2] https://hostingtribunal.com/blog/how-many-websites/#:~:text=A%20colossal%204.2%20billion%20pages,across%208.2%20million%20web%20servers https://hostingtribunal.com/blog/how-many-websites/#:~:text=.... [3] https://searchengineland.com/googles-search-indexes-hits-130-trillion-pages-documents-263378 https://searchengineland.com/googles-search-indexes-hits-130...
  - summerlight 6y ago
    We can say the exact same thing on all the technical debts that we've been accumulated in our code base. But I've never seen those technical debts properly payed off :D
ricardo81 6y ago
#1 can somewhat be substantiated by some technical info, without knowing the true number of pages. - Using DNS "zone files", the DNS database for TLDs (which are not available for all, but most) show there's circa 200 million domains registered at any moment - A large percentage of these are parked, i.e. no unique content. - Many domains are "tasted", i.e. bought, are alive for a few days then disappear, so potentially you waste time crawling them - Lots of sites are database driven and can result in millions of pages that can be created in a day - URL rewriting means you can have an almost infinite number of pages on any one site - Soft 404s and duplicate content can be hard to spot and can waste resources in gathering/removing them There's paid for resources like Majestic/Ahrefs/Moz that crawl the web to see who's linking to who and they all contain trillions of URLs. I think the most detrimental fact is that pages often disappear or change, I don't have a recent number but I'm fairly certain there's a 10-15% chance that any link you see this year, will be gone next year. "Link rot". Hard to build a DMOZ style directory on that scale with that problem. I don't think it is unmanageable, it just needs to be seen from different perspectives and managed by different groups of people.
- clarry 6y ago
  On the other hand, most of that is stuff that human curation can rightfully reject as not worth linking and indexing. We don't need to index the entire web, we need to index the good stuff (that is likely to stay there anyway and be worth archiving).
  - WalterBright 6y ago
    How does one know what the good stuff is without looking at it?

WalterBright 6y ago

Google says there are 2 billion web sites. Each one may consist of a huge number of web pages (see wikipedia.org). All I can say is, good luck with your human curation startup.

rkagerer 6y ago
Sure, it also estimates more than twice that many people in the world with access to the internet. And Wikipedia is already well-curated by humans. If there were orders magnitude more pages than humans, I'd agree. But I'd also ask: Who created them all?
- WalterBright 6y ago
  Wikipedia is one website. There are 2 billion. I don't believe wikipedia's methods scale to the internet. Also, any setup that allows everybody to be a moderator will be promptly gamed. What you want is what yahoo used to do - a hand-curated search engine. It worked when the internet was small, but got buried under the eventual avalanche of web sites.
- rkagerer 6y ago
  Granted it's silly, but I'm fascinated by this thought experiment. It's not easy to quantify the amount of useful content on the internet. The 2bn figure above seems to stem from registered domains, and depending who you ask [1][2] around three quarters of them are "inactive" (e.g. landing page for a parked domain). At the other end of the spectrum, Google's index surpassed 130 trillion pages four years ago [3]. Point in favour of my opponent! If everyone connected to the internet indexed one page a day over the course of their lifetime we might just about do it. And anyone creating a new page would need to [arrange to] index it themselves. [1] https://www.internetlivestats.com/total-number-of-websites/ https://www.internetlivestats.com/total-number-of-websites/ [2] https://hostingtribunal.com/blog/how-many-websites/#:~:text=A%20colossal%204.2%20billion%20pages,across%208.2%20million%20web%20servers https://hostingtribunal.com/blog/how-many-websites/#:~:text=.... [3] https://searchengineland.com/googles-search-indexes-hits-130-trillion-pages-documents-263378 https://searchengineland.com/googles-search-indexes-hits-130...
- summerlight 6y ago
  We can say the exact same thing on all the technical debts that we've been accumulated in our code base. But I've never seen those technical debts properly payed off :D

ricardo81 6y ago

#1 can somewhat be substantiated by some technical info, without knowing the true number of pages. - Using DNS "zone files", the DNS database for TLDs (which are not available for all, but most) show there's circa 200 million domains registered at any moment - A large percentage of these are parked, i.e. no unique content. - Many domains are "tasted", i.e. bought, are alive for a few days then disappear, so potentially you waste time crawling them - Lots of sites are database driven and can result in millions of pages that can be created in a day - URL rewriting means you can have an almost infinite number of pages on any one site - Soft 404s and duplicate content can be hard to spot and can waste resources in gathering/removing them There's paid for resources like Majestic/Ahrefs/Moz that crawl the web to see who's linking to who and they all contain trillions of URLs. I think the most detrimental fact is that pages often disappear or change, I don't have a recent number but I'm fairly certain there's a 10-15% chance that any link you see this year, will be gone next year. "Link rot". Hard to build a DMOZ style directory on that scale with that problem. I don't think it is unmanageable, it just needs to be seen from different perspectives and managed by different groups of people.

clarry 6y ago
On the other hand, most of that is stuff that human curation can rightfully reject as not worth linking and indexing. We don't need to index the entire web, we need to index the good stuff (that is likely to stay there anyway and be worth archiving).
- WalterBright 6y ago
  How does one know what the good stuff is without looking at it?