4 ms·

Google Dominates Thanks to an Unrivaled View of the Web

5 points by marban 6y ago

knuckleheads 6y ago
Hello! Zack Maril here. I've been working on this research for over two years now. I got started working on it after I read everybody on Hacker News talking about how they kept getting blocked when they were crawling websites[0] and also recent antitrust developments[1]. This has been a huge part of my life these past few years and it wouldn't have happened without the Hacker News community, so, from the bottom of my heart, thank you. If you would like to read more about this research, please check out Knuckleheads' Club[2]. I've published a lot of what I have so far there and plan on publishing more soon. You're all invited to join the club and I'll be checking this thread all day and responding to folks as best I can. [0] https://news.ycombinator.com/item?id=9612345 https://news.ycombinator.com/item?id=9612345 [1] https://news.ycombinator.com/item?id=17941804 https://news.ycombinator.com/item?id=17941804 [2] http://knuckleheads.club http://knuckleheads.club
- marban 6y ago
  Never heard of this initiative but kudos for bringing it up. I run a popular news site[0] and merely aggregating ~150 sources costs me upwards of $1k/m just to crawl the basics. [0] https://upstract.com/ https://upstract.com/
  - knuckleheads 6y ago
    Yes!! Exactly!! Crawling is super expensive to do and it's hard to even get the money to start doing the crawling. And even then once you get started, a lot of websites block you out or only let Googlebot through the paywall. And we are launching the club today with the publishing of the NYT article, we have been pretty quiet about it so far for the last two years or so, doing the research in private.
dkjaudyeqooe 6y ago
An effective breakup of Google would entail requiring them to put their crawling infrastructure into a seperate company and require them to make it available to other businesses at a reasonable cost, or get out of the crawling business entirely. Might be tricky to implement but without something like this no one has a chance of competing with them.
- knuckleheads 6y ago
  Absolutely, that's direct precedence with AT&T and the Yellow Pages when the government broke up AT&T. There's a really interesting angle to this where you consider Googlebot to be an "internet phonebook" and it becomes clear that Google needs to be regulated in order for anybody to be able to compete. I've written up an argument like this here: https://knuckleheads.club/the-secretaries-and-the-phonebooks/ https://knuckleheads.club/the-secretaries-and-the-phonebooks...

5 points by marban 6y ago

Hello! Zack Maril here. I've been working on this research for over two years now. I got started working on it after I read everybody on Hacker News talking about how they kept getting blocked when they were crawling websites[0] and also recent antitrust developments[1]. This has been a huge part of my life these past few years and it wouldn't have happened without the Hacker News community, so, from the bottom of my heart, thank you. If you would like to read more about this research, please check out Knuckleheads' Club[2]. I've published a lot of what I have so far there and plan on publishing more soon. You're all invited to join the club and I'll be checking this thread all day and responding to folks as best I can. [0] https://news.ycombinator.com/item?id=9612345 https://news.ycombinator.com/item?id=9612345 [1] https://news.ycombinator.com/item?id=17941804 https://news.ycombinator.com/item?id=17941804 [2] http://knuckleheads.club http://knuckleheads.club

marban 6y ago
Never heard of this initiative but kudos for bringing it up. I run a popular news site[0] and merely aggregating ~150 sources costs me upwards of $1k/m just to crawl the basics. [0] https://upstract.com/ https://upstract.com/
- knuckleheads 6y ago
  Yes!! Exactly!! Crawling is super expensive to do and it's hard to even get the money to start doing the crawling. And even then once you get started, a lot of websites block you out or only let Googlebot through the paywall. And we are launching the club today with the publishing of the NYT article, we have been pretty quiet about it so far for the last two years or so, doing the research in private.

dkjaudyeqooe 6y ago

An effective breakup of Google would entail requiring them to put their crawling infrastructure into a seperate company and require them to make it available to other businesses at a reasonable cost, or get out of the crawling business entirely. Might be tricky to implement but without something like this no one has a chance of competing with them.

knuckleheads 6y ago
Absolutely, that's direct precedence with AT&T and the Yellow Pages when the government broke up AT&T. There's a really interesting angle to this where you consider Googlebot to be an "internet phonebook" and it becomes clear that Google needs to be regulated in order for anybody to be able to compete. I've written up an argument like this here: https://knuckleheads.club/the-secretaries-and-the-phonebooks/ https://knuckleheads.club/the-secretaries-and-the-phonebooks...