6 ms·

Crawling the internet is a natural monopoly. Nobody wants an endless stream of bots crawling their site, so googlebot wins because they’re the dominant search e

Crawling the internet is a natural monopoly. Nobody wants an endless stream of bots crawling their site, so googlebot wins because they’re the dominant search engine. It makes sense to break that out so everyone has access to the same dataset at FRAND pricing. My heart just wants Google to burn to the ground, but my brain says this is the more reasonable approach.

hkpack 1y ago
Most of the tech is set for being a monopoly due to the negligible variable cost associated with serving a customer. Thus being even slightly in front of others is reinforced and the gap only widens.
oceanplexian 1y ago
> Crawling the internet is a natural monopoly. How so? A caching proxy costs you almost nothing and will serve thousands of requests per second on ancient hardware. Actually there's never been a better time in the history of the Internet to have competing search engines since there's never been so much abundance of performance, bandwidth, and software available at historic low prices or for free.
- deleted 1y ago
  [deleted]
- stackskipton 1y ago
  Not everyone wants to deal with caching proxy because they think the load on their site under normal operations is fine if it's rendered server side.
- Onavo 1y ago
  In the past month there were dozens of posts about using proof of work and other methods to defeat crawlers. I don't think most websites tolerate heavy crawling in the era of Vercel/AWS's serverless "per request" and bandwidth billing.
- immibis 1y ago
  You don't get to tell site owners what to do. The actual facts on the ground are that they're trying to block your bot. It would be nice if they didn't block your bot, but the other, completely unnatural and advertising-driven, monopoly of hosting providers with insane per-request costs makes that impossible until they switch away.
  - AlexandrB 1y ago
    They try to block your bot because Google is a monopoly and there's little to no cost for blocking everything except Google. This isn't a "natural" monopoly, it's more like Internet Explorer 6.0 and everyone designing their sites to use ActiveX and IE-specific quirks.
    - luckylion 1y ago
      One possible answer: pay them for their trouble until you provide value to them, e.g. by paying some fraction of a cent for each (document) request.
      - BobaFloutist 1y ago
        Cool, you wanna solve micropayments now or wait until we've got cold fusion rolling first...?
        luckylion 1y ago
        You wouldn't have to make them micropayments, you can pay out once some threshold is reached. Of course, it would incentivize the sites to make you want to crawl them more, but that might be a good thing. There would be pressure on you to focus on quality over quantity, which would probably be a good thing for your product.
        BobaFloutist 1y ago
        >You wouldn't have to make them micropayments, you can pay out once some threshold is reached. Believe it or not, this is a potential solution for micropayments that has been explored.
- sokoloff 1y ago
  Costs almost nothing, but returns even less.* There are so many other bots/scrapers out there that literally return zero that I don’t blame site owners for blocking all bots except googlebot. Would it be nice if they also allowed altruist-bot or common-crawler-bot? Maybe, but that’s their call and a lot of them have made it on a rational basis. * - or is perceived to return
  - Aurornis 1y ago
    > that I don’t blame site owners for blocking all bots except googlebot. I doubt this is happening outside of a few small hobbyist websites where crawler traffic looks significant relative to human traffic. Even among those, it’s so common to move to static hosting with essentially zero cost and/or sign up for free tiers of CDNs that it’s just not worth it outside of edge cases like trying to host public-facing Gitlab instances with large projects. Even then, the ROI on setting up proper caching and rate limiting far outweighs the ROI on trying to play whack-a-mole with non-Google bots. Even if someone did go to all the lengths to try to block the majority of bots, I have a really hard time believing they wouldn’t take the extra 10 minutes to look up the other major crawlers and put those on the allow list, too. This whole argument about sites going to great lengths to block search indexers but then stopping just short of allowing a couple more of the well-known ones feels like mental gymnastics for a situation that doesn’t occur.
    - luckylion 1y ago
      Regarding allowlisting the other major crawlers: I've never seen any significant amount of traffic coming from anything but Google or Bing. There's the occasional click from one of the resellers (ecosia, brave search, duckduckgo etc), but that's about it. Yahoo? haven't seen them in ages, except in Japan. Baidu or Yandex? might be relevant if you're in their primary markets, but I've never seen them. Huawei's Petal Search? Apple Search? Nothing. Ahrefs & friends? No need to crawl _my_ website, even if I wanted to use them for competitor analysis. So practically, there's very little value in allowing those. I usually don't bother blocking them, but if my content wasn't easy to cache, I probably would.
    - fc417fc802 1y ago
toomuchtodo 1y ago
https://commoncrawl.org/ https://commoncrawl.org/ This is similar to the natural monopoly of root DNS servers (managed as a public good). There is no reason more money couldn't go into either Common Crawl, or something like it. The Internet Archive can persist the data for ~$2/GB in perpetuity (although storing it elsewhere is also fine imho) as the storage system of last resort. How you provide access to this data is, I argue, similar to how access to science datasets is provided by custodian institutions (examples would be NOAA, CERN, etc). Build foundations on public goods, very broadly speaking (think OSI model, but for entire systems). This helps society avoid the grasp of Big Tech and their endless desire to build moats for value capture.
- b112 1y ago
  One problem, it leaves one place to censor. I agree that each front end should do it, but you can bet it will be a core service.
- vasco 1y ago
  > The Internet Archive can persist the data for ~$2/GB in perpetuity No they can't but do you have a source?
  - toomuchtodo 1y ago
    https://help.archive.org/help/archive-org-information/ https://help.archive.org/help/archive-org-information/ and first hand conversations with their engineering team > We estimate that permanent storage costs us approximately $2.00US per gigabyte. https://webservices.archive.org/pages/vault/ https://webservices.archive.org/pages/vault/ > Vault offers a low-cost pricing model based on a one-time price per-gigabyte/terabyte for data deposited in the system, with no additional annual storage fees or data egress costs. https://blog.dshr.org/2017/08/economic-model-of-long-term-storage.html https://blog.dshr.org/2017/08/economic-model-of-long-term-st...
    - dmoy 1y ago
      What's the read throughout to get the data back out, and does it scale to what you'd need to have N search indexes building on top of this shared crawl?
mattmaroon 1y ago
Are sites really that averse to having a few more crawlers than they already do? It would seem that it’s only a monopoly insofar as it’s really expensive to do and almost nobody else thinks they can recoup the cost.
- robinsonb5 1y ago
  A "few" more would be fine - but the sheer scale of the malicious AI training bot crawling that's happening now is enough to cause real availability problems (and expense) for numerous sites. One web forum I regularly read went through a patch a few months ago where it was unavailable for about 90% of the time due to being hammered by crawlers. It's only up again now because the owner managed to find a way to block them that hasn't yet been circumvented. So it's easy to see why people would allow googlebot and little else.
- natebc 1y ago
  A few? We routinely are fighting off hundreds of bots at any moment. Thousands and Thousands per day, easily. US, China, Brazil from hundreds of different IPs, dozens of different (and falsified!) user agents all ignoring robots.txt and pushing over services that are needed by human beings trying to get work done. EDIT: Just checked our anubis stats for the last 24h CHALLENGE: 829,586 DENY: 621,462 ALLOW: 96,810 This is with a pretty aggressive "DENY" rule for a lot of the AI related bots and on 2 pretty small sites at $JOB. We have hundreds, if not thousands of different sites that aren't protected by Anubis (yet). Anubis and efforts like it are a xesend for companies that don't want to pay off Cloudflare or some other "security" company peddling a WAF.
  - zrm 1y ago
    This seems like two different issues. One is, suppose there are a thousand search engine bots. Then what you want is some standard facility to say "please give me a list of every resources on this site that has changed since <timestamp>" so they can each get a diff from the last time they crawled your site. Uploading each resource on the site to each of a thousand bots once is going to be irrelevant to a site serving millions of users (because it's a trivial percentage) and to a site with a small amount of content (because it's a small absolute number), which together constitute the vast majority of all sites. The other is, there are aggressive bots that will try to scrape your entire site five times a day even if nothing has changed and ignore robots.txt. But then you set traps like disallowing something in robots.txt and then ban anything that tries to access it, which doesn't affect legitimate search engine crawlers because they respect robots.txt.
wslh 1y ago
> so googlebot wins because they’re the dominant search engine. I think it's also important to highlight that sites explicitly choose which bots to allow in their robots.txt files, prioritizing Google which reinforces its position as the de-facto monopoly. Even when other bots are technically able to crawl them.
shadowgovt 1y ago
Of all the bad ideas I've heard of where to slice Google to break it up, this... Is actually the best idea. The indexer, without direct Google influence, is primarily incentivized to play nice with site administrators. This gives them reasons to improve consideration of both network integrity and privacy concerns (though Google has generally been good about these things, I think the damage is done regarding privacy that the brand name is toxic, regardless of the behaviors).
tananaev 1y ago
Google search is a monopoly not because of crawling. It's because of the all the data it knows about website stats and user behavior. Original Google idea of ranking based on links doesn't work because it's too easily gamed. You have to know what websites are good based on user preferences and that's where you need to have data. It's impossible to build anything similar to Google without access to large amounts of user data.
- AtlasBarfed 1y ago
  Page ranking sounds like a perfect application of artificial intelligence. If China can apply it for total information awareness on their population, Google can apply it on page reliability
  - fc417fc802 1y ago
    I'm fairly certain many people have already tried to apply magical AI pixie dust to this problem. Presumably it isn't so simple in practice.
- luckylion 1y ago
  Sounds like you're implying that they are using Google Analytics to feed their ranking, but that's much easier to game than links are. User-signals on SERP clicks? There's a niche industry supplying those to SEOs (I've seen it a few times, I haven't seen it have any reliable impact).
Aurornis 1y ago
> Crawling the internet is a natural monopoly. Nobody wants an endless stream of bots crawling their site, Companies want traffic from any source they can get. They welcome every search engine crawler that comes along because every little exposure translates to incremental chances at revenue or growing audience. I doubt many people are doing things to allow Googlebot but also ban other search crawlers. > My heart just wants Google to burn to the ground I think there’s a lot of that in this thread and it’s opening the door to some mental gymnastics like the above claim about Google being the only crawler allowed to index the internet.
- nulld3v 1y ago
  > I doubt many people are doing things to allow Googlebot but also ban other search crawlers. Sadly this is just not the case.[1][2] Google knows this too so they explicitly crawl from a specific IP range that they publish.[3] I also know this, because I had a website that blocked any bots outside of that IP range. We had honeypot links (hidden to humans via CSS) that insta-banned any user or bot that clicked/fetched them. User-Agent from curl, wget, or any HTTP lib = insta-ban. Crawling links sequentially across multiple IPs = all banned. Any signal we found that indicated you were not a human using a web browser = ban. We were listed on Google and never had traffic issues. [1] https://onescales.com/blogs/main/the-bot-blocklist https://onescales.com/blogs/main/the-bot-blocklist [2] Chart in the middle of this page: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo... (note: Google-Extended != Googlebot) [3] https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/...
deleted 1y ago
[deleted]
1vuio0pswjnm7 1y ago
CommonCrawl is not a vlaid comparison. Most robots.txt target CCBot.

hkpack 1y ago

Most of the tech is set for being a monopoly due to the negligible variable cost associated with serving a customer. Thus being even slightly in front of others is reinforced and the gap only widens.

oceanplexian 1y ago

> Crawling the internet is a natural monopoly. How so? A caching proxy costs you almost nothing and will serve thousands of requests per second on ancient hardware. Actually there's never been a better time in the history of the Internet to have competing search engines since there's never been so much abundance of performance, bandwidth, and software available at historic low prices or for free.

deleted 1y ago
[deleted]
stackskipton 1y ago
Not everyone wants to deal with caching proxy because they think the load on their site under normal operations is fine if it's rendered server side.
Onavo 1y ago
In the past month there were dozens of posts about using proof of work and other methods to defeat crawlers. I don't think most websites tolerate heavy crawling in the era of Vercel/AWS's serverless "per request" and bandwidth billing.
immibis 1y ago
You don't get to tell site owners what to do. The actual facts on the ground are that they're trying to block your bot. It would be nice if they didn't block your bot, but the other, completely unnatural and advertising-driven, monopoly of hosting providers with insane per-request costs makes that impossible until they switch away.
- AlexandrB 1y ago
  They try to block your bot because Google is a monopoly and there's little to no cost for blocking everything except Google. This isn't a "natural" monopoly, it's more like Internet Explorer 6.0 and everyone designing their sites to use ActiveX and IE-specific quirks.
  - luckylion 1y ago
    One possible answer: pay them for their trouble until you provide value to them, e.g. by paying some fraction of a cent for each (document) request.
    - BobaFloutist 1y ago
      Cool, you wanna solve micropayments now or wait until we've got cold fusion rolling first...?
      - luckylion 1y ago
        You wouldn't have to make them micropayments, you can pay out once some threshold is reached. Of course, it would incentivize the sites to make you want to crawl them more, but that might be a good thing. There would be pressure on you to focus on quality over quantity, which would probably be a good thing for your product.
        BobaFloutist 1y ago
        >You wouldn't have to make them micropayments, you can pay out once some threshold is reached. Believe it or not, this is a potential solution for micropayments that has been explored.
sokoloff 1y ago
Costs almost nothing, but returns even less.* There are so many other bots/scrapers out there that literally return zero that I don’t blame site owners for blocking all bots except googlebot. Would it be nice if they also allowed altruist-bot or common-crawler-bot? Maybe, but that’s their call and a lot of them have made it on a rational basis. * - or is perceived to return
- Aurornis 1y ago
  > that I don’t blame site owners for blocking all bots except googlebot. I doubt this is happening outside of a few small hobbyist websites where crawler traffic looks significant relative to human traffic. Even among those, it’s so common to move to static hosting with essentially zero cost and/or sign up for free tiers of CDNs that it’s just not worth it outside of edge cases like trying to host public-facing Gitlab instances with large projects. Even then, the ROI on setting up proper caching and rate limiting far outweighs the ROI on trying to play whack-a-mole with non-Google bots. Even if someone did go to all the lengths to try to block the majority of bots, I have a really hard time believing they wouldn’t take the extra 10 minutes to look up the other major crawlers and put those on the allow list, too. This whole argument about sites going to great lengths to block search indexers but then stopping just short of allowing a couple more of the well-known ones feels like mental gymnastics for a situation that doesn’t occur.
  - luckylion 1y ago
    Regarding allowlisting the other major crawlers: I've never seen any significant amount of traffic coming from anything but Google or Bing. There's the occasional click from one of the resellers (ecosia, brave search, duckduckgo etc), but that's about it. Yahoo? haven't seen them in ages, except in Japan. Baidu or Yandex? might be relevant if you're in their primary markets, but I've never seen them. Huawei's Petal Search? Apple Search? Nothing. Ahrefs & friends? No need to crawl _my_ website, even if I wanted to use them for competitor analysis. So practically, there's very little value in allowing those. I usually don't bother blocking them, but if my content wasn't easy to cache, I probably would.
  - fc417fc802 1y ago

toomuchtodo 1y ago

https://commoncrawl.org/ https://commoncrawl.org/ This is similar to the natural monopoly of root DNS servers (managed as a public good). There is no reason more money couldn't go into either Common Crawl, or something like it. The Internet Archive can persist the data for ~$2/GB in perpetuity (although storing it elsewhere is also fine imho) as the storage system of last resort. How you provide access to this data is, I argue, similar to how access to science datasets is provided by custodian institutions (examples would be NOAA, CERN, etc). Build foundations on public goods, very broadly speaking (think OSI model, but for entire systems). This helps society avoid the grasp of Big Tech and their endless desire to build moats for value capture.

b112 1y ago
One problem, it leaves one place to censor. I agree that each front end should do it, but you can bet it will be a core service.
vasco 1y ago
> The Internet Archive can persist the data for ~$2/GB in perpetuity No they can't but do you have a source?
- toomuchtodo 1y ago
  https://help.archive.org/help/archive-org-information/ https://help.archive.org/help/archive-org-information/ and first hand conversations with their engineering team > We estimate that permanent storage costs us approximately $2.00US per gigabyte. https://webservices.archive.org/pages/vault/ https://webservices.archive.org/pages/vault/ > Vault offers a low-cost pricing model based on a one-time price per-gigabyte/terabyte for data deposited in the system, with no additional annual storage fees or data egress costs. https://blog.dshr.org/2017/08/economic-model-of-long-term-storage.html https://blog.dshr.org/2017/08/economic-model-of-long-term-st...
  - dmoy 1y ago
    What's the read throughout to get the data back out, and does it scale to what you'd need to have N search indexes building on top of this shared crawl?

mattmaroon 1y ago

Are sites really that averse to having a few more crawlers than they already do? It would seem that it’s only a monopoly insofar as it’s really expensive to do and almost nobody else thinks they can recoup the cost.

robinsonb5 1y ago
A "few" more would be fine - but the sheer scale of the malicious AI training bot crawling that's happening now is enough to cause real availability problems (and expense) for numerous sites. One web forum I regularly read went through a patch a few months ago where it was unavailable for about 90% of the time due to being hammered by crawlers. It's only up again now because the owner managed to find a way to block them that hasn't yet been circumvented. So it's easy to see why people would allow googlebot and little else.
natebc 1y ago
A few? We routinely are fighting off hundreds of bots at any moment. Thousands and Thousands per day, easily. US, China, Brazil from hundreds of different IPs, dozens of different (and falsified!) user agents all ignoring robots.txt and pushing over services that are needed by human beings trying to get work done. EDIT: Just checked our anubis stats for the last 24h CHALLENGE: 829,586 DENY: 621,462 ALLOW: 96,810 This is with a pretty aggressive "DENY" rule for a lot of the AI related bots and on 2 pretty small sites at $JOB. We have hundreds, if not thousands of different sites that aren't protected by Anubis (yet). Anubis and efforts like it are a xesend for companies that don't want to pay off Cloudflare or some other "security" company peddling a WAF.
- zrm 1y ago
  This seems like two different issues. One is, suppose there are a thousand search engine bots. Then what you want is some standard facility to say "please give me a list of every resources on this site that has changed since <timestamp>" so they can each get a diff from the last time they crawled your site. Uploading each resource on the site to each of a thousand bots once is going to be irrelevant to a site serving millions of users (because it's a trivial percentage) and to a site with a small amount of content (because it's a small absolute number), which together constitute the vast majority of all sites. The other is, there are aggressive bots that will try to scrape your entire site five times a day even if nothing has changed and ignore robots.txt. But then you set traps like disallowing something in robots.txt and then ban anything that tries to access it, which doesn't affect legitimate search engine crawlers because they respect robots.txt.

wslh 1y ago

> so googlebot wins because they’re the dominant search engine. I think it's also important to highlight that sites explicitly choose which bots to allow in their robots.txt files, prioritizing Google which reinforces its position as the de-facto monopoly. Even when other bots are technically able to crawl them.

shadowgovt 1y ago

Of all the bad ideas I've heard of where to slice Google to break it up, this... Is actually the best idea. The indexer, without direct Google influence, is primarily incentivized to play nice with site administrators. This gives them reasons to improve consideration of both network integrity and privacy concerns (though Google has generally been good about these things, I think the damage is done regarding privacy that the brand name is toxic, regardless of the behaviors).

tananaev 1y ago

Google search is a monopoly not because of crawling. It's because of the all the data it knows about website stats and user behavior. Original Google idea of ranking based on links doesn't work because it's too easily gamed. You have to know what websites are good based on user preferences and that's where you need to have data. It's impossible to build anything similar to Google without access to large amounts of user data.

AtlasBarfed 1y ago
Page ranking sounds like a perfect application of artificial intelligence. If China can apply it for total information awareness on their population, Google can apply it on page reliability
- fc417fc802 1y ago
  I'm fairly certain many people have already tried to apply magical AI pixie dust to this problem. Presumably it isn't so simple in practice.
luckylion 1y ago
Sounds like you're implying that they are using Google Analytics to feed their ranking, but that's much easier to game than links are. User-signals on SERP clicks? There's a niche industry supplying those to SEOs (I've seen it a few times, I haven't seen it have any reliable impact).

Aurornis 1y ago

> Crawling the internet is a natural monopoly. Nobody wants an endless stream of bots crawling their site, Companies want traffic from any source they can get. They welcome every search engine crawler that comes along because every little exposure translates to incremental chances at revenue or growing audience. I doubt many people are doing things to allow Googlebot but also ban other search crawlers. > My heart just wants Google to burn to the ground I think there’s a lot of that in this thread and it’s opening the door to some mental gymnastics like the above claim about Google being the only crawler allowed to index the internet.

nulld3v 1y ago
> I doubt many people are doing things to allow Googlebot but also ban other search crawlers. Sadly this is just not the case.[1][2] Google knows this too so they explicitly crawl from a specific IP range that they publish.[3] I also know this, because I had a website that blocked any bots outside of that IP range. We had honeypot links (hidden to humans via CSS) that insta-banned any user or bot that clicked/fetched them. User-Agent from curl, wget, or any HTTP lib = insta-ban. Crawling links sequentially across multiple IPs = all banned. Any signal we found that indicated you were not a human using a web browser = ban. We were listed on Google and never had traffic issues. [1] https://onescales.com/blogs/main/the-bot-blocklist https://onescales.com/blogs/main/the-bot-blocklist [2] Chart in the middle of this page: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo... (note: Google-Extended != Googlebot) [3] https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot https://developers.google.com/search/docs/crawling-indexing/...

deleted 1y ago

[deleted]

1vuio0pswjnm7 1y ago

CommonCrawl is not a vlaid comparison. Most robots.txt target CCBot.