3 ms·

> Crawling the internet is a natural monopoly. How so? A caching proxy costs you almost nothing and will serve thousands of requests per second on ancient har

> Crawling the internet is a natural monopoly. How so? A caching proxy costs you almost nothing and will serve thousands of requests per second on ancient hardware. Actually there's never been a better time in the history of the Internet to have competing search engines since there's never been so much abundance of performance, bandwidth, and software available at historic low prices or for free.

deleted 1y ago
[deleted]
stackskipton 1y ago
Not everyone wants to deal with caching proxy because they think the load on their site under normal operations is fine if it's rendered server side.
Onavo 1y ago
In the past month there were dozens of posts about using proof of work and other methods to defeat crawlers. I don't think most websites tolerate heavy crawling in the era of Vercel/AWS's serverless "per request" and bandwidth billing.
immibis 1y ago
You don't get to tell site owners what to do. The actual facts on the ground are that they're trying to block your bot. It would be nice if they didn't block your bot, but the other, completely unnatural and advertising-driven, monopoly of hosting providers with insane per-request costs makes that impossible until they switch away.
- AlexandrB 1y ago
  They try to block your bot because Google is a monopoly and there's little to no cost for blocking everything except Google. This isn't a "natural" monopoly, it's more like Internet Explorer 6.0 and everyone designing their sites to use ActiveX and IE-specific quirks.
  - luckylion 1y ago
    One possible answer: pay them for their trouble until you provide value to them, e.g. by paying some fraction of a cent for each (document) request.
    - BobaFloutist 1y ago
      Cool, you wanna solve micropayments now or wait until we've got cold fusion rolling first...?
      - luckylion 1y ago
        You wouldn't have to make them micropayments, you can pay out once some threshold is reached. Of course, it would incentivize the sites to make you want to crawl them more, but that might be a good thing. There would be pressure on you to focus on quality over quantity, which would probably be a good thing for your product.
        BobaFloutist 1y ago
        >You wouldn't have to make them micropayments, you can pay out once some threshold is reached. Believe it or not, this is a potential solution for micropayments that has been explored.
        immibis 1y ago
        I could even pay a fixed amount to my ISP every month for a fixed amount of data transfer.
- threeseed 1y ago
  > The actual facts on the ground are that they're trying to block your bot Based on what evidence.
  - immibis 1y ago
    based on them matching the user-agent and sending you a block page? I don't know what else to tell you. It's in plain sight.
sokoloff 1y ago
Costs almost nothing, but returns even less.* There are so many other bots/scrapers out there that literally return zero that I don’t blame site owners for blocking all bots except googlebot. Would it be nice if they also allowed altruist-bot or common-crawler-bot? Maybe, but that’s their call and a lot of them have made it on a rational basis. * - or is perceived to return
- Aurornis 1y ago
  > that I don’t blame site owners for blocking all bots except googlebot. I doubt this is happening outside of a few small hobbyist websites where crawler traffic looks significant relative to human traffic. Even among those, it’s so common to move to static hosting with essentially zero cost and/or sign up for free tiers of CDNs that it’s just not worth it outside of edge cases like trying to host public-facing Gitlab instances with large projects. Even then, the ROI on setting up proper caching and rate limiting far outweighs the ROI on trying to play whack-a-mole with non-Google bots. Even if someone did go to all the lengths to try to block the majority of bots, I have a really hard time believing they wouldn’t take the extra 10 minutes to look up the other major crawlers and put those on the allow list, too. This whole argument about sites going to great lengths to block search indexers but then stopping just short of allowing a couple more of the well-known ones feels like mental gymnastics for a situation that doesn’t occur.
  - luckylion 1y ago
    Regarding allowlisting the other major crawlers: I've never seen any significant amount of traffic coming from anything but Google or Bing. There's the occasional click from one of the resellers (ecosia, brave search, duckduckgo etc), but that's about it. Yahoo? haven't seen them in ages, except in Japan. Baidu or Yandex? might be relevant if you're in their primary markets, but I've never seen them. Huawei's Petal Search? Apple Search? Nothing. Ahrefs & friends? No need to crawl _my_ website, even if I wanted to use them for competitor analysis. So practically, there's very little value in allowing those. I usually don't bother blocking them, but if my content wasn't easy to cache, I probably would.
  - fc417fc802 1y ago

deleted 1y ago

[deleted]

stackskipton 1y ago

Not everyone wants to deal with caching proxy because they think the load on their site under normal operations is fine if it's rendered server side.

Onavo 1y ago

In the past month there were dozens of posts about using proof of work and other methods to defeat crawlers. I don't think most websites tolerate heavy crawling in the era of Vercel/AWS's serverless "per request" and bandwidth billing.

immibis 1y ago

You don't get to tell site owners what to do. The actual facts on the ground are that they're trying to block your bot. It would be nice if they didn't block your bot, but the other, completely unnatural and advertising-driven, monopoly of hosting providers with insane per-request costs makes that impossible until they switch away.

AlexandrB 1y ago
They try to block your bot because Google is a monopoly and there's little to no cost for blocking everything except Google. This isn't a "natural" monopoly, it's more like Internet Explorer 6.0 and everyone designing their sites to use ActiveX and IE-specific quirks.
- luckylion 1y ago
  One possible answer: pay them for their trouble until you provide value to them, e.g. by paying some fraction of a cent for each (document) request.
  - BobaFloutist 1y ago
    Cool, you wanna solve micropayments now or wait until we've got cold fusion rolling first...?
    - luckylion 1y ago
      You wouldn't have to make them micropayments, you can pay out once some threshold is reached. Of course, it would incentivize the sites to make you want to crawl them more, but that might be a good thing. There would be pressure on you to focus on quality over quantity, which would probably be a good thing for your product.
      - BobaFloutist 1y ago
        >You wouldn't have to make them micropayments, you can pay out once some threshold is reached. Believe it or not, this is a potential solution for micropayments that has been explored.
      - immibis 1y ago
        I could even pay a fixed amount to my ISP every month for a fixed amount of data transfer.
threeseed 1y ago
> The actual facts on the ground are that they're trying to block your bot Based on what evidence.
- immibis 1y ago
  based on them matching the user-agent and sending you a block page? I don't know what else to tell you. It's in plain sight.

sokoloff 1y ago

Costs almost nothing, but returns even less.* There are so many other bots/scrapers out there that literally return zero that I don’t blame site owners for blocking all bots except googlebot. Would it be nice if they also allowed altruist-bot or common-crawler-bot? Maybe, but that’s their call and a lot of them have made it on a rational basis. * - or is perceived to return

Aurornis 1y ago
> that I don’t blame site owners for blocking all bots except googlebot. I doubt this is happening outside of a few small hobbyist websites where crawler traffic looks significant relative to human traffic. Even among those, it’s so common to move to static hosting with essentially zero cost and/or sign up for free tiers of CDNs that it’s just not worth it outside of edge cases like trying to host public-facing Gitlab instances with large projects. Even then, the ROI on setting up proper caching and rate limiting far outweighs the ROI on trying to play whack-a-mole with non-Google bots. Even if someone did go to all the lengths to try to block the majority of bots, I have a really hard time believing they wouldn’t take the extra 10 minutes to look up the other major crawlers and put those on the allow list, too. This whole argument about sites going to great lengths to block search indexers but then stopping just short of allowing a couple more of the well-known ones feels like mental gymnastics for a situation that doesn’t occur.
- luckylion 1y ago
  Regarding allowlisting the other major crawlers: I've never seen any significant amount of traffic coming from anything but Google or Bing. There's the occasional click from one of the resellers (ecosia, brave search, duckduckgo etc), but that's about it. Yahoo? haven't seen them in ages, except in Japan. Baidu or Yandex? might be relevant if you're in their primary markets, but I've never seen them. Huawei's Petal Search? Apple Search? Nothing. Ahrefs & friends? No need to crawl _my_ website, even if I wanted to use them for competitor analysis. So practically, there's very little value in allowing those. I usually don't bother blocking them, but if my content wasn't easy to cache, I probably would.
- fc417fc802 1y ago