4 ms·

> Switching costs for a search engine are pretty damn low. Are they, though? I seem to recall conversations here where some larger sites were only allowing goo

> Switching costs for a search engine are pretty damn low. Are they, though? I seem to recall conversations here where some larger sites were only allowing googlebot to crawl their site in robots.txt rules. Their thinking was that the vast majority of traffic is coming from google, and it wasn't worth it to have multiple also-ran search engines crawling their site frequently. There's a case to be made that search engines are actually a natural monopoly.

type0 6y ago
> Their thinking was that the vast majority of traffic is coming from google, and it wasn't worth it to have multiple also-ran search engines crawling their site frequently. There's a case to be made that search engines are actually a natural monopoly. The other case to be made that people doing this sort of decisions should loose their jobs.
- judge2020 6y ago
  Why?
zaroth 6y ago
I meant end-user switching costs, but you raise good points. Barrier to entry and startup costs to a competitor have never been lower. I believe recent court wins make “unauthorized” scraping more openly available, and anti-trust regulations should look at allowing competitors to ignore a Google-only robots.txt as long as reasonable throttling is enforced.
- dj_mc_merlin 6y ago
  Is it not already legal?
Guest19023892 6y ago
I blocked a number of non Google search bots on one of my smaller sites the other day, increasing the number of bots I have blocked from about 20 to 30. That reduced the load on my database server by over 80%. It's a small site that only generates about $5/day from AdSense, so I need to keep the expenses down on it, and I don't want all of the revenue paying for bots that don't bring in any significant traffic. If I didn't block any search bots, I bet the bots would make up at least 95% of the traffic. On my larger sites I don't mind serving a million page views a day to random search bots, but on that smaller one it doesn't make sense.
escape_goat 6y ago
Web crawling is a natural monopoly, and perhaps should be treated as such. That's easily compartmentalized from search itself.

type0 6y ago

> Their thinking was that the vast majority of traffic is coming from google, and it wasn't worth it to have multiple also-ran search engines crawling their site frequently. There's a case to be made that search engines are actually a natural monopoly. The other case to be made that people doing this sort of decisions should loose their jobs.

judge2020 6y ago
Why?

zaroth 6y ago

I meant end-user switching costs, but you raise good points. Barrier to entry and startup costs to a competitor have never been lower. I believe recent court wins make “unauthorized” scraping more openly available, and anti-trust regulations should look at allowing competitors to ignore a Google-only robots.txt as long as reasonable throttling is enforced.

dj_mc_merlin 6y ago
Is it not already legal?

Guest19023892 6y ago

I blocked a number of non Google search bots on one of my smaller sites the other day, increasing the number of bots I have blocked from about 20 to 30. That reduced the load on my database server by over 80%. It's a small site that only generates about $5/day from AdSense, so I need to keep the expenses down on it, and I don't want all of the revenue paying for bots that don't bring in any significant traffic. If I didn't block any search bots, I bet the bots would make up at least 95% of the traffic. On my larger sites I don't mind serving a million page views a day to random search bots, but on that smaller one it doesn't make sense.

escape_goat 6y ago

Web crawling is a natural monopoly, and perhaps should be treated as such. That's easily compartmentalized from search itself.