3 ms·

Being only tangentially familiar with the indexing bots…what legal barriers are in place to prevent a competitor from impersonating a google indexing bot? Is i

Being only tangentially familiar with the indexing bots…what legal barriers are in place to prevent a competitor from impersonating a google indexing bot? Is it just a matter of the google bot originates from x subnet so that’s the only one webmasters allow? What’s to stop a competitor from running their own bot but sending user-agent: totally-the-google-indexing-bot-and-not-a-competitor?

jiggawatts 3y ago
Source addresses: the Google bot traffic comes from a small set of Google-owned IP address blocks. Third parties bake this into things like Web Application Firewall (WAF) rules. For example, Azure App Gateway WAF has a policy category for “known bots” which includes Google but excludes your tiny AI startup. It’s a moat built by giant corporations to keep tiny players in their place. Google "helpfully" publishes their bot source IP addresses: https://developers.google.com/static/search/apis/ipranges/googlebot.json https://developers.google.com/static/search/apis/ipranges/go... AWS also provides named rules such as "bot:name:googlebot": https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-bot.html https://docs.aws.amazon.com/waf/latest/developerguide/aws-ma...
- creato 3y ago
  I agree it’s a moat, but why would azure restrain competition from google? I think it’s just yet another example of an anti-abuse collateral damage, like email anti-spam blocking small unknown servers or Cloudflare blocking unknown IPs.
  - whatshisface 3y ago
    Moats are anticompetitive, that's the entire idea behind why founders want them. Who else would a moat keep away, customers? ;)
    - kazinator 3y ago
      Customers are kept in by moats.
      - whatshisface 3y ago
        That's lock-in, moats are things that keep your competitors out. Source: https://www.investopedia.com/ask/answers/05/economicmoat.asp https://www.investopedia.com/ask/answers/05/economicmoat.asp
  - sokoloff 3y ago
    Because Azure customers want Google to be able to index their sites. Would you host your e-commerce or social media site on a cloud provider that blocked Googlebot? It's not Microsoft giving Google a handout out of the goodness of their hearts; it's Azure customers demanding that functionality. (Those Azure customers also don't care about a random little search startup and probably don't want to pay any egress fees to serve traffic to it.) AWS offers the same: https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control-example-allow-verified-bots.html https://docs.aws.amazon.com/waf/latest/developerguide/waf-bo... Oracle as well: https://docs.oracle.com/en-us/iaas/Content/WAF/Bot/good_bot_allowlist.htm https://docs.oracle.com/en-us/iaas/Content/WAF/Bot/good_bot_... Akamai as well: https://www.akamai.com/products/bot-manager https://www.akamai.com/products/bot-manager
- nova22033 3y ago
  For example, Azure App Gateway WAF has a policy category for “known bots” So why should we take Satya's testimony seriously? Azure is a Microsoft product and if what you're saying is true, Microsoft is part of the problem.
  - sokoloff 3y ago
    Almost surely that Azure customers want that. https://news.ycombinator.com/item?id=37746480 https://news.ycombinator.com/item?id=37746480

jiggawatts 3y ago

Source addresses: the Google bot traffic comes from a small set of Google-owned IP address blocks. Third parties bake this into things like Web Application Firewall (WAF) rules. For example, Azure App Gateway WAF has a policy category for “known bots” which includes Google but excludes your tiny AI startup. It’s a moat built by giant corporations to keep tiny players in their place. Google "helpfully" publishes their bot source IP addresses: https://developers.google.com/static/search/apis/ipranges/googlebot.json https://developers.google.com/static/search/apis/ipranges/go... AWS also provides named rules such as "bot:name:googlebot": https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-bot.html https://docs.aws.amazon.com/waf/latest/developerguide/aws-ma...

creato 3y ago
I agree it’s a moat, but why would azure restrain competition from google? I think it’s just yet another example of an anti-abuse collateral damage, like email anti-spam blocking small unknown servers or Cloudflare blocking unknown IPs.
- whatshisface 3y ago
  Moats are anticompetitive, that's the entire idea behind why founders want them. Who else would a moat keep away, customers? ;)
  - kazinator 3y ago
    Customers are kept in by moats.
    - whatshisface 3y ago
      That's lock-in, moats are things that keep your competitors out. Source: https://www.investopedia.com/ask/answers/05/economicmoat.asp https://www.investopedia.com/ask/answers/05/economicmoat.asp
- sokoloff 3y ago
  Because Azure customers want Google to be able to index their sites. Would you host your e-commerce or social media site on a cloud provider that blocked Googlebot? It's not Microsoft giving Google a handout out of the goodness of their hearts; it's Azure customers demanding that functionality. (Those Azure customers also don't care about a random little search startup and probably don't want to pay any egress fees to serve traffic to it.) AWS offers the same: https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control-example-allow-verified-bots.html https://docs.aws.amazon.com/waf/latest/developerguide/waf-bo... Oracle as well: https://docs.oracle.com/en-us/iaas/Content/WAF/Bot/good_bot_allowlist.htm https://docs.oracle.com/en-us/iaas/Content/WAF/Bot/good_bot_... Akamai as well: https://www.akamai.com/products/bot-manager https://www.akamai.com/products/bot-manager
nova22033 3y ago
For example, Azure App Gateway WAF has a policy category for “known bots” So why should we take Satya's testimony seriously? Azure is a Microsoft product and if what you're saying is true, Microsoft is part of the problem.
- sokoloff 3y ago
  Almost surely that Azure customers want that. https://news.ycombinator.com/item?id=37746480 https://news.ycombinator.com/item?id=37746480