2 ms·

Hope companies obey robots.txt.

• 0xEF 2y ago
  I hate this answer but have to agree with it. That we are in a position where small content creators are at the mercy of giants who regularly ignore the rules tells me the Internet is a lost cause. I am not even sure how we start to take back whatever power we can and think about this every time I publish an article to my tiny tech blog, which is super discouraging.
• 7373737373 2y ago
  They don't. Also, they use user agent strings used by other bots (e.g. Googlebot).