3 ms·
Google can afford to respect robots.txt because it has a monopoly on search and nobody would consider actually blocking them in said robots.txt anyway. SerpApi
by Nextgrid 10mo ago
Google can afford to respect robots.txt because it has a monopoly on search and nobody would consider actually blocking them in said robots.txt anyway.
SerpApi doesn't have that privilege.
- bitpush 10mo agobut SerpApi is not scraping websites, it is sending malicoius requests to google.com.
- Nextgrid 10mo agoSerpApi is scraping Google. The "maliciousness" if the requests is a matter of perspective. Of course Google considers it malicious; that doesn't necessarily make it true.
- xnx 10mo agoGoogle has respected robots.txt from the start.
- skybrian 10mo agoSome domains do block Google, often partially. There are some statistics here: https://radar.cloudflare.com/ai-insights#ai-user-agents-found-in-robotstxt https://radar.cloudflare.com/ai-insights#ai-user-agents-foun...