3 ms·
My logs tell me that Google ignores my robots.txt.
by forinti 9mo ago
My logs tell me that Google ignores my robots.txt.
- CDRdude 9mo agoIsn’t it somewhat likely that a lot of shady crawlers pretend to be a google bot with their user agent?
- ipaddr 9mo agoIf you check the ips they belong to Google. Plus if you run adsense google with ignore crawler rules and visit the page from google ips and from some shady ip. Wonder if it is the same for sites using Analytics.
- bflesch 9mo agoWhy you still have the idea in your head that they play by the rules. With the current administration they have been empowered to extract maximum value from us. In the early days of smartphone use, Google and Facebook uploaded contact lists of every single smartphone user to their servers.
- cookiengineer 9mo agoGotta dig -x to find out if they really are from Google
- 0x1ch 9mo agoI was going to say, this is an absolute win for everyone with a personal site they have been trying to prevent Google from crawling if true.