Some folks think it's a very big issue and have released data on their findings about robots.txt files: https://knuckleheads.club/the-evidence-we-found-so-far/

Some folks think it's a very big issue and have released data on their findings about robots.txt files: https://knuckleheads.club/the-evidence-we-found-so-far/ https://knuckleheads.club/the-evidence-we-found-so-far/ We see some problems but as an established, and respectful crawler, we are usually able to overcome most blocks at Mojeek; a international web search engine with no tracking (currently 4.3 billion pages)