5 ms·They do ignore robots.txt sometimes, the pages just font end up in serps.by angry-hacker 10y agoThey do ignore robots.txt sometimes, the pages just font end up in serps.