4 ms·
robots.txt stops crawling - you can get indexed via other mechanisms. You want no index robots tags on all your pages and let google see those. You can use GS
by walshemj 5y ago
robots.txt stops crawling - you can get indexed via other mechanisms.
You want no index robots tags on all your pages and let google see those.
You can use GSC (Google search console) to remove a site / page from the index