robots.txt prevents crawling, but not indexing and returning as search results. Previously discussed e.g. [0]. But of course shop updates won't be seen easily a

robots.txt prevents crawling, but not indexing and returning as search results. Previously discussed e.g. [0]. But of course shop updates won't be seen easily anymore. [0] https://news.ycombinator.com/item?id=31892299 https://news.ycombinator.com/item?id=31892299