4 ms·If one doesn’t want LLMs to scrape data and knows the LLMs will be ignoring the robots.txt file.by Diti 2y agoIf one doesn’t want LLMs to scrape data and knows the LLMs will be ignoring the robots.txt file.