3 ms·
Yep, just make sure to add them to your robots.txt file so that only the bad robots are harmed.
by RobKohr 2y ago
Yep, just make sure to add them to your robots.txt file so that only the bad robots are harmed.
- ethan_smith 2y agoTemporary solution and only works if only some of us are doing it. What if these bots have a "manager" LLM agent that takes the decision on what pages to scrape?