3 ms·
wget ignores robots.txt outside of recursive mode. I think it's correct to do so, and I think an AI loading a handful of pages in response to a command should
by Dylan16807 11d ago
wget ignores robots.txt outside of recursive mode. I think it's correct to do so, and I think an AI loading a handful of pages in response to a command should be about the same.