3 ms·
Curl and libcurl doesn't. Wget doesn't, I think.
by hamilyon2 2y ago
Curl and libcurl doesn't. Wget doesn't, I think.
- account42 2y agoCURL isn't a bot, it's a tool. It can be part of a bot (which may or may not respect robots.txt) but it can also just act as a user agent directly for a human operator in which case it SHOULD just do what asked. Chrome doesn't follow robots.txt either for the same reason.