4 ms·
DuckDuckGo returns 500 if user agent starts with “Java”
- oh_no_not_again 6y agoI wonder why. My guess is to prevent bots from aggressively scraping site generating useless load, but I failed to find any other user agent strings that lead to this behavior. Any suggestions?
- verdverm 6y agoCould be an actual bug, have you reported it to them?
- oh_no_not_again 6y agoNah, first I got very surprised, then did this post, then vent to sleep. Just tweeted at them.
- duskwuff 6y agoBecause it means you're a bot with a default user agent, and they've probably had some bad experiences with those. Well-behaved bots use descriptive user agents.
- oh_no_not_again 6y agoRight. But why Java default user agent, and not Python request's, libwww-perl or curl itself? Is there disproportionately more bad bots written in Java? This is not-intuitive to me, so that's why I posted this :)