3 ms·

I tried with a nonexistent path, here are the user agents I got: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.

• moneywoes 2y ago
  what sort of evasion technique is being used here, random user agents on each attempt?
  • CaptainOfCoit 2y ago
    > what sort of evasion technique is being used here Using a user-agent that looks like a desktop user-agent, rather than including the name of the actual project/product that is being used. I understand why though, plenty of websites block anything that looks like non-desktop/mobile user-agents, so makes sense. Besides the pragmatic reason, I also agree with menacingly that it doesn't matter, people should be able to use whatever user-agent they want to access the content you've put publicly on the internet.
• menacingly 2y ago
  the more reasonable assumption is that it uses a third party crawl service. but still, it doesn't matter, it's acting on behalf of a user, and you aren't entitled to know what software your users run. One consequence of putting stuff on the public internet is that it's like public.