2 ms·
Given these are trivially forged, presumably they aren't really using a Mac for scraping, right? Just to elicit a 'standard' end user response from the server?
by jcims 10mo ago
Given these are trivially forged, presumably they aren't really using a Mac for scraping, right? Just to elicit a 'standard' end user response from the server?
>useragent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; robots.txt;
- benjojo12 10mo agothe ip address the this comes from is a OpenAI search bot range: > "ipv4Prefix": "74.7.175.128/25" from https://openai.com/searchbot.json https://openai.com/searchbot.json
- Hrun0 10mo agoYes, it is very common to change your useragent for web scraping. Mainly because there are websites which will block you just based on that alone
- snowwrestler 10mo agoRight. Crawler user agent strings in general tend to include all sorts of legacy stuff for compatibility. This actually is a well-behaved crawler user agent because it identifies itself at the end.
- deathanatos 10mo ago… the UA is malformed, even. Makes me want to reconfigure my servers to just drop such traffic. If you can't be arsed to send a well-formed UA, I have doubts that you'll obey other conventions like robots.txt.