3 ms·
For what it's worth, Brave Search lies about their User Agent too. I found it fishy as well, but they claim that many websites only allow Googlebot to crawl and
by AlienRobot 2y ago
For what it's worth, Brave Search lies about their User Agent too. I found it fishy as well, but they claim that many websites only allow Googlebot to crawl and ban other UAs. I remember searching for alternative search engines and finding an article that said most new engines face this exact problem: they can't crawl because any unusual bots are blocked.
I have tried programming scrappers in the past and one thing I noticed is that there doesn't seem to be a guide in how to make a "good" bot, since there are so few bots with legitimate use cases. Most people use Chrome, too. So I guess now UA is pointless as the only valid UA is going to be Chrome or Googlebot.