Yeah, nobody cares that most sites will work with it. The number of sites that require it might be small, but that number tends to include significant and notab

Yeah, nobody cares that most sites will work with it. The number of sites that require it might be small, but that number tends to include significant and notable ones - the ones that most people actually wind up wanting to crawl. Cloudflare & co make this more difficult with each passing year. OP is making a very well known day-one point about this topic. It is somewhat surprising that the library doesn't offer a way to dynamically set it. Edit: and this isn't even getting into stuff like Tls fingerprinting, header order, etc