The protocol is also that you send a valid, non-faked User-Agent: "The User-Agent request-header field contains information about the user agent originating th

The protocol is also that you send a valid, non-faked User-Agent: "The User-Agent request-header field contains information about the user agent originating the request. This is for [...] the tracing of protocol violations [...]. User agents SHOULD include this field with requests" Many scrapers disregard this part of the protocol. Of course, whether a headless browser should send a different UA is an interesting question. https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html