3 ms·
I don't know if scrapy handles this, but I've run into issues with sites fingerprinting my browser. Proxies help, but there are other ways to identify site visi
by sram1337 8y ago
I don't know if scrapy handles this, but I've run into issues with sites fingerprinting my browser. Proxies help, but there are other ways to identify site visitors aside from IP addresses.
- JeanMarcS 8y agoRotating user agents ?
- dredmorbius 8y agoMinimally effective, or outright detrimental, in my own limited testing.
- jaytaylor 8y agoSome proxies will also include your IP address in the form of an "X-Forwarded-For" header.
- dingaling 8y agoCan't you use a service like Panopticlick to 'tune' the browser fingerprint to a majority percentile? Particularly if running in clean VM images.