4 ms·
The Internet is a wild place, and I reckon 90% of the complexity of a crawler is dealing with workarounds and non-compliant servers (cough www.apple.com cough).
by sph 3y ago
The Internet is a wild place, and I reckon 90% of the complexity of a crawler is dealing with workarounds and non-compliant servers (cough www.apple.com cough).
I'll have a look, thanks for the heads up.
- hasty_pudding 3y agoare you setting the headers to make the sites think it's a browser?? edit: User-agent: bernard/1.0" I bet thats going to cause issuses. Id fake a browser user agent for off domain sites.