3 ms·
How do you control for the faked user agents scrapers use? A bad scraper will be much more common in your logs than a well-behaved browser, after all.
by jeroenhd 4y ago
How do you control for the faked user agents scrapers use? A bad scraper will be much more common in your logs than a well-behaved browser, after all.
- stayml 4y agoI filter out any user agents that are invalid, but there's no way to see which are real or faked. The access logs include the useragent of every single site visitor - not only errors/bad actors.