3 ms·
You end up with a lot of noise from bots and crawlers (using bogus user agents) if you're just looking at server logs.
by eli 6y ago
You end up with a lot of noise from bots and crawlers (using bogus user agents) if you're just looking at server logs.
- justinclift 6y agoAny reasonable server side processing thing will exclude the obvious bots, which almost always have some kind of "Bot" wording in their user agent header.
- eli 6y agoThere are a LOT of bots and crawlers with bogus browser user agents. Some of the bad ones you can select see indirectly in logs because they pick UAs that almost no one uses any more. Go search your logs for IE8 or Firefox <= 70.0. Most just pick a random modern User Agent though and that's awfully hard to see in server logs.
- colechristensen 6y agoYeah, but so what? There are plenty of blacklists maintained for all sorts of things, and no metric is perfect. Some really rudimentary filtering or AI methods could get you pretty good data.