3 ms·

I feel like all I need is a good script to filter out bots and spiders. I could take the rest from there.

• luxurytent 6y ago
  This is an interesting thought. I wonder if I could pre process my log files to get rid of this noise ahead of piping to a log analyzer.
  • girzel 6y ago
    Or even a logging module in the web server itself...
  • Carpetsmoker 6y ago
    I worked on a library to do this for my own analytics thing[1]; adding a CLI so you can filter bots from logfiles isn't too hard: just need to parse the log lines to a http.Request{} with the correct fields filled in (mostly User-Agent, but also looks at IP address). One of the signals is from various JS properties though (navigator.webdriver, window._phantom), so you'll miss out on that. I have some other ideas as well. Either way, it's better than nothing? https://github.com/zgoat/isbot https://github.com/zgoat/isbot
    • luxurytent 6y ago
      This is cool! I've bookmarked it for future review. Thanks for sharing
  • nicbou 6y ago
    You could do this with nginx for sure. I don't know what the impact on performance would be.