4 ms·
Looking at my logs I only see 5 requests (maybe I'm grepping for the wrong thing?) but the UI shows many more links checked. grep www.drlinkcheck.com /var/l
by symkat 15y ago
Looking at my logs I only see 5 requests (maybe I'm grepping for the wrong thing?) but the UI shows many more links checked.
grep www.drlinkcheck.com /var/log/nginx/access.log | wc -l
5
All of those have the user agent 'Mozilla/5.0 (compatible; dlcbot/0.1; +http://www.drlinkcheck.com/) http://www.drlinkcheck.com/)
Also, I think it would be useful to add a section in 'About' that says the user agent the crawler will use.
- awulf 15y agoThat's strange, your grep looks fine to me. Is it possible that you have configured nginx to use different log files for different locations (or maybe don't log at all)? Other than that, I don't have any good explanation.
- symkat 15y agoOops, I'm a dumby and hadn't had enough coffee yet. That file was just logging 301s, and the actual target was going elsewhere, which has an appropriate number of requests. Overall pretty awesome, thank you!