3 ms·
I want to allow good spiders but not these, how do you differentiate them in access logs?
by throwaway290 4y ago
I want to allow good spiders but not these, how do you differentiate them in access logs?
- JohnFen 4y agoThat's not a use case I've put any thought into, honestly, but if I assume the "good spiders" are well-behaved, you should be able to identify them based on their user agent string. Or perhaps you can nail down what IP ranges the good spiders are coming from and allow them based on that.