Copying my comment here for additional discussion: Worth noting that most of these bots are 'good bots' (i.e. they will obey robots.txt). So you can avoid the

Copying my comment here for additional discussion: Worth noting that most of these bots are 'good bots' (i.e. they will obey robots.txt). So you can avoid the nginx resource usage entirely by adding suitable robots.txt entries. I think using nginx tests like this could have negative effects on showing OpenGraph metadata (including images). If choosing this approach however I would probably respond with a 403 code to mark forbidden as bots are more likely to continue making attempts if they think the server would come back online.