3 ms·
I would give insane kudos to the first person to implement this, or point me at one which already exists: I want an nginx module that allows a crawler once per
by maaku 13y ago
I would give insane kudos to the first person to implement this, or point me at one which already exists:
I want an nginx module that allows a crawler once per specified period (per day or per week, I would imagine, but configurable is better). That is to say, it allows the bot to finish its crawl, then bans that IP/User-Agent for the specified duration.