4 ms·
Recently I needed to block some scrapers to execessive load on a server, and here are some that I identified: BOTS=( "semrushbot" "petalbot" "aliyunsecbo
by ofrzeta 9mo ago
Recently I needed to block some scrapers to execessive load on a server, and here are some that I identified:
BOTS=(
"semrushbot"
"petalbot"
"aliyunsecbot"
"amazonbot"
"claudebot"
"thinkbot"
"perplexitybot"
"openai.com/bot"
)
This was really just emergency blocking and it included more than 1500 IP addresses.
Here's Amazon's page about their bot with more information including IP addresses
https://developer.amazon.com/amazonbot https://developer.amazon.com/amazonbot