I used to have what was basically a spider trap on my website while learning Django. The truth is that every bot that found it hammered my site with tens of tho

I used to have what was basically a spider trap on my website while learning Django. The truth is that every bot that found it hammered my site with tens of thousands of requests per day (even Googlebot). I don't think it stopped until I blocked the page in robots.txt