This is an old problem with an old solution. If you have an expensive-to-generate resource that you don't want automatically retrieved en masse, you use robots

This is an old problem with an old solution. If you have an expensive-to-generate resource that you don't want automatically retrieved en masse, you use robots.txt to deny access to it.