Ideally, there would be a better way for web hosts to block bandwidth-hogging, but not block bots. I don't think anyone cares if some automated user-agent reque

Ideally, there would be a better way for web hosts to block bandwidth-hogging, but not block bots. I don't think anyone cares if some automated user-agent requests a 1KB file from their web site. They care that automated systems are sucking down TBs per day, all day, every day.