On the other hand, I'm not sure if the whole idea of people visiting a website where they ought to behave and play by the house-rules is doing the world much go

On the other hand, I'm not sure if the whole idea of people visiting a website where they ought to behave and play by the house-rules is doing the world much good. And I'm not just saying this because one of my pet projects was crippled by an overzealous bot-prevention system (yes I was using a program I wrote myself to access your system, no that doesn't make me a bot). Quite a few things would be vastly better when the information that is on websites is not just publicly readable but publicised in the full sense of the word. Several industries are paying 20~40% revenue just to be findable. Now maybe scrapers aren't the best solution, but making stuff less accessible is usually not doing much good. Especially not for accessibility.