The only sound advice one can give is: there are two elements to consider: 1) ethics is different from law 1.1) the ethical way: respect robots.txt protocol

The only sound advice one can give is: there are two elements to consider: 1) ethics is different from law 1.1) the ethical way: respect robots.txt protocol 2) consult a lawyer 2.1) prior written consent, they will say, prevents you from being sued, and not much else.