3 ms·
We do have global rules: the Berne convention, which has been ratified by all 170 UN countries plus the Holy See and Niue, states that copyright is automatic an
by icebraining 10y ago
We do have global rules: the Berne convention, which has been ratified by all 170 UN countries plus the Holy See and Niue, states that copyright is automatic and mostly universal, so any unauthorized copying is illegal. By having certain paths listed on robots.txt, the site are explicitly saying they don't authorize people to crawl them, so unless you have a license granting you permission, your legal position is probably iffy - CFAA or not.
Obviously some countries have a more lax enforcement than others, but don't be surprised if the US starts squeezing and one day you suddenly get a knock on the door.