4 ms·

robots.txt is not DRM: it can be freely implemented by anyone without special knowledge – and ignoring it does not lead to a criminal investigation.