Yes, it's by design. It's to stave off the inevitable flood of legal threats and takedown demands. If someone requests content be taken off, they instruct them

Yes, it's by design. It's to stave off the inevitable flood of legal threats and takedown demands. If someone requests content be taken off, they instruct them to update their robots.txt. The content is not removed but will not be shown through archive.org as long as the robots.txt exclusion is in place. There was a court case where the plaintiff wanted to subpoena the Internet Archive for evidence (since the defendant had since blocked the content with robots.txt). They sent an expert to testify that complying with that kind of thing would be too much of a burden for them, and suggested that the court force the defendant to change their robots.txt. The court agreed.