But this still doesn't specifically answer the question, "What is the penalty for ignoring 'User-agent:* Disallow: /' in robots.txt, or meta tag no-index", only

But this still doesn't specifically answer the question, "What is the penalty for ignoring 'User-agent:* Disallow: /' in robots.txt, or meta tag no-index", only that without a robots.txt or no-index tag, crawlers can't be found liable for accessing information that is freely available to them. Interesting link, however. I was unaware of that case.