Assuming you are wanting to block bots and allow real interactive humans: these are not perfect One option - Make your own custom silly captcha. e.g. Picture

Assuming you are wanting to block bots and allow real interactive humans: these are not perfect One option - Make your own custom silly captcha. e.g. Picture of a motorcycle and the person has to type in motorcycle. Or say "What is 2 + 2 X 2". Or "What is the average unladen speed of a swallow?" but that last one may eliminate many guests. Another option - On a landing page, list the username and password for basic auth set on a sub-set of URL's that has your semi-sensitive data. Another option - Restrict the site to HTTP/2.0. Most bots can't talk 2.0 yet. Bing can. Google can't. Discord and Steam can't. Most of the blog-spam and malware bots can't. Real browsers can talk HTTP/2.0. Another thing that breaks some bots is enforcing strict-SNI. Another option, though not super effective - Restrict the semi-sensitive content to having a valid referrer. Easy to spoof and some bots will spoof. Along this line one can block a set of known bot user-agents. Again, super easy to spoof and Google intentionally spoofs on a percentage of requests to look like Android phones and to test if you are giving different data to search engines. Another option though much less effective - Add robots.txt entries but many search engines will still crawl and index the data, but won't show it in public results. Robots shows intent but enforces nothing and not all search engines obey it. It is good practice to have this anyway. Archive.org partially obeys robots, in that they crawl but will hide results if you tell them to. Once your robots is gone or your site is offline for a while they will unhide the data.