I can't say I'm surprised. They're a bunch of data science people. Thinking about how to not let embarrassing things get indexed by scrapers is probably second

I can't say I'm surprised. They're a bunch of data science people. Thinking about how to not let embarrassing things get indexed by scrapers is probably second nature to them in the same way that chaos engineering is second nature to plenty of programmers. I bet the question "should this be indexed" is the data scientist equivalent of "how could a malicious user exploit this".