There are ways to hornswoggle a trained machine learning model into generating new outputs that "kinda sorta" look like bits and pieces of an input that was use

There are ways to hornswoggle a trained machine learning model into generating new outputs that "kinda sorta" look like bits and pieces of an input that was used for training. It's tricky, and not always reliable, but it would be a reasonable thing to attempt. This is why you shouldn't use sensitive material as training data. If you're a startup and you're using all of your customers' private data to train a single model that then gets used by all of your customers -- guess what, you're doing it wrong! It's the same reason you wouldn't teach your five year old how to read by letting them pour over you and your spouse's text message history. I mean, you could, but then you shouldn't be surprised when they inadvertently utter something embarrassing that they don't necessarily understand, but whose origin is obvious to everyone else at the party.