What I don't get about the DP approach is how this would be reconciled with the "exact" question-answering functionality of LLMs. DP makes perfect sense if all

What I don't get about the DP approach is how this would be reconciled with the "exact" question-answering functionality of LLMs. DP makes perfect sense if all I care about is low-resolution statistical metrics or distributions of something and not the exact values - the entire purpose of DP is to prevent reconstructing the exact values. However, the expectation for LLMs is usually to ask a question (or request a task) and get an exact value as a response: If you ask "What's the phone number of John Smith?" the model will either tell you it doesn't know or it will answer you with an actual phone number (real or hallucinated). It will not tell you "the number is with 83% probability somewhere in New Jersey". So if the model is trained with DP, then either the data is scrambled enough that the it won't be able to return any kind of reliably correct data, effectively making it useless - or it's not scrambled enough, so that the model can successfully reconstuct data despite the scrambling process, effectively making the DP step useless. Or in other words, the OP defines "DP unlearning" as: > The intuition is that if an adversary cannot (reliably) tell apart the models, then it is as if this data point has never been learned—thus no need to unlearn. However, if my original model truthfully returns John Smith's phone number on request and the "unlearned" model must not be distinguishable by an outside observer from the original model, then the "unlearned" model will also return the phone number. While I could say that "technically" the model has never seen the phone number in the training data due to my DP scrambling, this doesn't solve the practical problem why the unlearning was requested in the first place, namely that John Smith doesn't want the model to return his phone number. He could probably care less about the specific details of the training process. So then, how would DP help here?