This is a good point. We have some concerns and evidence that neural networks do memorize their training data. Also, the "untraining" idea is an open research

This is a good point. We have some concerns and evidence that neural networks do memorize their training data. Also, the "untraining" idea is an open research question (I just saw a talk about it). We don't know exactly how to do it yet, definitely not in general, but "impossible" is too strong.