I think the main difficulty is that a neural decoder is allowed to make up lots of plausible phase information, which likely leads to pretty large L2 errors whi

I think the main difficulty is that a neural decoder is allowed to make up lots of plausible phase information, which likely leads to pretty large L2 errors while retaining perceptual quality. So then you'll end up with large residuals even though you might only barely discern the difference perceptually.