That does sound simple, but how can you be so sure? They never bothered to find a way of actually understanding what happens during inference. All we can do is

That does sound simple, but how can you be so sure? They never bothered to find a way of actually understanding what happens during inference. All we can do is guess, and while your explanation seems reasonable we can't actually know, and that's part of the problem.