I think it's fairly simple in that the overwhelming majority of people (including most physicists) haven't got an intuitive grasp of exactly how relativity work

I think it's fairly simple in that the overwhelming majority of people (including most physicists) haven't got an intuitive grasp of exactly how relativity works. This possibly includes Einstein. It wouldn't surprise me one bit to find out that the basic assumptions are not flawed at all, but that whole piles of stuff is being assumed in proofs regarding expected consequences of these things which prove to be complete nonsense. The other aspect of it is mathematics itself, which is probably not up to the task.