I think the bigger concern is the nature of the patch Intel supplied. The kernel developers at Intel are smart people, even if they're not putting in malicious

I think the bigger concern is the nature of the patch Intel supplied. The kernel developers at Intel are smart people, even if they're not putting in malicious code they know there's a risk that there might be something wrong with their random number generation so the logical thing would be for them to add their random source to the pool to mitigate against that. The fact that they chose not to do that puts red flags all over the patch. Sure it could have been purely accidental, but any patch that "accidentally" weakens the security of a system deserves thorough analysis regardless of who submitted it. Assuming you were a malicious party who wanted to backdoor a system in such a way that you could gain access without creating a massive security vulnerability then the random number source would be ideal as we've got a pretty good idea of how you could do it. It'd be harder to do with other parts of the architecture (although presumably not impossible). It would also give you attacks against a much wider range of systems (i.e. ones with which you can't directly interact so can't do timing-based attacks on).