Sure, if you're hardware RNG is inspecting your RAM and trying to compromise your entropy pool that could be done, but: 1 - The entropy mixing is more complex

Sure, if you're hardware RNG is inspecting your RAM and trying to compromise your entropy pool that could be done, but: 1 - The entropy mixing is more complex than simply XOR, making such a thing considerably harder 2 - If you expect this level of backdooring from your CPU, you have bigger problems :)