You're presenting your argument as if that's the only kind of malicious binary we have to worry about, and since it's (probably) impossible, then therefore we d

You're presenting your argument as if that's the only kind of malicious binary we have to worry about, and since it's (probably) impossible, then therefore we don't have to worry. But there are more ways to compromise a binary than via the compiler automatically backdooring them. Deterministic builds protect against compromised binaries of any kind, so we need to use it. The attack to be worried about goes like this: You build "Firefox Setup 23.0.1.exe" and intentionally insert a backdoor into the setup process. You make sure the setup process appears to function exactly the same as the clean installer (not hard). You then replace Firefox Setup 23.0.1.exe on various distribution websites with your malicious version. Or you MITM the distribution websites in order to send your malicious version in place of the one the user expects to be downloading. Deterministic builds defend against that attack vector, while also defending against any hypothetical compiler-backdoor-autoinjector. You get both defense layers for free, just by using deterministic builds. This is a necessary step for the future, not an optional security layer.