One way of defending against compiler-based backdoor would be to make the compiler binary as audit-able as possible (or to make it as easy as possible to be imp

One way of defending against compiler-based backdoor would be to make the compiler binary as audit-able as possible (or to make it as easy as possible to be implemented "from scratch"), i.e. for it to be as small as possible, and what you've suggested comes out as a consequence - not stacking abstractions on the compiler, or at least not on the compiler's lowest level (that gets translated into machine code). The simplest implementation as I see it is to rely on a very small core of language primitives like C (although a simpler core was possible even there).