It’s hard for me to blame developers for their misaligned mental models since, as you yourself point out, the relationship between your mental model of the hard

It’s hard for me to blame developers for their misaligned mental models since, as you yourself point out, the relationship between your mental model of the hardware’s behavior and what it actually does is complex and not really obvious. Mitigations etc means that the hardware you bought might totally change its behavior in the future. If all you can rely on is time measurements, it doesn’t explain that the software you wrote last year is now eg 40% slower due to bug mitigations. I think it’s easier to expect developers to put in the effort once they’re empowered with powerful tools to explain how the software gets mapped onto the hardware. I think tools that result in greater transparency at lower levels will probably emerge, something like a compiler-generated dynamic analysis + pre-flight checklist hybrid. But since it’s so low-level, it might actually affect results, which may be a problem. I think intel vtune seems interesting in that regard.