Partly that, and partly because they were/are built with constrained functionality (do one thing well) that significantly limits the interfaces exposed, and are

Partly that, and partly because they were/are built with constrained functionality (do one thing well) that significantly limits the interfaces exposed, and are designed to be secure under the assumption of physical attack (which most things aren't!) Partly because many secure enclave type applications have undergone formal verification and verification through something like CC EAL, which should reduce the likelihood of glaring oversights etc. One final aspect, more for hardware enclaves though, is that they're often designed specifically to resist the kinds of physical attacks people may try - as security products, they've very likely considered power rail glitching and how to reset the enclave if there's a glitch. Similarly, the physical ICs may be produced with physical features to try to prevent successfully decapping the chip, like sensing wires you'll disrupt, and inbuilt EM shielding.