I think it's less loss of potential (though that is a good rationalization) and more that people who weren't sad when they thought about their kids dying didn't

I think it's less loss of potential (though that is a good rationalization) and more that people who weren't sad when they thought about their kids dying didn't become ancestors to the current set of humans, via the obvious mechanism.