It would help because the private keys needed to decrypt the emails would not have been kept on the server, and, even if they were, they'd still need a passphra

It would help because the private keys needed to decrypt the emails would not have been kept on the server, and, even if they were, they'd still need a passphrase to get the content of the private key (though, it could have been the same insecure passphrase used elsewhere). Also, a common policy of encrypting and signing emails would have stopped the social engineering attack completely, as the sysadmin would've known not to accept an unsigned request to give out passwords. Kind of mind boggling that people don't do this generally already.