Access to the passphrase would not by itself provide access to anything. The malicious person leaving would also have to take along a copy of the encrypted priv

Access to the passphrase would not by itself provide access to anything. The malicious person leaving would also have to take along a copy of the encrypted private key. In practice you would just register 2 or more keys left in the care of 2 or more people. Each person would be individually responsible, as it should be. When someone left you would revoke the key. You would not have to go super hard on this, most of the requests would be routine and not time sensitive. In an emergency you do the best you can with what you have available.