Arguably you are one privilege escalation or boot disk away from being compromised to the point that the other user can acquire your passphrase or key from memo

Arguably you are one privilege escalation or boot disk away from being compromised to the point that the other user can acquire your passphrase or key from memory. If merely a passphrase it could even be gathered by a device between the keyboard and computer or plugged into a usb port. Security vs someone who has physical access to your machine while you aren't there will probably remain crappy.