You are missing an integrity checking step. You can do it by sending some sort of ephemeral binary over ssh that does integrity checking and requests a key with

You are missing an integrity checking step. You can do it by sending some sort of ephemeral binary over ssh that does integrity checking and requests a key with the resulting hash of the check to proceed, don't blindly trust sshd running from an unencrypted partition. But still at the end of the day it's all about obscurity and obfuscation, you can't make it provably secure. You can go far and make that binary one time randomly generated, obfuscated, bound its running time, you can use a TPM and what not, but it probably won't matter for pretty much any realistic threat model.