That could work, but the advantage of an HSM is that the key material is physically prevented from leaving the device, barring acid/x-rays/other attacks. All en

That could work, but the advantage of an HSM is that the key material is physically prevented from leaving the device, barring acid/x-rays/other attacks. All encryption happens on the device itself. With an open ssh connection you could still remove the key material from computer.