I understand your concern. I would just add that, if the main concern for you would be someone finding such a USB key with encrypted private keys and be willing

I understand your concern. I would just add that, if the main concern for you would be someone finding such a USB key with encrypted private keys and be willing to use a long time to recover them, that's not really an issue: you only need the encryption to be strong enough for you to notice that you lost the USB key and rotate the keys, if such an adversary needs even just a couple days to perform its attack, you still got enough time IMO.