You'd want to use a strong passphrase. Even if you do, it's protecting the physical key. If an attacker has access to the that they can perform an offline attac

You'd want to use a strong passphrase. Even if you do, it's protecting the physical key. If an attacker has access to the that they can perform an offline attack. i.e. attacking a passphrase offline with decent hardware and no likelihood for being noticed means it had better be strong!