I lost all my messages with my girlfriend (of course). Can anyone clarify: if I use their "server key backup" and set a passphrase, I am now two passwords away

I lost all my messages with my girlfriend (of course). Can anyone clarify: if I use their "server key backup" and set a passphrase, I am now two passwords away from giving the next hacker read access to all my messages, is that right?