Yeah this is a super common point of confusion. You need to worry about slowing down a brute force attacker when you're trying to protect a low-entropy secret,

Yeah this is a super common point of confusion. You need to worry about slowing down a brute force attacker when you're trying to protect a low-entropy secret, which basically always means user passwords. But when your secrets are big enough, like 128 bits, brute force becomes impossible just by virtue of the size of the search space. So for most cryptographic applications, it's the size of your key (and the fact that your hash or cipher doesn't _leak_ the key) that's protecting you from brute force, not the amount of time a single attempt takes.