I don’t follow. Why is it the size or structure of what’s being encrypted that matters for brute force time rather than the key space of the encryption algorith

I don’t follow. Why is it the size or structure of what’s being encrypted that matters for brute force time rather than the key space of the encryption algorithm. Password hashing is different because there is no key space.