I wish someone could explain in simpler terms to us casual users what this means. If only privileged users can SSH into my server, does this really affect me?

I wish someone could explain in simpler terms to us casual users what this means. If only privileged users can SSH into my server, does this really affect me? In other words, I already allow only SSH users to become root.