What you are describing is called a Bona Fide Occupational Qualification (BFOQ). The specifics of what sort of attributes might be covered for what jobs is som

What you are describing is called a Bona Fide Occupational Qualification (BFOQ). The specifics of what sort of attributes might be covered for what jobs is something courts hash out, but broadly: if you're hiring workers for a warehouse it's fine to require workers be able to lift boxes. If you're hiring airline pilots, it's fine to turn away blind people. Etc.