What does a single-person facility have to do with it? Why is it legitimate to segregate multi-person facilities based on gender identity but not on sex? If me

What does a single-person facility have to do with it? Why is it legitimate to segregate multi-person facilities based on gender identity but not on sex? If men’s restrooms are dangerous for trans women then certainly they are dangerous for gay or gender-nonconforming men as well. These men certainly have as much right to avoid a dangerous situation as trans women do, no?