When talking about fundamental rights (like say, the freedom of speech or the freedom of religion or ...), I don't understand the thinking that only some people

When talking about fundamental rights (like say, the freedom of speech or the freedom of religion or ...), I don't understand the thinking that only some people should have them based on some legal criteria. I'm talking out of my ass here but it feels like the kind of legacy that is carried by having stuff like slavery and the like where humans were not equals. I guess this is why in France we call those fundamental rights human rights and not citizens rights. My point being: I don't see how the difference you give provides anything other that than the ability to abuse some groups, I don't see any advantage from it.