Right, but I'd argue that if you believe that (for example) human rights are some kind of fundamental fact of the universe then you really aren't that different

Right, but I'd argue that if you believe that (for example) human rights are some kind of fundamental fact of the universe then you really aren't that different from someone who believes in God. If you just have a preference for a world where people behave as though human rights exist, then you can be a materialist.