I think it will always inherently affect some "freedom" but some of them may so contrived and the competing concerns so unevenly stacked between the "freedom to

I think it will always inherently affect some "freedom" but some of them may so contrived and the competing concerns so unevenly stacked between the "freedom to X" and the "freedom from X" that there's legitimate case to be made. Your interracial marriage example would clearly be one - it has significant impact on the quality of life on one side, and no remotely similar impact on the quality of life on the other side. It's only when the "freedom to" and the "freedom from" are legitimately at least somewhat evenly stacked in how it affects people that it ought to be an issue given careful consideration.