You're arguing "you don't have a [legal/moral] right" (which is true) They're arguing "it's a victimless crime" (which is also true) These are related things,

You're arguing "you don't have a [legal/moral] right" (which is true) They're arguing "it's a victimless crime" (which is also true) These are related things, but they aren't contradictory