This would be an issue if you were naively looking at a single mutation in a single gene in a single individual without context, but the problem disappears when

This would be an issue if you were naively looking at a single mutation in a single gene in a single individual without context, but the problem disappears when looking at large numbers of whole genes in many individuals with fossil evidence for context. You can see more basal forms of genes, combinations of genes which get passed on together, biological clocks indicating when a mutation occurred, and geographic information indicating where it originated. If a gene variant exists for hundreds of thousands of years in one population, and then a nearly identical variant suddenly starts showing up in another population, it's safe to assume that some hanky panky took place. We do indeed find neanderthals who descend from humans, interbreeding went both ways. For example we can see that around 100,000 years ago, the neanderthals' y chromosomes were almost completely replaced with a modern human y chromosome whereas even as late as 38,000 years ago there were still neanderthals with purely neanderthal mitochondrial DNA.