It's a little simpler than you're thinking: only fully matching configurations (of all particles etc) can interfere. If you have a setup where a particle can pa

It's a little simpler than you're thinking: only fully matching configurations (of all particles etc) can interfere. If you have a setup where a particle can pass through one of two slits and then end up in the same location (with the same energy etc) afterward, so that all particles everywhere are in the same arrangement including the particle that passed through one of the slits, then these two configurations resulting from the possible paths can interfere. If anything is different between these two resulting configurations, such as a detector's particles differently jostled out of position, then the configurations won't be able to interfere with each other. An interesting experiment to consider is the delayed-choice quantum eraser experiment, in which a special detector detects which path a particle went through, and then the full results of the detector are carefully fully stomped over so that the particles of the detector (and everything else) are in the same exact state no matter which path had been detected. The configurations are able to interfere once this erasure step happens and not if the erasure step isn't done. Another fun consequence of this all is that we can basically check what configurations count as the same to reality by seeing if you still get interference patterns in the results. You can have a setup where two particles 1 and 2 of the same kind have a chance to end up in locations A and B respectively or in locations B and A, and then run it a bunch of times and see if you get the interference patterns in the results you'd expect if the configurations were able to interfere. Successful experiments like this have been done with many kinds of particles including photons, subatomic particles, and atoms of a given element and isotope, implying that the individual particles of these kinds have no unique internal structure or tracked identity and are basically fungible.