Yes, the "drifting" component you speak of is called the peculiar velocity. It's responsible for the "finger of god" effect if we don't account for it (galaxies

Yes, the "drifting" component you speak of is called the peculiar velocity. It's responsible for the "finger of god" effect if we don't account for it (galaxies appear to be aligned along the line of sight of the observer) and becomes negligible at large enough distances. I guess technically it's hard to differentiate because it's usually random, but it enters the error analysis and is accounted for.