I've thought about this same issue a lot, especially while reading Crichton's Timeline. I decided to justify all of it in my head by making an assumption about

I've thought about this same issue a lot, especially while reading Crichton's Timeline. I decided to justify all of it in my head by making an assumption about the fictional time-travel technology. The assumption is that the function of the device is somehow tied to matter, either through the physical manifestation itself or through the distortion that matter has on spacetime. Either way, the mass of the earth would "drag" the path through time with it to about the same location from an earthling's perspective.