Hand-wavey sci-fi explanation - time travel being observed causes a divergent reality (think many worlds), we're in the branch that didn't notice. Seems like no

Hand-wavey sci-fi explanation - time travel being observed causes a divergent reality (think many worlds), we're in the branch that didn't notice. Seems like not much more of a leap than aliens/time travellers would definitely visit us _and_ we would notice.