Not sure if I understand this correctly, but the attacker's JS must load the pages in question to see if they are in cache, right? Wouldn't that put also them

Not sure if I understand this correctly, but the attacker's JS must load the pages in question to see if they are in cache, right? Wouldn't that put also them in cache, which means that next time this technique is used it will not work? Even more, there is now plausible deniability: "I never saw these pages, I guess some JS must have been snooping around and put them to my cache..." And the logical workaround is disabling cache, which helps fight against other tracking techniques too. All in all, this doesn't sound so worrying. Unless I missing something?