3 ms·
What does this even mean? If I break up all of the data on the internet up into pieces, wouldn't every chunk be a duplicate of another chunk if we exploded the
by thinkmcfly 4y ago
What does this even mean? If I break up all of the data on the internet up into pieces, wouldn't every chunk be a duplicate of another chunk if we exploded the samples into small enough pieces?
Did anyone else not click the link tho?
- hcs 4y agoThe light gray text in the slide [1] suggests that this is about multiple URLs returning the same content: > 1. remove protocol duplicates - favor HTTPS > 2. remove www/non-www > 3. remove URLs with useless parameters (sessionID?) > 4. remove slash/no-slash variant > 5. [...] [che]cksum dups I haven't seen the talk, but I guess that the 60% is after taking 1-4 into account and directly comparing checksums of the full response (body?) [1] https://twitter.com/suzukik/status/1595679289873428480 https://twitter.com/suzukik/status/1595679289873428480
- rchaud 4y agoThat just means that Google's index of URLs contain duplicates, not the Internet itself. Google indexes `/page` and `/page.aspx` as two unique pages with duplicate content. All that happened was that the search spider clicked a link to a server-rendered page that can be rendered with a non-canonical link.