3 ms·
The light gray text in the slide [1] suggests that this is about multiple URLs returning the same content: > 1. remove protocol duplicates - favor HTTPS > 2.
by hcs 4y ago
The light gray text in the slide [1] suggests that this is about multiple URLs returning the same content:
> 1. remove protocol duplicates - favor HTTPS
> 2. remove www/non-www
> 3. remove URLs with useless parameters (sessionID?)
> 4. remove slash/no-slash variant
> 5. [...] [che]cksum dups
I haven't seen the talk, but I guess that the 60% is after taking 1-4 into account and directly comparing checksums of the full response (body?)
[1] https://twitter.com/suzukik/status/1595679289873428480 https://twitter.com/suzukik/status/1595679289873428480
- rchaud 4y agoThat just means that Google's index of URLs contain duplicates, not the Internet itself. Google indexes `/page` and `/page.aspx` as two unique pages with duplicate content. All that happened was that the search spider clicked a link to a server-rendered page that can be rendered with a non-canonical link.