4 ms·
Not to hijack, but how do we stop esentially duplicate URLs in "new" see (http://mobile.nytimes.com/2016/01/05/technology/gm-invests-in-lyft.html?smprod=nytcore
by wheaties 11y ago
Not to hijack, but how do we stop esentially duplicate URLs in "new" see (http://mobile.nytimes.com/2016/01/05/technology/gm-invests-in-lyft.html?smprod=nytcore-iphone&smid=nytcore-iphone-share&referer=http://jalopnik.com/gm-to-invest-500-million-in-lyft-build-autonomous-on-1750886157&_r=0 http://mobile.nytimes.com/2016/01/05/technology/gm-invests-i...) not matching?
- BinaryIdiot 11y agoThe subdomain is probably the issue. Suggest updating it to strip out subdomain, query string, hashes etc and do a domain and path comparison. Too many sites have a mobile subdomain with the rest of the domain the same.
- gk1 11y agoAlso, sites like NYTimes add different URL parameters based on how you arrived at the article (eg, which menu item you clicked). I strip those out when submitting to HN, but others might not, so HN views it as a different URL.
- snthd 11y agouse the canonical url defined in the html (<link rel="canonical" href="http://www.nytimes.com/2016/01/05/technology/gm-invests-in-lyft.html"> http://www.nytimes.com/2016/01/05/technology/gm-invests-in-l... https://en.wikipedia.org/wiki/Canonical_link_element https://en.wikipedia.org/wiki/Canonical_link_element
- psadri 11y agoMaybe fetch the url and look at the <link rel="canonical"> tag?