3 ms·

detecting duplicate urls? seomoz's api will do that for you... url cruft is much more an aesthic thing. see http://www.mattcutts.com/blog/clean-up-extra-url-p

by a5seo 15y ago

detecting duplicate urls? seomoz's api will do that for you... url cruft is much more an aesthic thing. see http://www.mattcutts.com/blog/clean-up-extra-url-parameters-when-searching-google/ http://www.mattcutts.com/blog/clean-up-extra-url-parameters-...

nikcub 15y ago
there is an RFC section dedicated to URL normalization, this doesn't need an API: http://tools.ietf.org/html/rfc3986#section-6 http://tools.ietf.org/html/rfc3986#section-6 I am currently working on making a urllib for python which is 3986 compliant
- yahelc 15y ago
  Anywhere I can follow progress on that? I'd love a Python-based URL normalization tool for http://SharedCount.com http://SharedCount.com
  - nikcub 15y ago
    email me and ill ping you once it is up (email in profile)