3 ms·

I wish that tools which remove unnecessary information from URLs were more popular than traditional URL "shorteners". For example, if I go to some random produ

I wish that tools which remove unnecessary information from URLs were more popular than traditional URL "shorteners". For example, if I go to some random product page on Amazon, the link looks something like this: https://www.amazon.com/Really-Long-Product-Name/dp/PRODUCTID/?_encoding=UTF8&pd_rd_w=garbage&pf_rd_p=long-garbage-value-that-doesnt-matter-to-the-user&pf_rd_r=MORELONGGARBAGE&pd_rd_r=another-really-long-garbage-value&pd_rd_wg=garbage&ref_=even_more_garbage&th=1 That link looks ridiculous and isn't suited for sharing in many situations. For most people, this is the link they really want: https://www.amazon.com/dp/PRODUCTID/ I know there are tools out there that can shorten URLs like that. I used to have a browser extension which did exactly that. It doesn't work for everything, but at least your link wont die because a third-party service shut down or messed up.

karlicoss 4y ago
It's kind of tricky to do in general case, e.g. even hackernews is keeping meaningful semantic information in id= query parameter. Because of that it ultimately needs to a site-specific database/algorithm, perhaps with a fallback to the default behaviour like simply cleaning up the most common garbage like (_encoding/usg/etc). I suspect it's possible to use some sort of machine learning to guess the meaningful parts of the URL path/query/fragments, but even for that we need some human curation for the training set. I wish we could collaborate on a shared database/library for that, have sketched some ideas/applications/prior art here: https://beepb00p.xyz/exobrain/projects/cannon.html https://beepb00p.xyz/exobrain/projects/cannon.html I started thinking about it since I have a similar problem in Promnesia (https://github.com/karlicoss/promnesia#readme https://github.com/karlicoss/promnesia#readme), a knowledge management tool I'm working on. Ideally I want to normalise URLS, so they address the exact bit of information, and nothing more.
- zinekeller 4y ago
  It doesn't cover everyone, but you should know about rel="canonical" (https://datatracker.ietf.org/doc/html/rfc6596 https://datatracker.ietf.org/doc/html/rfc6596). For example, Amazon helpfully hints that this messy Kindle Paperwhite link (https://www.amazon.com/Kindle-Paperwhite-Signature-Essentials-including/dp/B09FNJZD68/ref=sr_1_4?crid=3H1D0BMHBNTQ0&keywords=kindle+paperwhite+2022&sprefix=kindl%2Caps%2C358&sr=8-4 https://www.amazon.com/Kindle-Paperwhite-Signature-Essential...) is actually https://www.amazon.com/Kindle-Paperwhite-Signature-Essentials-including/dp/B09FNJZD68 https://www.amazon.com/Kindle-Paperwhite-Signature-Essential.... Additionally, CleanURLs to the rescue! https://github.com/ClearURLs https://github.com/ClearURLs
  - karlicoss 4y ago
    Yeah, sadly, to get the canonical attribute, you need to fetch the URL first (which is slow and wasteful). Also sometimes canonical would still be different on the desktop and mobile version of the site, so it still has to be normalised after that
- CivBase 4y ago
  I don't even think you need a machine learning algorithm. I follow a simple process all the time to achieve this. load the page with the original URL for each part of the URL: remove that part of the URL load the page with the modified URL if the page rendered differently: put that part back You'd need to incorporate an ad blocker, otherwise changing ads on each reload could screw it up. Of course, you'd also probably want to hard-code in logic for popular websites like Amazon to avoid wasting time with reloads. And I know there are limitations to the concept. It wont work for pages which require authentication or any kind of session data. It wont work for pages which are intentionally dynamic. It wont work for sites which cannot be accessed by the service. It wont be useful for pages who simply have long URLs. But it probably covers the many common use cases for a URL shortener, and you could always fall back on traditional shortening methods (redirecting) when it doesn't work.
  - karlicoss 4y ago
    Oh nice, I like it! So it basically automates detecting useful bits for a particular URL, but it's kind of time consuming and flaky. It could be very helpful to populate the 'rules' database though, and then this database could be shared with other people so they don't have to scrape. I guess when I said ML (or preferably some fuzzy algorithm/heuristic), I was referring to generifying rules so they also work on the sites not in the rules database. If humans can detect garbage in the URL looking at a few examples, the computer can too :)
- deleted 4y ago
  [deleted]
password4321 4y ago
FYI: https://amzn.com/dp/PRODUCTID https://amzn.com/dp/PRODUCTID is official, to shave off 2 more bytes. Affiliate links used to catch some heat on HN but https://amzn.to https://amzn.to seems to be showing up in more and more comments without too much blowback. It wasn't mentioned much that affiliates could view everything people bought while under their cookie, but I'm not sure if that's still possible.

karlicoss 4y ago

It's kind of tricky to do in general case, e.g. even hackernews is keeping meaningful semantic information in id= query parameter. Because of that it ultimately needs to a site-specific database/algorithm, perhaps with a fallback to the default behaviour like simply cleaning up the most common garbage like (_encoding/usg/etc). I suspect it's possible to use some sort of machine learning to guess the meaningful parts of the URL path/query/fragments, but even for that we need some human curation for the training set. I wish we could collaborate on a shared database/library for that, have sketched some ideas/applications/prior art here: https://beepb00p.xyz/exobrain/projects/cannon.html https://beepb00p.xyz/exobrain/projects/cannon.html I started thinking about it since I have a similar problem in Promnesia (https://github.com/karlicoss/promnesia#readme https://github.com/karlicoss/promnesia#readme), a knowledge management tool I'm working on. Ideally I want to normalise URLS, so they address the exact bit of information, and nothing more.

zinekeller 4y ago
It doesn't cover everyone, but you should know about rel="canonical" (https://datatracker.ietf.org/doc/html/rfc6596 https://datatracker.ietf.org/doc/html/rfc6596). For example, Amazon helpfully hints that this messy Kindle Paperwhite link (https://www.amazon.com/Kindle-Paperwhite-Signature-Essentials-including/dp/B09FNJZD68/ref=sr_1_4?crid=3H1D0BMHBNTQ0&keywords=kindle+paperwhite+2022&sprefix=kindl%2Caps%2C358&sr=8-4 https://www.amazon.com/Kindle-Paperwhite-Signature-Essential...) is actually https://www.amazon.com/Kindle-Paperwhite-Signature-Essentials-including/dp/B09FNJZD68 https://www.amazon.com/Kindle-Paperwhite-Signature-Essential.... Additionally, CleanURLs to the rescue! https://github.com/ClearURLs https://github.com/ClearURLs
- karlicoss 4y ago
  Yeah, sadly, to get the canonical attribute, you need to fetch the URL first (which is slow and wasteful). Also sometimes canonical would still be different on the desktop and mobile version of the site, so it still has to be normalised after that
CivBase 4y ago
I don't even think you need a machine learning algorithm. I follow a simple process all the time to achieve this. load the page with the original URL for each part of the URL: remove that part of the URL load the page with the modified URL if the page rendered differently: put that part back You'd need to incorporate an ad blocker, otherwise changing ads on each reload could screw it up. Of course, you'd also probably want to hard-code in logic for popular websites like Amazon to avoid wasting time with reloads. And I know there are limitations to the concept. It wont work for pages which require authentication or any kind of session data. It wont work for pages which are intentionally dynamic. It wont work for sites which cannot be accessed by the service. It wont be useful for pages who simply have long URLs. But it probably covers the many common use cases for a URL shortener, and you could always fall back on traditional shortening methods (redirecting) when it doesn't work.
- karlicoss 4y ago
  Oh nice, I like it! So it basically automates detecting useful bits for a particular URL, but it's kind of time consuming and flaky. It could be very helpful to populate the 'rules' database though, and then this database could be shared with other people so they don't have to scrape. I guess when I said ML (or preferably some fuzzy algorithm/heuristic), I was referring to generifying rules so they also work on the sites not in the rules database. If humans can detect garbage in the URL looking at a few examples, the computer can too :)
deleted 4y ago
[deleted]

password4321 4y ago

FYI: https://amzn.com/dp/PRODUCTID https://amzn.com/dp/PRODUCTID is official, to shave off 2 more bytes. Affiliate links used to catch some heat on HN but https://amzn.to https://amzn.to seems to be showing up in more and more comments without too much blowback. It wasn't mentioned much that affiliates could view everything people bought while under their cookie, but I'm not sure if that's still possible.