7 ms·

Show HN: I built a FOSS service to protect against link shorteners tracking us

57 points by l1am0 7y ago

zelon88 7y ago
This is really cool. And it's all written in Go? Interesting. Do you have any plans on releasing the static JS/HTML? Or is there a way this could be run on standard Apache/Nginx/IIS web servers?
- l1am0 7y ago
  No I do not play to make it runnable on Apache/Nginx/ISS as there is more logic in it, than just the redirect. The js/html assets are in the subfolder server/static/ for your use
pmoriarty 7y ago
How do we know this site is not tracking us?
- l1am0 7y ago
  TBO you would never know, that's why the project is open source and if you want to be really sure what happens here, run it yourself. The browser extension can be easily (in the settings) tweaked to use your own server instead of unshort.link
- notduncansmith 7y ago
  You can run it yourself, and audit the source if you like: https://github.com/simonfrey/unshort.link/ https://github.com/simonfrey/unshort.link/
  - eat_veggies 7y ago
    If you run it yourself, do you not lose some of the privacy benefits?
    - notduncansmith 7y ago
      I believe you gain privacy by running it yourself, since you're no longer submitting a portion of your browsing history to a third party. The README for the server claims "You can build & run it yourself for even better privacy".
      - OJFord 7y ago
        Shorteners are frequently used to hide/prettify query parameters used for tracking; so if you run your own you're losing the aggregate anonymity-ish, and gaining that OP can't see what URLs you're unshortening. I suppose it depends on your use, but IMO if you run it yourself as the only user, you lose more privacy than you gain.
        notduncansmith 7y ago
        If the website you're ultimately visiting can see your IP address anyways, I fail to see the lost privacy. Their traffic will be sliced and diced into cohorts along many axes, and this is only one axis on which you are slightly anomalous. If anything, you'll be targeted less as a result since you won't show up in their funnels.
tyingq 7y ago
Pretty cool. You might consider adding some of the more popular vanity names that bitly runs for big sites. Like nyti.ms (New York Times), wapo.st (Washington Post), etc.
- l1am0 7y ago
  You can add them yourself :D Just unshort them on unshort.link (not the browser extensions) and the service learns that it found a new shortlink service and it will be deployed right away. After the next browser restart also the extension knows about that new service. (No updated required) You can see if your updated worked if e.g. wapo.st is listed in https://unshort.link/providers https://unshort.link/providers
OJFord 7y ago
Yes yes yes! I hate URL shorteners. It's not just privacy - I can't decide if I want to click it or jot before I do. I suppose that issue could also be solved with a browser extension. It'd be nice to do this with DNS rather than an extension though, above point aside, e.g. if already running Pi-hole or similar, send shortener URLs to unshort.link, and redirect immediately.
- tyingq 7y ago
  Pi-hole would only see the hostname, and not the "slug", so that wouldn't work.
  - edoceo 7y ago
    Pi-Hole sees the bit.ly DNS lookup,that responds with HTTP/30x to full.domain.tls, which is a second DNS lookup the Pi could catch/filter.
    - tyingq 7y ago
      But bitly.com/xyz can be mapped to full.domain/abcdef The pi-hole doesn't see the /xyz, or any of the http redirects either.
  - OJFord 7y ago
    What I mean is, you'd send `t.co` and similar to your unshort.link server, which then has the `Host` header `t.co` and the path; so it can do the redirect. I know it's not how it works today, but I wasn't describing how it works, I was describing how I'd like it to.
- l1am0 7y ago
  unshort.link has a browser extension for chrome & firefox, you find it on the site. If you directly want to get redirected without showing the url you can activate that feature in the plugins config. Or you can send the shortlinks to http://unshort.link/[http://short.link/sd] http://unshort.link/[http://short.link/sd] and it will directly redirect you
Wowfunhappy 7y ago
Where and why are you guys still seeing/using shortened URLs? I don't think I've personally made a shortened URL since Twitter built one into their service, and I stopped seeing shortened URLs around the same time. There are two exceptions: • I still see service-specific short URLs, e.g. goo.gl, nyti.ms, etc. Because these can only be created by the service in question, they don't introduce security/privacy concerns. • When a URL shortener is used to create a sort of vanity link to e.g. a Google doc. (OP's service could be useful for these!) Outside of those times... why would you use a URL shortener? When are you limited by number of characters?
- kick 7y ago
  • I still see service-specific short URLs, e.g. goo.gl, nyti.ms, etc. Because these can only be created by the service in question, they don't introduce security/privacy concerns. This isn't necessarily true; I've seen companies that offer to do the 'custom shortlink' thing for various companies. It's safe to assume that there are privacy concerns with at least some of them.
  - tyingq 7y ago
    Bitly runs nyti.ms, so yes, you aren't trusting just the NYT.
    - deleted 7y ago
      [deleted]
    - edoceo 7y ago
      Yes, and nyti.ms only ever pints to New York Times, a well known entity (well, that's what's intended, right?)
      - tyingq 7y ago
        Sure. Noting that you have to trust that bitly isn't fingerprinting your browser and tracking you across their domains.
myself248 7y ago
Related, the Archive Team realized a while ago, that when a link shortening service goes dark, all the links that went through it instantly break. This has bad implications for future historians trying to figure out how the present-day web was interlinked. So there's a project called URLteam to index and save them. I just started running a Warrior instance on the URLteam project, and it's really straightforward. I figure folks reading this may be interested. More info: https://www.archiveteam.org/index.php?title=URLTeam https://www.archiveteam.org/index.php?title=URLTeam https://www.archiveteam.org/index.php?title=ArchiveTeam_Warrior https://www.archiveteam.org/index.php?title=ArchiveTeam_Warr...
iudqnolq 7y ago
Reminds me when security researchers used data Bit.ly exposes to go from one url shortened by Russian military intelligence to the hundreds of other short URLs they made for phishing emails, many of which had state embedded in query parameters that gave away who they were targeting. All because the GRU forgot to set their accounts to private. https://www.vice.com/en_us/article/mg7xjb/how-hackers-broke-into-john-podesta-and-colin-powells-gmail-accounts https://www.vice.com/en_us/article/mg7xjb/how-hackers-broke-...
l1am0 7y ago
Oh wow thanks for all the feedback. Should not post a link on HN and than sleep directly after. I came to work on this as I wantend to not get bothered with the short link services (and the links behind them) tracking me. So unshort.link also tries to "learn" (by trying different url parameters) which paramters on the long url are required and which are not and are only tracking you (with a HTML diff). The service learns new shortlink services the moment you enter them on unshort.link and they than are also automatically used in the extension.
zzo38computer 7y ago
Is it possible to make such a thing working if using command-line programs such as curl or wget to download a file rather than opening it in the web browser?
- l1am0 7y ago
  It should totaly be possible. Same as with the DNS solution you can get the data via the api of unshort.link and than build a small bash script for it. If I have time I may try to do such a script, but not sure if and when I will be able to make it. Would be awesome if you build something like that! You can get the currently to unshort.link known short link providers in a nice json format from https://unshort.link/providers https://unshort.link/providers You can get the redirect info from unshort link also via its api without redirecting, just replace /d/ with /api/ in the GET request: e.g. https://unshort.link/api/https://tinyurl.com/unshortchromeex.. https://unshort.link/api/https://tinyurl.com/unshortchromeex.... If you build a solution, please do not forget to add the documentation on how to do it via a pull request to the unshort.link repo, so everyone could profit from it
- moehm 7y ago
  This is what I use which also removes utm_* tracking parameters. getloc () { curl -sI "$1" | grep -i "Location" | awk '/http.*/{print $2}' | sed 's/[%+]*//g' | sed 's/&*utm_[a-zA-Z0-9]*=[a-zA-Z0-9]*//g' | sed 's/?\s*$//g' }
gesman 7y ago
Add: Replacement of affiliate or referral links with your user' affiliate link and you can charge for this service.
gesman 7y ago
I wanted to make something like this with my C.GG domain but couldn't refuse an offer to sell the name for $10k

57 points by l1am0 7y ago

This is really cool. And it's all written in Go? Interesting. Do you have any plans on releasing the static JS/HTML? Or is there a way this could be run on standard Apache/Nginx/IIS web servers?

l1am0 7y ago
No I do not play to make it runnable on Apache/Nginx/ISS as there is more logic in it, than just the redirect. The js/html assets are in the subfolder server/static/ for your use

pmoriarty 7y ago

How do we know this site is not tracking us?

l1am0 7y ago
TBO you would never know, that's why the project is open source and if you want to be really sure what happens here, run it yourself. The browser extension can be easily (in the settings) tweaked to use your own server instead of unshort.link
notduncansmith 7y ago
You can run it yourself, and audit the source if you like: https://github.com/simonfrey/unshort.link/ https://github.com/simonfrey/unshort.link/
- eat_veggies 7y ago
  If you run it yourself, do you not lose some of the privacy benefits?
  - notduncansmith 7y ago
    I believe you gain privacy by running it yourself, since you're no longer submitting a portion of your browsing history to a third party. The README for the server claims "You can build & run it yourself for even better privacy".
    - OJFord 7y ago
      Shorteners are frequently used to hide/prettify query parameters used for tracking; so if you run your own you're losing the aggregate anonymity-ish, and gaining that OP can't see what URLs you're unshortening. I suppose it depends on your use, but IMO if you run it yourself as the only user, you lose more privacy than you gain.
      - notduncansmith 7y ago
        If the website you're ultimately visiting can see your IP address anyways, I fail to see the lost privacy. Their traffic will be sliced and diced into cohorts along many axes, and this is only one axis on which you are slightly anomalous. If anything, you'll be targeted less as a result since you won't show up in their funnels.

tyingq 7y ago

Pretty cool. You might consider adding some of the more popular vanity names that bitly runs for big sites. Like nyti.ms (New York Times), wapo.st (Washington Post), etc.

l1am0 7y ago
You can add them yourself :D Just unshort them on unshort.link (not the browser extensions) and the service learns that it found a new shortlink service and it will be deployed right away. After the next browser restart also the extension knows about that new service. (No updated required) You can see if your updated worked if e.g. wapo.st is listed in https://unshort.link/providers https://unshort.link/providers

OJFord 7y ago

Yes yes yes! I hate URL shorteners. It's not just privacy - I can't decide if I want to click it or jot before I do. I suppose that issue could also be solved with a browser extension. It'd be nice to do this with DNS rather than an extension though, above point aside, e.g. if already running Pi-hole or similar, send shortener URLs to unshort.link, and redirect immediately.

tyingq 7y ago
Pi-hole would only see the hostname, and not the "slug", so that wouldn't work.
- edoceo 7y ago
  Pi-Hole sees the bit.ly DNS lookup,that responds with HTTP/30x to full.domain.tls, which is a second DNS lookup the Pi could catch/filter.
  - tyingq 7y ago
    But bitly.com/xyz can be mapped to full.domain/abcdef The pi-hole doesn't see the /xyz, or any of the http redirects either.
- OJFord 7y ago
  What I mean is, you'd send `t.co` and similar to your unshort.link server, which then has the `Host` header `t.co` and the path; so it can do the redirect. I know it's not how it works today, but I wasn't describing how it works, I was describing how I'd like it to.
l1am0 7y ago
unshort.link has a browser extension for chrome & firefox, you find it on the site. If you directly want to get redirected without showing the url you can activate that feature in the plugins config. Or you can send the shortlinks to http://unshort.link/[http://short.link/sd] http://unshort.link/[http://short.link/sd] and it will directly redirect you

Wowfunhappy 7y ago

Where and why are you guys still seeing/using shortened URLs? I don't think I've personally made a shortened URL since Twitter built one into their service, and I stopped seeing shortened URLs around the same time. There are two exceptions: • I still see service-specific short URLs, e.g. goo.gl, nyti.ms, etc. Because these can only be created by the service in question, they don't introduce security/privacy concerns. • When a URL shortener is used to create a sort of vanity link to e.g. a Google doc. (OP's service could be useful for these!) Outside of those times... why would you use a URL shortener? When are you limited by number of characters?

kick 7y ago
• I still see service-specific short URLs, e.g. goo.gl, nyti.ms, etc. Because these can only be created by the service in question, they don't introduce security/privacy concerns. This isn't necessarily true; I've seen companies that offer to do the 'custom shortlink' thing for various companies. It's safe to assume that there are privacy concerns with at least some of them.
- tyingq 7y ago
  Bitly runs nyti.ms, so yes, you aren't trusting just the NYT.
  - deleted 7y ago
    [deleted]
  - edoceo 7y ago
    Yes, and nyti.ms only ever pints to New York Times, a well known entity (well, that's what's intended, right?)
    - tyingq 7y ago
      Sure. Noting that you have to trust that bitly isn't fingerprinting your browser and tracking you across their domains.

myself248 7y ago

Related, the Archive Team realized a while ago, that when a link shortening service goes dark, all the links that went through it instantly break. This has bad implications for future historians trying to figure out how the present-day web was interlinked. So there's a project called URLteam to index and save them. I just started running a Warrior instance on the URLteam project, and it's really straightforward. I figure folks reading this may be interested. More info: https://www.archiveteam.org/index.php?title=URLTeam https://www.archiveteam.org/index.php?title=URLTeam https://www.archiveteam.org/index.php?title=ArchiveTeam_Warrior https://www.archiveteam.org/index.php?title=ArchiveTeam_Warr...

iudqnolq 7y ago

Reminds me when security researchers used data Bit.ly exposes to go from one url shortened by Russian military intelligence to the hundreds of other short URLs they made for phishing emails, many of which had state embedded in query parameters that gave away who they were targeting. All because the GRU forgot to set their accounts to private. https://www.vice.com/en_us/article/mg7xjb/how-hackers-broke-into-john-podesta-and-colin-powells-gmail-accounts https://www.vice.com/en_us/article/mg7xjb/how-hackers-broke-...

l1am0 7y ago

Oh wow thanks for all the feedback. Should not post a link on HN and than sleep directly after. I came to work on this as I wantend to not get bothered with the short link services (and the links behind them) tracking me. So unshort.link also tries to "learn" (by trying different url parameters) which paramters on the long url are required and which are not and are only tracking you (with a HTML diff). The service learns new shortlink services the moment you enter them on unshort.link and they than are also automatically used in the extension.

zzo38computer 7y ago

Is it possible to make such a thing working if using command-line programs such as curl or wget to download a file rather than opening it in the web browser?

l1am0 7y ago
It should totaly be possible. Same as with the DNS solution you can get the data via the api of unshort.link and than build a small bash script for it. If I have time I may try to do such a script, but not sure if and when I will be able to make it. Would be awesome if you build something like that! You can get the currently to unshort.link known short link providers in a nice json format from https://unshort.link/providers https://unshort.link/providers You can get the redirect info from unshort link also via its api without redirecting, just replace /d/ with /api/ in the GET request: e.g. https://unshort.link/api/https://tinyurl.com/unshortchromeex.. https://unshort.link/api/https://tinyurl.com/unshortchromeex.... If you build a solution, please do not forget to add the documentation on how to do it via a pull request to the unshort.link repo, so everyone could profit from it
moehm 7y ago
This is what I use which also removes utm_* tracking parameters. getloc () { curl -sI "$1" | grep -i "Location" | awk '/http.*/{print $2}' | sed 's/[%+]*//g' | sed 's/&*utm_[a-zA-Z0-9]*=[a-zA-Z0-9]*//g' | sed 's/?\s*$//g' }

gesman 7y ago

Add: Replacement of affiliate or referral links with your user' affiliate link and you can charge for this service.

gesman 7y ago

I wanted to make something like this with my C.GG domain but couldn't refuse an offer to sell the name for $10k