12 ms·

Tell HN: How to ban url shorteners from linking to your site

22 points by pj 17y ago

This is just an idea I'm going to throw out there. It seems a bit odd to suggest it because any link is a good link, right? Maybe not. Anyway, something like this in a bit of javascript would prevent people from linking to your site using a url shortener: <script> if (document.referrer.indexOf('bit.ly') > 0 || document.referrer.indexOf('someothershortener')>0) document.location.href='http://yoursite.com/noshortener.html'; </script> Put in that noshortener.html some content that will tell people why you don't allow shorteners to your content, here are some suggestions: 1) You want to know where your visitors are coming from and shorteners steal the real referrer. 2) Shorteners are a layer of indirection that could be modified beyond the user of the shortener's discretion to do something like digg is doing now 3) If the shortener goes away, the link to the content is no longer valid 4) It slows down the web due to one more DNS lookup, one more server redirect, etc... Most users of shorteners, when they use one to link to a site are going to click the link just to make sure it works. When they do, they'll see your banned url shorteners page and it will help all of us "experts" in the web who understand why url shorteners are bad educate the lay users of the web as to why those url shorteners are bad. Yes, we take a hit, but we are sacrificing a little traffic for a better web experience in the future.

a2tech 17y ago
I tend to think that any link and visit is a good one. What I would prefer to do is using the above javascript, detect incoming traffic from URL shorteners and display my own informational bar across the top with the pages real URL and perhaps simple stats of how many people were referred to that page using a shortened link.
- pj 17y ago
  That's a good idea, instead of document.location.href="..."; You could use: document.getElementById('urlshorteners').style.display='block'; Where originally, the div is like this <body> <div id='urlshorteners' style='display:none;'> Url shorteners information... </div> I like that idea.
jerf 17y ago
Honest question: Why do you care? Consider this like an impromptu poll about why you care about not getting linked via a shortener. (If you don't care, I'd ask that you not post a reply to my message here, though I obviously can't enforce that.) I ask because I wouldn't care, but I am interested with an open mind in why someone would.
- calambrac 17y ago
  See current DiggBar scandal. These shorteners put a middleman in between your links and your content, allowing them to, at any point they like, do whatever they want, including sending all those links to themselves, to a competitor, to a landing page strewn with ads with your content framed inside, whatever. As much as we all love middlemen in other industries, seems like a great idea to let them in here, doesn't it? Joshua Schachter wrote the definitive anti-shortener argument: http://joshua.schachter.org/2009/04/on-url-shorteners.html http://joshua.schachter.org/2009/04/on-url-shorteners.html
- pj 17y ago
  I think really, because I care about the future of the web. I know there are myriad problems with url shorteners and they are jeopardizing the future of the web. My website is just one little piece of it, I know, but education is grand. I think people, when presented with rational reasons that benefit us all, will listen to them. The person linking to your content is interested more than anything in bringing users to your content. If you put up a page that prevents them from doing that, then they'll use another link, or not link to you at all. Really, it's about education and education comes at a cost. If you are educating, then you aren't focusing on coding or something else. On the page where the javascript redirects, you could say, "Thank you very much for linking to my content. I think it's great that you appreciate what I have written, but please understand that the method you are using to link to it is not good and here's why: yada yada yada..."
  - ubernostrum 17y ago
    You really think that being able to get a shortened URL to put in a post on Twitter is such a threat to the future of the Web that you need to, essentially, implement a huge "fuck you for trying to visit my site" message? If that's the case (and believe me, I've read Schachter's bit, and many of the zillions of "me too" posts following it, and so know the arguments inside and out at this point), I really don't have anything to say to you except "congratulations on your new job in the newspaper industry, I'm sure you'll fit right in".
zain 17y ago
None of the reasons you listed are significant to the person reading them. It is not the visitor's problem that you can't track referrers or that it slows down the web, and the fact that they're seeing this page means the shortened is working as intended. Furthermore, you're basically making it impossible to share links to your site on Twitter (where they get auto-shortened). All in all, you're doing little more than pissing off visitors you've probably worked pretty hard to get.
- pj 17y ago
  the fact that they're seeing this page means the shortened is working as intended They won't actually see the page they used the URL shortener to create a link to, because the browser will automatically redirect to another page where you have explained why you are disallowing url shortened links to your content. I also disagree with your premise. I think the reasons I mentioned are significant to all users of the web. I do understand the issue you have presented though, that the short term benefits to the user of using a shortened url, may outweigh the long term ramifications I have listed, but perhaps as a group here, we can come up with some arguments that may help us content producers take back the web from the url shorteners. Really, I just think when taken to the logical conclusion, a web saturated with url shortened links is really a not very good web. I'm trying to propose a solution here, it may not be the best one and it will take a coordinated effort among content producers. I suppose those who ignore the call to arms are going to "please" url shortener users more than those who don't, but with a little effort and a genuine interest in a sustainable web, I think we can win this war.
  - rex 17y ago
    ...the browser will automatically redirect to another page where you have explained why you are disallowing url shortened links to your content. If you're going to the trouble of serving up content from your site, why would you disallow the content they came to see anyway? Maybe the concept could work if you had support from higher-up in the redirection process (from url shortening services themselves), but to effectively re-direct from your own page to another "you an't access this content" page on your site will only make end users confused and disinterested, driving traffic away. I agree that URL shorteners are bad and acknowledge you're trying to do something about it, but this isn't the way to solve the problem.
callmeed 17y ago
I'm all for blocking framing sites like the Diggbar, but killing all URL shorteners is a bad idea, IMO. With Twitter growing so fast and automatically shortening pasted URLs, you could actually be losing some quality traffic if people start talking about your business/app/site.
- calambrac 17y ago
  Someone with a big vested interest in seeing the internet not break needs to step up and provide a shortening service that says "if you own the target, you own the shortened link, period".
  - barredo 17y ago
    like http://awe.sm http://awe.sm ?
- chaosprophet 17y ago
  I agree. Most of my sites have frame busting code in them, simply because most of the times, the frame spoils the look of my page. Also the frame sort of implies as though my page is part of the site that does the framing, and I find that annoying. As regards URL shorteners, I don't mind them one bit. The only problem I have with URL shorteners is that they may be used to disguise malicious links, and even that problem is being solved (bit.ly provides automatic warnings).
philfreo 17y ago
I think if you're going to do this it'd be better done as a server side redirect, like through .htaccess However even though I'm, in general, against URL shorteners, I wouldn't do this because I think there are some situations where they are helpful.
- pj 17y ago
  That's true. I just threw this together. I think you could even create a database table and generate the redirect on the server side or dynamically create the javascript. Most browsers have javascript enabled. Maybe a hacker with some time (maybe I'll create one) could create a site with a dynamically created .js file and a remote link like this: <script src="http://takebackthewebfromurlshorteners.com/bannedreferrerslist.js"> </script> This .js could be dynamically generated from a database table of known url shorteners. Such a service could even get really fancy and show stats from all the known shorteners, etc. You could add parameters like this: <script src=".../bannedlist.aspx?allow=digg.com,bit.ly,etc..."></script> Then the user could customize exactly which url shorteners they want to allow. All kinds of things a service like this could do!
  - nir 17y ago
    http://www.listable.org/json/url-shortening-sites http://www.listable.org/json/url-shortening-sites :)
    - pj 17y ago
      Yeah! That's what i'm talking about!! now it's just a FOR loop!
      - bhseo 17y ago
        http://www.blackhat-seo.com/2009/short-url-services/ http://www.blackhat-seo.com/2009/short-url-services/ A bigger list.
nir 17y ago
Might find this list of URL shorteners useful: http://www.listable.org/show/url-shortening-sites http://www.listable.org/show/url-shortening-sites (can be exported as JSON/SQL/plaintext)
tokenadult 17y ago
How long are the URLs on your site, that they are getting shortened so often?
- andrewf 17y ago
  Putting as much content in the URL as possible is a common SEO tactic. A bit silly, really - we're producing readable English links targeted at the machines, and short strings of unintelligible gibberish for humans to look at and click on.
  - jrockway 17y ago
    Putting as much content in the URL as possible is a common SEO tactic. Are there any controlled studies that shows that this does anything useful? Or is it an imagined effect? ("We think it should work this way, so it does!")
paul9290 17y ago
ALl links will die someday! Not every website is going to be around forever! The only ppl who hate on URL shorteners is this crowd here while millions of others usual them daily! We are hackers and thus if this is such a big issue and change, one of us should do something about it. Something that works for the users and solves the issue about 3% of all users rant about!
tallanvor 17y ago
Why not pass the url the user was trying to visit to your noshortener page? Then you can have a brief blurb and let them click the link to continue. You still don't get full referrer information, but it's better than nothing and the user gets to view the content.
keyist 17y ago
The reasons you give are all valid reasons. However, you miss the reason most likely to sway end-users: a shortener service can relate every page a user has seen to that user (or at least, to his ip). The privacy angle is the biggest factor imo. In your info page you may also want to promote Untiny (http://untiny.com/extra/ http://untiny.com/extra/) as a service which helps bypass shorteners with minimum fuss.
ErrantX 17y ago
As a user if you did that to me I would just close the link and lose interest in your site.... Yes, agreed, the DiggBar style thing IS annoying and we should stop that. But short links are important. Several forums I frequent insist you use shortners for URL's longer than 25 characters (you get a warning if you dont!) to keep things neat. Twitter - your isntantly killing traffic from that crowd. Now, showing a box on the page with some of that info would be useful / understandable.
jasonkester 17y ago
While we're talking about it, here's a feature I'd like to see these URL Shortening services impliment: - If the "Shortened" URL you are about to give me is actually longer than the URL I gave you, just give me back that URL. I see this sort of thing on Twitter all the time, and it drives me nuts. "hey, check out: http://tinyurl.com/mweytn http://tinyurl.com/mweytn ... which redirects you to... http://twiddla.com/ http://twiddla.com/ Uh... What exactly have you shortened again?
skwaddar 17y ago
I have no idea what your beef is, I think you have a stick up your ass about something. Why you think you're some sort of web "expert" is beyond me. A real expert wouldn't be using javascript for this for a start. Punishing third parties because of some moral crusade is dumb. Slows down the web hahahah jeesh.
jrockway 17y ago
This is exactly why I use noscript. But anyway, why not a standard API, similar to robots.txt or favicon.ico that will let each site control shortening? Requesting http://foo.com/shorten?url=http://foo.com/long/url/goes/here http://foo.com/shorten?url=http://foo.com/long/url/goes/here will return http://foo.com/lkjdhf http://foo.com/lkjdhf or whatever.
throw_away 17y ago
it would be nice if url shorteners did a double redirect so that the following happened: 1) user clicks on http://sho.rt/abcdef http://sho.rt/abcdef located on http://original-referrer.com http://original-referrer.com 2) sho.rt redirects to http://sho.rt/abcdef?original-referrer=http://original-referrer.com http://sho.rt/abcdef?original-referrer=http://original-refer... 3) second url redirects to the content. in this way, content owners will know where their traffic is coming from, people who place shortened urls don't have to do anything special and the pasted urls remain short. the only downside is a slight increase to latency to retrieve the second redirect.

22 points by pj 17y ago

a2tech 17y ago

I tend to think that any link and visit is a good one. What I would prefer to do is using the above javascript, detect incoming traffic from URL shorteners and display my own informational bar across the top with the pages real URL and perhaps simple stats of how many people were referred to that page using a shortened link.

pj 17y ago
That's a good idea, instead of document.location.href="..."; You could use: document.getElementById('urlshorteners').style.display='block'; Where originally, the div is like this <body> <div id='urlshorteners' style='display:none;'> Url shorteners information... </div> I like that idea.

jerf 17y ago

Honest question: Why do you care? Consider this like an impromptu poll about why you care about not getting linked via a shortener. (If you don't care, I'd ask that you not post a reply to my message here, though I obviously can't enforce that.) I ask because I wouldn't care, but I am interested with an open mind in why someone would.

calambrac 17y ago
See current DiggBar scandal. These shorteners put a middleman in between your links and your content, allowing them to, at any point they like, do whatever they want, including sending all those links to themselves, to a competitor, to a landing page strewn with ads with your content framed inside, whatever. As much as we all love middlemen in other industries, seems like a great idea to let them in here, doesn't it? Joshua Schachter wrote the definitive anti-shortener argument: http://joshua.schachter.org/2009/04/on-url-shorteners.html http://joshua.schachter.org/2009/04/on-url-shorteners.html
pj 17y ago
I think really, because I care about the future of the web. I know there are myriad problems with url shorteners and they are jeopardizing the future of the web. My website is just one little piece of it, I know, but education is grand. I think people, when presented with rational reasons that benefit us all, will listen to them. The person linking to your content is interested more than anything in bringing users to your content. If you put up a page that prevents them from doing that, then they'll use another link, or not link to you at all. Really, it's about education and education comes at a cost. If you are educating, then you aren't focusing on coding or something else. On the page where the javascript redirects, you could say, "Thank you very much for linking to my content. I think it's great that you appreciate what I have written, but please understand that the method you are using to link to it is not good and here's why: yada yada yada..."
- ubernostrum 17y ago
  You really think that being able to get a shortened URL to put in a post on Twitter is such a threat to the future of the Web that you need to, essentially, implement a huge "fuck you for trying to visit my site" message? If that's the case (and believe me, I've read Schachter's bit, and many of the zillions of "me too" posts following it, and so know the arguments inside and out at this point), I really don't have anything to say to you except "congratulations on your new job in the newspaper industry, I'm sure you'll fit right in".

zain 17y ago

None of the reasons you listed are significant to the person reading them. It is not the visitor's problem that you can't track referrers or that it slows down the web, and the fact that they're seeing this page means the shortened is working as intended. Furthermore, you're basically making it impossible to share links to your site on Twitter (where they get auto-shortened). All in all, you're doing little more than pissing off visitors you've probably worked pretty hard to get.

pj 17y ago
the fact that they're seeing this page means the shortened is working as intended They won't actually see the page they used the URL shortener to create a link to, because the browser will automatically redirect to another page where you have explained why you are disallowing url shortened links to your content. I also disagree with your premise. I think the reasons I mentioned are significant to all users of the web. I do understand the issue you have presented though, that the short term benefits to the user of using a shortened url, may outweigh the long term ramifications I have listed, but perhaps as a group here, we can come up with some arguments that may help us content producers take back the web from the url shorteners. Really, I just think when taken to the logical conclusion, a web saturated with url shortened links is really a not very good web. I'm trying to propose a solution here, it may not be the best one and it will take a coordinated effort among content producers. I suppose those who ignore the call to arms are going to "please" url shortener users more than those who don't, but with a little effort and a genuine interest in a sustainable web, I think we can win this war.
- rex 17y ago
  ...the browser will automatically redirect to another page where you have explained why you are disallowing url shortened links to your content. If you're going to the trouble of serving up content from your site, why would you disallow the content they came to see anyway? Maybe the concept could work if you had support from higher-up in the redirection process (from url shortening services themselves), but to effectively re-direct from your own page to another "you an't access this content" page on your site will only make end users confused and disinterested, driving traffic away. I agree that URL shorteners are bad and acknowledge you're trying to do something about it, but this isn't the way to solve the problem.

callmeed 17y ago

I'm all for blocking framing sites like the Diggbar, but killing all URL shorteners is a bad idea, IMO. With Twitter growing so fast and automatically shortening pasted URLs, you could actually be losing some quality traffic if people start talking about your business/app/site.

calambrac 17y ago
Someone with a big vested interest in seeing the internet not break needs to step up and provide a shortening service that says "if you own the target, you own the shortened link, period".
- barredo 17y ago
  like http://awe.sm http://awe.sm ?
chaosprophet 17y ago
I agree. Most of my sites have frame busting code in them, simply because most of the times, the frame spoils the look of my page. Also the frame sort of implies as though my page is part of the site that does the framing, and I find that annoying. As regards URL shorteners, I don't mind them one bit. The only problem I have with URL shorteners is that they may be used to disguise malicious links, and even that problem is being solved (bit.ly provides automatic warnings).

philfreo 17y ago

I think if you're going to do this it'd be better done as a server side redirect, like through .htaccess However even though I'm, in general, against URL shorteners, I wouldn't do this because I think there are some situations where they are helpful.

pj 17y ago
That's true. I just threw this together. I think you could even create a database table and generate the redirect on the server side or dynamically create the javascript. Most browsers have javascript enabled. Maybe a hacker with some time (maybe I'll create one) could create a site with a dynamically created .js file and a remote link like this: <script src="http://takebackthewebfromurlshorteners.com/bannedreferrerslist.js"> </script> This .js could be dynamically generated from a database table of known url shorteners. Such a service could even get really fancy and show stats from all the known shorteners, etc. You could add parameters like this: <script src=".../bannedlist.aspx?allow=digg.com,bit.ly,etc..."></script> Then the user could customize exactly which url shorteners they want to allow. All kinds of things a service like this could do!
- nir 17y ago
  http://www.listable.org/json/url-shortening-sites http://www.listable.org/json/url-shortening-sites :)
  - pj 17y ago
    Yeah! That's what i'm talking about!! now it's just a FOR loop!
    - bhseo 17y ago
      http://www.blackhat-seo.com/2009/short-url-services/ http://www.blackhat-seo.com/2009/short-url-services/ A bigger list.

nir 17y ago

Might find this list of URL shorteners useful: http://www.listable.org/show/url-shortening-sites http://www.listable.org/show/url-shortening-sites (can be exported as JSON/SQL/plaintext)

tokenadult 17y ago

How long are the URLs on your site, that they are getting shortened so often?

andrewf 17y ago
Putting as much content in the URL as possible is a common SEO tactic. A bit silly, really - we're producing readable English links targeted at the machines, and short strings of unintelligible gibberish for humans to look at and click on.
- jrockway 17y ago
  Putting as much content in the URL as possible is a common SEO tactic. Are there any controlled studies that shows that this does anything useful? Or is it an imagined effect? ("We think it should work this way, so it does!")

paul9290 17y ago

ALl links will die someday! Not every website is going to be around forever! The only ppl who hate on URL shorteners is this crowd here while millions of others usual them daily! We are hackers and thus if this is such a big issue and change, one of us should do something about it. Something that works for the users and solves the issue about 3% of all users rant about!

tallanvor 17y ago

Why not pass the url the user was trying to visit to your noshortener page? Then you can have a brief blurb and let them click the link to continue. You still don't get full referrer information, but it's better than nothing and the user gets to view the content.

keyist 17y ago

The reasons you give are all valid reasons. However, you miss the reason most likely to sway end-users: a shortener service can relate every page a user has seen to that user (or at least, to his ip). The privacy angle is the biggest factor imo. In your info page you may also want to promote Untiny (http://untiny.com/extra/ http://untiny.com/extra/) as a service which helps bypass shorteners with minimum fuss.

ErrantX 17y ago

As a user if you did that to me I would just close the link and lose interest in your site.... Yes, agreed, the DiggBar style thing IS annoying and we should stop that. But short links are important. Several forums I frequent insist you use shortners for URL's longer than 25 characters (you get a warning if you dont!) to keep things neat. Twitter - your isntantly killing traffic from that crowd. Now, showing a box on the page with some of that info would be useful / understandable.

jasonkester 17y ago

While we're talking about it, here's a feature I'd like to see these URL Shortening services impliment: - If the "Shortened" URL you are about to give me is actually longer than the URL I gave you, just give me back that URL. I see this sort of thing on Twitter all the time, and it drives me nuts. "hey, check out: http://tinyurl.com/mweytn http://tinyurl.com/mweytn ... which redirects you to... http://twiddla.com/ http://twiddla.com/ Uh... What exactly have you shortened again?

skwaddar 17y ago

I have no idea what your beef is, I think you have a stick up your ass about something. Why you think you're some sort of web "expert" is beyond me. A real expert wouldn't be using javascript for this for a start. Punishing third parties because of some moral crusade is dumb. Slows down the web hahahah jeesh.

jrockway 17y ago

This is exactly why I use noscript. But anyway, why not a standard API, similar to robots.txt or favicon.ico that will let each site control shortening? Requesting http://foo.com/shorten?url=http://foo.com/long/url/goes/here http://foo.com/shorten?url=http://foo.com/long/url/goes/here will return http://foo.com/lkjdhf http://foo.com/lkjdhf or whatever.

throw_away 17y ago

it would be nice if url shorteners did a double redirect so that the following happened: 1) user clicks on http://sho.rt/abcdef http://sho.rt/abcdef located on http://original-referrer.com http://original-referrer.com 2) sho.rt redirects to http://sho.rt/abcdef?original-referrer=http://original-referrer.com http://sho.rt/abcdef?original-referrer=http://original-refer... 3) second url redirects to the content. in this way, content owners will know where their traffic is coming from, people who place shortened urls don't have to do anything special and the pasted urls remain short. the only downside is a slight increase to latency to retrieve the second redirect.