3 ms·

If not, this would be a cool project for a third-party distributed project like SETI@home or folding@home.

by paulgb 17y ago

tibbon 17y ago
It's a lot of data, but not THAT much data. Nothing that a small amazon cluster Hadooping it couldn't do in just an hour or so. If anything, the bottleneck would probably be the URL shorting services themselves.
- chronomex 17y ago
  Yeah. I have a multithreaded perl script that can basically peg the database on the far end. This usually happens at about 20-30 threads. It sort of leaks memory, to the point that restarting it every day or so is advisable. http://github.com/chronomex/urlteam http://github.com/chronomex/urlteam if you're interested in my code.
chronomex 17y ago
I'm a member of ArchiveTeam (http://www.archiveteam.org/ http://www.archiveteam.org/). One of our current projects is to archive all the URL shorteners we can find. We're going to make the full dataset available for public download.
- jf 17y ago
  This is fantastic news! Please announce the public dataset on HN when it's ready for public download.
  - chronomex 17y ago
    I totally hadn't thought about that, but I will. :)
- alanh 17y ago
  Can we report shorteners? For example, I host a private-use one at http://ajh.us http://ajh.us that's powered by my project http://lessnmore.net http://lessnmore.net
  - Kadin 17y ago
    While they may well have a way for you to submit archives, I think the greater question to ask is "why?" URL shorteners are bad enough on their own, but arguably a necessary evil -- encouraging everyone to set up their own is in no way a good thing. I can't think of a worse service for an average user (or anyone without a datacenter, or better yet multiple datacenters) to run. By all means run your own email server -- if it craps out, at least you'll just lose your email. But if you run a URL shortening service and send out or post a lot of URLs using it, and then you go bankrupt, die, or just get bored and decide to stop running it ... now everyone who happens to have one of the links is S.O.L. It strikes me as the kind of service that's relatively easy to run, but unless someone is prepared to run it forever, they probably shouldn't.
    - alanh 17y ago
      I am using it primarily to shorten links back to http://alanhogan.com http://alanhogan.com, and since I run both domains, I intend to keep both of them running as long as possible -- and when one goes down, the other becomes pretty much useless. But yes, I agree that it is important to intend to run these services forever.
  - chronomex 17y ago
    You can add them to the master list at http://archiveteam.org/index.php?title=TinyURL http://archiveteam.org/index.php?title=TinyURL . If you know it, include the charset and algorithm that the short URLs are generated from. If you don't mind putting a CSV copy of your database up for public download, that'd be even more awesome! (http://ur1.ca/ http://ur1.ca/ does this, for example.) Plus then I don't load up your database server with my multithreaded, rather antisocial scraper of doom.

If not, this would be a cool project for a third-party distributed project like SETI@home or folding@home.

by paulgb 17y ago

If not, this would be a cool project for a third-party distributed project like SETI@home or folding@home.

It's a lot of data, but not THAT much data. Nothing that a small amazon cluster Hadooping it couldn't do in just an hour or so. If anything, the bottleneck would probably be the URL shorting services themselves.

chronomex 17y ago
Yeah. I have a multithreaded perl script that can basically peg the database on the far end. This usually happens at about 20-30 threads. It sort of leaks memory, to the point that restarting it every day or so is advisable. http://github.com/chronomex/urlteam http://github.com/chronomex/urlteam if you're interested in my code.

chronomex 17y ago

I'm a member of ArchiveTeam (http://www.archiveteam.org/ http://www.archiveteam.org/). One of our current projects is to archive all the URL shorteners we can find. We're going to make the full dataset available for public download.

jf 17y ago
This is fantastic news! Please announce the public dataset on HN when it's ready for public download.
- chronomex 17y ago
  I totally hadn't thought about that, but I will. :)
alanh 17y ago
Can we report shorteners? For example, I host a private-use one at http://ajh.us http://ajh.us that's powered by my project http://lessnmore.net http://lessnmore.net
- Kadin 17y ago
  While they may well have a way for you to submit archives, I think the greater question to ask is "why?" URL shorteners are bad enough on their own, but arguably a necessary evil -- encouraging everyone to set up their own is in no way a good thing. I can't think of a worse service for an average user (or anyone without a datacenter, or better yet multiple datacenters) to run. By all means run your own email server -- if it craps out, at least you'll just lose your email. But if you run a URL shortening service and send out or post a lot of URLs using it, and then you go bankrupt, die, or just get bored and decide to stop running it ... now everyone who happens to have one of the links is S.O.L. It strikes me as the kind of service that's relatively easy to run, but unless someone is prepared to run it forever, they probably shouldn't.
  - alanh 17y ago
    I am using it primarily to shorten links back to http://alanhogan.com http://alanhogan.com, and since I run both domains, I intend to keep both of them running as long as possible -- and when one goes down, the other becomes pretty much useless. But yes, I agree that it is important to intend to run these services forever.
- chronomex 17y ago
  You can add them to the master list at http://archiveteam.org/index.php?title=TinyURL http://archiveteam.org/index.php?title=TinyURL . If you know it, include the charset and algorithm that the short URLs are generated from. If you don't mind putting a CSV copy of your database up for public download, that'd be even more awesome! (http://ur1.ca/ http://ur1.ca/ does this, for example.) Plus then I don't load up your database server with my multithreaded, rather antisocial scraper of doom.