6 ms·

Show HN: Avoiding Imgur Link Rot

68 points by rvavruch 3y ago

Over the last decade I've built a number of different digital asset managers (mostly media files) that met the needs of my companies at the time. It is an area I enjoy working in. A month ago, when asked what was next for me, I jokingly said I would build another DAM. Then on Saturday, two weeks ago, I learnt that Imgur was going to delete all anonymous & NSFW files on the 15th of May. It was pointed out that this would mean broken links in communities that had relied on Imgur. By the Sunday I had decided that I would build another DAM, initially with the intent of avoiding Imgur link rot. It was challenging to find time to spend on this, the project was put together over about 8 evenings. It still has rough patches, this is an early MVP (a Michael Seibel "brick"). I have many ideas of where to take this project, but for now it only does one thing: backup Imgur files and produce new links that are easy to swap out for old soon-to-be-deleted Imgur links.

nicklaf 3y ago
Interesting work! Thanks for putting this together. One potentially problematic thing about the new Imgur policy is that it will break machine learning projects that depend on Imgur for NSFW data. For example, here's a project relying on Imgur links in order to train a classifier that detects adult content: https://github.com/alex000kim/nsfw_data_scraper https://github.com/alex000kim/nsfw_data_scraper https://github.com/EBazarov/nsfw_data_source_urls https://github.com/EBazarov/nsfw_data_source_urls
MayeulC 3y ago
That makes me think something like this could be a nice phpbb plugin. Let forum admins compile a list of linked imgur posts that should be archived. Storage and bandwidth are cheap compared to what they used to be, so some may be able to self-host images now.
rkagerer 3y ago
Curious why not a "recycling" approach where images that are about to expire on Imgur are simply reposted to Imgur gaining a fresh URL?
csydas 3y ago
Interesting service, but an earnest question about your Terms and Conditions[0]. How closely did you read this? Since I see you used an TOC generator (which was nice for me learn about), I'm just curious how closely you read some of the conditions. I just may be reading the Prohibited[1] section too closely as well as the User Generated Content [2] section, but it seems like your stated goal about protecting the NSFW content from Imgur, as I guess such content is potentially obscene, lewd, lascivious, filthy, violent, harassing, libellous, slanderous, or otherwise objectionable (as determined by [you and your team]). I'm not here to trash your service or trying to "gotcha", just since you're responding in the thread, I'm only curious how closely you've read the TOC and if you stand by all the elements of it. 0 - https://haasie.com/terms_and_conditions https://haasie.com/terms_and_conditions 1 - https://haasie.com/terms_and_conditions#prohibited https://haasie.com/terms_and_conditions#prohibited 2- https://haasie.com/terms_and_conditions#ugc https://haasie.com/terms_and_conditions#ugc edit: fixed link formatting
- rvavruch 3y ago
  Fair question. I did read the TOCs, and studied the current TOCs of Imgur as well as my hosting company to make sure Haasie's covered the most important aspects. Imgur atm still allows for NSFW, although presumably this will change. But as it stands the contents of these TOCs are not dissimilar from Imgur's or any other file hosts that allow for NSFW content. To clarify Haasie is not intended to be a specifically NSFW host - like RedGIF someone here mentioned - but rather to host any legal content, which includes NSFW. Emphasis here is on legal. The intention behind the TOCs is to specifically prohibit the use of the service for anything illegal or otherwise prohibited by the hosting company. And to state that Haasie will act on any content that is found to be in violation of the TOCs.
  - csydas 3y ago
    thanks for the answer :) really, not trying to "gotcha", just was curious if your read of your current TOC was compatible with your goals. I didn't meant to indicate you were exclusively looking for nsfw (but I see how what I wrote implied it), just more a coffee thought while reading this thread :) good luck on the project!
gkbrk 3y ago
> Well done, you’ve defeated link rot! Except you haven't, because anyone that wants to view these images where they are linked needs to know to change the URL to Haasie instead of Imgur. So all the links are still broken. And on top of that, even if the users know about Haasie, it only works if someone went to this website before the original image was deleted, and enter the URL on a form. Non-popular images will be long gone before people do this manually. Perhaps accepting any URL and transparently fetching it from Imgur would improve this a little. From my manual testing, this doesn't seem to be the case, just gives an S3 error from bucket ID haasie01, and nothing shows up there even if I wait and refresh. All of this is assuming this website won't go down or go bad. Probably the Archive Team, and Archive.org can handle this task better.
rvavruch 3y ago
Thanks for all the questions. It has really gotten me to thinking about how to navigate the future of the project. I have a question for all of you. My approach is that we are all understand that there is no such thing as a free service, and that for something like this to be long term sustainable donations (or some source of revenue) is crucial. If people get value from the service then it only makes sense to give back to the service to sustain it. With this in mind, I'm toying with an idea that I would like to get a temperature check from HN on. How would you react if, when donations are too low, Haasie serves a single image that requests donations to sustain the service in place of the original anonymously uploaded content, until the required donation level is reached? Once there are enough donations normal service is resumed.
quickthrower2 3y ago
First off: great work! The age old question is how you afford to run this, and how if you get popular do you not become an imgur yourself and have their same pressures to delete, moderate and make money.
- rvavruch 3y ago
  Thank you! My thinking is to start with donations. There is no such thing as a free service. If people get value from using it to upload anonymously and they want to keep it going, makes sense to donate. In the future I could also add a subscription account for additional features, access to an API, bigger upload limits, transformations, greater ability to organise, etc.
  - not_your_vase 3y ago
    Is there a backup plan? Not saying that relying on donations is impossible, but realistically there is a very small chance that enough donation will flow in continuously. As a sidenote, reddit is (was?) the one of main user of imgur, however https://www.google.com/search?q=%22haasie.com%22+site%3Areddit.com https://www.google.com/search?q=%22haasie.com%22+site%3Aredd... yields no results - maybe it is worth to promote it there also.
    - rvavruch 3y ago
      Your question honestly got me thinking. If everything goes completely sideways there are a number of shuttering options I can think of. One is to turn off any new uploads, and just serve the existing content, presumably the bandwidth on old content would be much lower - until some sustainable plan can be devised. Last resort is to approach the ArchiveTeam mentioned here to see if and we could to setup a maintenance service. To your side note, yes, I want to promote it on Reddit, as soon as I've added a few more features - like donations.
- flangola7 3y ago
  I say we need basic universal internet infrastructure in the same way we have roads, libraries, hospitals, mail service, etc. Fundamental services like email, image and video hosting, backup storage, small cloud nodes, and so on. All partially or fully publicly funded, with no bloat or wasted overhead trying to commercialize it into an advertising platform filled with spyware and dark patterns.
tennisflyi 3y ago
Is RedGIFs not Imgur for porn?
- newjersey 3y ago
  Red gifs is a gfycat product though, not an Imgur one.
- teej 3y ago
  Imgur hosts a lot of porn and was the preferred image host for some popular NSFW subreddits.
  - deleted 3y ago
    [deleted]
- CamperBob2 3y ago
  Only a matter of time until the same thing happens there.
cantsingh 3y ago
Could it be configured so if I go to, for example, https://i.haasie.com/something.png https://i.haasie.com/something.png and you haven't already backed up from imgur, you back it up when that URL is hit?
- rvavruch 3y ago
  That's a fantastic idea, thanks!
- jonhohle 3y ago
  Yeah, a read through cache seems more obvious than opt-in.
- Sakos 3y ago
  Definitely agree. It also feels like this would be perfect in combination with something like ipfs for a decentralized way to store and access these images.
  - rvavruch 3y ago
    Agreed. I've played with IPFS before. Very interesting, definitely something to consider for the future of Haasie.
pabs3 3y ago
Could you export a list of all imgur URLs saved, so that ArchiveTeam can help save those to archive.org too? https://wiki.archiveteam.org/index.php/Imgur https://wiki.archiveteam.org/index.php/Imgur
- rvavruch 3y ago
  Thanks, I will check them out.
_a9 3y ago
> Well done, you’ve defeated link rot Until this site succumbs to the challenges of being a free image host? FWIW ArchiveTeam is grabbing as much as possible so it could eventually show up in the wayback machine. Though what they have now, 47TB, is a drop to the petabytes imgur probably has now. There was an estimated 376 TiB in 2015. - https://wiki.archiveteam.org/index.php/Imgur https://wiki.archiveteam.org/index.php/Imgur - https://tracker.archiveteam.org/imgur/ https://tracker.archiveteam.org/imgur/
- rvavruch 3y ago
  My intention is not to be a free host though - that's not sustainable. I hope to fund it through donations and potentially a subscription for enhanced features.
  - scrollaway 3y ago
    Donations are not sustainable either. You need a business plan.
  - deleted 3y ago
    [deleted]
  - celsoazevedo 3y ago
    Sites like catbox[.]moe are popular and most months seem to fail to cover all their expenses with donations. And that's using cheap hosting (ovh, hetzner, etc, and no CDN), I imagine that your Fastly bill will be higher than theirs. For this to work, I think you need to find revenue sources right away and run it as cheaply as possible, otherwise you'll run into problems, creating more link rot in the process. Anyway, I hope you manage to make it work.
    - rvavruch 3y ago
      Thanks, agreed it will be a challenge.
- stavros 3y ago
  Free image hosts just aren't sustainable, the costs are high and it's fairly hard to make that money back. I made a paid image host a while back (https://imgz.org https://imgz.org) exactly because of that reason.

68 points by rvavruch 3y ago

nicklaf 3y ago

Interesting work! Thanks for putting this together. One potentially problematic thing about the new Imgur policy is that it will break machine learning projects that depend on Imgur for NSFW data. For example, here's a project relying on Imgur links in order to train a classifier that detects adult content: https://github.com/alex000kim/nsfw_data_scraper https://github.com/alex000kim/nsfw_data_scraper https://github.com/EBazarov/nsfw_data_source_urls https://github.com/EBazarov/nsfw_data_source_urls

MayeulC 3y ago

That makes me think something like this could be a nice phpbb plugin. Let forum admins compile a list of linked imgur posts that should be archived. Storage and bandwidth are cheap compared to what they used to be, so some may be able to self-host images now.

rkagerer 3y ago

Curious why not a "recycling" approach where images that are about to expire on Imgur are simply reposted to Imgur gaining a fresh URL?

csydas 3y ago

Interesting service, but an earnest question about your Terms and Conditions[0]. How closely did you read this? Since I see you used an TOC generator (which was nice for me learn about), I'm just curious how closely you read some of the conditions. I just may be reading the Prohibited[1] section too closely as well as the User Generated Content [2] section, but it seems like your stated goal about protecting the NSFW content from Imgur, as I guess such content is potentially obscene, lewd, lascivious, filthy, violent, harassing, libellous, slanderous, or otherwise objectionable (as determined by [you and your team]). I'm not here to trash your service or trying to "gotcha", just since you're responding in the thread, I'm only curious how closely you've read the TOC and if you stand by all the elements of it. 0 - https://haasie.com/terms_and_conditions https://haasie.com/terms_and_conditions 1 - https://haasie.com/terms_and_conditions#prohibited https://haasie.com/terms_and_conditions#prohibited 2- https://haasie.com/terms_and_conditions#ugc https://haasie.com/terms_and_conditions#ugc edit: fixed link formatting

rvavruch 3y ago
Fair question. I did read the TOCs, and studied the current TOCs of Imgur as well as my hosting company to make sure Haasie's covered the most important aspects. Imgur atm still allows for NSFW, although presumably this will change. But as it stands the contents of these TOCs are not dissimilar from Imgur's or any other file hosts that allow for NSFW content. To clarify Haasie is not intended to be a specifically NSFW host - like RedGIF someone here mentioned - but rather to host any legal content, which includes NSFW. Emphasis here is on legal. The intention behind the TOCs is to specifically prohibit the use of the service for anything illegal or otherwise prohibited by the hosting company. And to state that Haasie will act on any content that is found to be in violation of the TOCs.
- csydas 3y ago
  thanks for the answer :) really, not trying to "gotcha", just was curious if your read of your current TOC was compatible with your goals. I didn't meant to indicate you were exclusively looking for nsfw (but I see how what I wrote implied it), just more a coffee thought while reading this thread :) good luck on the project!

gkbrk 3y ago

> Well done, you’ve defeated link rot! Except you haven't, because anyone that wants to view these images where they are linked needs to know to change the URL to Haasie instead of Imgur. So all the links are still broken. And on top of that, even if the users know about Haasie, it only works if someone went to this website before the original image was deleted, and enter the URL on a form. Non-popular images will be long gone before people do this manually. Perhaps accepting any URL and transparently fetching it from Imgur would improve this a little. From my manual testing, this doesn't seem to be the case, just gives an S3 error from bucket ID haasie01, and nothing shows up there even if I wait and refresh. All of this is assuming this website won't go down or go bad. Probably the Archive Team, and Archive.org can handle this task better.

rvavruch 3y ago

Thanks for all the questions. It has really gotten me to thinking about how to navigate the future of the project. I have a question for all of you. My approach is that we are all understand that there is no such thing as a free service, and that for something like this to be long term sustainable donations (or some source of revenue) is crucial. If people get value from the service then it only makes sense to give back to the service to sustain it. With this in mind, I'm toying with an idea that I would like to get a temperature check from HN on. How would you react if, when donations are too low, Haasie serves a single image that requests donations to sustain the service in place of the original anonymously uploaded content, until the required donation level is reached? Once there are enough donations normal service is resumed.

quickthrower2 3y ago

First off: great work! The age old question is how you afford to run this, and how if you get popular do you not become an imgur yourself and have their same pressures to delete, moderate and make money.

rvavruch 3y ago
Thank you! My thinking is to start with donations. There is no such thing as a free service. If people get value from using it to upload anonymously and they want to keep it going, makes sense to donate. In the future I could also add a subscription account for additional features, access to an API, bigger upload limits, transformations, greater ability to organise, etc.
- not_your_vase 3y ago
  Is there a backup plan? Not saying that relying on donations is impossible, but realistically there is a very small chance that enough donation will flow in continuously. As a sidenote, reddit is (was?) the one of main user of imgur, however https://www.google.com/search?q=%22haasie.com%22+site%3Areddit.com https://www.google.com/search?q=%22haasie.com%22+site%3Aredd... yields no results - maybe it is worth to promote it there also.
  - rvavruch 3y ago
    Your question honestly got me thinking. If everything goes completely sideways there are a number of shuttering options I can think of. One is to turn off any new uploads, and just serve the existing content, presumably the bandwidth on old content would be much lower - until some sustainable plan can be devised. Last resort is to approach the ArchiveTeam mentioned here to see if and we could to setup a maintenance service. To your side note, yes, I want to promote it on Reddit, as soon as I've added a few more features - like donations.
flangola7 3y ago
I say we need basic universal internet infrastructure in the same way we have roads, libraries, hospitals, mail service, etc. Fundamental services like email, image and video hosting, backup storage, small cloud nodes, and so on. All partially or fully publicly funded, with no bloat or wasted overhead trying to commercialize it into an advertising platform filled with spyware and dark patterns.

tennisflyi 3y ago

Is RedGIFs not Imgur for porn?

newjersey 3y ago
Red gifs is a gfycat product though, not an Imgur one.
teej 3y ago
Imgur hosts a lot of porn and was the preferred image host for some popular NSFW subreddits.
- deleted 3y ago
  [deleted]
CamperBob2 3y ago
Only a matter of time until the same thing happens there.

cantsingh 3y ago

Could it be configured so if I go to, for example, https://i.haasie.com/something.png https://i.haasie.com/something.png and you haven't already backed up from imgur, you back it up when that URL is hit?

rvavruch 3y ago
That's a fantastic idea, thanks!
jonhohle 3y ago
Yeah, a read through cache seems more obvious than opt-in.
Sakos 3y ago
Definitely agree. It also feels like this would be perfect in combination with something like ipfs for a decentralized way to store and access these images.
- rvavruch 3y ago
  Agreed. I've played with IPFS before. Very interesting, definitely something to consider for the future of Haasie.

pabs3 3y ago

Could you export a list of all imgur URLs saved, so that ArchiveTeam can help save those to archive.org too? https://wiki.archiveteam.org/index.php/Imgur https://wiki.archiveteam.org/index.php/Imgur

rvavruch 3y ago
Thanks, I will check them out.

_a9 3y ago

> Well done, you’ve defeated link rot Until this site succumbs to the challenges of being a free image host? FWIW ArchiveTeam is grabbing as much as possible so it could eventually show up in the wayback machine. Though what they have now, 47TB, is a drop to the petabytes imgur probably has now. There was an estimated 376 TiB in 2015. - https://wiki.archiveteam.org/index.php/Imgur https://wiki.archiveteam.org/index.php/Imgur - https://tracker.archiveteam.org/imgur/ https://tracker.archiveteam.org/imgur/

rvavruch 3y ago
My intention is not to be a free host though - that's not sustainable. I hope to fund it through donations and potentially a subscription for enhanced features.
- scrollaway 3y ago
  Donations are not sustainable either. You need a business plan.
- deleted 3y ago
  [deleted]
- celsoazevedo 3y ago
  Sites like catbox[.]moe are popular and most months seem to fail to cover all their expenses with donations. And that's using cheap hosting (ovh, hetzner, etc, and no CDN), I imagine that your Fastly bill will be higher than theirs. For this to work, I think you need to find revenue sources right away and run it as cheaply as possible, otherwise you'll run into problems, creating more link rot in the process. Anyway, I hope you manage to make it work.
  - rvavruch 3y ago
    Thanks, agreed it will be a challenge.
stavros 3y ago
Free image hosts just aren't sustainable, the costs are high and it's fairly hard to make that money back. I made a paid image host a while back (https://imgz.org https://imgz.org) exactly because of that reason.