6 ms·

I've had a similar problem. In updating my portfolio site recently, I noticed a vast majority of links were dead. Not just live projects published maybe 3 years

I've had a similar problem. In updating my portfolio site recently, I noticed a vast majority of links were dead. Not just live projects published maybe 3 years or more ago (I expect those to die). But also links to articles and mentions from barely one year ago, or links to award sites, and the like. With a site listing projects going back ~15 years, one can imagine how bad things were. I had to end up creating a link component that would automatically link to an archive.org version of the link on every URL if I marked as "dead". It was so prevalent it had to be automated like that. Another reason why I've been contributing $100/year to the Internet Archive for the past 3 years and will continue to do so. They're doing some often unsung but important work.

deleted 9y ago
[deleted]
LoSboccacc 9y ago
beware: robots.txt can retroactively clear archive.org data
- deadalus 9y ago
  another reason to use archive.is
  - notyourwork 9y ago
    What is the difference between these two?
    - snerbles 9y ago
      archive.is does not crawl automatically, it must be pointed at a page by a user. While this makes it particularly useful for snapshotting frequently-changed pages, it is not a replacement for the proper Internet Archive.
  - dublinben 9y ago
    perma.cc is another option as well.
  - msla 9y ago
    archive.is is very nice, but they're a URL-shortener as well, so their links are utterly opaque strings of alphanumerics, whereas the Wayback Machine preserves both the full original URL and the date and time it was captured in the archival URL.
  - pilsetnieks 9y ago
    Considering the topic of discussion, how sure can you be that archive.is will still be around in a year? Three years? Ten? As much as I tried, all I could find about it is that it's run by one guy in Czech Republic who's paying $2000/month out of pocket for hosting, and apparently dislikes Finland.
    - exikyut 9y ago
      It's actually worse - http://blog.archive.is/post/151510917631/how-do-you-guys-keep-the-lights-on-i-gave-the http://blog.archive.is/post/151510917631/how-do-you-guys-kee... says it's $4k/mo. http://archive.is/robots.txt http://archive.is/robots.txt doesn't seem too bad, it looks like you could slowly inhale everything... in theory. There are no sitemaps (they're there, but empty placeholders); you have to know the site name to be able to get a workable list. The author hasn't ruled out/blocked archiving the snapshots, but apparently it's... big. http://blog.archive.is/post/154930531126/if-someone-was-willing-to-make-a http://blog.archive.is/post/154930531126/if-someone-was-will...
    - cooper12 9y ago
      I think http://www.webcitation.org/ http://www.webcitation.org/ might be better in that regard since it's a consortium of "editors, publishers, libraries". See "How can I be assured that archived material remains accessible and that webcitation.org doesn't disappear in the future?" in their FAQ (http://www.webcitation.org/faq http://www.webcitation.org/faq). Although from my perspective it seems to be more geared towards academic use.
- C4K3 9y ago
  Hopefully not anymore: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
- viddi 9y ago
  To clear things up: robots.txt can retroactively hide content from the archive. If it's changed back to allowing the archive's crawler, content from before the ban can be accessed again.
jedberg 9y ago
I actually downloaded a copy of the NYT article I was quoted in in 1996 specifically because I feared it would fall off the internet at some point. It's behind a paywall now, but at least I have a digital copy!
- Asooka 9y ago
  Isn't that illegal, since you don't own the copyright? Or are you not distributing it and keeping it for archive purposes?
  - jedberg 9y ago
    It's on my website, which is a blatant copyright violation. So far the NYT hasn't asked me to take it down. It could potentially be considered fair use, since I'm not making a profit and I provide commentary.
    - exergy 9y ago
      > It's on my website, which is a blatant copyright violation Tangential, but I long for a world where we can all be as candid with each other.
    - ringaroundthetx 9y ago
      > It could potentially be considered fair use, since I'm not making a profit and I provide commentary. Although people through that term around willy nilly, in our current framework that means being sued for a minimum of $100,000 per supposed violation, and making your fair use defense in front of a judge. Youtubers have reported spending $50,000 just to begin talking with lawyers and preparing briefs. Maybe our ISPs can start offering us insurance.
      - DiabloD3 9y ago
        If he is actually providing commentary and it does meet the fair use requirements, the EFF would probably end up representing him. Remember to donate to the EFF, they're literally the only thing between you and a world where the corporations rule the world.
cheapsteak 9y ago
For portfolios, you should really also look into https://webrecorder.io/ https://webrecorder.io/ It's _not_ a video recording service. It saves and can replay all network requests during a session (including authenticated requests). It's open source, you can self host, I'm not affiliated even though I'm very happy that it exists
- no_news_is 9y ago
  Thanks for mentioning this site. I've done this with mitmproxy but it's complicated, and this was super easy -- I could recommend to anyone.
- whatever_dude 9y ago
  That's impressive. I'll take a look, thanks for the tip.
nfriedly 9y ago
I've updated my portfolio before and noticed that as well. I usually include a screenshot or two when I first add a project, so at least that remains. If the site goes down later, I just remove the link and don't worry about it. My code from 15 years ago is probably atrocious, so I'll consider it a small blessing :P

deleted 9y ago

[deleted]

LoSboccacc 9y ago

beware: robots.txt can retroactively clear archive.org data

deadalus 9y ago
another reason to use archive.is
- notyourwork 9y ago
  What is the difference between these two?
  - snerbles 9y ago
    archive.is does not crawl automatically, it must be pointed at a page by a user. While this makes it particularly useful for snapshotting frequently-changed pages, it is not a replacement for the proper Internet Archive.
- dublinben 9y ago
  perma.cc is another option as well.
- msla 9y ago
  archive.is is very nice, but they're a URL-shortener as well, so their links are utterly opaque strings of alphanumerics, whereas the Wayback Machine preserves both the full original URL and the date and time it was captured in the archival URL.
- pilsetnieks 9y ago
  Considering the topic of discussion, how sure can you be that archive.is will still be around in a year? Three years? Ten? As much as I tried, all I could find about it is that it's run by one guy in Czech Republic who's paying $2000/month out of pocket for hosting, and apparently dislikes Finland.
  - exikyut 9y ago
    It's actually worse - http://blog.archive.is/post/151510917631/how-do-you-guys-keep-the-lights-on-i-gave-the http://blog.archive.is/post/151510917631/how-do-you-guys-kee... says it's $4k/mo. http://archive.is/robots.txt http://archive.is/robots.txt doesn't seem too bad, it looks like you could slowly inhale everything... in theory. There are no sitemaps (they're there, but empty placeholders); you have to know the site name to be able to get a workable list. The author hasn't ruled out/blocked archiving the snapshots, but apparently it's... big. http://blog.archive.is/post/154930531126/if-someone-was-willing-to-make-a http://blog.archive.is/post/154930531126/if-someone-was-will...
  - cooper12 9y ago
    I think http://www.webcitation.org/ http://www.webcitation.org/ might be better in that regard since it's a consortium of "editors, publishers, libraries". See "How can I be assured that archived material remains accessible and that webcitation.org doesn't disappear in the future?" in their FAQ (http://www.webcitation.org/faq http://www.webcitation.org/faq). Although from my perspective it seems to be more geared towards academic use.
C4K3 9y ago
Hopefully not anymore: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
viddi 9y ago
To clear things up: robots.txt can retroactively hide content from the archive. If it's changed back to allowing the archive's crawler, content from before the ban can be accessed again.

jedberg 9y ago

I actually downloaded a copy of the NYT article I was quoted in in 1996 specifically because I feared it would fall off the internet at some point. It's behind a paywall now, but at least I have a digital copy!

Asooka 9y ago
Isn't that illegal, since you don't own the copyright? Or are you not distributing it and keeping it for archive purposes?
- jedberg 9y ago
  It's on my website, which is a blatant copyright violation. So far the NYT hasn't asked me to take it down. It could potentially be considered fair use, since I'm not making a profit and I provide commentary.
  - exergy 9y ago
    > It's on my website, which is a blatant copyright violation Tangential, but I long for a world where we can all be as candid with each other.
  - ringaroundthetx 9y ago
    > It could potentially be considered fair use, since I'm not making a profit and I provide commentary. Although people through that term around willy nilly, in our current framework that means being sued for a minimum of $100,000 per supposed violation, and making your fair use defense in front of a judge. Youtubers have reported spending $50,000 just to begin talking with lawyers and preparing briefs. Maybe our ISPs can start offering us insurance.
    - DiabloD3 9y ago
      If he is actually providing commentary and it does meet the fair use requirements, the EFF would probably end up representing him. Remember to donate to the EFF, they're literally the only thing between you and a world where the corporations rule the world.

cheapsteak 9y ago

For portfolios, you should really also look into https://webrecorder.io/ https://webrecorder.io/ It's _not_ a video recording service. It saves and can replay all network requests during a session (including authenticated requests). It's open source, you can self host, I'm not affiliated even though I'm very happy that it exists

no_news_is 9y ago
Thanks for mentioning this site. I've done this with mitmproxy but it's complicated, and this was super easy -- I could recommend to anyone.
whatever_dude 9y ago
That's impressive. I'll take a look, thanks for the tip.

nfriedly 9y ago

I've updated my portfolio before and noticed that as well. I usually include a screenshot or two when I first add a project, so at least that remains. If the site goes down later, I just remove the link and don't worry about it. My code from 15 years ago is probably atrocious, so I'll consider it a small blessing :P