6 ms·

I've been working on this problem for a while. Website upkeep is hard to quantify, but basically every disk fails and every operating system eventually needs a

I've been working on this problem for a while. Website upkeep is hard to quantify, but basically every disk fails and every operating system eventually needs a serious upgrade. The timeframe that a system can run continuously is not that long compared to the timeframe that information is relevant. So the most lightweight way to keep something up and running is to make it trivial to port to many hosting configurations by simplifying the toolchain needed to rehost it. (Note that humans are part of that workflow, if it's a company) I've written a manifesto about making a commitment to keep websites online and maintained for 10-30 years, for people who are maintaining web content: https://jeffhuang.com/designed_to_last/ https://jeffhuang.com/designed_to_last/ And on the flipside (from a user's point of view), I've also been working on a background process that automatically captures full-resolution screenshots of every website you visit, creating your own searchable personal web archive: https://irchiver.com/ https://irchiver.com/ I've personally been trying to make a commitment to keep my web projects and writing online for 30 years. My original internal goal when I started thinking about this, was to outlast all the content on Twitter, Google+, and facebook.com. One of those has already been met, kind of sadly.

anonymoushn 4y ago
irchiver seems incredible. I hope there's a comparable product for other OSes one day.
- ryanfox 4y ago
  I’ve been working on a very similar thing which runs on Windows, Mac, and Linux: https://apse.io https://apse.io
10000truths 4y ago
It’s not uncommon to hear of rack servers with several years of continuous uptime. I wouldn’t be surprised if you could keep a website online for a decade without touching anything by using an LTS distro, enabling unattended upgrades, and running something like nginx.
EddieDante 4y ago
> I've also been working on a background process that automatically captures full-resolution screenshots of every website you visit, creating your own searchable personal web archive: How are screenshots searchable? They aren't plain text. You can't grep them.
- anonymoushn 4y ago
  The tool captures screenshots in addition to text.
  - doodles33 4y ago
    That seems awfully simillar to archive.org wayback machine. I do like to see all these archival projects though, they are certainly worthwhile.
    - lazyjeff 4y ago
      irchiver captures text on the page, and separately OCRs the screenshots (specifically, the screenshot from your viewport). So you can search just what was shown on the page, or what was in the page. Both techniques have pros and cons. While archive.org is fantastic, it can only capture pages that are both 1) publicly accessible (i.e. no social media content) that it happens to crawl, and 2) static content (you're out of luck if the content you want is loaded dynamically, or changes depending on user input).
      - pabs3 4y ago
        IIRC archive.org does save the JS and things it downloads, so you can replay them when you visit the archived site later.
        doodlesdev 4y ago
        I guess the difference in this case is that the JS on web archive relies on future browsers being backwards compatible, whereas irchiver relies on much less to stay timeless which is good. Although I don't think JavaScript will ever get a major update (as in breaking comparability) I believe relying on that is not a perfect way to archive web content. This kind of backwards compatibility breakage is something we have seen before with the deprecation of Adobe Flash and it could theoretically happen elsewhere on the web stack.
SoftTalker 4y ago
I think it's less a technological problem and more just that everyone who used to care about that site or its content no longer does. Or the company behind it has gone out of business -- who is going to maintain a website for a defunct organization, and why would anyone want to? There's no obligation for a person to maintain anything longer than he wants to do it. Putting a blog online is not a lifetime committment. Interests change, or you simply realize nobody much cares about your online musings, and you move on to other interests.
- gowld 4y ago
  But what can you do when you do care, to make your website as durable as a printed book?
  - hypertele-Xii 4y ago
    You could literally print it in a book. With links suffixed by a page number [915].
  - bombcar 4y ago
    Depends on if you want it to survive without you maintaining it. If so, something like a bog-standard Wordpress blog hosted by them might work until they decide they don't want to bother anymore. Otherwise, some setup using S3-as-a-website or GitHub pages may work, but those also depend on the company maintaining that service. If your entire website can be dropped into a ZIP file and served anywhere, you have a greater chance in it surviving, especially if the internet archive got a copy at some point. But if you die and nobody cares about the content, eventually it will disappear.
    - ElectricalUnion 4y ago
      > If your entire website can be dropped into a ZIP file and served anywhere, you have a greater chance in it surviving, especially if the internet archive got a copy at some point. Well, your entire website can be a zip: https://redbean.dev/ https://redbean.dev/
  - 4y ago
chubot 4y ago
Hm interesting, I read over your 7 guidelines [1] and I would say I agree 50%. So the most lightweight way to keep something up and running is to make it trivial to port to many hosting configurations by simplifying the toolchain needed to rehost it -- I agree with this, although I would use the words "standards" and multiple implementations. The linked article doesn't appear to emphasize this. 1. Return to vanilla HTML/CSS Again I feel the relevant issue is "standards", multiple implementations, and the fallback option of "old code" like GNU coreutils (i.e. taking advantage of the Lindy Effect). Not just just HTML/CSS (which certainly meet all of those criteria.) I thought about this when designing https://www.oilshell.org/ https://www.oilshell.org/ and the toolchain - The site started in Markdown but is now standardized on CommonMark [1], which has multiple implementations. So I don't see any reason to stick to HTML/CSS. - The tools are written in Python and shell [2]. Both languages have multiple implementations. (Ironically, Oil is a second implementation of bash! My bash scripts run under both bash and Oil.) - Python and shell rely on Unix, which is standardized (e.g. Linux and FreeBSD have a large common subset that can build my site). This is perhaps unconventional, but I avoided using languages that I don't know how to bootstrap like node.js (has a complex JIT and build process), Go (doesn't use libc) or Rust (complex bootstrap). On the other hand, C has multiple implementations, and Python and shell are written in C, and easy to compile. I'd say Lua also falls in this category, but node.js doesn't. I feel like this is at least 60% of the way to making a website that lasts 30 years. The other 40% is the domain name and hosting. This is a pretty a big site now, you can tar a lot of it up and browse it locally (well it's true you may have to rewrite some self links). Of course, this is my solution as a technical user. If you're trying to solve this problem for the general public, then that's much harder. [1] https://www.oilshell.org/blog/2018/02/14.html https://www.oilshell.org/blog/2018/02/14.html [2] https://www.oilshell.org/site.html https://www.oilshell.org/site.html 2. Don't minimize that HTML Mildly agree if only because it's useful to be able to read HTML to rewrite self links and so forth. Tools don't need it, but it's nice for humans. 3. Prefer one page over several If you're allowed to use Python and shell to automate stuff, then this isn't an issue. I suppose another risk is that people besides me might not understand how to maintain my code. But I don't think it needs to be maintained -- I think it will run forever on the basis of Unix, shell, and Python. Those technologies have "crossed the chasm", where again I think the jury is still out on others. 4. End all forms of hotlinking Yes, my site hosts all its own JS and CSS. 5. Stick with native fonts Yes, custom fonts have a higher chance of being unreadable in the future. I just use the browser's font preference to avoid this issue. It's more future proof and in keeping with the spirit of the web (semantic, not pixel perfect). 6. Obsessively compress your images Agree 7. Eliminate the broken URL risk I think too few people are using commodity shared hosting ... I've been meaning to write some blog posts about that. I use Dreamhost but NearlyFreeSpeech is basically the same idea. It's a Unix box and a web server that somebody else maintains. I absolutely don't care about CPUs, disk drives, even ipv4 vs. ipv6, and I've never had to. The key point is writing to an interface and not an implementation. Commodity shared hosting is a de facto standard. The main difference between a tarball of HTML and a shared hosting site is that say "index.html" is always respected as /, and a few other minor things. So I expect Heroku and similar platforms to come and go, but the de-facto standard of a shared hosting interface will stay forever. It's basically any Unix + any static web server, e.g. I think OpenBSD has a pretty good httpd that's like Apache/Nginx for all the relevant purposes. Github pages also qualifies. So I guess this is adding sort of a programmer's slant to it. To be honest it took me a long time to be fluent enough in Python and shell to make a decent website :) Markdown/CommonMark definitely helps too. I had of course made HTML before this, but it was the odd page here and there. Making a whole site does require some automation, and I agree that lots of common/popular tools and hosting platforms will rot very quickly. (and like you I've seen that happen multiple times in my life!) And I think what your guidelines might be missing is a guideline on how to make "progress". For example CommonMark being standardized is progress that has happened in the last 10 years. You don't want to be tied to the old forever. At some point you have to introduce new technologies, and the way to do that is once there's wide agreement on them and multiple implementations. (just like there's wide agreement on HTML) I think there is / can be progress in dynamic sites too, so you don't have to stick to static!
samsquire 4y ago
One of my ideas is called Wantsfiles at the root of domains. The Wantsfiles contains information about what you want and the price you're willing to pay. An order matching system matches your Wantsfiles.txt to Sellfiles.txt and transacts on your behalf. So I could host link to my tgz on my Wantsfile and offer $5 to host the tarball a month. There we go a perpetually hosted website. You could call it autohosting.
powersnail 4y ago
> system font stack that matches the default font to the operating system of your visitor If the list just matches the default font of the OS, is there a difference between using such a stack vs. not specifying a font? Wouldn't the browser naturally use the OS's default if font-family is unset?
8n4vidtmkvmk 4y ago
disk failure and os upgrades are a non-issue with the right technology stack.... but then you have other problems (more complicated setup)
shadowofneptune 4y ago
Interested personally in the font stack point. The system font stacks listed focus on font styles more than individual typefaces. If I want say, Bookman, is there any harm to long-term use to use a font stack tailored to that font?
- chipotle_coyote 4y ago
  There isn't, no, and there's no particular harm in self-hosted web fonts when it comes to long-term stability. Not only is it relatively unlikely that future browsers will deprecate web fonts but not anything else in current CSS (in which case there's a bigger issue than your fonts), it's even less likely that they'll do so in an entirely incompatible fashion. Assuming you've designed your font stack with decent fallbacks -- which you should -- then it's just like any other font stack, e.g., you may not be presenting in the typeface that you really want to, but nothing's going to break. ("Web fonts are okay, actually" is evidently the windmill I will keep tilting at on Hacker News.)

anonymoushn 4y ago

irchiver seems incredible. I hope there's a comparable product for other OSes one day.

ryanfox 4y ago
I’ve been working on a very similar thing which runs on Windows, Mac, and Linux: https://apse.io https://apse.io

10000truths 4y ago

It’s not uncommon to hear of rack servers with several years of continuous uptime. I wouldn’t be surprised if you could keep a website online for a decade without touching anything by using an LTS distro, enabling unattended upgrades, and running something like nginx.

EddieDante 4y ago

> I've also been working on a background process that automatically captures full-resolution screenshots of every website you visit, creating your own searchable personal web archive: How are screenshots searchable? They aren't plain text. You can't grep them.

anonymoushn 4y ago
The tool captures screenshots in addition to text.
- doodles33 4y ago
  That seems awfully simillar to archive.org wayback machine. I do like to see all these archival projects though, they are certainly worthwhile.
  - lazyjeff 4y ago
    irchiver captures text on the page, and separately OCRs the screenshots (specifically, the screenshot from your viewport). So you can search just what was shown on the page, or what was in the page. Both techniques have pros and cons. While archive.org is fantastic, it can only capture pages that are both 1) publicly accessible (i.e. no social media content) that it happens to crawl, and 2) static content (you're out of luck if the content you want is loaded dynamically, or changes depending on user input).
    - pabs3 4y ago
      IIRC archive.org does save the JS and things it downloads, so you can replay them when you visit the archived site later.
      - doodlesdev 4y ago
        I guess the difference in this case is that the JS on web archive relies on future browsers being backwards compatible, whereas irchiver relies on much less to stay timeless which is good. Although I don't think JavaScript will ever get a major update (as in breaking comparability) I believe relying on that is not a perfect way to archive web content. This kind of backwards compatibility breakage is something we have seen before with the deprecation of Adobe Flash and it could theoretically happen elsewhere on the web stack.

SoftTalker 4y ago

I think it's less a technological problem and more just that everyone who used to care about that site or its content no longer does. Or the company behind it has gone out of business -- who is going to maintain a website for a defunct organization, and why would anyone want to? There's no obligation for a person to maintain anything longer than he wants to do it. Putting a blog online is not a lifetime committment. Interests change, or you simply realize nobody much cares about your online musings, and you move on to other interests.

gowld 4y ago
But what can you do when you do care, to make your website as durable as a printed book?
- hypertele-Xii 4y ago
  You could literally print it in a book. With links suffixed by a page number [915].
- bombcar 4y ago
  Depends on if you want it to survive without you maintaining it. If so, something like a bog-standard Wordpress blog hosted by them might work until they decide they don't want to bother anymore. Otherwise, some setup using S3-as-a-website or GitHub pages may work, but those also depend on the company maintaining that service. If your entire website can be dropped into a ZIP file and served anywhere, you have a greater chance in it surviving, especially if the internet archive got a copy at some point. But if you die and nobody cares about the content, eventually it will disappear.
  - ElectricalUnion 4y ago
    > If your entire website can be dropped into a ZIP file and served anywhere, you have a greater chance in it surviving, especially if the internet archive got a copy at some point. Well, your entire website can be a zip: https://redbean.dev/ https://redbean.dev/
- 4y ago

chubot 4y ago

Hm interesting, I read over your 7 guidelines [1] and I would say I agree 50%. So the most lightweight way to keep something up and running is to make it trivial to port to many hosting configurations by simplifying the toolchain needed to rehost it -- I agree with this, although I would use the words "standards" and multiple implementations. The linked article doesn't appear to emphasize this. 1. Return to vanilla HTML/CSS Again I feel the relevant issue is "standards", multiple implementations, and the fallback option of "old code" like GNU coreutils (i.e. taking advantage of the Lindy Effect). Not just just HTML/CSS (which certainly meet all of those criteria.) I thought about this when designing https://www.oilshell.org/ https://www.oilshell.org/ and the toolchain - The site started in Markdown but is now standardized on CommonMark [1], which has multiple implementations. So I don't see any reason to stick to HTML/CSS. - The tools are written in Python and shell [2]. Both languages have multiple implementations. (Ironically, Oil is a second implementation of bash! My bash scripts run under both bash and Oil.) - Python and shell rely on Unix, which is standardized (e.g. Linux and FreeBSD have a large common subset that can build my site). This is perhaps unconventional, but I avoided using languages that I don't know how to bootstrap like node.js (has a complex JIT and build process), Go (doesn't use libc) or Rust (complex bootstrap). On the other hand, C has multiple implementations, and Python and shell are written in C, and easy to compile. I'd say Lua also falls in this category, but node.js doesn't. I feel like this is at least 60% of the way to making a website that lasts 30 years. The other 40% is the domain name and hosting. This is a pretty a big site now, you can tar a lot of it up and browse it locally (well it's true you may have to rewrite some self links). Of course, this is my solution as a technical user. If you're trying to solve this problem for the general public, then that's much harder. [1] https://www.oilshell.org/blog/2018/02/14.html https://www.oilshell.org/blog/2018/02/14.html [2] https://www.oilshell.org/site.html https://www.oilshell.org/site.html 2. Don't minimize that HTML Mildly agree if only because it's useful to be able to read HTML to rewrite self links and so forth. Tools don't need it, but it's nice for humans. 3. Prefer one page over several If you're allowed to use Python and shell to automate stuff, then this isn't an issue. I suppose another risk is that people besides me might not understand how to maintain my code. But I don't think it needs to be maintained -- I think it will run forever on the basis of Unix, shell, and Python. Those technologies have "crossed the chasm", where again I think the jury is still out on others. 4. End all forms of hotlinking Yes, my site hosts all its own JS and CSS. 5. Stick with native fonts Yes, custom fonts have a higher chance of being unreadable in the future. I just use the browser's font preference to avoid this issue. It's more future proof and in keeping with the spirit of the web (semantic, not pixel perfect). 6. Obsessively compress your images Agree 7. Eliminate the broken URL risk I think too few people are using commodity shared hosting ... I've been meaning to write some blog posts about that. I use Dreamhost but NearlyFreeSpeech is basically the same idea. It's a Unix box and a web server that somebody else maintains. I absolutely don't care about CPUs, disk drives, even ipv4 vs. ipv6, and I've never had to. The key point is writing to an interface and not an implementation. Commodity shared hosting is a de facto standard. The main difference between a tarball of HTML and a shared hosting site is that say "index.html" is always respected as /, and a few other minor things. So I expect Heroku and similar platforms to come and go, but the de-facto standard of a shared hosting interface will stay forever. It's basically any Unix + any static web server, e.g. I think OpenBSD has a pretty good httpd that's like Apache/Nginx for all the relevant purposes. Github pages also qualifies. So I guess this is adding sort of a programmer's slant to it. To be honest it took me a long time to be fluent enough in Python and shell to make a decent website :) Markdown/CommonMark definitely helps too. I had of course made HTML before this, but it was the odd page here and there. Making a whole site does require some automation, and I agree that lots of common/popular tools and hosting platforms will rot very quickly. (and like you I've seen that happen multiple times in my life!) And I think what your guidelines might be missing is a guideline on how to make "progress". For example CommonMark being standardized is progress that has happened in the last 10 years. You don't want to be tied to the old forever. At some point you have to introduce new technologies, and the way to do that is once there's wide agreement on them and multiple implementations. (just like there's wide agreement on HTML) I think there is / can be progress in dynamic sites too, so you don't have to stick to static!

samsquire 4y ago

One of my ideas is called Wantsfiles at the root of domains. The Wantsfiles contains information about what you want and the price you're willing to pay. An order matching system matches your Wantsfiles.txt to Sellfiles.txt and transacts on your behalf. So I could host link to my tgz on my Wantsfile and offer $5 to host the tarball a month. There we go a perpetually hosted website. You could call it autohosting.

powersnail 4y ago

> system font stack that matches the default font to the operating system of your visitor If the list just matches the default font of the OS, is there a difference between using such a stack vs. not specifying a font? Wouldn't the browser naturally use the OS's default if font-family is unset?

8n4vidtmkvmk 4y ago

disk failure and os upgrades are a non-issue with the right technology stack.... but then you have other problems (more complicated setup)

shadowofneptune 4y ago

Interested personally in the font stack point. The system font stacks listed focus on font styles more than individual typefaces. If I want say, Bookman, is there any harm to long-term use to use a font stack tailored to that font?

chipotle_coyote 4y ago
There isn't, no, and there's no particular harm in self-hosted web fonts when it comes to long-term stability. Not only is it relatively unlikely that future browsers will deprecate web fonts but not anything else in current CSS (in which case there's a bigger issue than your fonts), it's even less likely that they'll do so in an entirely incompatible fashion. Assuming you've designed your font stack with decent fallbacks -- which you should -- then it's just like any other font stack, e.g., you may not be presenting in the typeface that you really want to, but nothing's going to break. ("Web fonts are okay, actually" is evidently the windmill I will keep tilting at on Hacker News.)