6 ms·

The biggest mistake I made was high uptime. arjie.com was up for 10 years plus on a Hetzner VPS so that by the time they wanted to sunset the machine underlying

by arjie 6mo ago

The biggest mistake I made was high uptime. arjie.com was up for 10 years plus on a Hetzner VPS so that by the time they wanted to sunset the machine underlying I had no idea what my teenage self had set up. I have the backups but the site hasn’t been up in a decade… Nowadays I build things so that they move and I have moved things about a bit so I know they work.

deleted 6mo ago
[deleted]
nine_k 6mo ago
Indeed, for a VM, high uptime makes little sense, because a reboot takes a few seconds, and an upgrade requires no downtime, just switching the DNS to a new instance. For a physical machine which you can't easily copy, it's a different story.
- mx7zysuj4xew 5mo ago
  We're talking what, 15 minutes to reach post?
bfivyvysj 6mo ago
I started putting things in a big ansible playbook repo. Don't need to have it fully managed by ansible either I mostly just have setup configured there I still do lots of by hand management.
- arjie 6mo ago
  I have the same. The infra management is in one place, the apps hold their own, and there’s a docs folder on the server where each guy puts his stuff. The install is idempotent deploy scripts. But back then my stuff was more ramshackle.
gerdesj 6mo ago
"The biggest mistake I made was high uptime" Quite. I'm old enough to remember machine uptime being a badge of honour. However, being older and not really wiser, I look for service uptime these days. Yes we did have similar back in the day, that's why MX and the like DNS records exist. Old school clusters were pretty esoteric but the lessons were learned (split brain n that) and that's why we still argue the toss with kiddies about why a Proxmox cluster with two nodes is fucked and why we recommend an additional "witness". I don't care that VMware glossed over the whole two node HA cluster thing years ago with a massive bodge. They were wrong then and they are probably still wrong because that nonsense is probably still baked in. Sorry, slight digression. High uptime implies no patching. We all love patching.
- andai 6mo ago
  https://en.wikipedia.org/wiki/Split-brain_(computing) https://en.wikipedia.org/wiki/Split-brain_(computing) The more you know! >a Proxmox cluster with two nodes is fucked and why we recommend an additional "witness". Reminds me of the three Magi from Evangelion: https://magi.kinta.ma/ https://magi.kinta.ma/
  - red-iron-pine 5mo ago
    "a man with two watches can never be sure as to the time" need a third one to confirm which of the 2 is accurate
- pjmlp 5mo ago
  There is something like live patching. One reason mainframes and micros are still around us, is that you can change almost everything between hardware and software without downtime. It is also available in commercial surviving UNIXes, and as paid for feature in some Linux distros, although not to the extent that those grandparent systems are capable of.
  - Scramblejams 5mo ago
    I’ve long wanted that amazing uptime and virtualization and huge I/O and all that cool stuff mainframes offered, but on the desktop or in the closet, with modern CPUs. I think I’m gonna hafta keep waiting...
niel 5mo ago
This reminds me of Ise Shrine in Japan, which is completely dismantled then rebuilt every 20 years. This is top of mind because I recently read Breakneck by Dan Wang. He makes the case that this practice of rebuilding the shrine preserves knowledge that would otherwise have been lost to time. Wang contrasts Ise Shrine with Notre Dame, where rebuilding the roof is apparently quite difficult, perhaps in part due to the loss of knowledge. I'm not familiar enough with either structure to judge whether this is a fair comparison, but I like the principle. (Edit to add: This is only a minor analogy from the book, which I highly recommend overall.)
- arjie 5mo ago
  Thank you for the recommendation! I love that reference, and particularly because I am fond of the story of the shrine for a different reason https://wiki.roshangeorge.dev/w/Constancy_Preference#Conceptual_Constancy https://wiki.roshangeorge.dev/w/Constancy_Preference#Concept...
culi 5mo ago
Sometimes I leave Architectural Decision Records for personal projects. It feels silly but it honestly comes in handy more times than expected
- gofreddygo 5mo ago
  I keep them embedded in the codebase or an artifact right next to the source. And the key thing is that i dont need too many details at all. A few cues and its all back in my head.
walletdrainer 5mo ago
> The biggest mistake I made was high uptime. arjie.com was up for 10 years plus on a Hetzner VPS so that by the time they wanted to sunset the machine underlying I had no idea what my teenage self had set up. I have the backups but the site hasn’t been up in a decade LLMs have solved this problem, they’ll happily deal with the software archaeology on your behalf. This is the kind of task they really excel at.
- arjie 5mo ago
  You're right, of course. At this point it's inertia. It's been dead a decade.
bradley13 5mo ago
I hear you. On the other hand, not having to mess with something is good. I just make extensive notes in a README somewhere - usually in KeePass right next to the system info.
unethical_ban 5mo ago
I stood up a dokuwiki instance recently and then documented how to stand up dokuwiki, haha. I disabled revision history viewing and have a public portion and a private portion. I use it to track things I'm learning and document rollout procedures and commands I need for things. So far I have rclone backups into S3 Glacier, Tuwunel(Matrix) server deployment with voice/video support, and various little tutorials on server stuff I'm learning. TLDR use a wiki!

by arjie 6mo ago

deleted 6mo ago

[deleted]

nine_k 6mo ago

Indeed, for a VM, high uptime makes little sense, because a reboot takes a few seconds, and an upgrade requires no downtime, just switching the DNS to a new instance. For a physical machine which you can't easily copy, it's a different story.

mx7zysuj4xew 5mo ago
We're talking what, 15 minutes to reach post?

bfivyvysj 6mo ago

I started putting things in a big ansible playbook repo. Don't need to have it fully managed by ansible either I mostly just have setup configured there I still do lots of by hand management.

arjie 6mo ago
I have the same. The infra management is in one place, the apps hold their own, and there’s a docs folder on the server where each guy puts his stuff. The install is idempotent deploy scripts. But back then my stuff was more ramshackle.

gerdesj 6mo ago

"The biggest mistake I made was high uptime" Quite. I'm old enough to remember machine uptime being a badge of honour. However, being older and not really wiser, I look for service uptime these days. Yes we did have similar back in the day, that's why MX and the like DNS records exist. Old school clusters were pretty esoteric but the lessons were learned (split brain n that) and that's why we still argue the toss with kiddies about why a Proxmox cluster with two nodes is fucked and why we recommend an additional "witness". I don't care that VMware glossed over the whole two node HA cluster thing years ago with a massive bodge. They were wrong then and they are probably still wrong because that nonsense is probably still baked in. Sorry, slight digression. High uptime implies no patching. We all love patching.

andai 6mo ago
https://en.wikipedia.org/wiki/Split-brain_(computing) https://en.wikipedia.org/wiki/Split-brain_(computing) The more you know! >a Proxmox cluster with two nodes is fucked and why we recommend an additional "witness". Reminds me of the three Magi from Evangelion: https://magi.kinta.ma/ https://magi.kinta.ma/
- red-iron-pine 5mo ago
  "a man with two watches can never be sure as to the time" need a third one to confirm which of the 2 is accurate
pjmlp 5mo ago
There is something like live patching. One reason mainframes and micros are still around us, is that you can change almost everything between hardware and software without downtime. It is also available in commercial surviving UNIXes, and as paid for feature in some Linux distros, although not to the extent that those grandparent systems are capable of.
- Scramblejams 5mo ago
  I’ve long wanted that amazing uptime and virtualization and huge I/O and all that cool stuff mainframes offered, but on the desktop or in the closet, with modern CPUs. I think I’m gonna hafta keep waiting...

niel 5mo ago

This reminds me of Ise Shrine in Japan, which is completely dismantled then rebuilt every 20 years. This is top of mind because I recently read Breakneck by Dan Wang. He makes the case that this practice of rebuilding the shrine preserves knowledge that would otherwise have been lost to time. Wang contrasts Ise Shrine with Notre Dame, where rebuilding the roof is apparently quite difficult, perhaps in part due to the loss of knowledge. I'm not familiar enough with either structure to judge whether this is a fair comparison, but I like the principle. (Edit to add: This is only a minor analogy from the book, which I highly recommend overall.)

arjie 5mo ago
Thank you for the recommendation! I love that reference, and particularly because I am fond of the story of the shrine for a different reason https://wiki.roshangeorge.dev/w/Constancy_Preference#Conceptual_Constancy https://wiki.roshangeorge.dev/w/Constancy_Preference#Concept...

culi 5mo ago

Sometimes I leave Architectural Decision Records for personal projects. It feels silly but it honestly comes in handy more times than expected

gofreddygo 5mo ago
I keep them embedded in the codebase or an artifact right next to the source. And the key thing is that i dont need too many details at all. A few cues and its all back in my head.

walletdrainer 5mo ago

> The biggest mistake I made was high uptime. arjie.com was up for 10 years plus on a Hetzner VPS so that by the time they wanted to sunset the machine underlying I had no idea what my teenage self had set up. I have the backups but the site hasn’t been up in a decade LLMs have solved this problem, they’ll happily deal with the software archaeology on your behalf. This is the kind of task they really excel at.

arjie 5mo ago
You're right, of course. At this point it's inertia. It's been dead a decade.

bradley13 5mo ago

I hear you. On the other hand, not having to mess with something is good. I just make extensive notes in a README somewhere - usually in KeePass right next to the system info.

unethical_ban 5mo ago

I stood up a dokuwiki instance recently and then documented how to stand up dokuwiki, haha. I disabled revision history viewing and have a public portion and a private portion. I use it to track things I'm learning and document rollout procedures and commands I need for things. So far I have rclone backups into S3 Glacier, Tuwunel(Matrix) server deployment with voice/video support, and various little tutorials on server stuff I'm learning. TLDR use a wiki!