9 ms·

I run a small home lab, about 30 services One day I decided to change my main disk and used the opportunity to rebuild everything from scratch and from backups

I run a small home lab, about 30 services One day I decided to change my main disk and used the opportunity to rebuild everything from scratch and from backups. I was up in about an hour. And then I spent a week fixing this and that, ah yes I changed that too and, crap, I cannot remember why this thingie is set up this way. And some more. This is a one-man lab, with simple services, all on docker. I also work in IT. Recovering from scratch a whole infrastructure managed by many people over the years is a titanic task. I helped to recover my nearby hospital as a volunteer when it was ransomwared. The poor two IT guys over there has no idea how to recover and the official help was pityful. I also helped with a ransomware attack on a large company. The effort people had to do to remember why something was that way, or just remember whatever was colossal. Sure a lot of things were "documented" and "tested" but reality hit hard.

WhyNotHugo 1y ago
Modern IT practices don’t really contemplate disaster recovery. Even organisations with strict backup procedures seldom test recovery (most never at all). Everything is quickly strapped together due to teams being understaffed. Preparing infrastructure in a way such that it can easily be recreated is easily twice the effort as “just” setting it up the usual way.
- andrelaszlo 1y ago
  Just the other day one of my clients had a production critical server failing and we started restoring it from backups. Turns out some of the software running on it had some weird licensing checks tied to the hardware so it refused to start on the new server. It turns out that the company that made this important piece of software doesn't even exist anymore.
  - tialaramex 1y ago
    Virtualization really helps. We have a lot of weird software which requires hardware dongles, but they're all USB dongles and they're all virtualized, one of the DC racks has a few U worth of just USB socket -> dongle wired up so that if we spin up a VM it can say "Hey, give me a USB socket with a FooCorp OmniBloat dongle on it" and get one unless they're all used.
    - 2YwaZHXV 1y ago
      would certainly be interested to learn more about this
  - BrandoElFollito 1y ago
    This is a nightmare kind of discovery. I had a similar one, but fortunately, it wasn't as impactful. This is why I like docker, if you keep the sources, you recover no matter what (at least until the "no matter what" holds water)
    - znpy 1y ago
      > This is why I like docker, my understanding is that docker would not have helped in that scenario
      - BrandoElFollito 1y ago
        it really depends on the scenario but if the application was dockerized and they had an image, it would be just starting it again, somewhere else. Possibly with the same network settings if the licensing check was based on that. But of course it can easily go south, though testing the recovery of a container based off an image and mounted volume is simple and quickly shows you if it works or not. But of course it may work today but not tomorrow because the software was not ready for Y2K and according to it we are in the XX century or something and the license is 156 years ... young. Cannot allow this nonsense to proceed, call us at <defunct number> IT is full of joy and happiness
        znpy 1y ago
        > it really depends on the scenario yeah and that scenario was clear: > Turns out some of the software running on it had some weird licensing checks tied to the hardware so it refused to start on the new server.
        BrandoElFollito 1y ago
        "hardware" does not mean "bare metal". It could be a MAC, a serial number or similar things that may be linked to a generic or clonable value in virtualization.
        1y ago
  - 15155 1y ago
    > Turns out some of the software running on it had some weird licensing checks tied to the hardware so it refused to start on the new server. This is around the time when you call that one guy on your team that can reverse engineer and patch out the license check.
    - aspenmayer 1y ago
      Interoperability exception might allow this in exigent circumstances when you do have a valid license, but I wouldn’t do this without running it by the software vendor whose license you are using. In a recovery situation, you’ll probably need to be on the phone a lot, so I can see how you might think it’s quicker to bypass the license check, but that is one person giving some or all of their attention just to that. Disaster recovery isn’t a one person job unless that one person was the whole team anyway, so I think this idea needs to be calibrated somewhat to expectations.
- benterix 1y ago
  > Modern IT practices don’t really contemplate disaster recovery. Even organisations with strict backup procedures seldom test recovery (most never at all). I think this is an outdated view. In modern enterprises DR is often one of the most crucial (and difficult) steps in building the whole infra. You select what is crucial for you, you allocate the budget, you test it, and you plan the date of the next test. However, I'd say it's very rare to do DR of everything. It's terribly expensive and problematic. You need to choose what's really important to you based on defined budgets.
  - rimbo789 1y ago
    Budgets - and lowering them - win every time. I do budgeting and forecasting for SaaS companies and this kind of work is always the first cut
    - edoceo 1y ago
      Is there a recurring theme for why? There is huge risk exposure.
      - supertrope 1y ago
        People round down small risk to zero risk. Meanwhile the cost to run a full DR drill is a certain and immediate cost to their budget.
- 9dev 1y ago
  Actually I think this is hard to properly implement. If you're a small shop, really setting up backups with redundancies, writing the documentation, and testing disaster recovery, that's so much more work than people anticipate, and it has implications on all areas of the business, not just IT. So usually it's hard to justify to management why you would put in all that work and slow down operations—which leads to everyone postponing it. Either that bites you sooner or later, or you're lucky and grow; suddenly, you're a larger organisation, and there are way too many moving parts to start from scratch. So you do a half-hearted attempt of creating a backup strategy held together by duct-tape and hope, that kinda-sorta should work in the worst case, write some LLM-assisted documentation that nobody ever reads, and carry on. You're understaffed and overworked anyway, people are engaging in shadow IT, your actual responsibilities demand attention, so that's the best you can do. And then you've grown even bigger, you're a reputable company now, and then the consultants and auditors and customers with certification requirements come in. So that's when you actually have to put in the work, and it's going to be a long, gruesome, exhausting, and expensive project. Given, of course, that nobody fucks up in the mean time.
  - prmph 1y ago
    Indeed. Setting up infrastructure properly and documenting it properly is even more complex than coding, to me. I can go back to code I wrote months or years ago, and assuming I architectured and documented it idiomatically, I takes me only a bit of time to start being able to reason about it effectively. With infrastructure is it a whole different story. Within weeks of not touching it (which happens if it just works) I start to have trouble retaining a good mental model of it. if I have to dig into it, I'll have to spend a lot of time getting re-acquainted with how it all fits together again.
    - macintux 1y ago
      As much as Cloudformation and Terraform annoy me (thankfully I’ve never been burdened with k8s) there is something magical about having your infrastructure captured in code.
- jon-wood 1y ago
  That's a choice that companies make. I've certainly worked at places which don't test DR, while at my current job we do annual DR runs, where we'll bring up a complete production ready environment from scratch to prove that the backups work, and the runbook for doing a restore actually works.
  - amanaplanacanal 1y ago
    I'm retired now, but the last place I worked estimated it would take months to do a full restore from off site backups, assuming that the data center and hardware were intact. If the data center was destroyed... Longer.
- madaxe_again 1y ago
  If you’re doing it right, the DR process is basically the deployment process, and gets tested every time you do a deployment. We used chef, docker, stored snapshot images, and every deploy basically spun up a new infrastructure from scratch, and once it had passed the automated tests, the load balancers would switch to the new instance. DBs were created from binary snapshots which would then slave off the live DB to catch up (never more than an hour of diff), which also ensured we had a continuously tested DB backup process. The previous instance would get torn down after 8 hours, which was long enough to allow any straggling processes to finish and to have somewhere to roll back to if needed. This all got stored in the cloud, but also locally in our office, and also written onto a DVD-R, all automatically, all verified each time. Our absolute worst case scenario would be less than an hour of downtime, less than an hour of data loss. Similarly our dev environments were a watered down version of the live environment, and so if they were somehow lost, they could be restored in the same manner - and again, frequently tested, as any merge into the preprod branch would trigger a new dev environment to automatically spin up with that codebase. It takes up-front engineering effort to get in place, but it ended up saving our bacon twice, and made our entire pipeline much easier and faster to manage.
- readthenotes1 1y ago
  I used to find it amusing how many people thought Backup was a requirement. "No, Restore is" I would say to stunned faces...
- readlikeasloth 1y ago
  Say what you want about European financial organizations but they are legally obliged to practice their recovery strategies. So every other month production clusters with all user data get teared down in one cloud region and set up in another one over night. This works surprisingly well. I guess they would never do that without the legal requirements.
hu3 1y ago
This is why documenting is so crucial. Even on a software architecture level. A few months from now, I'd love to have written down decisions for my current project: - Why did I decided to use Kysely over Drizzle, Knex, Prisma, TypeORM or other ORM/SQL tool? - How am I going to do migrations? - Why am I using one of Deno/Bun over sticking to nodejs? - Why did I structure the project as a directory per feature over controllers/models/services directories? - Why did I fork this library and what are the steps to keep this thing updated? Do I plan to upstream my changes? Is there a GitHub issue or PR about it? - Why am I hosting in one of AWS/GCP/Azure? Why not lambda functions? Why docker? - Why did I pick this specific distribution of kubernetes over the other also lightweight alternatives? - Why did I even start this project and what do I aim to accomplish with it? So I created a # Decisions section in README.md This way I don't keep doubting my own decisions and wasting time opening 20 documentation tabs to compare solutions yet again.
- dotancohen 1y ago
  Every project I work on has a technical-decisions.org file. Also a daily-notes.org file with every failed experiment, test, install command, etc. The top level headings are dates. Technical decisions used to be in the daily-notes.org file, but keeping in a separate file makes it more accessible to LLMs. I actually started that practice before LLMs were in common use, I struggle to remember why.
  - sjs382 1y ago
    > I struggle to remember why. Should that "why" be in technical-decisions.org or daily-notes.org?
    - dotancohen 1y ago
      It should have been the first entry in some project's technical-decisions.org file!
  - qntmfred 1y ago
    this is why in 2023 i started livestreaming whenever I work at my PC. I also take these kinds of daily and project notes, but it's a bit tedious and can take you out of the flow. so I just let youtube capture everything I'm doing and if I need to go back and remind myself of something (or ask an LLM a question about my livestream history, in the not too distant future) it's all right there.
v3ss0n 1y ago
Thats why Infra as Code is very very important.
- bilekas 1y ago
  Not really, the OP was already using docker, but even with IAC on a small home lab like this you're going to modify one or two small things manually here and there over the years. Sure it can help, but it's just not a one fix solution people thing. If you want a good test of your IAC, just provisioning a brand new environment first time using only your iac.
  - xyzzy_plugh 1y ago
    > you're going to modify one or two small things manually here and there over the years. Huh? This is a strange assumption to make. Is your premise that IAC can't ever be truly reproducible? If you are modifying things manually then you're not doing IAC.
    - v3ss0n 1y ago
      Yeah as soon as you start hand tweaking the system it breaks IAC.
  - v3ss0n 1y ago
    If you hand modify the already IAC system , you are not doing IAC. IAC with CI/CD is what we do. We don't even use AWS Console , we do everything in terraform/opentofu code.
  - wwweston 1y ago
    The first rule of IAC club is that you do not hand modify your infrastructure. The second rule of IAC club is that you do not hand modify your infrastructure.
eptcyka 1y ago
Yeah, I had similar experiences, but now I use nix, which solves these problems.
- bilekas 1y ago
  > but now I use nix, which solves these problems Um, sorry but what do you mean ?
  - eptcyka 1y ago
    Everything is configured via nix, I can swap out the hardware and redeploy everything from 0 with a single command invocation.
    - throwawayffffas 1y ago
      What if something happens to where you keep your configuration?
      - eptcyka 1y ago
        It is in git, I have backups. The secrets are not backed up, but those I can recycle if need be.
- SoftTalker 1y ago
  I don't really know nix, but have used Ansible to try to have all configuration version-controlled and automated. But if there's any possibility of making changes outside of that, you have to be very disciplined. As soon as someone makes a one-off manual change to a crontab or a systemd unit, you're screwed.
  - eptcyka 1y ago
    NixOS just doesn’t let you do that in the nominal case, most of /etc consists of symlinks to a read only partition that is managed by nix - it is actually more difficult to do one-off scripts or config changes via files than it is to do so via nix, at least nominally - there are of course software that has it’s own special config format or that keeps its config in a database - but those get snapshot(ed?) and backed up anyway. Imo, nix is more finnicky but more of a complete solution than ansible.
jl6 1y ago
On the other hand, I’ve worked in places where the total destruction of IT (so as to start again from a clean slate) was within the Overton window of options for how to transform the business.
feynmanalgo 1y ago
> I helped to recover my nearby hospital as a volunteer when it was ransomwared. How did they prevent threat actors presenting themselves as volunteers, were you vetted?
- zulban 1y ago
  A real person showing up is a huge cost and risk. No threat actor will continue an attack on just a hospital like that. The economics make no sense and any money is already extracted. Ransomware shops are very happy to just shotgun the internet from afar. A far bigger risk is accepting incompetent volunteers if anything.
  - feynmanalgo 1y ago
    I didn't say anything about original attacker continuing the attack.
    - stavros 1y ago
      The same answer still applies: That attack vector doesn't have a positive ROI.
- BrandoElFollito 1y ago
  The nature of my work helped to quickly sort that out
slightwinder 1y ago
> Recovering from scratch a whole infrastructure managed by many people over the years is a titanic task. Half of the work is to know what you need, the other half is to know how you do it, while the third half is to cope with all the undocumented tinkering which happened along the way. So in that regard, starting from scratch can be acceptable, as long you are not starting from zero, and can build up on the knowledge and experience of the previous run(s). I mean, there is a whole gaming-genre about this, which is quite popular. And usually you have the benefit that you might be able to fix some fundamental failures which you had to ignore because nobody wanted to take the risk.
- fragmede 1y ago
  wait, what games are in that genre?
  - ac29 1y ago
    I think they are talking about roguelikes/roguelites
froh 1y ago
side remark: I like the ambiguity of titanic (giant) task and Titanic (1912) task :-)
- kayge 1y ago
  Don't forget Titanic (1997)! :D
BLKNSLVR 1y ago
I had to rebuild a significant percentage of my homelab after my house was raided by the police and they took about $10k-worth of my gear; desktop, laptop, NAS, hard drives. However, because in a previous life I'd been responsible for backups and involved in disaster recovery planning I was already kind of prepared with: - a mirrored on site copy of backups (that they either didn't find or chose to leave behind) - older hardware that had once been performing the duties of the existing seized gear (I'm a bit of a hoarder, I like repurposing or keeping for just such an occasion) - multiple off site backups - pretty good documentation of my setup I was back up and running within a day or two and had lost maybe a couple of days of data. And it's a home lab, so nothing super important anyway, but a (not really) nice resilience test. It also gave me the experience to work out a few structural changes to further limit the impact of an event that takes out a bunch of processing and storage. (After 8 months they told me to pick up all my gear, they found nothing, but thanks for traumatising my kids)
- bapak 1y ago
  Why did they raid you?
  - BLKNSLVR 1y ago
    Short version: Possibly the worst thing to be raided for: distribution of CSAM. Apparently based purely on the 'evidence' of my IP address being on some list - that's the only explanation I ever got. Funny thing is, they did so little background research they didn't even know to expect kids in the house when they raided at 6:30am. It still triggers me. This was in August 2022. I wrote pages and pages of my memories and thoughts about it, and it still makes me angry for about ten different reasons. The long version I haven't written yet and probably never will. I don't want to dwell on it, I want to get on with my life and have an even worse drama to deal with at the moment: https://news.ycombinator.com/item?id=44533637 https://news.ycombinator.com/item?id=44533637 I know I'm alive, that's for sure. I'm trying to make lemonade by the goddamn bucket load. P.S. I have written prior HN comments referring to the raid if you care enough to go back that far.
    - 1y ago
QuiCasseRien 1y ago
- i'm scripting at best the full setup of my servers (mostly Nixos and some debian). - daily backup locale + remote (blackbaze with 60 readonly retention strategy, separated bucket by service) - monthly offline backup - a preprod server where my users can restore entiere environment for testing purpose (CI) in case of full house fire, i can be back online in an working day. PS: i have only some TBs of data so quite easy to do.
bluGill 1y ago
In the 1990s mainframes got so stable and redundant that some were not rebooted in over a decade - they could even upgrade the kernel without rebooting. Then one company had a power failure andthe backup generators failed. When the power came back it was months before they figured out everything it was doing and then how to start that service where the guy who started it originally quit years ago. most companies started rebooting the mainframe every six months to ensure they could restart it.
- BrandoElFollito 1y ago
  I was very supportive of the infrastructure IT team when they moved their datacenter. I also had popcorn when watching the switch being figuratively flipped on. It went surprisingly well despite having stayed 15 years in the old DC without rebooting. They were super scared of exactly the case you described but except for some minor issues (and a lot of cussing) it was OK.
  - SoftTalker 1y ago
    The data center where I work self-tests this stuff unintentionally a couple of times a year. The typical case: UPS maintenance, room is put on bypass, load drops when switching back.
    - BrandoElFollito 1y ago
      This is arguably better than not having any tests. This is why I reboot my server from time to time after having applied patches or made more significant changes, despite the fact that "it should not change anything". This is a good moment to realize that it did change something and you have the opportunity to fix the issue while it I sfresh in your mind, and possibly with more time.
- stavros 1y ago
  That's why I delete all my company's data stores every quarter too!
freddealmeida 1y ago
Chamath's new company 80/90 is targeting this pain. Large firms often have no idea what their software is trying to do. Rebuilding it is cheaper and leads to better software.
tmiku 1y ago
> I helped to recover my nearby hospital as a volunteer when it was ransomwared. I'm curious about how you got in the door here. Very cool, but isn't healthcare IT notoriously cagey about access? I've had to do PHI training and background checks before getting into the system at my (admittedly only 2) PHI-centered jobs. Granted, if it was such an emergency, I could see them rushing you through a lite version of the HR onboarding process. Did you have a connection in the hospital through whom you offered your services?
- BrandoElFollito 1y ago
  The nature and place of my work helped to quickly clear this. I volunteered to help because I knew that even broadly planning the recovery, evidence preservation etc. would be completely beyond the capabilities of the two IT folks (they were extremely nice and helpful, and glad that there was someone to help). I was there to draw things on the board and ask the questions that will help to recover. I would not have (nor want, not have the need) to access patient information. This is something I warned them about early in the process, as the chaos was growing. You need to imagine a large hospital completely blocked, with patients during an operation being stabilized and driven away. I am used to crisis situations and having someone who will anticipate things you do not think about (how to communicate, how to reach prople having planned procedures, who does what and who talks with whom) is a useful person to have before the authorities kick in. My wife had a planned operation that morning and I was on site when the ransomware hit, it is just this. Nothing James Bond like, just sheer luck to have been around. The hospital made a recovery but it took about a year IIRC
  - tmiku 1y ago
    That's really cool. I was mostly envisioning hands-on admin stuff (because that's the work I'm most familiar with), but I hadn't thought about how much of a boon it would be to have someone with incident management experience arriving to help out. If you ever do a write-up about your experience, I'd love to read it.
    - BrandoElFollito 1y ago
deleted 1y ago
[deleted]
groby_b 1y ago
If you are professionally responsible for infrastructure, and you haven't thought very hard about the "how do we rebuild from scratch" case, you're committing professional malpractice. The problem is environments like hospitals, who'll "cost center" their IT department to death, where even the most seasoned pro has no chance to ever do the right thing. There should be liability at the board level. There never will.
amy214 1y ago
>This is a one-man lab, with simple services, all on docker. I also work in IT. TBH your mistake was only running one layer of virtualization. What I do on my home setup is run a docker in a VM in a VM in a docker in a docker in a VM in a docker in a VM in a VM in a docker. This, I feel, ought to be the minimum level of indirection and virtualization in any technical configuration in perpetuity. Anything less is bush league and prone to errors.
martin82 1y ago
I would hope that military facilities follow better standards and are capable of recovering more quickly than a hobbyist...
2OEH8eoCRo0 1y ago
30 services is a small home lab?
- BrandoElFollito 1y ago
  Out of these 30 services, ~8 are useful on a daily basis, some from time to time,e and the rest is either forgotten or tests. What I meant I that I do not have the kind of setup a lot of people show: a DC-grade self, with a U2 switch etc. I have an old server, my ISP box, a UPS, a switch and maybe something else I forgot. But since the server runs both Home Assistant and Pi-hole, it is critically important. OTOH this criticality allows me not to invest too much in monitoring: I have family and friends yelling immediately when something is down :)
rkagerer 1y ago
I've worked in tech all my life, and long ago learned how important it is to be impeccably diligent in documenting build processes whenever creating, deploying or adjusting new architecture. Now it's simply become part of my engineering hygiene - as natural and effortless as brushing my teeth. Actually drilling your DR is also crucial. If you never put it to the test, your documentation isn't worth the paper it's printed on. In fact the last few years I've been thinking about ways for these systems to rebuild themselves on a continuous basis. Eg. I'd love a smartphone that competely restores itself from backup every night, even to brand new identical hardware, including secure element artifacts (either via private keys I securely control or reregistering everything in an automated fashion), with no user-noticeable impact.

WhyNotHugo 1y ago

Modern IT practices don’t really contemplate disaster recovery. Even organisations with strict backup procedures seldom test recovery (most never at all). Everything is quickly strapped together due to teams being understaffed. Preparing infrastructure in a way such that it can easily be recreated is easily twice the effort as “just” setting it up the usual way.

andrelaszlo 1y ago
Just the other day one of my clients had a production critical server failing and we started restoring it from backups. Turns out some of the software running on it had some weird licensing checks tied to the hardware so it refused to start on the new server. It turns out that the company that made this important piece of software doesn't even exist anymore.
- tialaramex 1y ago
  Virtualization really helps. We have a lot of weird software which requires hardware dongles, but they're all USB dongles and they're all virtualized, one of the DC racks has a few U worth of just USB socket -> dongle wired up so that if we spin up a VM it can say "Hey, give me a USB socket with a FooCorp OmniBloat dongle on it" and get one unless they're all used.
  - 2YwaZHXV 1y ago
    would certainly be interested to learn more about this
- BrandoElFollito 1y ago
  This is a nightmare kind of discovery. I had a similar one, but fortunately, it wasn't as impactful. This is why I like docker, if you keep the sources, you recover no matter what (at least until the "no matter what" holds water)
  - znpy 1y ago
    > This is why I like docker, my understanding is that docker would not have helped in that scenario
    - BrandoElFollito 1y ago
      it really depends on the scenario but if the application was dockerized and they had an image, it would be just starting it again, somewhere else. Possibly with the same network settings if the licensing check was based on that. But of course it can easily go south, though testing the recovery of a container based off an image and mounted volume is simple and quickly shows you if it works or not. But of course it may work today but not tomorrow because the software was not ready for Y2K and according to it we are in the XX century or something and the license is 156 years ... young. Cannot allow this nonsense to proceed, call us at <defunct number> IT is full of joy and happiness
      - znpy 1y ago
        > it really depends on the scenario yeah and that scenario was clear: > Turns out some of the software running on it had some weird licensing checks tied to the hardware so it refused to start on the new server.
        BrandoElFollito 1y ago
        "hardware" does not mean "bare metal". It could be a MAC, a serial number or similar things that may be linked to a generic or clonable value in virtualization.
        1y ago
- 15155 1y ago
  > Turns out some of the software running on it had some weird licensing checks tied to the hardware so it refused to start on the new server. This is around the time when you call that one guy on your team that can reverse engineer and patch out the license check.
  - aspenmayer 1y ago
    Interoperability exception might allow this in exigent circumstances when you do have a valid license, but I wouldn’t do this without running it by the software vendor whose license you are using. In a recovery situation, you’ll probably need to be on the phone a lot, so I can see how you might think it’s quicker to bypass the license check, but that is one person giving some or all of their attention just to that. Disaster recovery isn’t a one person job unless that one person was the whole team anyway, so I think this idea needs to be calibrated somewhat to expectations.
benterix 1y ago
> Modern IT practices don’t really contemplate disaster recovery. Even organisations with strict backup procedures seldom test recovery (most never at all). I think this is an outdated view. In modern enterprises DR is often one of the most crucial (and difficult) steps in building the whole infra. You select what is crucial for you, you allocate the budget, you test it, and you plan the date of the next test. However, I'd say it's very rare to do DR of everything. It's terribly expensive and problematic. You need to choose what's really important to you based on defined budgets.
- rimbo789 1y ago
  Budgets - and lowering them - win every time. I do budgeting and forecasting for SaaS companies and this kind of work is always the first cut
  - edoceo 1y ago
    Is there a recurring theme for why? There is huge risk exposure.
    - supertrope 1y ago
      People round down small risk to zero risk. Meanwhile the cost to run a full DR drill is a certain and immediate cost to their budget.
9dev 1y ago
Actually I think this is hard to properly implement. If you're a small shop, really setting up backups with redundancies, writing the documentation, and testing disaster recovery, that's so much more work than people anticipate, and it has implications on all areas of the business, not just IT. So usually it's hard to justify to management why you would put in all that work and slow down operations—which leads to everyone postponing it. Either that bites you sooner or later, or you're lucky and grow; suddenly, you're a larger organisation, and there are way too many moving parts to start from scratch. So you do a half-hearted attempt of creating a backup strategy held together by duct-tape and hope, that kinda-sorta should work in the worst case, write some LLM-assisted documentation that nobody ever reads, and carry on. You're understaffed and overworked anyway, people are engaging in shadow IT, your actual responsibilities demand attention, so that's the best you can do. And then you've grown even bigger, you're a reputable company now, and then the consultants and auditors and customers with certification requirements come in. So that's when you actually have to put in the work, and it's going to be a long, gruesome, exhausting, and expensive project. Given, of course, that nobody fucks up in the mean time.
- prmph 1y ago
  Indeed. Setting up infrastructure properly and documenting it properly is even more complex than coding, to me. I can go back to code I wrote months or years ago, and assuming I architectured and documented it idiomatically, I takes me only a bit of time to start being able to reason about it effectively. With infrastructure is it a whole different story. Within weeks of not touching it (which happens if it just works) I start to have trouble retaining a good mental model of it. if I have to dig into it, I'll have to spend a lot of time getting re-acquainted with how it all fits together again.
  - macintux 1y ago
    As much as Cloudformation and Terraform annoy me (thankfully I’ve never been burdened with k8s) there is something magical about having your infrastructure captured in code.
jon-wood 1y ago
That's a choice that companies make. I've certainly worked at places which don't test DR, while at my current job we do annual DR runs, where we'll bring up a complete production ready environment from scratch to prove that the backups work, and the runbook for doing a restore actually works.
- amanaplanacanal 1y ago
  I'm retired now, but the last place I worked estimated it would take months to do a full restore from off site backups, assuming that the data center and hardware were intact. If the data center was destroyed... Longer.
madaxe_again 1y ago
If you’re doing it right, the DR process is basically the deployment process, and gets tested every time you do a deployment. We used chef, docker, stored snapshot images, and every deploy basically spun up a new infrastructure from scratch, and once it had passed the automated tests, the load balancers would switch to the new instance. DBs were created from binary snapshots which would then slave off the live DB to catch up (never more than an hour of diff), which also ensured we had a continuously tested DB backup process. The previous instance would get torn down after 8 hours, which was long enough to allow any straggling processes to finish and to have somewhere to roll back to if needed. This all got stored in the cloud, but also locally in our office, and also written onto a DVD-R, all automatically, all verified each time. Our absolute worst case scenario would be less than an hour of downtime, less than an hour of data loss. Similarly our dev environments were a watered down version of the live environment, and so if they were somehow lost, they could be restored in the same manner - and again, frequently tested, as any merge into the preprod branch would trigger a new dev environment to automatically spin up with that codebase. It takes up-front engineering effort to get in place, but it ended up saving our bacon twice, and made our entire pipeline much easier and faster to manage.
readthenotes1 1y ago
I used to find it amusing how many people thought Backup was a requirement. "No, Restore is" I would say to stunned faces...
readlikeasloth 1y ago
Say what you want about European financial organizations but they are legally obliged to practice their recovery strategies. So every other month production clusters with all user data get teared down in one cloud region and set up in another one over night. This works surprisingly well. I guess they would never do that without the legal requirements.

hu3 1y ago

This is why documenting is so crucial. Even on a software architecture level. A few months from now, I'd love to have written down decisions for my current project: - Why did I decided to use Kysely over Drizzle, Knex, Prisma, TypeORM or other ORM/SQL tool? - How am I going to do migrations? - Why am I using one of Deno/Bun over sticking to nodejs? - Why did I structure the project as a directory per feature over controllers/models/services directories? - Why did I fork this library and what are the steps to keep this thing updated? Do I plan to upstream my changes? Is there a GitHub issue or PR about it? - Why am I hosting in one of AWS/GCP/Azure? Why not lambda functions? Why docker? - Why did I pick this specific distribution of kubernetes over the other also lightweight alternatives? - Why did I even start this project and what do I aim to accomplish with it? So I created a # Decisions section in README.md This way I don't keep doubting my own decisions and wasting time opening 20 documentation tabs to compare solutions yet again.

dotancohen 1y ago
Every project I work on has a technical-decisions.org file. Also a daily-notes.org file with every failed experiment, test, install command, etc. The top level headings are dates. Technical decisions used to be in the daily-notes.org file, but keeping in a separate file makes it more accessible to LLMs. I actually started that practice before LLMs were in common use, I struggle to remember why.
- sjs382 1y ago
  > I struggle to remember why. Should that "why" be in technical-decisions.org or daily-notes.org?
  - dotancohen 1y ago
    It should have been the first entry in some project's technical-decisions.org file!
- qntmfred 1y ago
  this is why in 2023 i started livestreaming whenever I work at my PC. I also take these kinds of daily and project notes, but it's a bit tedious and can take you out of the flow. so I just let youtube capture everything I'm doing and if I need to go back and remind myself of something (or ask an LLM a question about my livestream history, in the not too distant future) it's all right there.

v3ss0n 1y ago

Thats why Infra as Code is very very important.

bilekas 1y ago
Not really, the OP was already using docker, but even with IAC on a small home lab like this you're going to modify one or two small things manually here and there over the years. Sure it can help, but it's just not a one fix solution people thing. If you want a good test of your IAC, just provisioning a brand new environment first time using only your iac.
- xyzzy_plugh 1y ago
  > you're going to modify one or two small things manually here and there over the years. Huh? This is a strange assumption to make. Is your premise that IAC can't ever be truly reproducible? If you are modifying things manually then you're not doing IAC.
  - v3ss0n 1y ago
    Yeah as soon as you start hand tweaking the system it breaks IAC.
- v3ss0n 1y ago
  If you hand modify the already IAC system , you are not doing IAC. IAC with CI/CD is what we do. We don't even use AWS Console , we do everything in terraform/opentofu code.
- wwweston 1y ago
  The first rule of IAC club is that you do not hand modify your infrastructure. The second rule of IAC club is that you do not hand modify your infrastructure.

eptcyka 1y ago

Yeah, I had similar experiences, but now I use nix, which solves these problems.

bilekas 1y ago
> but now I use nix, which solves these problems Um, sorry but what do you mean ?
- eptcyka 1y ago
  Everything is configured via nix, I can swap out the hardware and redeploy everything from 0 with a single command invocation.
  - throwawayffffas 1y ago
    What if something happens to where you keep your configuration?
    - eptcyka 1y ago
      It is in git, I have backups. The secrets are not backed up, but those I can recycle if need be.
SoftTalker 1y ago
I don't really know nix, but have used Ansible to try to have all configuration version-controlled and automated. But if there's any possibility of making changes outside of that, you have to be very disciplined. As soon as someone makes a one-off manual change to a crontab or a systemd unit, you're screwed.
- eptcyka 1y ago
  NixOS just doesn’t let you do that in the nominal case, most of /etc consists of symlinks to a read only partition that is managed by nix - it is actually more difficult to do one-off scripts or config changes via files than it is to do so via nix, at least nominally - there are of course software that has it’s own special config format or that keeps its config in a database - but those get snapshot(ed?) and backed up anyway. Imo, nix is more finnicky but more of a complete solution than ansible.

jl6 1y ago

On the other hand, I’ve worked in places where the total destruction of IT (so as to start again from a clean slate) was within the Overton window of options for how to transform the business.

feynmanalgo 1y ago

> I helped to recover my nearby hospital as a volunteer when it was ransomwared. How did they prevent threat actors presenting themselves as volunteers, were you vetted?

zulban 1y ago
A real person showing up is a huge cost and risk. No threat actor will continue an attack on just a hospital like that. The economics make no sense and any money is already extracted. Ransomware shops are very happy to just shotgun the internet from afar. A far bigger risk is accepting incompetent volunteers if anything.
- feynmanalgo 1y ago
  I didn't say anything about original attacker continuing the attack.
  - stavros 1y ago
    The same answer still applies: That attack vector doesn't have a positive ROI.
BrandoElFollito 1y ago
The nature of my work helped to quickly sort that out

slightwinder 1y ago

> Recovering from scratch a whole infrastructure managed by many people over the years is a titanic task. Half of the work is to know what you need, the other half is to know how you do it, while the third half is to cope with all the undocumented tinkering which happened along the way. So in that regard, starting from scratch can be acceptable, as long you are not starting from zero, and can build up on the knowledge and experience of the previous run(s). I mean, there is a whole gaming-genre about this, which is quite popular. And usually you have the benefit that you might be able to fix some fundamental failures which you had to ignore because nobody wanted to take the risk.

fragmede 1y ago
wait, what games are in that genre?
- ac29 1y ago
  I think they are talking about roguelikes/roguelites

froh 1y ago

side remark: I like the ambiguity of titanic (giant) task and Titanic (1912) task :-)

kayge 1y ago
Don't forget Titanic (1997)! :D

BLKNSLVR 1y ago

I had to rebuild a significant percentage of my homelab after my house was raided by the police and they took about $10k-worth of my gear; desktop, laptop, NAS, hard drives. However, because in a previous life I'd been responsible for backups and involved in disaster recovery planning I was already kind of prepared with: - a mirrored on site copy of backups (that they either didn't find or chose to leave behind) - older hardware that had once been performing the duties of the existing seized gear (I'm a bit of a hoarder, I like repurposing or keeping for just such an occasion) - multiple off site backups - pretty good documentation of my setup I was back up and running within a day or two and had lost maybe a couple of days of data. And it's a home lab, so nothing super important anyway, but a (not really) nice resilience test. It also gave me the experience to work out a few structural changes to further limit the impact of an event that takes out a bunch of processing and storage. (After 8 months they told me to pick up all my gear, they found nothing, but thanks for traumatising my kids)

bapak 1y ago
Why did they raid you?
- BLKNSLVR 1y ago
  Short version: Possibly the worst thing to be raided for: distribution of CSAM. Apparently based purely on the 'evidence' of my IP address being on some list - that's the only explanation I ever got. Funny thing is, they did so little background research they didn't even know to expect kids in the house when they raided at 6:30am. It still triggers me. This was in August 2022. I wrote pages and pages of my memories and thoughts about it, and it still makes me angry for about ten different reasons. The long version I haven't written yet and probably never will. I don't want to dwell on it, I want to get on with my life and have an even worse drama to deal with at the moment: https://news.ycombinator.com/item?id=44533637 https://news.ycombinator.com/item?id=44533637 I know I'm alive, that's for sure. I'm trying to make lemonade by the goddamn bucket load. P.S. I have written prior HN comments referring to the raid if you care enough to go back that far.
  - 1y ago

QuiCasseRien 1y ago

- i'm scripting at best the full setup of my servers (mostly Nixos and some debian). - daily backup locale + remote (blackbaze with 60 readonly retention strategy, separated bucket by service) - monthly offline backup - a preprod server where my users can restore entiere environment for testing purpose (CI) in case of full house fire, i can be back online in an working day. PS: i have only some TBs of data so quite easy to do.

bluGill 1y ago

In the 1990s mainframes got so stable and redundant that some were not rebooted in over a decade - they could even upgrade the kernel without rebooting. Then one company had a power failure andthe backup generators failed. When the power came back it was months before they figured out everything it was doing and then how to start that service where the guy who started it originally quit years ago. most companies started rebooting the mainframe every six months to ensure they could restart it.

BrandoElFollito 1y ago
I was very supportive of the infrastructure IT team when they moved their datacenter. I also had popcorn when watching the switch being figuratively flipped on. It went surprisingly well despite having stayed 15 years in the old DC without rebooting. They were super scared of exactly the case you described but except for some minor issues (and a lot of cussing) it was OK.
- SoftTalker 1y ago
  The data center where I work self-tests this stuff unintentionally a couple of times a year. The typical case: UPS maintenance, room is put on bypass, load drops when switching back.
  - BrandoElFollito 1y ago
    This is arguably better than not having any tests. This is why I reboot my server from time to time after having applied patches or made more significant changes, despite the fact that "it should not change anything". This is a good moment to realize that it did change something and you have the opportunity to fix the issue while it I sfresh in your mind, and possibly with more time.
stavros 1y ago
That's why I delete all my company's data stores every quarter too!

freddealmeida 1y ago

Chamath's new company 80/90 is targeting this pain. Large firms often have no idea what their software is trying to do. Rebuilding it is cheaper and leads to better software.

tmiku 1y ago

> I helped to recover my nearby hospital as a volunteer when it was ransomwared. I'm curious about how you got in the door here. Very cool, but isn't healthcare IT notoriously cagey about access? I've had to do PHI training and background checks before getting into the system at my (admittedly only 2) PHI-centered jobs. Granted, if it was such an emergency, I could see them rushing you through a lite version of the HR onboarding process. Did you have a connection in the hospital through whom you offered your services?

BrandoElFollito 1y ago
The nature and place of my work helped to quickly clear this. I volunteered to help because I knew that even broadly planning the recovery, evidence preservation etc. would be completely beyond the capabilities of the two IT folks (they were extremely nice and helpful, and glad that there was someone to help). I was there to draw things on the board and ask the questions that will help to recover. I would not have (nor want, not have the need) to access patient information. This is something I warned them about early in the process, as the chaos was growing. You need to imagine a large hospital completely blocked, with patients during an operation being stabilized and driven away. I am used to crisis situations and having someone who will anticipate things you do not think about (how to communicate, how to reach prople having planned procedures, who does what and who talks with whom) is a useful person to have before the authorities kick in. My wife had a planned operation that morning and I was on site when the ransomware hit, it is just this. Nothing James Bond like, just sheer luck to have been around. The hospital made a recovery but it took about a year IIRC
- tmiku 1y ago
  That's really cool. I was mostly envisioning hands-on admin stuff (because that's the work I'm most familiar with), but I hadn't thought about how much of a boon it would be to have someone with incident management experience arriving to help out. If you ever do a write-up about your experience, I'd love to read it.
  - BrandoElFollito 1y ago

deleted 1y ago

[deleted]

groby_b 1y ago

If you are professionally responsible for infrastructure, and you haven't thought very hard about the "how do we rebuild from scratch" case, you're committing professional malpractice. The problem is environments like hospitals, who'll "cost center" their IT department to death, where even the most seasoned pro has no chance to ever do the right thing. There should be liability at the board level. There never will.

amy214 1y ago

>This is a one-man lab, with simple services, all on docker. I also work in IT. TBH your mistake was only running one layer of virtualization. What I do on my home setup is run a docker in a VM in a VM in a docker in a docker in a VM in a docker in a VM in a VM in a docker. This, I feel, ought to be the minimum level of indirection and virtualization in any technical configuration in perpetuity. Anything less is bush league and prone to errors.

martin82 1y ago

I would hope that military facilities follow better standards and are capable of recovering more quickly than a hobbyist...

2OEH8eoCRo0 1y ago

30 services is a small home lab?

BrandoElFollito 1y ago
Out of these 30 services, ~8 are useful on a daily basis, some from time to time,e and the rest is either forgotten or tests. What I meant I that I do not have the kind of setup a lot of people show: a DC-grade self, with a U2 switch etc. I have an old server, my ISP box, a UPS, a switch and maybe something else I forgot. But since the server runs both Home Assistant and Pi-hole, it is critically important. OTOH this criticality allows me not to invest too much in monitoring: I have family and friends yelling immediately when something is down :)

rkagerer 1y ago

I've worked in tech all my life, and long ago learned how important it is to be impeccably diligent in documenting build processes whenever creating, deploying or adjusting new architecture. Now it's simply become part of my engineering hygiene - as natural and effortless as brushing my teeth. Actually drilling your DR is also crucial. If you never put it to the test, your documentation isn't worth the paper it's printed on. In fact the last few years I've been thinking about ways for these systems to rebuild themselves on a continuous basis. Eg. I'd love a smartphone that competely restores itself from backup every night, even to brand new identical hardware, including secure element artifacts (either via private keys I securely control or reregistering everything in an automated fashion), with no user-noticeable impact.