4 ms·

The crux of SH is 6-pronged: - Deployment and configuration management - Platform maintenance (not app upgrades) - App upgrades - Security - Performance mo

The crux of SH is 6-pronged: - Deployment and configuration management - Platform maintenance (not app upgrades) - App upgrades - Security - Performance monitoring - Data backup and disaster recovery When considering taking on SH, it's important to be wise about all of the implications. Automating these, standardization, and making them more portable are what would make SH cheaper and more doable. Definitely create runbooks and detailed notes that are kept current, organized, and could be used to reconstruct production without automation.

TheNewsIsHere 2y ago
Being able to reconstruct production without either automation or, well, production, is a line I spend time to not cross. I have seen that so much though. It’s unfortunately common to find that the BCDR strategy, if it exists, has a reliance on production even if production is unavailable. So now you have to spend time manually reconstructing whatever dependencies exist in require production to be available. How well is that documented? If you have circular dependencies like that, probably not very well.
- banish-m4 2y ago
  Then it's fraud or corporate BS. Bare-metal recovery means: - a BOM that's shot to a major hardware vendor or already having network and server device spares; Dell is able to meet this sort of need - a USB boot device to kickstart a rebuild, restore environment - some sort of initial restore media or equivalent containing encrypted backups - encrypted offline backup of the infrastructure wiki - n-person split password key material to decrypt backups. DR/BCP also requires thoughtful derisking by inventorying, identifying dependencies, prioritizing restoration (DNS, DHCP, AD, etc.), and removing/untangling cyclic dependencies if they happen up or down the stack. There are semi-automated tools to help with most of this, and infrastructure automation should assist it as much as possible, because if you can't rebuild it with confidence, then you don't have anything. It requires significant thought, effort, standardization, continuous maintenance, and practiced at least yearly by someone(s) like SREs and with regular input of front-line IT-equivalent owners.
  - TheNewsIsHere 2y ago
    You won’t find any argument to the contrary from me. (Although I would nitpick that you could relax some of those points in certain circumstances. For example you may not necessarily need or want to have your OOB backups of your wiki encrypted if you variously don’t have credential-type secrets in it, or have satisfactory enough physical safeguards for your use case. But we could probably argue that kind of nuance back and forth for eternity.) It’s just that it’s fairly common to see such circular dependencies. I’ve seen many sitiations where one team just assumes that another team is looking after these things and that “it’s someone else’s responsibility” or some such. I am certainly not defending that. Plenty of teams see things like backup as something that is somewhere else in the stack. Plenty of executives just assume their technologists are baking these capabilities into product. A lot of junior people haven’t really been given a solid education on these areas and it’s not uncommon to see newer-to-career people making basic mistakes like conflating synchronization and backup, or geo-redundancy with wholesale BCDR.

TheNewsIsHere 2y ago

Being able to reconstruct production without either automation or, well, production, is a line I spend time to not cross. I have seen that so much though. It’s unfortunately common to find that the BCDR strategy, if it exists, has a reliance on production even if production is unavailable. So now you have to spend time manually reconstructing whatever dependencies exist in require production to be available. How well is that documented? If you have circular dependencies like that, probably not very well.

banish-m4 2y ago
Then it's fraud or corporate BS. Bare-metal recovery means: - a BOM that's shot to a major hardware vendor or already having network and server device spares; Dell is able to meet this sort of need - a USB boot device to kickstart a rebuild, restore environment - some sort of initial restore media or equivalent containing encrypted backups - encrypted offline backup of the infrastructure wiki - n-person split password key material to decrypt backups. DR/BCP also requires thoughtful derisking by inventorying, identifying dependencies, prioritizing restoration (DNS, DHCP, AD, etc.), and removing/untangling cyclic dependencies if they happen up or down the stack. There are semi-automated tools to help with most of this, and infrastructure automation should assist it as much as possible, because if you can't rebuild it with confidence, then you don't have anything. It requires significant thought, effort, standardization, continuous maintenance, and practiced at least yearly by someone(s) like SREs and with regular input of front-line IT-equivalent owners.
- TheNewsIsHere 2y ago
  You won’t find any argument to the contrary from me. (Although I would nitpick that you could relax some of those points in certain circumstances. For example you may not necessarily need or want to have your OOB backups of your wiki encrypted if you variously don’t have credential-type secrets in it, or have satisfactory enough physical safeguards for your use case. But we could probably argue that kind of nuance back and forth for eternity.) It’s just that it’s fairly common to see such circular dependencies. I’ve seen many sitiations where one team just assumes that another team is looking after these things and that “it’s someone else’s responsibility” or some such. I am certainly not defending that. Plenty of teams see things like backup as something that is somewhere else in the stack. Plenty of executives just assume their technologists are baking these capabilities into product. A lot of junior people haven’t really been given a solid education on these areas and it’s not uncommon to see newer-to-career people making basic mistakes like conflating synchronization and backup, or geo-redundancy with wholesale BCDR.