5 ms·
We are seriously looking at strategies for clean room rebuild of our IT infrastructure, potentially on a recurring basis via automation. Obviously, you cant mi
by bob1029 6y ago
We are seriously looking at strategies for clean room rebuild of our IT infrastructure, potentially on a recurring basis via automation.
Obviously, you cant mitigate 0-day exploits in any situation where reasonable/expected network access is possible. But our concern, despite not being directly impacted by this, is that we may have accumulated malware over the past decade+ that has never been discovered. How many exploits exist in the wild which have never been documented or even noticed? Do we think it's at least one?
The thinking we are getting into is - If we nuke-from-orbit and then reseed from trusted backups on a recurring basis, any malware that gets installed via some side-channel would not be able to persist for as long as it traditionally would. Keeping backups pure via deterministic cryptographic schemes is far easier to work with than running 100+ security suites across your IT stack in hopes you find something naughty. It is incredibly hard for malware to hide in a well-normalized SQL database without SP or other programmatic features.
What if we built a new IT stack that was designed to be obliterated and reconstructed every 24 hours with latest patch builds each time? Surely many businesses could tolerate 1-2 hours of downtime overnight. It certainly works for the stock market. There really isn't a reason you need to give an attacker a well-managed private island to hide on for 10+ years at a time.
- iforgotpassword 6y agoWe use netboot for most desktop computers and servers that are mostly stateless. Any changes are temporary ending up on a dedicated temp partition that gets wiped on boot, or in ram. Rebuilds are mostly automatic. Of course, netboot in itself opens new attack vectors where we're in early stages of exploring different approaches, even the painful secure boot crap. Honestly I think most of the security in our case right now comes from being an obscure in-house solution that you'd need to specifically target. Also, in case you do get pwned, a post mortem becomes mostly impossible since once you reboot a machine, everything is gone except stuff on network shares of course.
- mlac 6y agoI’ve thought a lot about this. I think from a tech standpoint and a security standpoint, my ideal approach would be to rotate out an A team and a B team. Every 2-3 years, the teams switch off. So year 1-3 A team is running the environment. B team is completely rebuilding and re-architecting the organization’s IT. The company is migrated to B teams infrastructure for 3 years. A team gets to re-build while B team is running and the cycle repeats. This has a few advantages, it keeps the org very current with tools and technology, everyone stays sharp on the latest tech, nothing is sacred, and teams get experience across the spectrum of design build implement and run. It also has good Disaster recovery properties if you idle the old environment so that you can fall back if some critical failure occurs in the new environment. This would be expensive, but please poke holes. I like your idea of clean rebuilds and can see a path to it with automation / terraform / cloud resources. And you don’t need the downtime if you stand up the second one in parallel and just fail over. There’s still persistent data that needs to carry through, so you’d need to figure out how to separate your persistent data from the elements that reset.
- djrogers 6y agoSince you asked us to poke holes: 1) turnover 2) skillset Some people are amazing at the architect/build side of things while either sucking at or hating the run side, and vise versa. Mismatched skill sets leads to higher turnover, which makes running an a/b team routine even harder.
- mlac 6y agoFair points. I would say if it is done well that turnover would go down. I’d think re-architecting from scratch every 4-6 years would be extremely engaging and keep the role interesting. Or it would be extremely tiring and lead to burnout. Not saying the architects would need to run the application for 3 years - during the three years of run for their cycle they could determine issues with their architecture to fix for next time, work with the other architecture team to make improvements for the next cycle, and perform research for their next design. I think the main drawback is cost - it essentially doubles the cost of staffing for the organization’s IT. I guess there is core functionality that could be shared and stay consistent.
- bopbeepboop 6y agoNot the person you’re talking to, but I want to check my inference. Maybe I misunderstood, but I’m trying to recap your point: The teams are on a 3 year production deployment cycle — 0.5y - design next gen system 1.5y - implement next gen system 0.5y - deploy next gen system 3.0y - production (primary, solo, backup; 1y each) 0.5y - decommission Is that what you had in mind? I think what a lot of people aren’t seeing is what it looks like with multiple cycles overlapping: You begin architecture design on gen3 1.5y after deploying gen1. The coding team rolls smoothly from implementing gen1, deploying gen1, and running gen1 into implementing gen3, deploying gen3, and running gen3. (Assuming minimal coders for the backup phase.) It even works out roughly for promotion cycles: an SDE 1 at the start of implementation for gen1 manages a service as an SDE 2 (2yr experience) and can get promoted to SDE 3 part way through gen3 in time for them to design gen5 (having seen two implement-to-maintain cycles). On the production side, operations are continuous: your 3 years of production overlap with the other team by 1, making the entire cycle for a single team 4 years in length. Your production crew spends their entire time on a commission-operation-decommission loop. There’s no downtime: they go straight from decommissioning gen1 to commissioning gen3. Expense is the negative: each team needs a full set of architects, coders, and operators. But nuclear submarines have two teams for a reason, so I think there’s certain domains where operating two full development teams in lockstep like this makes sense. I think it would help a lot with “legacy” bloat: to have upgrade cycles be a fact of the business structure.
- vsareto 6y ago>What if we built a new IT stack that was designed to be obliterated and reconstructed every 24 hours with latest patch builds each time? Inevitably an update is going to break something. So even if you can automate all of that, how can you make sure it doesn't break something? This requirement isn't just the automation and technology gathering, it's testing too. It seems to me like you'd need a lot more benefits to make this worth the time/money/effort. You'd probably be better off having 2 networks for employees: 1 for public internet and 1 for internal company stuff. I think the intelligence community has something like that?
- jka 6y agoCould the same principles from application development apply? Given sufficient coverage, running unit and integration tests for each "infrastructure build artifact" could help to provide assurance. (and if your infrastructure service provider(s) don't have suitable test coverage they can offer, perhaps it's time for a conversation with them about that)
- mleonhard 6y agoWhen you're deploying from backup every day, rollbacks are easy.