6 ms·

I’ve thought a lot about this. I think from a tech standpoint and a security standpoint, my ideal approach would be to rotate out an A team and a B team. Every

by mlac 6y ago

I’ve thought a lot about this. I think from a tech standpoint and a security standpoint, my ideal approach would be to rotate out an A team and a B team. Every 2-3 years, the teams switch off. So year 1-3 A team is running the environment. B team is completely rebuilding and re-architecting the organization’s IT. The company is migrated to B teams infrastructure for 3 years. A team gets to re-build while B team is running and the cycle repeats. This has a few advantages, it keeps the org very current with tools and technology, everyone stays sharp on the latest tech, nothing is sacred, and teams get experience across the spectrum of design build implement and run. It also has good Disaster recovery properties if you idle the old environment so that you can fall back if some critical failure occurs in the new environment. This would be expensive, but please poke holes. I like your idea of clean rebuilds and can see a path to it with automation / terraform / cloud resources. And you don’t need the downtime if you stand up the second one in parallel and just fail over. There’s still persistent data that needs to carry through, so you’d need to figure out how to separate your persistent data from the elements that reset.

djrogers 6y ago
Since you asked us to poke holes: 1) turnover 2) skillset Some people are amazing at the architect/build side of things while either sucking at or hating the run side, and vise versa. Mismatched skill sets leads to higher turnover, which makes running an a/b team routine even harder.
- mlac 6y ago
  Fair points. I would say if it is done well that turnover would go down. I’d think re-architecting from scratch every 4-6 years would be extremely engaging and keep the role interesting. Or it would be extremely tiring and lead to burnout. Not saying the architects would need to run the application for 3 years - during the three years of run for their cycle they could determine issues with their architecture to fix for next time, work with the other architecture team to make improvements for the next cycle, and perform research for their next design. I think the main drawback is cost - it essentially doubles the cost of staffing for the organization’s IT. I guess there is core functionality that could be shared and stay consistent.
  - bopbeepboop 6y ago
    Not the person you’re talking to, but I want to check my inference. Maybe I misunderstood, but I’m trying to recap your point: The teams are on a 3 year production deployment cycle — 0.5y - design next gen system 1.5y - implement next gen system 0.5y - deploy next gen system 3.0y - production (primary, solo, backup; 1y each) 0.5y - decommission Is that what you had in mind? I think what a lot of people aren’t seeing is what it looks like with multiple cycles overlapping: You begin architecture design on gen3 1.5y after deploying gen1. The coding team rolls smoothly from implementing gen1, deploying gen1, and running gen1 into implementing gen3, deploying gen3, and running gen3. (Assuming minimal coders for the backup phase.) It even works out roughly for promotion cycles: an SDE 1 at the start of implementation for gen1 manages a service as an SDE 2 (2yr experience) and can get promoted to SDE 3 part way through gen3 in time for them to design gen5 (having seen two implement-to-maintain cycles). On the production side, operations are continuous: your 3 years of production overlap with the other team by 1, making the entire cycle for a single team 4 years in length. Your production crew spends their entire time on a commission-operation-decommission loop. There’s no downtime: they go straight from decommissioning gen1 to commissioning gen3. Expense is the negative: each team needs a full set of architects, coders, and operators. But nuclear submarines have two teams for a reason, so I think there’s certain domains where operating two full development teams in lockstep like this makes sense. I think it would help a lot with “legacy” bloat: to have upgrade cycles be a fact of the business structure.
    - mlac 6y ago
      I wrote this in another comment, but I should have been more clear. Since this is the discussion about the Exchange hack, I was thinking in the context of large organizations and their internal IT architecture and being able to build from scratch without legacy bloat. The challenge is “how do we run this organization as efficiently and securely as possible? What tools does the business need in place to get the job done? Is our current set sufficient?”. The fact that any company hands a new employee a Windows 7 laptop in 2021 shouldn’t be happening, but a surprising number of Fortune 500 companies are in that state because of legacy dependencies that require Win 7 to operate. I think the ability to give an organization the opportunity to reset every 3 years would keep things efficient, better integrated, and identify legacy issues that often come up and cause emergencies (the guy who wrote that script left the company 5 years ago and it runs on a server under this desk... we just don’t touch it). Right now, upgrading a payment system may be difficult due to certain dependencies or other legacy internal systems. If the whole architecture is being re-done, there is a lot more flexibility.
bob1029 6y ago
I think the need to maintain multiple team may not be as urgent if you constrain the timeline. The biggest requirement I see is automation. For this to be feasible in a general sense, it has to be down to a single method invocation completes in 1 hour what those teams are doing in 2-3 years. The biggest challenge that will emerge from trying to meet this objective is the import/export of data to/from these now-highly-ephemeral IT systems. The ability to easily import pure business data back into a fresh instance of the system will likely constrain the vendor & product choices as well. Very soon, you might find yourself building a 100% custom vertical to support these objectives explicitly. I think this is ultimately inevitable and desirable though. We just need to learn how to build these things quickly & reliably.
- mlac 6y ago
  Yeah I see it as two items - total architecture rebuilds and redesigns for an organization’s IT system vs. blowing away enterprise resources each night and restoring with a known good application each day or week. That would be amazing but incredibly complex. Each week I guess you would run a script to re-build your architecture in AWS with the latest builds and patches. Then run a config script to re-import all your data. It would be painful to figure out, but you could essentially store a copy of your data at another AWS location and fail over within a day or two just given your two install scripts (the architecture build out and then the config script to read in the data), Depending how often and on how many systems you did this on, you’d basically make attackers restart every night or week. And ideally you’re patching as quickly as possible, so it might block some of them out quickly.
oauea 6y ago
Sounds like an excellent strategy for resume padding.
RijilV 6y ago
doubles the cost of implementing anything. Say a customer wants feature-X. Unless you're magically at the point where in your 2-3 year cycle where you're switching, both the A and B team need to implement feature. Of course, that's assuming you don't just tell the customer to stuff it and wait 2-3 years. You're also assuming that you know ahead of time all use cases and interfaces. It's surprising how dependencies are taken. I've seen large scale systems break when a HTTP 204 was changed to a HTTP 206, or a base36 field changed to base62. Now again maybe you're thinking the consumer can stuff it and update everything whenever you decide to switch over, or that you'll have captured everything and have tests around it. But.. for any sufficiently complex system with a sufficiently large customer base everything about your interface becomes your customer contract. Changing everything all at once is going to break a ton of things nobody ever thought about. Doing upgrades every 2-3 years means you're pretty much never going to be good at them. Institutional knowledge seems to have a 2-3 year memory horizon. Sure, you get that one person who is a bit of an archeologist/historian but tenure at most shops is not long ("The median number of years wage and salaried employees stayed with their current employer in 2018 was 4.2 years" - first hit on Google). While you're upgrading every 3 years, each team only does so every 6 years. Nobody is gonna remember what it looked like. There's also a meta point, which is what are you actually trying to solve? Is it so hard to go from architecture A.v0 -> A.v1 -> architecture B that you need to build A, maintain A and simultaneously build B? If moving between architectures is so hard but moving between versions of an architecture isn't - why is that the case and why can't you make the former case easier? I'm assuming that your plan has you upgrading the A-architecture within those 2-3 years. Maybe you're saying you wouldn't touch it at all and just hope there are no security issues or features or scaling you need to do. There's also another point which is you've coupled all changes to a particular cadence. Maybe you want to upgrade your network, servers, storage systems, OS, application services, etc on different cycles. At the very least you're sorta hoping that all of those things have similar release cycles, which realistically you're going to be picking some network switch that's been out for 2 years and marrying it to a storage product that was released last month (because the previous one is 5 years old and will be out of support before your next refresh). And scaling... what happens when you can't get the same server you were ordering 2 years ago? Tell users they can't have nice things until the other team rolls out their massive platform shift in a year? Or would you adopt a new platform to scale on, in which case, why are you doing this A and B team thing again? And not only do you need two teams, but you need two sets of hardware which means you need twice as much datacenter space, etc etc. Do folks need to two desk phones when you roll that out? And ... I'm gonna stop here...

by mlac 6y ago

djrogers 6y ago

Since you asked us to poke holes: 1) turnover 2) skillset Some people are amazing at the architect/build side of things while either sucking at or hating the run side, and vise versa. Mismatched skill sets leads to higher turnover, which makes running an a/b team routine even harder.

mlac 6y ago
Fair points. I would say if it is done well that turnover would go down. I’d think re-architecting from scratch every 4-6 years would be extremely engaging and keep the role interesting. Or it would be extremely tiring and lead to burnout. Not saying the architects would need to run the application for 3 years - during the three years of run for their cycle they could determine issues with their architecture to fix for next time, work with the other architecture team to make improvements for the next cycle, and perform research for their next design. I think the main drawback is cost - it essentially doubles the cost of staffing for the organization’s IT. I guess there is core functionality that could be shared and stay consistent.
- bopbeepboop 6y ago
  Not the person you’re talking to, but I want to check my inference. Maybe I misunderstood, but I’m trying to recap your point: The teams are on a 3 year production deployment cycle — 0.5y - design next gen system 1.5y - implement next gen system 0.5y - deploy next gen system 3.0y - production (primary, solo, backup; 1y each) 0.5y - decommission Is that what you had in mind? I think what a lot of people aren’t seeing is what it looks like with multiple cycles overlapping: You begin architecture design on gen3 1.5y after deploying gen1. The coding team rolls smoothly from implementing gen1, deploying gen1, and running gen1 into implementing gen3, deploying gen3, and running gen3. (Assuming minimal coders for the backup phase.) It even works out roughly for promotion cycles: an SDE 1 at the start of implementation for gen1 manages a service as an SDE 2 (2yr experience) and can get promoted to SDE 3 part way through gen3 in time for them to design gen5 (having seen two implement-to-maintain cycles). On the production side, operations are continuous: your 3 years of production overlap with the other team by 1, making the entire cycle for a single team 4 years in length. Your production crew spends their entire time on a commission-operation-decommission loop. There’s no downtime: they go straight from decommissioning gen1 to commissioning gen3. Expense is the negative: each team needs a full set of architects, coders, and operators. But nuclear submarines have two teams for a reason, so I think there’s certain domains where operating two full development teams in lockstep like this makes sense. I think it would help a lot with “legacy” bloat: to have upgrade cycles be a fact of the business structure.
  - mlac 6y ago
    I wrote this in another comment, but I should have been more clear. Since this is the discussion about the Exchange hack, I was thinking in the context of large organizations and their internal IT architecture and being able to build from scratch without legacy bloat. The challenge is “how do we run this organization as efficiently and securely as possible? What tools does the business need in place to get the job done? Is our current set sufficient?”. The fact that any company hands a new employee a Windows 7 laptop in 2021 shouldn’t be happening, but a surprising number of Fortune 500 companies are in that state because of legacy dependencies that require Win 7 to operate. I think the ability to give an organization the opportunity to reset every 3 years would keep things efficient, better integrated, and identify legacy issues that often come up and cause emergencies (the guy who wrote that script left the company 5 years ago and it runs on a server under this desk... we just don’t touch it). Right now, upgrading a payment system may be difficult due to certain dependencies or other legacy internal systems. If the whole architecture is being re-done, there is a lot more flexibility.

bob1029 6y ago

I think the need to maintain multiple team may not be as urgent if you constrain the timeline. The biggest requirement I see is automation. For this to be feasible in a general sense, it has to be down to a single method invocation completes in 1 hour what those teams are doing in 2-3 years. The biggest challenge that will emerge from trying to meet this objective is the import/export of data to/from these now-highly-ephemeral IT systems. The ability to easily import pure business data back into a fresh instance of the system will likely constrain the vendor & product choices as well. Very soon, you might find yourself building a 100% custom vertical to support these objectives explicitly. I think this is ultimately inevitable and desirable though. We just need to learn how to build these things quickly & reliably.

mlac 6y ago
Yeah I see it as two items - total architecture rebuilds and redesigns for an organization’s IT system vs. blowing away enterprise resources each night and restoring with a known good application each day or week. That would be amazing but incredibly complex. Each week I guess you would run a script to re-build your architecture in AWS with the latest builds and patches. Then run a config script to re-import all your data. It would be painful to figure out, but you could essentially store a copy of your data at another AWS location and fail over within a day or two just given your two install scripts (the architecture build out and then the config script to read in the data), Depending how often and on how many systems you did this on, you’d basically make attackers restart every night or week. And ideally you’re patching as quickly as possible, so it might block some of them out quickly.

oauea 6y ago

Sounds like an excellent strategy for resume padding.

RijilV 6y ago

doubles the cost of implementing anything. Say a customer wants feature-X. Unless you're magically at the point where in your 2-3 year cycle where you're switching, both the A and B team need to implement feature. Of course, that's assuming you don't just tell the customer to stuff it and wait 2-3 years. You're also assuming that you know ahead of time all use cases and interfaces. It's surprising how dependencies are taken. I've seen large scale systems break when a HTTP 204 was changed to a HTTP 206, or a base36 field changed to base62. Now again maybe you're thinking the consumer can stuff it and update everything whenever you decide to switch over, or that you'll have captured everything and have tests around it. But.. for any sufficiently complex system with a sufficiently large customer base everything about your interface becomes your customer contract. Changing everything all at once is going to break a ton of things nobody ever thought about. Doing upgrades every 2-3 years means you're pretty much never going to be good at them. Institutional knowledge seems to have a 2-3 year memory horizon. Sure, you get that one person who is a bit of an archeologist/historian but tenure at most shops is not long ("The median number of years wage and salaried employees stayed with their current employer in 2018 was 4.2 years" - first hit on Google). While you're upgrading every 3 years, each team only does so every 6 years. Nobody is gonna remember what it looked like. There's also a meta point, which is what are you actually trying to solve? Is it so hard to go from architecture A.v0 -> A.v1 -> architecture B that you need to build A, maintain A and simultaneously build B? If moving between architectures is so hard but moving between versions of an architecture isn't - why is that the case and why can't you make the former case easier? I'm assuming that your plan has you upgrading the A-architecture within those 2-3 years. Maybe you're saying you wouldn't touch it at all and just hope there are no security issues or features or scaling you need to do. There's also another point which is you've coupled all changes to a particular cadence. Maybe you want to upgrade your network, servers, storage systems, OS, application services, etc on different cycles. At the very least you're sorta hoping that all of those things have similar release cycles, which realistically you're going to be picking some network switch that's been out for 2 years and marrying it to a storage product that was released last month (because the previous one is 5 years old and will be out of support before your next refresh). And scaling... what happens when you can't get the same server you were ordering 2 years ago? Tell users they can't have nice things until the other team rolls out their massive platform shift in a year? Or would you adopt a new platform to scale on, in which case, why are you doing this A and B team thing again? And not only do you need two teams, but you need two sets of hardware which means you need twice as much datacenter space, etc etc. Do folks need to two desk phones when you roll that out? And ... I'm gonna stop here...