3 ms·

I work at a very large IT org (1000+) and the only way I have ever seen anything change substantially and persistently was when the change was introduced and ma

by hpoe 7y ago

I work at a very large IT org (1000+) and the only way I have ever seen anything change substantially and persistently was when the change was introduced and made it far easier to do the right thing. Ultimately if you want people to do the right thing you can't force them into it, you have to make them want it. It seems to me there are lots of opinions on how to change an organization or introduce improvements but ultimately if you aren't making people's lives easier, making them want the change you are trying to bring about, you'll find it almost impossible to succeed, human inertia is just too great.

pjc50 7y ago
This is the "poka-yoke" philosophy.
- deleted 6y ago
  [deleted]
Cthulhu_ 6y ago
The only major changes I've seen in IT organizations of companies was where the existing IT department had declined or was let go over time and any development was in slumber mode. Then an excited manager comes around and erects a new IT organization from nothing (usually leaning heavily on consultants and freelancers), having sold a Digital Transformation to the company. The new IT / developer department spends millions a year in conjuring up a new server stack (cloud native) and a new software architecture (microservices woo), often building on top of existing ancient software (probably running on a mainframe three further abstraction levels down), with the intent of eventually replacing them. Eventually. Maybe. Usually those things got bogged down in the end because after 1-2 years, the first wave of developers get bored and move on to other opportunities, and the architecture they made is poorly abstracted microservices communicating via undocumented and unstructured REST APIs, with not nearly enough end-to-end tests or traceability to reliably make changes or add features. [/rant]
- DerArzt 6y ago
  That got a little to real for me, I started thinking about work.
davedx 6y ago
If only CloudFormation followed this philosophy. Unfortunately, doing things "The Right Way (tm)" within AWS means torturously long feedback loops, tearing down stacks and rebuilding them again over and over again (and when you're working with RDS databases, "tear down" is 10 mins and "rebuild" is 15 mins), then some stacks fail to delete because an IAM role that was just created is now somehow unable to be deleted... Our company has restricted it so CloudFormation is the only way to provision infrastructure because doing things the right way makes our lives significantly harder. And so the "native cloud" tax continues to accumulate and grow until you spend more and more time and money on just building and maintaining infrastructure...
- carlsborg 6y ago
  Why are you bringing down the RDS stack to make cloudformation udpates? Can you share an example of something that makes you need to do this over and over again?
  - davedx 6y ago
    I'm creating a database with parameters, roles, backup buckets and so on. Usually each time I run the stack I either get a template error, or the stack partly runs then hits some other error. Then the stack rolls back, and re-running it will fail because it's in rolled_back state. So then I need to delete the whole thing and start again. When I can simply re-run and update the stack without deleting it I do (for example if I just change a property of one of the resources). But I spend a lot of time with stacks that are in broken states for various reasons.
    - acdha 6y ago
      I share this pain - it’s a major reason why we switched to Terraform, along with not having to wait 6-12 months for support for new AWS features - but I would recommend using CDK to generate your CF stacks. The cfn-python-lint tool was helpful for catching many things without a 20 minute delay but CDK goes even further on the boilerplate reduction and a real programming language beats YAML in every way.
user5994461 6y ago
>>> ... when the change was introduced and made it far easier to do the right thing. Correction: When the change was introduced and made it far easier for people to do the thing they are trying to do. Nobody cares about doing the right thing, employees just do whatever they have to do now then move on to the next task/project. More often than not, changes are making it harder for people to do their job and it's no surprise that it's not welcome.
- piva00 6y ago
  Exactly, from my personal experience that is the hardest ball to juggle: there are changes that will be painful, that won't have an immediate impact but that over time will compound to it. How do you sell this to people that are going to go through the pain but very likely won't be there to see its benefits?
hedora 6y ago
A lot of people fail to realize there are large multiplicative factors when the infrastructure team makes other parts of the organization implement a “simple” change. In large companies, there might be dozens or hundreds of teams that rely on the infrastructure. If the change requires each team to learn the tool of the month, and make a one line configuration change, it will cost the organization hundreds of developer days to implement. If engineering and release management have to schedule that work to align with business constraints, the change is costing years of people time. If you’re even further down the rabbit hole, you’ve adopted devops, and are forcing people that actively hate operations stuff to learn obscure in-house operations tools. That, combined with preventing them from doing the work they signed up for, leads to burn out and resignations. It can get even worse than that. I routinely see all of the above compounded with infra bikeshedding that delays rollouts by months, and results in weeks of work per team. At that point, the organizational damage ends up being many times the budget of the IT department. So, yeah, IT needs to figure out how to help people be more productive while simultaneously implementing best practices, and it’s hard. I deeply appreciate people that are good at that (and have worked with some).

by hpoe 7y ago

pjc50 7y ago

This is the "poka-yoke" philosophy.

deleted 6y ago
[deleted]

Cthulhu_ 6y ago

The only major changes I've seen in IT organizations of companies was where the existing IT department had declined or was let go over time and any development was in slumber mode. Then an excited manager comes around and erects a new IT organization from nothing (usually leaning heavily on consultants and freelancers), having sold a Digital Transformation to the company. The new IT / developer department spends millions a year in conjuring up a new server stack (cloud native) and a new software architecture (microservices woo), often building on top of existing ancient software (probably running on a mainframe three further abstraction levels down), with the intent of eventually replacing them. Eventually. Maybe. Usually those things got bogged down in the end because after 1-2 years, the first wave of developers get bored and move on to other opportunities, and the architecture they made is poorly abstracted microservices communicating via undocumented and unstructured REST APIs, with not nearly enough end-to-end tests or traceability to reliably make changes or add features. [/rant]

DerArzt 6y ago
That got a little to real for me, I started thinking about work.

davedx 6y ago

If only CloudFormation followed this philosophy. Unfortunately, doing things "The Right Way (tm)" within AWS means torturously long feedback loops, tearing down stacks and rebuilding them again over and over again (and when you're working with RDS databases, "tear down" is 10 mins and "rebuild" is 15 mins), then some stacks fail to delete because an IAM role that was just created is now somehow unable to be deleted... Our company has restricted it so CloudFormation is the only way to provision infrastructure because doing things the right way makes our lives significantly harder. And so the "native cloud" tax continues to accumulate and grow until you spend more and more time and money on just building and maintaining infrastructure...

carlsborg 6y ago
Why are you bringing down the RDS stack to make cloudformation udpates? Can you share an example of something that makes you need to do this over and over again?
- davedx 6y ago
  I'm creating a database with parameters, roles, backup buckets and so on. Usually each time I run the stack I either get a template error, or the stack partly runs then hits some other error. Then the stack rolls back, and re-running it will fail because it's in rolled_back state. So then I need to delete the whole thing and start again. When I can simply re-run and update the stack without deleting it I do (for example if I just change a property of one of the resources). But I spend a lot of time with stacks that are in broken states for various reasons.
  - acdha 6y ago
    I share this pain - it’s a major reason why we switched to Terraform, along with not having to wait 6-12 months for support for new AWS features - but I would recommend using CDK to generate your CF stacks. The cfn-python-lint tool was helpful for catching many things without a 20 minute delay but CDK goes even further on the boilerplate reduction and a real programming language beats YAML in every way.

user5994461 6y ago

>>> ... when the change was introduced and made it far easier to do the right thing. Correction: When the change was introduced and made it far easier for people to do the thing they are trying to do. Nobody cares about doing the right thing, employees just do whatever they have to do now then move on to the next task/project. More often than not, changes are making it harder for people to do their job and it's no surprise that it's not welcome.

piva00 6y ago
Exactly, from my personal experience that is the hardest ball to juggle: there are changes that will be painful, that won't have an immediate impact but that over time will compound to it. How do you sell this to people that are going to go through the pain but very likely won't be there to see its benefits?

hedora 6y ago

A lot of people fail to realize there are large multiplicative factors when the infrastructure team makes other parts of the organization implement a “simple” change. In large companies, there might be dozens or hundreds of teams that rely on the infrastructure. If the change requires each team to learn the tool of the month, and make a one line configuration change, it will cost the organization hundreds of developer days to implement. If engineering and release management have to schedule that work to align with business constraints, the change is costing years of people time. If you’re even further down the rabbit hole, you’ve adopted devops, and are forcing people that actively hate operations stuff to learn obscure in-house operations tools. That, combined with preventing them from doing the work they signed up for, leads to burn out and resignations. It can get even worse than that. I routinely see all of the above compounded with infra bikeshedding that delays rollouts by months, and results in weeks of work per team. At that point, the organizational damage ends up being many times the budget of the IT department. So, yeah, IT needs to figure out how to help people be more productive while simultaneously implementing best practices, and it’s hard. I deeply appreciate people that are good at that (and have worked with some).