9 ms·

Slow deployment causes meetings (2015)

188 points by fagnerbrack 2y ago

dang 2y ago
Related: Slow Deployment Causes Meetings - https://news.ycombinator.com/item?id=10622834 https://news.ycombinator.com/item?id=10622834 - Nov 2015 (26 comments)
yarg 2y ago
I had a boss who actually acknowledged that he was deliberately holding up my development process - this was a man who refused to allow me a four day working week.
Sparkyte 2y ago
Sounds like a process problem. 2024 development cycles should be able to handle multiple lanes of development and deployments. Also why things moved to microservices so you can deploy with minimal impact as long as you don't tightly couple your dependencies.
- m00x 2y ago
  You don't need microservices to do this. It's actually easier deploying a monolith with internal dependencies than deploying microservices that depend on each other.
  - adrianpike 2y ago
    This is very accurate - microservices can be great as a forcing function to revisit your architectural boundaries, but if all you do is add a network hop and multiple components to update when you tweak a data model, all you'll get is headcount sprawl and deadlock to the moon. I'm a huge fan of migrating to microservices as a secondary outcome of revisiting your component boundaries, but just moving to separate repos & artifacts so we can all deploy independently is a recipe for pain.
    - jrs235 2y ago
      and a recipe for "career" driven managers and directors to grow department head count, budget oversight, and self importance.
    - Sparkyte 2y ago
      Network hop isn't needed if you're deploying your microservices correctly. So you can make pod groups inside of kubernetes and application that depends on another can call that lightweight container contained in that pod group. Pods inherently know the other is there in their group it has some or like network call without traversing hardware.
  - Sparkyte 2y ago
    I know microservices and monoliths are a heated topic. However breaking up complicated code to preserve user experience is sometimes essential. However you can have machines that contain many services and that interact with each for performance if needed. You would put them into pod groups while deploying to kubernetes and have them call their service inside of the pod. This can increase performance and through put.
lizzas 2y ago
Microservices lets you horizontally scale deployment frequency too.
- theptip 2y ago
  Not a silver bullet; you increase api versioning overhead between services for example.
  - whateveracct 2y ago
    True but your API won't be changing that rapidly especially in a backwards-incompatible way.
    - dhfuuvyvtt 2y ago
      What's that got to do with microservices? Edit, because you can avoid those things in a monolith.
      - whateveracct 2y ago
        You avoid a lot of things in a monolith. But normal services mapped to your org chart tend to be pretty nice. I hate how people spam "micro" service talk as if that's the end game for putting a physical boundary between software.
    - theptip 2y ago
      The more/smaller microservices you have, the more frequently your APIs must change. It’s more fruitful to recognize that this is a dimension of freedom rather than a binary decision.
      - whateveracct 2y ago
        I mean I recommend that you don't go micro right away. But a few well-placed service boundaries that align with your eng org chart pay dividends and help build discipline and rigor. Even if the services are all running on the same box to start and just IPCing over localhost or w/e. I prefer my teams build habits while it's easy and fun rather than once the monolith gets unwieldy and drawing boundaries becomes painful.
qaq 2y ago
A bit tangential but why is CloudFormation so slowww?
- justin_oaks 2y ago
  I figure it's because AWS can get away with it.
  - shepherdjerred 2y ago
    AWS deploys using cfn internally
    - bobnamob 2y ago
      This is only partly true. The foundational services do not use cfn for the vast majority of their deployments.
- Aeolun 2y ago
  The reason by boss tends to give is that it’s made by AWS, so it cannot possibly be bad. Also, it’s free. Which is never given as anything more than a tangentially related reason, but…
  - Uehreka 2y ago
    It… definitely isn’t free. Have you ever looked at the “Config” category of your AWS bill?
- hk1337 2y ago
  This is just anecdotal but I have found anytime a network interface is involved, it can slow down the deployment. I had a case where I was deleting lambdas in a VPC, and connected to EFS, that the deployment was rather quick but it took ~20 minutes for cloudformation to cleanup and finish.
- motorest 2y ago
  > A bit tangential but why is CloudFormation so slowww? It's not that CloudFormation is slow. It's that the whole concept of infrastructure-as code-as-codd is slow by nature. Each time you deploy a change to a state as a transaction, you need to assert preconditions and post-conditions at each step. If you have to roll out a set of changes that have any semblance of interdependence, you have no option other than to deploy each change as sequential steps. Each step requires many network calls to apply changes, go through auth, poll state, each one taking somewhere between 50-200ms. That quickly adds up. If you deploy the same app on a different cloud provider with Terraform or Ansible, you get the same result. If you deploy the same changes manually you turn a few minutes into a day-long ordeal. The biggest problem with IaC is that it is so high-level and does so much under the hood that some people have no idea what changes they are actually applying or what they are doing. Then they complain it takes so long.
khana 2y ago
[dead]
jojobas 2y ago
Fast deployment causes incident war rooms.
- DougBTX 2y ago
  Maybe the opposite, slow rollbacks cause escalating incidents.
- Trasmatta 2y ago
  In my experience, there's very little correlation. I've been on projects with 1 deployment every six weeks, and there were just as many production incidents as projects with daily deployments.
- boxed 2y ago
  I was on a team that went from every 3 weeks to multiple times per day. The number of incidents in production dropped drastically. But much more important than that drop, was that when things went wrong is was MUCH MUCH faster to find the problem. It was also much safer and easier to roll back, since there were so few changes that would be rolled back. No one wants to back off 3 weeks of work. That's chaos.
- wussboy 2y ago
  That is the opposite of my experience. Slow deploys mean bigger deploys mean more complexity going live mean more nervousness and more testing mean more hesitation mean more chance that something unforeseen mean errors that no one understands mean war rooms.
- wasmitnetzen 2y ago
  Yeah, and slow ones as well.
sourceless 2y ago
I think unfortunately the conclusion here is a bit backwards; de-risking deployments by improving testing and organisational properties is important, but is not the only approach that works. The author notes that there appears to be a fixed number of changes per deployment and that it is hard to increase - I think the 'Reversie Thinkie' here (as the author puts it) is actually to decrease the number of changes per deployment. The reason those meetings exist is because of risk! The more changes in a deployment, the higher the risk that one of them is going to introduce a bug or operational issue. By deploying small changes often, you get deliver value much sooner and fail smaller. Combine this with techniques such as canarying and gradual rollout, and you enter a world where deployments are no longer flipping a switch and either breaking or not breaking - you get to turn outages into degradations. This approach is corroborated by the DORA research[0], and covered well in Accelerate[1]. It also features centrally in The Phoenix Project[2] and its spiritual ancestor, The Goal[3]. [0] https://dora.dev/ https://dora.dev/ [1] https://www.amazon.co.uk/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339 https://www.amazon.co.uk/Accelerate-Software-Performing-Tech... [2] https://www.amazon.co.uk/Phoenix-Project-Helping-Business-Anniversary/dp/B00VBEBRK6 https://www.amazon.co.uk/Phoenix-Project-Helping-Business-An... [3] https://www.amazon.co.uk/Goal-Process-Ongoing-Improvement/dp/0566086654 https://www.amazon.co.uk/Goal-Process-Ongoing-Improvement/dp...
- tomxor 2y ago
  I tend to agree. Whenever I've removed artificial technical friction, or made a fundamental change to an approach, the processes that grew around them tend to evaporate, and not be replaced. I think many of these processes are a rational albeit non-technical response to making the best of a bad situation in the absence of a more fundamental solution. But that doesn't mean they are entirely harmless. I've come across some scenarios where the people driving decisions continued to reach for human processes as the solution rather than a workaround, for both new projects and projects designated specifically to remove existing inefficiencies. They either lacked the technical imagination, or were too stuck in the existing framing of the problem, and this is where people who do have that imagination need to speak up and point out that human processes need to be minimised with technical changes where possible. Not all human processes can be obviated through technical changes, but we don't want to spread ourselves thin on unnecessary ones.
andy_ppp 2y ago
The organisation will actively prevent you from trying to improve deployments though, they will say things like “Jenkins shouldn’t be near production” or “we can’t possibly put things live without QA being involved” or “we need this time to make sure the quality of the software is high enough”. All with a straight face while having millions of production bugs and a product that barely meets any user requirements (if there are any). In the end fighting the bureaucracy is actually impossible in most organisations, especially if you’re not part of the 200 layers of management that create these meetings. I would sack everyone but programmers and maybe two designers and let everyone fight it out without any agile coaches and product owners and scrum master and product experts. Slow deployment is a problem but it’s not the problem.
- gleenn 2y ago
  You sound very defeatist about fighting bureaucracy. If you work at an org with too much management, you can slowly push to move it in the direction you hope for or leave. If you keep ending up at places that seem impossible to change, perhaps you should ask more questions about this during the interview. I've worked at many small companies where there wasn't crazy bureaucracy because that's definitely what I preferred. I also currently work at a megacorp and yes there is difficulty, but being consistent and persuasive has lead to many things slowly heading in the right direction. Things take time. You have to realize why people have made things some way and then find convincing arguments to make things better. Sometimes places do just suck so don't stick around. But being hopeless doesn't seem helpful.
  - mewpmewp2 2y ago
    My issue is that it just pays more. I have no similar paying option that would have good experience in that sense.
- gavmor 2y ago
  > Jenkins shouldn’t be near production > we can’t possibly put things live without QA being involved > we need this time to make sure the quality of the software is high enough I've only developed software professionally since 2012, but in that time not only have I never encountered such sentiments, but (and, perhaps, because) it has always been a top priority of leadership to emphatically insist on the very opposite: day one of any initiative is Jenkins to production—often directly via trunk-based development—and quality is every developer's responsibility. At the IC level, there was no "fighting bureaucracy," although I don't doubt leadership debated these things vigorously, from time to time, especially as external partners and stakeholders were often intimately involved. > I would sack everyone but programmers and maybe two designers and let everyone fight it out That works for me! But it doesn't scale. We definitely have to keep at least one product "owner" or "expert" or "manager" to enqueue stakeholder priorities and, while this can be a "hat" that devs and designers trade off, it's also a skill at which some individuals uniquely excel. All that being said, I don't want to come across as pearl-clutching, shocked Pikachu face about this. I understand that many organizations don't operate this way. The way I've helped firms make this change is via the introduction of a single, experimental team of volunteers dedicated to these practices—one protected (but not dictated to) by a mandate from on high. But, then again, this is California.
austin-cheney 2y ago
While this is mostly correct it’s also just as irrelevant. TLDR; software performance, thus human performance, is all that matters. Risk management/acceptance can be measured with numbers. In software this is actually far more straightforward than in many other careers, because software engineers can only accept risk within the restrictions of their known operating constraints and everything else is deferred. If you want to go faster you need to maximize the frequency of human iteration above absolutely everything else. If a person cannot iterate, such as waiting on permissions, they are blocked. If they are waiting on a build or screen refresh they are slowed. This can also be measured with numbers. If person A can iterate 100x faster than person B correctness becomes irrelevant. Person B must maximize upon correctness because they are slow. To be faster and more correct person A has extreme flexibility to learn, fail, and improve beyond what person B can deliver. Part of iterating faster AND reducing risk is fast test automation. If person A can execute 90+% test coverage in time of 4 human iterations then that test automation is still 25x faster than one person B iteration with a 90+% lower risk of regression.
black_13 2y ago
[dead]
vegetablepotpie 2y ago
I have personal experience with this in my professional career. Before Christmas break I had a big change, and there was fear. My org responded by increasing testing (regression testing, which increased overhead). This increased the risk that changes on dev would break changes on my branch (not a code merging way, but in a complex adaptive system way). I responded to this risk by making a meeting. I presented our project schedule, and told my colleagues about their expectations, I.e. if they drop code style comments on the PRs they will be deferred to a future PR (and then ignored and never done). What we needed is fine grained testing with better isolation between components. The problem is is that our management is at a high level, they don’t see meetings as a means to an end, they see meetings as a worthy goal in and of itself self to achieve. More meetings means more collaboration, means good. I’d love to see advice on how to lead technical changes with non-technical management.
lifeisstillgood 2y ago
I am trying to expound a concept I call “software literacy” - where a business can be run via code just as much as today a company can be run by English words (policy documents, emails etc). This leads to a few corollaries - things like “If GPUs do the work then coders are the new managers” or we need whole-org-test-rigs to be clear about the impacts of chnages. This seems directly related to this excellent article - to my mind if all the decision makers are not looking at the code as the first class object in a chnage process (is opposed to Jiras or project plans) then not all decision makers are (software) literate - and this comes up a lot in the threads here (“how do I discuss with non-technical management”) - the answer is you cannot - that management must be changed. This is an enormous generational road block that I thought was a problem thirty years ago but naively assumed would disappear as coders grew up. Of course the problem is that to “run” a company one does not need to code - so until not coding is something embarrassing like not writing is for a newspaper editor we won’t get past it. The main point is that we need companies that can be run with the new set of self-reinforcing concepts - sops, testing, not meetings but systems as communication. I will try and rewrite this comment later - it needs work
- gavmor 2y ago
  You had me at "whole org test" harness. This is a very, very interesting idea. Especially in conjunction with the concept of corporation as "slow AI" that I don't hear referenced often enough. I don't see why you call it "literacy," though. I think Maturana & Varela's term "autopoiesis" more closely orbits the kernel, and I'll bet Stafford Beer's Autopoietic Systems would contribute to a good intellectual foundation. At a certain point, though, I wonder if a purely software "business" doesn't just look like... SaaS?
- fduran 2y ago
  I've called that "Organization as Code" some years back :-)
braza 2y ago
A marginally related point but I do not know if others faced the following situation: I worked in a place with a CI pipeline room ~25 minutes with the unit/integration tests (3000+) taking 18 minutes. When something happens in production we ended up placing more tests; and of course when things goes south at least 50 minutes were necessary to recover. After a lot of consideration we decided to focus on the recovery and relax and simply some tests and focus on recovery (i.e. have the full thing in less than 5 minutes) combined with a canary as deployment strategy (instead rolling updates). At least for us was a so refreshing experience but sounded wrong in some ways.
- wussboy 2y ago
  I’ve often said that it is the speed of deployment that matters. If it takes you 50 minutes to deploy, it takes you 50 minutes to fix a problem. If it takes you 50 seconds to deploy, it takes you 50 seconds to fix a problem. Of course all kinds of things are rolled up in that speed to deploy, but almost all of them are good.
tpoacher 2y ago
Meetings (used right) are a great tool, in the same sense that project planners (used right) are a great tool. But then there's Jira. /s

188 points by fagnerbrack 2y ago

dang 2y ago

Related: Slow Deployment Causes Meetings - https://news.ycombinator.com/item?id=10622834 https://news.ycombinator.com/item?id=10622834 - Nov 2015 (26 comments)

yarg 2y ago

I had a boss who actually acknowledged that he was deliberately holding up my development process - this was a man who refused to allow me a four day working week.

Sparkyte 2y ago

Sounds like a process problem. 2024 development cycles should be able to handle multiple lanes of development and deployments. Also why things moved to microservices so you can deploy with minimal impact as long as you don't tightly couple your dependencies.

m00x 2y ago
You don't need microservices to do this. It's actually easier deploying a monolith with internal dependencies than deploying microservices that depend on each other.
- adrianpike 2y ago
  This is very accurate - microservices can be great as a forcing function to revisit your architectural boundaries, but if all you do is add a network hop and multiple components to update when you tweak a data model, all you'll get is headcount sprawl and deadlock to the moon. I'm a huge fan of migrating to microservices as a secondary outcome of revisiting your component boundaries, but just moving to separate repos & artifacts so we can all deploy independently is a recipe for pain.
  - jrs235 2y ago
    and a recipe for "career" driven managers and directors to grow department head count, budget oversight, and self importance.
  - Sparkyte 2y ago
    Network hop isn't needed if you're deploying your microservices correctly. So you can make pod groups inside of kubernetes and application that depends on another can call that lightweight container contained in that pod group. Pods inherently know the other is there in their group it has some or like network call without traversing hardware.
- Sparkyte 2y ago
  I know microservices and monoliths are a heated topic. However breaking up complicated code to preserve user experience is sometimes essential. However you can have machines that contain many services and that interact with each for performance if needed. You would put them into pod groups while deploying to kubernetes and have them call their service inside of the pod. This can increase performance and through put.

lizzas 2y ago

Microservices lets you horizontally scale deployment frequency too.

theptip 2y ago
Not a silver bullet; you increase api versioning overhead between services for example.
- whateveracct 2y ago
  True but your API won't be changing that rapidly especially in a backwards-incompatible way.
  - dhfuuvyvtt 2y ago
    What's that got to do with microservices? Edit, because you can avoid those things in a monolith.
    - whateveracct 2y ago
      You avoid a lot of things in a monolith. But normal services mapped to your org chart tend to be pretty nice. I hate how people spam "micro" service talk as if that's the end game for putting a physical boundary between software.
  - theptip 2y ago
    The more/smaller microservices you have, the more frequently your APIs must change. It’s more fruitful to recognize that this is a dimension of freedom rather than a binary decision.
    - whateveracct 2y ago
      I mean I recommend that you don't go micro right away. But a few well-placed service boundaries that align with your eng org chart pay dividends and help build discipline and rigor. Even if the services are all running on the same box to start and just IPCing over localhost or w/e. I prefer my teams build habits while it's easy and fun rather than once the monolith gets unwieldy and drawing boundaries becomes painful.

qaq 2y ago

A bit tangential but why is CloudFormation so slowww?

justin_oaks 2y ago
I figure it's because AWS can get away with it.
- shepherdjerred 2y ago
  AWS deploys using cfn internally
  - bobnamob 2y ago
    This is only partly true. The foundational services do not use cfn for the vast majority of their deployments.
Aeolun 2y ago
The reason by boss tends to give is that it’s made by AWS, so it cannot possibly be bad. Also, it’s free. Which is never given as anything more than a tangentially related reason, but…
- Uehreka 2y ago
  It… definitely isn’t free. Have you ever looked at the “Config” category of your AWS bill?
hk1337 2y ago
This is just anecdotal but I have found anytime a network interface is involved, it can slow down the deployment. I had a case where I was deleting lambdas in a VPC, and connected to EFS, that the deployment was rather quick but it took ~20 minutes for cloudformation to cleanup and finish.
motorest 2y ago
> A bit tangential but why is CloudFormation so slowww? It's not that CloudFormation is slow. It's that the whole concept of infrastructure-as code-as-codd is slow by nature. Each time you deploy a change to a state as a transaction, you need to assert preconditions and post-conditions at each step. If you have to roll out a set of changes that have any semblance of interdependence, you have no option other than to deploy each change as sequential steps. Each step requires many network calls to apply changes, go through auth, poll state, each one taking somewhere between 50-200ms. That quickly adds up. If you deploy the same app on a different cloud provider with Terraform or Ansible, you get the same result. If you deploy the same changes manually you turn a few minutes into a day-long ordeal. The biggest problem with IaC is that it is so high-level and does so much under the hood that some people have no idea what changes they are actually applying or what they are doing. Then they complain it takes so long.

khana 2y ago

[dead]

jojobas 2y ago

Fast deployment causes incident war rooms.

DougBTX 2y ago
Maybe the opposite, slow rollbacks cause escalating incidents.
Trasmatta 2y ago
In my experience, there's very little correlation. I've been on projects with 1 deployment every six weeks, and there were just as many production incidents as projects with daily deployments.
boxed 2y ago
I was on a team that went from every 3 weeks to multiple times per day. The number of incidents in production dropped drastically. But much more important than that drop, was that when things went wrong is was MUCH MUCH faster to find the problem. It was also much safer and easier to roll back, since there were so few changes that would be rolled back. No one wants to back off 3 weeks of work. That's chaos.
wussboy 2y ago
That is the opposite of my experience. Slow deploys mean bigger deploys mean more complexity going live mean more nervousness and more testing mean more hesitation mean more chance that something unforeseen mean errors that no one understands mean war rooms.
wasmitnetzen 2y ago
Yeah, and slow ones as well.

sourceless 2y ago

I think unfortunately the conclusion here is a bit backwards; de-risking deployments by improving testing and organisational properties is important, but is not the only approach that works. The author notes that there appears to be a fixed number of changes per deployment and that it is hard to increase - I think the 'Reversie Thinkie' here (as the author puts it) is actually to decrease the number of changes per deployment. The reason those meetings exist is because of risk! The more changes in a deployment, the higher the risk that one of them is going to introduce a bug or operational issue. By deploying small changes often, you get deliver value much sooner and fail smaller. Combine this with techniques such as canarying and gradual rollout, and you enter a world where deployments are no longer flipping a switch and either breaking or not breaking - you get to turn outages into degradations. This approach is corroborated by the DORA research[0], and covered well in Accelerate[1]. It also features centrally in The Phoenix Project[2] and its spiritual ancestor, The Goal[3]. [0] https://dora.dev/ https://dora.dev/ [1] https://www.amazon.co.uk/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339 https://www.amazon.co.uk/Accelerate-Software-Performing-Tech... [2] https://www.amazon.co.uk/Phoenix-Project-Helping-Business-Anniversary/dp/B00VBEBRK6 https://www.amazon.co.uk/Phoenix-Project-Helping-Business-An... [3] https://www.amazon.co.uk/Goal-Process-Ongoing-Improvement/dp/0566086654 https://www.amazon.co.uk/Goal-Process-Ongoing-Improvement/dp...

tomxor 2y ago
I tend to agree. Whenever I've removed artificial technical friction, or made a fundamental change to an approach, the processes that grew around them tend to evaporate, and not be replaced. I think many of these processes are a rational albeit non-technical response to making the best of a bad situation in the absence of a more fundamental solution. But that doesn't mean they are entirely harmless. I've come across some scenarios where the people driving decisions continued to reach for human processes as the solution rather than a workaround, for both new projects and projects designated specifically to remove existing inefficiencies. They either lacked the technical imagination, or were too stuck in the existing framing of the problem, and this is where people who do have that imagination need to speak up and point out that human processes need to be minimised with technical changes where possible. Not all human processes can be obviated through technical changes, but we don't want to spread ourselves thin on unnecessary ones.

andy_ppp 2y ago

The organisation will actively prevent you from trying to improve deployments though, they will say things like “Jenkins shouldn’t be near production” or “we can’t possibly put things live without QA being involved” or “we need this time to make sure the quality of the software is high enough”. All with a straight face while having millions of production bugs and a product that barely meets any user requirements (if there are any). In the end fighting the bureaucracy is actually impossible in most organisations, especially if you’re not part of the 200 layers of management that create these meetings. I would sack everyone but programmers and maybe two designers and let everyone fight it out without any agile coaches and product owners and scrum master and product experts. Slow deployment is a problem but it’s not the problem.

gleenn 2y ago
You sound very defeatist about fighting bureaucracy. If you work at an org with too much management, you can slowly push to move it in the direction you hope for or leave. If you keep ending up at places that seem impossible to change, perhaps you should ask more questions about this during the interview. I've worked at many small companies where there wasn't crazy bureaucracy because that's definitely what I preferred. I also currently work at a megacorp and yes there is difficulty, but being consistent and persuasive has lead to many things slowly heading in the right direction. Things take time. You have to realize why people have made things some way and then find convincing arguments to make things better. Sometimes places do just suck so don't stick around. But being hopeless doesn't seem helpful.
- mewpmewp2 2y ago
  My issue is that it just pays more. I have no similar paying option that would have good experience in that sense.
gavmor 2y ago
> Jenkins shouldn’t be near production > we can’t possibly put things live without QA being involved > we need this time to make sure the quality of the software is high enough I've only developed software professionally since 2012, but in that time not only have I never encountered such sentiments, but (and, perhaps, because) it has always been a top priority of leadership to emphatically insist on the very opposite: day one of any initiative is Jenkins to production—often directly via trunk-based development—and quality is every developer's responsibility. At the IC level, there was no "fighting bureaucracy," although I don't doubt leadership debated these things vigorously, from time to time, especially as external partners and stakeholders were often intimately involved. > I would sack everyone but programmers and maybe two designers and let everyone fight it out That works for me! But it doesn't scale. We definitely have to keep at least one product "owner" or "expert" or "manager" to enqueue stakeholder priorities and, while this can be a "hat" that devs and designers trade off, it's also a skill at which some individuals uniquely excel. All that being said, I don't want to come across as pearl-clutching, shocked Pikachu face about this. I understand that many organizations don't operate this way. The way I've helped firms make this change is via the introduction of a single, experimental team of volunteers dedicated to these practices—one protected (but not dictated to) by a mandate from on high. But, then again, this is California.

austin-cheney 2y ago

While this is mostly correct it’s also just as irrelevant. TLDR; software performance, thus human performance, is all that matters. Risk management/acceptance can be measured with numbers. In software this is actually far more straightforward than in many other careers, because software engineers can only accept risk within the restrictions of their known operating constraints and everything else is deferred. If you want to go faster you need to maximize the frequency of human iteration above absolutely everything else. If a person cannot iterate, such as waiting on permissions, they are blocked. If they are waiting on a build or screen refresh they are slowed. This can also be measured with numbers. If person A can iterate 100x faster than person B correctness becomes irrelevant. Person B must maximize upon correctness because they are slow. To be faster and more correct person A has extreme flexibility to learn, fail, and improve beyond what person B can deliver. Part of iterating faster AND reducing risk is fast test automation. If person A can execute 90+% test coverage in time of 4 human iterations then that test automation is still 25x faster than one person B iteration with a 90+% lower risk of regression.

black_13 2y ago

[dead]

vegetablepotpie 2y ago

I have personal experience with this in my professional career. Before Christmas break I had a big change, and there was fear. My org responded by increasing testing (regression testing, which increased overhead). This increased the risk that changes on dev would break changes on my branch (not a code merging way, but in a complex adaptive system way). I responded to this risk by making a meeting. I presented our project schedule, and told my colleagues about their expectations, I.e. if they drop code style comments on the PRs they will be deferred to a future PR (and then ignored and never done). What we needed is fine grained testing with better isolation between components. The problem is is that our management is at a high level, they don’t see meetings as a means to an end, they see meetings as a worthy goal in and of itself self to achieve. More meetings means more collaboration, means good. I’d love to see advice on how to lead technical changes with non-technical management.

lifeisstillgood 2y ago

I am trying to expound a concept I call “software literacy” - where a business can be run via code just as much as today a company can be run by English words (policy documents, emails etc). This leads to a few corollaries - things like “If GPUs do the work then coders are the new managers” or we need whole-org-test-rigs to be clear about the impacts of chnages. This seems directly related to this excellent article - to my mind if all the decision makers are not looking at the code as the first class object in a chnage process (is opposed to Jiras or project plans) then not all decision makers are (software) literate - and this comes up a lot in the threads here (“how do I discuss with non-technical management”) - the answer is you cannot - that management must be changed. This is an enormous generational road block that I thought was a problem thirty years ago but naively assumed would disappear as coders grew up. Of course the problem is that to “run” a company one does not need to code - so until not coding is something embarrassing like not writing is for a newspaper editor we won’t get past it. The main point is that we need companies that can be run with the new set of self-reinforcing concepts - sops, testing, not meetings but systems as communication. I will try and rewrite this comment later - it needs work

gavmor 2y ago
You had me at "whole org test" harness. This is a very, very interesting idea. Especially in conjunction with the concept of corporation as "slow AI" that I don't hear referenced often enough. I don't see why you call it "literacy," though. I think Maturana & Varela's term "autopoiesis" more closely orbits the kernel, and I'll bet Stafford Beer's Autopoietic Systems would contribute to a good intellectual foundation. At a certain point, though, I wonder if a purely software "business" doesn't just look like... SaaS?
fduran 2y ago
I've called that "Organization as Code" some years back :-)

braza 2y ago

A marginally related point but I do not know if others faced the following situation: I worked in a place with a CI pipeline room ~25 minutes with the unit/integration tests (3000+) taking 18 minutes. When something happens in production we ended up placing more tests; and of course when things goes south at least 50 minutes were necessary to recover. After a lot of consideration we decided to focus on the recovery and relax and simply some tests and focus on recovery (i.e. have the full thing in less than 5 minutes) combined with a canary as deployment strategy (instead rolling updates). At least for us was a so refreshing experience but sounded wrong in some ways.

wussboy 2y ago
I’ve often said that it is the speed of deployment that matters. If it takes you 50 minutes to deploy, it takes you 50 minutes to fix a problem. If it takes you 50 seconds to deploy, it takes you 50 seconds to fix a problem. Of course all kinds of things are rolled up in that speed to deploy, but almost all of them are good.

tpoacher 2y ago

Meetings (used right) are a great tool, in the same sense that project planners (used right) are a great tool. But then there's Jira. /s