11 ms·

Development slowness in big and legacy applications

110 points by olvy0 3y ago

devjab 3y ago
I stated my carer developing systems in an enterprise setting but have since moved into the wonderful space of companies transitioning from startup to enterprise. So I’ve worked a lot in this space, and while I’m not sure that I necessarily agree that old complex software is unmanageable legacy, there is a lot of truths in this article. In my experience software doesn’t really have to be “legacy” to encounter these issues. In fact a lot of 30+ year old systems are both better and more maintainable than things developed within the past 5 years. Which is probably an interesting case on its own, but let’s keep on topic! The way I like to handle these systems is to slowly transition things into “microservices”. I put in in quotes because it doesn’t necessarily have to be what many think of when they read the word “microservices”, and in truth they aren’t at all since they often build upon complex shared data models with all the “fun” that comes with that. Eventually I like to build things in to actual microservices, but it’s often a loooong if not impossible struggle to get there within most organisations which view IT as a cost center on par with HR but without the communication charms. Anyway, what I like to do is to separate complex business logic into their own “microservices” so that anything related to X lives in a single smallish services instead of a big complex system. One which is example is how we’ve got a “budget upload and computation”, a ”liquidity (not entirely sure that’s the correct English word)”, and, a “bank account” service all of which used to be part of this huge complex system, but are now separated completely in terms of the responsibility for their particular businesses logic. Eventually this breaks complex systems into a ton of minor services, which comes with its own set of issues and is obviously against a lot of what you’re taught in academic computer science because often you’ll have similar code in multiple systems, but it’s sooo much more maintainable and it’s so much easier to move fast and implement what the particular teams in an enterprise organisation needs for their particular business logic. In my example, we’ve gone from five teams having to wait literal months for changes to being able to implement them within a week, and often the same day they are requested. We’ve also lowered our incident reports on problems from less than 60% to above 90% each month. I’m not sure why we call it lowered when higher is better, but whatever.
- WJW 3y ago
  > In fact a lot of 30+ year old systems are both better and more maintainable than things developed within the past 5 years. Just would be true just through survivorship bias, no? The systems that were already unmaintainable 30 years ago could not be maintained and either fell over or got replaced, while the maintainable systems from 30 years ago are the only ones left.
  - VBprogrammer 3y ago
    I don't think I've ever seen a system fall over and be replaced. Systems don't fail and get replaced - companies do. Systems at profitable companies keep on chugging along. Occasionally someone suggests a major rewrite but that usually fails when it doesn't reach feature parity, nevermind meet the expectations of all the new features which were thrown at it to justify the expenditure.
rtpg 3y ago
Getting the time from “everyone agrees the change looks good” to “it’s in front of users” down is so satisfying. Work that goes into reducing this tends to have wonderful knock on effects all over, from faster builds, quicker rollbacks, smaller CI bills (yes it’s possible!). The biggest difficulty I’ve seen in practice is going from “somebody hits the button” to “thing happens”. But even just making sure that “somebody hits the button” is a 1 button thing instead of a 20 button thing can be a major win! Totally valuable work On the other side of the coin: having good observability into prod (talking mainly about APM stuff and tracing) gives devs way more confidence in the changes they’re making as well. Make it easy to deploy stuff, but also make it easy for systems to log things, and make it easy for developers to poke into those logs (my favorite flavor of this right now is honeycomb, but mainly cuz you can do perf analysis and answer questions from PMs about usage quite easily) The nice thing about all of this if you’re a mid sized team is that you can totally outsource this with the right kind of person. Somebody who comes in, has the right to write some automation code (maybe it’s Zapier even!), and is not part of office infighting about process. The mandate being “make this pipeline smoother”. Probably among the easier problems to fix compared to deeper business or product problems. And it’s also super visible to other teams (“wow the bug is already fixed?”) EDIT: one thing I’ve seen done that “never works” (I have not seen everything) is splitting up a repo into smaller repos. If you find yourself thinking “with smaller repos things could be smoother” it’s likely that what you want is not multiple repos but a build tool like Bazel or Buck, that will let you describe multiple projects in one repo, without causing inter-repo friction come review time. Integrating these tools can save you loads of money on CI and enforce decent test suite separation. The unfortunate flip side is that at least Bazel is very idiosyncratic, and most other tools are based on Bazel. I have heard good things about Buck 2 but have yet to see it in prod
- kridsdale1 3y ago
  At Apple, we did indeed use many small repos. Some were even in different SCM systems! It was indeed a nightmare to integrate with “everything on main is broken, hold off on pulling” being a common multi day experience. At Google we use Blaze which is Bazel and it works very well, as you say. At Meta (where I was before Google) we used Buck2 and it was the best. Much faster and more laptop-friendly than Blaze. It was like Christmas when they switched us over from Buck1. Meta is very very good at “20,000 contributors to this repo”. Google is about 85% as good. Getting work done (especially on mobile apps that take 7 hours to build from scratch) at that scale without these tools would make me jump off the building.
The_Colonel 3y ago
That's a good summary, I agree with basically everything. I think a lot of it boils down to strong engineering, which is not getting mauled into submission by product / business. Strong engineering will naturally gravitate to optimizing long processes. The other big factor which has many consequences is the overall organization culture - are people encouraged to be helpful, reach out to other teams without organizing meetings in the calendar? Is there a blaming/finger pointing happening? Are many disputes largely about ego, dominance, politics? The "Meetings" section might be controversial, but I agree as well. There's just no getting away from lots of communication/alignment in large organizations, and synchronous/low latency communication should be encouraged. Yes, that means disruptions, but being on the same page/unblocking others is usually more important. The amount of lines of code produced is not the largest cause of productivity loss in the large organizations.
- VBprogrammer 3y ago
  > The "Meetings" section might be controversial I can't really say I found it all that controversial. If I wanted to be controversial I'd say that most recurring meetings and any meeting with more than 5 people are a waste of time. I quite often ask someone who messages me out of the blue with a problem to book time in my calendar (which is usually at least 50% free). Because having a specific time to look at something is usually much better than me either dropping what I'm doing to think about something else or forgetting about their message and getting chased by them 3 days later.
  - The_Colonel 3y ago
    > Because having a specific time to look at something is usually much better than me either dropping what I'm doing to think about something else There's no hard and fast rule, but if you can help them out in 10 minutes, then it's much healthier for the culture/organization to unblock them immediately (within let's say 30 minutes) than ask them to organize a meeting.
  - Silhouette 3y ago
    If I wanted to be controversial I'd say that most recurring meetings and any meeting with more than 5 people are a waste of time. You're probably right but it's sad that this is regarded as controversial rather than self-evident. Maybe any meeting with more than 5 people being a waste of time is a stretch and most again would have been a more defensible claim? But the spirit is right. Good meetings happen when there is a good reason to have a meeting. Recurring meetings are often symptoms of other management problems that have better solutions available.
drra 3y ago
It all boils down to unmanaged technical debt that can have many forms - code, standards, architecture, documentation, tooling, compliance. There are two major indicators of tech debt - slow development pace and high amount of unexpected bugs/problems. Big projects have big capacities for such debt so it's visibility is also higher. Replacing "legacy" solution with a shiny new thing is declaring bankruptcy and starting fresh - it does not change the cause of poor tech debt management.
carlmr 3y ago
>Long build times This is one of the major ones for me. It's a multiplier for everything else. local builds, CI builds, feedback cycles. If you get your build times down, everything else gets faster. Migrating to Bazel was a godsend, while Bazel isn't perfect, it's one of the best build systems for large C++ (or mixed) code bases I've seen. buck2 looks promising, too, in this regard.
- kridsdale1 3y ago
  Having used both I vote for Buck2 being a bit better in dev experience for performance, resource usage, and sensible config language.
  - carlmr 3y ago
    I'm intrigued by Buck2, do you have good resources on learning it? Last time I checked it was still a bit early and I had issues finding out how to migrate our Bazel build to Buck 2. The single phase build is amazing as a concept.
    - kridsdale1 3y ago
      It was an internal (not yet open source) tool when I used it, so the docs and people behind it were much more available to me than it would be now as a public project.
      - carlmr 3y ago
        Ah, gotcha. I tried installing it and the first thing it gave me was an error message. I googled it, couldn't find the answer. Got an answer now, it worked for a small toy example at least. But I haven't gotten around to try it out further. I feel like the dev experience - without internal help for initial setup - isn't quite there yet, especially regarding error messages and documentation.
danielovichdk 3y ago
A big pile of all things of kinds of things. Big Project Complexities - has nothing to do with software business. It's an organization challenge which has been known way before anyone wrote anything remotely complex with software. See planning fallacy, optimism bias, read about how car and structural companies has done it. Look up "GM, Toyota and NUMMI" as an example. Cross Team Dependencies. Again, this is an organizational challenge. Cognitive loads, poor communication structures based on industrial org designs and army-based leadership with control and command. Hierachies etc. Read Fred Brooks and Mel Conway on these for a more specific software approach. Meetings, meetings, meetings. Of course there is a "going around it" solution for this. And it's called a manager. No manager should allow any team member to attend meetings if no thourough agenda is available, no actions for attendees, no proposed actions afterwards and someone who controls the meeting. Meetings will then turn into writing instead of verbal.
- ponector 3y ago
  Great! Let's schedule a meeting to discuss your thoughts!
- punnerud 3y ago
  There is also a different way around this; Skip all the meeting and ship code into production, and iterate into good solutions to complex problems based on feedback
  - pjc50 3y ago
    > Skip all the meeting and ship code into production This works until you have an outage, disaster, or embarrasment of some kind, at which point everyone is banned from doing that without approval. This leads to a kind of "trauma driven development"; when you suggest a change, you see a flash of fear in the eyes of people around you. If you punish people for improvements often enough, you don't get any improvements.
    - hef19898 3y ago
      Imagine a surgeon would work like this: no prior diagnostics, no procedures to follow, just cut people open and take it from there! Honestly, you shouldn't deploy into production without testing. Development and deployment needs to be tracked and scheduled. Because worst case, it has to be reversible. If you start deploying directly, without coordination with others (skip all the meetings, right?), you are begging for desaster.
pjc50 3y ago
> Encouraging a culture of helpfulness is, well, helpful. If you incentivize developers to assist each other, they will do so. On the other hand, if you encourage a culture of competitiveness, they might feel less likely to help. They might even sabotage each other’s efforts. Yes! "Stack ranking" and inter-departmental competition and all these other dog-eat-dog things turn your employees against each other. People will waste unlimited amounts of company money to prevent another team improving their situation. > One idea to encourage helpfulness is to include it in a company’s mission statement. Or better yet, add helpfulness to every employee’s performance review goals. If money depends on it, you can be sure people will make an effort. Noooooo! You've gamified a metric! You know what happens when you gamify a metric? You destroy its value as a metric. And encourage people to cheat it, again. You've made money contingent on appearing helpful, not on actually being helpful. This is a great way to get interdepartmental meetings that achieve nothing but that everyone can cite on their "helpfulness score".
thraxil 3y ago
Good article. I highly recommend the book "Kill it with Fire: Manage Aging Computer Systems (And Future Proof Modern Ones)" by Marianne Bellotti if you've ever had to deal with these kinds of legacy systems.
pydry 3y ago
One thing that has always puzzled me is how many managers are seemingly less uninterested in low hanging fruit which would speed up development and more interested in creating metrics to measure the slowness.
- hamandcheese 3y ago
  How do you know it's low-hanging fruit if you can't measure the impact?
  - pydry 3y ago
    Ask your developers who can feel the impact.
- xxpor 3y ago
  How does a manager argue for funding to make improvements without some sort of metric? You can't just say it's self evident, because the people making these decisions haven't complied code in years.
  - pydry 3y ago
    Low hanging means you dont need budgetary approval.
  - ath3nd 3y ago
    I find it both puzzling but also sad that we as managers have to argue with somebody for giving us the time and space to make improvements. Who is that parental figure? Why have we reached this state as an industry? Why do we have to beg for time and space just so we can do our job right? To me, the problem is in today's management methodologies there are too many people loosely related to software who nevertheless participate in the process of software development. Scrum master bothering my teams with some rituals that loosely relate to software but are actually tailor fit for a Japanese factory floor from the 80's, and have a weird cult-y smell? Out of the door with scrum. Product owner wasting the time of my teams with childish poker card games and refinements and sprints and calling the shots on priority, considering they are the least technically qualified in the team? Why do we give the decision what to build to the least knowledgeable person? This is insane! Out of the door with this guy. Agile coach (yes, that exists) just talking and talking about how this is not the right flavor of scrum? Out with this guy also. Managers or sales attending the dev meetings trying to pressure them into delivering something faster? Throw them out and very sternly explain to them to stop with that, things will be done when they are done. But wait, how do developers then know what the business wants, you ask? Well, I am glad you brought it up. Engineers should decide what's the best way to build something awesome. They can do that by requesting feedback from sales or business representatives, talk to clients, view usage metrics, whatever. Whatever it is, it's their business and responsibility how to do it, what features to develop and how long each feature takes. They should also damn make sure the product is stable, free of bugs and reliable. Nobody else should have a vote in making engineering and build decisions. That's it. End of story. C łevel dudes better take a note of that, cause me and my teams, we are paid to build an awesome product, so leave that to us and don't waste our time trying to interfere. If anyone who is not an engineer has objections, I don't care. I don't go to your sales meetings (those things are noisy and chaotic anyway) and teach you how to do your sales magic, right? I don't make you play poker cards trying to squeeze out of you how much time till you make a sale, right? And I damn right don't tell you what client to focus on and that each client has to be acquired in a 2 week period, or you somehow failed the sprint. And who can sprint for two weeks, no rest, and sprint again? That's just harmful bollocks. So why should I accept a non engineer having any say of how and what engineers are building? I am at a fairly high executive position in a scale up, but that has been my approach and line of thinking for all my career. Remove anyone not engineer from having any say on what engineering does and you will build good, stable, reliable and sellable software. Even the guys who pay our salaries, they hired you not to execute their instructions like a trained monkey but to use your skills and expertise to do it yourself. So if you find that some management types are sniffing around you and your teams trying to make you make too many compromises, remind them that road leads to you using only 5% of your potential, and if they don't listen, find another place to work. We can build good software if we focus on that and push away all constraints trying to make us do otherwise.
datadrivenangel 3y ago
It all comes back around to having cross-functional teams that can act without dependencies.
cedws 3y ago
One of the most damaging things for productivity is not being able to run the stack locally. This generally happens due to laziness and corner cutting. If developers have to commit and wait for a build + test + deploy pipeline to see if their code works, that should be priority #1 to fix.
- pyjarrett 3y ago
  > laziness and corner cutting My experience is that it just isn't prioritized, and you have to do it early to enforce good engineering solutions to keep it this way. It's much easier to keep something locally runnable than try to set it up that way six months into a project. If you can't run the stack locally, it makes it harder to set up performance and acceptance testing later. Running locally also makes long build times more noticeable and it's easier to address these issues earlier. Being able to run locally looks low value at first, but if you do it, and sustain it, you get extremely large project-wide benefits later.
  - icedchai 3y ago
    I took a job at a place that was into "serverless". That's fine, but there was no way to run anything locally due to how the application was built (wrong framework, no framework, etc.) Also, they failed to give each developer their own environment, so we had people editing the same code in the AWS console. I've never seen anything like it. Everywhere else I worked prioritized local first.
  - PH95VuimJjqBqy 3y ago
    > My experience is that it just isn't prioritized my experience as well and it blows my fucking mind every time I see it. It tells me developers have _no clue_ what it is they're doing that is helping or hindering them.
- mrkeen 3y ago
  "Can you actually run your software on your laptop?" has become one of my interview questions for that reason. I think a lot of dev shops just genuinely don't realise it can be done. I've never been in a startup right at the beginning, but I don't understand how a stack would ever be allowed to transition away from being locally-runnable.
highfrequency 3y ago
”Features are often added with minimal effort and without refactoring the system to make the abstractions reflect the new feature set. So these additions, hacks, and patches are added on top of each other until the codebase becomes a big ball of mud… When a developer faces the choice of completing a task in 1 day with a patch versus asking for two additional weeks to refactor, the result will usually be the first. Multiply that by 1000 over five years, and you’ll get a typical enterprise software project.” Well put!

110 points by olvy0 3y ago

I stated my carer developing systems in an enterprise setting but have since moved into the wonderful space of companies transitioning from startup to enterprise. So I’ve worked a lot in this space, and while I’m not sure that I necessarily agree that old complex software is unmanageable legacy, there is a lot of truths in this article. In my experience software doesn’t really have to be “legacy” to encounter these issues. In fact a lot of 30+ year old systems are both better and more maintainable than things developed within the past 5 years. Which is probably an interesting case on its own, but let’s keep on topic! The way I like to handle these systems is to slowly transition things into “microservices”. I put in in quotes because it doesn’t necessarily have to be what many think of when they read the word “microservices”, and in truth they aren’t at all since they often build upon complex shared data models with all the “fun” that comes with that. Eventually I like to build things in to actual microservices, but it’s often a loooong if not impossible struggle to get there within most organisations which view IT as a cost center on par with HR but without the communication charms. Anyway, what I like to do is to separate complex business logic into their own “microservices” so that anything related to X lives in a single smallish services instead of a big complex system. One which is example is how we’ve got a “budget upload and computation”, a ”liquidity (not entirely sure that’s the correct English word)”, and, a “bank account” service all of which used to be part of this huge complex system, but are now separated completely in terms of the responsibility for their particular businesses logic. Eventually this breaks complex systems into a ton of minor services, which comes with its own set of issues and is obviously against a lot of what you’re taught in academic computer science because often you’ll have similar code in multiple systems, but it’s sooo much more maintainable and it’s so much easier to move fast and implement what the particular teams in an enterprise organisation needs for their particular business logic. In my example, we’ve gone from five teams having to wait literal months for changes to being able to implement them within a week, and often the same day they are requested. We’ve also lowered our incident reports on problems from less than 60% to above 90% each month. I’m not sure why we call it lowered when higher is better, but whatever.

WJW 3y ago
> In fact a lot of 30+ year old systems are both better and more maintainable than things developed within the past 5 years. Just would be true just through survivorship bias, no? The systems that were already unmaintainable 30 years ago could not be maintained and either fell over or got replaced, while the maintainable systems from 30 years ago are the only ones left.
- VBprogrammer 3y ago
  I don't think I've ever seen a system fall over and be replaced. Systems don't fail and get replaced - companies do. Systems at profitable companies keep on chugging along. Occasionally someone suggests a major rewrite but that usually fails when it doesn't reach feature parity, nevermind meet the expectations of all the new features which were thrown at it to justify the expenditure.

rtpg 3y ago

Getting the time from “everyone agrees the change looks good” to “it’s in front of users” down is so satisfying. Work that goes into reducing this tends to have wonderful knock on effects all over, from faster builds, quicker rollbacks, smaller CI bills (yes it’s possible!). The biggest difficulty I’ve seen in practice is going from “somebody hits the button” to “thing happens”. But even just making sure that “somebody hits the button” is a 1 button thing instead of a 20 button thing can be a major win! Totally valuable work On the other side of the coin: having good observability into prod (talking mainly about APM stuff and tracing) gives devs way more confidence in the changes they’re making as well. Make it easy to deploy stuff, but also make it easy for systems to log things, and make it easy for developers to poke into those logs (my favorite flavor of this right now is honeycomb, but mainly cuz you can do perf analysis and answer questions from PMs about usage quite easily) The nice thing about all of this if you’re a mid sized team is that you can totally outsource this with the right kind of person. Somebody who comes in, has the right to write some automation code (maybe it’s Zapier even!), and is not part of office infighting about process. The mandate being “make this pipeline smoother”. Probably among the easier problems to fix compared to deeper business or product problems. And it’s also super visible to other teams (“wow the bug is already fixed?”) EDIT: one thing I’ve seen done that “never works” (I have not seen everything) is splitting up a repo into smaller repos. If you find yourself thinking “with smaller repos things could be smoother” it’s likely that what you want is not multiple repos but a build tool like Bazel or Buck, that will let you describe multiple projects in one repo, without causing inter-repo friction come review time. Integrating these tools can save you loads of money on CI and enforce decent test suite separation. The unfortunate flip side is that at least Bazel is very idiosyncratic, and most other tools are based on Bazel. I have heard good things about Buck 2 but have yet to see it in prod

kridsdale1 3y ago
At Apple, we did indeed use many small repos. Some were even in different SCM systems! It was indeed a nightmare to integrate with “everything on main is broken, hold off on pulling” being a common multi day experience. At Google we use Blaze which is Bazel and it works very well, as you say. At Meta (where I was before Google) we used Buck2 and it was the best. Much faster and more laptop-friendly than Blaze. It was like Christmas when they switched us over from Buck1. Meta is very very good at “20,000 contributors to this repo”. Google is about 85% as good. Getting work done (especially on mobile apps that take 7 hours to build from scratch) at that scale without these tools would make me jump off the building.

The_Colonel 3y ago

That's a good summary, I agree with basically everything. I think a lot of it boils down to strong engineering, which is not getting mauled into submission by product / business. Strong engineering will naturally gravitate to optimizing long processes. The other big factor which has many consequences is the overall organization culture - are people encouraged to be helpful, reach out to other teams without organizing meetings in the calendar? Is there a blaming/finger pointing happening? Are many disputes largely about ego, dominance, politics? The "Meetings" section might be controversial, but I agree as well. There's just no getting away from lots of communication/alignment in large organizations, and synchronous/low latency communication should be encouraged. Yes, that means disruptions, but being on the same page/unblocking others is usually more important. The amount of lines of code produced is not the largest cause of productivity loss in the large organizations.

VBprogrammer 3y ago
> The "Meetings" section might be controversial I can't really say I found it all that controversial. If I wanted to be controversial I'd say that most recurring meetings and any meeting with more than 5 people are a waste of time. I quite often ask someone who messages me out of the blue with a problem to book time in my calendar (which is usually at least 50% free). Because having a specific time to look at something is usually much better than me either dropping what I'm doing to think about something else or forgetting about their message and getting chased by them 3 days later.
- The_Colonel 3y ago
  > Because having a specific time to look at something is usually much better than me either dropping what I'm doing to think about something else There's no hard and fast rule, but if you can help them out in 10 minutes, then it's much healthier for the culture/organization to unblock them immediately (within let's say 30 minutes) than ask them to organize a meeting.
- Silhouette 3y ago
  If I wanted to be controversial I'd say that most recurring meetings and any meeting with more than 5 people are a waste of time. You're probably right but it's sad that this is regarded as controversial rather than self-evident. Maybe any meeting with more than 5 people being a waste of time is a stretch and most again would have been a more defensible claim? But the spirit is right. Good meetings happen when there is a good reason to have a meeting. Recurring meetings are often symptoms of other management problems that have better solutions available.

drra 3y ago

It all boils down to unmanaged technical debt that can have many forms - code, standards, architecture, documentation, tooling, compliance. There are two major indicators of tech debt - slow development pace and high amount of unexpected bugs/problems. Big projects have big capacities for such debt so it's visibility is also higher. Replacing "legacy" solution with a shiny new thing is declaring bankruptcy and starting fresh - it does not change the cause of poor tech debt management.

carlmr 3y ago

>Long build times This is one of the major ones for me. It's a multiplier for everything else. local builds, CI builds, feedback cycles. If you get your build times down, everything else gets faster. Migrating to Bazel was a godsend, while Bazel isn't perfect, it's one of the best build systems for large C++ (or mixed) code bases I've seen. buck2 looks promising, too, in this regard.

kridsdale1 3y ago
Having used both I vote for Buck2 being a bit better in dev experience for performance, resource usage, and sensible config language.
- carlmr 3y ago
  I'm intrigued by Buck2, do you have good resources on learning it? Last time I checked it was still a bit early and I had issues finding out how to migrate our Bazel build to Buck 2. The single phase build is amazing as a concept.
  - kridsdale1 3y ago
    It was an internal (not yet open source) tool when I used it, so the docs and people behind it were much more available to me than it would be now as a public project.
    - carlmr 3y ago
      Ah, gotcha. I tried installing it and the first thing it gave me was an error message. I googled it, couldn't find the answer. Got an answer now, it worked for a small toy example at least. But I haven't gotten around to try it out further. I feel like the dev experience - without internal help for initial setup - isn't quite there yet, especially regarding error messages and documentation.

danielovichdk 3y ago

A big pile of all things of kinds of things. Big Project Complexities - has nothing to do with software business. It's an organization challenge which has been known way before anyone wrote anything remotely complex with software. See planning fallacy, optimism bias, read about how car and structural companies has done it. Look up "GM, Toyota and NUMMI" as an example. Cross Team Dependencies. Again, this is an organizational challenge. Cognitive loads, poor communication structures based on industrial org designs and army-based leadership with control and command. Hierachies etc. Read Fred Brooks and Mel Conway on these for a more specific software approach. Meetings, meetings, meetings. Of course there is a "going around it" solution for this. And it's called a manager. No manager should allow any team member to attend meetings if no thourough agenda is available, no actions for attendees, no proposed actions afterwards and someone who controls the meeting. Meetings will then turn into writing instead of verbal.

ponector 3y ago
Great! Let's schedule a meeting to discuss your thoughts!
punnerud 3y ago
There is also a different way around this; Skip all the meeting and ship code into production, and iterate into good solutions to complex problems based on feedback
- pjc50 3y ago
  > Skip all the meeting and ship code into production This works until you have an outage, disaster, or embarrasment of some kind, at which point everyone is banned from doing that without approval. This leads to a kind of "trauma driven development"; when you suggest a change, you see a flash of fear in the eyes of people around you. If you punish people for improvements often enough, you don't get any improvements.
  - hef19898 3y ago
    Imagine a surgeon would work like this: no prior diagnostics, no procedures to follow, just cut people open and take it from there! Honestly, you shouldn't deploy into production without testing. Development and deployment needs to be tracked and scheduled. Because worst case, it has to be reversible. If you start deploying directly, without coordination with others (skip all the meetings, right?), you are begging for desaster.

pjc50 3y ago

> Encouraging a culture of helpfulness is, well, helpful. If you incentivize developers to assist each other, they will do so. On the other hand, if you encourage a culture of competitiveness, they might feel less likely to help. They might even sabotage each other’s efforts. Yes! "Stack ranking" and inter-departmental competition and all these other dog-eat-dog things turn your employees against each other. People will waste unlimited amounts of company money to prevent another team improving their situation. > One idea to encourage helpfulness is to include it in a company’s mission statement. Or better yet, add helpfulness to every employee’s performance review goals. If money depends on it, you can be sure people will make an effort. Noooooo! You've gamified a metric! You know what happens when you gamify a metric? You destroy its value as a metric. And encourage people to cheat it, again. You've made money contingent on appearing helpful, not on actually being helpful. This is a great way to get interdepartmental meetings that achieve nothing but that everyone can cite on their "helpfulness score".

thraxil 3y ago

Good article. I highly recommend the book "Kill it with Fire: Manage Aging Computer Systems (And Future Proof Modern Ones)" by Marianne Bellotti if you've ever had to deal with these kinds of legacy systems.

pydry 3y ago

One thing that has always puzzled me is how many managers are seemingly less uninterested in low hanging fruit which would speed up development and more interested in creating metrics to measure the slowness.

hamandcheese 3y ago
How do you know it's low-hanging fruit if you can't measure the impact?
- pydry 3y ago
  Ask your developers who can feel the impact.
xxpor 3y ago
How does a manager argue for funding to make improvements without some sort of metric? You can't just say it's self evident, because the people making these decisions haven't complied code in years.
- pydry 3y ago
  Low hanging means you dont need budgetary approval.
- ath3nd 3y ago
  I find it both puzzling but also sad that we as managers have to argue with somebody for giving us the time and space to make improvements. Who is that parental figure? Why have we reached this state as an industry? Why do we have to beg for time and space just so we can do our job right? To me, the problem is in today's management methodologies there are too many people loosely related to software who nevertheless participate in the process of software development. Scrum master bothering my teams with some rituals that loosely relate to software but are actually tailor fit for a Japanese factory floor from the 80's, and have a weird cult-y smell? Out of the door with scrum. Product owner wasting the time of my teams with childish poker card games and refinements and sprints and calling the shots on priority, considering they are the least technically qualified in the team? Why do we give the decision what to build to the least knowledgeable person? This is insane! Out of the door with this guy. Agile coach (yes, that exists) just talking and talking about how this is not the right flavor of scrum? Out with this guy also. Managers or sales attending the dev meetings trying to pressure them into delivering something faster? Throw them out and very sternly explain to them to stop with that, things will be done when they are done. But wait, how do developers then know what the business wants, you ask? Well, I am glad you brought it up. Engineers should decide what's the best way to build something awesome. They can do that by requesting feedback from sales or business representatives, talk to clients, view usage metrics, whatever. Whatever it is, it's their business and responsibility how to do it, what features to develop and how long each feature takes. They should also damn make sure the product is stable, free of bugs and reliable. Nobody else should have a vote in making engineering and build decisions. That's it. End of story. C łevel dudes better take a note of that, cause me and my teams, we are paid to build an awesome product, so leave that to us and don't waste our time trying to interfere. If anyone who is not an engineer has objections, I don't care. I don't go to your sales meetings (those things are noisy and chaotic anyway) and teach you how to do your sales magic, right? I don't make you play poker cards trying to squeeze out of you how much time till you make a sale, right? And I damn right don't tell you what client to focus on and that each client has to be acquired in a 2 week period, or you somehow failed the sprint. And who can sprint for two weeks, no rest, and sprint again? That's just harmful bollocks. So why should I accept a non engineer having any say of how and what engineers are building? I am at a fairly high executive position in a scale up, but that has been my approach and line of thinking for all my career. Remove anyone not engineer from having any say on what engineering does and you will build good, stable, reliable and sellable software. Even the guys who pay our salaries, they hired you not to execute their instructions like a trained monkey but to use your skills and expertise to do it yourself. So if you find that some management types are sniffing around you and your teams trying to make you make too many compromises, remind them that road leads to you using only 5% of your potential, and if they don't listen, find another place to work. We can build good software if we focus on that and push away all constraints trying to make us do otherwise.

datadrivenangel 3y ago

It all comes back around to having cross-functional teams that can act without dependencies.

cedws 3y ago

One of the most damaging things for productivity is not being able to run the stack locally. This generally happens due to laziness and corner cutting. If developers have to commit and wait for a build + test + deploy pipeline to see if their code works, that should be priority #1 to fix.

pyjarrett 3y ago
> laziness and corner cutting My experience is that it just isn't prioritized, and you have to do it early to enforce good engineering solutions to keep it this way. It's much easier to keep something locally runnable than try to set it up that way six months into a project. If you can't run the stack locally, it makes it harder to set up performance and acceptance testing later. Running locally also makes long build times more noticeable and it's easier to address these issues earlier. Being able to run locally looks low value at first, but if you do it, and sustain it, you get extremely large project-wide benefits later.
- icedchai 3y ago
  I took a job at a place that was into "serverless". That's fine, but there was no way to run anything locally due to how the application was built (wrong framework, no framework, etc.) Also, they failed to give each developer their own environment, so we had people editing the same code in the AWS console. I've never seen anything like it. Everywhere else I worked prioritized local first.
- PH95VuimJjqBqy 3y ago
  > My experience is that it just isn't prioritized my experience as well and it blows my fucking mind every time I see it. It tells me developers have _no clue_ what it is they're doing that is helping or hindering them.
mrkeen 3y ago
"Can you actually run your software on your laptop?" has become one of my interview questions for that reason. I think a lot of dev shops just genuinely don't realise it can be done. I've never been in a startup right at the beginning, but I don't understand how a stack would ever be allowed to transition away from being locally-runnable.

highfrequency 3y ago

”Features are often added with minimal effort and without refactoring the system to make the abstractions reflect the new feature set. So these additions, hacks, and patches are added on top of each other until the codebase becomes a big ball of mud… When a developer faces the choice of completing a task in 1 day with a patch versus asking for two additional weeks to refactor, the result will usually be the first. Multiply that by 1000 over five years, and you’ll get a typical enterprise software project.” Well put!