4 ms·

I've come to suspect the normal way large companies are organised makes performant software almost impossible. Features are celebrated and make money - whereas

I've come to suspect the normal way large companies are organised makes performant software almost impossible. Features are celebrated and make money - whereas performance can be gradually salami-sliced away, with nobody advocating for it. If you can add a feature and it adds just 50ms to a 300ms pageload - it's unlikely anyone is going to block its release on performance grounds. And once the feature is released, there's no removing it. This means performance gradually ratchets slower and slower - until it becomes intolerable. And by then, different performance issues will be everywhere in the codebase, so fixing them will seem an insurmountable challenge. Far cheaper to ignore the problem. "Jira is only slow because customers insist on creating tickets"

marginalia_nu 3y ago
There's a lot of overhead and inefficiencies in how large companies do anything. It's hard to organize thousands of people well. It's the social equivalent of the old 'Command-line Tools can be 235x Faster than your Hadoop Cluster'[1] conclusion. You put a hundred developers on a project and you'll generously get twice the meaningful output of ten developers (but 50 times the LOC; as demanded by conway's law). The upside is that this makes it all the more feasible for a small group or even single individual to be surprisingly competitive. Peak productivity per person is probably is probably found in a self-organized 1-3 person team. [1] https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
- mirekrusin 3y ago
  People take holidays, get sick and change jobs - having 1-3 ppl team means you'll hit brick wall that will hurt sooner or later, more so if at least one of devs was really good.
  - marginalia_nu 3y ago
    This is a real concern if you're looking at it as someone who is managing people... but a 1-3 person team does not need a manager to tell them what to do, and generally is not hired but spontaneously organized. It's an inherently more unpredictable way of working, but that definitely cuts both ways. There are definitely more risks, but also far greater payoffs in terms of what you can do working in this fashion. The principle of ensuring predictable velocity across a team comes a cost of limiting maximal produtivity amongst talented individuals.
  - Jensson 3y ago
    It is better to sometimes be hurt than always be hurting. Organizations are so afraid of only 1 person understanding something that they create processes so that 0 persons understand it.
- aleph_minus_one 3y ago
  > You put a hundred developers on a project and you'll generously get twice the meaningful output of ten developers (but 50 times the LOC; as demanded by conway's law). Based on my job experience, this is rather an artifact of how software projects are (badly?) managed. Give each of the developers some small "hotspots" to work on (i.e. critical code fragment that is - hard to implement correctly and fast - which is critical to implement very fast and correctly - a correct and intended implementation will likely consist of few highly sophisticated lines of code ), and the scaling will be much better. This is a little bit like theses are assigned to students: each thesis is one isolated non-trivial scientific challenge (often open research problem) that is to be solved by the student. I guess the "management incentives" nevertheless prevent such a method: managers prefer/are incentivised to "tell stories" instead of explaining the deep, sophisticated challenges and solutions in making great implementations of these "hotspots".
  - dasil003 3y ago
    This assumes you know what you need to build. 90% of the challenge in large scale development is nailing down requirements, especially when the problem space is too big for any individual to fully understand.
    - aleph_minus_one 3y ago
      > This assumes you know what you need to build. This is what in my opinion senior programmers or project managers are for. If they are not capable of handling this task decently, these people simply are not (yet) ready for this role.
      - falling_myshkin 3y ago
        Young engineer here in charge of a project and feeling quite out of their depth, I agree with this. Currently no mentorship and it will take a couple months for a senior hire. Do you have any advice? What does a senior engineer love/hate to see when they come onto a project started by engineers earlier in their career? How can I be most helpful?
ozim 3y ago
Fun part in context of big corporations is when performance becomes awful business will call in some guys from outside because clearly people currently working don’t know how to make it perform better. Business skips part where for years they insist on pushing features out. Where there most likely are tons of easy fixes that can speed up performance if you give the team time to focus on that. Going back to “outside help” then you get people who don’t understand current system and propose some silver bullet like throwing noSQL because that “will fix your issues” - but they don’t say cost is basically redoing the system, project goes under, third party guys count money, put another success story on CV because no one will validate it anyway…
lp4vn 3y ago
Development in any corp today, even middle-sized ones, is a big red tape festival. You have a project divided in sprints that are divided in (often meaningless) tasks, then you have a completely bloated CI/CD system and then things get deployed to three or four environments where people pretend to test everything. Things that might be developed by a journeyman developer in five days take six weeks to be done by a team of eight developers. The whole process today exists to support a big class of managerial people(including scrum masters, PO's and PM's) whose salary depends on not understanding that the process is completely broken.
- mirekrusin 3y ago
  On the bright side remote development means you can work 1-2h per day and make management clapping their hands in happiness.
- mjfisher 3y ago
  > The whole process today exists to support a big class of managerial people (including scrum masters, PO's and PM's) whose salary depends on not understanding that the process is completely broken. I think it's even worse than that - those unnecessary processes only exist to try to help management attempt to manage content they don't understand in the first place. And I say this as someone who regularly implements said delivery processes for customers.
- zozbot234 3y ago
  Wasn't "Agile" development supposed to fix this, by keeping team sizes small and workflows as simple and iterative as possible? AIUI, the fabled "two-pizzas team" size was intended as an upper bound for the most complex projects only, not as the new normal. (The "test everything" culture was also AIUI in the service of quick iteration; the first versions of Agile are from the early 2000s when dynamic languages were starting to get popular and there wasn't really any kind of static analysis.)
  - whstl 3y ago
    Yes. Agile depends on not having a “big class of managerial people” micromanaging [1] every move done by the squad. No methodology will work when someone is actively fucking up the process, and agile is no exception. [1] In before some smartass comes to say “so you don’t want managers”… I qualified my statement with the word “micromanaging”.
monkeydust 3y ago
Depends. In certain fields like trading software, performance is a feature.
Jensson 3y ago
At Google they argued over microseconds, but there microseconds are money. If every software engineer working on search or ads added 10 microseconds each to load times Google would become unbearably slow extremely quickly and no longer make money.
fransje26 3y ago
Goodbye, Atom editor..
saagarjha 3y ago
> If you can add a feature and it adds just 50ms to a 300ms pageload - it's unlikely anyone is going to block its release on performance grounds. I worked on a team that did this. It was easy to justify because the benefits of the feature were almost always counterbalanced by less usage of the product, and we had numbers to back it up.
NL807 3y ago
>If you can add a feature and it adds just 50ms to a 300ms pageload - it's unlikely anyone is going to block its release on performance grounds. And once the feature is released, there's no removing it. Unfortunately this is happening, because compute power is cheap these days. It allows programmers to be lazy, take short cuts and not think about good software architecture. There is no economic incentive to optimise, because human perception won't perceive a difference and thus nobody complains. It's wild to think that we have basically supercomputers in our home and yet developers managed it to bring it down to a crawl with things like electron apps.
- rusk 3y ago
  > compute power is cheap these days. It allows programmers to be ecobomical, take short cuts and select appropriate software architecture for the business problem Just an alternative take that I don’t entirely agree with but I often find rephrasing a problem in alternative tone it can help to understand it in non adversarial terms. In this case I see strong parallels with the auto industry and food distribution
- ZaoLahma 3y ago
  > It allows programmers to be lazy, take short cuts and not think about good software architecture. I think it has little to do with laziness, and more to do with the taboo of reinventing wheels. If there's a very generic and very broad solution to your very specific and very narrow problem, you're encouraged to use (or invent!) the very generic and very broad solution. This leads to everything being abstracted to the point where most of your application does absolutely nothing. Just look at the call stack of a modern java web application. The depth of the layers of abstractions that you need to navigate to reach anything that performs any real work is vertigo inducing and nauseating. It far from ideal and I agree that economics is behind it, but I strongly oppose the wording that it "allows" programmers to be lazy.
  - NL807 3y ago
    I do believe laziness is a big part of it. Because laziness IS about economics - in this case, personal economics. Why expend effort when you can get away with less and get rewarded the same?
benrutter 3y ago
> Features are celebrated and make money I like your take, but I'm not 100% sure this is accurate. Obviously to some extent features make money, and "featureless" software by definition couldn't exist. But I think a lot of software contains features that don't drive demand at all - the worst contender for this that I can think of is [Microsoft Teams' Together Mode](https://www.microsoft.com/en-us/microsoft-teams/teams-together-mode https://www.microsoft.com/en-us/microsoft-teams/teams-togeth...) which I doubt has led to a single sale of the product worldwide, but probably adds some maintenance burden. I think a bigger part is, like you mentioned, the organisation of companies. Engineers like building things, and although maintaining software and improving performance can be rewarding, you'd struggle to find developers that are happy to only do that. Features also lead to promotions more than maintenance, so there's a strong personal incentive to create features even if they don't have a business benefit.
mbf1 3y ago
Your competition will benefit when you stumble. All software has competition.
markb139 3y ago
>I've come to suspect the normal way large companies are organised makes performant software almost impossible. I think this is a bit of an over generalisation. There were (and still are) many large companies producing performant and reliable software. I suspect when things just work (think of the global phone system) people don't realise how much software is involved. When was the last time you picked up a fixed line hand set and there was a software bug preventing you from calling any other handset on the planet - without varying delays or dropped packets?
discreteevent 3y ago
> Features are celebrated and make money - whereas performance can be gradually salami-sliced away, with nobody advocating for it. - Optimists tend to be promoted, so the higher up in the organization you are, the more optimistic you tend to be. If one manager says "I can do that in 4 months", and another only promises it in 6 months, the 4 month guy gets the job. When the software is 4 months late, the overall system complexity makes it easy to assign blame elsewhere, so there's no way to judge mis-management when it's time for promotions. - There's a disconnect between engineering and marketing. It's not surprising -- marketing wants all the whiz-bang features, it wants to run in 16 megabytes, and it wants it yesterday. Although engineering would like the same things, it is faced with the reality of time limits, fixed costs, and the laws of nature. - The complexity of our system software has surpassed the ability of average SGI programmers to understand it. And perhaps not just average programmers. Get a room full of 10 of our best software people, and you'll get 10 different opinions of what's causing the lousy performance and bloat. What's wrong is that the software has simply become too complicated for anyone to understand. - There was never an overall software architect - We should sell 'bloat credits', the way the government sells pollution credits - SGI software has a cracked engine block, and we're trying to fix it with a tune-up. From 1993: https://www.seriss.com/people/erco/sgi-irix-bloat-document.txt https://www.seriss.com/people/erco/sgi-irix-bloat-document.t...
- peteradio 3y ago
  Get 10 different opinions and then have them actually implement them 10 different ways pick your favorite. Software development is already eye watering inefficient, getting people out of the way of each other at the cost of ten different implementations may just come out ahead.
GTP 3y ago
Your comment remebered me of the speed-up loop: https://thedailywtf.com/articles/The-Speedup-Loop https://thedailywtf.com/articles/The-Speedup-Loop

marginalia_nu 3y ago

There's a lot of overhead and inefficiencies in how large companies do anything. It's hard to organize thousands of people well. It's the social equivalent of the old 'Command-line Tools can be 235x Faster than your Hadoop Cluster'[1] conclusion. You put a hundred developers on a project and you'll generously get twice the meaningful output of ten developers (but 50 times the LOC; as demanded by conway's law). The upside is that this makes it all the more feasible for a small group or even single individual to be surprisingly competitive. Peak productivity per person is probably is probably found in a self-organized 1-3 person team. [1] https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...

mirekrusin 3y ago
People take holidays, get sick and change jobs - having 1-3 ppl team means you'll hit brick wall that will hurt sooner or later, more so if at least one of devs was really good.
- marginalia_nu 3y ago
  This is a real concern if you're looking at it as someone who is managing people... but a 1-3 person team does not need a manager to tell them what to do, and generally is not hired but spontaneously organized. It's an inherently more unpredictable way of working, but that definitely cuts both ways. There are definitely more risks, but also far greater payoffs in terms of what you can do working in this fashion. The principle of ensuring predictable velocity across a team comes a cost of limiting maximal produtivity amongst talented individuals.
- Jensson 3y ago
  It is better to sometimes be hurt than always be hurting. Organizations are so afraid of only 1 person understanding something that they create processes so that 0 persons understand it.
aleph_minus_one 3y ago
> You put a hundred developers on a project and you'll generously get twice the meaningful output of ten developers (but 50 times the LOC; as demanded by conway's law). Based on my job experience, this is rather an artifact of how software projects are (badly?) managed. Give each of the developers some small "hotspots" to work on (i.e. critical code fragment that is - hard to implement correctly and fast - which is critical to implement very fast and correctly - a correct and intended implementation will likely consist of few highly sophisticated lines of code ), and the scaling will be much better. This is a little bit like theses are assigned to students: each thesis is one isolated non-trivial scientific challenge (often open research problem) that is to be solved by the student. I guess the "management incentives" nevertheless prevent such a method: managers prefer/are incentivised to "tell stories" instead of explaining the deep, sophisticated challenges and solutions in making great implementations of these "hotspots".
- dasil003 3y ago
  This assumes you know what you need to build. 90% of the challenge in large scale development is nailing down requirements, especially when the problem space is too big for any individual to fully understand.
  - aleph_minus_one 3y ago
    > This assumes you know what you need to build. This is what in my opinion senior programmers or project managers are for. If they are not capable of handling this task decently, these people simply are not (yet) ready for this role.
    - falling_myshkin 3y ago
      Young engineer here in charge of a project and feeling quite out of their depth, I agree with this. Currently no mentorship and it will take a couple months for a senior hire. Do you have any advice? What does a senior engineer love/hate to see when they come onto a project started by engineers earlier in their career? How can I be most helpful?

ozim 3y ago

Fun part in context of big corporations is when performance becomes awful business will call in some guys from outside because clearly people currently working don’t know how to make it perform better. Business skips part where for years they insist on pushing features out. Where there most likely are tons of easy fixes that can speed up performance if you give the team time to focus on that. Going back to “outside help” then you get people who don’t understand current system and propose some silver bullet like throwing noSQL because that “will fix your issues” - but they don’t say cost is basically redoing the system, project goes under, third party guys count money, put another success story on CV because no one will validate it anyway…

lp4vn 3y ago

Development in any corp today, even middle-sized ones, is a big red tape festival. You have a project divided in sprints that are divided in (often meaningless) tasks, then you have a completely bloated CI/CD system and then things get deployed to three or four environments where people pretend to test everything. Things that might be developed by a journeyman developer in five days take six weeks to be done by a team of eight developers. The whole process today exists to support a big class of managerial people(including scrum masters, PO's and PM's) whose salary depends on not understanding that the process is completely broken.

mirekrusin 3y ago
On the bright side remote development means you can work 1-2h per day and make management clapping their hands in happiness.
mjfisher 3y ago
> The whole process today exists to support a big class of managerial people (including scrum masters, PO's and PM's) whose salary depends on not understanding that the process is completely broken. I think it's even worse than that - those unnecessary processes only exist to try to help management attempt to manage content they don't understand in the first place. And I say this as someone who regularly implements said delivery processes for customers.
zozbot234 3y ago
Wasn't "Agile" development supposed to fix this, by keeping team sizes small and workflows as simple and iterative as possible? AIUI, the fabled "two-pizzas team" size was intended as an upper bound for the most complex projects only, not as the new normal. (The "test everything" culture was also AIUI in the service of quick iteration; the first versions of Agile are from the early 2000s when dynamic languages were starting to get popular and there wasn't really any kind of static analysis.)
- whstl 3y ago
  Yes. Agile depends on not having a “big class of managerial people” micromanaging [1] every move done by the squad. No methodology will work when someone is actively fucking up the process, and agile is no exception. [1] In before some smartass comes to say “so you don’t want managers”… I qualified my statement with the word “micromanaging”.

monkeydust 3y ago

Depends. In certain fields like trading software, performance is a feature.

Jensson 3y ago

At Google they argued over microseconds, but there microseconds are money. If every software engineer working on search or ads added 10 microseconds each to load times Google would become unbearably slow extremely quickly and no longer make money.

fransje26 3y ago

Goodbye, Atom editor..

saagarjha 3y ago

> If you can add a feature and it adds just 50ms to a 300ms pageload - it's unlikely anyone is going to block its release on performance grounds. I worked on a team that did this. It was easy to justify because the benefits of the feature were almost always counterbalanced by less usage of the product, and we had numbers to back it up.

NL807 3y ago

>If you can add a feature and it adds just 50ms to a 300ms pageload - it's unlikely anyone is going to block its release on performance grounds. And once the feature is released, there's no removing it. Unfortunately this is happening, because compute power is cheap these days. It allows programmers to be lazy, take short cuts and not think about good software architecture. There is no economic incentive to optimise, because human perception won't perceive a difference and thus nobody complains. It's wild to think that we have basically supercomputers in our home and yet developers managed it to bring it down to a crawl with things like electron apps.

rusk 3y ago
> compute power is cheap these days. It allows programmers to be ecobomical, take short cuts and select appropriate software architecture for the business problem Just an alternative take that I don’t entirely agree with but I often find rephrasing a problem in alternative tone it can help to understand it in non adversarial terms. In this case I see strong parallels with the auto industry and food distribution
ZaoLahma 3y ago
> It allows programmers to be lazy, take short cuts and not think about good software architecture. I think it has little to do with laziness, and more to do with the taboo of reinventing wheels. If there's a very generic and very broad solution to your very specific and very narrow problem, you're encouraged to use (or invent!) the very generic and very broad solution. This leads to everything being abstracted to the point where most of your application does absolutely nothing. Just look at the call stack of a modern java web application. The depth of the layers of abstractions that you need to navigate to reach anything that performs any real work is vertigo inducing and nauseating. It far from ideal and I agree that economics is behind it, but I strongly oppose the wording that it "allows" programmers to be lazy.
- NL807 3y ago
  I do believe laziness is a big part of it. Because laziness IS about economics - in this case, personal economics. Why expend effort when you can get away with less and get rewarded the same?

benrutter 3y ago

> Features are celebrated and make money I like your take, but I'm not 100% sure this is accurate. Obviously to some extent features make money, and "featureless" software by definition couldn't exist. But I think a lot of software contains features that don't drive demand at all - the worst contender for this that I can think of is [Microsoft Teams' Together Mode](https://www.microsoft.com/en-us/microsoft-teams/teams-together-mode https://www.microsoft.com/en-us/microsoft-teams/teams-togeth...) which I doubt has led to a single sale of the product worldwide, but probably adds some maintenance burden. I think a bigger part is, like you mentioned, the organisation of companies. Engineers like building things, and although maintaining software and improving performance can be rewarding, you'd struggle to find developers that are happy to only do that. Features also lead to promotions more than maintenance, so there's a strong personal incentive to create features even if they don't have a business benefit.

mbf1 3y ago

Your competition will benefit when you stumble. All software has competition.

markb139 3y ago

>I've come to suspect the normal way large companies are organised makes performant software almost impossible. I think this is a bit of an over generalisation. There were (and still are) many large companies producing performant and reliable software. I suspect when things just work (think of the global phone system) people don't realise how much software is involved. When was the last time you picked up a fixed line hand set and there was a software bug preventing you from calling any other handset on the planet - without varying delays or dropped packets?

discreteevent 3y ago

> Features are celebrated and make money - whereas performance can be gradually salami-sliced away, with nobody advocating for it. - Optimists tend to be promoted, so the higher up in the organization you are, the more optimistic you tend to be. If one manager says "I can do that in 4 months", and another only promises it in 6 months, the 4 month guy gets the job. When the software is 4 months late, the overall system complexity makes it easy to assign blame elsewhere, so there's no way to judge mis-management when it's time for promotions. - There's a disconnect between engineering and marketing. It's not surprising -- marketing wants all the whiz-bang features, it wants to run in 16 megabytes, and it wants it yesterday. Although engineering would like the same things, it is faced with the reality of time limits, fixed costs, and the laws of nature. - The complexity of our system software has surpassed the ability of average SGI programmers to understand it. And perhaps not just average programmers. Get a room full of 10 of our best software people, and you'll get 10 different opinions of what's causing the lousy performance and bloat. What's wrong is that the software has simply become too complicated for anyone to understand. - There was never an overall software architect - We should sell 'bloat credits', the way the government sells pollution credits - SGI software has a cracked engine block, and we're trying to fix it with a tune-up. From 1993: https://www.seriss.com/people/erco/sgi-irix-bloat-document.txt https://www.seriss.com/people/erco/sgi-irix-bloat-document.t...

peteradio 3y ago
Get 10 different opinions and then have them actually implement them 10 different ways pick your favorite. Software development is already eye watering inefficient, getting people out of the way of each other at the cost of ten different implementations may just come out ahead.

GTP 3y ago

Your comment remebered me of the speed-up loop: https://thedailywtf.com/articles/The-Speedup-Loop https://thedailywtf.com/articles/The-Speedup-Loop