8 ms·

Heat loss from big corporate projects

71 points by r4um 9y ago

hawktheslayer 9y ago
I work in a corporate environment in business intelligence and this rings true. This year we are tasked with retiring an arbitrary number of reports (mostly old Crystal reports) since they take too much of our time to maintain them. This will mean telling business partners that we are shutting down their reports, which in turn will cause them to create more Excel spreadsheets and Access databases. And in the process there will be the "heat loss" described in the article.
- lifeisstillgood 9y ago
  Isn't this the perfect example of lack of markets inside companies? i mean the consumers of your reports are either perfectly prepared to pay your department enough to keep the reports coming, or they don't want them that much. but without a free floating price the larger company will never know. Roald Coase probably has something to say on it
  - kwhitefoot 9y ago
    How can you have a market inside a company? Markets depend on having multiple suppliers and multiple customers. Nokia tried having internal competition and it didn't go well. Something like it can work though. Many years ago I worked for Philip Semiconductor at the Mullard factory in Southampton in a department that designed and built production and test equipment. We had a formal project process where if the customer department wanted something they had to ask us first and allow us to perform a feasibility study or produce a prototype while spending up to ten percent of the rough estimate of the project cost. At the end of that we would present a detailed plan and cost. The customer department could then decide whether to offer the job to us or ask an external player such as British Aerospace to do it.
    - joshuamorton 9y ago
      Its not a market so much as an economy. Instead of necessarily saying "you must do things this way" its "you must meet these requirements and these SLAs/goals/dates, and you have this many Magic Corporate Resource Bucks to spend". Those Magic Corporate Resource Bucks aren't real dollars necessarily, but they are representative of the costs of things that other divisions of the company may provide. You can swap your MCRBs for real dollars and go out and purchase your own hardware, for example, or do your own service monitoring, or you can "pay" the team that manages hardware to maintain it for you. And the service monitoring team can integrate your project into the company-wide monitoring system. If you do the first option, it might be cheaper, but when things inevitably break, you're the one who takes the blame and has to fix it, instead of having in house experts who fix it. There's central planning, so you don't have monopoly issues. The hardware provisioning team can't just jack up the price to infinity, because the company can see what the actual cost is and force the team to provide the stuff at cost. Then you do a little magic like loosely tying bonuses to the amount of MCRBs you receive if your team is an infrastructural one, and perhaps allow management to redeem MCRBs for things like additional capacity to hire employees, and potentially even use MCRB allocation by end users as a way of figuring out what pieces of infrastructure are the highest impact/most in need of additional staffing/etc. You end up with a system where infra teams have decent incentives to offer good tooling and to fix bugs/implement features (because there's always the looming threat of a client team leaving you and developing their own solution). User facing teams have an incentive to use existing tools and rely on the infrastructure that exists, unless there are compelling reasons not to, but when there are such reasons, they have some level of autonomy to develop unique solutions to fit their needs, and you have indirect, but still useful, signals of where upper management needs to invest additional resources, because something is being used a lot.
      - jcrites 9y ago
        Do you know if any companies have actually tried this? I'd love to read a case study if you know of one.
    - Applejinx 9y ago
      Sears tried having internal competition (and markets. IIRC) and it really didn't go well.
RyanZAG 9y ago
The whole system where all of these projects generally go towards some form of advertising where we convince people to buy things they don't really need is a far worse form of heat loss than this. Buying the latest shiny smartphone every year isn't just the heat loss of thousands of engineers making it slightly thinner, it's the actual loss of all the resources that went into last year's batch of smartphones that end up in a landfill somewhere. And yet, none of this is really a loss as people need something that feels important to do with their time, and building that new smartphone that is the world's thinnest definitely feels like you're making accomplishments. And people are pretty happy when they open the box and see their new smartphone. Moving corporate resources around is just one more step in a world we don't truly need, but which we create for ourselves anyway. In this case, it's part of the game of proving that our company has the biggest numbers so that our share holders get a smile on their faces when they open their retirement portfolios.
- Nition 9y ago
  Isn't it funny, the amount of things like this that go on in the system we've created. Like how weird is it that "competition" - two companies both working on the same things against each other, without sharing what they know - can be better for the consumer than if both companies worked together. None of this is really going to change unless we can somehow make a world where making the world a truly better place can be a more powerful driver than money.
  - OscarCunningham 9y ago
    It seems to me like competing companies actually are incentivised to share information with each other. That would allow them both to cut costs. Does anyone know why they don't do this? Or know of examples where they do do this? Of course competition would force them both to lower prices after they cut costs, but this will lead to increased demand so both companies should be making more profit. Also if they have a third competitior then they will both be gaining customers from them. EDIT: Actually this happens all the time, just mediated by money. My economist brain switched off for a second. I was imagining a situation like "You tell us that trade secret in exchange for this one". But this situtation would require a "coincidence of wants". Money alleviates that problem. So this does happen all the time, it just looks more like "Even though you're our competitors we'll give you secret X in exchange for $n".
kwhitefoot 9y ago
This happens at the company I recently retired from. At intervals of between five and ten years someone decides that R&D should either be centralised or that the development engineers should be closer to the factories. The people are the same people, the work they do is the same, but the location is different. The justification for centralisation is usually that there will be better communication between the development engineers and the justification for putting them in the factories is that they will be able to communicate better with the management and workers of the factories. The reorganization is always intended to have a valuable side effect. I could never understand why they don't go directly for the side effect.
scottmcdot 9y ago
What if the 'heat loss' is just the required 'activation energy' to get from State A to State B, where B is far superior for the customer.
wellboy 9y ago
Isn't this just switching costs? The author assumes that a change in a process never amortize. However, some changes cost time and money yes, but they amortize after a few months ans are worth it. Yes, some changes might lead to worse processes than others, but most changes aren't. It all comes down to a good CEO, if you make too many changes that make it worse, the business becomes bankrupt after a few years. If your changes amortize, even only slightly, your business stays afloat or improves.
baxtr 9y ago
In that sense, isn’t all of life somehow “heat loss”?
- Nition 9y ago
  Depends what you consider meaningful in life I think. Certainly a nihilist could say so.
  - baxtr 9y ago
    I find it difficult to draw a line. Is it meaningful to build smartphones?
    - Nition 9y ago
      Well, I think it is difficult to draw a line or we wouldn't be having this conversation, and everyone's line is going to be in a different place (and move around a bit as they change their own opinions as well). A large part of philosophy is built on trying to decide what is meaningful to do. I can't give you an answer because I'm not sure what the best answer is myself.
realusername 9y ago
I completely agree with this. Something I would add here is the great danger of wanting to recreate products from scratch. You have the "heat loss" effect as stated here but you also have a massive amount of expectation from the users of the existing product. It can quickly become "the product which will fix all our issues" over time and the goalposts are being moved so much that the release day is just pushed forward forever.
roadbeats 9y ago
Reinvention of desktop apps in web could be also an example for this.
dnautics 9y ago
as a chemist, love the analogy, resonates a ton with me. a little bit nit-picky, but: "That is in addition to the actual cost of pumping the water from one tank to the other." should be something like "that is in addition to the actual cost of lifting net 25 units of water higher". The cost of moving from one tank to the other is, also heat loss, but the lifting part is recoverable.
brain5ide 9y ago
I like the last sentence nod to Douglas Adams.
kokey 9y ago
I think many businesses are implementing productivity improvement work which consists of a) an investment in engineering time and b) moving things around. I think to most people moving things around is the only really visible part. I think this leads to others simply copying the 'moving around' part in order to attempt to gain productivity improvements. The most common example of this nowadays is people moving their infrastructure into a cloud provider. If this is not done right and the investment made in the engineering time to deliver a more productive result, it's going to be worse than not moving around.
madez 9y ago
The example with the halved memory space isn't a good one. There might be a real net benefit to society by using up less resources.
- UncleEntity 9y ago
  There might also be a net loss to society for any number of reasons, Bastiat's That Which is Seen, and That Which is Not Seen is of special interest here.
UncleEntity 9y ago
Has anyone blogged about a software refactoring project being "heat loss"? Same thing really -- it comes out better, the same or worse but the underlying principles are identical. You could avoid the "heat loss" by just hacking on the old crufty code or redesign it into a fancy-smancy state of the art system. I would also bring up The Economic Calculation Problem and how it applies to large monolithic corporations equally as well but that's a discussion for another day.
- xaedes 9y ago
  Refactorings can safe future heat losses by enabling easier maintainance and extensibility.
alxlaz 9y ago
This is a lovely account and, though it may not seem when you first read it -- in my experience, it is somewhat optimistic. For instance: > The process of changing the system to have a new balancing point took work. It involved investments of engineer time, design work, tech writers and documentation changes, and all kinds of other "meta" stuff. In most places I've worked, this involves investments of two interns' time and a few design meetings which result in a design document, half of which will never make it into the actual implementation. If it's a customer-facing thing, it will result in documentation changes -- otherwise, it won't. There will be no time to adjust the documentation and internal knowledge about how we do it now is going to float around as oral wisdom. Why we do it like that is going to be folklore two years down the road.
shoo 9y ago
> But, it kept them busy, and let them show off, and they can take that all the way to the bank by way of a performance review system that optimizes for Shiny New Things. Yup. At least some of the generated heat can be captured in the form of bullet points on resumes, which can be used as leverage for promotions or new jobs. Perhaps from a whole-system or whole-company perspective the result may be a net loss, but the endeavour may be quite postive for many of the people involved who are making or executing the decisions.
theyinwhy 9y ago
In my current project we get moved around teams on a monthly (!) basis. The heat loss generated is so enormous you start to feel like you are in a powerplant.
greenleafjacob 9y ago
Common tropes in software for this: * Switching from polling to push or vice versa * Merging two overlapping systems into one * Splitting one system into two * Rewriting from language X to Y * Moving from centralized to decentralized or vice versa * Moving logic from one layer to another (eg smart client vs thin client)
borman 9y ago
Another analogy that comes to mind would be 'going to gym'. Short-term, working out is exactly a heat loss (doing hard work with no useful results). But long-term, it makes you healthier and able to do more things. Back to corporate IT, it's not that hard to stabilize a decent big ol' legacy revenue generating system, while avoiding making changes. But if the company has to compete against smaller and leaner startups, avoiding changes might become a major risk, as the cost of all changes tends to go up. So, it would be reasonable to keep the system is a constant state of flux, so that it has no choice but to become fitter and learn to change quickly without breaking: stuff like reproducible builds, reproducible deployment, CI/CD don't matter a lot for monthly stable releases, but one would have a hard time doing nightly stable releases without them.

71 points by r4um 9y ago

I work in a corporate environment in business intelligence and this rings true. This year we are tasked with retiring an arbitrary number of reports (mostly old Crystal reports) since they take too much of our time to maintain them. This will mean telling business partners that we are shutting down their reports, which in turn will cause them to create more Excel spreadsheets and Access databases. And in the process there will be the "heat loss" described in the article.

lifeisstillgood 9y ago
Isn't this the perfect example of lack of markets inside companies? i mean the consumers of your reports are either perfectly prepared to pay your department enough to keep the reports coming, or they don't want them that much. but without a free floating price the larger company will never know. Roald Coase probably has something to say on it
- kwhitefoot 9y ago
  How can you have a market inside a company? Markets depend on having multiple suppliers and multiple customers. Nokia tried having internal competition and it didn't go well. Something like it can work though. Many years ago I worked for Philip Semiconductor at the Mullard factory in Southampton in a department that designed and built production and test equipment. We had a formal project process where if the customer department wanted something they had to ask us first and allow us to perform a feasibility study or produce a prototype while spending up to ten percent of the rough estimate of the project cost. At the end of that we would present a detailed plan and cost. The customer department could then decide whether to offer the job to us or ask an external player such as British Aerospace to do it.
  - joshuamorton 9y ago
    Its not a market so much as an economy. Instead of necessarily saying "you must do things this way" its "you must meet these requirements and these SLAs/goals/dates, and you have this many Magic Corporate Resource Bucks to spend". Those Magic Corporate Resource Bucks aren't real dollars necessarily, but they are representative of the costs of things that other divisions of the company may provide. You can swap your MCRBs for real dollars and go out and purchase your own hardware, for example, or do your own service monitoring, or you can "pay" the team that manages hardware to maintain it for you. And the service monitoring team can integrate your project into the company-wide monitoring system. If you do the first option, it might be cheaper, but when things inevitably break, you're the one who takes the blame and has to fix it, instead of having in house experts who fix it. There's central planning, so you don't have monopoly issues. The hardware provisioning team can't just jack up the price to infinity, because the company can see what the actual cost is and force the team to provide the stuff at cost. Then you do a little magic like loosely tying bonuses to the amount of MCRBs you receive if your team is an infrastructural one, and perhaps allow management to redeem MCRBs for things like additional capacity to hire employees, and potentially even use MCRB allocation by end users as a way of figuring out what pieces of infrastructure are the highest impact/most in need of additional staffing/etc. You end up with a system where infra teams have decent incentives to offer good tooling and to fix bugs/implement features (because there's always the looming threat of a client team leaving you and developing their own solution). User facing teams have an incentive to use existing tools and rely on the infrastructure that exists, unless there are compelling reasons not to, but when there are such reasons, they have some level of autonomy to develop unique solutions to fit their needs, and you have indirect, but still useful, signals of where upper management needs to invest additional resources, because something is being used a lot.
    - jcrites 9y ago
      Do you know if any companies have actually tried this? I'd love to read a case study if you know of one.
  - Applejinx 9y ago
    Sears tried having internal competition (and markets. IIRC) and it really didn't go well.

RyanZAG 9y ago

The whole system where all of these projects generally go towards some form of advertising where we convince people to buy things they don't really need is a far worse form of heat loss than this. Buying the latest shiny smartphone every year isn't just the heat loss of thousands of engineers making it slightly thinner, it's the actual loss of all the resources that went into last year's batch of smartphones that end up in a landfill somewhere. And yet, none of this is really a loss as people need something that feels important to do with their time, and building that new smartphone that is the world's thinnest definitely feels like you're making accomplishments. And people are pretty happy when they open the box and see their new smartphone. Moving corporate resources around is just one more step in a world we don't truly need, but which we create for ourselves anyway. In this case, it's part of the game of proving that our company has the biggest numbers so that our share holders get a smile on their faces when they open their retirement portfolios.

Nition 9y ago
Isn't it funny, the amount of things like this that go on in the system we've created. Like how weird is it that "competition" - two companies both working on the same things against each other, without sharing what they know - can be better for the consumer than if both companies worked together. None of this is really going to change unless we can somehow make a world where making the world a truly better place can be a more powerful driver than money.
- OscarCunningham 9y ago
  It seems to me like competing companies actually are incentivised to share information with each other. That would allow them both to cut costs. Does anyone know why they don't do this? Or know of examples where they do do this? Of course competition would force them both to lower prices after they cut costs, but this will lead to increased demand so both companies should be making more profit. Also if they have a third competitior then they will both be gaining customers from them. EDIT: Actually this happens all the time, just mediated by money. My economist brain switched off for a second. I was imagining a situation like "You tell us that trade secret in exchange for this one". But this situtation would require a "coincidence of wants". Money alleviates that problem. So this does happen all the time, it just looks more like "Even though you're our competitors we'll give you secret X in exchange for $n".

kwhitefoot 9y ago

This happens at the company I recently retired from. At intervals of between five and ten years someone decides that R&D should either be centralised or that the development engineers should be closer to the factories. The people are the same people, the work they do is the same, but the location is different. The justification for centralisation is usually that there will be better communication between the development engineers and the justification for putting them in the factories is that they will be able to communicate better with the management and workers of the factories. The reorganization is always intended to have a valuable side effect. I could never understand why they don't go directly for the side effect.

scottmcdot 9y ago

What if the 'heat loss' is just the required 'activation energy' to get from State A to State B, where B is far superior for the customer.

wellboy 9y ago

Isn't this just switching costs? The author assumes that a change in a process never amortize. However, some changes cost time and money yes, but they amortize after a few months ans are worth it. Yes, some changes might lead to worse processes than others, but most changes aren't. It all comes down to a good CEO, if you make too many changes that make it worse, the business becomes bankrupt after a few years. If your changes amortize, even only slightly, your business stays afloat or improves.

baxtr 9y ago

In that sense, isn’t all of life somehow “heat loss”?

Nition 9y ago
Depends what you consider meaningful in life I think. Certainly a nihilist could say so.
- baxtr 9y ago
  I find it difficult to draw a line. Is it meaningful to build smartphones?
  - Nition 9y ago
    Well, I think it is difficult to draw a line or we wouldn't be having this conversation, and everyone's line is going to be in a different place (and move around a bit as they change their own opinions as well). A large part of philosophy is built on trying to decide what is meaningful to do. I can't give you an answer because I'm not sure what the best answer is myself.

realusername 9y ago

I completely agree with this. Something I would add here is the great danger of wanting to recreate products from scratch. You have the "heat loss" effect as stated here but you also have a massive amount of expectation from the users of the existing product. It can quickly become "the product which will fix all our issues" over time and the goalposts are being moved so much that the release day is just pushed forward forever.

roadbeats 9y ago

Reinvention of desktop apps in web could be also an example for this.

dnautics 9y ago

as a chemist, love the analogy, resonates a ton with me. a little bit nit-picky, but: "That is in addition to the actual cost of pumping the water from one tank to the other." should be something like "that is in addition to the actual cost of lifting net 25 units of water higher". The cost of moving from one tank to the other is, also heat loss, but the lifting part is recoverable.

brain5ide 9y ago

I like the last sentence nod to Douglas Adams.

kokey 9y ago

I think many businesses are implementing productivity improvement work which consists of a) an investment in engineering time and b) moving things around. I think to most people moving things around is the only really visible part. I think this leads to others simply copying the 'moving around' part in order to attempt to gain productivity improvements. The most common example of this nowadays is people moving their infrastructure into a cloud provider. If this is not done right and the investment made in the engineering time to deliver a more productive result, it's going to be worse than not moving around.

madez 9y ago

The example with the halved memory space isn't a good one. There might be a real net benefit to society by using up less resources.

UncleEntity 9y ago
There might also be a net loss to society for any number of reasons, Bastiat's That Which is Seen, and That Which is Not Seen is of special interest here.

UncleEntity 9y ago

Has anyone blogged about a software refactoring project being "heat loss"? Same thing really -- it comes out better, the same or worse but the underlying principles are identical. You could avoid the "heat loss" by just hacking on the old crufty code or redesign it into a fancy-smancy state of the art system. I would also bring up The Economic Calculation Problem and how it applies to large monolithic corporations equally as well but that's a discussion for another day.

xaedes 9y ago
Refactorings can safe future heat losses by enabling easier maintainance and extensibility.

alxlaz 9y ago

This is a lovely account and, though it may not seem when you first read it -- in my experience, it is somewhat optimistic. For instance: > The process of changing the system to have a new balancing point took work. It involved investments of engineer time, design work, tech writers and documentation changes, and all kinds of other "meta" stuff. In most places I've worked, this involves investments of two interns' time and a few design meetings which result in a design document, half of which will never make it into the actual implementation. If it's a customer-facing thing, it will result in documentation changes -- otherwise, it won't. There will be no time to adjust the documentation and internal knowledge about how we do it now is going to float around as oral wisdom. Why we do it like that is going to be folklore two years down the road.

shoo 9y ago

> But, it kept them busy, and let them show off, and they can take that all the way to the bank by way of a performance review system that optimizes for Shiny New Things. Yup. At least some of the generated heat can be captured in the form of bullet points on resumes, which can be used as leverage for promotions or new jobs. Perhaps from a whole-system or whole-company perspective the result may be a net loss, but the endeavour may be quite postive for many of the people involved who are making or executing the decisions.

theyinwhy 9y ago

In my current project we get moved around teams on a monthly (!) basis. The heat loss generated is so enormous you start to feel like you are in a powerplant.

greenleafjacob 9y ago

Common tropes in software for this: * Switching from polling to push or vice versa * Merging two overlapping systems into one * Splitting one system into two * Rewriting from language X to Y * Moving from centralized to decentralized or vice versa * Moving logic from one layer to another (eg smart client vs thin client)

borman 9y ago

Another analogy that comes to mind would be 'going to gym'. Short-term, working out is exactly a heat loss (doing hard work with no useful results). But long-term, it makes you healthier and able to do more things. Back to corporate IT, it's not that hard to stabilize a decent big ol' legacy revenue generating system, while avoiding making changes. But if the company has to compete against smaller and leaner startups, avoiding changes might become a major risk, as the cost of all changes tends to go up. So, it would be reasonable to keep the system is a constant state of flux, so that it has no choice but to become fitter and learn to change quickly without breaking: stuff like reproducible builds, reproducible deployment, CI/CD don't matter a lot for monthly stable releases, but one would have a hard time doing nightly stable releases without them.