5 ms·

This is the sneaky, flawed premise behind many instances of what we call optimization. I have a Law of Optimization: the closer you get to optimal on your chose

by sfink 5y ago

This is the sneaky, flawed premise behind many instances of what we call optimization. I have a Law of Optimization: the closer you get to optimal on your chosen metrics, the more cost is shifted to unmeasured externalities. (I imagine someone has named this.) A stronger version would be to say that the distance to optimal is inversely proportional to the externalized cost, so that the total externalized cost goes to infinity as you squeeze out ever-smaller increments of gain with respect to your chosen metrics. The important corollary is that you can get 80% of the way there with very little externalized cost, and that going too far will eventually reveal the oversimplification in your cost model. You can move an object with near-zero energy, but it'll take 1000 years. You can get to net zero carbon emission by shifting it to someone else or dumping it somewhere it isn't measured or is a temporary sink that will eventually re-release. In my own area, you can reduce garbage collection overhead to zero by either never freeing anything (so the externalized cost is memory) or shifting the GCs to happen in between the timed portions of a benchmark (and this often happens accidentally, especially with a "no regressions" policy!)

alexpetralia 5y ago
Great comment. This is something I've thought for a long while, especially as we see the suppressed costs of "optimized systems" rear their ugly head against unplanned contingencies.
jjtheblunt 5y ago
Your Law of Optimization reminds me of the (often popularly ignored) denominator in the famous e = m c^2 equation. I believe the full version has a (1 - (v/c)^2) denominator on the right side; I think the Lorentz transformations motivate this. The upshot matches your observation in spirit: the closer you get to the optimum (the celeritas/speed of light) the more energy you require.
- JadeNB 5y ago
  > Your Law of Optimization reminds me of the (often popularly ignored) denominator in the famous e = m c^2 equation. > I believe the full version has a (1 - (v/c)^2) denominator on the right side …. I think the question is what you mean by `m`. If it's inertial mass, then I believe no correction is necessary. The point is that the inertial mass `m` is the rest mass `m_0` divided by `\sqrt{1 - (v/c)^2}`, as you say.
  - jjtheblunt 5y ago
    yes!
- breck 5y ago
  Ironic the man who wrote `make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.` is most tagged with `e = mc^2`.
triska 5y ago
A consultant I worked with compared the organization we wanted to measure to an air mattress: If you press it somewhere, i.e., measure a specific point, the air goes elsewhere, and another part is inflated. He used this analogy to explain the importance of taking the entire situation into account, by applying full cost accounting.
dane-pgp 5y ago
If you don't want to take the credit for it, you could call it the Pareto-Goodhart Law of Optimization, and state it something like this: "The closer a system gets to an optimal state, the greater the propensity for the agents attempting to further optimize it to externalize the costs of the system."
Nekhrimah 5y ago
> You can get to net zero carbon emission by shifting it to someone else or dumping it somewhere it isn't measured or is a temporary sink that will eventually re-release. Often referred to as tragedy of the commons.
deleted 5y ago
[deleted]
canadaduane 5y ago
I think this may explain part of why people tend to think of companies as "efficient" and governments as "wasteful": companies have more leeway to choose their metrics and externalize the unmeasured things.
- Jensson 5y ago
  If you think government agencies doesn't externalize costs then I have bad news for you! If we look at how communism went in practice then you'd see that those agencies were much better at externalizing costs than western companies. As long as they could optimize for the target measure they were fine and could blame everyone else when things didn't work. People think of governments as wasteful since governments never seem to manage long supply chains well. They can run services like roads, electricity, school, healthcare etc that are simple as long as all the necessary tools and materials are supplied by the private sector. They seemingly can't run stuff like medicine manufacturing, heavy machinery engineering etc.
  - hutzlibu 5y ago
    "They seemingly can't run stuff like medicine manufacturing, heavy machinery engineering etc. " Well, I now have a image of soviet russias state factories spitting out tank after tank ... Governments can handle this. Also "government" can mean very different things, but I agree that on a general base, they are not able to handle a complex chain of supply. Or at least not as good as a distributed market of private entities.
    - Jensson 5y ago
      Yeah, I should have added "efficiently" in there. And manufacturing has gotten a lot more complex since back then, not sure they could properly manage a modern supply chain with related electronics etc, without relying on the private sector to provide things. But we know both China and Soviet gave up trying to accomplish things without the private sector, so I'd assume they couldn't.
- thristian 5y ago
aaron695 5y ago
> the closer you get to optimal on your chosen metrics, the more cost is shifted to unmeasured externalities. This is fiction. Not a single example is given anywhere in this thread. But people believe it. And that is from the Environmental Industrial Complex that earns trillions of $ each year keeping suckers in despair. There are externalities, there are unmeasured externalities and it's a big word (13 letters) so people like using it. Your law will work, just like Jevons paradox which is also fake and almost constantly misused because no one using it outside academia (and also in) understands it. I would suggest you also name it paradox, that allows people to feel extra clever thinking the future sucks.
Jensson 5y ago
A very common example of this is people suggesting to remove private profits from the modern economy. It seems like a really good way to cut down waste and save costs in theory, but it never worked in practice if you do it for the whole economy. It does work really well for some problems, but for many others those private profits seems to be a necessary evil for the system to work well. What those people are missing is that private profits is a really good instrument for allocating resources and performing quality control efficiently in many sectors. Removing those means you now need to manage all those resource allocations and quality controls via other means, and so far we haven't found anything better for most business to business supply chains. Governments often do well in the business to consumer space though, as in many cases cutting costs there means consumers gets worse service. But in the business to business step both sides wants to push costs to each other, so they negotiate on who should do what to make both earn money, or they will go and do business with someone else.
- beckman466 5y ago
  > A very common example of this is people suggesting to remove private profits from the modern economy. It seems like a really good way to cut down waste and save costs in theory, but it never worked in practice if you do it for the whole economy. It does work really well for some problems, but for many others those private profits seems to be a necessary evil for the system to work well. > What those people are missing is that private profits is a really good instrument for allocating resources and performing quality control efficiently in many sectors. Such a ‘whole economy’ system was being built by Allende’s socialist government in Chile in the form of project Cybersyn (with the help of a well known British cyberneticist). This was before the Chilean bourgeoisie, backed by US imperialist forces, coup-ed Chile: https://www.youtube.com/watch?v=RJLA2_Ho7X0 https://www.youtube.com/watch?v=RJLA2_Ho7X0 An exciting project called http://valueflo.ws http://valueflo.ws is being developed by http://mikorizal.org http://mikorizal.org and others that reminds me a lot of Cybersyn.
  - aluket 5y ago
    Interesting thread on Cybersyn: https://news.ycombinator.com/item?id=24764727 https://news.ycombinator.com/item?id=24764727
bob1029 5y ago
> the closer you get to optimal on your chosen metrics, the more cost is shifted to unmeasured externalities. This is not inevitable in my experience. You can achieve optimizations that also reduce the dependency on external things. For some of us, this is one of the objectives of optimization. 3rd party dependencies == externalized cost. The real game for me is to solve the problem using correctly-allocated empathy. Building a product to streamline back office operations? Empathize with the user and the buisness processes. Building an options exchange which must handle at least 1 million orders per second? Empathize with the computer hardware. Optimizing for one factor vs another and your willingness to take on external cost should all be driven from this perspective.
- taneq 5y ago
  There's still going to be something you're not measuring, and this optimisation process is gonna be making that whatever-it-is worse. As I'm reading it, the fundamental principle we're talking about here is that there's always a point beyond which optimization is a tradeoff.
simonh 5y ago
This can absolutely be true for many systems and optimisations, but Fuller's premise is that new technology will allow efficiency improvements or substitutions that would not be possible in an extant system. For example one might come up with many ways to optimise the use of meeting rooms in an office, maybe making the rooms smaller to allow more simultaneous meetings, but now they're too cramped and you can't get as many people in a meeting. Maybe you add screens and network connections for presentations, but that's expensive, Tradeoffs, tradeoffs. However if you switch to a new technology like desktop videoconferencing you can get rid of the meeting rooms completely. Or you can reduce the number of meeting rooms by 5x and make the remaining ones bigger and higher tech for much better in person meetings that remain, at much lower total cost. New technology can completely eliminate or restructure whole categories of optimisation tradeoff.

by sfink 5y ago

alexpetralia 5y ago

Great comment. This is something I've thought for a long while, especially as we see the suppressed costs of "optimized systems" rear their ugly head against unplanned contingencies.

jjtheblunt 5y ago

Your Law of Optimization reminds me of the (often popularly ignored) denominator in the famous e = m c^2 equation. I believe the full version has a (1 - (v/c)^2) denominator on the right side; I think the Lorentz transformations motivate this. The upshot matches your observation in spirit: the closer you get to the optimum (the celeritas/speed of light) the more energy you require.

JadeNB 5y ago
> Your Law of Optimization reminds me of the (often popularly ignored) denominator in the famous e = m c^2 equation. > I believe the full version has a (1 - (v/c)^2) denominator on the right side …. I think the question is what you mean by `m`. If it's inertial mass, then I believe no correction is necessary. The point is that the inertial mass `m` is the rest mass `m_0` divided by `\sqrt{1 - (v/c)^2}`, as you say.
- jjtheblunt 5y ago
  yes!
breck 5y ago
Ironic the man who wrote `make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.` is most tagged with `e = mc^2`.

triska 5y ago

A consultant I worked with compared the organization we wanted to measure to an air mattress: If you press it somewhere, i.e., measure a specific point, the air goes elsewhere, and another part is inflated. He used this analogy to explain the importance of taking the entire situation into account, by applying full cost accounting.

dane-pgp 5y ago

If you don't want to take the credit for it, you could call it the Pareto-Goodhart Law of Optimization, and state it something like this: "The closer a system gets to an optimal state, the greater the propensity for the agents attempting to further optimize it to externalize the costs of the system."

Nekhrimah 5y ago

> You can get to net zero carbon emission by shifting it to someone else or dumping it somewhere it isn't measured or is a temporary sink that will eventually re-release. Often referred to as tragedy of the commons.

deleted 5y ago

[deleted]

canadaduane 5y ago

I think this may explain part of why people tend to think of companies as "efficient" and governments as "wasteful": companies have more leeway to choose their metrics and externalize the unmeasured things.

Jensson 5y ago
If you think government agencies doesn't externalize costs then I have bad news for you! If we look at how communism went in practice then you'd see that those agencies were much better at externalizing costs than western companies. As long as they could optimize for the target measure they were fine and could blame everyone else when things didn't work. People think of governments as wasteful since governments never seem to manage long supply chains well. They can run services like roads, electricity, school, healthcare etc that are simple as long as all the necessary tools and materials are supplied by the private sector. They seemingly can't run stuff like medicine manufacturing, heavy machinery engineering etc.
- hutzlibu 5y ago
  "They seemingly can't run stuff like medicine manufacturing, heavy machinery engineering etc. " Well, I now have a image of soviet russias state factories spitting out tank after tank ... Governments can handle this. Also "government" can mean very different things, but I agree that on a general base, they are not able to handle a complex chain of supply. Or at least not as good as a distributed market of private entities.
  - Jensson 5y ago
    Yeah, I should have added "efficiently" in there. And manufacturing has gotten a lot more complex since back then, not sure they could properly manage a modern supply chain with related electronics etc, without relying on the private sector to provide things. But we know both China and Soviet gave up trying to accomplish things without the private sector, so I'd assume they couldn't.
thristian 5y ago

aaron695 5y ago

> the closer you get to optimal on your chosen metrics, the more cost is shifted to unmeasured externalities. This is fiction. Not a single example is given anywhere in this thread. But people believe it. And that is from the Environmental Industrial Complex that earns trillions of $ each year keeping suckers in despair. There are externalities, there are unmeasured externalities and it's a big word (13 letters) so people like using it. Your law will work, just like Jevons paradox which is also fake and almost constantly misused because no one using it outside academia (and also in) understands it. I would suggest you also name it paradox, that allows people to feel extra clever thinking the future sucks.

Jensson 5y ago

A very common example of this is people suggesting to remove private profits from the modern economy. It seems like a really good way to cut down waste and save costs in theory, but it never worked in practice if you do it for the whole economy. It does work really well for some problems, but for many others those private profits seems to be a necessary evil for the system to work well. What those people are missing is that private profits is a really good instrument for allocating resources and performing quality control efficiently in many sectors. Removing those means you now need to manage all those resource allocations and quality controls via other means, and so far we haven't found anything better for most business to business supply chains. Governments often do well in the business to consumer space though, as in many cases cutting costs there means consumers gets worse service. But in the business to business step both sides wants to push costs to each other, so they negotiate on who should do what to make both earn money, or they will go and do business with someone else.

beckman466 5y ago
> A very common example of this is people suggesting to remove private profits from the modern economy. It seems like a really good way to cut down waste and save costs in theory, but it never worked in practice if you do it for the whole economy. It does work really well for some problems, but for many others those private profits seems to be a necessary evil for the system to work well. > What those people are missing is that private profits is a really good instrument for allocating resources and performing quality control efficiently in many sectors. Such a ‘whole economy’ system was being built by Allende’s socialist government in Chile in the form of project Cybersyn (with the help of a well known British cyberneticist). This was before the Chilean bourgeoisie, backed by US imperialist forces, coup-ed Chile: https://www.youtube.com/watch?v=RJLA2_Ho7X0 https://www.youtube.com/watch?v=RJLA2_Ho7X0 An exciting project called http://valueflo.ws http://valueflo.ws is being developed by http://mikorizal.org http://mikorizal.org and others that reminds me a lot of Cybersyn.
- aluket 5y ago
  Interesting thread on Cybersyn: https://news.ycombinator.com/item?id=24764727 https://news.ycombinator.com/item?id=24764727

bob1029 5y ago

> the closer you get to optimal on your chosen metrics, the more cost is shifted to unmeasured externalities. This is not inevitable in my experience. You can achieve optimizations that also reduce the dependency on external things. For some of us, this is one of the objectives of optimization. 3rd party dependencies == externalized cost. The real game for me is to solve the problem using correctly-allocated empathy. Building a product to streamline back office operations? Empathize with the user and the buisness processes. Building an options exchange which must handle at least 1 million orders per second? Empathize with the computer hardware. Optimizing for one factor vs another and your willingness to take on external cost should all be driven from this perspective.

taneq 5y ago
There's still going to be something you're not measuring, and this optimisation process is gonna be making that whatever-it-is worse. As I'm reading it, the fundamental principle we're talking about here is that there's always a point beyond which optimization is a tradeoff.

simonh 5y ago

This can absolutely be true for many systems and optimisations, but Fuller's premise is that new technology will allow efficiency improvements or substitutions that would not be possible in an extant system. For example one might come up with many ways to optimise the use of meeting rooms in an office, maybe making the rooms smaller to allow more simultaneous meetings, but now they're too cramped and you can't get as many people in a meeting. Maybe you add screens and network connections for presentations, but that's expensive, Tradeoffs, tradeoffs. However if you switch to a new technology like desktop videoconferencing you can get rid of the meeting rooms completely. Or you can reduce the number of meeting rooms by 5x and make the remaining ones bigger and higher tech for much better in person meetings that remain, at much lower total cost. New technology can completely eliminate or restructure whole categories of optimisation tradeoff.