9 ms·

What I've noticed about models, or at least when people are talking about them or trying to prove a point about them, is that people forget models are a simplif

What I've noticed about models, or at least when people are talking about them or trying to prove a point about them, is that people forget models are a simplified version of a specific part of reality, much the same as models of airplanes or something. No matter how many variables you include, you can never capture the utterly massive and unpredictable amount of variables that exist in reality. But they're not supposed to. They're tools that are supposed to be used to help get an idea about how something might happen, that's marginally better than just guessing, due again to models always being incomplete, despite any amount of best efforts. Like any tools, using models incorrectly, fucking around with them until you just get what you want to see or using the wrong one for the wrong task can be potentially dangerous, especially when those models are used to guide large decisions with serious consequences and impacts. Reasoning and common sense need to be used alongside models and if the models don't seem to reflect reality, the problem is with the model and not reality and the model needs to be adjusted or thrown out otherwise continuing to use it will just lead to lousier and lousier decisions.

hinkley 7y ago
It’s funny, but when you point out that a bad analogy is actually pretty accurate if you actually know anything about the other concept, people don’t want to talk about it any more. We all know that hill climbing algorithms are often naive and sometimes hilariously wrong. Nobody will disagree with you about this, until you start talking about prioritizing work, and then everyone vigorously defends their favorite hill climbing algorithm, from task management to performance tuning. My preferred strategy for optimization more closely resembles how a fruit grower would pick fruit, and the term, “low-hanging fruit” galls me because you would go bankrupt using that strategy. And probably lose most of your trees to disease. It could be a quite good analogy, if the lessons taken from it weren’t just so childishly naive.
- edmundsauto 7y ago
  Can you elaborate on your preferred strategy?
  - hinkley 7y ago
    I'm gonna do this as two posts to split up the how and the why. Here is why and some hints at how: Perf analysis was how I got through some of my most mundane college homework assignments instead of (or sometimes, in addition to) procrastinating. Then during my formative years I saw the following script play out several times: Management and sales have a problem. They say the customers are upset that the code is so slow (maybe the customer has expectations, or maybe a competitor or their old system went a lot faster). It's taking 300 units of time now and they think it should take 100. So the leads and the heavies wander off and come back with 20% easy, the next 10 harder, 6, 4, and then they declare that they've done all they can. So now we're around 180 units of time. That's all the customer gets. See this <flame chart precursor> chart? Everything is nice and flat. Our hands are tied now. Business is... not happy. They're glad they have something to show the customer, but it's really just a bone. So between tasks I'd start poking around, because why not. A pinch here, an architectural tweak there, and maybe 25% just gruntwork, and I'd find another 40% out of the "we've done all we can" code, which is just a hair away from what the business asked for. You can imagine that you invite a tremendous amount of well-deserved scrutiny on yourself doing stuff like this. But that's okay: the most important part of optimization is not the code speed but the bang for the buck. You can certainly interpret "Low hanging fruit" to fulfill that concern, in practice most people don't (for some, lack of imagination, for others, ego-protection). But once you consider everything, including, for instance, externalities on QA, or the misapprehensions of customers who expect every release to be faster... there are a lot of more complicated things to worry about than how to rearrange this block of code to improve branch prediction. Not touching it for 10% is just a heuristic.
  - hinkley 7y ago
    Performance is just budgeting. It's opportunity costs (versus expected return on the investment). It's time, materials, risk/scheduling uncertainty, and in many cases, externalities. Interestingly enough, I don't think you need to explain any of this to game designers. They seem to know this. It even appears to be the dogmatic in some circles. When you are way over budget, every cost needs to go on the table, but your end goal is one of the most important pieces of information. That should go on the wall, in large block letters. Look at a real budget. If you are deep in the red, it doesn't matter that your car is only 10% of how much you spend a month. What matters is that the car is 25% of your target. That is a ridiculous amount to spend on your car, even though your house is way more. Even if you spend an ungodly amount on dry cleaning or coffee, you're going to have to trade down for a cheaper car. You don't need charts to tell you this. In fact all the charts can do is convince you that nothing needs to be done. They allow you to bargain. Therefore, I don't give a shit if you think that this function which accounts for "only 5%" of the current run time "is fine". It's 13% of our time budget on some mundane task that doesn't really make us money. It is not fine. You cannot justify 1/8th of our budget for this thing. It's gonna be fixed, and if you have no idea how, you'd better start thinking about it now, because we're gonna come back to it soon and if you don't have a solution then you might find that code is now someone else's responsibility. But I said something about the 'fruit tree' analogy and I'm a full page of text in without a peep. So the thing is that when you're thinking about big budgetary changes, it's more manageable to do it one subject matter at a time. Pick a 'ripe' area and glean it for everything it's worth. So I might pitch that I think I can get 30% improvement out of the edit/update code. Can I have this much time to do it? Okay. I might get half of that goal in the first couple weeks. Half of the rest in the next, but at the end of it, the last thing I touch is probably only going to get 3%. But 3% is 10% of what I promised, so I can justify it, and if they aren't complaining about the timeline, I may try to squeeze in some 1-2% things at the end in a way I can pull the plug if it looks too risky. The important thing to note here is that it will never, ever be as cheap as it is right now to touch that 3% code. Everyone is already thinking about it, everyone knows to keep an eye on it, and new test plans are being developed, documentation and customer training plans are being updated. If I don't touch it now, that target of opportunity might never come again. That 3% plays out everywhere in the code, and I will never be able to get budget to fix any of them (I'll have to sneak them in on a refactor or leave them forever). A dozen of these is not that hard to end up with, and our code is 40% slower than it could be (or worst of all, than a competitor). If that's just the time for a button click, fine. If it's on our slowest operation and things time out, customers with larger data can't function, or they have to buy the absolute most expensive machines instead of slightly better than average? That's not fine. But everyone feels perfectly justified in never doing anything about it. Next release, I can pitch for 25% in some other functional area, and another, and another. Last time I did this, it took over 20 months before I ran out of functional areas to work on. Every release for 'years' was faster than the previous. At the end, I knew that code better than anyone (an important point I omitted - focusing that intently on one section teaches people a hell of a lot about the code they otherwise would never know). I probably could have swung through for another 10% in every area, but by then I was ready for a new gig.
- tartoran 7y ago
  The low hanging fruit metaphor meaning do the simplest or easiest work first is not really applicable to fruit growing but seems to have been coopted by the busieness world instead, then we started using it too in our contexts. Not crazy about this metaphor either, but it could be useful in some contexts and while also misunderstood in others. The same goes for most jargon.
  - darkerside 7y ago
    Low hanging fruit is a point in time tactic, not a generalized recommendation
    - hinkley 7y ago
      And when you do that for a very long time they call it short-term thinking. Myopia. Missing the forest for the trees. Plenty of that going around.
  - noizejoy 7y ago
    The “low hanging fruit” makes a lot more sense in a hunter/gatherer context than in an agricultural context
- discreteevent 7y ago
  I doubt many people think of it from the perspective of a fruit grower. I think of it as picking wild fruit, in which case the analogy mostly works.
  - hinkley 7y ago
    It is a common attribute of 'civilian' fruit trees that the ground beneath them is littered with rotting fruit (and that fruit is a pathogen vector). We miss a lot of opportunities, and not taking them has lingering consequences.
BiteCode_dev 7y ago
Humans have the tendency to think they understand something if they have a label on it. It helps them take decisions faster, whether it's good or bad. If you tell people you don't know what causes their respiratory illness, they may think you are incompetent, and feel helpless. If you tell them they have acute bronchitis, suddenly they feel empowered. It just means they have an inflammation of the bronchi. It says nothing of the direct cause, nor the indirect cause: a virus, tobacco, dust, air pollution or asthma. It says nothing about how bad it is. They don't even know really what it implies for their body or their treatment. But now they think they know. Call something a democracy, and nobody will check if it is. Call something a privilege, and people will start to want it, working hard for it. Models can be used to help your interact with the world. But they are also used to avoid interacting with the world: if you use them as labels, instead of comparing the model with reality, you get a shortcut to take decisions. It becomes a mere name to justify a decision process instead of a tool for the decision process.
- LudwigNagasena 7y ago
  >If you tell them they have acute bronchitis, suddenly they feel empowered. In the The Imaginary Invalid (17th century) Molliere makes fun of it. Doctor explains how opium works through its "dormitive virtue."
- mattkrause 7y ago
  I'm not sure I agree. While technically bronchitis could be any inflammation of the brochial tubes, a diagnosis of bronchitis usually means that that it's the patient's major problem. It'd be odd to describe a cancer as "bronchitis" even if it involved brochi. So you do learn something, in a Griceian sense, from the relatively uninformative diagnosis. There's also the implication that there are some tools for managing it, the doctor's seen it before, etc all of which are reassuring.
- dchyrdvh 7y ago
  "Mind is the great destroyer of reality" - a quote from one of those ancient dusty books. Our mind is a machine that creates models and labels for everything, to avoid dealing with the illigible reality.
schnable 7y ago
Which leads to the saying that this headline is a play on: "All models are wrong, but some are useful." (George E.P. Box) We probably should be repeating that a lot to make sure people hear it.

hinkley 7y ago

It’s funny, but when you point out that a bad analogy is actually pretty accurate if you actually know anything about the other concept, people don’t want to talk about it any more. We all know that hill climbing algorithms are often naive and sometimes hilariously wrong. Nobody will disagree with you about this, until you start talking about prioritizing work, and then everyone vigorously defends their favorite hill climbing algorithm, from task management to performance tuning. My preferred strategy for optimization more closely resembles how a fruit grower would pick fruit, and the term, “low-hanging fruit” galls me because you would go bankrupt using that strategy. And probably lose most of your trees to disease. It could be a quite good analogy, if the lessons taken from it weren’t just so childishly naive.

edmundsauto 7y ago
Can you elaborate on your preferred strategy?
- hinkley 7y ago
  I'm gonna do this as two posts to split up the how and the why. Here is why and some hints at how: Perf analysis was how I got through some of my most mundane college homework assignments instead of (or sometimes, in addition to) procrastinating. Then during my formative years I saw the following script play out several times: Management and sales have a problem. They say the customers are upset that the code is so slow (maybe the customer has expectations, or maybe a competitor or their old system went a lot faster). It's taking 300 units of time now and they think it should take 100. So the leads and the heavies wander off and come back with 20% easy, the next 10 harder, 6, 4, and then they declare that they've done all they can. So now we're around 180 units of time. That's all the customer gets. See this <flame chart precursor> chart? Everything is nice and flat. Our hands are tied now. Business is... not happy. They're glad they have something to show the customer, but it's really just a bone. So between tasks I'd start poking around, because why not. A pinch here, an architectural tweak there, and maybe 25% just gruntwork, and I'd find another 40% out of the "we've done all we can" code, which is just a hair away from what the business asked for. You can imagine that you invite a tremendous amount of well-deserved scrutiny on yourself doing stuff like this. But that's okay: the most important part of optimization is not the code speed but the bang for the buck. You can certainly interpret "Low hanging fruit" to fulfill that concern, in practice most people don't (for some, lack of imagination, for others, ego-protection). But once you consider everything, including, for instance, externalities on QA, or the misapprehensions of customers who expect every release to be faster... there are a lot of more complicated things to worry about than how to rearrange this block of code to improve branch prediction. Not touching it for 10% is just a heuristic.
- hinkley 7y ago
  Performance is just budgeting. It's opportunity costs (versus expected return on the investment). It's time, materials, risk/scheduling uncertainty, and in many cases, externalities. Interestingly enough, I don't think you need to explain any of this to game designers. They seem to know this. It even appears to be the dogmatic in some circles. When you are way over budget, every cost needs to go on the table, but your end goal is one of the most important pieces of information. That should go on the wall, in large block letters. Look at a real budget. If you are deep in the red, it doesn't matter that your car is only 10% of how much you spend a month. What matters is that the car is 25% of your target. That is a ridiculous amount to spend on your car, even though your house is way more. Even if you spend an ungodly amount on dry cleaning or coffee, you're going to have to trade down for a cheaper car. You don't need charts to tell you this. In fact all the charts can do is convince you that nothing needs to be done. They allow you to bargain. Therefore, I don't give a shit if you think that this function which accounts for "only 5%" of the current run time "is fine". It's 13% of our time budget on some mundane task that doesn't really make us money. It is not fine. You cannot justify 1/8th of our budget for this thing. It's gonna be fixed, and if you have no idea how, you'd better start thinking about it now, because we're gonna come back to it soon and if you don't have a solution then you might find that code is now someone else's responsibility. But I said something about the 'fruit tree' analogy and I'm a full page of text in without a peep. So the thing is that when you're thinking about big budgetary changes, it's more manageable to do it one subject matter at a time. Pick a 'ripe' area and glean it for everything it's worth. So I might pitch that I think I can get 30% improvement out of the edit/update code. Can I have this much time to do it? Okay. I might get half of that goal in the first couple weeks. Half of the rest in the next, but at the end of it, the last thing I touch is probably only going to get 3%. But 3% is 10% of what I promised, so I can justify it, and if they aren't complaining about the timeline, I may try to squeeze in some 1-2% things at the end in a way I can pull the plug if it looks too risky. The important thing to note here is that it will never, ever be as cheap as it is right now to touch that 3% code. Everyone is already thinking about it, everyone knows to keep an eye on it, and new test plans are being developed, documentation and customer training plans are being updated. If I don't touch it now, that target of opportunity might never come again. That 3% plays out everywhere in the code, and I will never be able to get budget to fix any of them (I'll have to sneak them in on a refactor or leave them forever). A dozen of these is not that hard to end up with, and our code is 40% slower than it could be (or worst of all, than a competitor). If that's just the time for a button click, fine. If it's on our slowest operation and things time out, customers with larger data can't function, or they have to buy the absolute most expensive machines instead of slightly better than average? That's not fine. But everyone feels perfectly justified in never doing anything about it. Next release, I can pitch for 25% in some other functional area, and another, and another. Last time I did this, it took over 20 months before I ran out of functional areas to work on. Every release for 'years' was faster than the previous. At the end, I knew that code better than anyone (an important point I omitted - focusing that intently on one section teaches people a hell of a lot about the code they otherwise would never know). I probably could have swung through for another 10% in every area, but by then I was ready for a new gig.
tartoran 7y ago
The low hanging fruit metaphor meaning do the simplest or easiest work first is not really applicable to fruit growing but seems to have been coopted by the busieness world instead, then we started using it too in our contexts. Not crazy about this metaphor either, but it could be useful in some contexts and while also misunderstood in others. The same goes for most jargon.
- darkerside 7y ago
  Low hanging fruit is a point in time tactic, not a generalized recommendation
  - hinkley 7y ago
    And when you do that for a very long time they call it short-term thinking. Myopia. Missing the forest for the trees. Plenty of that going around.
- noizejoy 7y ago
  The “low hanging fruit” makes a lot more sense in a hunter/gatherer context than in an agricultural context
discreteevent 7y ago
I doubt many people think of it from the perspective of a fruit grower. I think of it as picking wild fruit, in which case the analogy mostly works.
- hinkley 7y ago
  It is a common attribute of 'civilian' fruit trees that the ground beneath them is littered with rotting fruit (and that fruit is a pathogen vector). We miss a lot of opportunities, and not taking them has lingering consequences.

BiteCode_dev 7y ago

Humans have the tendency to think they understand something if they have a label on it. It helps them take decisions faster, whether it's good or bad. If you tell people you don't know what causes their respiratory illness, they may think you are incompetent, and feel helpless. If you tell them they have acute bronchitis, suddenly they feel empowered. It just means they have an inflammation of the bronchi. It says nothing of the direct cause, nor the indirect cause: a virus, tobacco, dust, air pollution or asthma. It says nothing about how bad it is. They don't even know really what it implies for their body or their treatment. But now they think they know. Call something a democracy, and nobody will check if it is. Call something a privilege, and people will start to want it, working hard for it. Models can be used to help your interact with the world. But they are also used to avoid interacting with the world: if you use them as labels, instead of comparing the model with reality, you get a shortcut to take decisions. It becomes a mere name to justify a decision process instead of a tool for the decision process.

LudwigNagasena 7y ago
>If you tell them they have acute bronchitis, suddenly they feel empowered. In the The Imaginary Invalid (17th century) Molliere makes fun of it. Doctor explains how opium works through its "dormitive virtue."
mattkrause 7y ago
I'm not sure I agree. While technically bronchitis could be any inflammation of the brochial tubes, a diagnosis of bronchitis usually means that that it's the patient's major problem. It'd be odd to describe a cancer as "bronchitis" even if it involved brochi. So you do learn something, in a Griceian sense, from the relatively uninformative diagnosis. There's also the implication that there are some tools for managing it, the doctor's seen it before, etc all of which are reassuring.
dchyrdvh 7y ago
"Mind is the great destroyer of reality" - a quote from one of those ancient dusty books. Our mind is a machine that creates models and labels for everything, to avoid dealing with the illigible reality.

schnable 7y ago

Which leads to the saying that this headline is a play on: "All models are wrong, but some are useful." (George E.P. Box) We probably should be repeating that a lot to make sure people hear it.