6 ms·

The Dangers of Intermediate Targets: IQ, Cholesterol, and 99%-ile Latency

66 points by kryptiskt 12y ago

emmfr 12y ago
Useful related concept in the social sciences: Goodhart's law [1] “When a measure becomes a target, it ceases to be a good measure.” [1] https://en.wikipedia.org/wiki/Goodhart's_law https://en.wikipedia.org/wiki/Goodhart's_law
- paol 12y ago
  With the corollary that every measure always becomes a target...
majke 12y ago
I hate blog posts without dates. Is that a new blog post or old? Anyway, to the merit: > But the only trick is that there is no trick Yes a thousand times. Very well put. This is my understanding of the "Premature Optimization" mantra: it's either you got everything right, and there is little need for miro-optimizing, or everything wrong and micro-optimizing won't get you much.
- venomsnake 12y ago
  Well no ... premature optimization is something else. When you start a project - you make some assumptions about load, shape whatever. Optimizing based on these assumptions before being validated by the real world testing is sub optimal. You don't know where the bottlenecks will be, or how users will interact with your software.
  - tjradcliffe 12y ago
    Premature optimization includes both what you describe and what the GP talks about. Adopting the general rule, "No optimization without quantification" helps in all cases: validate those assumptions early in the design process (with prototypes, user observations, whatever) and with regard to optimization of code later on, profile it intelligently to identify bottlenecks, don't just assume the issue arise from what seem like the obvious culprits.
- peloton 12y ago
  I, too, am not a fan of blog posts without dates. The archive view suggests it's new: http://danluu.com/blog/archives/ http://danluu.com/blog/archives/
- adsofhoads 12y ago
  I found the date on the RSS feed (2015-3-5 21:54).
atemerev 12y ago
This is how market crashes now happen. People at hedge funds optimize to be right 99.99% of times, ignoring those huge black swans at 0.01%.
- grandalf 12y ago
  The idea of a black swan event is that it's beyond normal expectations. This is a bit misleading because we all know such events will occur. Because we know this, how should we act? Suppose you are on a small island with 50 other people. You know that there will eventually be a tsunami. What percentage of your time do you spend on tsunami preparedness vs other productive pursuits? When it comes to regulation, we know catastrophic events will occur eventually, but the time horizon is distant enough (where the probability of such events approaches 1) that we collectively decide to roll the dice and accept the risk. Then when such an event occurs we pretend that nobody knew it would happen. We did know it would happen, we just blindly hoped it would not happen during our lifetime|tenure|time-in-office, etc. The further we are from a previous event, the easier it becomes (politically) to erode the protections that are in place. So much lobbying successfully attempts to do this. This is predictable, and as our society evolves in finance more and more risk can be arbitraged, and so transfer of risk from the individual or firm to the state is easier than ever before. Not only will a firm fail but its pension fund will go away and the insurance company that insured its pension fund's underwriting capital is only sufficient to handle two such events per year because firms lobbied to have the requirements reduced. The capital freed due to lessened underwriting requirements now funds some other industry which is propped on this risk transfer. So over time we move in lock step to ignore systemic risk and to double down and double down again on existing assumptions about how much underwriting is necessary, etc. Quantitative Easing is the intentional erosion of risk capital standards, intended to put more capital into the economy to do exactly this. While it's a relief that the bailout largely worked, it's discomfiting to think about the many perverse incentives that have now been added to the mix.
- yellowstuff 12y ago
  This is pretty much what happened to LTCM (http://en.wikipedia.org/wiki/Long-Term_Capital_Management http://en.wikipedia.org/wiki/Long-Term_Capital_Management) in 1998. They owed so much money that the Fed had to help organize a bailout from their creditors (although no government money was involved.) This didn't cause a market crash, though. I don't think hedge funds have ever played a major role in causing a market crash. If you were to assign blame for the 2008 crash I think that hedge funds played a smaller role than banks, mortgage originators, AIG, Fannie and Freddie, the ratings agencies, and the US Congress.
carbocation 12y ago
The treatment of cholesterol could use some clarification. As the author notes, statins reduce MI risk and they reduce cholesterol (LDL cholesterol specifically). He then addresses torcetrapib, which was designed to raise HDL-cholesterol. This was testing quite a different hypothesis from that of statins. I don't think it makes sense to lump these two together and say "sometimes cholesterol works and sometimes it doesn't". Rather, it's more fruitful to see the biomarkers separately.

66 points by kryptiskt 12y ago

emmfr 12y ago

Useful related concept in the social sciences: Goodhart's law [1] “When a measure becomes a target, it ceases to be a good measure.” [1] https://en.wikipedia.org/wiki/Goodhart's_law https://en.wikipedia.org/wiki/Goodhart's_law

paol 12y ago
With the corollary that every measure always becomes a target...

majke 12y ago

I hate blog posts without dates. Is that a new blog post or old? Anyway, to the merit: > But the only trick is that there is no trick Yes a thousand times. Very well put. This is my understanding of the "Premature Optimization" mantra: it's either you got everything right, and there is little need for miro-optimizing, or everything wrong and micro-optimizing won't get you much.

venomsnake 12y ago
Well no ... premature optimization is something else. When you start a project - you make some assumptions about load, shape whatever. Optimizing based on these assumptions before being validated by the real world testing is sub optimal. You don't know where the bottlenecks will be, or how users will interact with your software.
- tjradcliffe 12y ago
  Premature optimization includes both what you describe and what the GP talks about. Adopting the general rule, "No optimization without quantification" helps in all cases: validate those assumptions early in the design process (with prototypes, user observations, whatever) and with regard to optimization of code later on, profile it intelligently to identify bottlenecks, don't just assume the issue arise from what seem like the obvious culprits.
peloton 12y ago
I, too, am not a fan of blog posts without dates. The archive view suggests it's new: http://danluu.com/blog/archives/ http://danluu.com/blog/archives/
adsofhoads 12y ago
I found the date on the RSS feed (2015-3-5 21:54).

atemerev 12y ago

This is how market crashes now happen. People at hedge funds optimize to be right 99.99% of times, ignoring those huge black swans at 0.01%.

grandalf 12y ago
The idea of a black swan event is that it's beyond normal expectations. This is a bit misleading because we all know such events will occur. Because we know this, how should we act? Suppose you are on a small island with 50 other people. You know that there will eventually be a tsunami. What percentage of your time do you spend on tsunami preparedness vs other productive pursuits? When it comes to regulation, we know catastrophic events will occur eventually, but the time horizon is distant enough (where the probability of such events approaches 1) that we collectively decide to roll the dice and accept the risk. Then when such an event occurs we pretend that nobody knew it would happen. We did know it would happen, we just blindly hoped it would not happen during our lifetime|tenure|time-in-office, etc. The further we are from a previous event, the easier it becomes (politically) to erode the protections that are in place. So much lobbying successfully attempts to do this. This is predictable, and as our society evolves in finance more and more risk can be arbitraged, and so transfer of risk from the individual or firm to the state is easier than ever before. Not only will a firm fail but its pension fund will go away and the insurance company that insured its pension fund's underwriting capital is only sufficient to handle two such events per year because firms lobbied to have the requirements reduced. The capital freed due to lessened underwriting requirements now funds some other industry which is propped on this risk transfer. So over time we move in lock step to ignore systemic risk and to double down and double down again on existing assumptions about how much underwriting is necessary, etc. Quantitative Easing is the intentional erosion of risk capital standards, intended to put more capital into the economy to do exactly this. While it's a relief that the bailout largely worked, it's discomfiting to think about the many perverse incentives that have now been added to the mix.
yellowstuff 12y ago
This is pretty much what happened to LTCM (http://en.wikipedia.org/wiki/Long-Term_Capital_Management http://en.wikipedia.org/wiki/Long-Term_Capital_Management) in 1998. They owed so much money that the Fed had to help organize a bailout from their creditors (although no government money was involved.) This didn't cause a market crash, though. I don't think hedge funds have ever played a major role in causing a market crash. If you were to assign blame for the 2008 crash I think that hedge funds played a smaller role than banks, mortgage originators, AIG, Fannie and Freddie, the ratings agencies, and the US Congress.

carbocation 12y ago

The treatment of cholesterol could use some clarification. As the author notes, statins reduce MI risk and they reduce cholesterol (LDL cholesterol specifically). He then addresses torcetrapib, which was designed to raise HDL-cholesterol. This was testing quite a different hypothesis from that of statins. I don't think it makes sense to lump these two together and say "sometimes cholesterol works and sometimes it doesn't". Rather, it's more fruitful to see the biomarkers separately.