4 ms·
I've also found very similar issues with extremely "data driven" organizations that live and die by A/B test performance. It's not that there is anything fundam
by hn_throwaway_99 5y ago
I've also found very similar issues with extremely "data driven" organizations that live and die by A/B test performance. It's not that there is anything fundamentally wrong with the science behind A/B testing, it's just that individuals are incentivized to run tests on things that are easily measured. Things where results will take a long time to show, or things that may be initially disruptive but then beneficial, are discounted.
I see this theme in many, many areas: business, sports, politics, academia, etc. When we have tons of data and a desire to make things as "objective" as possible, it's easy to get stuck in homogeneous "local maxima" because we just grade by the things that are easiest to measure.
- User23 5y agoI was working at a SV company and one pf my peers told the PMs that their A/B test design was terrible and no meaningful conclusions could be drawn from it. Naturally he was ignored. And he is a physics PhD who has worked at Los Alamos, so I think he could fairly be called an expert in experiment design.
- burnished 5y agoThe flip side is that his standards could be too high for a practically useful test, the standards for searching for something like ground truth and the standards for searching for some more money are different. In the former being right for the wrong reasons is still being wrong, in the latter no one cares why you were right.
- bpodgursky 5y agoA good role of thumb I use is — if you need a P-test to tell you your result is significant, there's basically no chance you have found something revolutionary. Real breakthroughs are... obvious, both qualitatively and quantitatively.
- tux3 5y agoIt should be easy to find counterexamples, look at experimental physics. The measure of the Higgs was not a small insignificant result, but it didn't go from unexpected to obvious. It was expected and honestly unknown, until new instruments and fancy statistics could just barely push it over the significance line. It was in a sense a disapointment, but arguably the last big breakthrough for the standard model.. Look at gravitational waves. Suddenly we have a whole new force we can measure the universe with! This one breakthrough opened a whole new field of study. And yet it's at the edge of what we can measure, and even when the data looks consistent with the explanation, it's far from an obvious thing.
- a1369209993 5y agoBoth of those are things that are not revolutionary, but rather logical extensions of what the established models already predict. The Higgs boson was a direct confirmation of what is literally called the "standard model" of particle physics, and gravitational waves follow fairly directly from orbital decay + conservation of energy. Actually detecting them is a fairly impressive feat of engineering, but scientifically it's "yep, the thing we figured was probably true turned out to be true".
- heisenzombie 5y agoI should point out that many fields have commonly used p ≤ .05 as a "significance line". The significance line in particle physics is generally 5-sigma, or p ≤ 3x10⁻⁷. The Higgs is now at p ≤ 1x10⁻⁹. I think that meets GP's criteria of obviousness, at least in spirit. Also -- you can look at the plots in the Higgs discovery papers and go "oh yeah, there's definitely a bump there".
- bpodgursky 5y agoYeah exactly. Once identified, the measurements and graphs made it blindingly obvious that they'd found it.
- asplake 5y agoRelated to that, if (in business) you need detailed quantitative analysis to tell you which of two paths to pursue, it probably doesn’t much matter either way. And extrapolations are a dangerous business anyway.
- tomp 5y agoHm... both Tesla and SpaceX took about 10 years to be "obvious" breakthroughs.
- bpodgursky 5y agoAnd neither of them ever needed a P-test to show it.
- dnautics 5y agoI think this is categorically different. What you are talking about is a bias to work on things that are measurable followed by goodharts law; the linked paper is more like "if you throw money at a problem and increase participation in the field it becomes more difficult to separate wheat from chaff due to sheer volume".
- Bohbc 5y agoYup its like sorting through YouTubes 7 trillion 'How to make an omelette' videos. You know there are great ones in there but its a crap shoot what gets found. More awareness reqd about the Explore-Exploit tradeoff.
- hn_throwaway_99 5y agoMy point is that researchers are "graded" on things that are easily measured, things like publication quantity and "citation impact factor", which leads to a sort of monoculture in academia.
- dnautics 5y agothat is at best tangentially related to the issue in OP.
- makeitdouble 5y agoFrom the abstract I see OP's point to directly apply. Simplistic metrics (number of citations for instance) applied to papers push people toward "easy" decisions (reading the "best" papers in priority), and they don't get to discuss the more innovative work. I see the parallel on people too focused on data and easily measurable properties, missing "riskier" works that would rely on subjective evaluation or less clear selection criteria.