9 ms·
What are the most important statistical ideas of the past 50 years?
- oneoff786 5y agoShap values and other methods to parse out the inner workings for “black box” machine learning models. They’re good enough that I’ve grown fond of just throwing a light gbm model at everything and calling it a day for that sweet spot of predictive power and ease of implementation.
- hackernewds 5y agoWould be so kind to share an example or resources to learn about this?
- screye 5y ago[1] Scott is the leading authority on all things SHAP (he wrote the seminal paper on it). [2] Chris Molnar's interpretable learning book has chapters on Shapeley values and SHAP. If you'd prefer text instead of video. [1] https://www.youtube.com/watch?v=B-c8tIgchu0 https://www.youtube.com/watch?v=B-c8tIgchu0 [2] https://christophm.github.io/interpretable-ml-book/shapley.html https://christophm.github.io/interpretable-ml-book/shapley.h...
- magneticnorth 5y agoSeconding Chris Molnar's excellent writeup. I also find the readme & example notebooks in Scott Lundberg's github repo to be a great way to get started. There are also references there for the original papers, which are surprisingly readable, imo. https://github.com/slundberg/shap https://github.com/slundberg/shap
- hervature 5y agoStrongly disagree. Shapley values and LIME give a very crude and extremely limited understanding of the model. They basically amount to a random selection of local slopes. For instance, if I tell you the (randomly selected) slopes of a function are [0.5, 0.2, 12.4, 1.1, 2.6] which the average is 3.3, can you guess anything? You might notice it is monotonic (maybe) but you certainly don't guess it is e^x.
- LeanderK 5y agoI would say that our ML models are not predictable enough yet at the local neighbourhood to really trust LIME. Adversarial examples prove that you just can't select a small enough range since you can always find those even for super tiny distances.
- oli5679 5y agoIt depends on your domain. If you have a relatively small number of features and your features have relatively stable but not necessarily linear partial dependence, then shap can unpack them pretty well. E.g. value of a house it might not necessarily increase linearly in squarefoot, distance from city centre and other factors so a random-forest/gbt might do better than a linear model without careful feature engineering. Shap is a brilliant tool to unpack the dependencies this model has learned. There will be other cases where a linear model is adequate, so you can get the explainability just from the coefficients, or where there are very large numbers of features with complicated nested structure. In either case shap is not too useful. But the 'fit lightgbm and explain it with shap' is a decently pragmatic approach for lots of tabular data machine-learning problems.
- oneoff786 5y agoMaybe. But I don’t care about estimating the slopes. I just need to know big values of feature X pull the model prediction up and Small values do nothing. If you really need to unpack it, cluster your shaps. To see what kinds of predictions are being made
- teruakohatu 5y ago> I’ve grown fond of just throwing a light gbm model at everything and calling it a day It is not always a good idea to do that. Always try different methods, there is no ultimate method. At the very least OLS should be tried and some other fully explainable methods, even a simple CART like method.
- csee 5y agoOLS is my default go to. It outperforms a random forest so often in small data real world applications over nonstationary data, and model explainability is built into it. If I'm working in a domain with stationary data then I'd tilt more to the forest (due to not having to engineer features, and the inbuilt ability to detect non-linear relationships and interactions between features).
- deleted 5y ago[deleted]
- sockpuppet69 5y ago
- deepsquirrelnet 5y agoGood review article. I always enjoy browsing the references, and found "Computer Age Statistical Inference" among them. Looks like a good read, with a pdf available online.
- mjb 5y agoIt's a great book. Short and to-the-point. Highly recommended.
- appleflaxen 5y agoFor others: https://hastie.su.domains/CASI_files/PDF/casi.pdf https://hastie.su.domains/CASI_files/PDF/casi.pdf
- mlcrypto 5y ago
- ThouYS 5y agoBootstrap resampling is such a black magic thing
- CrazyStat 5y agoOne way to approach the bootstrap is as sampling from the posterior mean of a Dirichlet Process model with a noninformative prior (alpha=0).
- derbOac 5y agoIt's just Monte Carlo simulation using the observed distribution as the population distribution.
- civilized 5y agoIt's not the mere description of the procedure that people find mysterious.
- btown 5y agoThis has big "A monad is just a monoid in the category of endofunctors" energy
- grayclhn 5y ago1) It’s not — there are lots of procedures called “the bootstrap” that act differently. 2) The fact that “substitute the data for the population distribution” both works and is sometimes provably better than other more sensible approaches is a little mind blowing. Most things called the bootstrap feel like cheating, ie “this part seems hard, let’s do the easiest thing possible instead and hope it works.”
- derbOac 5y agoI think it's brilliant as an idea but not particularly mysterious after the fact or something. I love it, but think it's brilliant in part because it's so transparent and simple. I agree "bootstrap" has expanded a bit in meaning but I think it's basically the same idea. I used to think it was cheating but have realized there is a cost to it, which is replication. For many things it's just impractical in terms of computation time. So although it's great, it requires a lot.
- oxff 5y ago"Throw more compute at it"
- Q6T46nT668w6i3m 5y agoThe authors agree. It’s mentioned a handful of times.
- oraoraoraoraora 5y agoStatistical process control has been significant over the last 100 years.
- avs733 5y agoI would agree with you but they are speaking to a different audience. This is in a journal for statistics researchers and theorists. These would all be things that would inform the creation of pragmatic tools like SPC.
- bell-cot 5y agoThe most important statistical idea of the past 50 years is the same as the most important statistical idea of the 50 years before that: "Due to reduced superstition, better education, and general awareness & progress, humans who are neither meticulous statistics experts, nor working in very constrained & repetitive circumstances, will understand and apply statistics more objectively and correctly than they generally have in the past." Sadly, this idea is still wrong.
- urthor 5y agoI disagree in the very strongest terms. We barely have any concept of how education has penetrated our society, to a far greater degree, now than fifty years ago. We as a population are FAR more educated, far less ignorant, and generally speaking extraordinarily better off in our daily lives than in the past. It's only because the phenomenon of social media, which has crowd sourced ignorance and hurled it in front of our eyes, that we perceive we're in an unenlighted age/trajectory.
- aabajian 5y agoDidn't read the document, but hopefully it mentions PageRank, the prime example of using probabilistic graphical models to rank nodes in a directed graph. More info: https://www.amazon.com/Probabilistic-Graphical-Models-Principles-Computation/dp/0262013193 https://www.amazon.com/Probabilistic-Graphical-Models-Princi... I've heard that Google and Baidu essentially started at the same time, with the same algorithm discovery (PageRank). Maybe someone can comment on if there was idea sharing or if both teams derived it independently.
- mach1ne 5y agoDidn't Larry Page and Sergey Brin openly publicize the PageRank algorithm? It'd seem more likely that Baidu just copypasted the idea.
- swyx 5y agoBaidu's patent predated theirs by 2 years https://news.ycombinator.com/item?id=30419414 https://news.ycombinator.com/item?id=30419414
- oneoff786 5y agoThe basic concept behind page rank is pretty obvious. If you stare at a graph for a while, it’ll probably be your big idea if you try to imagine centrality calculations. Implementing it and catching edge cases isn’t trivial
- andi999 5y agoI heard that the first approach of google was using/adapting an published algorithm which was used to rank scientific publications from the network of citations. Not sure if this is the algorithm you mentioned though.
- divbzero 5y agoThe ranking of scientific publications based on citations you’re describing is impact factor [1]. I haven’t heard that as an inspiration for Larry Page’s PageRank [2] but that is plausible. [1]: https://en.wikipedia.org/wiki/Impact_factor https://en.wikipedia.org/wiki/Impact_factor [2]: https://en.wikipedia.org/wiki/PageRank https://en.wikipedia.org/wiki/PageRank
- andi999 5y agoIdentifying p value hacking.
- hackernewds 5y agowhat does that mean?
- ldiracdelta 5y agoI believe it is referencing "The Replication Crisis" https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
- pacbard 5y agop-hacking is a research "dark pattern" where a researcher fits several similar models and reports only the one that has the significant p-value for the relationship of interest. This strategy is possible because p-values are themselves stochastic and a researcher will find one significant p-value for every 20 models that they run (at least on average). p-hacking could also refer to pushing a p-value close to the significant cut-off (usually 0.05) by modifying the statistical model slightly until the desired result is achieved. This process usually involves the inclusion of control variables that are not really related to the outcome but that will change the standard errors/p-values. Another way to p-hack is to drop specific observations until the desired p-value is reached. This process usually involves removing participants from a sample for a seemingly legitimate reason until the desired p-value is achieved. Usually identifying and eliminating a few high leverage observations is enough to change the significance level of a point estimate. Multiple strategies to address p-hacking have been proposed and discussed. One of the most popular ones is pre-registration of research designs and models. The idea here is that a researcher would publish their research design and models before conducting the experiment and they will report only the results from the pre-registered models. This process eliminates the "fishing expedition" nature of p-hacking. Other strategies involve better research designs that are not sensitive to model respecification. These are usually experimental and quasi-experimental methods that leverage an external source of variation (external to both the researcher and the studied system, like random assignment to conditions) to isolate the relationship between two variables.
- prout69098132 5y ago
- prout69098132 5y ago
- prout69098132 5y ago
- prout69098132 5y ago
- csee 5y agoMeta-analysis techniques like funnel plots.
- pacbard 5y agoMeta-analysis is an application of idea #4 (Bayesian Multilevel Models) in the article. What makes meta-analysis special within a multilevel framework is that you know the level 1 variance. This creates a special case of a generalized multilevel model where you leverage your knowledge of L1 mean and variance (from each individual study's results) to estimate the possible mean and variance of the population effect. The population mean and variance is usually presented in funnel plots where you can see the expected distribution of effect sizes/point estimates given a sample size/standard error. Researchers have also started to plot actual point estimates from published papers in this plot, showing that most of the published results are "inside the funnel", a result that is usually cited as evidence of publication bias. In other words, the missing studies end up in researchers' file drawers instead of being published somewhere.
- bobbyd2323 5y agoBootstrap
- ModernMech 5y agoKalman published his filter in 1960... a little over 50 years but I'd say it's worthy to mention given its huge impact. The idea that we can use multiple noisy sensors to get a more accurate reading than any one of them could provide enables all kinds of autonomous systems. Basically everything that uses sensors (which is essentially every device these days) is better due to this fact.
- savant_penguin 5y agoAnd it uses almost all of the tricks in the book in a single analytic model Statistics+optimization+dynamic systems+linear algebra
- vanattab 5y agoThe most important to me is "There are three kinds of lies in this world. Lies, damn lies, and statistics." Not attacking the mathematically field of statistics just pointing out that lots of people abuse statistics in an attempt to get people to behave as they would prefer.
- klysm 5y agoI would argue the problem lies in our complete lack of teaching around statistical epistemology. It’s such a critical concept in my eyes and it seems just totally glazed over looking back at my education.
- uoaei 5y agoIMO, kernel-based computational methods are by far the most important overlooked advances in the statistical sciences. Kernel methods are linear methods on data projected into very-high-dimensional spaces, and you get basically all the benefits of linear methods (convexity, access to analytical techniques/manipulations, etc.) while being much more computationally tractable and data-efficient than a naive approach. Maximum mean discrepancy (MMD) is a particularly shiny result from the last few years. The tradeoff is that you must use an adequate kernel for whatever procedure you intend, and these can sometimes have sneaky pitfalls. A crass example would be the relative failure of tSNE and similar kernel-based visualization tools: in the case of tSNE the Cauchy kernel's tails are extremely fat which ends up degrading the representation of intra- vs inter-cluster distances.
- teshier-A 5y agoMy experience with MMD is that unless you've been using it and are familiar with other kernel methods, you probably won't know what to do with it : what kernel do I use ? how can I test for significance (in any sense of the word) ? Add the (last I checked) horrendous complexity, to me it looks like a less usable mutual information (or KL div) and without all the nice information theory around it.
- NavinF 5y agoI see a ton of papers about kernel-based stuff, but zero real applications. Feels like that whole subfield came and went before I got into ml.
- westcort 5y agoNot in the past 50 years, but more like the past 80 years, nonparametric statistics in general are pretty amazing, though underused. Look at the Mann-Whitney U test and the Wilcoxon Rank Sum. Those tests require very few assumptions.
- dannykwells 5y agoThese are the standard tests in most of biology at this time. Not underused at all. Very powerful and lovely to not have to assume normality.
- pandoro 5y agoSolomonoff Induction. Although proven to be uncomputable (there are people working on formalizing efficient approximations) it is such a mind-blowing idea. It brings together Occam's razor, Epicurus' Principle of multiple explanations, Bayes' theorem, Algorithmic Information Theory and Universal Turing machines in a theory of universal induction. The mathematical proof and details are way above my head but I cannot help but feel like it is very underrated.
- spekcular 5y agoStatistics is an applied science, and Solomonoff induction has had zero practical impact. So I feel it's not underrated at all, and perhaps overrated among a certain crowd.
- derbOac 5y agoMy impression, for what it's worth or not worth, is that this area is sort of what Bayesian statistics was 50 years ago. Something that is fundamental but is difficult to implement so doesn't have much impact practically speaking at a given moment in time.
- jll29 5y agoOff-the-cuff, i.e. without digging deeply into a set of history of statistics books: Tied 1st place: • Markov chain Monte Carlo (MCMC) and the Metropolis–Hastings algorithm • Hidden Markov Models and the Viterbi algorithm for most probable sequence in linear time • Vapnik–Chervonenkis theory of statistical learning (Vladimir Naumovich Vapnik & Alexey Chervonenkis) and SVMs 4th place: • Edwin Jaynes: maximum entropy for constructing priors (borderline: 1957) Honorable mentions: • Breiman et al.'s CART (Classification and Regression Trees) algorithm (and Quinlan's C5.0 extension) • Box–Jenkins method (autoregressive moving average (ARMA) / autoregressive integrated moving average (ARIMA) to find the best fit of a time-series model to past values of a time series) (The beginning of the 20th century was much more fertile in comparison - Kolmogorov, Fisher, Gosset, Aitken, Cox, de Finetti, Kullback, the Pearsons, Spearman etc.)
- dlg 5y agoI generally agree with you. However, as a pedantic note, Metrolopis, Rosenbluth, Rosenbluth, Teller and Teller was in 1953 and Hastings was 1970.
- enriquto 5y agoMy favourite is Mandelbrot's heuristic converse of the central limit theorem: the only variables that are normal are those that are sums of many variables of finite variance.
- klysm 5y agoI have not heard that before, but that seems like a very strong statement to know! How does interact with sample size though? Seems hard to formulate precisely…
- enriquto 5y ago> Seems hard to formulate precisely… Well, it's an heuristic ;) The idea is that if you ever find a distribution that seems to be normal, you look for the finite variance variables that are lurking behind it. If you do not find these variables, then your distribution was not likely normal.
- datastoat 5y agoValidation on holdout sets. When I was a student in the 1990s, I was taught about hypothesis testing (and all the hassle of p-fishing etc.), and about Bayesian inference (which is lovely, until you have to invent priors over the model space -- e.g. a prior over neural network architectures). These are both systems that tie themselves in epistemological knots when trying to answer the simple question "What model shall I use?" Holdout set validation is such a clean simple idea, and so easy to use (as long as you have big data), and it does away with all the frequentist and Bayesian tangle, which is why it's so widespread in ML nowadays. It also aligns statistical inference with Popper's idea of scientific falsifiability -- scientists test their models against a new experimental data, data scientists can test their model against qualitatively different holdout sets. (Just make sure you don't get your holdout set by shuffling, since that's not what Popper would call a "genuine risky validation".) The article mentions Breiman's "alternative view of the foundations of statistics based on prediction rather than modeling". Breiman does make a big deal of evaluation on holdout sets; but his "prediction" idea isn't general enough, since it doesn't accommodate generative modelling (e.g. GPT, GANs). I think it's better to frame ML in terms of "evaluating model fit on a holdout set", since that accommodates both predictive and generative modelling.
- anxrn 5y agoVery much agree with the simplicity and power of separation of training, validation and test sets. Is this really a 'big data' era notion though? This was fairly standard in 90s era language and speech work.
- datastoat 5y agoBig enough data that you can afford not to use some of it for training! Different disciplines hit this threshold at different times -- language and speech much earlier, as you say; clinical trials not there yet. Maybe we could talk about two cardinalities of "big" data. The first is when you can afford not to use all of your data for training. The second is when you can usefully fit highly overparameterized models.
- 5y ago
- codeulike 5y agoPublishing your data, your code and your results in an open format (workbooks etc)
- analog31 5y agoThis is a side issue, but I wonder what changes could be made to the teaching of statistics, most importantly "stats for scientists" taught in college and even high school. What occurs to me right off the bat is that I learned stats by the use of formulas to boil data sets down to answers. For example, null hypothesis significance testing and the dreaded p-value. We had to do it this way, as widespread availability of computers was still on the horizon. Today, computation is easy and cheap. I wonder if we could learn stats in a different way, perhaps starting by just playing with data, and simulated random numbers, graphing, and so forth. Can something like null hypothesis testing be taught primarily through bootstrapping, with the formulas introduced as an aside? Yes I know that statistics is a formal branch of math, with theorems and proofs. I took that class. But the students who take "stats for scientists" don't ever see that side of it. Understanding the formulas without seeing the proofs is like trying to learn freshman physics without calculus.
- 41b696ef1113 5y agoThat is essentially how Allen Downey approaches statistical education: the analytical solutions came first because we lacked the computational power. Now that we have cheap computation, we should exploit that to develop better intuition. His Bayesian book[0] is available as Jupyter notebooks. [0]: https://allendowney.github.io/ThinkBayes2/ https://allendowney.github.io/ThinkBayes2/
- analog31 5y agoThat's pretty cool. I actually would like to see K-12 math education use more computation and data, with less of a focus on algebra (expression manipulation). This would be more reflective of how regular people, including STEM people, use math outside of school.
- in9 5y agoIt is a very good idea, but some resources can give very wrong/opaque views about procedures, by showing only model.fit(X,y) and model.predict(X).
- kqr 5y ago
- j7ake 5y agoIn today’s world of big data, I would say The James-Stein estimator is one of the most important concepts (although it was 1950s). It was a shock to the world when it was shown that least squares methods (a method used since Gauss) are actually not optimal in most cases. https://en.m.wikipedia.org/wiki/James–Stein_estimator https://en.m.wikipedia.org/wiki/James–Stein_estimator
- abiro 5y agoSurprised that nobody mentioned the first one in the list: Counterfactual Causal Inference The formalisation of casual inference by Pearl is in my opinion a development comparable to the invention of calculus in mathematics: Suddenly the problem space solvable with statistics became at least an order of magnitude larger and we're only starting to see the benefits in other sciences.
- disgruntledphd2 5y agoI would love to believe this is true, and I think that theoretically it's super important, but I've never had success with this in practical applications. Hopefully one day I'll be able to make use of it.
- morelandjs 5y agoIt’s a tough field to become proficient in because you rarely get to validate your process. But I’m starting to get a better understanding of double ML, and it really does work in a provable way.
- morelandjs 5y agoCausal inference has come a long way, e.g. double machine learning is pretty nifty.
- thedudeabides5 5y agoLetting users do stats on business data in spreadsheets