5 ms·
> Traditionally, it’s a cardinal sin in academia to use parameters like these because you can’t say anything interesting about the parameters, but the trick in
by Konohamaru 6y ago
> Traditionally, it’s a cardinal sin in academia to use parameters like these because you can’t say anything interesting about the parameters, but the trick in machine learning is that you don’t need to say anything about the parameters. In machine learning, your focus is on describing y-hat, not β-hat.
This kind of philosophy will cause future generations to see machine learning as something worse than a fad, almost as something in between a fad and crank science. If this encapsulates how all (generally speaking) machine learning operates then we will enter big trouble, if we have not already.
> In machine learning, bad results are wrong if they catastrophically fail to predict the future, and nobody cares much how your crystal ball works, they only care that it works.
This has moved from Cargo Cult Science into numeromancy. It's leveraging the occult (=hidden, incomprehensible parameters) for predicting the future. Because there exist no first principles, nothing can be further interpreted. Only more of the occult can be leveraged in order to make more predictions not amenable to interpretation, which will in turn require MORE occult to make MORE inscrutable predictions, until the heat death of the universe....
And appealing to 80's AI (neural networks) as case precedence further harms the author's case. If ML operates like how AI neural network technology went, then this whole rigamole will go tits up by case precedence as well.
- teruakohatu 6y agoI agree from the perspective that often complex models that cannot be interpreted are created because Neural Networks sound fancy, when much simpler models could be used instead. But on the other hand complex models such as Neural Networks are being used for very wide datasets with incredible amounts of parameters, so that understanding a single parameter's contribution is not useful in the real world. For example, using mouse movement data to classify users into age groups. Knowing that some movement vector adds 0.000067 to the probability of a user being in the 16-25 age group, and so being allowed to watch a movie rated 16+, is not very useful.
- Konohamaru 6y ago> But on the other hand complex models such as Neural Networks are being used for very wide datasets with incredible amounts of parameters, so that understanding a single parameter's contribution is not useful in the real world. In the near future AI and statisticians will need to cooperate by: 1. Finding means for extracting rules/general principles from neural networks. 2. Creating new fields of statistics that can handle multi-dimensional data sets, such as by representing them as small-dimensional datasets that interact (?), which converges to the same multidimensional model when the modeler sutures these interactions (?) together according to a topological structure. We know that we can do this because somehow human beings reason about complex systems successfully.
- hoseja 6y ago>human beings reason about complex systems successfully Have you SEEN economics?
- jwilber 6y agoThere’s a whole field around 1 called explainable ai. Interestingly, one of the SOTA techniques, SHOP values, comes from economics (game theory). Regarding the article, I think it was a good read, but as a data scientist on a research at Amazon, there’s a reason our interviews have shifted from stats-heavy to more cs-heavy: the ability to actually implement and maintain analytic products is just more useful. (Note, this isn’t across the board - we still hire PhD-level candidates to do research).
- NateEag 6y agoIt's conceivable that humans minds are not perfectly modeled by software neural networks. If neural networks are not perfect models of our minds, then your last paragraph does not hold.
- hoseja 6y agoHave you considered that useful predictive models for reality are simply irreducible to being comprehensible by swollen savannah monkey brains? It's not magic. It simply cannot both be explained to a human and be worth anything.
- Konohamaru 6y agoBy the Good Regulator Theorem, if models of reality were incapable of being made comprehensible, then we would have went extinct. What we need to figure out then becomes how to externalize what we know unconsciously.
- Isinlor 6y agoSounds interesting, but I'm not sure if I understand. From Wikipedia: It is stated that "every good regulator of a system must be a model of that system". How is that related to comprehensibility? Or going extinct? A lot of smart people seem to be surprised that anything can be comprehended at all: https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness... The lack of mathematics effectiveness in description of intelligence seems unsurprising to me.
- rob74 6y agoCan a swollen savannah monkey brain drive a car safely on public roads, using nothing but its attached eyes and ears? Yes (most of the time). Can AI do it? Not yet...
- yeellow 6y agoSure you can drive, but can you explain in details HOW you do it? Probably not and that's why a machine cannot do it yet (until it learns by itself, as we all did).
- natch 6y agoNot an AI in a publicly available release that can be obtained by Joe consumer. Supposedly the development versions we the general public can’t get yet are getting pretty good and would meet your “most of the time” bar on public roads. But we don’t really know how good AI is at any given moment, because those that are leading in any AI area sometimes have reasons for not telling yet. Or because we don’t believe them when they do tell us.
- QuesnayJr 6y agoI think essentially the opposite is true, and academia's notion of cardinal sins have held back the analysis of data by decades. And I say this as an academic. You make predictions all the time, and you don't know how. You don't know how you walk, how you drive a car, how you know the rules of English grammar. Your mind is a black box. And yet, you can do things that are still beyond the power of what we can understand using first principles. In many domains, first principles have achieved nothing. Years and years of effort by some of the smartest people in the world, and we pushed the ball one inch down the football field. Supervised machine learning asks the question "To what extent can we predict this outcome, given these predictors?" This is a perfectly valid question, one that we can try to answer given enough data. And we can do it under the exact same conditions we can do standard statistical inference. Ultimately, we want to answer "why" questions. But sometimes we can't even answer "what" questions. Most data is so complex that it's hard to say what the data even says, let alone why it says it. We could have been using what as a stepping stone to why, but our own provincialism as statisticians prevented us. I hope now we are learning to do better.
- Konohamaru 6y agoHow does one interpret what a black box means? In truth, any black box must remain beyond interpretation as a consequence of the fact that Verificationalism failed. There exist hidden verificationalist presuppositions embedded in your conclusions. If verificationalism doesn't hold, then by-first-principles stands as the best approach.
- bregma 6y agoWell, statistics is for those who understood that showing your work on an exam question was the whole point of the exam. ML is for those who just wrote down the final answer and dismissed showing their work as a waste of time. There's no need to understand the 'why' if you already know the 'what'.
- avs733 6y agoI love this and I'm stealing this but I think you maybe didn't take it far enough. To extend your analogy, Statistics is for places where making meaning of or taking action on the answer must (for practical or normative reasons) include the process Machine learning is for places where meaning or action can be done without regard for the process.
- srean 6y ago> This kind of philosophy will cause future generations to see machine learning as something worse than a fad Why do you think so ? Parameters are a piece of fiction, no one has actually seen them. Sometimes they are an useful piece of fiction, but not a very falsifiable notion. Prediction accuracy, well, that I can definitely measure without resorting to pieces of fiction that are epistemologically unknowable.