8 ms·
The problem with ML in my opinion is not that we're missing some sort of fundamental theory, but that there simply is none. ML is essentially fancy pattern matc
by Barrin92 8y ago
The problem with ML in my opinion is not that we're missing some sort of fundamental theory, but that there simply is none. ML is essentially fancy pattern matching roughly resembling the human visual system, which is why it happens to be good at tasks related to perception.
It's not some master algorithm, it's not going to produce sci-fi AI, and it probably isn't even suited to solve most problems in the realm of intelligence.
In fact it basically hits the worst possible spot on the problem solving scale. It barely learns anything given the amount of computation effort an data that goes into it, but it just happens to be good enough to be practically preferable to old symbolic systems.
It is completely mysterious to me how networks that approximate some utility function are a huge step forward to giving insight into cognition, reasoning, modelling, creation of counterfactuals and the sort of mechanisms we actually need to produce human like performance.
- typon 8y agoThe annoying part is that ML is not sold to the world like this. I would say the truth (I believe) in this comment is the "dirty little secret" of our industry. Everyone working on it knows this but the research and VC dollars are flowing in so no one wants to talk about it too much.
- chrstphrhrt 8y agoCan confirm, worked at an AI hype company for over a year and built all their systems outside the NN internals. There is going to be a correction.
- kuhhk 8y agoMay I ask if this insight is why you left? What do you work on now?
- chrstphrhrt 8y agoActually left due to anti-engineering culture (no tech leaders), succumbed to the distortions of Conway's law due to imbalance between professional services and core/platform. That said, the nature of the lack of leadership was around the MBA mindset that data science is the be-all end-all of AI, which IME is so far from the truth. There needs to be many layers/spirals of naive methods (i.e. straight ahead engineering) as a vanguard on the front lines with the DS/NNs/NLP bringing up the flanks for aspects of the domains that become more well understood over time. So yeah, tail wagging the dog, cart before horse etc. If you just barrel ahead with a monolithic NN pretending that all classes are created equal (structureless blob), which is what our DS PhDs were doing, it will quickly ossify. In our case it was intent prediction so we got to pretty high accuracy, but the labels were indicating many different categories that actually had relations in nature that could not be expressed. Ironically it took some regular engineers to research HMC (hierarchical multi-label classification) and ensembles and implement a new model training framework to support them. Not sure what they teach in school but it doesn't seem to be very practical. EDIT: now working on messaging (SMS/MMS/RCS/ABC) with a focus on dialog management and logical rules with layering-in of progressively less naive stats methods strategically rather than blindly. It helps to have an existing revenue stream that can be leveraged to create even more value with AI features rather than the backwards hail Mary of not iterating from a foothold of existing traction.
- RA_Fisher 8y agoStatistics is quite valuable and grounded in theory. It's a trusty tool if used with respect (like a knife).
- candiodari 8y agoYeah, too bad the theory requires a number of things to be true for data and models ... that are essentially never true. Once you think about 1) is the independance assumption really applicable here ? 2) does the underlying model satisfy the law of large numbers ? (NOT your model, the underlying model, ie. "reality") 3) am I really only interested in predictions within the data range ? The answer for all these questions is almost universally "no", and the theory says that statistics is not guaranteed to work under those circumstances. Correct statistical answers: 1) which average percentage of borrowers will default ? There is no answer to that question, because many non-random influences will make borrowers default in synchronized, and therefore very much non-independent ways (say, an economic crisis, an earthquake, an epidemic). Therefore you cannot correctly even calculate an average. 2) which average percentage of borrowers will default ? A model that describes whether an individual will default is going to be non-differentiable, and therefore certainly violate the law of large numbers (there will be sudden jumps in the default rate everywhere because of a million random reasons. An idiot tv anchor declares now the time to sell, and half the town finds out their loan is underwater, say). So underlying models involving humans essentially never satisfy the law of large numbers. 3) which average percentage of borrowers will default ? The number of factors changing that value do not match between the period your data is from (ie. the past) and the period you're predicting. Because there are so many real-world factors that affect your variable you can never avoid this situation. Therefore you cannot predict. All of these issues allow one to construct realistic scenarios where even trivial statistics will fail spectacularly.
- RA_Fisher 8y agoIt's true, statistics can't answer every question. There are some limits.
- soared 8y ago> It barely learns anything given the amount of computation effort an data that goes into it, but it just happens to be good enough to be practically preferable to old symbolic systems. I don’t follow this. Are you implying there haven’t been absolutely massive gains in computer vision, nlg, nlp, etc?
- laretluval 8y agoThere’s going to be people plugging their ears and shouting “but it’s just nonlinear function approximation!” all the way into the singularity.
- ggm 8y agoSingularity be like fusion... Always far enough away to be hinted at from thin evidence
- perl4ever 8y agoHuman intelligence involves choosing which context to use at any given point. So far, so called AI appears to take an external source of context for granted, which is why it seems to me fundamentally different from "real" intelligence, just like your eyes are different from your entire nervous system.
- wetpaws 8y agoEvery time there is some progress in AI people would yell "but this is not a real AI 'cause humans can..." ( play go/understand pictures/language/semantics/etc, goalpost is always in motion)
- pdimitar 8y agoStop playing the victim. Just make a driverless car that never makes mistakes already -- and I promise you the goalpost won't move for a while. Go and chess mean nothing out there. The world has a lot of very practical and concrete needs that this magical "AI" is nowhere near solving yet. I find the constant defensiveness of the ML proponents tiring and cringe-worthy. Somebody should remind you that what you are doing is not a backyard hobby. The people who fund you make a lot of promises and I don't think any of them have been delivered.
- laretluval 8y agoMaybe the concepts of “cognition, reasoning, modelling, creation of counterfactuals” etc. were not well-formed to begin with. Maybe intelligence, like gravity, is nothing like what we came up with using intuition. The fact is that black-box function approximators regularly beat systems designed around intuitive notions of cognition on observable metrics. Yes, the function approximators are bad. But the fact that they do better than more intuitively-satisfying symbolic systems suggests that our intuitive ideas about intelligence are even worse.
- QML 8y agoWait, what? Are you confusing ML with AI? The last two courses I took on ML were all theory. For example, when we use linear regression, we make assumptions about the noise — i.i.d. and Gaussian.
- nurettin 8y agoWould you describe AI as: automatic categorization and decision making? If so, then Machine Learning is a part of modeling AI. Regardless of how they are taught in terms of University lectures.
- tnecniv 8y agoI believe OP is speculating about state-of-the-art ML. The classical techniques, like linear regression, are well studied. However, at this point, there is a race to explain theoretically _why_ deep learning is so successful at generalization when, by classical standards, it shouldn't be. OP is speculating that this quest is in vain.
- notafraudster 8y agoI agree with your interpretation and pushback against your parent comment, but wanted to elaborate that the "problem" here is much deeper. Deep learning is basically magic, okay, fine, so let's start much closer to linear regression. Say we have a predictor matrix X with 2 predictors. We fit a model using a penalized linear regression (say LASSO) adding to our predictor matrix an interaction terms, arbitrary polynomial and logarithmic transformations of each X, and interactions between the transformations of the Xes. Ideally we motivate this because of some case knowledge about relevant nonlinear transformations of the predictors. Or maybe second best we use the kernel trick to run a kernelized regression that uses an infinite dimensional prediction of all possible transformations of the predictors. But realistically, we toss some shit in the model and run it. The LASSO spits out X1, X2^2, and X1^3 * log(X2) as being the cross-validation selected non-zero parameters. What real world scenario could possibly generate a causal process that is linear in X1 (say income), quadratic in X2 (say age, which often displays quadratic forms in regressions), but also predicted by a bizarre non-linear interaction of nonsense transformations? What a practitioner would probably do is fit the model. In a lot of ML contexts, interpretation would lead to the practitioner saying "Well, okay, ML sometimes produces nonsense models, but you can't argue with the predictive results". Or maybe the practitioner is more sensitive to interpretability and instead takes another tack. Maybe the practitioner might say "clearly this interaction is nonsense, but there must be some interaction, I'll re-run with a linear interaction". Or else they'd re-run the LASSO with conditions about not including nonlinear terms without including the lower dimensional terms. Or else they'd run a grouped LASSO and make up some justification for the groups. All of these reveal that most ML practitioners are basically just doing alchemy. And this is talking about what amounts to a minor version increment of linear regression, so probably the simplest possible technique we'd still call part of ML.
- protonfish 8y agoIt is a misconception that ANNs are based on human visual processing. Maybe "inspired by" is a better term. The 1959 paper "What the Frog's Eye Tells the Frogs Brain" disproved the belief that ANNs have anything to do real neural networks.
- oldgradstudent 8y agoBut it's a great brand name.
- deleted 8y ago[deleted]
- tjpnz 8y ago>It barely learns anything given the amount of computation effort an data that goes into it, but it just happens to be good enough to be practically preferable to old symbolic systems. This is the part that's always bothered me. For many ML has just become the default tool to throw at a problem where other solutions (albeit less shiny) exist, that might give the same or even better results (and are very likely going to end up being more efficient). Would it be too much of a leap to suggest that we're forgetting to think?
- berkut 8y agoI'd agree with this assessment 100% - ML can achieve some very impressive results - in my domain (computer graphics for movie VFX) with regards to image generation / identification / modification, e.g. denoising for the latter - however the result is almost always effectively a black box of functionality which: A: isn't always understood what it is actually doing or how. B: often needs to be retrained when input data varies slightly outside original training data - this is expected at a technical level, but is annoying in practice and often then starts deviating and losing its effectiveness or gets inconsistent results with larger less homogeneous input data sets. C: Can't really (at least easily) be used as a stand-alone / packaged component/library to re-use nicely in other places as a conventional algorithm would be.
- groestl 8y agoI like that all of the issues you bring up around ML can be read as applying to a human expert/craftsperson as well! (still with a wider margin, of course)
- yesenadam 8y agoML is essentially fancy pattern matching roughly resembling the human visual system, which is why it happens to be good at tasks related to perception. It seems that an application like cancer diagnosis - feeding in test results and medical information to a NN, it finding patterns better than any human could, has nothing to do with human perception systems. It's just much better and faster at detecting patterns in complex things than humans are.
- kuu 8y agoML != Deep Learning. Deep Learning is just a branch (or even a sub-branch inside NN) of many ML techniques. Not all of them suffer the same problems or are based in the same theoretical background.
- make3 8y agothere is huge with being done right now on the theoretical backing of deep learning. classical machine learning has huge, sound theoretical backing. scaling well is extremely important, and is the main advantage of deep learning. as planes don't need a bird's feathers to be useful, deep learning is already way sufficiently successful enough to justify their use. no one serious claims that deep nets are biologically possible. they are still promising, and useful, powering translation, phone keyboard autocomplete, search, camera stuff like low light mode and cell phone portait mode, medical imaging diagnosis, malware detection, new planet detection in astronomy, plasma modeling in energy engineering, etc
- logicallee 8y ago> It barely learns anything given the amount of computation effort and data that goes into it You know what else barely learns anything given the amount of computation that goes into it? A kid. With 100 billion neurons -- so on the order of a hundred gigabytes of RAM -- after 72 months of closely supervised learning they're still far from being able to do many rudimentary tasks. It takes over 10 years of training to do that. Something tells me most ML researchers wouldn't be too unhappy with such an awful performance.
- baxtr 8y agoThe moment I realized that Deep Learning is nothing more than a non-linear matrix operation I lost a big deal of respect for the field. I still believe the potential of Deep Learning is huge, since many, especially larger companies, have lots of data that just sits there waiting for some innovation. DL can deliver more productivity, i.e. higher quality, faster processes etc. That is great. But this has very little to do with the fancy sci-fi version of AI.
- andreyk 8y agoWhile it's true that the basics of Deep Learning (eg ImageNet) are not that conceptually interesting, it is also worth noting people are working on tons of interesting directions within this framework combined with other ideas from AI (see eg 'Learning to Reason with Third-Order Tensor Products' - https://arxiv.org/abs/1811.12143 https://arxiv.org/abs/1811.12143 | 'Beyond imitation: Zero-shot task transfer on robots by learning concepts as cognitive programs' - https://arxiv.org/abs/1812.02788 https://arxiv.org/abs/1812.02788)
- Farradfahren 8y agoActually it could produce SciFi AI - if combined with evolutionary algos. Just look at how every toddler does it- act and produce a lot of data, then check for signal -> reaction reproduceability. Once that is stable, fine tune. Then use the new gained abilitys to repeat the process.