16 ms·

The revolution of machine learning has been exaggerated

339 points by benryon 7y ago

deminature 7y ago
This is a perfect example of the 'AI effect' in action: >"It's part of the history of the field of artificial intelligence that every time somebody figured out how to make a computer do something—play good checkers, solve simple but relatively informal problems—there was a chorus of critics to say, 'that's not thinking'." AIS researcher Rodney Brooks complains: "Every time we figure out a piece of it, it stops being magical; we say, 'Oh, that's just a computation.'" https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect
- username90 7y ago
  No, it talks about the other AI effect, where AI researchers thinks that generalizing the model will be relatively easy. Like, lets say someone makes an AI to play tic-tac-toe at the same level as the best human and then says that with just a little bit more work it will be able to beat a grandmaster at Go.
  - Animats 7y ago
    Right. Someone linked to "Artificial Intelligence Meets Natural Stupidity" last week, which makes that argument in detail. People in the field have been claiming Strong AI Real Soon Now since the General Problem Solver of the 1960s, which is a simple tree search algorithm. I heard a lot of that at Stanford in the 1980s, when the expert systems boom was about to crash and people thought Symbolics LISP machines were magical. There is progress, though. Machine learning does do a lot. And it makes money, so effort will continue at a high level. AI used to be a dinky field - maybe 20-50 people at MIT, CMU, and Stanford. Now it's huge.
  - NoodleIncident 7y ago
    I've only seen the headlines whizzing by, but isn't this exactly what Deepmind(/AlphaGo/AlphaZero/MuZero) has been doing lately?
    - username90 7y ago
      My point was that machines mastered tic-tac-toe 50 years ago and mastered Go just last year. Making a statistical model to predict a small subset of three body problems isn't very impressive at all, kinda like tic-tac-toe. It is a bit interesting that it works, but not much more than that. Adding another body, or adding starting velocities to these three bodies, would make the problem a lot less tractable, so it is very unlikely the same techniques will work there.
- blauditore 7y ago
  I think the root the issue is that "AI" is a vague term where almost any kind of program, no none at all could go under. The fact that a machine can automatically calculate the sum of two numbers could be considered artificially intelligent by someone who's never seen a computer before. At the same time, no matter how far you go, any program (based on turing-like machines) will always be a deterministic chain of cause-and-effect, which is "just a computation". But AI in its meaning use contains a certain set of fields in a way that makes sense. It's just the puplic misinterpration nonsense that makes it look so weird.
  - sjy 7y ago
    Are computers meaningfully different to brains in that “deterministic” respect? Both are apparently based on deterministic interactions at the nanometre scale and exhibit simple behaviour that can be explained from first principles (eg. simple arithmetic, or a stroke). Both also exhibit complex emergent behaviour that can’t be explained from first principles (eg. the output of complex ML systems, or most human behaviour), which we sometimes call “non-deterministic.” I’m happy to concede that software is artificially intelligent just as I’m prepared to concede that most animals are naturally intelligent. It’s just a question of drawing an arbitrary line when things are intelligent enough to warrant the description. The AI effect is the process of that line getting moved closer and closer to “capable of doing everything an average human can do.”
    - meowface 7y ago
      Exactly. That's why there's the AI vs. AGI distinction.
    - perl4ever 7y ago
      The definition of a modern human is someone who does what machines do not do, so as machines become more capable it changes what humans do. I guess it seems very likely to me that we will have Skynet/paperclip maximizers before AGI. I mean, I think we kind of do right now in what the internet is growing into. We're like single cells imagining what a brain would be like but we'll never know because probably we'll be part of something brainless that dominates the ecosystem, or if there is an organism with a brain, we won't be able to perceive it because we're components.
- feral 7y ago
  We really need another name, which refers to people uninformedly quoting 'the AI effect'. How about "the AI effect effect"? This happens all the time: - X suggests that success at a particular task is a good proxy to general intelligence. - Years later Y solves that task with a clever hack and lots of computation, but no general intelligence. - The media breathlessly suggests we've made progress on general intelligence. - Z points out we haven't really. - Someone says "Oh that's the AI effect!", by which they mean the goalposts have been unfairly moved on AI. No, they have not. There's something called general intelligence, humans can do it, and Deep Blue cannot, and it's fine to say that Deep Blue beating humans at Chess was not thinking. It just turned out that Chess wasn't as good a proxy to thinking as we initially thought, and mentioning "The AI effect" just clouds the discussion.
  - whatisthiseven 7y ago
    That is a good point about something I hadn't considered. I was going to disagree with you until the last sentence, which then clicked. As AI is developed, it is also championed as "closer to real thought" and might someday solve the more general intelligence problem, only for those same experts to later realize that would never be the case and then retroactively proclaim "this was never going to solve general intelligence, but it is still AI". The history is full of this kind of revisionism, so I agree with you.
  - didibus 7y ago
    I agree with you, but just want to clarify for others who don't understand why the parent misused the term AI effect. AI effect is more related to the fact that once you understand the trick behind the magic you no longer believe the task to require intelligence, even though it is still able to do an amazing task that wasn't possible before and had for centuries been thought off as requiring intelligence and something that only humans could do. That the current AI techniques couldn't work for every imaginable task, and that people believed they might isn't a case of the AI effect. That's just hype, and possibly another coming case of the AI freeze. The AI effect will be how in 5 or 10 years, CS students will learn neural nets as part of their first or second year curriculum and the whole machinery will be behind a single API to some Apache library. And someone will then use that to some cool effect and say they're doing AI, and others will laugh and say, you're not doing AI, you just used a neural net to learn weights from a big data set silly, that's not AI. This effect happens for every task, even when AI is not involved. Until you understand how, someone doing something you can't comprehend will have you think they are a genius and maybe of a higher IQ and probably very intelligent. But learn the "how" for yourself and it'll stop being so impressive. Rubik's cube is a good example of this. Once you realize there's a trick to it, it stops being as impressive. Some relevant quotes from the wikipedia article: > AI effect is: As soon as AI successfully solves a problem, the problem is no longer a part of AI. > Software and algorithms developed by AI researchers are now integrated into many applications throughout the world, without really being called AI. > AI advances are not trumpeted as artificial intelligence so much these days, but are often seen as advances in some other field > practical AI successes, computational programs that actually achieved intelligent behavior, were soon assimilated into whatever application domain they were found to be useful in, and became silent partners alongside other problem-solving approaches, which left AI researchers to deal only with the "failures", the tough nuts that couldn't yet be cracked > The great practical benefits of AI applications and even the existence of AI in many software products go largely unnoticed by many despite the already widespread use of AI techniques in software. This is the AI effect. Many marketing people don't use the term 'artificial intelligence' even when their company's products rely on some AI techniques. Why not? https://en.m.wikipedia.org/wiki/AI_effect https://en.m.wikipedia.org/wiki/AI_effect
- domnomnom 7y ago
  That link honestly reads like more kool aid and deflection of criticism written by AI professionals. Why aren’t mathematicians complaining about their work being largely unknown?
- debt 7y ago
  The concept of AI to the average person outside of the computer bubble is "human intelligence made artificial" So if it's not actually thinking, then technically, from the perspective of an average layperson, it is not real AI. However, the "AI effect" is still real, but it's related to a problem local to academics and AI laypeople. Most non-AI-laypeople/AI academics believe AI is a machine that can think like how a person thinks.
orasis 7y ago
From a professional perspective, if you think ML is a fad you are completely missing the train.
- semiotagonal 7y ago
  From a professional perspective, does it make sense to get on a train that is so crowded already? Step 0 is probably to take Andrew Ng's on Coursera, but as of right now, you'd be among "2,647,287 already enrolled!" [0] [0] https://www.coursera.org/learn/machine-learning https://www.coursera.org/learn/machine-learning
  - orasis 7y ago
    You have to be joking. Demand for ML skills, especially scaling in production, is off the charts.
    - username90 7y ago
      Scaling ML in production doesn't take much ML skills though.
      - omicron1 7y ago
        The bulk of industry ML work isn't actually suited to ML phds without engineering skills. I work in FAANG and this is a huge problem where the ML phds have poor communication with skilled engineers and a lack of engineering experience. They often even look down upon people who don't have fancy credentials. Unfortunately they just end up creating a money wasting disaster of a system.
    - deleted 7y ago
      [deleted]
  - usaar333 7y ago
    I'd guess they have a very high drop-out rate. Regardless, "machine learning" is a very broad field and honestly I have no idea what an "ML engineer" is doing if they are one. It can cover any of the following: 1. Cutting-edge academic research (do better on this test set) 2. Doing data analysis to identify prediction ability 3. Creatively thinking of useful features to evaluate. 4. Implementing data pipelines/logging to obtain the features needed for #3. 5. Production systems to evaluate/train ML systems. (multiple places in the stack). Because the spectrum is so wide, if you are already an engineer, you can readily get into "ML" categories 3-5 and even 2. Andrew Ng's course is a valuable introduction and not that heavy of an investment -- I found that just with it (alongside my product and infra background), I could readily contribute to ML groups at my company.
lettergram 7y ago
In this particular case I think the headline of the title is probably better than the one on HN. Yes, I’m many fields ML has been exaggerated, but it’s just a matter of time. Most companies aren’t using ML effectively yet. But I know for a fact when you improve NLP, you _can_ improve almost all white collar jobs. NLP the past two years has been crazy, and we haven’t seen anything yet.
- dmreedy 7y ago
  I agree that there has been some impressive progress on NLP lately; am curious what you think the corresponding improvements in white collar jobs have been, though? Genuinely asking! Or do you think they are mostly yet to come?
  - rpedela 7y ago
    I can at least give you some examples from my recent experience. A combination of text classification and NER can solve many business problems if the accuracy is high enough or you can mitigate errors. For e-commerce search, a common problem is how to give relevant results for misspelled queries or unique queries. Text classification and/or NER can often identify the product type, brand, etc and hopefully give a relevant result which leads to a sale. Another example is data entry or business analysis roles. Often they will manually extract data from sources and put it into Excel or some other BI tool. NLP ML has gotten good enough in just the last couple years that much of the data extraction can be automated, but you still need a human to verify accuracy.
  - lettergram 7y ago
    From my own work: * NER (name-entity recognition) is becoming insanely good (99+%), this can help with everything from data security to automated form filling, to automated medical billing, etc. Multiple billion dollar industries will be shook up by this. * My own work on data generation is pointing to reduced surface area of security purposes and improved access to quasi-data for data science and application testing[1] * Translation from images to text descriptions are also a major thing, which when combined with other systems can do things like make suggested diagnosis. * Text generation systems are impacting how consumers interact. The chat bots these days are getting to the point where it's very difficult to identify if it's a human or not. I can't go into too many details, but see[2] * There's also my side business which is a search engine for people who would know an answer to a questions: https://insideropinion.com/ https://insideropinion.com/ In general, NLP is probably going to have a larger impact that driverless cars on our day-to-day lives. We're seeing insanely good story generation (e.g. GPT-2), chat bots, billing, etc. I also think WAY WAY more is yet to come. The transformer (which is spuring most of this) is only a couple years old. We'll see lots more applications as time goes on. [1] https://medium.com/capital-one-tech/why-you-dont-necessarily-need-data-for-data-science-48d7bf503074 https://medium.com/capital-one-tech/why-you-dont-necessarily... [2] https://arxiv.org/pdf/1908.01841.pdf https://arxiv.org/pdf/1908.01841.pdf
FiberBundle 7y ago
"The widespread misimpression that data + neural networks is a universal formula has real consequences: [...] in what students choose to study, and in what universities choose to teach." I think this is a problem that deserves more attention. A large portion of CS students nowadays choose to focus on Machine Learning, many brilliant CS students decide to get a PhD in ML in hope of contributing to a field that will continue to develop at a pace similar to the progress we have seen in the last 7 years. This view, I think, is not justified at all. The challenges that AI faces such as incorporating a world model into the techniques that work well will be significantly harder than what most researchers openly admit, probably to the detriment of other CS research areas.
- didibus 7y ago
  I've also noticed a trend of college graduates who in interview struggle with general software engineering practices and more fundamental coding skills and CS knowledge, but have knowledge and proficiency of ML. If you're a CS BA or MS, and don't plan to do an ML PhD, I'd suggest asking yourself if you want a data science job or a software engineering one. If the latter, remember to focus more on that. Surface knowledge of ML is a bonus, but the rest is more important.
  - aidos 7y ago
    That’s definitely something I’ve seen too. Most of the cvs I get through, especially from younger devs, talk about ML. I’ve got my work cut out for me keeping my existing codebase understandably without deliberately introducing opacity!
  - weego 7y ago
    My partner has ditched crosstraining into data science and ML because of how ridiculously 'ductape and string' the entire sub-indistry is. Tooling that is pre-2000 quality if it even exists, people with math skills yet only the most rudimentary ability in best practice around data storage and maintenance, coding, domain knowledge and understanding of how much bias they're introducing into results. Her view is its just a free for all of people who have little to no need to justify their output because everyone is waiting and hoping for the magic to happen.
sam0x17 7y ago
A problem is that the current mode of input for neural networks (one floating point value per input node) makes neural networks problematic for tackling problems with variable input size (read: most problems). My current research looks into how we can maybe fix that in certain cases, but in general it's a big limitation. Recurrent networks fix this to a certain extent, but they have not had the revolution that CNN's have enjoyed just yet. We can also do things like represent things in binary, but this adds complexity, and often greatly inhibits learning.
tus88 7y ago
Deep fakes are pretty good. It's kindof interesting....ML has allowed computers to generate realistic images that humans cannot discern....while image recognition remains woefully hopeless compared to human perception. So rather than competing with the human brain...AI will try and trick the human brain instead.
- EpicEng 7y ago
  >ML has allowed computers to generate realistic images that humans cannot discern I've yet to see a deep fake that's even remotely indiscernible from the real thing.
  - andrewnicolalde 7y ago
    How do you know?
    - EpicEng 7y ago
      Well, technically I don't, for the reason I don't know if the abominable snowman exists. I have yet to see one I knew to be a df which was even close to being convincing.
      - joycian 7y ago
        Have you seen CTRL Shift Face, the YouTube channel?
  - afterburner 7y ago
    You probably haven't been noticing them in movies though. Such as when they stuck Margot Robbie's entire head on an actual skater for I, Tonya. https://www.youtube.com/watch?v=bqgD6lHrQ_8 https://www.youtube.com/watch?v=bqgD6lHrQ_8
    - EpicEng 7y ago
      Was that a DF though?
dfan 7y ago
I assume from the title that this is either written by Gary Marcus or quotes him extensively. (I just checked and was correct.) It would be nice to hear this sort of critique from more than one person.
- ycombinete 7y ago
  Is he a good source of critique? Sincere question, I don’t know anything about the field.
  - high_derivative 7y ago
    He is rather controversial in machine learning circles. I would say he is essentially right about the fact that a lot of ml work is crap. Like most of anything is crap, so that is not a groundbreaking insight and it comes off as somewhat grandstanding to keep saying that. He has however really turned this into a personal brand which many suspect he is using to advertise his startups. Many people feel he is sort of disingenuous about the actual progress being made in computer vision and NLP, even if there is many ways to go. To answer your question: You can read a single piece of his and maybe get some value from that, but he keeps rehashing the same points to sh*t over different new work, and there is not much to be learned from that after a while. Disclaimer: I did not read the above article in full, I am just familiar with Gary Marcus + work in ML industry. Edit: Keep expanding this but it got me thinking. Earlier in my career, I was also eager for the 'gotchas', for being able to point out why something is crap, exaggerated, cannot do what it promises, and so forth. As I outgrew this impulse to focus more on what is good about some work, I began spotting it on new students a lot. They would come into seminars and smugly crap all over recent papers. I have since come to the conclusion that it is useful to have realistic internal assessment of progress in your field, but that sharing your harshest criticisms is not necessarily a good way to look smart or make progress. It often turns out that while feeling clever about realizing limitations of some work, the field just moves forward. The people you just criticized leapfrogged while you were busy thinking about how to one-up them.
    - t_serpico 7y ago
personjerry 7y ago
That's because the media hype is completely different from the industry hype. This article is a form of media hype. The pragmatic, already realized effects of machine learning are mostly in improving existing systems, especially classification systems, and have increased metrics by big percentages at big data companies, and more applications are being found every day. That is why real demand and interest is so high for ML. These generative, speculative models are interesting experiments by researchers but inconsistent and far overstudied by the media, because "AI invents new physics" is more understandable and sexier than "AI improves lead generation by 6.8%"
- spamizbad 7y ago
  I would say the industry definitely has its own hype. And honestly, I think the news media is a trailing indicator on that hype. For example, there's already an ongoing replication crisis that's been ongoing for several years with ML papers in computer science. I've spoken with people in the industry who have even speculated that there's straight-up academic fraud taking place. If you get a splashy, sexy finding you can get scooped up by a FAANG-like company with a fat salary before anyone gives your work a second look.
  - omicron1 7y ago
    From working with many ML researchers, it's not speculation. Some aren't very proud, but if you want to get ahead and stay competitive, you have to publish your un-replicable results. The results aren't usually completely fake, but they depend on enough dishonest sleight of hand that they might as well be.
- mobjack 7y ago
  You can improve lead generation by 6.8% without ML too. A/B testing a few changes can easily accomplish the same thing. The AI could be optimizing to attract people who are willing to give out their email address than actual potential customers so it doesn't help the business. We tried using Google ads to optimize for leads before and we saw ads were on sketchy websites promising people free stuff. Lead generation was a bad KPI to use but it was chosen can't optimize for sales because the volume is to low to train the ML models. This is the reality of many businesses that experiment in AI and they produce junk. ML does work well for certain domains where you have the scale of data to pull it off, but now the hype has spread where it is not appropriate.
deleted 7y ago
[deleted]
KaoruAoiShiho 7y ago
I'm not expert but I have a feeling this article will age poorly.
m3kw9 7y ago
DeepFake issues has definitely been exaggerated
genz9000 7y ago
ok boomer
egocodedinsol 7y ago
"You can see the computer age everywhere but in the productivity statistics." This was true for a few years, until it wasn't.
yters 7y ago
Not if we measure the revolution in games and kitten pictures! There has been tremendous progress in those domains!
- xamuel 7y ago
  What revolution in games do you have in mind? Is there any particular video game that is making really interesting use of ML?
  - yters 7y ago
    I am referring to the playing of said games like with alphago and alphazero.
  - friendlybus 7y ago
    All the production tools for making games/video are cramming ml tools in. It's one of the first places to be swimming in it. The two minute papers guy made a material predictor to cut down on preview render time. Quixel uses ml for search. Substance and Houdini are gaining tools soon. Blender has used it behind the scenes for its new neblua tool. Mocap & animation are bathed in it. There's probably a lot more I can't recall right now.
andbberger 7y ago
In my experience companies just aren't willing to invest, and get turned off when they realize just how long the timelines on bringing DL to production are. It certainly works, but magic it ain't. You need an experienced practitioner and lots of patience.
jokoon 7y ago
Once you realize there's "artificial" in AI, it quickly dispel the interest. Science cannot even agree on a practical definition of intelligence. Unless you let biologists, psychologists and neurologists work with computer scientists to advance the discussion, I don't think it's worth it to explore machine learning techniques to pursue intelligence. Don't call it ML or AI, call it "improved statistical prediction". The buzz will quickly fade.
hinkley 7y ago
Question for the collective wisdom: What, in your estimation, are some current or near-future trends that represent an objective improvement to the state of the art?
hinkley 7y ago
I would like to humbly submit, from my nose-bleed seat, that ML would serve us best by adding it to our arsenal of fuzzing techniques. Train a model and then set it loose to do exploratory testing, looking for valid-looking inputs that cause the wrong answer to be returned from our entirely pedestrian Plain Old Business Logic. The AI proposes a scenario, but the human is the final arbiter.
dgudkov 7y ago
>tendency among science journalists ... to overstate the significance of new advances in AI and machine learning It's very hard to not overstate when overstating brings you clicks, likes, retweets, and dopamine spikes.
buboard 7y ago
These articles (and a lot of what g. marcus writes) are attacking strawmen. I ve never heard no one claiming that NNs will invent new theories and i dont think that 2008 article is widely read. But, for things that are hard both computationally and theoretically, like protein folding, NNs may really be revolutionizing the field even if we don't know how they do it. Scientists do not buy-in foolishly into every AI hype. The problems lie with VCs and funding bodies, which are indeed swayed by adding "AI" to every kind of proposal. That's a separate problem, though
- skywhopper 7y ago
  That's the whole point... at the very beginning the post cites articles that suggest or even outright say that AI is coming up with new theories (or in the CS world, see all the articles and bloviators who predict AI will replace programming as a job). Regardless of what the scientists in the field would say one-on-one, the outright-wrong portrayal of ML in the press, egged on by VCs out to flip their investments, is causing problems for anyone who wants to actually make progress in their field, or recruit students honestly.
  - buboard 7y ago
    > articles that suggest or even outright say that AI is coming up with new theories I would contest that there are widely read articles saying those things. Yes, surely someone publishes those , but they are generally dismissed. the field is conscious of its hype (e.g. check r/machinelearning), and i don't think students are stupid either
justicezyx 7y ago
The ratio of ML interested or majored job candidates I saw nowadays is staggering, appears >70%.
jhrmnn 7y ago
This year, I’ve been working on representing electronic wave functions with neural networks: https://arxiv.org/abs/1909.08423 https://arxiv.org/abs/1909.08423 I come from a physics background, and my impression is that NNs are a cool computational tool that every computational physicist probably should have in their toolbox (at least having a sense of their capability). But so far I haven’t seen any truly revolutionary advance with NNs acrose the whole of materials science.
Gatsky 7y ago
Still waiting for a revolution which has the optimal amount of hype. I wonder if this is an application for GANs.
deleted 7y ago
[deleted]
geoalchimista 7y ago
A recipe for the typical media hype "AI learns physics" discovery 1. Find a well-known classical problem with a battle-tested deterministic model representation 2. Generate synthetic data using the model 3. Train a 1-layer neural network to score 99% in predicting the synthetic data (even if the deterministic model can score 99.999% but let's ignore the minor difference) 4. Claim AI "learns" the underlying physics and does a better job than classical models 5. World dominance
- knzhou 7y ago
  This is exactly it. Earlier this year there was a hyped up paper [0] going around claiming that they could derive physics with AI. On inspection I found that what they actually did was this: 1. Take one simple physics equation, such as F = ma 2. Generate 10^6 tuples exactly satisfying this equation (F = 6, m = 2, a = 3), with absolutely no redundant, erroneous, or extraneous information 3. Feed these to a primitive "AI" system which brute-force tries the simplest relations that could relate the three provided variables (m = Fa, a = Fm, F = ma, ...) 4. Recover F = ma, declare physicists obsolete It's such a poor imitation of the real process of discovery in science that it's frankly insulting. When people hear AI is involved with something they turn their brains off. 0: https://arxiv.org/abs/1905.11481 https://arxiv.org/abs/1905.11481
Lapalux 7y ago
Machine learning = stats + computing power. The key thing here is that the core ingenuity of ML is statistics - the limits of which are well known. ML hype is false promises based on uninformed wild extrapolation.
blankdebut 7y ago
People doing research in ML have been saying something along these lines for years.
jowdones 7y ago
When I started with computers back in 1992, Eastern Europe, noone around me had the slightest idea what computers can do. I sort of imagined (and hoped) they will (magically) solve math problems for me so I won't need to actually make the hard effort to understand. 27 years later, not just 99.99% of the humans in general but of my "fellow" programmers, if I can call them so, are still stuck at this petty ignorance level.
- le3dh4x0r 7y ago
  Is that really so? I think this field has been hyped pretty hard and lots of people want a piece of that deep neural cake.
thecleaner 7y ago
Most ML will eventually be commoditized so it is unlikely that specializing only in ML will make things better for people who dont come from permier schools. If you have a so called MiML from any premier school you may not know jackshit about production Ml and still earn hundreds of thousands.

339 points by benryon 7y ago

This is a perfect example of the 'AI effect' in action: >"It's part of the history of the field of artificial intelligence that every time somebody figured out how to make a computer do something—play good checkers, solve simple but relatively informal problems—there was a chorus of critics to say, 'that's not thinking'." AIS researcher Rodney Brooks complains: "Every time we figure out a piece of it, it stops being magical; we say, 'Oh, that's just a computation.'" https://en.wikipedia.org/wiki/AI_effect https://en.wikipedia.org/wiki/AI_effect

username90 7y ago
No, it talks about the other AI effect, where AI researchers thinks that generalizing the model will be relatively easy. Like, lets say someone makes an AI to play tic-tac-toe at the same level as the best human and then says that with just a little bit more work it will be able to beat a grandmaster at Go.
- Animats 7y ago
  Right. Someone linked to "Artificial Intelligence Meets Natural Stupidity" last week, which makes that argument in detail. People in the field have been claiming Strong AI Real Soon Now since the General Problem Solver of the 1960s, which is a simple tree search algorithm. I heard a lot of that at Stanford in the 1980s, when the expert systems boom was about to crash and people thought Symbolics LISP machines were magical. There is progress, though. Machine learning does do a lot. And it makes money, so effort will continue at a high level. AI used to be a dinky field - maybe 20-50 people at MIT, CMU, and Stanford. Now it's huge.
- NoodleIncident 7y ago
  I've only seen the headlines whizzing by, but isn't this exactly what Deepmind(/AlphaGo/AlphaZero/MuZero) has been doing lately?
  - username90 7y ago
    My point was that machines mastered tic-tac-toe 50 years ago and mastered Go just last year. Making a statistical model to predict a small subset of three body problems isn't very impressive at all, kinda like tic-tac-toe. It is a bit interesting that it works, but not much more than that. Adding another body, or adding starting velocities to these three bodies, would make the problem a lot less tractable, so it is very unlikely the same techniques will work there.
blauditore 7y ago
I think the root the issue is that "AI" is a vague term where almost any kind of program, no none at all could go under. The fact that a machine can automatically calculate the sum of two numbers could be considered artificially intelligent by someone who's never seen a computer before. At the same time, no matter how far you go, any program (based on turing-like machines) will always be a deterministic chain of cause-and-effect, which is "just a computation". But AI in its meaning use contains a certain set of fields in a way that makes sense. It's just the puplic misinterpration nonsense that makes it look so weird.
- sjy 7y ago
  Are computers meaningfully different to brains in that “deterministic” respect? Both are apparently based on deterministic interactions at the nanometre scale and exhibit simple behaviour that can be explained from first principles (eg. simple arithmetic, or a stroke). Both also exhibit complex emergent behaviour that can’t be explained from first principles (eg. the output of complex ML systems, or most human behaviour), which we sometimes call “non-deterministic.” I’m happy to concede that software is artificially intelligent just as I’m prepared to concede that most animals are naturally intelligent. It’s just a question of drawing an arbitrary line when things are intelligent enough to warrant the description. The AI effect is the process of that line getting moved closer and closer to “capable of doing everything an average human can do.”
  - meowface 7y ago
    Exactly. That's why there's the AI vs. AGI distinction.
  - perl4ever 7y ago
    The definition of a modern human is someone who does what machines do not do, so as machines become more capable it changes what humans do. I guess it seems very likely to me that we will have Skynet/paperclip maximizers before AGI. I mean, I think we kind of do right now in what the internet is growing into. We're like single cells imagining what a brain would be like but we'll never know because probably we'll be part of something brainless that dominates the ecosystem, or if there is an organism with a brain, we won't be able to perceive it because we're components.
feral 7y ago
We really need another name, which refers to people uninformedly quoting 'the AI effect'. How about "the AI effect effect"? This happens all the time: - X suggests that success at a particular task is a good proxy to general intelligence. - Years later Y solves that task with a clever hack and lots of computation, but no general intelligence. - The media breathlessly suggests we've made progress on general intelligence. - Z points out we haven't really. - Someone says "Oh that's the AI effect!", by which they mean the goalposts have been unfairly moved on AI. No, they have not. There's something called general intelligence, humans can do it, and Deep Blue cannot, and it's fine to say that Deep Blue beating humans at Chess was not thinking. It just turned out that Chess wasn't as good a proxy to thinking as we initially thought, and mentioning "The AI effect" just clouds the discussion.
- whatisthiseven 7y ago
  That is a good point about something I hadn't considered. I was going to disagree with you until the last sentence, which then clicked. As AI is developed, it is also championed as "closer to real thought" and might someday solve the more general intelligence problem, only for those same experts to later realize that would never be the case and then retroactively proclaim "this was never going to solve general intelligence, but it is still AI". The history is full of this kind of revisionism, so I agree with you.
- didibus 7y ago
  I agree with you, but just want to clarify for others who don't understand why the parent misused the term AI effect. AI effect is more related to the fact that once you understand the trick behind the magic you no longer believe the task to require intelligence, even though it is still able to do an amazing task that wasn't possible before and had for centuries been thought off as requiring intelligence and something that only humans could do. That the current AI techniques couldn't work for every imaginable task, and that people believed they might isn't a case of the AI effect. That's just hype, and possibly another coming case of the AI freeze. The AI effect will be how in 5 or 10 years, CS students will learn neural nets as part of their first or second year curriculum and the whole machinery will be behind a single API to some Apache library. And someone will then use that to some cool effect and say they're doing AI, and others will laugh and say, you're not doing AI, you just used a neural net to learn weights from a big data set silly, that's not AI. This effect happens for every task, even when AI is not involved. Until you understand how, someone doing something you can't comprehend will have you think they are a genius and maybe of a higher IQ and probably very intelligent. But learn the "how" for yourself and it'll stop being so impressive. Rubik's cube is a good example of this. Once you realize there's a trick to it, it stops being as impressive. Some relevant quotes from the wikipedia article: > AI effect is: As soon as AI successfully solves a problem, the problem is no longer a part of AI. > Software and algorithms developed by AI researchers are now integrated into many applications throughout the world, without really being called AI. > AI advances are not trumpeted as artificial intelligence so much these days, but are often seen as advances in some other field > practical AI successes, computational programs that actually achieved intelligent behavior, were soon assimilated into whatever application domain they were found to be useful in, and became silent partners alongside other problem-solving approaches, which left AI researchers to deal only with the "failures", the tough nuts that couldn't yet be cracked > The great practical benefits of AI applications and even the existence of AI in many software products go largely unnoticed by many despite the already widespread use of AI techniques in software. This is the AI effect. Many marketing people don't use the term 'artificial intelligence' even when their company's products rely on some AI techniques. Why not? https://en.m.wikipedia.org/wiki/AI_effect https://en.m.wikipedia.org/wiki/AI_effect
domnomnom 7y ago
That link honestly reads like more kool aid and deflection of criticism written by AI professionals. Why aren’t mathematicians complaining about their work being largely unknown?
debt 7y ago
The concept of AI to the average person outside of the computer bubble is "human intelligence made artificial" So if it's not actually thinking, then technically, from the perspective of an average layperson, it is not real AI. However, the "AI effect" is still real, but it's related to a problem local to academics and AI laypeople. Most non-AI-laypeople/AI academics believe AI is a machine that can think like how a person thinks.

orasis 7y ago

From a professional perspective, if you think ML is a fad you are completely missing the train.

semiotagonal 7y ago
From a professional perspective, does it make sense to get on a train that is so crowded already? Step 0 is probably to take Andrew Ng's on Coursera, but as of right now, you'd be among "2,647,287 already enrolled!" [0] [0] https://www.coursera.org/learn/machine-learning https://www.coursera.org/learn/machine-learning
- orasis 7y ago
  You have to be joking. Demand for ML skills, especially scaling in production, is off the charts.
  - username90 7y ago
    Scaling ML in production doesn't take much ML skills though.
    - omicron1 7y ago
      The bulk of industry ML work isn't actually suited to ML phds without engineering skills. I work in FAANG and this is a huge problem where the ML phds have poor communication with skilled engineers and a lack of engineering experience. They often even look down upon people who don't have fancy credentials. Unfortunately they just end up creating a money wasting disaster of a system.
  - deleted 7y ago
    [deleted]
- usaar333 7y ago
  I'd guess they have a very high drop-out rate. Regardless, "machine learning" is a very broad field and honestly I have no idea what an "ML engineer" is doing if they are one. It can cover any of the following: 1. Cutting-edge academic research (do better on this test set) 2. Doing data analysis to identify prediction ability 3. Creatively thinking of useful features to evaluate. 4. Implementing data pipelines/logging to obtain the features needed for #3. 5. Production systems to evaluate/train ML systems. (multiple places in the stack). Because the spectrum is so wide, if you are already an engineer, you can readily get into "ML" categories 3-5 and even 2. Andrew Ng's course is a valuable introduction and not that heavy of an investment -- I found that just with it (alongside my product and infra background), I could readily contribute to ML groups at my company.

lettergram 7y ago

In this particular case I think the headline of the title is probably better than the one on HN. Yes, I’m many fields ML has been exaggerated, but it’s just a matter of time. Most companies aren’t using ML effectively yet. But I know for a fact when you improve NLP, you _can_ improve almost all white collar jobs. NLP the past two years has been crazy, and we haven’t seen anything yet.

dmreedy 7y ago
I agree that there has been some impressive progress on NLP lately; am curious what you think the corresponding improvements in white collar jobs have been, though? Genuinely asking! Or do you think they are mostly yet to come?
- rpedela 7y ago
  I can at least give you some examples from my recent experience. A combination of text classification and NER can solve many business problems if the accuracy is high enough or you can mitigate errors. For e-commerce search, a common problem is how to give relevant results for misspelled queries or unique queries. Text classification and/or NER can often identify the product type, brand, etc and hopefully give a relevant result which leads to a sale. Another example is data entry or business analysis roles. Often they will manually extract data from sources and put it into Excel or some other BI tool. NLP ML has gotten good enough in just the last couple years that much of the data extraction can be automated, but you still need a human to verify accuracy.
- lettergram 7y ago
  From my own work: * NER (name-entity recognition) is becoming insanely good (99+%), this can help with everything from data security to automated form filling, to automated medical billing, etc. Multiple billion dollar industries will be shook up by this. * My own work on data generation is pointing to reduced surface area of security purposes and improved access to quasi-data for data science and application testing[1] * Translation from images to text descriptions are also a major thing, which when combined with other systems can do things like make suggested diagnosis. * Text generation systems are impacting how consumers interact. The chat bots these days are getting to the point where it's very difficult to identify if it's a human or not. I can't go into too many details, but see[2] * There's also my side business which is a search engine for people who would know an answer to a questions: https://insideropinion.com/ https://insideropinion.com/ In general, NLP is probably going to have a larger impact that driverless cars on our day-to-day lives. We're seeing insanely good story generation (e.g. GPT-2), chat bots, billing, etc. I also think WAY WAY more is yet to come. The transformer (which is spuring most of this) is only a couple years old. We'll see lots more applications as time goes on. [1] https://medium.com/capital-one-tech/why-you-dont-necessarily-need-data-for-data-science-48d7bf503074 https://medium.com/capital-one-tech/why-you-dont-necessarily... [2] https://arxiv.org/pdf/1908.01841.pdf https://arxiv.org/pdf/1908.01841.pdf

FiberBundle 7y ago

"The widespread misimpression that data + neural networks is a universal formula has real consequences: [...] in what students choose to study, and in what universities choose to teach." I think this is a problem that deserves more attention. A large portion of CS students nowadays choose to focus on Machine Learning, many brilliant CS students decide to get a PhD in ML in hope of contributing to a field that will continue to develop at a pace similar to the progress we have seen in the last 7 years. This view, I think, is not justified at all. The challenges that AI faces such as incorporating a world model into the techniques that work well will be significantly harder than what most researchers openly admit, probably to the detriment of other CS research areas.

didibus 7y ago
I've also noticed a trend of college graduates who in interview struggle with general software engineering practices and more fundamental coding skills and CS knowledge, but have knowledge and proficiency of ML. If you're a CS BA or MS, and don't plan to do an ML PhD, I'd suggest asking yourself if you want a data science job or a software engineering one. If the latter, remember to focus more on that. Surface knowledge of ML is a bonus, but the rest is more important.
- aidos 7y ago
  That’s definitely something I’ve seen too. Most of the cvs I get through, especially from younger devs, talk about ML. I’ve got my work cut out for me keeping my existing codebase understandably without deliberately introducing opacity!
- weego 7y ago
  My partner has ditched crosstraining into data science and ML because of how ridiculously 'ductape and string' the entire sub-indistry is. Tooling that is pre-2000 quality if it even exists, people with math skills yet only the most rudimentary ability in best practice around data storage and maintenance, coding, domain knowledge and understanding of how much bias they're introducing into results. Her view is its just a free for all of people who have little to no need to justify their output because everyone is waiting and hoping for the magic to happen.

sam0x17 7y ago

A problem is that the current mode of input for neural networks (one floating point value per input node) makes neural networks problematic for tackling problems with variable input size (read: most problems). My current research looks into how we can maybe fix that in certain cases, but in general it's a big limitation. Recurrent networks fix this to a certain extent, but they have not had the revolution that CNN's have enjoyed just yet. We can also do things like represent things in binary, but this adds complexity, and often greatly inhibits learning.

tus88 7y ago

Deep fakes are pretty good. It's kindof interesting....ML has allowed computers to generate realistic images that humans cannot discern....while image recognition remains woefully hopeless compared to human perception. So rather than competing with the human brain...AI will try and trick the human brain instead.

EpicEng 7y ago
>ML has allowed computers to generate realistic images that humans cannot discern I've yet to see a deep fake that's even remotely indiscernible from the real thing.
- andrewnicolalde 7y ago
  How do you know?
  - EpicEng 7y ago
    Well, technically I don't, for the reason I don't know if the abominable snowman exists. I have yet to see one I knew to be a df which was even close to being convincing.
    - joycian 7y ago
      Have you seen CTRL Shift Face, the YouTube channel?
- afterburner 7y ago
  You probably haven't been noticing them in movies though. Such as when they stuck Margot Robbie's entire head on an actual skater for I, Tonya. https://www.youtube.com/watch?v=bqgD6lHrQ_8 https://www.youtube.com/watch?v=bqgD6lHrQ_8
  - EpicEng 7y ago
    Was that a DF though?

dfan 7y ago

I assume from the title that this is either written by Gary Marcus or quotes him extensively. (I just checked and was correct.) It would be nice to hear this sort of critique from more than one person.

ycombinete 7y ago
Is he a good source of critique? Sincere question, I don’t know anything about the field.
- high_derivative 7y ago
  He is rather controversial in machine learning circles. I would say he is essentially right about the fact that a lot of ml work is crap. Like most of anything is crap, so that is not a groundbreaking insight and it comes off as somewhat grandstanding to keep saying that. He has however really turned this into a personal brand which many suspect he is using to advertise his startups. Many people feel he is sort of disingenuous about the actual progress being made in computer vision and NLP, even if there is many ways to go. To answer your question: You can read a single piece of his and maybe get some value from that, but he keeps rehashing the same points to sh*t over different new work, and there is not much to be learned from that after a while. Disclaimer: I did not read the above article in full, I am just familiar with Gary Marcus + work in ML industry. Edit: Keep expanding this but it got me thinking. Earlier in my career, I was also eager for the 'gotchas', for being able to point out why something is crap, exaggerated, cannot do what it promises, and so forth. As I outgrew this impulse to focus more on what is good about some work, I began spotting it on new students a lot. They would come into seminars and smugly crap all over recent papers. I have since come to the conclusion that it is useful to have realistic internal assessment of progress in your field, but that sharing your harshest criticisms is not necessarily a good way to look smart or make progress. It often turns out that while feeling clever about realizing limitations of some work, the field just moves forward. The people you just criticized leapfrogged while you were busy thinking about how to one-up them.
  - t_serpico 7y ago

personjerry 7y ago

That's because the media hype is completely different from the industry hype. This article is a form of media hype. The pragmatic, already realized effects of machine learning are mostly in improving existing systems, especially classification systems, and have increased metrics by big percentages at big data companies, and more applications are being found every day. That is why real demand and interest is so high for ML. These generative, speculative models are interesting experiments by researchers but inconsistent and far overstudied by the media, because "AI invents new physics" is more understandable and sexier than "AI improves lead generation by 6.8%"

spamizbad 7y ago
I would say the industry definitely has its own hype. And honestly, I think the news media is a trailing indicator on that hype. For example, there's already an ongoing replication crisis that's been ongoing for several years with ML papers in computer science. I've spoken with people in the industry who have even speculated that there's straight-up academic fraud taking place. If you get a splashy, sexy finding you can get scooped up by a FAANG-like company with a fat salary before anyone gives your work a second look.
- omicron1 7y ago
  From working with many ML researchers, it's not speculation. Some aren't very proud, but if you want to get ahead and stay competitive, you have to publish your un-replicable results. The results aren't usually completely fake, but they depend on enough dishonest sleight of hand that they might as well be.
mobjack 7y ago
You can improve lead generation by 6.8% without ML too. A/B testing a few changes can easily accomplish the same thing. The AI could be optimizing to attract people who are willing to give out their email address than actual potential customers so it doesn't help the business. We tried using Google ads to optimize for leads before and we saw ads were on sketchy websites promising people free stuff. Lead generation was a bad KPI to use but it was chosen can't optimize for sales because the volume is to low to train the ML models. This is the reality of many businesses that experiment in AI and they produce junk. ML does work well for certain domains where you have the scale of data to pull it off, but now the hype has spread where it is not appropriate.

deleted 7y ago

[deleted]

KaoruAoiShiho 7y ago

I'm not expert but I have a feeling this article will age poorly.

m3kw9 7y ago

DeepFake issues has definitely been exaggerated

genz9000 7y ago

ok boomer

egocodedinsol 7y ago

"You can see the computer age everywhere but in the productivity statistics." This was true for a few years, until it wasn't.

yters 7y ago

Not if we measure the revolution in games and kitten pictures! There has been tremendous progress in those domains!

xamuel 7y ago
What revolution in games do you have in mind? Is there any particular video game that is making really interesting use of ML?
- yters 7y ago
  I am referring to the playing of said games like with alphago and alphazero.
- friendlybus 7y ago
  All the production tools for making games/video are cramming ml tools in. It's one of the first places to be swimming in it. The two minute papers guy made a material predictor to cut down on preview render time. Quixel uses ml for search. Substance and Houdini are gaining tools soon. Blender has used it behind the scenes for its new neblua tool. Mocap & animation are bathed in it. There's probably a lot more I can't recall right now.

andbberger 7y ago

In my experience companies just aren't willing to invest, and get turned off when they realize just how long the timelines on bringing DL to production are. It certainly works, but magic it ain't. You need an experienced practitioner and lots of patience.

jokoon 7y ago

Once you realize there's "artificial" in AI, it quickly dispel the interest. Science cannot even agree on a practical definition of intelligence. Unless you let biologists, psychologists and neurologists work with computer scientists to advance the discussion, I don't think it's worth it to explore machine learning techniques to pursue intelligence. Don't call it ML or AI, call it "improved statistical prediction". The buzz will quickly fade.

hinkley 7y ago

Question for the collective wisdom: What, in your estimation, are some current or near-future trends that represent an objective improvement to the state of the art?

hinkley 7y ago

I would like to humbly submit, from my nose-bleed seat, that ML would serve us best by adding it to our arsenal of fuzzing techniques. Train a model and then set it loose to do exploratory testing, looking for valid-looking inputs that cause the wrong answer to be returned from our entirely pedestrian Plain Old Business Logic. The AI proposes a scenario, but the human is the final arbiter.

dgudkov 7y ago

>tendency among science journalists ... to overstate the significance of new advances in AI and machine learning It's very hard to not overstate when overstating brings you clicks, likes, retweets, and dopamine spikes.

buboard 7y ago

These articles (and a lot of what g. marcus writes) are attacking strawmen. I ve never heard no one claiming that NNs will invent new theories and i dont think that 2008 article is widely read. But, for things that are hard both computationally and theoretically, like protein folding, NNs may really be revolutionizing the field even if we don't know how they do it. Scientists do not buy-in foolishly into every AI hype. The problems lie with VCs and funding bodies, which are indeed swayed by adding "AI" to every kind of proposal. That's a separate problem, though

skywhopper 7y ago
That's the whole point... at the very beginning the post cites articles that suggest or even outright say that AI is coming up with new theories (or in the CS world, see all the articles and bloviators who predict AI will replace programming as a job). Regardless of what the scientists in the field would say one-on-one, the outright-wrong portrayal of ML in the press, egged on by VCs out to flip their investments, is causing problems for anyone who wants to actually make progress in their field, or recruit students honestly.
- buboard 7y ago
  > articles that suggest or even outright say that AI is coming up with new theories I would contest that there are widely read articles saying those things. Yes, surely someone publishes those , but they are generally dismissed. the field is conscious of its hype (e.g. check r/machinelearning), and i don't think students are stupid either

justicezyx 7y ago

The ratio of ML interested or majored job candidates I saw nowadays is staggering, appears >70%.

jhrmnn 7y ago

This year, I’ve been working on representing electronic wave functions with neural networks: https://arxiv.org/abs/1909.08423 https://arxiv.org/abs/1909.08423 I come from a physics background, and my impression is that NNs are a cool computational tool that every computational physicist probably should have in their toolbox (at least having a sense of their capability). But so far I haven’t seen any truly revolutionary advance with NNs acrose the whole of materials science.

Gatsky 7y ago

Still waiting for a revolution which has the optimal amount of hype. I wonder if this is an application for GANs.

deleted 7y ago

[deleted]

geoalchimista 7y ago

A recipe for the typical media hype "AI learns physics" discovery 1. Find a well-known classical problem with a battle-tested deterministic model representation 2. Generate synthetic data using the model 3. Train a 1-layer neural network to score 99% in predicting the synthetic data (even if the deterministic model can score 99.999% but let's ignore the minor difference) 4. Claim AI "learns" the underlying physics and does a better job than classical models 5. World dominance

knzhou 7y ago
This is exactly it. Earlier this year there was a hyped up paper [0] going around claiming that they could derive physics with AI. On inspection I found that what they actually did was this: 1. Take one simple physics equation, such as F = ma 2. Generate 10^6 tuples exactly satisfying this equation (F = 6, m = 2, a = 3), with absolutely no redundant, erroneous, or extraneous information 3. Feed these to a primitive "AI" system which brute-force tries the simplest relations that could relate the three provided variables (m = Fa, a = Fm, F = ma, ...) 4. Recover F = ma, declare physicists obsolete It's such a poor imitation of the real process of discovery in science that it's frankly insulting. When people hear AI is involved with something they turn their brains off. 0: https://arxiv.org/abs/1905.11481 https://arxiv.org/abs/1905.11481

Lapalux 7y ago

Machine learning = stats + computing power. The key thing here is that the core ingenuity of ML is statistics - the limits of which are well known. ML hype is false promises based on uninformed wild extrapolation.

blankdebut 7y ago

People doing research in ML have been saying something along these lines for years.

jowdones 7y ago

When I started with computers back in 1992, Eastern Europe, noone around me had the slightest idea what computers can do. I sort of imagined (and hoped) they will (magically) solve math problems for me so I won't need to actually make the hard effort to understand. 27 years later, not just 99.99% of the humans in general but of my "fellow" programmers, if I can call them so, are still stuck at this petty ignorance level.

le3dh4x0r 7y ago
Is that really so? I think this field has been hyped pretty hard and lots of people want a piece of that deep neural cake.

thecleaner 7y ago

Most ML will eventually be commoditized so it is unlikely that specializing only in ML will make things better for people who dont come from permier schools. If you have a so called MiML from any premier school you may not know jackshit about production Ml and still earn hundreds of thousands.