14 ms·

Good grief. When will people wake up and realize that AI today is just capable of "curve fitting"? Yes, that is a bit of a simplification. But not far off. N

Good grief. When will people wake up and realize that AI today is just capable of "curve fitting"? Yes, that is a bit of a simplification. But not far off. Neural networks depend on back propagation. They are really just another type of optimizer for maximum likelihood, using gradient descent. They work better on high dimensional, non linear data than other methods before. But if the function you are attempting to model is non differentiable, neural networks won't help you. They certainly aren't capable of performing magic tricks like writing an app for you.

navigatesol 7y ago
>They certainly aren't capable of doing magic tricks like writing an app for you. It's the money-men that need to realize it. Practitioners know that it's simply rebranded statistical methods. The usage is exactly the same. That said, some of the things these guys offer are useful; but branding it as "AI" is another story. By the way, is SoftBank going to implode, or what?
- EGreg 7y ago
  Softbank implode? Why?
  - sgarman 7y ago
    Presumably the OP is referring to the insane amount of money Softbank has invested in the AI space.
  - navigatesol 7y ago
    Because they are a massive fund throwing money into every half-baked idea coming out of Silicon Valley? https://www.zdnet.com/article/softbank-group-looking-to-ride-ai-unicorns-into-the-future/ https://www.zdnet.com/article/softbank-group-looking-to-ride... That image was copped form an investor presentation. How can anyone take that seriously?
- MiroF 7y ago
  I get the backlash against the hype, but to suggest that DL is simply "rebranded statistical methods" is just wrong. Yes, backprop existed before the recent explosion, that doesn't mean they're haven't been major leaps forward.
  - tachyonbeam 7y ago
    It's not that DL is simply "rebranded statistical methods", it's that the companies with names ending in "dot AI" are simply using "rebranded statistical methods". Lots of money is being poured into anything that calls itself "AI" right now, and a lot of people are starting to roll their eyes when they hear that term. I work with DL researchers on a daily basis, and I feel like there have been a lot of cool developments, but much of it is not ready for prime time. I would really like to see a list of successful DL deployments in the wild, with some info about accuracy if possible.
    - MiroF 7y ago
      I work in NLP and the industry has pretty much completely shifted to DL based approaches (ie. translation vendors pretty much only use neural engines now, language modeling and classification is very often neural as well now). There's definitely a lot that's not quite ready for prime time (reinforcement learning comes to mind as something that is particularly promising but not working really yet). > It's not that DL is simply "rebranded statistical methods", it's that the companies with names ending in "dot AI" are simply using "rebranded statistical methods". Lots of money is being poured into anything that calls itself "AI" right now, and a lot of people are starting to roll their eyes when they hear that term. This is definitely true - but I took it as a bit different from your original comment.
  - navigatesol 7y ago
    >Yes, backprop existed before the recent explosion, that doesn't mean they're haven't been major leaps forward. I didn't mean to imply that things were not advancing, only that what we claim is "AI" are processes we've been doing in statistics forever. But, yes, of course the tools and techniques improve.
rohan404 7y ago
Disclaimer - I'm a VP E at Engineer.ai AI is certainly not magic, and as an industry we're super far away from what would be considered real AI in the technical sense. That being said, AI has become a catch all term for everything as simple as linear regressions, all the way through to neural networks. We don't claim to be able to write apps using AI, we're a platform that is trying to use AI and general automation in order to optimize the traditional SDLC. Actual code generation/synthesis is years away in my opinion and there is far more impact that can be had by going after other manual aspects of software development.
- hitekker 7y ago
  Great, since you're here: what's your comment on the allegation that engineer.ai defrauded investors with fake AI?
  - rohan404 7y ago
    Disclaimer - I'm a VP E at Engineer.ai We've been very transparent with our investors on where we are in the process of creating this platform both pre-investment and post. They actually responded to the WSJ article: A spokeswoman for Deepcore said it has complete confidence in Mr. Duggal’s vision and team. A spokesman for Jungle Ventures said it is a proud investor in Engineer.ai and its technology, adding that “the AI landscape is a varied spectrum.” A Lakestar spokeswoman said it also has confidence in Engineer.ai and its team, adding that “growth in the AI space does not happen overnight.” It said Engineer.ai had been very careful in presenting its technology to Lakestar and other investors
    - mikeryan 7y ago
      All due respect but I wouldn't expect a VC to say anything else. These guys want to be able to peddle their stake onto someone else in a future round. I'll be more interested in what those investors do in your next funding round.
- vanadium 7y ago
  I was trying to read up on the site to evaluate what you said, but the UI's horribly broken on Firefox latest. Not the best first impression.
YeGoblynQueenne 7y ago
>> When will people wake up and realize that AI today is just capable of "curve fitting"? You are (possibly inadvertently) misquoting Judea Pearl and he was talking specifically about deep learning, not "AI": “All the impressive achievements of deep learning amount to just curve fitting,” he said recently. [1] "AI today" still means many more techniques and algorithms than deep learning. For example, SAT Solvers, logic programming and theorem provers, classical planning, classical search, adversarial search (MCTS) etc are alll AI techniques that have nothing to do with "curve fitting". So could we please all not throw about big proclamations about what "AI today" is ("just" or not), without first making sure that we have a thorough understanding of what we are saying? I thank us all in advance. ____________________ [1] https://www.quantamagazine.org/to-build-truly-intelligent-machines-teach-them-cause-and-effect-20180515/ https://www.quantamagazine.org/to-build-truly-intelligent-ma...
- codesushi42 7y ago
  For example, SAT Solvers, logic programming and theorem provers, classical planning, classical search, adversarial search (MCTS) etc are alll AI techniques that have nothing to do with "curve fitting". None of those techniques are new, nor are they fueling the AI hype cycle. I purposely conflated AI with "deep learning" because it is the source of the hype. And in reality, what most AI startups claim to be using.
  - YeGoblynQueenne 7y ago
    Deep learning is not new either. LSTMs are from the 1990s. CNNs are from the 1980's (and note well that most recent successes of deep learning are basically tweaks of these two architectures). What is new is the hype around deep learning that took off after 2012, and because Google and Facebook decided to champion it. In any case, as far as I can tell "AI today" is anything that is "AI" and that exists "today". How do you mean "AI today"? >> I purposely conflated AI with "deep learning" because it is the source of the hype. I don't understand why you would do that. You are aware that there is hype and that it is increased by misuse of the term AI. And you purposefully misuse the term AI in a way that increases the hype? Why?
Balgair 7y ago
For me, the bar is not high. If you can just replace the knowledge worker that is lower than 1 standard deviation less than the mean worker, that will be enough for me to say that it is General AI. What is a good test for this?: In under 4 hours can you have a program where I write something like a very detailed email (maybe I spend 10 minutes trying to craft the email) where I give a direct link to an internal file folder where there is a .pdf file and a .xlsx file. Each file contains names and birthdates amoung other things. I need the program to combine the data in these files and output it to .xlsx for me, giving me a list of names and birthdates from each file, combined in some sort of coherent manner that I can make sense of. If you can get a program to do something like that via a detailed email-like directive before lunch, not a bunch of python that constantly breaks every 6 months, that's beating about 30% of the workforce and might as well be General AI.
- michaelmrose 7y ago
  You will get a system that can do specifically that or very very similar tasks given a carefully worded email which amounts to an English like programming language created by people that cost an order of magnitude more than the people that would have massaged the data into a spreadsheet. This is actually probably a decent idea but it wouldn't be anything like general AI this year or next.
  - Balgair 7y ago
    I agree totally. But I hope you get the drift here. To get to a GAI system, it has to be something like the test provided. You have to go into detail, but it's not something that you'd have to hard code either. If I wanted the GAI to take in .docx files, or just first names, or to hop over to another directory and find things there, it should be able to do those things too. Basically, think of a task that you would give to a not-too-bright nephew that is in the firm for a summer internship. You'd need to give very clear instructions, but you should be able to coax him into getting some of the busy work done for you.
    - perl4ever 7y ago
tsss 7y ago
> But if the function you are attempting to model is non differentiable, neural networks won't help you. What? Any classification function is non-differentiable because it can only ever take discrete values. Yet this is a task that neural networks do every day. You can approximate a non-differentiable function with a differentiable one.
- codesushi42 7y ago
  That is not what I was referring to. Your relationship (neuron) needs to have a gradient, i.e. it needs to be differentiable, in order for backprop to work. There are many situations that are not differentiable. Images work well because adjusting color values during back prop can easily be done with a gradient. For example, during backprop it is feasible to increase the level of blue by 10% by increasing its RGB value. Likewise for 3%, 15%, -7%, etc. For other types of data, a gradient may not exist.
  - martindbp 7y ago
    > For other types of data, a gradient may not exist. You don't calculate a gradient on the data, you calculate a gradient on the parameters of the model. But I'm going to assume you mean that some types of data are not well represented by floating point numbers, which is true, but there are workarounds, e.g. vector embeddings for words or simply one-hot encodings for categorical values. What is true is that if you need to make a hard decision in one part of the network, for example deciding that the image is a cat, in order to further process that decision. This would not be differentiable. But we can represent objects in a non-discrete way, with embeddings, hidden state vectors etc. My point is, people are getting around this problem. I also want to add that I agree that we are way far off from building apps automatically, but I think you're giving way too little importance to the advances that's been made. Just take a look at the latest GANs, or the advances in machine translation using the Transformer and attention architectures, or Neural Turing Machines.
    - codesushi42 7y ago
      You don't calculate a gradient on the data, you calculate a gradient on the parameters of the model. Whether a gradient exists will depend on the type of data and what you're trying to predict. But we can represent objects in a non-discrete way, with embeddings, hidden state vectors etc. My point is, people are getting around this problem. Great progress has been made using that, e.g. advancements in NLU. But it is not a complete workaround. Embeddings don't work if your relationship is non differentiable. And you will lose the gradient for very long contexts (deep networks), even using new techniques like attention and transformers.
pojzon 7y ago
This one really hits the nail: "They are really just another type of optimizer for maximum likelihood" But like always in our industry - we have to hype something beyond believe to keep new projects rolling. Ill let ppl dream but I know enough to see how it all will end.. like always..
- MiroF 7y ago
  Do you actually work in the ML industry? Or are you just saying software writ large? If you actually work in the ML industry, I find it astonishing that you don't think that there have been major breakthroughs in the last 7 years
  - rurp 7y ago
    If only people directly working in ML notice these advancements, I don't think it makes sense to be astonished that some people do not consider them "major breakthroughs".
    - MiroF 7y ago
      Wait, so is ML overhyped or not? First I thought that everybody was covering AI as the next big thing which is absurd. But now, "nobody notices these advancements"? Which is it? My astonishment that you didn't think there had been any breakthroughs was only if you directly worked in ML, otherwise you are just misinformed.
  - pojzon 7y ago
    Should I be astonished by some field which recently got a lot of traction* made some breakthroughs ? (*investors money) I'm just stating that "again" in our industry, masses will now flock to "new shiny thing" simply because it is hyped and investors throw money at it. While managers will try to fit ML/DL/AI into pretty much anything either if it makes sense or not.
    - MiroF 7y ago
      > recently got a lot of traction This flips causality - funding didn't really really pick up until the last 5 years or so - about two years after the initial breakthrough. Look, I agree with this comment - managers are super eager to apply ML to tasks it has no business being applied to (at least yet). But your original claim was that all the new techniques are just glorified maximum likelihood optimization. That's just false.
dlphn___xyz 7y ago
they really should drop the term ‘AI’ and call it what it is - statistics
- drdeca 7y ago
  Would you say that AlphaGo is really just statistics? Even AlphaZero, which, iirc, is trained entirely using self-play, with no starting data from other players?
  - deepnotderp 7y ago
    While AI learners are not actually perfect generalizers so to speak, they are also quite clearly not purely statistical correlation machines, and there is a lot of evidence to show for this, such as the surprisingly similarities of ConvNets with low level human perception, etc. It seems to me that there is a category of people who are eager to dismiss deep learning altogether and say "iT's JuSt stAtIStics" even though there is a good amount of evidence to show that it isn't the case. That isn't real science, it's human bias.
    - jcranberry 7y ago
      Sounds to me like you're being overly reductive of statistics and probability.
      - codesushi42 7y ago
        There are a lot of folks in this thread who are AI "experts", fighting tooth and nail to defend their startup/PhD/career path, even if it means defending complete nonsense.
    - codesushi42 7y ago
      they are also quite clearly not purely statistical correlation machines, and there is a lot of evidence to show for this, such as the surprisingly similarities of ConvNets with low level human perception, etc. Wrong. If you actually study signal processing, you will find out that CNNs aren't something magically rooted in something other than "statistical correlation machines." CNNs in fact work because they're used to calculate cross correlation! "Convolving" in terms of a CNN is a misnomer, it's the same as calculating cross correlation in terms of signal processing.
GistNoesis 7y ago
Neural nets are one way to program which use the computation power available. It's a way to keep getting the benefits of Moore's law. The hardware is co-evolving with the software. GPUs with SIMD architecture, or TPU which are even more specialized will give a computational advantage to methods like neural nets which they have been designed for. The magic comes from the memory/computation power combo.
quotemstr 7y ago
Yet these neural networks are capable of exceeding human performance on tasks that just a few years ago people like you were saying no machine could do. Look: maybe ML is just curve fitting. But maybe human consciousness is too. As the scope of ML expands, we're going to have to confront the reality that there's nothing special about our own minds.
- deleted 7y ago
  [deleted]
- rightbyte 7y ago
  Maybe consciousness are just a bunch of conditional goto statements executed parallized with stocastic lag? My wild guess is that digital computers are not very efficient at AI. It would be really interesting if nets could be implemented in an analogue fashion.
deepnotderp 7y ago
This is not true at all, and no offense, is a misunderstanding of how NNs can be used in other applications. For one, what you are modeling itself does not need to be differentiable, only the network itself needs to be. Second, using neural networks in combination with other techniques for program synthesis is an active area of research currently, and although it is currently at around 50 lines or so, your fundamental assertion here is wrong. Third, there are a number of ways that deep learning could be leveraged in app development. The easiest way would be to make heuristic-type decisions around UI/UX or to build parts of said GUIs using existing code blocks. This has already been used to some extent in website design (e.g. https://arxiv.org/abs/1705.07962 https://arxiv.org/abs/1705.07962 and https://blog.floydhub.com/turning-design-mockups-into-code-with-deep-learning/ https://blog.floydhub.com/turning-design-mockups-into-code-w...). So it's certainly possible that it could be used in conjunction with templates to build common app types. Now, this startup is clearly not doing that, but, that doesn't mean that it's a) impossible to leverage AI for app development and b) all of deep learning is "just curve fitting"
- YeGoblynQueenne 7y ago
  To be fair, neural program synthesis has been an active area of research for a while and it doesn't seem that it's going to ever take off for good, like machine vision or NLP. I keep linking to this page: https://blog.keras.io/the-limitations-of-deep-learning.html https://blog.keras.io/the-limitations-of-deep-learning.html But what Chollet says is still the case. Machine-learning a mapping from arbitrary specifications to programs is many, many times more difficult than classification. Unless someone comes up with a completely new architecture that is for neural program synthesis what CNNs are for vision and LSTMs for sequence learning, and then some, then there's not going to be any big advances in the field. Source: I study algorithms that learn programs from examples from my PhD and you need three things for it that neural nets lack: a) generalisation, b) the ability to learn recursive functions and c) higher-order representations (i.e. quantified variables). Personally, I was very excited with DeepMind's differentiable neural computers, but it seems very hard to train on anything but toy problems.
  - marmaduke 7y ago
tlb 7y ago
A lot of things can be cast as curve fitting! Speech is just a curve of air pressure over time, so a function that takes the curve from a microphone and sends a new curve to a speaker can, if the curves correspond in interesting ways, be a conversational AGI. Curve fitting in millions of dimensions is qualitatively more interesting than the 2D graph paper exercise people think of.
- christiansakai 7y ago
  Are you trolling?
mark_l_watson 7y ago
I don’t want to sound critical, instead just providing some information: deep learning models like RoBERTa (a type of BERT, I experimented with Facebook’s just released large model this morning) can perform anaphora resolution (coreference) answer general questions, score whether two sentences contradict each other, etc. One model solves several very difficult problems whose solutions have evaded hand-coding efforts for decades. There are experimental seq models that transform paper text into figures or joint models that transform figures and text into some code, but you are correct that these are not production ready.
- YeGoblynQueenne 7y ago
  I think I read this paper because someone posted it on HN (or linked to it in a comment): https://arxiv.org/abs/1907.07355 https://arxiv.org/abs/1907.07355 We are surprised to find that BERT's peak performance of 77% on the Argument Reasoning Comprehension Task reaches just three points below the average untrained human baseline. However, we show that this result is entirely accounted for by exploitation of spurious statistical cues in the dataset. We analyze the nature of these cues and demonstrate that a range of models all exploit them. This analysis informs the construction of an adversarial dataset on which all models achieve random accuracy. Our adversarial dataset provides a more robust assessment of argument comprehension and should be adopted as the standard in future work. So it seems to be the benchmarks that are flawed and not BERT and friends who are that good at text comprehension. And that is not surprising. It would be surprising if language understanding just arose spontaneously by training a very big net on a very big corpus.
  - jeremysalwen 7y ago
    Referencing that paper is a red herring. It's a flaw in one specific dataset, not with BERT or any model. Everything GP posted about about BERT models answering questions, performing coreference resolution, etc, is still true and is not at all affected by the flaw in this one dataset (out of many that these models have been tested on). Heck, you can just try out any of these models yourself on completely novel questions you come up with, and see that they work.
chillacy 7y ago
When will people wake up and realize that computers today are just capable of "fetch decode execute" in a loop?
lone_haxx0r 7y ago
I am ignorant, might be really wrong and stupid, but aren't most problems in the world just curve-fitting in high dimensional spaces? Isn't human education simply discovering spaces that somewhat resemble a reality with arbitrarily many dimensions and finding functions that match curves/surfaces/bodies to some degree of precison and exactitude?
XnoiVeX 7y ago
You may want to read this paper. https://arxiv.org/abs/1904.01557 https://arxiv.org/abs/1904.01557
andreyk 7y ago
It's good to be wary of hype, but this comment seems overly down on where we are by focusing on AI today doing 'curve fitting' ; sure, supervised learning is at it's core that, but when it achieves things like image captioning or super accurate translation calling it just curve fitting is reductionist and downplays how powerful these techniques are. Furthermore, the notion that neural networks can only be used as part of supervised learning (and in particular with backpropagation) is totally off. Just to cite a few examples: http://proceedings.mlr.press/v48/taylor16.pdf http://proceedings.mlr.press/v48/taylor16.pdf https://arxiv.org/pdf/1908.01580v1.pdf https://arxiv.org/pdf/1908.01580v1.pdf And, well, all of deep RL and unsupervised learning is not 'curve fitting'. So yes, even if your 'model' (and really this should be 'objective' or 'loss', the neural net is the model, but whatever) neural nets can indeed help you. Though it's true they can't magically start doing super complicated things like app development yet ; the whole point is to understand how to use them and build hybrid systems that play to neural nets' strengths (as with eg self driving systems).
mattlondon 7y ago
Yeah I broadly agree that there is a lot of confusion about capabilities. Some years ago I was working somewhere and the management had caught the AI/ML bug and were obsessed with the idea of using ML to generate business "insights". They'd get some vague & unspecified data about a client's business operations, we'd input it into the ML and voila: "insights" about how to improve their business (and make us money) They didn't know what these "insights" would be, they expected machine learning to magically generate them on its own. I tried explaining that at a high level, ML can only really give you answers you already know are possibilities. It won't offer up some totally novel answer that you've not trained it for - i.e. you've got to know what the answers could be before you even start. We got shut down by the parent company not long after that.
- deleted 7y ago
  [deleted]
bigred100 7y ago
You can train a neural network without backpropagation (with a gradient method) and it will just be slower. There are plenty of derivative free methods as well. It’s not clear whether a function being nondifferentiable means anything without further specification since classification is practiced quite successfully in many cases.

navigatesol 7y ago

>They certainly aren't capable of doing magic tricks like writing an app for you. It's the money-men that need to realize it. Practitioners know that it's simply rebranded statistical methods. The usage is exactly the same. That said, some of the things these guys offer are useful; but branding it as "AI" is another story. By the way, is SoftBank going to implode, or what?

EGreg 7y ago
Softbank implode? Why?
- sgarman 7y ago
  Presumably the OP is referring to the insane amount of money Softbank has invested in the AI space.
- navigatesol 7y ago
  Because they are a massive fund throwing money into every half-baked idea coming out of Silicon Valley? https://www.zdnet.com/article/softbank-group-looking-to-ride-ai-unicorns-into-the-future/ https://www.zdnet.com/article/softbank-group-looking-to-ride... That image was copped form an investor presentation. How can anyone take that seriously?
MiroF 7y ago
I get the backlash against the hype, but to suggest that DL is simply "rebranded statistical methods" is just wrong. Yes, backprop existed before the recent explosion, that doesn't mean they're haven't been major leaps forward.
- tachyonbeam 7y ago
  It's not that DL is simply "rebranded statistical methods", it's that the companies with names ending in "dot AI" are simply using "rebranded statistical methods". Lots of money is being poured into anything that calls itself "AI" right now, and a lot of people are starting to roll their eyes when they hear that term. I work with DL researchers on a daily basis, and I feel like there have been a lot of cool developments, but much of it is not ready for prime time. I would really like to see a list of successful DL deployments in the wild, with some info about accuracy if possible.
  - MiroF 7y ago
    I work in NLP and the industry has pretty much completely shifted to DL based approaches (ie. translation vendors pretty much only use neural engines now, language modeling and classification is very often neural as well now). There's definitely a lot that's not quite ready for prime time (reinforcement learning comes to mind as something that is particularly promising but not working really yet). > It's not that DL is simply "rebranded statistical methods", it's that the companies with names ending in "dot AI" are simply using "rebranded statistical methods". Lots of money is being poured into anything that calls itself "AI" right now, and a lot of people are starting to roll their eyes when they hear that term. This is definitely true - but I took it as a bit different from your original comment.
- navigatesol 7y ago
  >Yes, backprop existed before the recent explosion, that doesn't mean they're haven't been major leaps forward. I didn't mean to imply that things were not advancing, only that what we claim is "AI" are processes we've been doing in statistics forever. But, yes, of course the tools and techniques improve.

rohan404 7y ago

Disclaimer - I'm a VP E at Engineer.ai AI is certainly not magic, and as an industry we're super far away from what would be considered real AI in the technical sense. That being said, AI has become a catch all term for everything as simple as linear regressions, all the way through to neural networks. We don't claim to be able to write apps using AI, we're a platform that is trying to use AI and general automation in order to optimize the traditional SDLC. Actual code generation/synthesis is years away in my opinion and there is far more impact that can be had by going after other manual aspects of software development.

hitekker 7y ago
Great, since you're here: what's your comment on the allegation that engineer.ai defrauded investors with fake AI?
- rohan404 7y ago
  Disclaimer - I'm a VP E at Engineer.ai We've been very transparent with our investors on where we are in the process of creating this platform both pre-investment and post. They actually responded to the WSJ article: A spokeswoman for Deepcore said it has complete confidence in Mr. Duggal’s vision and team. A spokesman for Jungle Ventures said it is a proud investor in Engineer.ai and its technology, adding that “the AI landscape is a varied spectrum.” A Lakestar spokeswoman said it also has confidence in Engineer.ai and its team, adding that “growth in the AI space does not happen overnight.” It said Engineer.ai had been very careful in presenting its technology to Lakestar and other investors
  - mikeryan 7y ago
    All due respect but I wouldn't expect a VC to say anything else. These guys want to be able to peddle their stake onto someone else in a future round. I'll be more interested in what those investors do in your next funding round.
vanadium 7y ago
I was trying to read up on the site to evaluate what you said, but the UI's horribly broken on Firefox latest. Not the best first impression.

YeGoblynQueenne 7y ago

>> When will people wake up and realize that AI today is just capable of "curve fitting"? You are (possibly inadvertently) misquoting Judea Pearl and he was talking specifically about deep learning, not "AI": “All the impressive achievements of deep learning amount to just curve fitting,” he said recently. [1] "AI today" still means many more techniques and algorithms than deep learning. For example, SAT Solvers, logic programming and theorem provers, classical planning, classical search, adversarial search (MCTS) etc are alll AI techniques that have nothing to do with "curve fitting". So could we please all not throw about big proclamations about what "AI today" is ("just" or not), without first making sure that we have a thorough understanding of what we are saying? I thank us all in advance. ____________________ [1] https://www.quantamagazine.org/to-build-truly-intelligent-machines-teach-them-cause-and-effect-20180515/ https://www.quantamagazine.org/to-build-truly-intelligent-ma...

codesushi42 7y ago
For example, SAT Solvers, logic programming and theorem provers, classical planning, classical search, adversarial search (MCTS) etc are alll AI techniques that have nothing to do with "curve fitting". None of those techniques are new, nor are they fueling the AI hype cycle. I purposely conflated AI with "deep learning" because it is the source of the hype. And in reality, what most AI startups claim to be using.
- YeGoblynQueenne 7y ago
  Deep learning is not new either. LSTMs are from the 1990s. CNNs are from the 1980's (and note well that most recent successes of deep learning are basically tweaks of these two architectures). What is new is the hype around deep learning that took off after 2012, and because Google and Facebook decided to champion it. In any case, as far as I can tell "AI today" is anything that is "AI" and that exists "today". How do you mean "AI today"? >> I purposely conflated AI with "deep learning" because it is the source of the hype. I don't understand why you would do that. You are aware that there is hype and that it is increased by misuse of the term AI. And you purposefully misuse the term AI in a way that increases the hype? Why?

Balgair 7y ago

For me, the bar is not high. If you can just replace the knowledge worker that is lower than 1 standard deviation less than the mean worker, that will be enough for me to say that it is General AI. What is a good test for this?: In under 4 hours can you have a program where I write something like a very detailed email (maybe I spend 10 minutes trying to craft the email) where I give a direct link to an internal file folder where there is a .pdf file and a .xlsx file. Each file contains names and birthdates amoung other things. I need the program to combine the data in these files and output it to .xlsx for me, giving me a list of names and birthdates from each file, combined in some sort of coherent manner that I can make sense of. If you can get a program to do something like that via a detailed email-like directive before lunch, not a bunch of python that constantly breaks every 6 months, that's beating about 30% of the workforce and might as well be General AI.

michaelmrose 7y ago
You will get a system that can do specifically that or very very similar tasks given a carefully worded email which amounts to an English like programming language created by people that cost an order of magnitude more than the people that would have massaged the data into a spreadsheet. This is actually probably a decent idea but it wouldn't be anything like general AI this year or next.
- Balgair 7y ago
  I agree totally. But I hope you get the drift here. To get to a GAI system, it has to be something like the test provided. You have to go into detail, but it's not something that you'd have to hard code either. If I wanted the GAI to take in .docx files, or just first names, or to hop over to another directory and find things there, it should be able to do those things too. Basically, think of a task that you would give to a not-too-bright nephew that is in the firm for a summer internship. You'd need to give very clear instructions, but you should be able to coax him into getting some of the busy work done for you.
  - perl4ever 7y ago

tsss 7y ago

> But if the function you are attempting to model is non differentiable, neural networks won't help you. What? Any classification function is non-differentiable because it can only ever take discrete values. Yet this is a task that neural networks do every day. You can approximate a non-differentiable function with a differentiable one.

codesushi42 7y ago
That is not what I was referring to. Your relationship (neuron) needs to have a gradient, i.e. it needs to be differentiable, in order for backprop to work. There are many situations that are not differentiable. Images work well because adjusting color values during back prop can easily be done with a gradient. For example, during backprop it is feasible to increase the level of blue by 10% by increasing its RGB value. Likewise for 3%, 15%, -7%, etc. For other types of data, a gradient may not exist.
- martindbp 7y ago
  > For other types of data, a gradient may not exist. You don't calculate a gradient on the data, you calculate a gradient on the parameters of the model. But I'm going to assume you mean that some types of data are not well represented by floating point numbers, which is true, but there are workarounds, e.g. vector embeddings for words or simply one-hot encodings for categorical values. What is true is that if you need to make a hard decision in one part of the network, for example deciding that the image is a cat, in order to further process that decision. This would not be differentiable. But we can represent objects in a non-discrete way, with embeddings, hidden state vectors etc. My point is, people are getting around this problem. I also want to add that I agree that we are way far off from building apps automatically, but I think you're giving way too little importance to the advances that's been made. Just take a look at the latest GANs, or the advances in machine translation using the Transformer and attention architectures, or Neural Turing Machines.
  - codesushi42 7y ago
    You don't calculate a gradient on the data, you calculate a gradient on the parameters of the model. Whether a gradient exists will depend on the type of data and what you're trying to predict. But we can represent objects in a non-discrete way, with embeddings, hidden state vectors etc. My point is, people are getting around this problem. Great progress has been made using that, e.g. advancements in NLU. But it is not a complete workaround. Embeddings don't work if your relationship is non differentiable. And you will lose the gradient for very long contexts (deep networks), even using new techniques like attention and transformers.

pojzon 7y ago

This one really hits the nail: "They are really just another type of optimizer for maximum likelihood" But like always in our industry - we have to hype something beyond believe to keep new projects rolling. Ill let ppl dream but I know enough to see how it all will end.. like always..

MiroF 7y ago
Do you actually work in the ML industry? Or are you just saying software writ large? If you actually work in the ML industry, I find it astonishing that you don't think that there have been major breakthroughs in the last 7 years
- rurp 7y ago
  If only people directly working in ML notice these advancements, I don't think it makes sense to be astonished that some people do not consider them "major breakthroughs".
  - MiroF 7y ago
    Wait, so is ML overhyped or not? First I thought that everybody was covering AI as the next big thing which is absurd. But now, "nobody notices these advancements"? Which is it? My astonishment that you didn't think there had been any breakthroughs was only if you directly worked in ML, otherwise you are just misinformed.
- pojzon 7y ago
  Should I be astonished by some field which recently got a lot of traction* made some breakthroughs ? (*investors money) I'm just stating that "again" in our industry, masses will now flock to "new shiny thing" simply because it is hyped and investors throw money at it. While managers will try to fit ML/DL/AI into pretty much anything either if it makes sense or not.
  - MiroF 7y ago
    > recently got a lot of traction This flips causality - funding didn't really really pick up until the last 5 years or so - about two years after the initial breakthrough. Look, I agree with this comment - managers are super eager to apply ML to tasks it has no business being applied to (at least yet). But your original claim was that all the new techniques are just glorified maximum likelihood optimization. That's just false.

dlphn___xyz 7y ago

they really should drop the term ‘AI’ and call it what it is - statistics

drdeca 7y ago
Would you say that AlphaGo is really just statistics? Even AlphaZero, which, iirc, is trained entirely using self-play, with no starting data from other players?
- deepnotderp 7y ago
  While AI learners are not actually perfect generalizers so to speak, they are also quite clearly not purely statistical correlation machines, and there is a lot of evidence to show for this, such as the surprisingly similarities of ConvNets with low level human perception, etc. It seems to me that there is a category of people who are eager to dismiss deep learning altogether and say "iT's JuSt stAtIStics" even though there is a good amount of evidence to show that it isn't the case. That isn't real science, it's human bias.
  - jcranberry 7y ago
    Sounds to me like you're being overly reductive of statistics and probability.
    - codesushi42 7y ago
      There are a lot of folks in this thread who are AI "experts", fighting tooth and nail to defend their startup/PhD/career path, even if it means defending complete nonsense.
  - codesushi42 7y ago
    they are also quite clearly not purely statistical correlation machines, and there is a lot of evidence to show for this, such as the surprisingly similarities of ConvNets with low level human perception, etc. Wrong. If you actually study signal processing, you will find out that CNNs aren't something magically rooted in something other than "statistical correlation machines." CNNs in fact work because they're used to calculate cross correlation! "Convolving" in terms of a CNN is a misnomer, it's the same as calculating cross correlation in terms of signal processing.

GistNoesis 7y ago

Neural nets are one way to program which use the computation power available. It's a way to keep getting the benefits of Moore's law. The hardware is co-evolving with the software. GPUs with SIMD architecture, or TPU which are even more specialized will give a computational advantage to methods like neural nets which they have been designed for. The magic comes from the memory/computation power combo.

quotemstr 7y ago

Yet these neural networks are capable of exceeding human performance on tasks that just a few years ago people like you were saying no machine could do. Look: maybe ML is just curve fitting. But maybe human consciousness is too. As the scope of ML expands, we're going to have to confront the reality that there's nothing special about our own minds.

deleted 7y ago
[deleted]
rightbyte 7y ago
Maybe consciousness are just a bunch of conditional goto statements executed parallized with stocastic lag? My wild guess is that digital computers are not very efficient at AI. It would be really interesting if nets could be implemented in an analogue fashion.

deepnotderp 7y ago

This is not true at all, and no offense, is a misunderstanding of how NNs can be used in other applications. For one, what you are modeling itself does not need to be differentiable, only the network itself needs to be. Second, using neural networks in combination with other techniques for program synthesis is an active area of research currently, and although it is currently at around 50 lines or so, your fundamental assertion here is wrong. Third, there are a number of ways that deep learning could be leveraged in app development. The easiest way would be to make heuristic-type decisions around UI/UX or to build parts of said GUIs using existing code blocks. This has already been used to some extent in website design (e.g. https://arxiv.org/abs/1705.07962 https://arxiv.org/abs/1705.07962 and https://blog.floydhub.com/turning-design-mockups-into-code-with-deep-learning/ https://blog.floydhub.com/turning-design-mockups-into-code-w...). So it's certainly possible that it could be used in conjunction with templates to build common app types. Now, this startup is clearly not doing that, but, that doesn't mean that it's a) impossible to leverage AI for app development and b) all of deep learning is "just curve fitting"

YeGoblynQueenne 7y ago
To be fair, neural program synthesis has been an active area of research for a while and it doesn't seem that it's going to ever take off for good, like machine vision or NLP. I keep linking to this page: https://blog.keras.io/the-limitations-of-deep-learning.html https://blog.keras.io/the-limitations-of-deep-learning.html But what Chollet says is still the case. Machine-learning a mapping from arbitrary specifications to programs is many, many times more difficult than classification. Unless someone comes up with a completely new architecture that is for neural program synthesis what CNNs are for vision and LSTMs for sequence learning, and then some, then there's not going to be any big advances in the field. Source: I study algorithms that learn programs from examples from my PhD and you need three things for it that neural nets lack: a) generalisation, b) the ability to learn recursive functions and c) higher-order representations (i.e. quantified variables). Personally, I was very excited with DeepMind's differentiable neural computers, but it seems very hard to train on anything but toy problems.
- marmaduke 7y ago

tlb 7y ago

A lot of things can be cast as curve fitting! Speech is just a curve of air pressure over time, so a function that takes the curve from a microphone and sends a new curve to a speaker can, if the curves correspond in interesting ways, be a conversational AGI. Curve fitting in millions of dimensions is qualitatively more interesting than the 2D graph paper exercise people think of.

christiansakai 7y ago
Are you trolling?

mark_l_watson 7y ago

I don’t want to sound critical, instead just providing some information: deep learning models like RoBERTa (a type of BERT, I experimented with Facebook’s just released large model this morning) can perform anaphora resolution (coreference) answer general questions, score whether two sentences contradict each other, etc. One model solves several very difficult problems whose solutions have evaded hand-coding efforts for decades. There are experimental seq models that transform paper text into figures or joint models that transform figures and text into some code, but you are correct that these are not production ready.

YeGoblynQueenne 7y ago
I think I read this paper because someone posted it on HN (or linked to it in a comment): https://arxiv.org/abs/1907.07355 https://arxiv.org/abs/1907.07355 We are surprised to find that BERT's peak performance of 77% on the Argument Reasoning Comprehension Task reaches just three points below the average untrained human baseline. However, we show that this result is entirely accounted for by exploitation of spurious statistical cues in the dataset. We analyze the nature of these cues and demonstrate that a range of models all exploit them. This analysis informs the construction of an adversarial dataset on which all models achieve random accuracy. Our adversarial dataset provides a more robust assessment of argument comprehension and should be adopted as the standard in future work. So it seems to be the benchmarks that are flawed and not BERT and friends who are that good at text comprehension. And that is not surprising. It would be surprising if language understanding just arose spontaneously by training a very big net on a very big corpus.
- jeremysalwen 7y ago
  Referencing that paper is a red herring. It's a flaw in one specific dataset, not with BERT or any model. Everything GP posted about about BERT models answering questions, performing coreference resolution, etc, is still true and is not at all affected by the flaw in this one dataset (out of many that these models have been tested on). Heck, you can just try out any of these models yourself on completely novel questions you come up with, and see that they work.

chillacy 7y ago

When will people wake up and realize that computers today are just capable of "fetch decode execute" in a loop?

lone_haxx0r 7y ago

I am ignorant, might be really wrong and stupid, but aren't most problems in the world just curve-fitting in high dimensional spaces? Isn't human education simply discovering spaces that somewhat resemble a reality with arbitrarily many dimensions and finding functions that match curves/surfaces/bodies to some degree of precison and exactitude?

XnoiVeX 7y ago

You may want to read this paper. https://arxiv.org/abs/1904.01557 https://arxiv.org/abs/1904.01557

andreyk 7y ago

It's good to be wary of hype, but this comment seems overly down on where we are by focusing on AI today doing 'curve fitting' ; sure, supervised learning is at it's core that, but when it achieves things like image captioning or super accurate translation calling it just curve fitting is reductionist and downplays how powerful these techniques are. Furthermore, the notion that neural networks can only be used as part of supervised learning (and in particular with backpropagation) is totally off. Just to cite a few examples: http://proceedings.mlr.press/v48/taylor16.pdf http://proceedings.mlr.press/v48/taylor16.pdf https://arxiv.org/pdf/1908.01580v1.pdf https://arxiv.org/pdf/1908.01580v1.pdf And, well, all of deep RL and unsupervised learning is not 'curve fitting'. So yes, even if your 'model' (and really this should be 'objective' or 'loss', the neural net is the model, but whatever) neural nets can indeed help you. Though it's true they can't magically start doing super complicated things like app development yet ; the whole point is to understand how to use them and build hybrid systems that play to neural nets' strengths (as with eg self driving systems).

mattlondon 7y ago

Yeah I broadly agree that there is a lot of confusion about capabilities. Some years ago I was working somewhere and the management had caught the AI/ML bug and were obsessed with the idea of using ML to generate business "insights". They'd get some vague & unspecified data about a client's business operations, we'd input it into the ML and voila: "insights" about how to improve their business (and make us money) They didn't know what these "insights" would be, they expected machine learning to magically generate them on its own. I tried explaining that at a high level, ML can only really give you answers you already know are possibilities. It won't offer up some totally novel answer that you've not trained it for - i.e. you've got to know what the answers could be before you even start. We got shut down by the parent company not long after that.

deleted 7y ago
[deleted]

bigred100 7y ago

You can train a neural network without backpropagation (with a gradient method) and it will just be slower. There are plenty of derivative free methods as well. It’s not clear whether a function being nondifferentiable means anything without further specification since classification is practiced quite successfully in many cases.