10 ms·

The future of deep learning

242 points by nicolrx 9y ago

therajiv 9y ago
TLDR is that models will become more abstract (current pattern recognition will blend with formal reasoning and abstraction), modular (think transfer learning, but taken to its extreme - every trained model's learned representations should be applicable to other tasks), and automated (ML experts will spend less time in the repetitive training/optimization cycle, instead focusing more on how models apply to their specific domain).
- radarsat1 9y ago
  I think it's true, but I hope this synergy between logic and pattern recognition actually happens, as I feel like this has been proposed for years but never really come to fruition. However, with recent work on differentiable communicating agents, differentiable memory etc., perhaps it now has a chance to get there.
  - eli_gottlieb 9y ago
    The author says not everything should be differentiable. Intuitively, I agree, but the question is how to do a sufficiently fast search through a high-dimensional space when you don't have a gradient.
    - guicho271828 9y ago
      Guided by domain-independent lowerbound functions, obviously. There are lots of research in this area, see every year there are papers on heuristic search in AAAI. ML people are simply not reading them and citing them.
      - eli_gottlieb 9y ago
        Could you point some out to me? I haven't been deliberately avoiding them by any means, but I'd like to see some that allow fast search. And especially, if you've got it, fast stochastic search for Monte Carlo methods.
        guicho271828 9y ago
        I bet you already know MCTS. But I see you have a problem with defining "fast". Define "fast". You can usually only say something is faster or slower. Absolute speed depends on the machine resource and the low-level optimization. If you take MCTS it is "enough fast" considering it has already beat Lee Sedol. There are multiple means to measure the speed. Blind search is "fast" re: node expansion but takes exponential time to find a solution and quite likely exhausts memory. Search with heavy heuristic functions has "slow" expansion but can solve the problem quickly, despite the cost of evaluating them, since it can prune a large part of the state space and expands fewer nodes. Only the latter is practically meaningful. Just read the classical AI textbooks like AIMA first, then [this one](https://www.amazon.co.uk/Heuristic-Search-Applications-Stefan-Edelkamp/dp/0123725127 https://www.amazon.co.uk/Heuristic-Search-Applications-Stefa...) if you want to know more. Maybe [this short paper](https://www.cs.cmu.edu/~maxim/files/searchcomesofage_aaai12_invitedpaper.pdf https://www.cs.cmu.edu/~maxim/files/searchcomesofage_aaai12_...) may give you the gist.
        Radim 9y ago
        On the topic of "likely exhausts memory": The best search methods allow reversible state updates. The reversibility makes things super-fast -- you no longer need to copy all data structures on path expansion. Instead, you modify only a single representation, incrementally. And when retracting a search step, you "undo" the same modifications again, arriving at the exact same state as when you started. This is of course non-trivial -- it is much easier to copy everything, then throw away the entire copy when it's not needed, rather than keeping a single state incrementally consistent. But the effects due to data locality (excellent caching), better memory management (no allocations, fragmentation) and less work (only touch and update parts of the state that matter) can be tremendous. I haven't seen any discussion of DeepMind's implementation details for AlphaGo, but since they come from a game development background (David Silver was the CTO and lead dev at Elixir Studios), where each cycle counts, I have no doubt they're well familiar with all these concepts. But then the TPU throws a wrench into it again...
    - CuriouslyC 9y ago
      If you don't have a gradient, one tactic is to make the most of the situation. Give your model the Bayesian treatment, and sample from the posterior using MCMC. This is slow, but you end up with posteriors on your parameter values, which is a huge win.
      - eli_gottlieb 9y ago
        Yeah, I've been a big fan of probabilistic programming for a while. The real problem is that getting Monte Carlo methods to converge and produce a large sample from the posterior takes orders of magnitude more time than running an optimizer to descend a gradient. Hey, you can even make it a probabilistic gradient: variational inference! But then you still have a hard time with discrete, nondifferentiable structure.
ipunchghosts 9y ago
Great work! Glad someone can finally explain this to the masses in an easy to understand way. Looking forward to the future!
kirillkh 9y ago
Seeing how gradient descent is such a pinnacle of deep learning, I can't help wondering: is this how our brain learns? If not, then what prevents us from implementing deep learning the same way?
- ipunchghosts 9y ago
  Our brains do not use gradient descent. Last i read they use some kind of local optimization. Can someone elaborate on this?
- rsiqueira 9y ago
  One of the most consistent theory about how our brain learns is described in HTM (Hierarchical Temporal Memory), a more biologically inspired neural network. See Jeff Hawkins' "On Intelligence". It is based on: * Input of continuous unlabeled time-based patterns. * Associative Hebbian Learning (when distinct inputs/patterns come together, they are neuron-wired together). Synapses can be modified via experience. See "Hebbian Theory". * The brain is a prediction machine: it is always trying to predict the future based on past learned patterns. Learning happens when reality does not match the originally prediction and we rewire the world model based on new input. See "Bayesian approaches to brain function". * Input signals are processed by many layers, each one creating more abstraction from the previous one, from sensory neurons to the highest cortex layers. * Each region of the hierarchy forms invariant memories (what a typical region of cortex learns is sequences of invariant representations). * There is lots of feedback (highest level neurons back to the lowest levels). In some structures (e.g. the thalamus, that is a kind of "hub of information") connections going backward (toward the input) exceed the connections going forward by almost a factor of ten. * Brain uses Sparse Distributed Memory (SDM). See SDM by Pentti Kanerva (NASA researcher). * Neuron models have many more variable/parameters (that can be used to transfer or process information) than usual nodes/links from artificial neural networks. E.g.: Long-term potentiation vs Long-term depression, neuronal Habituation vs Sensitization, inhibitory vs excitatory neurons, firing rates, synchronization, neuromodulation, homeostasis and more. * The backward propagation of errors in artificial neural networks only occurs during the learning phase. But the brain is always learning and updating weights and relationships between patterns, given new inputs. * During repetitive learning, representations of objects move down the cortical hierarchy (from short-term memory to long-term memory), forming invariant memories. * The brain needs to replay the memory (memory rehearsal) of a learned stimulus so it can be stored in long-term memory. * The job of any cortical region is to find out how inputs are related (pattern recognition), to memorize the sequence of correlations between them, and to use this memory to predict how the inputs will behave in the future. * Predictive coding: the brain is constantly generating and updating hypotheses that predict sensory input at varying levels of abstraction.
guicho271828 9y ago
Regarding logic and DL, there is NeSy workshop in London http://neural-symbolic.org/ http://neural-symbolic.org/
jdonaldson 9y ago
Glad to see Deep Learning "coming down to earth". This is the first high profile post I've seen that spells out exactly how DL models will become reconfigurable, purpose-built tools, and what a workflow might look like. We're still a long way aways from treating them like software components.
- Cacti 9y ago
  I mean, these are topics that have been discussed countless times over the years and in some cases decades. It's all well and good to say we need generalizable machines, and something other than backprop, and something closer to traditional programs, but we all know this. The issue is that no one knows what this would even mean, never mind how one would go about implementing it. In the few cases we do know how, the results are horrible compared to the methods we already use. We use the methods we do today because they work, not because we think they are the best, or because we don't understand the limitations of our models.
  - jdonaldson 9y ago
    True, there's been discussions, but from what I've seen it's mostly flag planting or vague pop-eng fodder that project directors dish out to tech journalists. Having Keras make a statement on this carries far more weight, because fchollet is not selling a product, or pushing an agenda, or creating a walled garden of some sort. The only thing that's a bit off about Keras is that it's mostly the efforts of one guy. Sure, there's many other contributors, but they don't seem to be acknowledged. I've never seen anyone else speak for the project. I'd really like to see a neutral party emerge for deep learning practice and tooling, before the whole industry gets sucked into a single dominant ecosystem like AWS.
    - droidist2 9y ago
      Do you think with Google's adoption of Keras for TensorFlow it'll get more resources dedicated to it?
nextstar 9y ago
I'm excited for the easy to use tools that have to be coming out relatively soon. There are a lot right now, but the few I've used weren't super intuitive like I feel like they could be.
computerex 9y ago
> In DeepMind's AlphaGo, for example, most of the "intelligence" on display is designed and hard-coded by expert programmers (e.g. Monte-Carlo tree search); Not true. This paraphrases the original paper: https://www.tastehit.com/blog/google-deepmind-alphago-how-it-works/ https://www.tastehit.com/blog/google-deepmind-alphago-how-it... > They tested their best-performing policy network against Pachi, the strongest open-source Go program, and which relies on 100,000 simulations of MCTS at each turn. AlphaGo's policy network won 85% of the games against Pachi! I find this result truly remarkable. A fast feed-forward architecture (a convolutional network) was able to outperform a system that relies extensively on search. Also, this article reeked of AGI ideas. Deep learning isn't trying to solve AGI. Reasoning and abstraction and high level AGI concepts that I don't think apply to deep learning. I don't know the path to AGI but I don't think it'll be deep learning. I think it would have to be fundamentally different.
- nojvek 9y ago
  > Reasoning and abstraction and high level AGI concepts that I don't think apply to deep learning What do you mean? Each layer is selecting features and abstracting previous layers. A cat neuron abstracts all possible pixels that form cats. He's just saying that we need better ways of composing this `deep` abstractions.
  - espadrine 9y ago
    > we need better ways of composing this `deep` abstractions Tensor2Tensor[0], from the Google Brain team, has some strong recent results in that direction. Related paper: "One Model To Learn Them All"[1]. [0]: https://github.com/tensorflow/tensor2tensor https://github.com/tensorflow/tensor2tensor [1]: https://arxiv.org/pdf/1706.05137.pdf https://arxiv.org/pdf/1706.05137.pdf
- landon32 9y ago
  >Also, this article reeked of AGI ideas. Deep learning isn't trying to solve AGI. Reasoning and abstraction and high level AGI concepts that I don't think apply to deep learning. I don't know the path to AGI but I don't think it'll be deep learning. I think it would have to be fundamentally different. I think that this is actually what the article is arguing. from the article: >Models closer to general-purpose computer programs, built on top of far richer primitives than our current differentiable layers—this is how we will get to reasoning and abstraction, the fundamental weakness of current models. This means not using current deep learning ideas, and instead finding ways to integrate other types of programs (conventional algorithms, other types of ML) alongside Deep Networks.
nzonbi 9y ago
Interesting article, in a difficult topic. Speculating about the future of deep learning. The author deserves recognition for writing about this. In my personal opinion, within the next 10 years, there will be systems exhibiting basic general intelligence behavior. I am currently doing early hobbist research on it, and I see it as feasible. These system will not be very powerful initially. They will exist and work in simpler simulated environments. Eventually we will be able to make these systems powerful enough to handle the real world. Although that will probably not be easy. I somewhat disagree with the author. I don't think that deep learning systems of the future are going to generate "programs", composed of programming primitives. In my speculative view, the key for general intelligence is not very far from our current knowledge. Deep learning, as currently we have, is a good enough basic tool. There are no magic improvements to the current deep learning algorithms, hidden around the corner. Rather what I think will enable general intelligence, is assembling systems of deep learning networks in the right setup. Some of the structure of these systems will be similar to traditional programs. But the models they generate will not resemble computer programs. They will be more like data graphs. I expect within 10 years there will be computer agents capable of communicating in simplified, but functional languages. Full human language capability will come after that. And within 20 years I expect artificial general intelligence to exist. At least in a basic form. That is my personal view. I am currently working on this.
- ColanR 9y ago
  what kind of research have you done on this? Most of what I hear about this is more pessimistic than anything else. I'm curious what you've learned that's made you optimistic.
  - nzonbi 9y ago
    I am working on creating basic artificial general intelligence. My research is still early, and I don't want to give much details for now. For example, in my opinion it is wrong to feed a giant neural network with tons of text, and expect it to understand human language. That will never work. Language understanding requires a grasp of a world where the agent exist first. A general understanding of things. Then, on top of that, it can build basic communication. And then it can be hoped that this communication can become sophisticated enough, to reach human language level. Another example: Feeding a neural network with millions of product reviews, and expect it to be capable to understand and write product reviews is hopelessly laughable. Not even with petabytes of data. I started working on this, because I think that not enough focus is being put on AGI, or research is not creative enough. At least not that is being published. I am optimist with my work, and soon I will reveal more. But even if I don't reach my goals, I think that it is just a matter of persistence. At any moment, someone will solve the problem of AGI.
toisanji 9y ago
This is part 2 from the post yesterday: https://news.ycombinator.com/item?id=14790251 https://news.ycombinator.com/item?id=14790251 And the author posted a comment on hn: "fchollet: Hardly a "made-up" conclusion -- just a teaser for the next post, which deals with how we can achieve "extreme generalization" via abstraction and reasoning, and how we can concretely implement those in machine learning models." I like the ideas presented in the post, but its not concrete or new at all.Basically he writes "everything will get better". I do agree with the point that we need to move away from strictly differential learning though. All deep learning problems only work on systems that have derivates so we can do backpropagation. I dont think the brain learns with backpropagation at all. * AutoML, there are dozens of these type of systems already, he mentions one already in the post called HyperOpt. So we will continue to use this systems and they will get smarter? Many of these systems are basically grid search/brute force. Do you think the brain is doing brute force at all? We have to use these now because there are no universal correct hyperparameters for tuning these models. As long as we build AI models the way we do now, we will have to do this hyperparameter tuning. Yes, these will get better, again, nothing new here. * He talks about reusable modules. Everyone in the deep learning community has been talking about this a lot, its called transfer learning and people are using it now, and working on making it better all the time. We currently have "model zoos" which are databases of pretrained models that you can use. If you want to see a great scifi short piece on what neural network mini programs could look like written by the head of computer vision at tesla, check out this post: http://karpathy.github.io/2015/11/14/ai/ http://karpathy.github.io/2015/11/14/ai/
- ipunchghosts 9y ago
  Everyone makes the assumption that computers should get to be as smart as humans but in some ways, its the other way around. For example, the human brain is not a turing machine, it doesnt have memory (in the sense that its lossy). You need memory to have a turing machine so with a paper and pencil, a human is a turing machining but a very slow run. Compare the difference to read and write on paper than a computer has to access ram. I think there will be some kind of meta deep learing (still using deep learning but compose of algebras which are augmented compared to today's standards). We have already started this by using pretrained networks for tasks. There is no reason RNNs won't go this way (i imagine they already are but this isnt my research area specifically) after all, RNNS are a turing machine.
lopatin 9y ago
I'm also interested to see how the worlds of program synthesis (specifically type directed, proof-checking, dependently typed stuff) can combine with deep learning. If recent neural nets have such great results on large amounts of unstructured data, imagine what they can do with a type lattice.
- gtani 9y ago
  Recent baby steps in gradient checking: https://news.ycombinator.com/item?id=14739491 https://news.ycombinator.com/item?id=14739491
crypticlizard 9y ago
Are there popular modern libraries that do program synthesis? Although I've thought about this and read about the concept on hn, I've not heard it mentioned seriously or frequently or strenuously as a thing to do either just for fun or to get a job doing it. This could be a popular way to solve programming problems without needing programmers. I think this truly would kick off AI as a very personal experience for the masses because they would use AI basically like they do already do now with a search engine. People would use a virtual editor to design their software using off the shelf parts freely available. The level of program complexity could really skyrocket as people now have more control over what and how they run programs because they can easily design it themselves. Everyone could design their own personal Facebook or Twitter and probably a whole new series of websites too complex or for other reasons not invented yet. For instance, you want to program the personality of a toy, so you search around using the AI search engine for parts that might work. Or you want a relationship advice coach so you put it together using personalities you like, taking only the parts you want from each personality. Or another example would be just to make remixes of media you like. Because everything works without programming anyone can participate.
- randcraw 9y ago
  Check out Genetic Programming: https://en.wikipedia.org/wiki/Genetic_programming https://en.wikipedia.org/wiki/Genetic_programming AFAIK GP remains the primary means to automate the synthesis of software. Though it was introduced perhaps 30 years ago, it hasn't been an active area of research for the past 20, AFAIK.
scientist 9y ago
Self-driving cars are expected to take over the roads, however no programmer is able to write code that does this directly, without machine learning. However, programmers have built all kinds of software of great value, from operating systems to databases, desktop software and so on. Much of this software is open source and artificial systems can learn from it. Therefore, it could well be that, in the end, it would be easier to build artificial systems that learn to automatically develop such software than systems that autonomously drive cars, if the right methodologies are used. The author is right to say that neural program synthesis is the next big thing, and this also motivated me to switch my research to this field. If you have a PhD and are interested in working in neural program synthesis, please check out these available positions: http://rist.ro/job-a3 http://rist.ro/job-a3
amelius 9y ago
What about the future of jobs in the field of deep learning? EDIT: I'm thinking deep learning will become much like web development is today. Everybody can do it, and only a few experts will work at the technological frontier and develop tools and libraries for everybody else to use. Therefore, if one invests time in DL, then I suppose it better be a serious effort (at research level), rather than at the level of invoking a library, because soon everybody can do that.
- droidist2 9y ago
  Everybody can do web development? Then why do so many people complain about how complicated it is?
  - deleted 9y ago
    [deleted]
  - hossbeast 9y ago
    anybody can do it poorly
  - eli_gottlieb 9y ago
    Because it was somehow designed to be as complicated, difficult, and utterly un-modular as possible. I actually have a more difficult time fully testing a commit's worth of Rails dev than I ever did with a commit's worth of embedded firmware.
- CuriouslyC 9y ago
  I highly doubt machine learning is going to become a commodity skill any time soon. The optimal model structure is tightly coupled to the specific data and the types of questions you want to answer. Additionally, understanding your model well enough to really trust it requires significant knowledge. I expect that common deep learning tasks are going to end up being performed by large pre-trained networks behind an API. Image tagging, sentiment analysis, etc. It will be possible to fine-tune these networks for specific tasks by retraining top layers using new data.
amelius 9y ago
I'm wondering if we will ever figure out how nature performs the equivalent of backpropagation, and if that will change how we work with artificial neural networks.
Kunix 9y ago
About libraries of models, it would be useful to have open source pre-trained models which can be augmented through github-like push requests of training data together with label sets. It would allow to maintain versioned versions of always improving models everyone can update with a `npm update`, `git pull` or equivalent.
primaryobjects 9y ago
Here are the results of my research into program synthesis using genetic algorithms. Using Artificial Intelligence to Write Self-Modifying/Improving Programs http://www.primaryobjects.com/2013/01/27/using-artificial-intelligence-to-write-self-modifying-improving-programs/ http://www.primaryobjects.com/2013/01/27/using-artificial-in... There is always a research paper, if you prefer the sciency format. BF-Programmer: A Counterintuitive Approach to Autonomously Building Simplistic Programs Using Genetic Algorithms http://www.primaryobjects.com/bf-programmer-2017.pdf http://www.primaryobjects.com/bf-programmer-2017.pdf
MR4D 9y ago
Compression. That one word disrupts his whole point of view. This idea that we need orders and orders of magnitude more data seems insane. What we need is to figure out how to be more effective with each layer of data, and be able to have compression between the tensor layers. The brain does a great job of throwing away information, and yet we can reconstruct pretty detailed memories. Somehow I find it hard to believe that all of that data is orders of magnitude above where we are today. Much more efficient, yes. And that's through compression.
- deleted 9y ago
  [deleted]
- MR4D 9y ago
  Crap. I just realized why this got voted down - I posted my comment on the wrong article. I guess that's what I get after walking away for 30 minutes before posting. Doh!
randcraw 9y ago
I enjoyed part 1 of Chollet's two articles today but am less fond of this one. It suggests that deep learning will expand from its present capabilities of recognizing patterns to one day master logical relations and employ a rich knowledge base of general facts, growing into a situated general problem solver that one day may equal or surpass human cognition. Maybe. But he then proposes that deep nets will rise to these heights of self-organization and purposefulness using one of the weakest and slowest forms of AI, namely evolutionary strategies? I don't think so. The many problems bedeviling the expansion of an AI's competence at one specific task into mastery of more general and more complex tasks are legend. Alas neither deep nets nor genetic algorithms have shown any way to address classic AGI roadblocks like: 1) the enormity of the possible solution space when synthesizing candidate solutions, and 2) the enormous number of training examples needed to learn the multitude of common sense facts common to all problem spaces, and 3) how to translate existing specific problem solutions into novel general ones. Wait, wait, there's more... These roadblocks are common to all forms of AI. The prospect of replacing heuristic strategies with zero knowledge techniques (like GA trial and error) or curated knowledge bases with only example-based learning is unrealistic and infeasible. Likewise, the notion that a sufficient number of deep nets can span all the info and problem spaces that will be needed for AGI is quite implausible. While quite impressive at the lowest levels of AI (pattern matching), deep learning has yet to address intermediate and high level AI implementation challenges like these. Until it does, there's little reason to believe DL will be equally good at implementing executive cognitive functions. Yes DeepMind solved Go using AlphaGo's deep nets (and monte carlo tree search). But 10 and 20 years before that IBM Watson solved Jeopardy and IBM Deep Blue solved chess. At the time, everyone was duly impressed. Yet today nobody is suggesting that the AI methods at the heart of those game solutions will one day pave the yellow brick road to AI Oz. In another 10 years, I predict it's just as likely that AlphaGo's deep nets will be a bust as a boom, at least when it comes to building deep AI like HAL 9000.

242 points by nicolrx 9y ago

TLDR is that models will become more abstract (current pattern recognition will blend with formal reasoning and abstraction), modular (think transfer learning, but taken to its extreme - every trained model's learned representations should be applicable to other tasks), and automated (ML experts will spend less time in the repetitive training/optimization cycle, instead focusing more on how models apply to their specific domain).

radarsat1 9y ago
I think it's true, but I hope this synergy between logic and pattern recognition actually happens, as I feel like this has been proposed for years but never really come to fruition. However, with recent work on differentiable communicating agents, differentiable memory etc., perhaps it now has a chance to get there.
- eli_gottlieb 9y ago
  The author says not everything should be differentiable. Intuitively, I agree, but the question is how to do a sufficiently fast search through a high-dimensional space when you don't have a gradient.
  - guicho271828 9y ago
    Guided by domain-independent lowerbound functions, obviously. There are lots of research in this area, see every year there are papers on heuristic search in AAAI. ML people are simply not reading them and citing them.
    - eli_gottlieb 9y ago
      Could you point some out to me? I haven't been deliberately avoiding them by any means, but I'd like to see some that allow fast search. And especially, if you've got it, fast stochastic search for Monte Carlo methods.
      - guicho271828 9y ago
        I bet you already know MCTS. But I see you have a problem with defining "fast". Define "fast". You can usually only say something is faster or slower. Absolute speed depends on the machine resource and the low-level optimization. If you take MCTS it is "enough fast" considering it has already beat Lee Sedol. There are multiple means to measure the speed. Blind search is "fast" re: node expansion but takes exponential time to find a solution and quite likely exhausts memory. Search with heavy heuristic functions has "slow" expansion but can solve the problem quickly, despite the cost of evaluating them, since it can prune a large part of the state space and expands fewer nodes. Only the latter is practically meaningful. Just read the classical AI textbooks like AIMA first, then [this one](https://www.amazon.co.uk/Heuristic-Search-Applications-Stefan-Edelkamp/dp/0123725127 https://www.amazon.co.uk/Heuristic-Search-Applications-Stefa...) if you want to know more. Maybe [this short paper](https://www.cs.cmu.edu/~maxim/files/searchcomesofage_aaai12_invitedpaper.pdf https://www.cs.cmu.edu/~maxim/files/searchcomesofage_aaai12_...) may give you the gist.
        Radim 9y ago
        On the topic of "likely exhausts memory": The best search methods allow reversible state updates. The reversibility makes things super-fast -- you no longer need to copy all data structures on path expansion. Instead, you modify only a single representation, incrementally. And when retracting a search step, you "undo" the same modifications again, arriving at the exact same state as when you started. This is of course non-trivial -- it is much easier to copy everything, then throw away the entire copy when it's not needed, rather than keeping a single state incrementally consistent. But the effects due to data locality (excellent caching), better memory management (no allocations, fragmentation) and less work (only touch and update parts of the state that matter) can be tremendous. I haven't seen any discussion of DeepMind's implementation details for AlphaGo, but since they come from a game development background (David Silver was the CTO and lead dev at Elixir Studios), where each cycle counts, I have no doubt they're well familiar with all these concepts. But then the TPU throws a wrench into it again...
  - CuriouslyC 9y ago
    If you don't have a gradient, one tactic is to make the most of the situation. Give your model the Bayesian treatment, and sample from the posterior using MCMC. This is slow, but you end up with posteriors on your parameter values, which is a huge win.
    - eli_gottlieb 9y ago
      Yeah, I've been a big fan of probabilistic programming for a while. The real problem is that getting Monte Carlo methods to converge and produce a large sample from the posterior takes orders of magnitude more time than running an optimizer to descend a gradient. Hey, you can even make it a probabilistic gradient: variational inference! But then you still have a hard time with discrete, nondifferentiable structure.

ipunchghosts 9y ago

Great work! Glad someone can finally explain this to the masses in an easy to understand way. Looking forward to the future!

kirillkh 9y ago

Seeing how gradient descent is such a pinnacle of deep learning, I can't help wondering: is this how our brain learns? If not, then what prevents us from implementing deep learning the same way?

ipunchghosts 9y ago
Our brains do not use gradient descent. Last i read they use some kind of local optimization. Can someone elaborate on this?
rsiqueira 9y ago
One of the most consistent theory about how our brain learns is described in HTM (Hierarchical Temporal Memory), a more biologically inspired neural network. See Jeff Hawkins' "On Intelligence". It is based on: * Input of continuous unlabeled time-based patterns. * Associative Hebbian Learning (when distinct inputs/patterns come together, they are neuron-wired together). Synapses can be modified via experience. See "Hebbian Theory". * The brain is a prediction machine: it is always trying to predict the future based on past learned patterns. Learning happens when reality does not match the originally prediction and we rewire the world model based on new input. See "Bayesian approaches to brain function". * Input signals are processed by many layers, each one creating more abstraction from the previous one, from sensory neurons to the highest cortex layers. * Each region of the hierarchy forms invariant memories (what a typical region of cortex learns is sequences of invariant representations). * There is lots of feedback (highest level neurons back to the lowest levels). In some structures (e.g. the thalamus, that is a kind of "hub of information") connections going backward (toward the input) exceed the connections going forward by almost a factor of ten. * Brain uses Sparse Distributed Memory (SDM). See SDM by Pentti Kanerva (NASA researcher). * Neuron models have many more variable/parameters (that can be used to transfer or process information) than usual nodes/links from artificial neural networks. E.g.: Long-term potentiation vs Long-term depression, neuronal Habituation vs Sensitization, inhibitory vs excitatory neurons, firing rates, synchronization, neuromodulation, homeostasis and more. * The backward propagation of errors in artificial neural networks only occurs during the learning phase. But the brain is always learning and updating weights and relationships between patterns, given new inputs. * During repetitive learning, representations of objects move down the cortical hierarchy (from short-term memory to long-term memory), forming invariant memories. * The brain needs to replay the memory (memory rehearsal) of a learned stimulus so it can be stored in long-term memory. * The job of any cortical region is to find out how inputs are related (pattern recognition), to memorize the sequence of correlations between them, and to use this memory to predict how the inputs will behave in the future. * Predictive coding: the brain is constantly generating and updating hypotheses that predict sensory input at varying levels of abstraction.

guicho271828 9y ago

Regarding logic and DL, there is NeSy workshop in London http://neural-symbolic.org/ http://neural-symbolic.org/

jdonaldson 9y ago

Glad to see Deep Learning "coming down to earth". This is the first high profile post I've seen that spells out exactly how DL models will become reconfigurable, purpose-built tools, and what a workflow might look like. We're still a long way aways from treating them like software components.

Cacti 9y ago
I mean, these are topics that have been discussed countless times over the years and in some cases decades. It's all well and good to say we need generalizable machines, and something other than backprop, and something closer to traditional programs, but we all know this. The issue is that no one knows what this would even mean, never mind how one would go about implementing it. In the few cases we do know how, the results are horrible compared to the methods we already use. We use the methods we do today because they work, not because we think they are the best, or because we don't understand the limitations of our models.
- jdonaldson 9y ago
  True, there's been discussions, but from what I've seen it's mostly flag planting or vague pop-eng fodder that project directors dish out to tech journalists. Having Keras make a statement on this carries far more weight, because fchollet is not selling a product, or pushing an agenda, or creating a walled garden of some sort. The only thing that's a bit off about Keras is that it's mostly the efforts of one guy. Sure, there's many other contributors, but they don't seem to be acknowledged. I've never seen anyone else speak for the project. I'd really like to see a neutral party emerge for deep learning practice and tooling, before the whole industry gets sucked into a single dominant ecosystem like AWS.
  - droidist2 9y ago
    Do you think with Google's adoption of Keras for TensorFlow it'll get more resources dedicated to it?

nextstar 9y ago

I'm excited for the easy to use tools that have to be coming out relatively soon. There are a lot right now, but the few I've used weren't super intuitive like I feel like they could be.

computerex 9y ago

> In DeepMind's AlphaGo, for example, most of the "intelligence" on display is designed and hard-coded by expert programmers (e.g. Monte-Carlo tree search); Not true. This paraphrases the original paper: https://www.tastehit.com/blog/google-deepmind-alphago-how-it-works/ https://www.tastehit.com/blog/google-deepmind-alphago-how-it... > They tested their best-performing policy network against Pachi, the strongest open-source Go program, and which relies on 100,000 simulations of MCTS at each turn. AlphaGo's policy network won 85% of the games against Pachi! I find this result truly remarkable. A fast feed-forward architecture (a convolutional network) was able to outperform a system that relies extensively on search. Also, this article reeked of AGI ideas. Deep learning isn't trying to solve AGI. Reasoning and abstraction and high level AGI concepts that I don't think apply to deep learning. I don't know the path to AGI but I don't think it'll be deep learning. I think it would have to be fundamentally different.

nojvek 9y ago
> Reasoning and abstraction and high level AGI concepts that I don't think apply to deep learning What do you mean? Each layer is selecting features and abstracting previous layers. A cat neuron abstracts all possible pixels that form cats. He's just saying that we need better ways of composing this `deep` abstractions.
- espadrine 9y ago
  > we need better ways of composing this `deep` abstractions Tensor2Tensor[0], from the Google Brain team, has some strong recent results in that direction. Related paper: "One Model To Learn Them All"[1]. [0]: https://github.com/tensorflow/tensor2tensor https://github.com/tensorflow/tensor2tensor [1]: https://arxiv.org/pdf/1706.05137.pdf https://arxiv.org/pdf/1706.05137.pdf
landon32 9y ago
>Also, this article reeked of AGI ideas. Deep learning isn't trying to solve AGI. Reasoning and abstraction and high level AGI concepts that I don't think apply to deep learning. I don't know the path to AGI but I don't think it'll be deep learning. I think it would have to be fundamentally different. I think that this is actually what the article is arguing. from the article: >Models closer to general-purpose computer programs, built on top of far richer primitives than our current differentiable layers—this is how we will get to reasoning and abstraction, the fundamental weakness of current models. This means not using current deep learning ideas, and instead finding ways to integrate other types of programs (conventional algorithms, other types of ML) alongside Deep Networks.

nzonbi 9y ago

Interesting article, in a difficult topic. Speculating about the future of deep learning. The author deserves recognition for writing about this. In my personal opinion, within the next 10 years, there will be systems exhibiting basic general intelligence behavior. I am currently doing early hobbist research on it, and I see it as feasible. These system will not be very powerful initially. They will exist and work in simpler simulated environments. Eventually we will be able to make these systems powerful enough to handle the real world. Although that will probably not be easy. I somewhat disagree with the author. I don't think that deep learning systems of the future are going to generate "programs", composed of programming primitives. In my speculative view, the key for general intelligence is not very far from our current knowledge. Deep learning, as currently we have, is a good enough basic tool. There are no magic improvements to the current deep learning algorithms, hidden around the corner. Rather what I think will enable general intelligence, is assembling systems of deep learning networks in the right setup. Some of the structure of these systems will be similar to traditional programs. But the models they generate will not resemble computer programs. They will be more like data graphs. I expect within 10 years there will be computer agents capable of communicating in simplified, but functional languages. Full human language capability will come after that. And within 20 years I expect artificial general intelligence to exist. At least in a basic form. That is my personal view. I am currently working on this.

ColanR 9y ago
what kind of research have you done on this? Most of what I hear about this is more pessimistic than anything else. I'm curious what you've learned that's made you optimistic.
- nzonbi 9y ago
  I am working on creating basic artificial general intelligence. My research is still early, and I don't want to give much details for now. For example, in my opinion it is wrong to feed a giant neural network with tons of text, and expect it to understand human language. That will never work. Language understanding requires a grasp of a world where the agent exist first. A general understanding of things. Then, on top of that, it can build basic communication. And then it can be hoped that this communication can become sophisticated enough, to reach human language level. Another example: Feeding a neural network with millions of product reviews, and expect it to be capable to understand and write product reviews is hopelessly laughable. Not even with petabytes of data. I started working on this, because I think that not enough focus is being put on AGI, or research is not creative enough. At least not that is being published. I am optimist with my work, and soon I will reveal more. But even if I don't reach my goals, I think that it is just a matter of persistence. At any moment, someone will solve the problem of AGI.

toisanji 9y ago

This is part 2 from the post yesterday: https://news.ycombinator.com/item?id=14790251 https://news.ycombinator.com/item?id=14790251 And the author posted a comment on hn: "fchollet: Hardly a "made-up" conclusion -- just a teaser for the next post, which deals with how we can achieve "extreme generalization" via abstraction and reasoning, and how we can concretely implement those in machine learning models." I like the ideas presented in the post, but its not concrete or new at all.Basically he writes "everything will get better". I do agree with the point that we need to move away from strictly differential learning though. All deep learning problems only work on systems that have derivates so we can do backpropagation. I dont think the brain learns with backpropagation at all. * AutoML, there are dozens of these type of systems already, he mentions one already in the post called HyperOpt. So we will continue to use this systems and they will get smarter? Many of these systems are basically grid search/brute force. Do you think the brain is doing brute force at all? We have to use these now because there are no universal correct hyperparameters for tuning these models. As long as we build AI models the way we do now, we will have to do this hyperparameter tuning. Yes, these will get better, again, nothing new here. * He talks about reusable modules. Everyone in the deep learning community has been talking about this a lot, its called transfer learning and people are using it now, and working on making it better all the time. We currently have "model zoos" which are databases of pretrained models that you can use. If you want to see a great scifi short piece on what neural network mini programs could look like written by the head of computer vision at tesla, check out this post: http://karpathy.github.io/2015/11/14/ai/ http://karpathy.github.io/2015/11/14/ai/

ipunchghosts 9y ago
Everyone makes the assumption that computers should get to be as smart as humans but in some ways, its the other way around. For example, the human brain is not a turing machine, it doesnt have memory (in the sense that its lossy). You need memory to have a turing machine so with a paper and pencil, a human is a turing machining but a very slow run. Compare the difference to read and write on paper than a computer has to access ram. I think there will be some kind of meta deep learing (still using deep learning but compose of algebras which are augmented compared to today's standards). We have already started this by using pretrained networks for tasks. There is no reason RNNs won't go this way (i imagine they already are but this isnt my research area specifically) after all, RNNS are a turing machine.

lopatin 9y ago

I'm also interested to see how the worlds of program synthesis (specifically type directed, proof-checking, dependently typed stuff) can combine with deep learning. If recent neural nets have such great results on large amounts of unstructured data, imagine what they can do with a type lattice.

gtani 9y ago
Recent baby steps in gradient checking: https://news.ycombinator.com/item?id=14739491 https://news.ycombinator.com/item?id=14739491

crypticlizard 9y ago

Are there popular modern libraries that do program synthesis? Although I've thought about this and read about the concept on hn, I've not heard it mentioned seriously or frequently or strenuously as a thing to do either just for fun or to get a job doing it. This could be a popular way to solve programming problems without needing programmers. I think this truly would kick off AI as a very personal experience for the masses because they would use AI basically like they do already do now with a search engine. People would use a virtual editor to design their software using off the shelf parts freely available. The level of program complexity could really skyrocket as people now have more control over what and how they run programs because they can easily design it themselves. Everyone could design their own personal Facebook or Twitter and probably a whole new series of websites too complex or for other reasons not invented yet. For instance, you want to program the personality of a toy, so you search around using the AI search engine for parts that might work. Or you want a relationship advice coach so you put it together using personalities you like, taking only the parts you want from each personality. Or another example would be just to make remixes of media you like. Because everything works without programming anyone can participate.

randcraw 9y ago
Check out Genetic Programming: https://en.wikipedia.org/wiki/Genetic_programming https://en.wikipedia.org/wiki/Genetic_programming AFAIK GP remains the primary means to automate the synthesis of software. Though it was introduced perhaps 30 years ago, it hasn't been an active area of research for the past 20, AFAIK.

scientist 9y ago

Self-driving cars are expected to take over the roads, however no programmer is able to write code that does this directly, without machine learning. However, programmers have built all kinds of software of great value, from operating systems to databases, desktop software and so on. Much of this software is open source and artificial systems can learn from it. Therefore, it could well be that, in the end, it would be easier to build artificial systems that learn to automatically develop such software than systems that autonomously drive cars, if the right methodologies are used. The author is right to say that neural program synthesis is the next big thing, and this also motivated me to switch my research to this field. If you have a PhD and are interested in working in neural program synthesis, please check out these available positions: http://rist.ro/job-a3 http://rist.ro/job-a3

amelius 9y ago

What about the future of jobs in the field of deep learning? EDIT: I'm thinking deep learning will become much like web development is today. Everybody can do it, and only a few experts will work at the technological frontier and develop tools and libraries for everybody else to use. Therefore, if one invests time in DL, then I suppose it better be a serious effort (at research level), rather than at the level of invoking a library, because soon everybody can do that.

droidist2 9y ago
Everybody can do web development? Then why do so many people complain about how complicated it is?
- deleted 9y ago
  [deleted]
- hossbeast 9y ago
  anybody can do it poorly
- eli_gottlieb 9y ago
  Because it was somehow designed to be as complicated, difficult, and utterly un-modular as possible. I actually have a more difficult time fully testing a commit's worth of Rails dev than I ever did with a commit's worth of embedded firmware.
CuriouslyC 9y ago
I highly doubt machine learning is going to become a commodity skill any time soon. The optimal model structure is tightly coupled to the specific data and the types of questions you want to answer. Additionally, understanding your model well enough to really trust it requires significant knowledge. I expect that common deep learning tasks are going to end up being performed by large pre-trained networks behind an API. Image tagging, sentiment analysis, etc. It will be possible to fine-tune these networks for specific tasks by retraining top layers using new data.

amelius 9y ago

I'm wondering if we will ever figure out how nature performs the equivalent of backpropagation, and if that will change how we work with artificial neural networks.

Kunix 9y ago

About libraries of models, it would be useful to have open source pre-trained models which can be augmented through github-like push requests of training data together with label sets. It would allow to maintain versioned versions of always improving models everyone can update with a `npm update`, `git pull` or equivalent.

primaryobjects 9y ago

Here are the results of my research into program synthesis using genetic algorithms. Using Artificial Intelligence to Write Self-Modifying/Improving Programs http://www.primaryobjects.com/2013/01/27/using-artificial-intelligence-to-write-self-modifying-improving-programs/ http://www.primaryobjects.com/2013/01/27/using-artificial-in... There is always a research paper, if you prefer the sciency format. BF-Programmer: A Counterintuitive Approach to Autonomously Building Simplistic Programs Using Genetic Algorithms http://www.primaryobjects.com/bf-programmer-2017.pdf http://www.primaryobjects.com/bf-programmer-2017.pdf

MR4D 9y ago

Compression. That one word disrupts his whole point of view. This idea that we need orders and orders of magnitude more data seems insane. What we need is to figure out how to be more effective with each layer of data, and be able to have compression between the tensor layers. The brain does a great job of throwing away information, and yet we can reconstruct pretty detailed memories. Somehow I find it hard to believe that all of that data is orders of magnitude above where we are today. Much more efficient, yes. And that's through compression.

deleted 9y ago
[deleted]
MR4D 9y ago
Crap. I just realized why this got voted down - I posted my comment on the wrong article. I guess that's what I get after walking away for 30 minutes before posting. Doh!

randcraw 9y ago

I enjoyed part 1 of Chollet's two articles today but am less fond of this one. It suggests that deep learning will expand from its present capabilities of recognizing patterns to one day master logical relations and employ a rich knowledge base of general facts, growing into a situated general problem solver that one day may equal or surpass human cognition. Maybe. But he then proposes that deep nets will rise to these heights of self-organization and purposefulness using one of the weakest and slowest forms of AI, namely evolutionary strategies? I don't think so. The many problems bedeviling the expansion of an AI's competence at one specific task into mastery of more general and more complex tasks are legend. Alas neither deep nets nor genetic algorithms have shown any way to address classic AGI roadblocks like: 1) the enormity of the possible solution space when synthesizing candidate solutions, and 2) the enormous number of training examples needed to learn the multitude of common sense facts common to all problem spaces, and 3) how to translate existing specific problem solutions into novel general ones. Wait, wait, there's more... These roadblocks are common to all forms of AI. The prospect of replacing heuristic strategies with zero knowledge techniques (like GA trial and error) or curated knowledge bases with only example-based learning is unrealistic and infeasible. Likewise, the notion that a sufficient number of deep nets can span all the info and problem spaces that will be needed for AGI is quite implausible. While quite impressive at the lowest levels of AI (pattern matching), deep learning has yet to address intermediate and high level AI implementation challenges like these. Until it does, there's little reason to believe DL will be equally good at implementing executive cognitive functions. Yes DeepMind solved Go using AlphaGo's deep nets (and monte carlo tree search). But 10 and 20 years before that IBM Watson solved Jeopardy and IBM Deep Blue solved chess. At the time, everyone was duly impressed. Yet today nobody is suggesting that the AI methods at the heart of those game solutions will one day pave the yellow brick road to AI Oz. In another 10 years, I predict it's just as likely that AlphaGo's deep nets will be a bust as a boom, at least when it comes to building deep AI like HAL 9000.