3 ms·

That is not what I was referring to. Your relationship (neuron) needs to have a gradient, i.e. it needs to be differentiable, in order for backprop to work. Th

That is not what I was referring to. Your relationship (neuron) needs to have a gradient, i.e. it needs to be differentiable, in order for backprop to work. There are many situations that are not differentiable. Images work well because adjusting color values during back prop can easily be done with a gradient. For example, during backprop it is feasible to increase the level of blue by 10% by increasing its RGB value. Likewise for 3%, 15%, -7%, etc. For other types of data, a gradient may not exist.

martindbp 7y ago
> For other types of data, a gradient may not exist. You don't calculate a gradient on the data, you calculate a gradient on the parameters of the model. But I'm going to assume you mean that some types of data are not well represented by floating point numbers, which is true, but there are workarounds, e.g. vector embeddings for words or simply one-hot encodings for categorical values. What is true is that if you need to make a hard decision in one part of the network, for example deciding that the image is a cat, in order to further process that decision. This would not be differentiable. But we can represent objects in a non-discrete way, with embeddings, hidden state vectors etc. My point is, people are getting around this problem. I also want to add that I agree that we are way far off from building apps automatically, but I think you're giving way too little importance to the advances that's been made. Just take a look at the latest GANs, or the advances in machine translation using the Transformer and attention architectures, or Neural Turing Machines.
- codesushi42 7y ago
  You don't calculate a gradient on the data, you calculate a gradient on the parameters of the model. Whether a gradient exists will depend on the type of data and what you're trying to predict. But we can represent objects in a non-discrete way, with embeddings, hidden state vectors etc. My point is, people are getting around this problem. Great progress has been made using that, e.g. advancements in NLU. But it is not a complete workaround. Embeddings don't work if your relationship is non differentiable. And you will lose the gradient for very long contexts (deep networks), even using new techniques like attention and transformers.
  - drdeca 7y ago
    Can you give an example of a relationship which is non-differentiable? It seems to me that there is no difficulty in using it to model fixed functions {0,1}^n -> {0,1}^m . I don’t know for sure that all such functions can be learned reasonably efficiently, but I do know that there are feedforward standard neural nets which, with some weights, can approximate any such function arbitrarily closely. But, again, I’m not saying that these functions can be learned effectively by backprop, idk if they can be. Are these functions examples of relations you would say aren’t differentiable. I’m confident that SOME of these functions can be learned by backprop, but I don’t know what would make these ones “more differentiable” than others, So, I don’t see how differentiability could be the obstacle here? But I may be misunderstanding.
    - codesushi42 7y ago
      The problem is that the true, hidden function being modeled is non differentiable. We can try to approximate that hidden function with one that is differentiable, yes. But no good approximation is guaranteed to exist depending on the nature of the problem. An example from op: What is true is that if you need to make a hard decision in one part of the network, for example deciding that the image is a cat, in order to further process that decision. This would not be differentiable. CNNs can work for image classification because a smooth gradient will exist for incrementally "stepping" color values during backprop. But asking the question "is this an image of a cat" is much different than "what do I do if this is an image of a cat". Also very deep neural networks (i.e. many, many time steps) essentially lose their gradient, or context. Even with attention and transformers. Which is partly why we haven't seen AIs that can write lengthy programs, books etc that are coherent and grammatically correct. And we probably never will by only relying on current "curve fitting" techniques.
      - p1esk 7y ago
        "what do I do if this is an image of a cat" You can train another NN to predict the best course of action for each decision of the first net. Or even train a single net to choose actions based on the initial input. Not sure what's the problem here. we haven't seen AIs that can write lengthy programs, books etc that are coherent and grammatically correct. And we probably never will Have you completely missed the recent NLP breakthroughs (BERT, GPT-2, XLNet, etc)? OpenAI even refused to publish their model because it could generate long coherent and grammatically correct text.
tsss 7y ago
That may not be what you meant but it is what you said. The "function you are trying to model" does not have to be differentiable as we have already established. Only the model itself (which is the neural network) needs to be differentiable.
- deleted 7y ago
  [deleted]
- codesushi42 7y ago
  The model needs to be differentiable. The function needs to be continuous. Discontinuous functions are not differentiable. You can approximate discontinuous functions with continuous ones (e.g. with logistic functions). In the end, we are still talking about curve fitting and optimization, not artificial general intelligence.

martindbp 7y ago

> For other types of data, a gradient may not exist. You don't calculate a gradient on the data, you calculate a gradient on the parameters of the model. But I'm going to assume you mean that some types of data are not well represented by floating point numbers, which is true, but there are workarounds, e.g. vector embeddings for words or simply one-hot encodings for categorical values. What is true is that if you need to make a hard decision in one part of the network, for example deciding that the image is a cat, in order to further process that decision. This would not be differentiable. But we can represent objects in a non-discrete way, with embeddings, hidden state vectors etc. My point is, people are getting around this problem. I also want to add that I agree that we are way far off from building apps automatically, but I think you're giving way too little importance to the advances that's been made. Just take a look at the latest GANs, or the advances in machine translation using the Transformer and attention architectures, or Neural Turing Machines.

codesushi42 7y ago
You don't calculate a gradient on the data, you calculate a gradient on the parameters of the model. Whether a gradient exists will depend on the type of data and what you're trying to predict. But we can represent objects in a non-discrete way, with embeddings, hidden state vectors etc. My point is, people are getting around this problem. Great progress has been made using that, e.g. advancements in NLU. But it is not a complete workaround. Embeddings don't work if your relationship is non differentiable. And you will lose the gradient for very long contexts (deep networks), even using new techniques like attention and transformers.
- drdeca 7y ago
  Can you give an example of a relationship which is non-differentiable? It seems to me that there is no difficulty in using it to model fixed functions {0,1}^n -> {0,1}^m . I don’t know for sure that all such functions can be learned reasonably efficiently, but I do know that there are feedforward standard neural nets which, with some weights, can approximate any such function arbitrarily closely. But, again, I’m not saying that these functions can be learned effectively by backprop, idk if they can be. Are these functions examples of relations you would say aren’t differentiable. I’m confident that SOME of these functions can be learned by backprop, but I don’t know what would make these ones “more differentiable” than others, So, I don’t see how differentiability could be the obstacle here? But I may be misunderstanding.
  - codesushi42 7y ago
    The problem is that the true, hidden function being modeled is non differentiable. We can try to approximate that hidden function with one that is differentiable, yes. But no good approximation is guaranteed to exist depending on the nature of the problem. An example from op: What is true is that if you need to make a hard decision in one part of the network, for example deciding that the image is a cat, in order to further process that decision. This would not be differentiable. CNNs can work for image classification because a smooth gradient will exist for incrementally "stepping" color values during backprop. But asking the question "is this an image of a cat" is much different than "what do I do if this is an image of a cat". Also very deep neural networks (i.e. many, many time steps) essentially lose their gradient, or context. Even with attention and transformers. Which is partly why we haven't seen AIs that can write lengthy programs, books etc that are coherent and grammatically correct. And we probably never will by only relying on current "curve fitting" techniques.
    - p1esk 7y ago
      "what do I do if this is an image of a cat" You can train another NN to predict the best course of action for each decision of the first net. Or even train a single net to choose actions based on the initial input. Not sure what's the problem here. we haven't seen AIs that can write lengthy programs, books etc that are coherent and grammatically correct. And we probably never will Have you completely missed the recent NLP breakthroughs (BERT, GPT-2, XLNet, etc)? OpenAI even refused to publish their model because it could generate long coherent and grammatically correct text.

tsss 7y ago

That may not be what you meant but it is what you said. The "function you are trying to model" does not have to be differentiable as we have already established. Only the model itself (which is the neural network) needs to be differentiable.

deleted 7y ago
[deleted]
codesushi42 7y ago
The model needs to be differentiable. The function needs to be continuous. Discontinuous functions are not differentiable. You can approximate discontinuous functions with continuous ones (e.g. with logistic functions). In the end, we are still talking about curve fitting and optimization, not artificial general intelligence.