7 ms·

Is there anything deep here? There's a parametrized representation of a function and it's being interpreted as a neural network, why is this surprising? It's li

by pontus 5y ago

Is there anything deep here? There's a parametrized representation of a function and it's being interpreted as a neural network, why is this surprising? It's like saying that Newton's second law is a neural network: F=ma can be written log(F) = log(m) + log(a). Aha, this is a neural network! The inputs are m and a, the first layer is sparsely connected with log activation functions, the second layer is fully connected with an exponential activation function: F = exp(c1 * o1 + c2 * o2) o1 = log(c3 * a + c4 * m) o2 = log(c5 * a + c6 * m) If you feed it enough data you'll find c1=c2=c3=c6=1 and c4=c5=0. But saying that Newton's second law is a Neural Network, while correct, seems a bit deceptive in that it's not a deep idea at all.

volta83 5y ago
> But saying that Newton's second law is a Neural Network, while correct, seems a bit deceptive in that it's not a deep idea at all. I guess the point is that neither is the idea of a Neural Network.
- Nasrudith 5y ago
  A neural network can also be set to learn unconditionally return a fixed value with no learning feedback. I don't think lower bounds on capabilities are very informative. So could many arbitrarily complex arrangements that do massive amounts of work only to discard it and return a constant. An upper bound of what an approach is capable of is more useful. Say no matter how vast a look up table is it will never return a different value for the same input regardless of prior sequence.
candiodari 5y ago
Well the point is that it does not make sense to DFT data before feeding it into a multilayered neural network (or calculating the force generated by mass and acceleration as you point out). Those formulas make no sense: the network can just learn them on the fly. In fact you'll find that this does not just work for the Fourier transform, but for any FIR filter (and some other classes), and therefore neural networks can deal with signals and construct low-pass filters, high-pass filters, bandgap filters, ... as required for the task at hand without the network (or it's designer) having any idea at all what is happening. I mean there's some basic assumptions these reasonings make (main one is that you need to feed many discretized values from a time window). Of course, a problem remains: local optima. Just because a neural network can construct a filterbank or do a DFT, doesn't mean that it will actually do it when the situation warrants it. If there's a local optimum without filters ... well, you may get unlucky. It there's many local optima without filters ... sucks to be you.
- ska 5y ago
  > Those formulas make no sense: the network can just learn them on the fly. > Just because [...] can construct a filterbank or do a DFT, doesn't mean that it will actually do it when the situation warrants it. These statements seem in conflict, no?
- nonameiguess 5y ago
  I don't think I can agree with that. You do a Fourier transform when the data you're working with doesn't admit easily or at all certain operations in the time domain but it does in frequency domain. If you already know that to be the case, preprocessing with a FFT is a better idea than hoping a neural network with enough layers uses a few of those to much less efficiently perform a DFT. Always take advantage of pre-existing knowledge of structure in your data. With the FFT especially, depending on how your data is being ingested, you might be able to use specialized DSPs that implement the FFT directly in hardware. These are cheap and easy to find since they're used in frequency-division multiplexing.
  - dkarras 5y ago
    Practically, yes, absolutely! Theoretically, we want the "black box" learning system to do such feature extraction itself. At least, that is the goal.
    - whimsicalism 5y ago
      > does not make sense to DFT data before feeding it into a multilayered neural network is false. "It would be nice if we didn't have to DFT data before feeding it into a multilayered neural network" is true, but a completely different statement.
- monocasa 5y ago
  Which is weird, because biological neural nets seem to have some evolutionary pressure to do hardware fourier transforms before neurons even get involved. You can see this most clearly in the auditory system where the incoming signal is transformed into the frequency domain by the cochlea before the signal is received by the epithelial cells. Neurons absolutely love working in the frequency domain, but they seem to prefer to not be the ones to do the binning in the first place.
montebicyclelo 5y ago
Op here, IMO the "deepest" bit is [1] - the network learns the DFT in order to reconstruct the signal, and is not explicitly trained on the DFT values from the FFT. Admittedly, I should have mentioned that any linear transform can be considered to be a single layer neural network (if you want to see the world through a neural network lens), and will add this to the post at some point. In fact, I have a series of posts planned, which will reveal that well known algorithms/models are actually neural networks... [1] https://sidsite.com/posts/fourier-nets/#learning-the-fourier-transform-via-reconstruction https://sidsite.com/posts/fourier-nets/#learning-the-fourier...
- mochomocha 5y ago
  You might be interested in this line of work: https://eng.uber.com/neural-networks-jpeg/ https://eng.uber.com/neural-networks-jpeg/ Training straight from DCT coefficients to avoid spending time learning a similar representation in the bottom layers of the net. I've personally toyed with something similar on GANs to gauge the computational benefits of not doing convolutions in the bottom layers of a net but learning directly in a FFT-like compressed space instead.
  - touisteur 5y ago
    Isn't there something like Fouriernets and spdnets already, and stuff on Riemannian manifolds and my head is spinning? Followed Daniel Brook's work for some time, e.g. https://hal.archives-ouvertes.fr/hal-02290838 https://hal.archives-ouvertes.fr/hal-02290838
- aesthesia 5y ago
  As far as I can tell, you're still using a fixed inverse DFT as the reconstruction layer, so it's not just rediscovering the DFT on its own. Instead of learning a linear transformation from input-output pairs, it's learning the inverse of a linear transformation when that transformation is given as an oracle. It's not terribly surprising that this works, although there are probably some interesting issues of numerical conditioning in the general case.
- suvakov 5y ago
Imnimo 5y ago
I think this is the sort of thing that is very obvious if you are already comfortable with neural networks and the FFT. But if you're only comfortable with neural networks, and the FFT feels like arcane magic, this exercise might be very instructive.
korijn 5y ago
Perhaps it is the compsci glasses talking, but this is just one very specific instance where someone figured out a way to map the DFT problem to a neural net. I agree that it is unfortunate that it is being presented as some kind of big discovery, and that the fundamental lesson is either still undiscovered to the author or just unclearly communicated, but there is still good intention in there (sharing something you've learned and eliciting feedback).
haecceity 5y ago
Can be represented as a neural network is not the same as is a neural network??
- namelessone 5y ago
  I think the issue here is that almost anything can be represented as a neural network. You could create a neural network that does a xor operation, for example. There is nothing new about this.
windsignaling 5y ago
It's a stretch to call it a neural network. It was already well-known that the Fourier transform can be seen as a matrix multiply which minimizes some least squares problem. This can be found somewhere in S.M. Kay's Fundamentals of Statistical Signal Processing: Estimation Theory (Vol 1), Detection Theory (Vol 2).
xyzzy21 5y ago
The "Deep" part ia to realize what this really means in terms of limitations of ML/NN!

by pontus 5y ago

volta83 5y ago

> But saying that Newton's second law is a Neural Network, while correct, seems a bit deceptive in that it's not a deep idea at all. I guess the point is that neither is the idea of a Neural Network.

Nasrudith 5y ago
A neural network can also be set to learn unconditionally return a fixed value with no learning feedback. I don't think lower bounds on capabilities are very informative. So could many arbitrarily complex arrangements that do massive amounts of work only to discard it and return a constant. An upper bound of what an approach is capable of is more useful. Say no matter how vast a look up table is it will never return a different value for the same input regardless of prior sequence.

candiodari 5y ago

Well the point is that it does not make sense to DFT data before feeding it into a multilayered neural network (or calculating the force generated by mass and acceleration as you point out). Those formulas make no sense: the network can just learn them on the fly. In fact you'll find that this does not just work for the Fourier transform, but for any FIR filter (and some other classes), and therefore neural networks can deal with signals and construct low-pass filters, high-pass filters, bandgap filters, ... as required for the task at hand without the network (or it's designer) having any idea at all what is happening. I mean there's some basic assumptions these reasonings make (main one is that you need to feed many discretized values from a time window). Of course, a problem remains: local optima. Just because a neural network can construct a filterbank or do a DFT, doesn't mean that it will actually do it when the situation warrants it. If there's a local optimum without filters ... well, you may get unlucky. It there's many local optima without filters ... sucks to be you.

ska 5y ago
> Those formulas make no sense: the network can just learn them on the fly. > Just because [...] can construct a filterbank or do a DFT, doesn't mean that it will actually do it when the situation warrants it. These statements seem in conflict, no?
nonameiguess 5y ago
I don't think I can agree with that. You do a Fourier transform when the data you're working with doesn't admit easily or at all certain operations in the time domain but it does in frequency domain. If you already know that to be the case, preprocessing with a FFT is a better idea than hoping a neural network with enough layers uses a few of those to much less efficiently perform a DFT. Always take advantage of pre-existing knowledge of structure in your data. With the FFT especially, depending on how your data is being ingested, you might be able to use specialized DSPs that implement the FFT directly in hardware. These are cheap and easy to find since they're used in frequency-division multiplexing.
- dkarras 5y ago
  Practically, yes, absolutely! Theoretically, we want the "black box" learning system to do such feature extraction itself. At least, that is the goal.
  - whimsicalism 5y ago
    > does not make sense to DFT data before feeding it into a multilayered neural network is false. "It would be nice if we didn't have to DFT data before feeding it into a multilayered neural network" is true, but a completely different statement.
monocasa 5y ago
Which is weird, because biological neural nets seem to have some evolutionary pressure to do hardware fourier transforms before neurons even get involved. You can see this most clearly in the auditory system where the incoming signal is transformed into the frequency domain by the cochlea before the signal is received by the epithelial cells. Neurons absolutely love working in the frequency domain, but they seem to prefer to not be the ones to do the binning in the first place.

montebicyclelo 5y ago

Op here, IMO the "deepest" bit is [1] - the network learns the DFT in order to reconstruct the signal, and is not explicitly trained on the DFT values from the FFT. Admittedly, I should have mentioned that any linear transform can be considered to be a single layer neural network (if you want to see the world through a neural network lens), and will add this to the post at some point. In fact, I have a series of posts planned, which will reveal that well known algorithms/models are actually neural networks... [1] https://sidsite.com/posts/fourier-nets/#learning-the-fourier-transform-via-reconstruction https://sidsite.com/posts/fourier-nets/#learning-the-fourier...

mochomocha 5y ago
You might be interested in this line of work: https://eng.uber.com/neural-networks-jpeg/ https://eng.uber.com/neural-networks-jpeg/ Training straight from DCT coefficients to avoid spending time learning a similar representation in the bottom layers of the net. I've personally toyed with something similar on GANs to gauge the computational benefits of not doing convolutions in the bottom layers of a net but learning directly in a FFT-like compressed space instead.
- touisteur 5y ago
  Isn't there something like Fouriernets and spdnets already, and stuff on Riemannian manifolds and my head is spinning? Followed Daniel Brook's work for some time, e.g. https://hal.archives-ouvertes.fr/hal-02290838 https://hal.archives-ouvertes.fr/hal-02290838
aesthesia 5y ago
As far as I can tell, you're still using a fixed inverse DFT as the reconstruction layer, so it's not just rediscovering the DFT on its own. Instead of learning a linear transformation from input-output pairs, it's learning the inverse of a linear transformation when that transformation is given as an oracle. It's not terribly surprising that this works, although there are probably some interesting issues of numerical conditioning in the general case.
suvakov 5y ago

Imnimo 5y ago

I think this is the sort of thing that is very obvious if you are already comfortable with neural networks and the FFT. But if you're only comfortable with neural networks, and the FFT feels like arcane magic, this exercise might be very instructive.

korijn 5y ago

Perhaps it is the compsci glasses talking, but this is just one very specific instance where someone figured out a way to map the DFT problem to a neural net. I agree that it is unfortunate that it is being presented as some kind of big discovery, and that the fundamental lesson is either still undiscovered to the author or just unclearly communicated, but there is still good intention in there (sharing something you've learned and eliciting feedback).

haecceity 5y ago

Can be represented as a neural network is not the same as is a neural network??

namelessone 5y ago
I think the issue here is that almost anything can be represented as a neural network. You could create a neural network that does a xor operation, for example. There is nothing new about this.

windsignaling 5y ago

It's a stretch to call it a neural network. It was already well-known that the Fourier transform can be seen as a matrix multiply which minimizes some least squares problem. This can be found somewhere in S.M. Kay's Fundamentals of Statistical Signal Processing: Estimation Theory (Vol 1), Detection Theory (Vol 2).

xyzzy21 5y ago

The "Deep" part ia to realize what this really means in terms of limitations of ML/NN!