3 ms·

Show HN: Nevermind XOR – Deep Learning Has an Issue with Sin

34 points by lostmsu 4y ago

rryan 4y ago
You're using a pointwise loss, which means you are treating each timepoint as conditionally independent. That's a deeply flawed assumption, probabilistically. Try an autoregressive model of the joint probability distribution of the sine wave timepoints -- like WaveNet [1]. It will nail your sine wave -- just as it nails mixtures of sine waves (speech and music). :) [1] https://arxiv.org/abs/1609.03499 https://arxiv.org/abs/1609.03499
- lostmsu 4y ago
  I am sure a LSTM or even a small 2 layer network can learn to predict the next numbers on the clock from the numbers on the clock exactly one second ago, but that is not the point. Given such a network it will take you forever to make a prediction for second number N=10^315, yet it takes rather short time to divide with modulo.
  - azalemeth 4y ago
    I guess a broader point is that while the universal approximation theorem is a get-out-of-jail-free card for Neural Nets, nobody said "how quickly". A bit like trying to approximate sin(x) with a Taylor series for large x is suboptimal – Chebyshev polynomials are better (and Padé approximants probably better still), but both "work". Unless you have data points on the boundary of the domain, or some analytic knowledge that the solution space is bounded somehow, expect weird shit...
    - lostmsu 4y ago
      The remark about the Chebyshev polynomials just sent me down another rabbit hole.
shreyshnaccount 4y ago
>Their x + sin(x) * 2 activation only really worked when the number of full cycles in the dataset was less than the number of paramters of the network, which completely misses the point. interesting, how did you come to this conclusion?
- lostmsu 4y ago
  It is just a guess, that tries to explain the observed behavior.
shreyshnaccount 4y ago
> def forward(self, x): return torch.sin(self.freq * x) I don't think this qualifies as deep learning anyway
- lostmsu 4y ago
  Well, the point is that neither large models could be trained, nor could a trivial model which is known to have a set of parameter values, that satisfy the given task.
  - shreyshnaccount 4y ago
    yeah, and the way the problem is being treated is flawed, but that's another thing
natch 4y ago
“stohastic” should that be stochastic?

34 points by lostmsu 4y ago

You're using a pointwise loss, which means you are treating each timepoint as conditionally independent. That's a deeply flawed assumption, probabilistically. Try an autoregressive model of the joint probability distribution of the sine wave timepoints -- like WaveNet [1]. It will nail your sine wave -- just as it nails mixtures of sine waves (speech and music). :) [1] https://arxiv.org/abs/1609.03499 https://arxiv.org/abs/1609.03499

lostmsu 4y ago
I am sure a LSTM or even a small 2 layer network can learn to predict the next numbers on the clock from the numbers on the clock exactly one second ago, but that is not the point. Given such a network it will take you forever to make a prediction for second number N=10^315, yet it takes rather short time to divide with modulo.
- azalemeth 4y ago
  I guess a broader point is that while the universal approximation theorem is a get-out-of-jail-free card for Neural Nets, nobody said "how quickly". A bit like trying to approximate sin(x) with a Taylor series for large x is suboptimal – Chebyshev polynomials are better (and Padé approximants probably better still), but both "work". Unless you have data points on the boundary of the domain, or some analytic knowledge that the solution space is bounded somehow, expect weird shit...
  - lostmsu 4y ago
    The remark about the Chebyshev polynomials just sent me down another rabbit hole.

shreyshnaccount 4y ago

>Their x + sin(x) * 2 activation only really worked when the number of full cycles in the dataset was less than the number of paramters of the network, which completely misses the point. interesting, how did you come to this conclusion?

lostmsu 4y ago
It is just a guess, that tries to explain the observed behavior.

shreyshnaccount 4y ago

> def forward(self, x): return torch.sin(self.freq * x) I don't think this qualifies as deep learning anyway

lostmsu 4y ago
Well, the point is that neither large models could be trained, nor could a trivial model which is known to have a set of parameter values, that satisfy the given task.
- shreyshnaccount 4y ago
  yeah, and the way the problem is being treated is flawed, but that's another thing

natch 4y ago

“stohastic” should that be stochastic?