3 ms·

It’s hard to walk if you can’t feel your legs. The problem is improperly evaluating the neural network as a function of time, instead of evaluating the network

It’s hard to walk if you can’t feel your legs. The problem is improperly evaluating the neural network as a function of time, instead of evaluating the network as a function of previous state. When we humans approximate functions (let’s say you’re drawing it on a piece of paper, or waving your arm around) we do not simply look at a clock and feed forward that information directly into our motor neurons. Rather, we have sensory neurons that feed in the current state of the function we are approximating as an input, then approximating the next output of a periodic function becomes trivial. It’s very easy to train a neural network to look at a piece of a sine wave and predict what the next value should be - the fact that the function is periodic actually helps you.

omrjml 5y ago
Periodic function like sin(x) are not a dynamical system so its previous state does not determine the current state. So it should be approximated in that way.
- doubleunplussed 5y ago
  Periodic functions like sin(x) are the solutions to differential equations like dy/dx = -y that describe for example, oscillations of springs, to name but one of an extremely large number of dynamical systems that behave this way.
  - omrjml 5y ago
    Of course sine can appear in the solution for dynamical systems but the function itself is not dynamical. When evaluating sin(x) you do not need to know about the previous state.
    - zmgsabst 5y ago
      That is true, the problem is with the conclusion you drew from that fact: > So it should not be approximated in that way. We can’t conclude a dynamic approximation is a bad approach based purely on the fact the underlying function isn’t dynamic. The function might nevertheless be easily approximated via dynamics — as in the case of predicting sine from seeing the recent history.
  - freemint 5y ago
    sin(x) only arises as solution in second order systems. So d^2x/dt^2 = -x
    - doubleunplussed 5y ago
      Sorry, you're absolutely correct. Brain fart. The equation I wrote actually is for an exponential, whoops.
- omrjml 5y ago
  Edit: should not be approximated
- em500 5y ago
  It's completely valid and equivalent to formulate trigonometric functions as a dynamic system (x,y-coordinates as you go around the circle using a rotation matrix).
- vidarh 5y ago
  It sounds like you're trying to say something different than the comment you replied to. sin(x) for a range of x is trivial to draw by looking at earlier parts of the curve to determine what to draw next. Back in the slow 1980's home computer days, on machines without floating point and multiply/divide, we quickly got used to approximate sine waves numerically by simply typing out sequences of integers with a rough idea of the wave it'd produce. If you gave me a subset of such a sequence and asked me to complete it, I wouldn't need much of it to assume you were wanting to approximate sin.
dahart 5y ago
> It’s very easy to train a neural network to look at a piece of a sine wave and predict what the next value should be - the fact that the function is periodic actually helps you. I’m not sure what you mean. Are you talking about training a network to predict cos(x)? In this case nothing at all changes by training on the derivative. Or do you mean train the net to take as input sin(t) and produce sin(t+0.01)? The problem with this is that any given value of sin(t) has 2 answers for sin(t+0.01), therefore your optimizer is going to spit out 0 as the answer. Plus, this is a different problem entirely, and you lose the ability to infer sin(t) based on t. It doesn’t answer the same question. Your suggestion is further going to be seriously confounded if the periodic function is more complex, say, the sum of several sin waves. There can be an arbitrary number of values y that correctly match f(x). You should try it before making assumptions that it’s easy, see what it takes to train a net to predict sin(x), without embedding knowledge of the fact that sin is periodic. > The problem is improperly evaluating the neural network as a function of time, instead of evaluating the network as a function of state. This also sounds like an assumption to me that somehow the entire world of research has failed to consider the most obvious of ideas. The point of both optimizers and neural networks is that they can be black boxes, right? It doesn’t matter at all whether the input is time based or position based or a function of money. The network, in theory, can learn any function, time or otherwise, and there’s nothing special about time. But, neural networks function better with domain knowledge. When you know the function domain is time and that the output is periodic, you can do things to make a network easier to train, like using a periodic activation function. As a side note, RNNs explicitly model an NN based on previous state. Also all layered NNs can be viewed as a series of smaller nets that feed state to the next net. In some sense, NNs always evaluate as a function of state.

omrjml 5y ago

Periodic function like sin(x) are not a dynamical system so its previous state does not determine the current state. So it should be approximated in that way.

doubleunplussed 5y ago
Periodic functions like sin(x) are the solutions to differential equations like dy/dx = -y that describe for example, oscillations of springs, to name but one of an extremely large number of dynamical systems that behave this way.
- omrjml 5y ago
  Of course sine can appear in the solution for dynamical systems but the function itself is not dynamical. When evaluating sin(x) you do not need to know about the previous state.
  - zmgsabst 5y ago
    That is true, the problem is with the conclusion you drew from that fact: > So it should not be approximated in that way. We can’t conclude a dynamic approximation is a bad approach based purely on the fact the underlying function isn’t dynamic. The function might nevertheless be easily approximated via dynamics — as in the case of predicting sine from seeing the recent history.
- freemint 5y ago
  sin(x) only arises as solution in second order systems. So d^2x/dt^2 = -x
  - doubleunplussed 5y ago
    Sorry, you're absolutely correct. Brain fart. The equation I wrote actually is for an exponential, whoops.
omrjml 5y ago
Edit: should not be approximated
em500 5y ago
It's completely valid and equivalent to formulate trigonometric functions as a dynamic system (x,y-coordinates as you go around the circle using a rotation matrix).
vidarh 5y ago
It sounds like you're trying to say something different than the comment you replied to. sin(x) for a range of x is trivial to draw by looking at earlier parts of the curve to determine what to draw next. Back in the slow 1980's home computer days, on machines without floating point and multiply/divide, we quickly got used to approximate sine waves numerically by simply typing out sequences of integers with a rough idea of the wave it'd produce. If you gave me a subset of such a sequence and asked me to complete it, I wouldn't need much of it to assume you were wanting to approximate sin.

dahart 5y ago

> It’s very easy to train a neural network to look at a piece of a sine wave and predict what the next value should be - the fact that the function is periodic actually helps you. I’m not sure what you mean. Are you talking about training a network to predict cos(x)? In this case nothing at all changes by training on the derivative. Or do you mean train the net to take as input sin(t) and produce sin(t+0.01)? The problem with this is that any given value of sin(t) has 2 answers for sin(t+0.01), therefore your optimizer is going to spit out 0 as the answer. Plus, this is a different problem entirely, and you lose the ability to infer sin(t) based on t. It doesn’t answer the same question. Your suggestion is further going to be seriously confounded if the periodic function is more complex, say, the sum of several sin waves. There can be an arbitrary number of values y that correctly match f(x). You should try it before making assumptions that it’s easy, see what it takes to train a net to predict sin(x), without embedding knowledge of the fact that sin is periodic. > The problem is improperly evaluating the neural network as a function of time, instead of evaluating the network as a function of state. This also sounds like an assumption to me that somehow the entire world of research has failed to consider the most obvious of ideas. The point of both optimizers and neural networks is that they can be black boxes, right? It doesn’t matter at all whether the input is time based or position based or a function of money. The network, in theory, can learn any function, time or otherwise, and there’s nothing special about time. But, neural networks function better with domain knowledge. When you know the function domain is time and that the output is periodic, you can do things to make a network easier to train, like using a periodic activation function. As a side note, RNNs explicitly model an NN based on previous state. Also all layered NNs can be viewed as a series of smaller nets that feed state to the next net. In some sense, NNs always evaluate as a function of state.