6 ms·

BTW this older article of mine is extended with a new one that shows how to handle multiple variables (: http://blog.demofox.org/2017/02/20/multivariable-dual-n

BTW this older article of mine is extended with a new one that shows how to handle multiple variables (: http://blog.demofox.org/2017/02/20/multivariable-dual-numbers-automatic-differentiation/ http://blog.demofox.org/2017/02/20/multivariable-dual-number...

imurray 10y ago
> If I had to guess, I’d say that dual numbers may be slightly slower than backpropagation The cost of Dual numbers (a form of forward-mode differentiation) scales linearly with the number of derivatives (just like finite differencing, but more accurate). Backpropagation, or reverse-mode differentiation, is a constant factor times the cost of a function evaluation. For neural nets with millions of parameters, backpropagation is going to be millions of times faster than dual numbers.
- Atrix256 10y ago
  Thanks!
- dnautics 10y ago
  http://www.denizyuret.com/2015/02/beginning-deep-learning-with-500-lines.html http://www.denizyuret.com/2015/02/beginning-deep-learning-wi... This guy did neural nets with dual numbers in julia and found it to take ~1.2 times as long as standard backpropagation, which suggests that if you had dedicated hardware for it it would be very nice.
  - imurray 10y ago
    I think you're assuming that "automatic differentiation" (AD) is the same thing as "using dual numbers", but AD is a family of methods including dual numbers. The blog post you link to says they're using https://github.com/denizyuret/AutoGrad.jl https://github.com/denizyuret/AutoGrad.jl which uses reverse-mode differentiation not Dual numbers. That means they're performing the same operations (or nearly the same) as standard backpropagation. Dual numbers are just the wrong approach for neural nets, which have many parameters. The amazing thing about backprop / reverse-mode differentiation is that you get the derivatives wrt all the weights in one reverse sweep through the network.
    - dnautics 10y ago
      Huh. I stand corrected. I thought for sure the package used automatic differentiation using dual numbers.

imurray 10y ago

> If I had to guess, I’d say that dual numbers may be slightly slower than backpropagation The cost of Dual numbers (a form of forward-mode differentiation) scales linearly with the number of derivatives (just like finite differencing, but more accurate). Backpropagation, or reverse-mode differentiation, is a constant factor times the cost of a function evaluation. For neural nets with millions of parameters, backpropagation is going to be millions of times faster than dual numbers.

Atrix256 10y ago
Thanks!
dnautics 10y ago
http://www.denizyuret.com/2015/02/beginning-deep-learning-with-500-lines.html http://www.denizyuret.com/2015/02/beginning-deep-learning-wi... This guy did neural nets with dual numbers in julia and found it to take ~1.2 times as long as standard backpropagation, which suggests that if you had dedicated hardware for it it would be very nice.
- imurray 10y ago
  I think you're assuming that "automatic differentiation" (AD) is the same thing as "using dual numbers", but AD is a family of methods including dual numbers. The blog post you link to says they're using https://github.com/denizyuret/AutoGrad.jl https://github.com/denizyuret/AutoGrad.jl which uses reverse-mode differentiation not Dual numbers. That means they're performing the same operations (or nearly the same) as standard backpropagation. Dual numbers are just the wrong approach for neural nets, which have many parameters. The amazing thing about backprop / reverse-mode differentiation is that you get the derivatives wrt all the weights in one reverse sweep through the network.
  - dnautics 10y ago
    Huh. I stand corrected. I thought for sure the package used automatic differentiation using dual numbers.