I think you're assuming that "automatic differentiation" (AD) is the same thing as "using dual numbers", but AD is a family of methods including dual numbers. T

I think you're assuming that "automatic differentiation" (AD) is the same thing as "using dual numbers", but AD is a family of methods including dual numbers. The blog post you link to says they're using https://github.com/denizyuret/AutoGrad.jl https://github.com/denizyuret/AutoGrad.jl which uses reverse-mode differentiation not Dual numbers. That means they're performing the same operations (or nearly the same) as standard backpropagation. Dual numbers are just the wrong approach for neural nets, which have many parameters. The amazing thing about backprop / reverse-mode differentiation is that you get the derivatives wrt all the weights in one reverse sweep through the network.