1) It's not restricted to the variables being passed, for example Zygote [1] has an example in the README with IO gradient(x -> fs[readline()](x), 1), and it's

1) It's not restricted to the variables being passed, for example Zygote [1] has an example in the README with IO gradient(x -> fs[readline()](x), 1), and it's not using numerical differentiation (varying inputs to check the output), but finding a formula (approximate, not a closed form like a symbolic differentiation) of how changing the output affects the input. Genetic algorithms is a black box metaheuristic, so people would favor using gradient descent given that the code is differentiable, but how the model is optimized is technically open to any method. 2) Yes, it can differentiate all control flow. What happens is that each forward/backward pass will have a different graph based on what branches it would follow (effectively a subderivative). Each one of these graphs is independently differentiable and do not contain the control flow by themselves. 3) It doesn't need a language extension (Julia doesn't, but that's because you can access Julia IR using the language itself, not many language support this level of metaprogramming). The OOP strategy (like Pytorch) overload methods so every time you call the overloaded method it builds the graph. It depends on customized types and methods and does not support code that isn't using those custom types that can hold the graph (native/custom types), native control flow (only implicitly by changing what graph is being created), reduces the possible optimizations (you only have one subderivative, not the complete graph to optimize) and end up creating a DSL with it's own error messages and quirks that is less natural than just using the host language. I can't say how it would look using monads though. [1] https://github.com/FluxML/Zygote.jl https://github.com/FluxML/Zygote.jl