> you can't really implement it in a library in a normal programming language, because it has to introspect on how your program is computing everything AFAIK a

> you can't really implement it in a library in a normal programming language, because it has to introspect on how your program is computing everything AFAIK autograd generically refers to a computational graph's gradient being automatically derived from those of it's constituent computational nodes/operators. It's not limited to cases like PyTorch that presents itself as "differentiable programming" where the graph is implicitly built by code execution. The alternative to a differentiable language is something like the original Lua-based Torch, or TensorFlow's original non-eager mode, where the computational graph is explicitly built by library calls out of predefined nodes/operators, each of which knows it's own gradient.