3 ms·
> Transformer models are universal function approximators. You have to be very careful with the wording here. The class of Transformer models is a universal (c
by tsimionescu 4y ago
> Transformer models are universal function approximators.
You have to be very careful with the wording here. The class of Transformer models is a universal (continuous) function approximator. A single Transformer model can only approximate 1 particular function.
Also, backpropagation/gradient descent are not universal function approximator constructors - they can't be used to construct an ANN that approximates an arbitrary (continuous) function even with arbitrarily many random input/output pairs from that function's domain.
For example, given a function that is f(x) = floor(x) if floor(x) is even; x if floor(x) is odd, defined for any `double x`, there is no guarantee that doing back-propagation over a random sample of (x, f(x)) pairs will yield the correct function approximator for some arbitrary precision (especially if you don't start with the right architecture).
- mjburgess 4y agoPeople confuse the "representational capacity" of `recurse(AW^X)` with the algorithm which sets `W`.