3 ms·
There are layers of abstraction on top of “the math”. The back propagation math for a transformer is no different than for a multi-layer perception, yet a tran
by shwaj 2y ago
There are layers of abstraction on top of “the math”. The back propagation math for a transformer is no different than for a multi-layer perception, yet a transformer is vastly more capable than a MLP. More to the point, it took a series of non-trivial steps to arrive at the transformer architecture. In other words, understanding the lowest-level math is no guarantee that you understand the whole thing, otherwise the transformer architecture would have been obvious.
- theGnuMe 2y agoWe know architecture and training procedures matter in practice. MLPs and transformers are ultimately theoretically equivalent. That means there is an MLP that represent the any function a given transformer can. However, that MLP is hard to identify and train. Also the transformer contains MLPs as well...
- barrell 2y agoI don’t disagree that it’s non-trivial, but we’re comparing this to conciousness, intelligence, even life. Personally I think it’s apples and an orange grove, but I guess we’ll get our answer eventually. Pretty sure we’re on the path to take transformers to their limit, wherever that may be