4 ms·
A lot of intimidating math that will make all self-attention tutorials seem like a walk in the park in comparison. Luckily subsequent state space models buildin
by imjonse 3y ago
A lot of intimidating math that will make all self-attention tutorials seem like a walk in the park in comparison. Luckily subsequent state space models building on S4 (DSS, S4D and newer ones like Mamba) simplified the primitives and the math used.
- marmaduke 3y agoThe math is not designed to intimidate but rather approach the "how to build sequence model" in a principled way from state space models, which draws from an arguably longer literature than neural networks. Some of concepts are better explained here than anywhere else, and make it straightforward to make sense of Mamba, which is increasingly popular.
- imjonse 3y agoI did not mean it in a negative way, this is a great resource. But the math will be intimidating regardless for most devs who don't have a solid math/signal processing background. It's way beyond the simple linear algebra plus chain rule from calculus that are required to understand basic neural networks training.
- ptojr 3y agoCan someone point me to DSS and S4D papers?
- efm 3y agoDSS: Diagonal State Spaces are as Effective as Structured State Spaces (https://arxiv.org/abs/2203.14343 https://arxiv.org/abs/2203.14343) S4D: On the Parameterization and Initialization of Diagonal State Space Models (https://arxiv.org/abs/2206.11893 https://arxiv.org/abs/2206.11893)
- ptojr 3y agoThank you!
- ssivark 3y agoWell, but this stuff is also much more principled and much better understood (by construction) than why/how a transformer works. The price of actual understanding, and being able to make precise statements, is that the statements will be precise and detailed (ie likely involve math).