3 ms·
Isn't this simply context pruning/optimization?
by jgreid 4mo ago
Isn't this simply context pruning/optimization?
- kylemaxwell 4mo agoFrom the abstract, it looks like it's actually doing something deeper, updating weights in part of the model?
- samsartor 4mo agoThe abstract and method sections only mention updating the SSM state during "sleep" (ie the same vectors that change after each token in stock Mamba) not any of the actual weight matrices. AFAICT this is just another attention compaction paper, with misleading tile? It is not very clearly written
- colechristensen 4mo agoNo, they're actually training weights based on context before compaction. Context is context, this is splitting the model into persistent weights and malleable ones which are periodically updated.
- delis-thumbs-7e 4mo agoWouldn’t that be extremely computationaly expensive considering how resource incentive training is?
- colechristensen 4mo agoNo, training a state of the art model involves training on the order of 10 trillion tokens. We're talking about a step that updates weights based on say between 10k and 1M tokens.
- delis-thumbs-7e 4mo agoI learned something. Thank you!