6 ms·
Zoology 1: Measuring and Improving Recall in Efficient Language Models
- convexstrictly 3y agoResearch suggesting that much of the power of the transformer architecture comes from associative recall over long sequences that does not require scaling model dimensions. They design state space models that narrow the gap. Overview https://hazyresearch.stanford.edu/blog/2023-12-11-zoology0-intro https://hazyresearch.stanford.edu/blog/2023-12-11-zoology0-i... Zoology 2 https://hazyresearch.stanford.edu/blog/2023-12-11-zoology2-based https://hazyresearch.stanford.edu/blog/2023-12-11-zoology2-b... Monarchs and Butterflies: Towards Sub-Quadratic Scaling in Model Dimension. Scaling in model dimension as opposed to sequence dimension scaling in the previous posts. https://hazyresearch.stanford.edu/blog/2023-12-11-truly-subquadratic https://hazyresearch.stanford.edu/blog/2023-12-11-truly-subq...