7 ms·
Tree Attention: Topology-Aware Decoding for Long-Context
- brrrrrm 2y agohow does this approach differ from Nvidia's 2019 writeup on using trees to improve allreduce operations? https://developer.nvidia.com/blog/massively-scale-deep-learning-training-nccl-2-4/ https://developer.nvidia.com/blog/massively-scale-deep-learn...
- mrfox321 2y agoSimilar, but specialized to softmax(QK)V computation
- Narhem 2y agoHow often do papers like this make it to industry applications/published research. Seems stuck in between the two.
- mithametacs 2y agoThis is going to shred when it reaches industry. But yeah very math heavy for a software engineering paper.
- ynniv 2y agoHow long can a page of python take? https://github.com/Zyphra/tree_attention/blob/main/tree_shard_test.py https://github.com/Zyphra/tree_attention/blob/main/tree_shar...
- mithametacs 2y agoYou seem to know your stuff. Will this technique work with existing model weights?
- kasmura 2y agoYes, it is just a way of computing the self-attention in a distributed way
- mjburgess 2y agoIt's from industry. It's a funded research startup.
- mjburgess 2y agoI recall reading recently that someone went back and trained an RNN at a similar scale to a GPT and got similar performance on modern hardware (perhaps someone can link me that paper?). ie., the innovation in statistical AI isn't in making the algorithms "smarter", it's finding ways to align the computation with modern GPU hardware -- this has been the story since 2012. In the end, the function all such algs are approximating is a conditional probability. ie., the perfect answer to any prompt is to ignore training entirely, and at inference time, compute an expectation across all historical data. All training does is essentially optimally cache a large part of that computation. This is very different to how it's typically sold/understood, in the sense that there's an appearance that at inference-time some unbounded computation is going on, ie., "thinking"/"reasoning"/etc. But at inference time for any prompt the same amount of computation is used, regardless of the question complexity. So the system will appear to reason (etc.) if it can sample convincingly from its pre-cached computation. This means "innovation" here follows a moore's law S-curve for GPU hardware.
- doctorpangloss 2y agoBut often, less training performs better.
- mjburgess 2y agoYou'd have to give me an example. If less training performs better, it's because it's overfitting. In the case of NLP/CV you, in general, cannot really "overfit" because most of the goal isnt predictive, but rather representational -- so training=templating, rather than predicting. In these cases training for longer should mostly improve the quality of the output, since there isnt any ground truth to overfit against.
- deleted 2y ago[deleted]
- kasmura 2y agoI disagree. Ring attention and tree attention are so general that the core ideas are independent of details about modern GPUs. Maybe so for flash attention but not this. I also disagree because these algorithms are fundamentally about enabling long context by distributing across gpus and this would not be enabled by “moored law for gpu hardware”
- cs702 2y agoInteresting. The authors claim this outperforms Ring Attention for distributed computation of self-attention over multiple GPUs. Distributing computation is necessary whenever context is too long for self-attention's computation to fit in a single GPU's available memory. Github link: https://github.com/Zyphra/tree_attention https://github.com/Zyphra/tree_attention
- tveita 2y agoThe same authors also have a language model at https://github.com/Zyphra/Zamba2 https://github.com/Zyphra/Zamba2 but it's not clear to me if that model is connected to tree attention. The announcement at https://www.zyphra.com/post/zamba2-small https://www.zyphra.com/post/zamba2-small links to this paper, but the paper doesn't actually mention Zamba2 anywhere.