4 ms·
They are using RingAttention that scales self attention computation linearly by number of devices by passing KV result blocks in a ring: https://arxiv.org/abs/
by xiphias2 3y ago
They are using RingAttention that scales self attention computation linearly by number of devices by passing KV result blocks in a ring:
https://arxiv.org/abs/2310.01889 https://arxiv.org/abs/2310.01889 (Submitted on 3 Oct 2023)