2 ms·
AFAIK generating a single token requires reading all the weights from RAM. So 5300 GB/s total memory bandwidth / 142 GB weights = ~37.2 tokens per second.
by wmf 2y ago
AFAIK generating a single token requires reading all the weights from RAM. So 5300 GB/s total memory bandwidth / 142 GB weights = ~37.2 tokens per second.