3 ms·

273 GB/s / 70GB = 3.9 tokens/sec

• mdp2021 1y ago
  Are you sure that kind of computation can be a general rule? Did you mean that the maximum rate it could be obtained is "bandwidth/size"?
  • deleted 1y ago
    [deleted]
  • hnuser123456 1y ago
    Yes, for most LLMs the transformer means the entire model and context is read from VRAM for every token.