4 ms·

How does this sort of thing perform with 70b models?

• hnuser123456 1y ago
  273 GB/s / 70GB = 3.9 tokens/sec
  • mdp2021 1y ago
    Are you sure that kind of computation can be a general rule? Did you mean that the maximum rate it could be obtained is "bandwidth/size"?
    • deleted 1y ago
      [deleted]
    • hnuser123456 1y ago
      Yes, for most LLMs the transformer means the entire model and context is read from VRAM for every token.