4 ms·
Why 3 words per minute as opposed to second? Is that a typo? If you have enough RAM (but not VRAM), does it basically become limited by the PCIE lanes? So for t
by bick_nyers 4y ago
Why 3 words per minute as opposed to second? Is that a typo? If you have enough RAM (but not VRAM), does it basically become limited by the PCIE lanes? So for the 112GB model with a Gen 5 GPU (64 GB/s PCIE bandwidth) that would be roughly 2 seconds per word right?
- flangola7 4y agoIf read is 1GB/s then it takes 20s to infer across a 20GB model. That's 3 tokens a minute.
- bick_nyers 4y agoYeah I'm not sure what kind of math I was subscribed to yesterday, thanks
- flangola7 4y ago:-) I keep saying that we don't have to stop AI from hallucinating, we only need to bring the rate to below human level.