4 ms·

I bought two (relatively) old datacenter GPUs with 48gb VRAM total for €200 that gets me 7 token/s for a 70b model.

• 383toast 2y ago
  which GPUs?
  • zargon 2y ago
    Not the GP, but I bought a few P40s over the summer for $150 each. Last I checked they're more expensive now, but it's still cheap vram and fast enough at inference for me.
  • apexalpha 2y ago
    Nvidia M40 and P40.