3 ms·

Which model? I am talking about 70b as mentioned clearly. 70b 8b is 70GB just for the model itself. How much token/second are you getting with single 4090?

• ramesh31 3y ago
  Offloading 40% of layers to CPU, about 50t/s with 16 threads.
  • pocketarc 3y ago
    That is more than an order of magnitude better than my experience; I get around 2 t/s with similar hardware. I had also seen others reporting similar figures to mine so I assumed it was normal. Is there a secret to what you're doing?
    • ramesh31 3y ago
      >Is there a secret to what you're doing? Core speed and memory bandwidth matter a lot. This is on a Ryzen 7950 with DDR5.
  • jpdus 3y ago
    Care to share your detailed stack and command to reach 50t/s? I also have a 7950 with DDR 5 and I don't even get 50 t/s on my two RTX 4090s....