3 ms·

one 4090 + 3090-Ti https://github.com/turboderp/exllama https://github.com/turboderp/exllama

• trifurcate 3y ago
  That's not a minimum, just what the author had on hand. I can get 65B and 70B running just fine with exllama on 2 3090s.