4 ms·
How much disk & RAM does it need? What's your tokens/sec rate (and on which device)?
by mathfailure 1y ago
How much disk & RAM does it need?
What's your tokens/sec rate (and on which device)?
- simonw 1y agoI've been running it on a 64GB M2. My favorite models to run tend to be about 20GB to download (eg Mistral Small 3.1) and use about 20GB of RAM while they are running. I don't have a token/second figure to hand but it's fast enough that I'm not frustrated by it.