28 ms·
Stupid question: Does "total model footprint of 5.9GB" mean it will run in 8GB of RAM ? Or is that the size on disk ?
by euroderf 14d ago
Stupid question: Does "total model footprint of 5.9GB" mean it will run in 8GB of RAM ? Or is that the size on disk ?
- Aurornis 14d agoThat’s the size of the model. Running it requires additional space for the KV cache depending on how much context you use. You can probably get it running in 8GB of RAM for short outputs but to get okay context length you’d want more.