4 ms·
Any idea how much ram this requires?
by sunflowerfly 2y ago
Any idea how much ram this requires?
- vineyardmike 2y agoThe model sizes are: > 270M, 450M, 1.1B and 3B parameters Which roughly translates to 3GB for the highest end one, depending on context length used.
- SushiHippie 2y ago* ~3GB with 8bit quantization. Without quantization it is ~6GB [0]. 8 bits = 1 byte 3 billion * 1 byte = 3 gigabyte + Some memory for the context of the LLM [0] 3b-instruct has a total file size of 4.94GB + 1.13GB which is 6.07GB which can be seen here: https://huggingface.co/apple/OpenELM-3B-Instruct/tree/main https://huggingface.co/apple/OpenELM-3B-Instruct/tree/main A bit of overhead will always be there, as you probably want to store some metadata next to the raw weights.