4 ms·
RAM is the limiting factor, but Mixtral 8x7 is probably the state of the art for self-hosted LLMs right now. If you are low on RAM you can run a quantized versi
by LeoPanthera 3y ago
RAM is the limiting factor, but Mixtral 8x7 is probably the state of the art for self-hosted LLMs right now. If you are low on RAM you can run a quantized version, at the expense of reduced quality.