3 ms·
well, we can AI inference on our desktops for $500 today, just with smaller models and far slower.
by stefs 2y ago
well, we can AI inference on our desktops for $500 today, just with smaller models and far slower.
- ryao 2y agoThere is no need to use smaller models. You can run the biggest models such as llama 3.1 405B on a fairly low end desktop today: https://github.com/lyogavin/airllm https://github.com/lyogavin/airllm However, it will be far slower as you said.