4 ms·
RAM and what GPU you have are big determinants of how fast it will run, and how smart a model you can run. A large amount of RAM and GPU memory is required for
by jonnycomputer 2y ago
RAM and what GPU you have are big determinants of how fast it will run, and how smart a model you can run. A large amount of RAM and GPU memory is required for larger models without significant slowdown because its much faster if it can keep the entire model in memory. Small models range from 3-8 gigabytes, but a 70B parameter model will be 30-50 gigabytes.
- neop1x 2y agoI am running 70B models on M2 Max with 96 GB of RAM and it works very well. As HW evolves, it will become a standard