4 ms·
inference can still be a bottleneck i think since you usually load the whole thing into memory which is 32-64GB+ usually?
by ausbah 4y ago
inference can still be a bottleneck i think since you usually load the whole thing into memory which is 32-64GB+ usually?
- visarga 4y agoLanguage models range from 1 to 300+ GB when loaded. It depends on how you load them, if you load in int8 you get 4x reduction.