4 ms·
I haven't been following llama closely but I thought the latest model was too big for inference on 4090's, and that you can't fine tune on 4090's either, but fu
by fragmede 2y ago
I haven't been following llama closely but I thought the latest model was too big for inference on 4090's, and that you can't fine tune on 4090's either, but furthermore, the other question is if the market is there for running inference on 4090s.
- vineyardmike 2y agoWell, (1) there are a ton of GPUs out there of various specs, and you can also use an inference provider who can use a H100 or similar to serve multiple inference requests at once. (2) there are a ton of LLAMA sizes, from 1b, 2b, 8b, 70b, and 400b. The smaller ones can even run on phone GPUs.