6 ms·
> Ironically, by supporting the LLM community with free compute-intense models, they’re decreasing demand (and price) for the compute. For other people that th
by jorvi 2y ago
> Ironically, by supporting the LLM community with free compute-intense models, they’re decreasing demand (and price) for the compute.
For other people that that sentence didn't make sense for at first glance: "by supporting the LLM community with free compute-intense models [to run on their own hardware] they’re decreasing demand (and price) for the compute [server supply]."
- fragmede 2y agoHow fungible is that compute though? Having even a single H100 is different than having a bunch of 4090's, nevermind a properly networked supercomputer of H100s.
- vineyardmike 2y agoThat’s the point. You can run inference on a 4090 but training is better on a H100. If you use llama, you don’t need to train on an H100, so you can free that supply up for meta.
- fragmede 2y agoI haven't been following llama closely but I thought the latest model was too big for inference on 4090's, and that you can't fine tune on 4090's either, but furthermore, the other question is if the market is there for running inference on 4090s.
- vineyardmike 2y agoWell, (1) there are a ton of GPUs out there of various specs, and you can also use an inference provider who can use a H100 or similar to serve multiple inference requests at once. (2) there are a ton of LLAMA sizes, from 1b, 2b, 8b, 70b, and 400b. The smaller ones can even run on phone GPUs.
- vineyardmike 2y agoSorry, I should have been more clear. They’re decreasing demand for expensive GPUs that would be required to train a model. Fine-tuning and inference are less compute intense, so overall demand for top-end GPU performance is decreased even if inference compute demand is increased. Basically, why train an LLM from scratch, and spend millions on GPUs, when you can fine tune LLAMA and spend hundreds instead.
- jorvi 2y agoThank you for the extra clarification, I hadn’t even thought of inference vs training!