4 ms·
What an utter bullshit take. As if the ATI could ever be competitive at LLMs given the resource requirements needed to train foundational models. If we want UK
by anonymousDan 2y ago
What an utter bullshit take. As if the ATI could ever be competitive at LLMs given the resource requirements needed to train foundational models. If we want UK universities to continue to make contributions in this sphere we either need to massively reduce the cost of training SOTA LLMs, create some national shared GPU infrastructure, or resource universities to access cloud infra. Unfortunately the latter is just introducing further dependencies on overseas (and increasingly problematic) private clouds. If things continue as they are then having the capability to train SOTA models will be a strategic imperative for every nation.
- frozenseven 2y agoWhat was the budget for DeepSeek's V3/R1 again?
- anonymousDan 2y agoWho actually knows? Far beyond what a UK university can afford.
- frozenseven 2y agoI see. Let's just assume that DeepSeek's V3/R1 budget of ~$5.5M was a lie and the Alan Turing Institute is just too poor to compete with their nine-figure sums. I guess I have no further questions.
- bubbler 2y agoYeah, the DeepSeek budget wasn't 6M by any means. > the $5-6M cost of training is misleading. It comes from the claim that 2048 H800 cards were used for one training, which at market prices is upwards of $5-6M. Developing such a model, however, requires running this training, or some variation of it, many times, and also many other experiments (item 3 below). That makes the cost to be many times above that, not to mention data collection and other things, a process which can be very expensive (why? item 4 below). Also, 2048 H800 cost between $50-100M. The company that deals with DC is owned by a large Chinese investment fund, where there are many times more GPUs than 2048 H800. https://therecursive.com/martin-vechev-of-insait-deepseek-6m-cost-of-training-is-misleading/ https://therecursive.com/martin-vechev-of-insait-deepseek-6m...
- frozenseven 2y agoOof, sounds like the budget for V3/R1 was exactly what I said. Having access to compute and running experiments is kind of the bare minimum for a supposed AI lab. And since this is a Western lab, their options for training are far more advantageous. But even if I were to accept all of those ridiculous fudged numbers, that's still within the Alan Turing Institute's budget. Of course their anti-deep learning "most senior scientist" hasn't heard of DeepSeek, lol.