3 ms·
People are announcing the death of foundational models too early. Don’t people realize that the big AI players will take all of the proprietary things they’ve b
by blackeyeblitzar 2y ago
People are announcing the death of foundational models too early. Don’t people realize that the big AI players will take all of the proprietary things they’ve been building up behind closed doors and simply layer onto them all the winning techniques everyone else is publishing (like what DeepSeek has used)? DeepSeek itself is taking ideas that have proven out in various other papers and stacking them up to produce their gains (which they’ve been transparent about in their papers).
I also still don’t believe their cost figures, and think they’re leaving out the capital to acquire their secret GPU stash and the cost of pre training their base model (DeepSeek-V3-base). I also suspect their training corpus, which they’ve only vaguely described, would reveal the savings came from working off other foundational models’ work without counting those costs in their figure.
For now, I treat the cost claim as simply a calculated strategy for China to not look like they’re behind in the most important race, to prevent investors from continuing to boost US technology by causing them to doubt the ROI, and to take value out of the US stock market as they did today.
- bambax 2y agoYeah, the cost figures need more scrutiny; they started with Llama3 which they got for free; had they had to build it from scratch it would have cost more than $6M. But as for your first paragraph: even if the "big AI players" have some secret sauce that will make their products better (and that they can actually keep secret), it seems unlikely it would be enough to command higher prices durably. A model would have to be incredibly superior to justify paying for it, when there are so many free (or dirt cheap) alternatives that are simply good enough.
- blackeyeblitzar 2y agoYou make a good point, that maybe the models won’t perform much better with those improvements, or at least not enough to get people to pay more. I’m curious about the Llama3 bit - do you have a source for that? I’ve been hearing they trained using OpenAI outputs (not sure how that would work).
- Palmik 2y agoI don't know where you're getting your information from. Maybe you're confusing DeepSeek v3/r1 and the distilled r1 models. DeepSeek V3/R1 architecture isn't anything like Llama 3. Llama 3 isn't even a mixture of experts, not to mention the various other differences like attention compression etc
- bambax 2y agoIndeed I got confused. DeepSeek V3 is not based on Llama 3. Sorry about that.