4 ms·

I can't wait for the 2000 dot com crash for AI. What gets me about the deepseek work is that it's neither novel or theoretically difficult. I've been using a

I can't wait for the 2000 dot com crash for AI. What gets me about the deepseek work is that it's neither novel or theoretically difficult. I've been using a similar synthetic data pipeline to generate examples for sota bert embedding models. That low-hanging fruits like this are left unexplored by meta or alphabet - and if we're to be believed lead to crisis meetings for a week straight - makes one wonder wtf the engineers there are doing?

spamizbad 2y ago
This is just a baseless conspiracy theory that I have, but I do wonder if they intentionally avoided certain avenues of research because it could erode the “moat” major tech firms have by dramatically reducing the capital costs required for training and inference. If you focus your research around work that mandates high-end hardware at scale you can lock out tons of potential incumbents
- llm_trw 2y ago
  There is nothing in the deepseek paper that suggests you can't use the order of magnitude in hardware costs you saved to just train models that are ten times as large.
  - spamizbad 2y ago
    But there are thresholds of commercial viability in all of this. DeepSeek's technology gets you over that line with less sophisticated hardware. There's already some pretty impressive work being done with folks using just a pair of M2 Ultras with r1 in a "home lab" context that goes way beyond what you could previously do with llama.
    - llm_trw 2y ago
      Yes, but the r1 model is shiny and new. I've been using it locally for a day and I'm already finding odd spots. One thing that I've seen it do a number of times is get the correct answer in the thinking tokens, then ignore it, generate thousands more tokens until it reaches it context length then give a wrong answer. There are things, like doc strings for functions, that llama models give better results for.
  - YurgenJurgensen 2y ago
    …and even if you can’t, you could train models that are ten times more specific, or update them ten times more frequently.

spamizbad 2y ago

This is just a baseless conspiracy theory that I have, but I do wonder if they intentionally avoided certain avenues of research because it could erode the “moat” major tech firms have by dramatically reducing the capital costs required for training and inference. If you focus your research around work that mandates high-end hardware at scale you can lock out tons of potential incumbents

llm_trw 2y ago
There is nothing in the deepseek paper that suggests you can't use the order of magnitude in hardware costs you saved to just train models that are ten times as large.
- spamizbad 2y ago
  But there are thresholds of commercial viability in all of this. DeepSeek's technology gets you over that line with less sophisticated hardware. There's already some pretty impressive work being done with folks using just a pair of M2 Ultras with r1 in a "home lab" context that goes way beyond what you could previously do with llama.
  - llm_trw 2y ago
    Yes, but the r1 model is shiny and new. I've been using it locally for a day and I'm already finding odd spots. One thing that I've seen it do a number of times is get the correct answer in the thinking tokens, then ignore it, generate thousands more tokens until it reaches it context length then give a wrong answer. There are things, like doc strings for functions, that llama models give better results for.
- YurgenJurgensen 2y ago
  …and even if you can’t, you could train models that are ten times more specific, or update them ten times more frequently.