3 ms·

How does DeepSeek work: An inside look

55 points by thunderbong 2y ago

texan_dev123 2y ago
"DeepSeek’s policy states that it stores the information for 'further training' of the chatbot in Chinese servers. While it’s not something to get panicked about (most of the applications follow the same principle, despite not being overly open about it)" Is this really true?
- relyks 2y ago
  Why wouldn't it be? OpenAI and Anthropic keep everyone's prompts and use them for training too
  - astrange 2y ago
    Anthropic says > To date we have not used any customer or user-submitted data to train our generative models. https://www.anthropic.com/news/claude-3-5-sonnet https://www.anthropic.com/news/claude-3-5-sonnet There's an obvious problem with the concept of training on user prompts; how would training on a bunch of questions cause it to know the answers?
    - space_fountain 2y ago
      User queries were at least historically useful to train smaller models from larger models. You need to know the kind of questions real people ask to train a model that’s good at answering those questions
    - lukan 2y ago
      "There's an obvious problem with the concept of training on user prompts; how would training on a bunch of questions cause it to know the answers?" I imagine by analysing the chat? If the user says thanks in the end, or gives a thumps up, it likely was a useful and correct answer, that could be included in further training. Or at least considered for future training and I cannot imagine them not considering and experimenting with it.
    - CarRamrod 2y ago
      Back when I started using LLMs for writing code I would type out long, gently phrased explanations about why it was wrong, as if I was teaching a pupil, hoping it would help. I'm sure a lot of us did. If they can parse and mine those prompts, they'll have a nice little metacorpus to build on. Now I just tell it to stop being stupid over and over until it does a good job. I wonder if it would improve the model to keep all of the beratement in the training data. Edit: Apparently a 'metacorpus' is a swollen nematode ass. My sincerest apologies, bros.
  - cheshire_cat 2y ago
    Anthropic states that they don't train on the inputs and outputs of their commercial offerings unless you explicitly opt-in: https://privacy.anthropic.com/en/articles/7996868-i-want-to-opt-out-of-my-prompts-and-results-being-used-for-training-models https://privacy.anthropic.com/en/articles/7996868-i-want-to-... Do you think they're lying or where you speaking about free tier offerings?
    - WhereIsTheTruth 2y ago
      If they lied about copyright infringements, why wouldn't they lie for data collection too?
    - anon373839 2y ago
      The bigger question is what ELSE are Anthropic/OpenAI/et al. doing with your data? Training is just one of many ways to exploit users’ data. Some of the other possibilities are truly chilling.
  - energy123 2y ago
    Because of how corporations and state are tightly fused in China's governance. > A Leninist system features an authoritarian regime in which the ruling elite monopolizes political power in the name of a revolutionary ideology through a highly articulated party structure that parallels, penetrates, and dominates the state at all levels and extends to workplaces, residential areas, and local institutions. From: https://www.csis.org/analysis/soviet-lessons-china-watching https://www.csis.org/analysis/soviet-lessons-china-watching All user data submitted to DeepSeek is accessible to the CCP.
    - csmpltn 2y ago
      As opposed to the US?
      - noduerme 2y ago
        Yes. These are not comparable political systems. In the US, the information you share can be accessed by law enforcement with the approval of a judge if there's a crime suspected. But in cases where the government improperly accesses your data, they actually destroy their own case against you, because anything from that poisoned tree of evidence can be thrown out in court. Even when governmental power is abused in the US, it is nothing like the routine surveillance and suppression that chills free thought and speech in a totalitarian dictatorship like China.
        okasaki 2y ago
        https://en.wikipedia.org/wiki/Parallel_construction https://en.wikipedia.org/wiki/Parallel_construction
        csmpltn 2y ago
        > "In the US..." I'm sorry, but your idea of how the US works is a complete fairytale. You need to get a serious reality check on how the US actually works in real life. The law in the US is applied selectively (depending on the profiles involved, severity of case, political backdrop, etc). There's plenty of corruption, misaligned incentives, and corporate meddling. I can't count the number of cases from the past 30+ years that demonstrate this.
cubefox 2y ago
The headline says "an in-depth look", but the whole post is quite short and doesn't go into much detail. I found these overviews better: https://www.lesswrong.com/posts/a9GR7m4nyBsqjjL8d/deepseek-r1-for-beginners https://www.lesswrong.com/posts/a9GR7m4nyBsqjjL8d/deepseek-r... https://newsletter.languagemodels.co/p/the-illustrated-deepseek-r1 https://newsletter.languagemodels.co/p/the-illustrated-deeps...
- krackers 2y ago
  Also https://epoch.ai/gradient-updates/how-has-deepseek-improved-the-transformer-architecture https://epoch.ai/gradient-updates/how-has-deepseek-improved-..., it's a much better overview with technical details, but still very understandable to the educated layman.
bingzhuwuhen 2y ago
if you know Chinese，this may help you。 https://www.meoai.net/deepseek-r1.html https://www.meoai.net/deepseek-r1.html
netfortius 2y ago
How could such systems prevent purposeful declining/refusal of correct answers, followed (within the "chat") by demand for "corrections" in misleading ways, and stopping the "chat" only when the answer is obviously wrong? Couldn't instructions in one such solution, meant to DoS (not in volume, but in malicious purposely constructed "conversations") the competitor, lead to an overall degradation of quality in all, eventually?
- llm_trw 2y ago
  They don't. R1 gets the right answer in the thinking part of the response and ignores it in the response more than half the time in my tests.
qeternity 2y ago
Was this written by DeepSeek? Aside from not being in depth, it’s also inaccurate (MoE details and MTP misunderstanding).
- rez-havaei 2y ago
  I'm a bit curious to know what was inaccurate in it.
  - qeternity 2y ago
    How about you go study first, instead of just trying to crank out AI generated slop. Absolutely no point in helping you correct your articles when they should instead be left obviously bad, so that they can be flagged.
- loveparade 2y ago
  IMO the more interesting question is why low-quality stuff like this keeps getting upvoted here. Feels like any submission that has AI in it automatically gets to the front page no matter the quality. Sad state of HN. I just can't imagine that people actually read this stuff and then decide to upvote because they found it useful. It's probably upvoted by people/bots who only read the title. The whole reason I come to HN in the first place is to filter out BS clickbait articles exactly like this one, not to have them fill the front page.
  - quietbritishjim 2y ago
    I'm certainly not an extreme HN old timer, but I've been visiting for a fair number of years and I've seen this sort of complaint since I started, while article quality doesn't seem to have gone down noticeably. In fact, the site rules even caution against complaining that HN is "becoming Reddit", which is essentially the old version of this comment. The fact is that, even here, there will always be a few poor quality articles that slip through. BTW, pointing out that a particular article is poor, like qeternity's comment, is worthwhile. It's just comments that complain all of HN is going downhill that are tiresome.
deleted 2y ago
[deleted]
SebFender 2y ago
"But for me, there’s another reason: DeepSeek feels unbiased and direct" Is it just me or this person hasn't read much on the subject?
- buyucu 2y ago
  DeepSeek feels a lot less censored than OpenAI models.

55 points by thunderbong 2y ago

"DeepSeek’s policy states that it stores the information for 'further training' of the chatbot in Chinese servers. While it’s not something to get panicked about (most of the applications follow the same principle, despite not being overly open about it)" Is this really true?

relyks 2y ago
Why wouldn't it be? OpenAI and Anthropic keep everyone's prompts and use them for training too
- astrange 2y ago
  Anthropic says > To date we have not used any customer or user-submitted data to train our generative models. https://www.anthropic.com/news/claude-3-5-sonnet https://www.anthropic.com/news/claude-3-5-sonnet There's an obvious problem with the concept of training on user prompts; how would training on a bunch of questions cause it to know the answers?
  - space_fountain 2y ago
    User queries were at least historically useful to train smaller models from larger models. You need to know the kind of questions real people ask to train a model that’s good at answering those questions
  - lukan 2y ago
    "There's an obvious problem with the concept of training on user prompts; how would training on a bunch of questions cause it to know the answers?" I imagine by analysing the chat? If the user says thanks in the end, or gives a thumps up, it likely was a useful and correct answer, that could be included in further training. Or at least considered for future training and I cannot imagine them not considering and experimenting with it.
  - CarRamrod 2y ago
    Back when I started using LLMs for writing code I would type out long, gently phrased explanations about why it was wrong, as if I was teaching a pupil, hoping it would help. I'm sure a lot of us did. If they can parse and mine those prompts, they'll have a nice little metacorpus to build on. Now I just tell it to stop being stupid over and over until it does a good job. I wonder if it would improve the model to keep all of the beratement in the training data. Edit: Apparently a 'metacorpus' is a swollen nematode ass. My sincerest apologies, bros.
- cheshire_cat 2y ago
  Anthropic states that they don't train on the inputs and outputs of their commercial offerings unless you explicitly opt-in: https://privacy.anthropic.com/en/articles/7996868-i-want-to-opt-out-of-my-prompts-and-results-being-used-for-training-models https://privacy.anthropic.com/en/articles/7996868-i-want-to-... Do you think they're lying or where you speaking about free tier offerings?
  - WhereIsTheTruth 2y ago
    If they lied about copyright infringements, why wouldn't they lie for data collection too?
  - anon373839 2y ago
    The bigger question is what ELSE are Anthropic/OpenAI/et al. doing with your data? Training is just one of many ways to exploit users’ data. Some of the other possibilities are truly chilling.
- energy123 2y ago
  Because of how corporations and state are tightly fused in China's governance. > A Leninist system features an authoritarian regime in which the ruling elite monopolizes political power in the name of a revolutionary ideology through a highly articulated party structure that parallels, penetrates, and dominates the state at all levels and extends to workplaces, residential areas, and local institutions. From: https://www.csis.org/analysis/soviet-lessons-china-watching https://www.csis.org/analysis/soviet-lessons-china-watching All user data submitted to DeepSeek is accessible to the CCP.
  - csmpltn 2y ago
    As opposed to the US?
    - noduerme 2y ago
      Yes. These are not comparable political systems. In the US, the information you share can be accessed by law enforcement with the approval of a judge if there's a crime suspected. But in cases where the government improperly accesses your data, they actually destroy their own case against you, because anything from that poisoned tree of evidence can be thrown out in court. Even when governmental power is abused in the US, it is nothing like the routine surveillance and suppression that chills free thought and speech in a totalitarian dictatorship like China.
      - okasaki 2y ago
        https://en.wikipedia.org/wiki/Parallel_construction https://en.wikipedia.org/wiki/Parallel_construction
      - csmpltn 2y ago
        > "In the US..." I'm sorry, but your idea of how the US works is a complete fairytale. You need to get a serious reality check on how the US actually works in real life. The law in the US is applied selectively (depending on the profiles involved, severity of case, political backdrop, etc). There's plenty of corruption, misaligned incentives, and corporate meddling. I can't count the number of cases from the past 30+ years that demonstrate this.

cubefox 2y ago

The headline says "an in-depth look", but the whole post is quite short and doesn't go into much detail. I found these overviews better: https://www.lesswrong.com/posts/a9GR7m4nyBsqjjL8d/deepseek-r1-for-beginners https://www.lesswrong.com/posts/a9GR7m4nyBsqjjL8d/deepseek-r... https://newsletter.languagemodels.co/p/the-illustrated-deepseek-r1 https://newsletter.languagemodels.co/p/the-illustrated-deeps...

krackers 2y ago
Also https://epoch.ai/gradient-updates/how-has-deepseek-improved-the-transformer-architecture https://epoch.ai/gradient-updates/how-has-deepseek-improved-..., it's a much better overview with technical details, but still very understandable to the educated layman.

bingzhuwuhen 2y ago

if you know Chinese，this may help you。 https://www.meoai.net/deepseek-r1.html https://www.meoai.net/deepseek-r1.html

netfortius 2y ago

How could such systems prevent purposeful declining/refusal of correct answers, followed (within the "chat") by demand for "corrections" in misleading ways, and stopping the "chat" only when the answer is obviously wrong? Couldn't instructions in one such solution, meant to DoS (not in volume, but in malicious purposely constructed "conversations") the competitor, lead to an overall degradation of quality in all, eventually?

llm_trw 2y ago
They don't. R1 gets the right answer in the thinking part of the response and ignores it in the response more than half the time in my tests.

qeternity 2y ago

Was this written by DeepSeek? Aside from not being in depth, it’s also inaccurate (MoE details and MTP misunderstanding).

rez-havaei 2y ago
I'm a bit curious to know what was inaccurate in it.
- qeternity 2y ago
  How about you go study first, instead of just trying to crank out AI generated slop. Absolutely no point in helping you correct your articles when they should instead be left obviously bad, so that they can be flagged.
loveparade 2y ago
IMO the more interesting question is why low-quality stuff like this keeps getting upvoted here. Feels like any submission that has AI in it automatically gets to the front page no matter the quality. Sad state of HN. I just can't imagine that people actually read this stuff and then decide to upvote because they found it useful. It's probably upvoted by people/bots who only read the title. The whole reason I come to HN in the first place is to filter out BS clickbait articles exactly like this one, not to have them fill the front page.
- quietbritishjim 2y ago
  I'm certainly not an extreme HN old timer, but I've been visiting for a fair number of years and I've seen this sort of complaint since I started, while article quality doesn't seem to have gone down noticeably. In fact, the site rules even caution against complaining that HN is "becoming Reddit", which is essentially the old version of this comment. The fact is that, even here, there will always be a few poor quality articles that slip through. BTW, pointing out that a particular article is poor, like qeternity's comment, is worthwhile. It's just comments that complain all of HN is going downhill that are tiresome.

deleted 2y ago

[deleted]

SebFender 2y ago

"But for me, there’s another reason: DeepSeek feels unbiased and direct" Is it just me or this person hasn't read much on the subject?

buyucu 2y ago
DeepSeek feels a lot less censored than OpenAI models.