3 ms·

Alexandr Wang did not even say they lied in the paper. Here's the interview: https://www.youtube.com/watch?v=x9Ekl9Izd38 https://www.youtube.com/watch?v=x9Ekl9

by Leary 2y ago

Alexandr Wang did not even say they lied in the paper. Here's the interview: https://www.youtube.com/watch?v=x9Ekl9Izd38 https://www.youtube.com/watch?v=x9Ekl9Izd38. "My understanding is that is that Deepseek has about 50000 a100s, which they can't talk about obviously, because it is against the export controls that the United States has put in place. And I think it is true that, you know, I think they have more chips than other people expect..." Plus, how exactly did Deepseek lie. The model size, data size are all known. Calculating the number of FLOPS is an exercise in arithmetics, which is perhaps the secret Deepseek has because it seemingly eludes people.

leetharris 2y ago
> Plus, how exactly did Deepseek lie. The model size, data size are all known. Calculating the number of FLOPS is an exercise in arithmetics, which is perhaps the secret Deepseek has because it seemingly eludes people. Model parameter count and training set token count are fixed. But other things such as epochs are not. In the same amount of time, you could have 1 epoch or 100 epochs depending on how many GPUs you have. Also, what if their claim on GPU count is accurate, but they are using better GPUs they aren't supposed to have? For example, they claim 1,000 GPUs for 1 month total. They claim to have H800s, but what if they are using illegal H100s/H200s, B100s, etc? The GPU count could be correct, but their total compute is substantially higher. It's clearly an incredible model, they absolutely cooked, and I love it. No complaints here. But the likelihood that there are some fudged numbers is not 0%. And I don't even blame them, they are likely forced into this by US exports laws and such.
- kd913 2y ago
  It should be trivially easy to reproduce the results no? Just need to wait for one of the giant companies with many times the GPUs to reproduce the results. I don't expect a #180 AUM hedgefund to have as many GPUs than meta, msft or Google.
  - sudosysgen 2y ago
    AUM isn't a good proxy for quantitative hedge fund performance, many strategies are quite profitable and don't scale with AUM. For what it's worth, they seemed to have some excellent returns for many years for any market, let alone the difficult Chinese markets.
- sudosysgen 2y ago
  > In the same amount of time, you could have 1 epoch or 100 epochs depending on how many GPUs you have. This is just not true for RL and related algorithms, having more GPU/agents encounters diminishing returns, and is just not the equivalent to letting a single agent go through more steps.

by Leary 2y ago

leetharris 2y ago

> Plus, how exactly did Deepseek lie. The model size, data size are all known. Calculating the number of FLOPS is an exercise in arithmetics, which is perhaps the secret Deepseek has because it seemingly eludes people. Model parameter count and training set token count are fixed. But other things such as epochs are not. In the same amount of time, you could have 1 epoch or 100 epochs depending on how many GPUs you have. Also, what if their claim on GPU count is accurate, but they are using better GPUs they aren't supposed to have? For example, they claim 1,000 GPUs for 1 month total. They claim to have H800s, but what if they are using illegal H100s/H200s, B100s, etc? The GPU count could be correct, but their total compute is substantially higher. It's clearly an incredible model, they absolutely cooked, and I love it. No complaints here. But the likelihood that there are some fudged numbers is not 0%. And I don't even blame them, they are likely forced into this by US exports laws and such.

kd913 2y ago
It should be trivially easy to reproduce the results no? Just need to wait for one of the giant companies with many times the GPUs to reproduce the results. I don't expect a #180 AUM hedgefund to have as many GPUs than meta, msft or Google.
- sudosysgen 2y ago
  AUM isn't a good proxy for quantitative hedge fund performance, many strategies are quite profitable and don't scale with AUM. For what it's worth, they seemed to have some excellent returns for many years for any market, let alone the difficult Chinese markets.
sudosysgen 2y ago
> In the same amount of time, you could have 1 epoch or 100 epochs depending on how many GPUs you have. This is just not true for RL and related algorithms, having more GPU/agents encounters diminishing returns, and is just not the equivalent to letting a single agent go through more steps.