5 ms·

No information to be found about it. Hopefully we get benchmarks soon. Reminds me of the days when Mistral would just tweet a torrent magnet link

by _lvbh 1y ago

aibrother 1y ago
getting a similar vibe yeah. given how adjacent they are, wouldn't be surprised if this was an intentional nod from DeepSeek
swyx 1y ago
i think usually deepseek posts a paper after a model release about a day later. no idea why they cant just wait a bit to coordinate stuff. bit messy in the news cycle.
- Destiner 1y ago
  honestly a power move. it's almost as if they don't care about creating a proper buzz.
  - wyre 1y ago
    From what I understand, isn’t DeepSeek just a pet project from a Chinese hedge fund? They have much less reason to create a buzz compared to openAI, Anthropic, or Google.
    - TeMPOraL 1y ago
      None of those players you mention actually need to create a buzz. People will do it for them for free. DeepSeek joined this group after releasing R1. Despite constant protestations of hype among the tech crowd, GenAI really is big enough of a deal that new developments don't need to be pushed onto market; people are voluntarily seeking them out.
      - wongarsu 1y ago
        OpenAI does a lot of work hyping themselves up and creating buzz around things they do or have a vague idea that they might try to do in the future. Not to make people aware of GenAI, but to make sure OpenAI continues to be perceived as the AI company. The company that leads and revolutionizes, with everyone just copying them and trying to match them. That perception is a significant part of their value and probably their biggest moat
        ktallett 1y ago
        Considering just how quickly others followed it's also obviously not the case. Infact the best AI software as in most useful is not theirs. Claude is far more reliable.
        TeMPOraL 1y ago
        Only goes to shown how strong their brand already is in the eyes of the public. I question how much active maintenance it takes, though. In my eyes, they already won big with ChatGPT - the name still is synonymous with LLMs to the general population. Everyone knows what ChatGPT is. Few known what Claude or Gemini is; arguably, more people know what Deepseek is thanks to the splash they made tanking NVidia stock and becoming part of general news coverage for a few days. Still, for regular folks (including business folks in tech industry, too), they all are, respectively, "that other ChatGPT", "ChatGPT from Google" and "that Chinese ChatGPT". It's a pretty sticky perception.
      - janalsncm 1y ago
        Well OpenAI is constantly asking for and raising money. If “buzz” isn’t the right word maybe mystique? Because “race to the bottom in a hyper-commoditized space” probably doesn’t get you billions. No, Sam Altman wants people to believe they are very close to AGI and a TAM of global GDP.
chvid 1y ago
Benchmarks seem like a fools errand at this point; overly tuning models just to specific test already published tests, rather than focusing on making them generalize. Hugging face has a leader board and it seems dominated by models that are finetunings of various common open source models, yet don't seem be broader used: https://huggingface.co/open-llm-leaderboard https://huggingface.co/open-llm-leaderboard
- lossolo 1y ago
  The only benchmarks that match my experience with different models are here https://livebench.ai/#/ https://livebench.ai/#/
  - ribelo 1y ago
    livebench was good, but now it's a joke. Gemini flash is better in coding than pro and sonnet 3.7. And this is only the beginning of weird results.
    - pdimitar 1y ago
      Flash is better than Pro in coding? Whoa... [makes a note to try a few things later this day] Out of curiosity, how did you gauge that?
      - code_biologist 1y ago
        I think your parent comment is citing that as an example of why livebench is no longer a good benchmark. That said, the new Flash is very good for what it is, and IMO after the Pro 05-06 nerfs the two models are much closer in performance for many tasks than they really should be — Pro should be / was way better (RIP 03-25 release). That livebench result may be wrong about the specific ranking, but I think it's right that Flash is in the same class of coding strength as Sonnet 3.7.
        pdimitar 1y ago
z2 1y ago
There's a table here showing some "Overall" and "Median" score, but no context on what exactly was tested. It appears to be in the ballpark as the latest models, but with some cost advantages with the downside of being just as slow as the original r1 (likely lots of thinking tokens). https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd.it%2Fdeepseek-r1-0528-v0-09patvqurk3f1.jpeg%3Fwidth%3D1080%26format%3Dpjpg%26auto%3Dwebp%26s%3D38580df0e04cb8e29921a6a55b24a7f76b5b30ff https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd....
- xelos 1y ago
  It’s appeared on the Livecodebench leaderboard too. Performance on par with O4 Mini - https://livecodebench.github.io/leaderboard.html https://livecodebench.github.io/leaderboard.html

No information to be found about it. Hopefully we get benchmarks soon. Reminds me of the days when Mistral would just tweet a torrent magnet link

by _lvbh 1y ago

No information to be found about it. Hopefully we get benchmarks soon. Reminds me of the days when Mistral would just tweet a torrent magnet link

aibrother 1y ago

getting a similar vibe yeah. given how adjacent they are, wouldn't be surprised if this was an intentional nod from DeepSeek

swyx 1y ago

i think usually deepseek posts a paper after a model release about a day later. no idea why they cant just wait a bit to coordinate stuff. bit messy in the news cycle.

Destiner 1y ago
honestly a power move. it's almost as if they don't care about creating a proper buzz.
- wyre 1y ago
  From what I understand, isn’t DeepSeek just a pet project from a Chinese hedge fund? They have much less reason to create a buzz compared to openAI, Anthropic, or Google.
  - TeMPOraL 1y ago
    None of those players you mention actually need to create a buzz. People will do it for them for free. DeepSeek joined this group after releasing R1. Despite constant protestations of hype among the tech crowd, GenAI really is big enough of a deal that new developments don't need to be pushed onto market; people are voluntarily seeking them out.
    - wongarsu 1y ago
      OpenAI does a lot of work hyping themselves up and creating buzz around things they do or have a vague idea that they might try to do in the future. Not to make people aware of GenAI, but to make sure OpenAI continues to be perceived as the AI company. The company that leads and revolutionizes, with everyone just copying them and trying to match them. That perception is a significant part of their value and probably their biggest moat
      - ktallett 1y ago
        Considering just how quickly others followed it's also obviously not the case. Infact the best AI software as in most useful is not theirs. Claude is far more reliable.
        TeMPOraL 1y ago
        Only goes to shown how strong their brand already is in the eyes of the public. I question how much active maintenance it takes, though. In my eyes, they already won big with ChatGPT - the name still is synonymous with LLMs to the general population. Everyone knows what ChatGPT is. Few known what Claude or Gemini is; arguably, more people know what Deepseek is thanks to the splash they made tanking NVidia stock and becoming part of general news coverage for a few days. Still, for regular folks (including business folks in tech industry, too), they all are, respectively, "that other ChatGPT", "ChatGPT from Google" and "that Chinese ChatGPT". It's a pretty sticky perception.
    - janalsncm 1y ago
      Well OpenAI is constantly asking for and raising money. If “buzz” isn’t the right word maybe mystique? Because “race to the bottom in a hyper-commoditized space” probably doesn’t get you billions. No, Sam Altman wants people to believe they are very close to AGI and a TAM of global GDP.

chvid 1y ago

Benchmarks seem like a fools errand at this point; overly tuning models just to specific test already published tests, rather than focusing on making them generalize. Hugging face has a leader board and it seems dominated by models that are finetunings of various common open source models, yet don't seem be broader used: https://huggingface.co/open-llm-leaderboard https://huggingface.co/open-llm-leaderboard

lossolo 1y ago
The only benchmarks that match my experience with different models are here https://livebench.ai/#/ https://livebench.ai/#/
- ribelo 1y ago
  livebench was good, but now it's a joke. Gemini flash is better in coding than pro and sonnet 3.7. And this is only the beginning of weird results.
  - pdimitar 1y ago
    Flash is better than Pro in coding? Whoa... [makes a note to try a few things later this day] Out of curiosity, how did you gauge that?
    - code_biologist 1y ago
      I think your parent comment is citing that as an example of why livebench is no longer a good benchmark. That said, the new Flash is very good for what it is, and IMO after the Pro 05-06 nerfs the two models are much closer in performance for many tasks than they really should be — Pro should be / was way better (RIP 03-25 release). That livebench result may be wrong about the specific ranking, but I think it's right that Flash is in the same class of coding strength as Sonnet 3.7.
      - pdimitar 1y ago

z2 1y ago

There's a table here showing some "Overall" and "Median" score, but no context on what exactly was tested. It appears to be in the ballpark as the latest models, but with some cost advantages with the downside of being just as slow as the original r1 (likely lots of thinking tokens). https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd.it%2Fdeepseek-r1-0528-v0-09patvqurk3f1.jpeg%3Fwidth%3D1080%26format%3Dpjpg%26auto%3Dwebp%26s%3D38580df0e04cb8e29921a6a55b24a7f76b5b30ff https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd....

xelos 1y ago
It’s appeared on the Livecodebench leaderboard too. Performance on par with O4 Mini - https://livecodebench.github.io/leaderboard.html https://livecodebench.github.io/leaderboard.html