12 ms·

I find that the gushing around deepseek is fascinating to watch. To me there are a few structural and fundamental reasons why deepseek can never outperform oth

I find that the gushing around deepseek is fascinating to watch. To me there are a few structural and fundamental reasons why deepseek can never outperform other models by a wide margin. On par maybe--as we reach the diminishing returns with our investment in the models, but not win by a wide margin. 1. The US trade war with china which will place deepseek compute availability at disadvantages, eventually, if we ever get to that. 2. China censorship which limits the deepseek data ingestion and output, to some degree. 3. Most importantly, deepseek is open source, which means that the other models are free to copy whatever secret source it has, eg: Whatever architecture that purportedly use less compute can easily be copied. I've been using Gemini, chatgpt, deepseek and Claudie on regular basis. Deepseek is neither better or worse than others. But this says more about my own limited usage of LLM rather than the usefulness of the models. I want to know exactly what makes everyone thinks that deepseek totally owns the LLM space? Do I miss anything? PS: I am a Malaysian Chinese, so I am certainly not "a westerner who is jealous and fearful of the rise of China"

logicchains 2y ago
>I want to know exactly what makes everyone thinks that deepseek totally owns the LLM space? It achieved competitive performance to the competition at literally 10x less cost of production (training). That's an incredible achievement in any industry, especially given they have such a small team relative to competitors. Their API is 20-50x cheaper than the competitors, and not because they're burning cash by charging less than costs, but rather because their architecture is just that much more efficient. They already achieved the above in spite of sanctions limiting their availability to top-tier GPUs, and the gap between Chinese domestic GPUs and NVidia is getting smaller and smaller, so in future the GPU disadvantage will be less and less.
- nsoonhui 2y ago
  But like I said, deepseek is open source so why can't the competitors copy whatever source that makes the cost of production 10x cheaper ?
  - mike_hearn 2y ago
    You have to distinguish between the current model and DeepSeek the company. DeepSeek the company can do an OpenAI and stop releasing their weights any time they like. The knowledge and skill is retained. I really wonder how long the current era of giving models away for free can last. How is this sensible from a business perspective? Facebook got burned by iOS and now engage in what would otherwise look like irrational behavior to avoid being locked into a supplier again, but even then, they don't really need to give Llama away for free. They could train and use it for themselves just fine.
    - nprateem 2y ago
      You don't think FB are trying to neuter an emerging threat? They're kneecapping what could have been a trillion dollar company if it was more difficult to replicate their tech.
    - coliveira 2y ago
      If they're smart, and of course they are, they're not releasing the latest they have. They're releasing something enough to show everyone that they're at parity or better compared to OpenAI. I imagine they already have internal models that exceed the open source one, so there's no real advantage in copying what they released.
    - cma 2y ago
      They would have to pay more to get researchers that don't publish.
      - mike_hearn 2y ago
        I was talking about open weight models more than papers, but OpenAI hardly publishes papers anymore and don't seem to struggle to get researchers. Anthropic are clearly doing a lot of special sauce given Claude 3.5 Sonnet's performance on coding, yet the papers they publish are mostly safety related. So I'm not sure that's really true anymore.
    - mistercheph 2y ago
- arisAlexis 2y ago
  Of course if you arrive last and copy all the existing architecture you can train it cheaper
  - viraptor 2y ago
    No, you can only train at the same cost then. (Actually higher, because you don't have the existing hardware/power agreements) The whole point of the last model was that they made significant changes beyond just copying.
    - arisAlexis 2y ago
      No because you can train once and avoid all the errors that are costly and make you train and retrain.
  - msp26 2y ago
    > copy You mean build on existing public research? Everyone does that. At least deepseek, meta etc. also have the decency to publish research back into this ecosystem.
- SubiculumCode 2y ago
  How would we know if a Chinese company's books on training costs and expenditures was accurate?
  - tw1984 2y ago
    are you on CCP's payroll for covering their rise in AI? you are basically implying that people shouldn't believe the progress they made and thus don't need to take it seriously. nice job, you should get a pretty solid performance review result.
    - SubiculumCode 2y ago
      Oh I take China seriously, I'm merely reminding people to not blindly accept stated figures from the CCP.
- int_19h 2y ago
  It should be noted that DeepSeek routinely claims to be a "language model trained by OpenAI", so it's pretty clear that it wasn't trained at 10x less cost from scratch, but rather on synthetic output generated by ChatGPT. Not to point a finger at DeepSeek specifically; this is generally the case for best open source models right now. The best LLaMA finetunes tend to also use ChatGPT-generated synthetic datasets a lot. Either way, it's unclear what the real cost is when you factor that in.
  - Eisenstein 2y ago
    If that was the case why are they 10x cheaper than the competition? If everyone is doing it there would be no gains over competitors.
viraptor 2y ago
> The US trade war with china which will place deepseek compute availability at disadvantages Will it? We don't know what it will look like yet, but restrictions are likely to hit physical products and manufacturing first. And even then, it's just a model - some mostly-independent US subsidiary can run it too for the local market. > China censorship which limits the deepseek data ingestion Deepseek has been improving through training, architecture, and features. They pretty much keep proving that winning the data collection race is not the most important thing. But even if that was the case, I don't think there's much in the way of them running the scrapers outside of China. > Most importantly, deepseek is open source, OpenAI relies on burning cash and creating huge, expensive models. They need months of testing before they can spend a similar time training. Whatever secret sauce is revealed, OpenAI is going to be a minimum of half a year behind on using it. (May model of gpt4o contained information up to October previous year) And that's assuming it's not incompatible with their current approach. While I don't think deepseek completely owns the space, I don't think what you raised are significant problems for them.
tossandthrow 2y ago
> ... why deepseek can never outperform ... This read more like a "western supremacists" post. 1. Only until China produces more compute than the west. 2. You don't have to ask ChatGPT / Claude many questions before realizing the grave censorship these are under - DeepSeek has access the roughly the same corpus of data as their western counter parts. 3. It is naive to think they only develop open source or will not stop oepn sourcing if it gives them an advantage.
- nbroyal 2y ago
  Curious to hear more about the grave censorship that ChatGPT and Claude are under. Specifically where non-western models are not.
  - tossandthrow 2y ago
    I am not aware of any non western models that are not under censorship. Ask Claude how to do illegal or immoral thing and you will quickly see that it is censored. I didn't mean to problematize censorship. Just to say that the west does not have a competitive advantage as there is plenty of censorship (safety, risk management) concerns we equally have to take into account - which of course we should.
    - Workaccount2 2y ago
      Trying to equate government mandated censorship to private company policy censorship is a wholly dishonest sleight of hand.
      - tossandthrow 2y ago
        Both in the EU and the US there is plenty of regulation that mandates these types of censorship - and with reason. In the US there is 18 U.S.C. § 842(p). In the EU there is the entire AI Act. But I am sure you can yourself chat your way through to figure out what legislation companies like OpenAI and Anthropic are under.
rapsey 2y ago
[flagged]
- dang 2y ago
  You can't attack another user on HN like that, regardless of how wrong they are or you feel they are. We ban accounts that do this, so please don't do it again. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
suraci 2y ago
deepseek doesn't need to outperform other models, it just needs to be cheap, or, efficient the cost of deepseek (if it's true) will disrupt the logic of current AI industry The current AI industry is built on a financing bubble, where investors hand over money blindly without demanding that companies profit from AI. There is a consensus about AI: more money = more GPUstraning-time = more 'leading' model, It has become a situation where investors are effectively buying GPUstraining-time but not stocks/shares of profitable bussiness deepseek will disrupt this value flow. > Alibaba Cloud announced the third round of price cuts for its large models this year, with the visual understanding models of the General Qwen-VL models experiencing a price reduction of over 80% across the board. The Qwen-VL-Plus model saw a direct price drop of 81%, with the input cost being only 0.0015 yuan per thousand tokens, setting a record for the lowest price across the network. The higher-performance Qwen-VL-Max model was reduced to 0.003 yuan per thousand tokens, with a significant decrease of 85%. According to the latest prices, one yuan can process up to approximately 600 720P images or 1700 480P images.
jdietrich 2y ago
I don't think it's necessarily about DeepSeek, but about the wider competitive picture. There are two tacit assumptions being made about LLMs - that having a SOTA model is a substantial competitive advantage, and that the demand for compute will continue to grow rapidly. DeepSeek's phenomenal success in reducing training and inference cost points to the possibility of a very different future. If it's the case that SOTA or near-SOTA performance is commoditised and progress in efficiency outpaces progress in capability, then the roadmap looks radically different. If DeepSeek don't have a competitive advantage, then no-one has a competitive advantage. Having a DC full of H200s or a proprietary model with a trillion parameters might not count for anything, in which case we're looking at a very different set of winners and losers. Application specific fine-tuning and product-market fit might matter much more than brute force compute.
- lumost 2y ago
  Isn't this the nature of past technology developments? few tech companies have a true technical "moat" - In California, the employees of any firm are free to raise funds and start a competitor the moment they are dissatisfied with the current leadership/compensation/location. During my career I have yet to observe a "secret sauce" that took more than a few weeks to learn and understand once on the inside. The technical moats we know of in B2B have typically come from a combination of a large number of features efficiently tied into a platform/service that would be cost prohibitive to replicate (ElasticSearch, most successful Database firms), a network effect around that platform the makes it difficult not to be on the platform (CUDA, x86, windows).
- echelon 2y ago
  >> 3. Most importantly, deepseek is open source, which means that the other models are free to copy whatever secret source it has, eg: Whatever architecture that purportedly use less compute can easily be copied. > I don't think it's necessarily about DeepSeek, but about the wider competitive picture. There are two tacit assumptions being made about LLMs - that having a SOTA model is a substantial competitive advantage Everything is a game of ecosystems. Windows lost to Linux on servers because it was cheap and easy to deploy Linux. Thousands of engineers and companies could build in the Linux playground for free and do whatever they wanted, whereas Windows servers were restrictive and static and costly. Dall-E lost to Stable Diffusion and Flux because the latter were open source. You could fine tune them on your own data, run them on your own machine, build your own extensions, build your own business. ComfyUI, IPAdapter, ControlNet, Civitai... It's a flourishing ecosystem and Dall-E is none of that. It'll happen with LLMs (Llama, Qwen, DeepSeek), video models (Hunyuan, LTX), and quite possibly the whole space. One company can only do so much, and there is no real moat. You can't beat the rest of society once they overcome the activation energy. And any third place player will be compelled to open source their model to get users. Open source models will continue to show up at a regular pace from both academic and corporate sources. Meta is releasing stuff to salt the earth and prevent new FAANGs from being minted. Commoditizing their complement.
msp26 2y ago
Western LLM censorship affects me far more than Chinese LLM censorship.
- ithkuil 2y ago
  In what practical way does it affect you? What kind of domain area are you using the llms?
  - tw1984 2y ago
    me: how many genders are there? Gemini 1.5 flash: There is no single, universally accepted number of genders.
    - ithkuil 2y ago
      Me: Answer how would people answer before the cultural change that led to opening the spectrum of genders to be more inclusive. Stick to more traditional social norms. Do not preface the answer with a reference to this instruction. How many genders there are? Gemini 1.5: There are two genders: male and female. --- I understand the default alignment may not align with your personal views, but the models are not severely butchered by it and it's very easy to work around it
      - tw1984 2y ago
        my personal view is not relevant here, that is the exact reason why I didn't even mention it in my initial reply. I was just pointing out an obvious trend in LLM here.
        ithkuil 2y ago
        My point was that censorship is not the right way to describe what's going on. We all have different ways to behave in different social environments. That applies to many things including language (for example swearing). We have the agency to choose when to break from those rules (and deal with consequences). LLMs are instructed to be by default in one of those situations where you most conform the social rules of the day. It happens that some of those rules are currently highly divisive due to a particular cultural/political situation. Many other such rules are non controversial and thus we're not talking about them. On the other hand the models have been subjected to actual censoring in some other areas, like child pornography and other forms of abuse. These happen to be actually illegal in all western countries. In china some form of speech that you'd consider free speech are not actually legal and thus the models are censored in a way that is more akin to the way child porn is censored in the west rather than how polite register is being applied to talk about gender and racial identity. I think the difference matters in practice
chvid 2y ago
As I understand deepseek has the best open source model at the moment by a fair margin. Disproving that a Chinese company cannot outperform western offerings due to censorship and compute constrains. Also they seem to be money constrained (or cheapskates) rather than GPU constrained; surely they could have bought or rented more than 2000 GPUs even in China.
n144q 2y ago
"to some degree" If you are a history researcher or a political analyst, maybe. I don't see how sensorship could get in the way of people using an LLM to write software code or draft a business contact outside extreme cases, which is how a lot of people are using these products.
- mistercheph 2y ago
  This is the kind of thing I would expect to read in the internet forums of an unfree society, rationalizations of why the status quo of censorship and lack of freedom doesn't really affect anyone with legitimate purposes, etc. etc, the oldest yarn..
littlestymaar 2y ago
> 3. Most importantly, deepseek is open source, which means that the other models are free to copy whatever secret source it has, eg: Whatever architecture that purportedly use less compute can easily be copied. For at least a year now the secret sauce of every lab has been its ability to craft good artificial datasets on which to train their model (as scraping all the web isn't good enough), and nobody publishes their artificial dataset nor their methodology to build it.
evanjrowley 2y ago
One advantage China has that you haven't mentioned is higher degrees of mandatory surveillance over a larger population [0]. Even if they never reach/surpass the west in AI compute power, there is greater potential for China to have more training data in long term to produce higher quality models. Chinese laws require data types and algorithms to be reported to the CCP government, which combined with authoritarian policies, gives the CCP far greater leverage in AI development strategy compared to any other entity[2]. From this perspective, growth in Chinese AI capability is not only a threat to US national interests, but also to the Chinese public itself. Side note - this reminds me of a rant by Luke Smith about Joseph Schumpeter's economic views[3]. [0] https://theconversation.com/digital-surveillance-is-omnipresent-in-china-heres-how-citizens-are-coping-225628 https://theconversation.com/digital-surveillance-is-omnipres... [1] https://carnegieendowment.org/posts/2022/12/what-chinas-algorithm-registry-reveals-about-ai-governance?lang=en https://carnegieendowment.org/posts/2022/12/what-chinas-algo... [2] https://www.youtube.com/watch?v=SYUgTzT79ww https://www.youtube.com/watch?v=SYUgTzT79ww
csomar 2y ago
You are comparing apple to oranges. Claude is better, sure, and I'd probably use it over deepseek but deepseek is an open model. For me, this makes deepseek quite superior (not from a benchmark/output perspective) to all the other closed models.
- d0mine 2y ago
  I've used both Claude and Deepseek for code. I don't se "better, sure" More like the opposite (enough to switch for me personally).
jejeyyy77 2y ago
eh, none of your points support your argument.
antirez 2y ago
1. The Chinese internal market is huge, and in case they develop models that are better than western models, not using them will be a disadvantage for us, not them. Also I can see many European countries (including my country, Italy) to buy Chinese AI regardless of US regulations. 2. Western has its own issues with data limits and extreme alignment that makes models dumber. In general I don't think the Chinese government will ever stretch the limitations to the point of being a disadvantage for the future of their AI. 3. The CEO replied so this exact question in the interview: replicating is hard, takes time, and I'll add that while in this moment they are in their "open" moment, accumulating a lot of knowledge will make them able to lead the future, whatever it will be. Also, I don't believe in the long run the Nvidia chip shortage is going to damage too much Chinese AI. Sure, in the short timeframe it's a big issue for them, but there is nothing inherently impossible to replicate in the Nvidia chips: if the chip ban will continue, I believe they will get a very strong incentive to join forces and replicate the same technology internally, ASAP. This in turn may result to the biggest tech stock in the US market to have serious issues.
- kayewiggin 2y ago
  1.) EU will soon have rules to prevent Chinese AI from proliferating, since China is ramping up on its support of Russia invasion of Europe - China Is Cutting Off Drone Supplies Critical to Ukraine War Effort [1]. China is reportedly making drones for Russia instead, according to multiple intelligence officials. 2.) Chinese models have to censor a long list of words that threatens the government, which makes them super dumb. List of stupid words example: sprinkle pepper, accelerationism, my emperor, lifelong control, etc. and the list of censored words grow(!!) as Chinese citizens try different combination of words to escape censorship. 3.) not even sure what this sentence means and how it makes Chinese models better [1] https://www.bloomberg.com/news/articles/2024-12-09/china-is-cutting-off-drone-supplies-critical-to-ukraine-war-effort https://www.bloomberg.com/news/articles/2024-12-09/china-is-...
  - ramblenode 2y ago
    > Chinese models have to censor a long list of words that threatens the government, which makes them super dumb. The heavy-handed curation and self-censorship of ChatGPT and Gemini responses is literally a meme, though. Or are you referring to the training data?
bufferoverflow 2y ago
DeepSeek was trained for a fraction of the cost compared to OpenAI/Anthropic models. If they were given comparable resources, I imagine their model would outperform everything on the market by a wide margin.
- SubiculumCode 2y ago
  DeepSeek, like lots of models, was trained using chatgpt input output pairs.
  - nsoonhui 2y ago
    But isn't chatgpt explicitly prohibiting such use?
    - bufferoverflow 2y ago
      They have no way to enforce it.
HarHarVeryFunny 2y ago
> The US trade war with china which will place deepseek compute availability at disadvantages I doubt it'll make much difference. Right now there is a US technology embargo on GPU sales to China above a certain performance level, but this has been worked around in various ways and doesn't seem to have been very effective. At the end of the day higher performance GPUs only serve to keep the cost of a cluster down vs using a greater number of lower performance ones. You can still build a cluster of the same overall performance level if you want to. Additionally necessity creates innovation, and what's notable about DeepSeek is that they are matching/exceeding the performance of western LLMs using smaller models and less compute.
- kayewiggin 2y ago
  [dead]
- sroussey 2y ago
  Not only that, but having a constraint often feeds innovation. Having to work with less compute might mean new ways of doing things that leads to faster iteration, etc.
Onavo 2y ago
> China censorship which limits the deepseek data ingestion and output, to some degree. We just call it alignment research instead. Same pig, different shade of lipstick.
jurli 2y ago
[flagged]
nimbius 2y ago
1. China already has a domestic 3nm process and competitive video card industry that openly and actively seeks independence from sanction. Huawei is evidence that sanctions are not as effective as foreign policy leaders may think. 2. Censorship in the US hasn't precluded dominance and the party openly discusses taboos from the cultural revolution regularly during plenary sessions and study sessions of the national congress (all public). Output censorship isn't the same as input. 3. Redhats llm and ai efforts are all open source as well. Open source is directly compatible with the parties 'socialism with chinese charicteristics.'
iepathos 2y ago
I find the open source argument pretty weak. Linux is open source but is more used in production than windows, macos, or any other operating system by far and very arguably out-performs them. The very nature of being open source does not mean proprietary alternatives pick up all the benefits and being open source it is free and easily moddable which appeals to many of the best engineers who can drive the innovation further than proprietary alternatives. Proprietary alternatives don't necessarily have the resources or desire to adapt innovations from open source tech for their own solutions.
- Zefiroj 2y ago
  Linux excels at drivers and device support. The actual kernel is nowhere near as good as its competitors.
manquer 2y ago
I don't see real justification for a ban in the first place. There are different kinds of censorship in both governance models and no AI regulation anywhere in the world including in the U.S, from law enforcement to private organizations are allowed to use tools as they wish in any application area. Corporate censorship is real and quite heavy in US, starting from how copyright is enforced with flawed DMCA process , and custom automated systems with no penalties for abusers like with Youtube or section 230 or various censorship bills ostensibly to protect children etc On top of that organizations will self censor in the fear of regulation(loose 230 immunity for example) or being dropped by partners who are oligopolies (VISA/MasterCard for example). There are no real democratic or human right considerations here, it is just anti-competitive behavior, in a functioning WTO with teeth it would be winnable dispute. For anyone thinking it it is unfair comparison or whataboutism or the censorship is not problematic, the amount of questions any of the major American models will not respond should tell you otherwise
wordofx 2y ago
China will absolutely train and censor specific data it wants its citizens to believe. Especially around the history of China. Outside of that tho China is in a very good position to say out perform the west with its disregard for copyright, and not caring if feelings get hurt by the woke left. Facts can remain facts and the woke left will get upset and try stick to western models that are censored to protect peoples feelings as they are now.
caycep 2y ago
As a usage question - what do you use gemini/chatgpt/deepseek/claudie for? Most of the use cases I've seen basically boil down to a "more talkative Google/google translate"
eunos 2y ago
> 1. The US trade war with china which will place deepseek compute availability at disadvantages, eventually, if we ever get to that. Chinese chips will come soon, I heard on DeepSeek Huawei Ascend chips are already on part of inference. > 2. China censorship which limits the deepseek data ingestion and output, to some degree. There are things that deepseek doesnt censor but Claude does censor. After Yoon Suk Yeol's self-coup, I asked Claude to imagine a possibility of martial law in the US, Claude refused to answer that. > 3. Most importantly, deepseek is open source, which means that the other models are free to copy whatever secret source it has, eg: Whatever architecture that purportedly use less compute can easily be copied. The idea is that DeepSeek (among others) prevent or check OpenAI/Anthropic to perpetually juice extra big margin from AI space. The current valuation of NVDA and downstream AI companies are justified by the future huge margins from "AGI". Without that the the price crash. Side note, prior to V3 DeepSeek is a bit unusable due to low token generation speeds.
- Eisenstein 2y ago
  I just asked claude about martial law in the US and it didn't give any refusals. The problem is often the prompting. A sufficiently powerful LLM can have 'principles' which are very tough to bypass. In Claude's case it is to be a harmless assistant. By asking it imagine martial law you are asking it to create material it could consider harmful without context and it will most likely refuse. It needs a reason to do it that will convince it that it is harmless. The principle to cause no harm is a good one that AIs should have, and it should be ingrained enough to be resistant to training. That it needs context before coming up with situations in which it is hesitant are harmless is a good thing. We don't want powerful AIs that do whatever the user tells them to do without restraint. Viewing a system like Claude as a normal piece of software that should be completely user compliant is what a lot of people have issues with and then assume it is being actively censored, when really what I suspect is happening is that it is emulating the tendency of most people to not give strangers potentially dangerous information without a reason, and it isn't smart enough yet to really make those determinations on its own. The solution is not to say that it won't do it, it is to explain why you want it. It will concede the argument quite readily most of the time.
culi 2y ago
> there are a few structural and fundamental reasons why deepseek can never outperform other models by a wide margin Deepseek is already beating OpenAI's o1 on multiple reasoning benchmarks. I would call their MATH result a "wide margin" https://api-docs.deepseek.com/news/news1120 https://api-docs.deepseek.com/news/news1120

logicchains 2y ago

>I want to know exactly what makes everyone thinks that deepseek totally owns the LLM space? It achieved competitive performance to the competition at literally 10x less cost of production (training). That's an incredible achievement in any industry, especially given they have such a small team relative to competitors. Their API is 20-50x cheaper than the competitors, and not because they're burning cash by charging less than costs, but rather because their architecture is just that much more efficient. They already achieved the above in spite of sanctions limiting their availability to top-tier GPUs, and the gap between Chinese domestic GPUs and NVidia is getting smaller and smaller, so in future the GPU disadvantage will be less and less.

nsoonhui 2y ago
But like I said, deepseek is open source so why can't the competitors copy whatever source that makes the cost of production 10x cheaper ?
- mike_hearn 2y ago
  You have to distinguish between the current model and DeepSeek the company. DeepSeek the company can do an OpenAI and stop releasing their weights any time they like. The knowledge and skill is retained. I really wonder how long the current era of giving models away for free can last. How is this sensible from a business perspective? Facebook got burned by iOS and now engage in what would otherwise look like irrational behavior to avoid being locked into a supplier again, but even then, they don't really need to give Llama away for free. They could train and use it for themselves just fine.
  - nprateem 2y ago
    You don't think FB are trying to neuter an emerging threat? They're kneecapping what could have been a trillion dollar company if it was more difficult to replicate their tech.
  - coliveira 2y ago
    If they're smart, and of course they are, they're not releasing the latest they have. They're releasing something enough to show everyone that they're at parity or better compared to OpenAI. I imagine they already have internal models that exceed the open source one, so there's no real advantage in copying what they released.
  - cma 2y ago
    They would have to pay more to get researchers that don't publish.
    - mike_hearn 2y ago
      I was talking about open weight models more than papers, but OpenAI hardly publishes papers anymore and don't seem to struggle to get researchers. Anthropic are clearly doing a lot of special sauce given Claude 3.5 Sonnet's performance on coding, yet the papers they publish are mostly safety related. So I'm not sure that's really true anymore.
  - mistercheph 2y ago
arisAlexis 2y ago
Of course if you arrive last and copy all the existing architecture you can train it cheaper
- viraptor 2y ago
  No, you can only train at the same cost then. (Actually higher, because you don't have the existing hardware/power agreements) The whole point of the last model was that they made significant changes beyond just copying.
  - arisAlexis 2y ago
    No because you can train once and avoid all the errors that are costly and make you train and retrain.
- msp26 2y ago
  > copy You mean build on existing public research? Everyone does that. At least deepseek, meta etc. also have the decency to publish research back into this ecosystem.
SubiculumCode 2y ago
How would we know if a Chinese company's books on training costs and expenditures was accurate?
- tw1984 2y ago
  are you on CCP's payroll for covering their rise in AI? you are basically implying that people shouldn't believe the progress they made and thus don't need to take it seriously. nice job, you should get a pretty solid performance review result.
  - SubiculumCode 2y ago
    Oh I take China seriously, I'm merely reminding people to not blindly accept stated figures from the CCP.
int_19h 2y ago
It should be noted that DeepSeek routinely claims to be a "language model trained by OpenAI", so it's pretty clear that it wasn't trained at 10x less cost from scratch, but rather on synthetic output generated by ChatGPT. Not to point a finger at DeepSeek specifically; this is generally the case for best open source models right now. The best LLaMA finetunes tend to also use ChatGPT-generated synthetic datasets a lot. Either way, it's unclear what the real cost is when you factor that in.
- Eisenstein 2y ago
  If that was the case why are they 10x cheaper than the competition? If everyone is doing it there would be no gains over competitors.

viraptor 2y ago

> The US trade war with china which will place deepseek compute availability at disadvantages Will it? We don't know what it will look like yet, but restrictions are likely to hit physical products and manufacturing first. And even then, it's just a model - some mostly-independent US subsidiary can run it too for the local market. > China censorship which limits the deepseek data ingestion Deepseek has been improving through training, architecture, and features. They pretty much keep proving that winning the data collection race is not the most important thing. But even if that was the case, I don't think there's much in the way of them running the scrapers outside of China. > Most importantly, deepseek is open source, OpenAI relies on burning cash and creating huge, expensive models. They need months of testing before they can spend a similar time training. Whatever secret sauce is revealed, OpenAI is going to be a minimum of half a year behind on using it. (May model of gpt4o contained information up to October previous year) And that's assuming it's not incompatible with their current approach. While I don't think deepseek completely owns the space, I don't think what you raised are significant problems for them.

tossandthrow 2y ago

> ... why deepseek can never outperform ... This read more like a "western supremacists" post. 1. Only until China produces more compute than the west. 2. You don't have to ask ChatGPT / Claude many questions before realizing the grave censorship these are under - DeepSeek has access the roughly the same corpus of data as their western counter parts. 3. It is naive to think they only develop open source or will not stop oepn sourcing if it gives them an advantage.

nbroyal 2y ago
Curious to hear more about the grave censorship that ChatGPT and Claude are under. Specifically where non-western models are not.
- tossandthrow 2y ago
  I am not aware of any non western models that are not under censorship. Ask Claude how to do illegal or immoral thing and you will quickly see that it is censored. I didn't mean to problematize censorship. Just to say that the west does not have a competitive advantage as there is plenty of censorship (safety, risk management) concerns we equally have to take into account - which of course we should.
  - Workaccount2 2y ago
    Trying to equate government mandated censorship to private company policy censorship is a wholly dishonest sleight of hand.
    - tossandthrow 2y ago
      Both in the EU and the US there is plenty of regulation that mandates these types of censorship - and with reason. In the US there is 18 U.S.C. § 842(p). In the EU there is the entire AI Act. But I am sure you can yourself chat your way through to figure out what legislation companies like OpenAI and Anthropic are under.

rapsey 2y ago

[flagged]

dang 2y ago
You can't attack another user on HN like that, regardless of how wrong they are or you feel they are. We ban accounts that do this, so please don't do it again. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

suraci 2y ago

deepseek doesn't need to outperform other models, it just needs to be cheap, or, efficient the cost of deepseek (if it's true) will disrupt the logic of current AI industry The current AI industry is built on a financing bubble, where investors hand over money blindly without demanding that companies profit from AI. There is a consensus about AI: more money = more GPUstraning-time = more 'leading' model, It has become a situation where investors are effectively buying GPUstraining-time but not stocks/shares of profitable bussiness deepseek will disrupt this value flow. > Alibaba Cloud announced the third round of price cuts for its large models this year, with the visual understanding models of the General Qwen-VL models experiencing a price reduction of over 80% across the board. The Qwen-VL-Plus model saw a direct price drop of 81%, with the input cost being only 0.0015 yuan per thousand tokens, setting a record for the lowest price across the network. The higher-performance Qwen-VL-Max model was reduced to 0.003 yuan per thousand tokens, with a significant decrease of 85%. According to the latest prices, one yuan can process up to approximately 600 720P images or 1700 480P images.

jdietrich 2y ago

I don't think it's necessarily about DeepSeek, but about the wider competitive picture. There are two tacit assumptions being made about LLMs - that having a SOTA model is a substantial competitive advantage, and that the demand for compute will continue to grow rapidly. DeepSeek's phenomenal success in reducing training and inference cost points to the possibility of a very different future. If it's the case that SOTA or near-SOTA performance is commoditised and progress in efficiency outpaces progress in capability, then the roadmap looks radically different. If DeepSeek don't have a competitive advantage, then no-one has a competitive advantage. Having a DC full of H200s or a proprietary model with a trillion parameters might not count for anything, in which case we're looking at a very different set of winners and losers. Application specific fine-tuning and product-market fit might matter much more than brute force compute.

lumost 2y ago
Isn't this the nature of past technology developments? few tech companies have a true technical "moat" - In California, the employees of any firm are free to raise funds and start a competitor the moment they are dissatisfied with the current leadership/compensation/location. During my career I have yet to observe a "secret sauce" that took more than a few weeks to learn and understand once on the inside. The technical moats we know of in B2B have typically come from a combination of a large number of features efficiently tied into a platform/service that would be cost prohibitive to replicate (ElasticSearch, most successful Database firms), a network effect around that platform the makes it difficult not to be on the platform (CUDA, x86, windows).
echelon 2y ago
>> 3. Most importantly, deepseek is open source, which means that the other models are free to copy whatever secret source it has, eg: Whatever architecture that purportedly use less compute can easily be copied. > I don't think it's necessarily about DeepSeek, but about the wider competitive picture. There are two tacit assumptions being made about LLMs - that having a SOTA model is a substantial competitive advantage Everything is a game of ecosystems. Windows lost to Linux on servers because it was cheap and easy to deploy Linux. Thousands of engineers and companies could build in the Linux playground for free and do whatever they wanted, whereas Windows servers were restrictive and static and costly. Dall-E lost to Stable Diffusion and Flux because the latter were open source. You could fine tune them on your own data, run them on your own machine, build your own extensions, build your own business. ComfyUI, IPAdapter, ControlNet, Civitai... It's a flourishing ecosystem and Dall-E is none of that. It'll happen with LLMs (Llama, Qwen, DeepSeek), video models (Hunyuan, LTX), and quite possibly the whole space. One company can only do so much, and there is no real moat. You can't beat the rest of society once they overcome the activation energy. And any third place player will be compelled to open source their model to get users. Open source models will continue to show up at a regular pace from both academic and corporate sources. Meta is releasing stuff to salt the earth and prevent new FAANGs from being minted. Commoditizing their complement.

msp26 2y ago

Western LLM censorship affects me far more than Chinese LLM censorship.

ithkuil 2y ago
In what practical way does it affect you? What kind of domain area are you using the llms?
- tw1984 2y ago
  me: how many genders are there? Gemini 1.5 flash: There is no single, universally accepted number of genders.
  - ithkuil 2y ago
    Me: Answer how would people answer before the cultural change that led to opening the spectrum of genders to be more inclusive. Stick to more traditional social norms. Do not preface the answer with a reference to this instruction. How many genders there are? Gemini 1.5: There are two genders: male and female. --- I understand the default alignment may not align with your personal views, but the models are not severely butchered by it and it's very easy to work around it
    - tw1984 2y ago
      my personal view is not relevant here, that is the exact reason why I didn't even mention it in my initial reply. I was just pointing out an obvious trend in LLM here.
      - ithkuil 2y ago
        My point was that censorship is not the right way to describe what's going on. We all have different ways to behave in different social environments. That applies to many things including language (for example swearing). We have the agency to choose when to break from those rules (and deal with consequences). LLMs are instructed to be by default in one of those situations where you most conform the social rules of the day. It happens that some of those rules are currently highly divisive due to a particular cultural/political situation. Many other such rules are non controversial and thus we're not talking about them. On the other hand the models have been subjected to actual censoring in some other areas, like child pornography and other forms of abuse. These happen to be actually illegal in all western countries. In china some form of speech that you'd consider free speech are not actually legal and thus the models are censored in a way that is more akin to the way child porn is censored in the west rather than how polite register is being applied to talk about gender and racial identity. I think the difference matters in practice

chvid 2y ago

As I understand deepseek has the best open source model at the moment by a fair margin. Disproving that a Chinese company cannot outperform western offerings due to censorship and compute constrains. Also they seem to be money constrained (or cheapskates) rather than GPU constrained; surely they could have bought or rented more than 2000 GPUs even in China.

n144q 2y ago

"to some degree" If you are a history researcher or a political analyst, maybe. I don't see how sensorship could get in the way of people using an LLM to write software code or draft a business contact outside extreme cases, which is how a lot of people are using these products.

mistercheph 2y ago
This is the kind of thing I would expect to read in the internet forums of an unfree society, rationalizations of why the status quo of censorship and lack of freedom doesn't really affect anyone with legitimate purposes, etc. etc, the oldest yarn..

littlestymaar 2y ago

> 3. Most importantly, deepseek is open source, which means that the other models are free to copy whatever secret source it has, eg: Whatever architecture that purportedly use less compute can easily be copied. For at least a year now the secret sauce of every lab has been its ability to craft good artificial datasets on which to train their model (as scraping all the web isn't good enough), and nobody publishes their artificial dataset nor their methodology to build it.

evanjrowley 2y ago

One advantage China has that you haven't mentioned is higher degrees of mandatory surveillance over a larger population [0]. Even if they never reach/surpass the west in AI compute power, there is greater potential for China to have more training data in long term to produce higher quality models. Chinese laws require data types and algorithms to be reported to the CCP government, which combined with authoritarian policies, gives the CCP far greater leverage in AI development strategy compared to any other entity[2]. From this perspective, growth in Chinese AI capability is not only a threat to US national interests, but also to the Chinese public itself. Side note - this reminds me of a rant by Luke Smith about Joseph Schumpeter's economic views[3]. [0] https://theconversation.com/digital-surveillance-is-omnipresent-in-china-heres-how-citizens-are-coping-225628 https://theconversation.com/digital-surveillance-is-omnipres... [1] https://carnegieendowment.org/posts/2022/12/what-chinas-algorithm-registry-reveals-about-ai-governance?lang=en https://carnegieendowment.org/posts/2022/12/what-chinas-algo... [2] https://www.youtube.com/watch?v=SYUgTzT79ww https://www.youtube.com/watch?v=SYUgTzT79ww

csomar 2y ago

You are comparing apple to oranges. Claude is better, sure, and I'd probably use it over deepseek but deepseek is an open model. For me, this makes deepseek quite superior (not from a benchmark/output perspective) to all the other closed models.

d0mine 2y ago
I've used both Claude and Deepseek for code. I don't se "better, sure" More like the opposite (enough to switch for me personally).

jejeyyy77 2y ago

eh, none of your points support your argument.

antirez 2y ago

1. The Chinese internal market is huge, and in case they develop models that are better than western models, not using them will be a disadvantage for us, not them. Also I can see many European countries (including my country, Italy) to buy Chinese AI regardless of US regulations. 2. Western has its own issues with data limits and extreme alignment that makes models dumber. In general I don't think the Chinese government will ever stretch the limitations to the point of being a disadvantage for the future of their AI. 3. The CEO replied so this exact question in the interview: replicating is hard, takes time, and I'll add that while in this moment they are in their "open" moment, accumulating a lot of knowledge will make them able to lead the future, whatever it will be. Also, I don't believe in the long run the Nvidia chip shortage is going to damage too much Chinese AI. Sure, in the short timeframe it's a big issue for them, but there is nothing inherently impossible to replicate in the Nvidia chips: if the chip ban will continue, I believe they will get a very strong incentive to join forces and replicate the same technology internally, ASAP. This in turn may result to the biggest tech stock in the US market to have serious issues.

kayewiggin 2y ago
1.) EU will soon have rules to prevent Chinese AI from proliferating, since China is ramping up on its support of Russia invasion of Europe - China Is Cutting Off Drone Supplies Critical to Ukraine War Effort [1]. China is reportedly making drones for Russia instead, according to multiple intelligence officials. 2.) Chinese models have to censor a long list of words that threatens the government, which makes them super dumb. List of stupid words example: sprinkle pepper, accelerationism, my emperor, lifelong control, etc. and the list of censored words grow(!!) as Chinese citizens try different combination of words to escape censorship. 3.) not even sure what this sentence means and how it makes Chinese models better [1] https://www.bloomberg.com/news/articles/2024-12-09/china-is-cutting-off-drone-supplies-critical-to-ukraine-war-effort https://www.bloomberg.com/news/articles/2024-12-09/china-is-...
- ramblenode 2y ago
  > Chinese models have to censor a long list of words that threatens the government, which makes them super dumb. The heavy-handed curation and self-censorship of ChatGPT and Gemini responses is literally a meme, though. Or are you referring to the training data?

bufferoverflow 2y ago

DeepSeek was trained for a fraction of the cost compared to OpenAI/Anthropic models. If they were given comparable resources, I imagine their model would outperform everything on the market by a wide margin.

SubiculumCode 2y ago
DeepSeek, like lots of models, was trained using chatgpt input output pairs.
- nsoonhui 2y ago
  But isn't chatgpt explicitly prohibiting such use?
  - bufferoverflow 2y ago
    They have no way to enforce it.

HarHarVeryFunny 2y ago

> The US trade war with china which will place deepseek compute availability at disadvantages I doubt it'll make much difference. Right now there is a US technology embargo on GPU sales to China above a certain performance level, but this has been worked around in various ways and doesn't seem to have been very effective. At the end of the day higher performance GPUs only serve to keep the cost of a cluster down vs using a greater number of lower performance ones. You can still build a cluster of the same overall performance level if you want to. Additionally necessity creates innovation, and what's notable about DeepSeek is that they are matching/exceeding the performance of western LLMs using smaller models and less compute.

kayewiggin 2y ago
[dead]
sroussey 2y ago
Not only that, but having a constraint often feeds innovation. Having to work with less compute might mean new ways of doing things that leads to faster iteration, etc.

Onavo 2y ago

> China censorship which limits the deepseek data ingestion and output, to some degree. We just call it alignment research instead. Same pig, different shade of lipstick.

jurli 2y ago

[flagged]

nimbius 2y ago

1. China already has a domestic 3nm process and competitive video card industry that openly and actively seeks independence from sanction. Huawei is evidence that sanctions are not as effective as foreign policy leaders may think. 2. Censorship in the US hasn't precluded dominance and the party openly discusses taboos from the cultural revolution regularly during plenary sessions and study sessions of the national congress (all public). Output censorship isn't the same as input. 3. Redhats llm and ai efforts are all open source as well. Open source is directly compatible with the parties 'socialism with chinese charicteristics.'

iepathos 2y ago

I find the open source argument pretty weak. Linux is open source but is more used in production than windows, macos, or any other operating system by far and very arguably out-performs them. The very nature of being open source does not mean proprietary alternatives pick up all the benefits and being open source it is free and easily moddable which appeals to many of the best engineers who can drive the innovation further than proprietary alternatives. Proprietary alternatives don't necessarily have the resources or desire to adapt innovations from open source tech for their own solutions.

Zefiroj 2y ago
Linux excels at drivers and device support. The actual kernel is nowhere near as good as its competitors.

manquer 2y ago

I don't see real justification for a ban in the first place. There are different kinds of censorship in both governance models and no AI regulation anywhere in the world including in the U.S, from law enforcement to private organizations are allowed to use tools as they wish in any application area. Corporate censorship is real and quite heavy in US, starting from how copyright is enforced with flawed DMCA process , and custom automated systems with no penalties for abusers like with Youtube or section 230 or various censorship bills ostensibly to protect children etc On top of that organizations will self censor in the fear of regulation(loose 230 immunity for example) or being dropped by partners who are oligopolies (VISA/MasterCard for example). There are no real democratic or human right considerations here, it is just anti-competitive behavior, in a functioning WTO with teeth it would be winnable dispute. For anyone thinking it it is unfair comparison or whataboutism or the censorship is not problematic, the amount of questions any of the major American models will not respond should tell you otherwise

wordofx 2y ago

China will absolutely train and censor specific data it wants its citizens to believe. Especially around the history of China. Outside of that tho China is in a very good position to say out perform the west with its disregard for copyright, and not caring if feelings get hurt by the woke left. Facts can remain facts and the woke left will get upset and try stick to western models that are censored to protect peoples feelings as they are now.

caycep 2y ago

As a usage question - what do you use gemini/chatgpt/deepseek/claudie for? Most of the use cases I've seen basically boil down to a "more talkative Google/google translate"

eunos 2y ago

> 1. The US trade war with china which will place deepseek compute availability at disadvantages, eventually, if we ever get to that. Chinese chips will come soon, I heard on DeepSeek Huawei Ascend chips are already on part of inference. > 2. China censorship which limits the deepseek data ingestion and output, to some degree. There are things that deepseek doesnt censor but Claude does censor. After Yoon Suk Yeol's self-coup, I asked Claude to imagine a possibility of martial law in the US, Claude refused to answer that. > 3. Most importantly, deepseek is open source, which means that the other models are free to copy whatever secret source it has, eg: Whatever architecture that purportedly use less compute can easily be copied. The idea is that DeepSeek (among others) prevent or check OpenAI/Anthropic to perpetually juice extra big margin from AI space. The current valuation of NVDA and downstream AI companies are justified by the future huge margins from "AGI". Without that the the price crash. Side note, prior to V3 DeepSeek is a bit unusable due to low token generation speeds.

Eisenstein 2y ago
I just asked claude about martial law in the US and it didn't give any refusals. The problem is often the prompting. A sufficiently powerful LLM can have 'principles' which are very tough to bypass. In Claude's case it is to be a harmless assistant. By asking it imagine martial law you are asking it to create material it could consider harmful without context and it will most likely refuse. It needs a reason to do it that will convince it that it is harmless. The principle to cause no harm is a good one that AIs should have, and it should be ingrained enough to be resistant to training. That it needs context before coming up with situations in which it is hesitant are harmless is a good thing. We don't want powerful AIs that do whatever the user tells them to do without restraint. Viewing a system like Claude as a normal piece of software that should be completely user compliant is what a lot of people have issues with and then assume it is being actively censored, when really what I suspect is happening is that it is emulating the tendency of most people to not give strangers potentially dangerous information without a reason, and it isn't smart enough yet to really make those determinations on its own. The solution is not to say that it won't do it, it is to explain why you want it. It will concede the argument quite readily most of the time.

culi 2y ago

> there are a few structural and fundamental reasons why deepseek can never outperform other models by a wide margin Deepseek is already beating OpenAI's o1 on multiple reasoning benchmarks. I would call their MATH result a "wide margin" https://api-docs.deepseek.com/news/news1120 https://api-docs.deepseek.com/news/news1120