6 ms·
That seems like a lot of rationalization to me. China is pursuing these because they cannot compete on the frontier. Yes, there is a possibility that all that c
by SubiculumCode 5mo ago
That seems like a lot of rationalization to me. China is pursuing these because they cannot compete on the frontier. Yes, there is a possibility that all that compute is not needed, but it is a rather remote possibility, and there is no doubt that, given the choice, China would be pursuing frontier model building with closed, propietary-only offerings.
- cyberge99 5mo agoForgive me if this is a naive assumption, but wouldn’t large language models be fundamentally different for a language that is largely symbols? Again, my understanding of Mandarin is limited if it exists at all.
- doph 5mo agoAll tokens are symbols. All of the frontier models speak Mandarin.
- boothby 5mo agoThis is why misspellings and homophones are tells of human righting. LLMs strongly prefer word-level tokens, and word substitutions follow semantic similarity and not the more human auditory similarity.
- mejutoco 5mo ago> righting. > LLMs strongly prefer word-level tokens, and word substitutions follow semantic similarity and not the more human auditory similarity. Is this an elaborate joke or your full-word misspelling of writing is both agreeing with your statement (word substitutions) and contradicting it (not semantic but only pronunciation similarity)
- calfuris 5mo agoI don't see the contradiction, unless you believe that the grandparent comment was written by an LLM.
- deleted 5mo ago[deleted]
- omneity 5mo agoFunny, I’ve been cracking[0] at this exact problem with a purpose-built model[1]: 0: https://huggingface.co/posts/omarkamali/593639295164067 https://huggingface.co/posts/omarkamali/593639295164067 1: https://omneitylabs.com/models/sawtone https://omneitylabs.com/models/sawtone
- jddj 5mo agoClaude the other day wrote code where one of the bytes in the array was 0xO5. That's zero ex oh (the letter) five
- wat10000 5mo ago"飞机" and "airplane" aren't fundamentally different in terms of how they're represented to a computer. Especially for an LLM, where tokenization likely turns each of those into a single token.
- throwaway27448 5mo ago> China is pursuing these because they cannot compete on the frontier. ? Claude, ChatGPT, etc are heinously expensive for tiny benefits lmao. Local + efficient is clearly the future
- ToucanLoucan 5mo agoAI boosters cling to this notion because it's the only way the massive data center buildouts make any sense at all. I guess you could say the US is winning the frontier AI race. Okay. I'm never going to grant a cloud service access to all the contents of my hard drive, that's just never going to happen, so if you expect me and a lot of people like me who feel similarly to get on this train, you better have a local, lightweight model too or we're not even having a discussion, the answer is just no.
- Our_Benefactors 5mo agoThe thing is, frontier model providers don’t take your feelings into account even a little bit. It’s totally irrelevant to the discussion about the service they can provide, because that service is predicated on access to high power GPU slices that local models can’t touch. Those providers won’t be in an existential crisis because some people choose the privacy route, it’s a cost of doing business.
- ToucanLoucan 5mo agoRight but that service being sold is predicated on products being sold to users, yes? Or are we still pretending that the hyperscalers can just pass the same $20 billion between themselves and that's going to be a growth industry forever?
- ElevenLathe 5mo agoI suppose its possible that all the value to pay back the datacenter construction can be squeezed out of enterprise contracts where your employer can assent on the privacy questions, probably with some kind of complicated contract and insurance regime regulating things. Even if so, if China is coming behind 6 months later selling laptops with hyper-efficient local models that are 80% as good as "frontier" ones, I imagine they'll get the consumer business AND a fair share of the enterprise business as IT managers look at their options during the next refresh cycle. Given economies of scale, I think it's ultimately inevitable that the enterprise more-or-less follows the consumer on this, and the consumer is going to prefer local models. There's no ongoing cost after the initial purchase, and your data at least nominally stays within your control.
- YetAnotherNick 5mo agoWell China is consistently 6 months behind the frontier labs(possibly because they can they harvest data from released frontier models). If the scaling continues, US will win, but if not then China will win as the models will converge.
- SubiculumCode 5mo agoThe non-release of Mythos tell you the future of that, so long as they can keep the weights from being exfiltrated. Once models become true national security threats, they won't be released in their full form. The hitch-a-ride approach becomes less capable of keeping up.
- nradov 5mo agoHow would they prevent distillation? That would seem pretty tough to block for any LLM available for commercial use.
- SubiculumCode 5mo agoBy only providing degraded models to use commercially outside national defense applications would be my guess. As soon as models are a threat in terms of enabling biowarfare etc, then they just are not going to be generally released. Honestly, I think its quite possible that models will be retrained with gaps in their knowledge. e.g. a coding model for commercial use probably doesn't need to have deep knowledge of biology, and training on biological sciences probably doesn't help those evals much. Honestly, I'd welcome such an approach.
- nradov 5mo agoWhat a hilariously uninformed comment. LLMs are not the limiting factor in biowarfare.
- SubiculumCode 5mo ago
- Matl 5mo agoI dunno, DeepSeek v4 Pro is rather on par as far as I can tell, maybe not with 5.5 Pro in all areas quite yet, but close. I think China is thinking more about the application layer on top of models as going to matter more than the models themselves, so they don't need to gatekeep the models as much.
- nradov 5mo agoAll that compute is not needed. We have an existence proof from biology in the form of natural intelligence that much greater efficiency is possible. However, achieving dramatic improvements in compute efficiency will depend on unpredictable scientific breakthroughs. Personally I suspect that an entirely new hardware architecture will be needed, although I don't have any hard evidence to back that up.
- ribosometronome 5mo ago>from biology ... much greater efficiency is possible Those are much more specialized models with pretty mediocre tokens per second.
- Hasslequest 5mo ago[dead]
- pwndByDeath 5mo agoPerhaps tokens is a dead end?
- SpicyLemonZest 5mo agoPerhaps! But perhaps whatever human brains use instead of tokens is not as amenable to scaling or copying.
- logicchains 5mo ago>We have an existence proof from biology in the form of natural intelligence that much greater efficiency is possible. It's only a proof that it's possible with 18+ years of training.
- nradov 5mo agoIn certain ways my dog has more generalized intelligence than any LLM, and I trained her in only a few months with a modest investment in dog treats.
- seanmcdirmid 5mo agoChina is competing in value AI because they cannot work at the frontier, but how is this bad at all? It’s like how the USA has the best drones but they are a few million dollars apiece while China has DJI. If China could work at the frontier, I don’t know, I kind of think they would still be dumping a lot of resources into exploring the value side since they have that culture already in place.
- SubiculumCode 5mo agoI did not imply it was bad. I implied that competing in value AI is the only option that China-based AI companies have due to limitations in compute.
- seanmcdirmid 5mo agoThis is true, but I don't think they would all be rushing to frontier if that option was available. Chinese are used to working with constraints to their benefit, they would see the price of working at frontier and make hard choices that maybe we can ignore in the states.