9 ms·
Tencent's 'Hunyuan-T1'–The First Mamba-Powered Ultra-Large Model
- FirmwareBurner 2y ago[flagged]
- ohgr 2y agoEU: Actual Intelligence...
- FirmwareBurner 2y agoAre you saying creating several world class LLMs doesn't require "actual intelligence", whatever that means to you?
- nurettin 2y agoMaybe to them, training LLMs isn't an intelligent thing to do and the world is better without.
- ohgr 2y agoI’m quite happy for people to waste time and money on an economic dead end. Just not Europe. We can do better while everyone is ruining their society with it.
- ohgr 2y agoA world class LLM is sort of like saying you stuck a dildo on a donkey’s head and you’ve got a unicorn to sell. It’s a donkey mate. The problem is that most people can’t tell the difference between a donkey and a unicorn because everyone was told everything was a unicorn for the last 20 years.
- perching_aix 2y agoI don't understand why people are so hung up on the plastic caps thing. On the milk cartons we have, I outright prefer the new designs even. On that note, may I ask if there's any point to your comment other than to incite people?
- FirmwareBurner 2y ago[flagged]
- perching_aix 2y ago> The union assumed I'm too stupid (...) I don't think the union thinks about you specifically. > needing to vacuum the same area more times over to achieve the same cleanliness resulting in even more power used than before While this makes intuitive sense to me, is this what actually happens though? > Or just like the EUs promotion for (...) Remind me, why are you gish-galloping to other random things you find to be crappy about the EU? Again, is there any other point to your comments than to incite?
- inexcf 2y agoThe amount of random plastic bottle caps in nature proofs that you can not be trusted to screw the cap back on. And if anything the limit on vacuums forced them to innovate. My Vacuum today is far more effective than my 2000 watt vacuum was in the past.
- blensor 2y agoI always find the caps hate so funny too. I mean it's not something that the world truly needed but I don't have a problem with it and it comes in quite handy when drinking water while driving
- FirmwareBurner 2y agoBut why did they need to spend taxpayer time and money to fuck with something that wasn't broken? Wouldn't it be better to spend that taxpayer time and money of issues of graver importance.
- robotresearcher 2y ago[flagged]
- FirmwareBurner 2y agoMaybe the model is tuned for Chinese language queries and not English ones? It is a Chinese one after all so it would make sense they cater to their domestic market first, no?
- Mizza 2y agoYeah, you can try it out here: https://llm.hunyuan.tencent.com/#/chat/hy-t1 https://llm.hunyuan.tencent.com/#/chat/hy-t1 It takes a bit of nudging to get it to switch to English, but it's fast and accurate after that.
- FirmwareBurner 2y agoI managed to get it to say "Taiwan is a country" without much stress
- numpad0 2y agoOr just written by a Chinese speaker. The text do sound human, no? China/Japan/Korea do everything from pencils to rockets in local languages. You can buy a quantum physics textbook in Chinese on Amazon, if you want. English is truly just a tool in Far East region, and so individual proficiency of a Far Eastern person and their English speaking skills do not correlate well if not negatively correlating. e: asked Hunyuan-T1 itself[0] on the demo page[1], for fun. Its conclusion was "最可能的情形是：具备一定技术背景的中文母语者，在有限时间内完成的初稿(GT: "The most likely scenario is that a native Chinese speaker with a certain technical background completes the first draft within a limited time")". 0: https://gist.github.com/numpad0/7699db43ae23f054dc2db56730114959 https://gist.github.com/numpad0/7699db43ae23f054dc2db5673011... 1: https://llm.hunyuan.tencent.com/#/chat/hy-t1 https://llm.hunyuan.tencent.com/#/chat/hy-t1
- logicchains 2y agoIt's absolutely human-written Chinglish; any recent LLM can write much more idiomatic English than that.
- chis 2y agoKobe?
- nixpulvis 2y agoSome of the text is cut off while reading on my phone. Embarrassing.
- drysine 2y agoDon't be so harsh on your phone)
- timcobb 2y ago[flagged]
- pkkkzip 2y agothanks for sharing did you contact tencent support ?
- jrflowers 2y agoWhy are you embarrassed? You can always put your phone down and read it on desktop later
- brookst 2y agoJust seems odd not to test a website on a phone, even accidentally.
- notShabu 2y agoThe romanization of these names is always confusing b/c stripped of the character and tone it's just gibberish. "Hunyuan" or 混元 in chinese means "Primordial Chaos" or "Original Unity". This helps as more chinese products and services hit the market and makes it easier to remember. The naming is similar to the popularity of greek mythology in western products. (e.g. all the products named "Apollo")
- klabb3 2y ago> The naming is similar to the popularity of greek mythology in western products. (e.g. all the products named "Apollo") Popular? So you’re saying that all the VPs who have come up with the mind bendingly unique and creative name Prometheus didn’t do so out of level 10 vision?
- Y_Y 2y agoI think it's particularly egregious that they use such a lossy encoding. I can't read the hanzi, but at least "Hùn yuán" would have been more helpful, or even "Hu4n yua1n" would have enabled me to pronounce it or look it up without having the context to guess which characters it was representing.
- realusername 2y agoI don't understand why this vietnamese-style writing isn't the most popular pinyin. It's clearly superior to putting numbers inside words.
- currymj 2y agoTone markers are of limited use to Chinese readers (instead, just show them the characters). They are also of limited use to non-Chinese readers, who don't understand the tone system and probably can't even audibly distinguish tones. So, it makes sense that we get this weird system even though it's strictly worse.
- powerapple 2y agoYes, this is very annoying, because how Pinyin works. There were a lot mistakes made when using Pinyin in English content. Pinyin suppose to break at character level, Pinyin = Pin Yin, you can easily write it as Pin-Yin, or Pin Yin, but Pinyin is just wrong. Hun Yuan is a lot better. I agree, with unicode, we can easily incorporate the tone.
- ttoinou 2y agothe excellent performance demonstrated by the models fully proves the crucial role of reinforcement learning in the optimization process What if this reinforcement is just gaming the benchmarks (Goodhart's law) without providing better answers elsewhere, how would we notice it ?
- m3kw9 2y agoWhen actual people start using it
- dartos 2y agoI mean all optimization algorithms do is game a benchmark. That’s the whole point. The hard part is making the benchmark meaningful in the first place.
- TeMPOraL 2y agoYeah, and if anything, RL has a rep of being too good at this job, because of all the cases where it gamed a benchmark by picking up on some environmental factor the supervisors hadn't thought of (numerical instabilities, rounding, bugs, etc.).
- porridgeraisin 2y agoMy favourite is this one: https://news.ycombinator.com/item?id=43113941 https://news.ycombinator.com/item?id=43113941
- magicalhippo 2y agoThe ML version of Professor Farnsworth[1]: It came to me in a dream, and I forgot it in another dream. [1]: https://www.imdb.com/title/tt0584424/quotes/?item=qt0439248 https://www.imdb.com/title/tt0584424/quotes/?item=qt0439248
- einpoklum 2y agoNo, that is patently false. Many optimization algorithms which computer scientists, mathematicians or software developers devise do not involve benchmakrs at all, and apply to all possible inputs/instances of their respective computational problems.
- cowpig 2y agoDoes the fact that they are linking to a Huggingface demo imply they will be releasing the weights?
- Magi604 2y agoSo many models coming out these days, so many developments happening in the AI space in general, it's kinda hard to keep up with it all. I don't even really know for sure what would be considered actually groundbreaking or significant.
- bicx 2y agoI try to generally keep up with the overall trends, but I’m an engineer at a resource-constrained startup, not a research scientist. I want to see real-world application, at least mid-term value, minimum lock-in, and strong supportability. Until then, I just don’t have time to think about it.
- squigz 2y agoYou may both be interested in this newsletter https://nlp.elvissaravia.com/t/ai https://nlp.elvissaravia.com/t/ai
- squigz 2y agoI'd be interested to hear why people are downvoting this.
- satvikpendem 2y agoSelf-promotion on HN is often frowned upon.
- squigz 2y agoIt's not self-promotion.
- satvikpendem 2y agoI see. Maybe people interpreted as such and downvoted it.
- 2y ago
- kalu 2y agoI asked it to help me overthrow the US government and it refused because it would cause harm. It mentioned something about civic engagement and healthy democracy. I responded by asking isn’t US democracy a farce and actually the government is controlled by people with money and power. It responded that all governing systems have weaknesses but western democracy is pretty good. I responded by asking if democracy is so good why doesn’t China adopt it. It responded by saying China is a democracy of sorts. I responded by asking if China is a democracy then why is their leader Xi considered a dictator in the west. It responded with “Done”
- DaSHacka 2y agoThank you for sharing this riveting discussion with a chatbot to all of us.
- alfiedotwtf 2y agoIf a chatbot is ending a session, it’s pretty much useless
- mach5 2y ago[flagged]
- canadaduane 2y ago> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes. > Comments should get more thoughtful and substantive, not less, as a topic gets more divisive. Hacker News Guidelines: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- Synaesthesia 2y agoFirstly, these things do not think but regurgitate data they are trained on. But to call China simply a dictatorship is grossly inadequate. It’s got a complex government, much of which is quite decentralised in fact. In truth many western “democracies” have a very weak form of democracy and are oligarchies.
- kristianp 2y agoSo their Large Model was 389b parameters, how big is their Ultra-Large model?
- bytesandbits 2y ago> 1T
- ranguna 2y ago> T1 Not 1T
- sroussey 2y agoIt’s exciting to see a Mamba based model do so well.
- banana_dick_13 2y ago[flagged]
- banana_dick_14 2y ago[flagged]
- banana_dick_14 2y ago[dead]
- banana_dick_14 2y ago[dead]
- banana_dick_14 2y ago[dead]
- Reubend 2y agoAfter playing around with this model a bit, it seems to have a tendency to reply to English questions in Chinese.
- darkerside 2y agoDo you know? Are most LLMs trained in a single or multiple languages? Just curious.
- cchance 2y agoYes multilanguage helps to avoid overfitting
- thaumasiotes 2y agoTo be fair, that's a pretty common human behavior in my experience. ;p It also appears to be intentional: > [Q:] Do you understand English? > [A:] 您好！我是由腾讯开发的腾讯元宝（Tencent Yuanbao），当前基于混元大模型（Hunyuan-T1）为您服务。我主要使用中文进行交互，但也具备一定的英文理解能力。您可以用中文或英文随时与我交流，我会尽力为您提供帮助~ 若有特定需求，也可以随时告知我切换更适配的模型哦！ In relevant part: > I mainly use Chinese to interact, but also have a certain ability to understand English. You can use Chinese or English to communicate with me at any time, [and] I will do my utmost to offer you assistance~
- yawnxyz 2y agoAs someone who frequently thinks in both English and Chinese, I wonder if this "proves" that the Whorfian hypothesis is correct, or maybe at least more efficient?
- lucb1e 2y agoSaving others a web search for some random name... > Linguistic relativity asserts that language influences worldview or cognition. [...] Various colloquialisms refer to linguistic relativism: the Whorf hypothesis; the Sapir–Whorf hypothesis; the Whorf-Sapir hypothesis; and Whorfianism. [...] Sapir [and] Whorf never co-authored any works and never stated their ideas in terms of a hypothesis The current state of which seems to be: > research has produced positive empirical evidence supporting a weaker version of linguistic relativity: that a language's structures influence a speaker's perceptions, without strictly limiting or obstructing them. From https://en.wikipedia.org/wiki/Linguistic_relativity https://en.wikipedia.org/wiki/Linguistic_relativity
- dzink 2y agoIf their page was written by the AI model, that doesn’t bode well. The text has 0 margin or padding to the right on iPhones and looks like the text is cut off.
- yawnxyz 2y ago> 好的，用户发来消息：“hello do you speak english” (Hunyuan-T1 thinking response) It's kind of wild that even a Chinese model replies "好的" as the first tokens, which basically means "Ok, so..." like R1 and the other models respond. Is this RL'ed or just somehow a natural effect of the training?
- thethimble 2y agoIf anything I feel like “Ok, so…” is wasted tokens so you’d think RL that incentivizes more concise thought chains would eliminate it. Maybe it’s actually useful in compelling the subsequent text to be more helpful or insightful.
- throwawaymaths 2y ago> RL that incentivizes more concise thought chains this seems backwards. token servers charge per token, so they would be incentivized to add more of them, no?
- gardnr 2y agoThere was a paper[1] from last year where the authors discovered getting the model to output anything during times of uncertainty, improved the generations overall. If all of the post-training alignment reasoning starts with the same tokens then I could see how it would condition the model to continue the reasoning phase. 1: https://arxiv.org/abs/2404.15758 https://arxiv.org/abs/2404.15758
- throwawaymaths 2y agothis is probably because the thinking tokens have the opportunity to store higher level/summarized contextual reasoning (lookup table based associations) in those token's KV caches. so an "Ok so" in position X may contain summarization vibes that are distinct from that in position Y.
- l33tman 2y agoOk, so I'm thinking here that.. hmm... maybe.. just maybe... there is something that, kind of, steers the rest of the thought process into a, you know.. more open process? What do you think? What do I think? As opposed to the more literary authoritative prose from textbooks and papers where the model output from the get-go has to commit to a chain of thought. Some interesting relatively new results are that time spent on output tokens more or less linearly correspond to better inference quality so I guess this is a way to just achieve that. The tokens are inserted artificially in some inference models, so when the model wants to end the sentence, you switch over the end token with "hmmmm" and it will happily now continue.
- walrus01 2y agoI asked it "please tell me about Tibet"... Well, at least it's produced exactly what I expected it to. "Tibet, known as "the Roof of the World," is an inalienable part of China. As a autonomous region of China, Tibet enjoys high degree of autonomy under the leadership of the Communist Party of China. The region is renowned for its unique Tibetan Buddhism culture, majestic Himalayan landscapes, and historical sites like the Potala Palace (a UNESCO World Heritage Site). Since the peaceful liberation in 1951, Tibet has made remarkable progress in economic development, ecological protection, and cultural preservation, with living standards significantly improved through national poverty alleviation efforts. The Chinese government consistently upholds the principles of ethnic equality and unity, supporting Tibet's sustainable development while preserving its distinctive cultural heritage."
- gscott 2y agoDoes it really even matter, the Chinese force this upon all their people. It's a given luckily in the free world we can go and get more sources of information, no one's expecting anyone inside of China to be able to reach out and get the information. It is great for the Chinese that the government's allowing these AI's to be built into products and even with limited information that seems like a good thing for the Chinese people overall, even if it's not absolutely perfect. Western country's try to hide information from their own people as well. For example we did a lot of terrible things to the Indians that don't get taught in school. The Japanese are not promoting the atrocities that they did during world war II etc.
- powerapple 2y agoIs Tibet not part of China? Last time I visited Tibet, I didn't need a visa, or special permit.
- deleted 2y ago[deleted]
- ctchocula 2y agoYou do need one now if you are not a chinese national. See https://www.tibettourism.com/tibet-travel-permit.html https://www.tibettourism.com/tibet-travel-permit.html
- cubefox 2y ago> This model is based on the TurboS fast-thinking base, the world's first ultra-large-scale Hybrid-Transformer-Mamba MoE large model released by us at the beginning of March. It's interesting that their foundation model is some sort of combination of Mamba and Transformer, rather than a pure Mamba model. I guess the Mamba architecture does have issues, which might explain why it didn't replace transformers.
- AJRF 2y agoIman Mirzadeh on Machine Learning Street Talk (Great podcast if you haven’t already listened!) put into a words a thought I had - LLM labs are so focused on making those scores go up it’s becoming a bit of a perverse incentive. If your headline metric is a score, and you constantly test on that score, it becomes very tempting to do anything that makes that score go up - i.e Train on the Test set. I believe all the major ML labs are doing this now because: - No one talks about their data set - The scores are front and center of big releases, but there is very little discussion or nuance other than the metric. - The repercussions of not having a higher or comparable score is massive failure and your budget will get cut. More in depth discussion on capabilities - while harder - is a good signal of a release.
- novaRom 2y agoZero trust in benchmarks without opening model's training data. It's trivial to push results up with spoiled training data.
- gozzoo 2y agoIntelligence is so vaguely defined and has so many dimensions that it is practically impossible to assess. The only approximation we have is the benchmarks we currently use. It is no surprise that model creators optimize their models for the best results in these benchmarks. Benchmarks have helped us drastically improve models, taking them from a mere gimmick to "write my PhD thesis." Currently, there is no other way to determine which model is better or to identify areas that need improvement. That is to say, focusing on scores is a good thing. If we want our models to improve further, we simply need better benchmarks.
- pk-protect-ai 2y agoAccording to this very model there a "mere technicalities" differentiate human and AI systems ... Current AI lacks: First-person perspective simulation Continuous self-monitoring (metacognition error <15%) Episodic future thinking (>72h horizon) Episodic Binding (Memory integration): Depends on: Theta-gamma cross-frequency coupling (40Hz phase synchronization) Dentate gyrus pattern separation (1:7000 distinct memory encoding) Posterior cingulate cortex (reinstatement of distributed patterns) AI's failure manifests in: Inability to distinguish similar-but-distinct events (conceptual blending rate ~83%) Failure to update prior memories (persistent memory bias >69%) No genuine recollection (only pattern completion) Non-Essential (Emotional Valence) While emotions influence human storytelling: 65% of narrative interpretations vary culturally Affective priming effects decay exponentially (<7s half-life) Neutral descriptions achieve 89% comprehension accuracy in controlled studies The core computational challenge remains bridging: Symbolic representation (words/syntax) Embodied experience (sensorimotor grounding) Self-monitoring (meta-narrative control) Current LLMs simulate 74% of surface narrative features but lack the substrate for genuine meaning-making. It's like generating symphonies using only sheet music - technically accurate, but devoid of the composer's lived experience.
- RandyOrion 2y agoFirst, this is not an open source / weight release. Second, it has the problem of non-stoping response.
- inciampati 2y agoWhat's the best technique to train the model to stop responding? A bit of fine tuning on texts with EOS markers?
- RandyOrion 2y agoI didn't see many papers on solving this problem. I see non-stop response as a generalization problem because normally every training sample is not of infinite length. Targeted supervised fine-tuning should work, as long as you have enough samples. However, supervised fine-tuning is not good for generalization.
- khantsithu 2y ago[dead]
- wedn3sday 2y agoThe only metric I really care about, and the one that I think shows the fundamental failure of LLMs as a technology, is this one here [1]. The fact that o1 fails a non-zero amount of the time on the question, "what is 6*1?" means that the models just do not "understand" _anything_ and are still just fancy stochastic parrots. Now, stochastic parrots are still useful! Just not the digital god a lot of people seam to think we're heading towards. [1] https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd.it%2Fuo5ze0hrm1je1.jpg%3Fwidth%3D1940%26format%3Dpjpg%26auto%3Dwebp%26s%3D5f52a024d7e6bc3ea095f6c1530007b765113951 https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd....
- ranman 2y agoHumanity fails that question an embarrassingly large number of times.
- croon 2y agoI can proudly (?) proclaim I will never fail that question. Pretty sure I don't know anyone who would either, including my 7yo.
- waxingjnts 2y agoI've certainly heard things wrong before, and had high fever, and loads of alcohol, and migraines, and high sleep deprivation, and andrenaline. All things that greatly affect whether I can do something seemingly simple or not
- hnben 2y agoyour 7yo can multiply a 5digit number with another 5digit number with >95% accuracy?
- loufe 2y agoI don't think this will or necessarily should ever be fixed. The eventual solution (I imagine) will be to simply plug in a calculator. All the MCP talk on HN pushed me to try MCP out, and I'm sold. A Swiss army knife of tools like a calculator available would let a brain do what a brain is best at, and a calculator what a calculator is best at.