7 ms·
01-AI/Yi: A series of large language models trained from scratch
- niel 3y agoPrevious discussion: https://news.ycombinator.com/item?id=38158554 https://news.ycombinator.com/item?id=38158554
- dang 3y agoThanks! Macroexpanded: Kai Fu Lee's New AI Company: Yi-Open Source - https://news.ycombinator.com/item?id=38158554 https://news.ycombinator.com/item?id=38158554 - Nov 2023 (66 comments)
- popinman322 3y agoWhen I saw "from scratch" instead of "pretrained" I assumed this was trained using some novel RL setup-- there's value to using the same verbiage as everyone else.
- leoff 3y agoIt's scratch, the programming language https://scratch.mit.edu/about https://scratch.mit.edu/about
- zapnuk 3y agoThese pedantic “when I saw X instead of Y I assumed Z” are the most annoying comments on this website. “From scratch” might be non-specific, but a fine phrasing in this case.
- huggingmouth 3y agoNo, they're not padentic or "anoying". Words have meaning. The more precise their use is, the clearer the communication becomes. Investing the time to properly learn pre-established termilology has tremendous value for everyone. Well, everyone but those who are too lazy and sloppy, and would rather just confuse everybody instead. I'm glad this disgusting sloppiness doesn't exist in more established fields. I hope people keep calling it out.
- nineteen999 3y ago> "disgusting sloppiness" > padentic > anoying > termilology Cute.
- huggingmouth 3y agoCute or not. My point stands
- popinman322 3y agoIt's valuable data sometimes. Those types of comments show you how other people perceived something. In this case my lens is colored by my interests; I've been thinking a lot on RL for LLMs and it slipped out here. For me I rely on phrases to quickly filter for (subjectively) interesting work. And assuming we value that subjective filtering of content, we make it easier for others to filter consistently when we use consistent language.
- synarchefriend 3y agoThey probably want to emphasize that it's not another llama derivative.
- Mougatine 3y ago"From scratch" is commonly used in the field.
- popinman322 3y agoAmong arxiv publications there are 217 results that contain "large language model" in the full text and "from scratch" in the title or abstract. There are 2873 results that contain "large language model" in the full text and use "pretrained" in the title or abstract. A 10x difference in publication count does make one feel more common than the other? I'd need to get into more involved queries to break down the semantic categories of those papers.
- light_hue_1 3y agoFrom scratch simply means that they didn't base it off some other llm. This is perfectly good language and exactly the correct thing for them to say.
- GaggiX 3y ago"Trained from scratch" is perfectly fine terminology to report that the model they published is not a finetuned, but was trained from randomly initialized weights.
- seydor 3y agoI would have liked to see how it compares with Mistral
- logicchains 3y agoChina's catching up fast in the open source model space, I wonder how long it'll take until they have a commercial model competitive with ChatGPT3.5 or Claude 2?
- antupis 3y agoWho cares if it is open source model and weights are available. Nightmare scenario is that AI is behind some paywall and some entity can decide what goes in and what goes out.
- FeepingCreature 3y agoInstead, let's have AI that nobody can control, that will surely not lead to nightmare scenarios.
- pjc50 3y agoControlling AI is just a subset of controlling humans; while US policymakers might like to dictate detailed policy to China to constrain competition, they're not going to be able to do that.
- btbuildem 3y agoWhat nightmare scenario are you envisioning? Now stop, go back, and examine who the bad actor is in that scenario. Leaving such powerful tools in the hands of secretive organizations or uber-wealthy individuals is a guarantee for bad outcomes. Distributing the tools, the know-how required to build and service them, and the benefits of their use -- that's the antidote.
- dvh 3y agoI downloaded the repository and it is 700kB (1900 LOC). It clearly doesn't contains what it claims to have. How is this considered "open source"?
- TheBlapse 3y agoWeights are available on huggingface: https://huggingface.co/01-ai/Yi-34B/ https://huggingface.co/01-ai/Yi-34B/
- ninjin 3y agoUnder a license which does not pass the open source litmus test: https://huggingface.co/01-ai/Yi-34B/blob/main/LICENSE https://huggingface.co/01-ai/Yi-34B/blob/main/LICENSE Still, better than being entirely closed I suppose. But there are properly open models out there and one wonders how they will compare.
- andy99 3y agoIt's better than being closed. But all the promotion I've see has been calling it "open source" which is harmful to all open source. If they'd just released it under their silly licensed and not played up open source it would be fine.
- ninjin 3y agoIndeed, as I keep on saying, we are not entitled for their work. However, attempting to draw good will by appropriating the meaning of the term is, to borrow an expression from Bryan Cantrill's amazing Lisa '11 talk about Illumos [1], "Shitting in the pool of open source and disgusting corporate behaviour". Shame on Facebook, 01-AI, and everyone else that keeps on doing this. [1]: https://yewtu.be/watch?v=-zRN7XLCRhc https://yewtu.be/watch?v=-zRN7XLCRhc
- pbronez 3y agoWow. That license asserts that Taiwan is part of China, requires you to “respect social ethics and moral standards”. It also forbids several specific uses, and they’re not what you might think on a naive reading of the English translation. When the CCP says “terrorism”, they’re justifying their genocidal policies towards ethnic and religious minorities https://www.hrw.org/report/2021/04/19/break-their-lineage-break-their-roots/chinas-crimes-against-humanity-targeting https://www.hrw.org/report/2021/04/19/break-their-lineage-br... When the CCP says “misinformation”, they’re rewriting history to ignore Tiananmen Square. https://www.britannica.com/event/Tiananmen-Square-incident https://www.britannica.com/event/Tiananmen-Square-incident When the CCP says “national security”, they condemning people like Naomi Wu who don’t fit cleanly into their worldview https://skepchick.org/2023/08/maker-naomi-wu-is-silenced-by-chinese-authorities-and-why-i-blame-elon-musk/ https://skepchick.org/2023/08/maker-naomi-wu-is-silenced-by-... Quotes: > “Laws and Regulations” refers to the laws and administrative regulations of the mainland of the People's Republic of China (for the purposes of this Agreement only, excluding Hong Kong, Macau, and Taiwan). > 1) Your use of the Yi Series Models must comply with the Laws and Regulations as well as applicable legal requirements of other countries/regions, and respect social ethics and moral standards, including but not limited to, not using the Yi Series Models for purposes prohibited by Laws and Regulations as well as applicable legal requirements of other countries/regions, such as harming national security, promoting terrorism, extremism, inciting ethnic or racial hatred, discrimination, violence, or pornography, and spreading false harmful information.
- anon23432343 3y agoAnother hour another new AI model. Which can't solve bubble sort correct and will output you a bad performing version of it. AI is the future.
- brucethemoose2 3y agoNot sure if its worth a new HN post, but a 200K context version of the base model just dropped: https://huggingface.co/01-ai/Yi-34B-200K https://huggingface.co/01-ai/Yi-34B-200K Even untuned, I am liking the 34B 4K model so far. I never really use raw llamav2 70b though.
- ChrisArchitect 3y ago[dupe] https://news.ycombinator.com/item?id=38158554 https://news.ycombinator.com/item?id=38158554
- ilaksh 3y agoLooks amazing. Too bad they don't allow commercial use of the model (without a license agreement).
- gs17 3y agoNon-commercial use is also limited. > Your use of the Yi Series Models must comply with the Laws and Regulations as well as applicable legal requirements of other countries/regions, and respect social ethics and moral standards ("Laws and Regulations" is specifically mainland China's)