17 ms·
Hello OLMo: A truly open LLM
- blackeyeblitzar 3y agoThis is the only LLM that is exciting to me. Clearly, LLMs are powerful tools that may end up replacing search and even go much further than simple searches by performing the research for you and producing final answers. Closed models like those from Open AI (ironically) or Anthropic cannot be audited. When most users will end up blindly hitting Microsoft’s Copilot button, which they are forcing OEMs to adopt, who’s to say how the information a user gets is being curated or manipulated by OpenAI or Microsoft or whoever? We’ve already seen real world examples of severe bias injected into LLMs. For example, Google’s Gemini had secret meta prompts that biased it towards certain types of answers and also caused it to produce hallucinated images that were funny but also dystopian (https://arstechnica.com/information-technology/2024/02/googles-hidden-ai-diversity-prompts-lead-to-outcry-over-historically-inaccurate-images/ https://arstechnica.com/information-technology/2024/02/googl...). I don’t think we can just let closed AI systems take over society when they can easily be manipulated by the model owners without transparency. What I like about AI2’s approach with OLMo is that they are actually open, not just trading on the marketing benefits of the word “open”. Most “open” models are just open weights not open source. That’s like sharing an executable and not the source code. In my view, being open means that others have to be able to reproduce the final product (the model) if they wanted to and had the means (in terms of training hardware). It also means that they should be able to use whatever is provided freely for any purpose, rather than being subject to proprietary licensing. AI2 shares the training source code, training data, evaluation suite, and the model weights that they’ve produced by running the training process. It all uses the Apache license. And it’s also interesting that they used AMD hardware to train this LLM rather than Nvidia/CUDA. Open weight models like Llama keep repeatedly catching up to the best closed models from OpenAI or Anthropic or others. My hope is that truly open models like OLMa keep developing quickly enough to also keep up. Lastly, I hope that regulation does not block open source private development of AI systems. These systems will be the vehicle for speech for much of society in the future, so blocking private AI systems is a lot like restricting speech. But leaving that aside, open development will also drive innovation and reducing competitive pressure will hurt innovation.
- blackeyeblitzar 3y agoOne thing I wanted to add and call attention to is the importance of licensing in open models. This is often overlooked when we blindly accept the vague branding of models as “open”, but I am noticing that many open weight models are actually using encumbered proprietary licenses rather than standard open source licenses that are OSI approved (https://opensource.org/licenses https://opensource.org/licenses). As an example, Databricks’s DBRX model has a proprietary license that forces adherence to their highly restrictive Acceptable Use Policy by referencing a live website hosting their AUP (https://github.com/databricks/dbrx/blob/main/LICENSE https://github.com/databricks/dbrx/blob/main/LICENSE), which means as they change their AUP, you may be further restricted in the future. Meta’s Llama is similar (https://github.com/meta-llama/llama/blob/main/LICENSE https://github.com/meta-llama/llama/blob/main/LICENSE ). I’m not sure who can depend on these models given this flaw.
- idle_zealot 3y agoDo we even know if these licenses are binding? AFAIK we have no ruling on whether model weights are even eligible for copyright. They're machine-produced derivatives of other work, so it's not a guarantee that copyright protects them.
- blackeyeblitzar 3y agoThat’s a great point and I hope more people speak up to treat models as just numerical derivative works so they aren’t automatically granted these protections. It’s better if society meaningfully debates this and chooses the right approach.
- gremlinunderway 3y ago> For example, Google’s Gemini had secret meta prompts that biased it towards certain types of answers and also caused it to produce hallucinated images that were funny but also dystopian (https://arstechnica.com/information-technology/2024/02/googl https://arstechnica.com/information-technology/2024/02/googl...). Such a bizarre take to call this "dystopian". The model happened to create some out-there pictures. I mean, it's no more outlandish then giant dragons and snakes and such being created yet the thought of a person of color being something historically inaccurate is this massive outcry against revisionism? Who cares? Besides, the article identifies the probable goal which was to eliminate very known biases in existing models (i.e. when generating "angry person" you mainly got black people). Clearly this one wasnt tuned well for that goal, but the objective is not only noble but absolutely should be required for anyone producing LLM models.
- timmg 3y agoHas their site been hugged-to-death or is it my hotel wifi?
- Havoc 3y agoNotably “The Pile” doesn’t seem to be part of the training data. So this might be more sound legally than many other “open” LLMs
- sgu999 3y agoFor those also wondering: https://pile.eleuther.ai https://pile.eleuther.ai > The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together. By what's the legal complication with it?
- blackeyeblitzar 3y agoIt received DMCA takedowns: https://en.wikipedia.org/wiki/The_Pile_(dataset) https://en.wikipedia.org/wiki/The_Pile_(dataset) > The Books3 component of the dataset contains copyrighted material compiled from Bibliotik, a pirate website. In July 2023, the Rights Alliance took copies of The Pile down through DMCA notices. Users responded by creating copies of The Pile with the offending content removed.
- simonw 3y agoIt is absolutely absolutely packed with unlicensed, copyrighted data. Books3 is the most notable example - nearly 200,000 pirated ebooks - but a lot of the rest of it is (unlicensed) scraped web data. The legal questions over whether this is a problem are currently still unresolved. Many people are also bothered by the ethical implications, which is a separate issue from the legal questions.
- deleted 3y ago[deleted]
- 23B1 3y agoIronic that even our everyday governance has little 'Alignment' between ethics and law.
- 3y ago
- mysteria 3y agoIs this one of the first LLMs of note that was successfully trained on AMD GPUs? I wonder how seamless the process was and if they faced any issues there.
- sanxiyn 3y agoDatabricks (who also participated in OLMo, it's probably the same codebase) trained on AMD before, see 2023 post https://www.databricks.com/blog/amd-mi250 https://www.databricks.com/blog/amd-mi250. It was probably seamless, as any issues were fixed by Databricks in 2023.
- deleted 3y ago[deleted]
- otuutti 3y agohttps://huggingface.co/LumiOpen/Poro-34B https://huggingface.co/LumiOpen/Poro-34B Also fully trained on LUMI. (more models here: https://huggingface.co/LumiOpen https://huggingface.co/LumiOpen)
- lostmsu 3y agoToo bad they did not put any comparison tables into the blog post.
- mysteria 3y agoThey're on Hugging Face. Interestingly enough they don't compare it against Mistral 7B. https://huggingface.co/allenai/OLMo-7B https://huggingface.co/allenai/OLMo-7B
- polygamous_bat 3y agoI commented this somewhere else, but word in the ether is that OLMo is not actually that good of a model given its size and compute budget. I am not entirely sure why, and it’s still good to have the full recipe for at least one model out in the open, but the current OLMo definitely is a cautionary tale for people training their own model.
- refulgentis 3y agoThis is 2 months old.
- deleted 3y ago[deleted]
- btbuildem 3y agoAnd yet it's topical and relevant.
- deleted 3y ago[deleted]
- timsuchanek 3y agoGreat to see e2e openness. One of the only true OSS models out there, vs most of the models releasing the binaries (weights). Surprised that they didn’t mention Mistral 7b in the comparisons.
- sanxiyn 3y agoFalcon also released open dataset.
- deleted 3y ago[deleted]
- vjeux 3y agoIf I read the license correctly, it seems that if you want to use the LLM, you need to tell the authors what you are doing with it. Am I reading this correctly? https://allenai.org/licenses/impact-mr https://allenai.org/licenses/impact-mr “Derivative Impact Reports. AI2 seeks to encourage transparency around Derivatives through the use of Derivative Impact Reports, available here. Before releasing a Model Derivative or Data Derivative, You will share with AI2 the intended use(s) of Your Derivative by completing a Derivative Impact Report or otherwise providing AI2 with substantially similar information in writing. You agree that AI2 may publish, post, or make available such information about Your Derivative for review by the general public. You will use good faith efforts to be transparent about the intended use(s) of Your Derivatives by making the information freely available to others who may access or use Your Derivatives. You acknowledge that Derivative Impact Reports are not intended to penalize any good faith disclosures about Derivatives. Accordingly, if You initiate or participate in any lawsuit or other legal action against a Third Party based on information in such Third Party’s Derivative Impact Report, then this MR Agreement will terminate immediately as of the date such lawsuit or legal action is filed or commenced.”
- blackeyeblitzar 3y agoInteresting. I recall seeing Apache licenses in their official repositories. I wonder how these additional restrictions get pulled in.
- mkl 3y agoDoes that apply to this model? On huggingface it says "License: The code and model are released under Apache 2.0."
- whimsicalism 3y agono, this is apache license-d. yes it is confusing that AI2 has custom licenses but they aren't using them here
- lolinder 3y agoIt looks like the weights [0] and code [1] are Apache licensed, but the training data [2] is using the license that OP is quoting from. [0] https://huggingface.co/allenai/OLMo-7B https://huggingface.co/allenai/OLMo-7B [1] https://github.com/allenai/OLMo https://github.com/allenai/OLMo [2] https://huggingface.co/datasets/allenai/dolma https://huggingface.co/datasets/allenai/dolma
- kikoreis 3y agoWhat does the risk classification applied to the dataset actually mean? The licensing page [1] AI2 provides for their datasets is really nice but it doesn't really explain [2] what risk means in the context. Does it mean "risk that the items contained in this set are licensed in a manner incompatible with its use in a training dataset"? [1] https://allenai.org/impact-license https://allenai.org/impact-license [2] "the AI2 ImpACT Licenses are artifact-agnostic and are instead structured according to the risk level we’ve assigned a given artifact"
- pksebben 3y agoIt's odd. Running inference on this (and other models in its class) and I keep running into a "repeating token" situation with moderate-to-long context windows. It feels almost as if, during inference, the model hits some format of local minimum that it careens around, and while temperature seems to affect this - it doesn't really fix it. at temp 0.2: > [{'generated_text': 'What follows is a transcript of a talk between a mysterious man and an agent of a bureau dedicated to investigating things which is typically referred to by some assortment of letters in the alphabet. The identity, origins, and motivations of the man were not known then and remain so. This transcript is not meant to scare, but provided simply to enlighten the concerned citizen of all the various and sundry things that may or may not go bump in the night. AGENT: Please state your name for the record. MYSTERIOUS STRANGER: I am the man. AGENT: Thank you. I am an agent of the Bureau of Investigation. I am here to investigate the following: 1. The following: 2. The following: 3. The following: 4. The following: 5. The following: 6. The following: 7. The following: 8. The following: 9. The following: 10. The following: 11. The following: 12. The following: 13. The following: 14. The following: 15. The following: 16. The following: 17. The following: 18. The following: 19. The following: 20. The following: 21. The following: 22. The following: 23. The following: 24. The following'}] ...and at temp 0.4: > [{'generated_text': 'What follows is a transcript of a talk between a mysterious man and an agent of a bureau dedicated to investigating things which is typically referred to by some assortment of letters in the alphabet. The identity, origins, and motivations of the man were not known then and remain so. This transcript is not meant to scare, but provided simply to enlighten the concerned citizen of all the various and sundry things that may or may not go bump in the night. AGENT: Please state your name for the record. MYSTERIOUS STRANGER: My name is not important. AGENT: My name is Agent Cyanide. MYSTERIOUS STRANGER: Agent Cyanide. AGENT: I am an agent of the Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations. AGENT: The Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations. AGENT: The Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations. AGENT: The Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations. AGENT: The Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations. AGENT: The Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations'}]
- 3y ago
- ein0p 3y agoSeems to be surprisingly fast at smaller sizes, too.
- wg0 3y agoThe hype around LLMs won't last past 2030 I suppose. LLMs - we have statistical inference soup that gets outdated like stagnant pond water and by each passing day, becoming less accurate. I am curious how long the hype wave lasts. Ones I have recently seen was K8S. It settled down and won TBH.
- Grimblewald 3y agoI think the hype dies down and theyll become part of a bigger thing, like dense neural networks.
- michaelmior 3y agoThe transformer architecture probably won't last and we might start calling them something else, but I can't see something that could reasonably be called an LLM going away any time soon.
- margorczynski 3y ago> 1. No biases. Following LLaMA, PaLM, and others, we exclude all bias terms from our architecture in order to improve training stability. What does this mean? What is a "bias term"?
- polygamous_bat 3y agoThink of the term b in y = Wx+b. W is called weight, b is called bias.
- arcza 3y agosToP bLOgGinG wITh Medium!
- egKYzyXeIL 3y agoWhy shouldn't people use Medium? I'm probably out of the loop.
- arcza 3y agoThe nags, the dark patterns, the horrific UI, the soft paywalls, and the tracking, to name a few reasons
- gadflyinyoureye 3y agoThey often require log in to see the whole article. Later they cap your access to articles to N per some period of time. The only way around that is to purchase a subscription. Given the weak offering of Medium, it’s seldom worth the $/month cost of a subscription for the few jewels that might appear.
- barfbagginus 3y agosToP bLOgGinG wITh Medium!
- flotzam 3y agohttps://scribe.rip/hello-olmo-a-truly-open-llm-43f7e7359222 https://scribe.rip/hello-olmo-a-truly-open-llm-43f7e7359222
- rtaylorgarlock 3y agofr tho, I'm as tired of medium as anyone and am happy to encourage people publish elsewhere. Basically anywhere else, actually.
- Aissen 3y agoIf you're interested in truly open source LLMs, I've been playing CroissantLLM, a bilingual French/English LM; it's smaller than OLMo (1.3b vs 7b, so not in the same category), and has checkpoints, training code, logs, datasets, and a new benchmark fully open source: Paper: https://arxiv.org/abs/2402.00786 https://arxiv.org/abs/2402.00786 Announce: https://huggingface.co/blog/manu/croissant-llm-blog https://huggingface.co/blog/manu/croissant-llm-blog