4 ms·

Feels like there must be 40 or so distinct open source llms now. What gives? We need some more new text to image models too... :(

• refulgentis 3y ago
  Languages, sizes, and degrees of open-ness.
  • chuckhend 3y ago
    There's some more commentary on their open-ness in this blog too https://www.interconnects.ai/p/olmo https://www.interconnects.ai/p/olmo
    • dwagnerkc 3y ago
      That post also very helpfully links to another paper they published alongside the OLMo paper just on the dataset. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research https://arxiv.org/abs/2402.00159 https://arxiv.org/abs/2402.00159
• chuckhend 3y ago
  The training datasets are also available, which sets them apart a bit IMO. https://huggingface.co/datasets/allenai/dolma https://huggingface.co/datasets/allenai/dolma
• hedgehog 3y ago
  There are few that are >1B params, competitive, and "open source" in the sense that the necessary ingredients to re-train are available. Models like Llama and thus its descendants (including Mistral's public models) have weights available but not the training data.
• wokwokwok 3y ago
  If you read around, training a 7B model costs on the order of $85,000; the 1.4 stable diffusion release cost around $600,000 to train. You don't see a lot of 70B or larger models being released for the same reason; it's expensive. We should just be grateful for what we're getting right now: basically, people are spending 100s of thousands of dollars on training and giving the results away for free. Hugging face is hosting them for free. ollama is hosting them for free. People are writing free inference engines (eg. llama.cpp) and giving them away. Don't complain. We've got it pretty damn good right now.
  • alchemist1e9 3y ago
    > If you read around, training a 7B model costs on the order of $85,000; the 1.4 stable diffusion release cost around $600,000 to train. That seems remarkably cheap actually and likely getting cheaper fairly quickly with improvements in training efficiencies I’d imagine.
  • sjwhevvvvvsj 3y ago
    On the other hand, the systems are trained on “free” data so it kinda should be public property by default. Claiming it’s fair use to suck up the entire web and pay wall the derived result is absurd argument. We all created the lifeblood of LLM and we’re entitled to the product.
    • visarga 3y ago
      > We all created the lifeblood of LLM and we’re entitled to the product. sounds so nice, yet there are going to be objections, NYT for example doesn't think we all should be entitled to the product
      • sjwhevvvvvsj 3y ago
        Of course, that is partially my point: if OpenAI et al wants to make the argument anything online is fair game, then they should release the weights. If not, they have no leg to stand on.
    • vlovich123 3y ago
      You’ve just described Google which derives most of its ad revenue from ads it places on the search engine that’s crawling the public web. It has always been thus that derivative products that provide a meaningful transformation of the input is a wholly separate piece of copyright.
      • sjwhevvvvvsj 3y ago
        No, this is very different. Google will link you to the NYT, you read there, and see ads. If GPT eats the web and pay walls it, they are 100% free riding. Now, I also think the Google model is proven at this point to be a bad model since the web is 90% ads and SEO dogshit. They strip mined the value, took them a while, but its nearly decimated.
• thawab 3y ago
  Open source means i have documentation to reproduce the same results. This is only true with tinyllama and this model. The other models (llama, mistral) are free to use and not open source.
  • arugulum 3y ago
    The Pythia models have all the training data, code, and configurations available.