5 ms·

Can you imagine how incredible an open source model would be for research / humanity beyond the buisness needs right in front of us? Open-Knowledge source with

Can you imagine how incredible an open source model would be for research / humanity beyond the buisness needs right in front of us? Open-Knowledge source with an Open-Inteligence that can guide you through the entire massive digital library of its own brain. Semantic data light-years beyond a Search Engine.

talldayo 2y ago
No, I really can't imagine it. Extrapolating from our free commercially-licensed offerings it would seem most people would ignore it or share stories on Reddit about how FreeGPT poisoned their family when generating a potato salad recipe.
- koolala 2y ago
  An open source model would be able to give you the sources of its potato salad recipe inspiration. It would be the best of both worlds. AI Knowledge + Real Open Human Knowledge.
  - JumpCrisscross 2y ago
    > open source model would be able to give you the source of its potaeo salad recipe Kagi’s LLM can already do that. I believe so can Perplexity’s. Citing sources isn’t something only open models can do.
    - koolala 2y ago
      I'm pretty sure Kagi is like a normal search engine with AI integration like Google. Not an AI designed to be open source with an open dataset of knowledge it was trained on.
      - JumpCrisscross 2y ago
        > pretty sure Kagi is like a normal search engine with AI integration like Google Sure. The point is the thing you said only an open-source model can do, it can do. Plenty of proprietary LLMs can cite sources. The plain truth is most of the benefits of open models are not on the consumer side. (Or at least, I haven't seen any articulated.) They're on the producers'. Open models are better for those of us training models. That's partly why the open data debate is academic--very few people are training large foundation models because the compute and electricity costs are prohibitive.
        koolala 2y ago
        I'm kinda hoping World Governments will use their Public Library infrastructure to train AI. Japan is my #1 hope with how they are opening public science knowledge. Super-computers have been prohibitive for a long time but national science institutions could be a great place for open source & open weight AI.
        JumpCrisscross 2y ago
        > hoping World Governments will use their Public Library infrastructure to train AI Genuinely blown away the EU isn't doing this. In the U.S., the solution may be in carving a legal safe harbor for companies that release their models per the OSI's draft definition of open source.
        talldayo 2y ago
        I bet Nvidia would quite like that too. Private and public-sector funding, theirs for the taking! Few businesses are ever so lucky.
  - valine 2y ago
    Just because you have the dataset doesn't mean you can generate a reference. Let's say I hand you a potato salad recipe and a copy of the entire internet. Say you somehow extract all potato salad recipes from the dataset (non trivial btw) and none of them are an exact match for the recipe the model generated. Now what?
JumpCrisscross 2y ago
> Open-Knowledge source with an Open-Inteligence that can guide you through the entire massive digital library of its own brain. Semantic data light-years beyond a Search Engine This sounds like the usual AI marketing with the word "open" thrown in. It's not articulating something youc an only do with an open source LLM (and doesn't define what that means). I'm personally not thrilled with how locked down LLMs are. But we'll need to do a better job at articulating (a) a definition and (b) the benefits of adhering to it versus the "you can run it on your own metal" definition Facebook is promulgating. Because a model meeting Facebook's definition has obvious benefits over proprietary models run on someone else's servers.
- koolala 2y ago
  You can't imagine it :( Open data :( I believe our world fights to destroy ideas like this because our economy drives our entire life.
  - JumpCrisscross 2y ago
    > Can you imagine >> No >>> You can't imagine it You haven’t articulated the idea you claim the “world fights to destroy”. (Just throwing around the word open without elaboration isn’t an idea.)
    - koolala 2y ago
      Data that is accessible. Knowledge. Truth. With an AI trained on it that can expose it in any expert / layman terms into any human language.
      - JumpCrisscross 2y ago
        You’re undermining the case for an open source LLM by stating things fully-proprietary models do.
        koolala 2y ago
valine 2y ago
If you have the model weights you have roughly the same opportunities as the company that trained the model. The code you need to run inference on the Llama weights is very much open source. The only thing you're missing out on is the training code, which is prohibitively expensive to run for most anyways. Open source training isn't going to give you any unique insights into the "digital brain library" of your model. Also just to be clear, if you want to set up a RAG with an open weight model and a large dataset there's nothing stopping you. Download Red Pajama and Llama and give it a try. https://github.com/togethercomputer/RedPajama-Data https://github.com/togethercomputer/RedPajama-Data
HarHarVeryFunny 2y ago
You're not really asking for an open source model though, you're asking for open source training data set(s), which isn't something that Meta can give you. There are open source web scrapes such as The Pile, but much of the more specialized data needs to be licensed.
- koolala 2y ago
  I'm asking for an "Open Source AI" and Meta and everyone supporting them is convinced its impossible in our lifetimes :( We are living in the Dark Ages where Information = $$$. I pray to AI we one day grow out of this pointless destructive economic spiral towards the heat death of the Earth and collect and share open knowledge across all human cultures and history.
  - HarHarVeryFunny 2y ago
    Well, as long as by "AI" you are referring to pre-trained transformers, then what you are effectively asking for is the data used to pre-train them. OTOH why you want the data is not clear. You don't need it to run Meta'a models for free, or to fine-tune them for your own needs. The only thing the data would allow you is to pre-train from scratch, in other words to obtain the exact same set of weights that Meta is giving you for free.
tourmalinetaco 2y ago
All of that data is already available, just look into “shadow libraries”. Now, I do wish Meta and other companies would publish their data sets and we, as humanity, could improve upon them and empower even better LLMs, but the unfortunate reality is copyright is holding us back. Most of what you say is essentially gibberish, but there is truth that LLMs would be better if it could not only utilize its weights, but reference and search its training data (that is collectively owned by humanity, by the way) and answer with that and not just what it “thinks”.

talldayo 2y ago

No, I really can't imagine it. Extrapolating from our free commercially-licensed offerings it would seem most people would ignore it or share stories on Reddit about how FreeGPT poisoned their family when generating a potato salad recipe.

koolala 2y ago
An open source model would be able to give you the sources of its potato salad recipe inspiration. It would be the best of both worlds. AI Knowledge + Real Open Human Knowledge.
- JumpCrisscross 2y ago
  > open source model would be able to give you the source of its potaeo salad recipe Kagi’s LLM can already do that. I believe so can Perplexity’s. Citing sources isn’t something only open models can do.
  - koolala 2y ago
    I'm pretty sure Kagi is like a normal search engine with AI integration like Google. Not an AI designed to be open source with an open dataset of knowledge it was trained on.
    - JumpCrisscross 2y ago
      > pretty sure Kagi is like a normal search engine with AI integration like Google Sure. The point is the thing you said only an open-source model can do, it can do. Plenty of proprietary LLMs can cite sources. The plain truth is most of the benefits of open models are not on the consumer side. (Or at least, I haven't seen any articulated.) They're on the producers'. Open models are better for those of us training models. That's partly why the open data debate is academic--very few people are training large foundation models because the compute and electricity costs are prohibitive.
      - koolala 2y ago
        I'm kinda hoping World Governments will use their Public Library infrastructure to train AI. Japan is my #1 hope with how they are opening public science knowledge. Super-computers have been prohibitive for a long time but national science institutions could be a great place for open source & open weight AI.
        JumpCrisscross 2y ago
        > hoping World Governments will use their Public Library infrastructure to train AI Genuinely blown away the EU isn't doing this. In the U.S., the solution may be in carving a legal safe harbor for companies that release their models per the OSI's draft definition of open source.
        talldayo 2y ago
        I bet Nvidia would quite like that too. Private and public-sector funding, theirs for the taking! Few businesses are ever so lucky.
- valine 2y ago
  Just because you have the dataset doesn't mean you can generate a reference. Let's say I hand you a potato salad recipe and a copy of the entire internet. Say you somehow extract all potato salad recipes from the dataset (non trivial btw) and none of them are an exact match for the recipe the model generated. Now what?

JumpCrisscross 2y ago

> Open-Knowledge source with an Open-Inteligence that can guide you through the entire massive digital library of its own brain. Semantic data light-years beyond a Search Engine This sounds like the usual AI marketing with the word "open" thrown in. It's not articulating something youc an only do with an open source LLM (and doesn't define what that means). I'm personally not thrilled with how locked down LLMs are. But we'll need to do a better job at articulating (a) a definition and (b) the benefits of adhering to it versus the "you can run it on your own metal" definition Facebook is promulgating. Because a model meeting Facebook's definition has obvious benefits over proprietary models run on someone else's servers.

koolala 2y ago
You can't imagine it :( Open data :( I believe our world fights to destroy ideas like this because our economy drives our entire life.
- JumpCrisscross 2y ago
  > Can you imagine >> No >>> You can't imagine it You haven’t articulated the idea you claim the “world fights to destroy”. (Just throwing around the word open without elaboration isn’t an idea.)
  - koolala 2y ago
    Data that is accessible. Knowledge. Truth. With an AI trained on it that can expose it in any expert / layman terms into any human language.
    - JumpCrisscross 2y ago
      You’re undermining the case for an open source LLM by stating things fully-proprietary models do.
      - koolala 2y ago

valine 2y ago

If you have the model weights you have roughly the same opportunities as the company that trained the model. The code you need to run inference on the Llama weights is very much open source. The only thing you're missing out on is the training code, which is prohibitively expensive to run for most anyways. Open source training isn't going to give you any unique insights into the "digital brain library" of your model. Also just to be clear, if you want to set up a RAG with an open weight model and a large dataset there's nothing stopping you. Download Red Pajama and Llama and give it a try. https://github.com/togethercomputer/RedPajama-Data https://github.com/togethercomputer/RedPajama-Data

HarHarVeryFunny 2y ago

You're not really asking for an open source model though, you're asking for open source training data set(s), which isn't something that Meta can give you. There are open source web scrapes such as The Pile, but much of the more specialized data needs to be licensed.

koolala 2y ago
I'm asking for an "Open Source AI" and Meta and everyone supporting them is convinced its impossible in our lifetimes :( We are living in the Dark Ages where Information = $$$. I pray to AI we one day grow out of this pointless destructive economic spiral towards the heat death of the Earth and collect and share open knowledge across all human cultures and history.
- HarHarVeryFunny 2y ago
  Well, as long as by "AI" you are referring to pre-trained transformers, then what you are effectively asking for is the data used to pre-train them. OTOH why you want the data is not clear. You don't need it to run Meta'a models for free, or to fine-tune them for your own needs. The only thing the data would allow you is to pre-train from scratch, in other words to obtain the exact same set of weights that Meta is giving you for free.

tourmalinetaco 2y ago

All of that data is already available, just look into “shadow libraries”. Now, I do wish Meta and other companies would publish their data sets and we, as humanity, could improve upon them and empower even better LLMs, but the unfortunate reality is copyright is holding us back. Most of what you say is essentially gibberish, but there is truth that LLMs would be better if it could not only utilize its weights, but reference and search its training data (that is collectively owned by humanity, by the way) and answer with that and not just what it “thinks”.