9 ms·

$330M is like 33 training runs of a large LLM. Not sure that would bring open source anywhere near GPT-3.5.

by treprinum 3y ago

abiraja 3y ago
Have you tried Mistral?
- dannyw 3y ago
  Mistral is genuinely groundbreaking, for a fast, locally-hosted model without content filtering at the base layer. You can try it online here: https://labs.perplexity.ai/ https://labs.perplexity.ai/ (switch to Mistral)
  - anonzzzies 3y ago
    Are more companies/teams than the creating team working to get this to copilot/chatgpt standards?
  - js4ever 3y ago
    Wow I was not expecting this, It's really something else in terms of speed, and results are not bad! Will test it more
  - eropple 3y ago
    It's very fast, but it doesn't seem very good. It doesn't take instruction well (acknowledges and spits back the same wrong stuff) and doesn't seem to have much of a corpus or it's dropping most of it on the floor because it successfully answers zero of my three basic smoke-test questions.
    - sigmar 3y ago
      >doesn't seem to have much of a corpus what do you mean by 'corpus'? It is only 13GB so questions that require recalling specific facts aren't going to work well with so little room for 'compression', but asking mistral to write emails or perform style revisions works quite well for me
    - mark_l_watson 3y ago
      Are you running mistral-7B or mistral-7B-instruct?
  - audessuscest 3y ago
    Thanks for the link, do you know any other similar services that support fine-tuning ?
gwervc 3y ago
Maybe at today cost. But this is a lab about AI not just LLM. Also given that French research is underfunded since decades, our researchers are accustomed to producing good results at a budget cost.
- nwoli 3y ago
  Also has incredibly good math institutions
- School-Cotton 3y ago
  “French research is underfunded since decades” is a French-ism, just so you know. Normally native English speakers would say “French research has been underfunded for decades”.
  - pokepim 3y ago
    [dead]
  - Obscurity4340 3y ago
    Depuis ?
    - School-Cotton 3y ago
      That can be used with either a duration (“depuis des décennies”) or a point in time (“depuis 1995”). English “since” can only be used with a point in time.
      - Obscurity4340 3y ago
        Merci, français ami! Edit: is that correct or is there an apostrophe Im missing? It would better if you were française, n'est-ce pas?
        School-Cotton 3y ago
        I think “merci, mon ami français” would be more correct, but at any rate, I’m not French!
nmfisher 3y ago
If they’re researching avenues other than LLMs, that money goes a lot further. We can only hope.
espadrine 3y ago
During the conference where it was announced, they indicated they have a partnership with Scaleway to access their Nabu 2023 supercomputer[0]. I expect it will be similar to the relationship between OpenAI and Microsoft Azure, with free credits as part of their investment in exchange for being at the forefront of AI datacenter design. Indeed, Xavier Niel is one of three investors in Kyutai, and founded Scaleway. [0]: https://www.scaleway.com/en/news/scaleway-releases-the-details-of-its-offering-based-on-nabuchodonosor-its-dedicated-ai-supercomputer-built-on-nvidia-dgx-h100-infrastructure/ https://www.scaleway.com/en/news/scaleway-releases-the-detai...
qwertox 3y ago
Maybe related: EU is starting a program where AI startups will get access to their supercomputers and also get a chance to win prizes. 16 November 2023: Commission opens access to EU supercomputers to speed up artificial intelligence development [0] - Launch of the Large AI grand challenge: This competition – launched today, is a collaboration led by the EU funded project AI-BOOST, with access to the European Supercomputers being facilitated by the EuroHPC JU. It encourages the wide participation from European start-ups with experience in large-scale AI models. The winners are expected to release the developed models under an open-source license for non-commercial use, or through publishing their research findings. The challenge will select up to four promising European AI start-ups that will be given access to EuroHPC supercomputing facilities to foster the development of large-scale AI models in Europe and a €1 million prize will be distributed among the winners. - Opening up European supercomputer capacity: Access will be established for ethical and responsible AI start-ups, enabling them to efficiently train their models using European supercomputers. - Enhanced activities and services: the EuroHPC JU will advance activities and services powered by High-Performance Computing to foster trustworthy AI in Europe. These efforts will aim to facilitate increased accessibility for AI communities and promote the optimal and efficient use of HPC technologies for scientific and industrial innovation. [0] https://ec.europa.eu/commission/presscorner/detail/en/ip_23_5739 https://ec.europa.eu/commission/presscorner/detail/en/ip_23_...
- konschubert 3y ago
  First they set up massive barriers such that you’d be insane to start an AI company in Europe. And then they dole up subsidies. Makes no sense. That money will go to waste.
  - zirgs 3y ago
    What massive barriers are there?
    - JumpCrisscross 3y ago
      > What massive barriers are there? Starting and scaling a business in Europe is harder than in America. This is true for fundamental reasons, like language and national borders. It’s also true due to regulation and the culture of work and towards commerce in general. Some of those come with reasonable tradeoffs, e.g. in respect of employee protections. Many do not, particularly when it comes to licensing, bureaucracy and the peculiar way most European tax law and regulation tries to compensate for its licensing and bureaucracy by adding more bureaucracy in front of a balancing subsidy.
gentleman11 3y ago
Sorry, could somebody show a source for that? An $11M bill, that’s nuts and just need a reference lol
- snypher 3y ago
  "But to put things into perspective, GPT-3 175B model required 3.14E23 FLOPS of computing for training. Even at theoretical 28 TFLOPS for V100 and lowest 3 year reserved cloud pricing we could find, this will take 355 GPU-years and cost $4.6M for a single training run." https://lambdalabs.com/blog/demystifying-gpt-3/ https://lambdalabs.com/blog/demystifying-gpt-3/
tyingq 3y ago
There's some compute also, which may not be counted in the 330M "Co-founded by the Iliad group...To do this, the laboratory will use the computing power made available to it by Scaleway, an iliad Group subsidiary. Scaleway’s supercomputer has the highest-performance computing power for AI applications deployed to date in Europe." https://kyutai.org/CP_Kyutai_AI_EN.pdf https://kyutai.org/CP_Kyutai_AI_EN.pdf
coding123 3y ago
Seems like it could be unlimited if people just bought hardware
logicchains 3y ago
> $330M is like 33 training runs of a large LLM. Not sure that would bring open source anywhere near GPT-3.5. Open source is already quite near GPT-3.5; it's reaching GPT-4 level that is the challenge.

$330M is like 33 training runs of a large LLM. Not sure that would bring open source anywhere near GPT-3.5.

abiraja 3y ago

Have you tried Mistral?

dannyw 3y ago
Mistral is genuinely groundbreaking, for a fast, locally-hosted model without content filtering at the base layer. You can try it online here: https://labs.perplexity.ai/ https://labs.perplexity.ai/ (switch to Mistral)
- anonzzzies 3y ago
  Are more companies/teams than the creating team working to get this to copilot/chatgpt standards?
- js4ever 3y ago
  Wow I was not expecting this, It's really something else in terms of speed, and results are not bad! Will test it more
- eropple 3y ago
  It's very fast, but it doesn't seem very good. It doesn't take instruction well (acknowledges and spits back the same wrong stuff) and doesn't seem to have much of a corpus or it's dropping most of it on the floor because it successfully answers zero of my three basic smoke-test questions.
  - sigmar 3y ago
    >doesn't seem to have much of a corpus what do you mean by 'corpus'? It is only 13GB so questions that require recalling specific facts aren't going to work well with so little room for 'compression', but asking mistral to write emails or perform style revisions works quite well for me
  - mark_l_watson 3y ago
    Are you running mistral-7B or mistral-7B-instruct?
- audessuscest 3y ago
  Thanks for the link, do you know any other similar services that support fine-tuning ?

gwervc 3y ago

Maybe at today cost. But this is a lab about AI not just LLM. Also given that French research is underfunded since decades, our researchers are accustomed to producing good results at a budget cost.

nwoli 3y ago
Also has incredibly good math institutions
School-Cotton 3y ago
“French research is underfunded since decades” is a French-ism, just so you know. Normally native English speakers would say “French research has been underfunded for decades”.
- pokepim 3y ago
  [dead]
- Obscurity4340 3y ago
  Depuis ?
  - School-Cotton 3y ago
    That can be used with either a duration (“depuis des décennies”) or a point in time (“depuis 1995”). English “since” can only be used with a point in time.
    - Obscurity4340 3y ago
      Merci, français ami! Edit: is that correct or is there an apostrophe Im missing? It would better if you were française, n'est-ce pas?
      - School-Cotton 3y ago
        I think “merci, mon ami français” would be more correct, but at any rate, I’m not French!

nmfisher 3y ago

If they’re researching avenues other than LLMs, that money goes a lot further. We can only hope.

espadrine 3y ago

During the conference where it was announced, they indicated they have a partnership with Scaleway to access their Nabu 2023 supercomputer[0]. I expect it will be similar to the relationship between OpenAI and Microsoft Azure, with free credits as part of their investment in exchange for being at the forefront of AI datacenter design. Indeed, Xavier Niel is one of three investors in Kyutai, and founded Scaleway. [0]: https://www.scaleway.com/en/news/scaleway-releases-the-details-of-its-offering-based-on-nabuchodonosor-its-dedicated-ai-supercomputer-built-on-nvidia-dgx-h100-infrastructure/ https://www.scaleway.com/en/news/scaleway-releases-the-detai...

qwertox 3y ago

Maybe related: EU is starting a program where AI startups will get access to their supercomputers and also get a chance to win prizes. 16 November 2023: Commission opens access to EU supercomputers to speed up artificial intelligence development [0] - Launch of the Large AI grand challenge: This competition – launched today, is a collaboration led by the EU funded project AI-BOOST, with access to the European Supercomputers being facilitated by the EuroHPC JU. It encourages the wide participation from European start-ups with experience in large-scale AI models. The winners are expected to release the developed models under an open-source license for non-commercial use, or through publishing their research findings. The challenge will select up to four promising European AI start-ups that will be given access to EuroHPC supercomputing facilities to foster the development of large-scale AI models in Europe and a €1 million prize will be distributed among the winners. - Opening up European supercomputer capacity: Access will be established for ethical and responsible AI start-ups, enabling them to efficiently train their models using European supercomputers. - Enhanced activities and services: the EuroHPC JU will advance activities and services powered by High-Performance Computing to foster trustworthy AI in Europe. These efforts will aim to facilitate increased accessibility for AI communities and promote the optimal and efficient use of HPC technologies for scientific and industrial innovation. [0] https://ec.europa.eu/commission/presscorner/detail/en/ip_23_5739 https://ec.europa.eu/commission/presscorner/detail/en/ip_23_...

konschubert 3y ago
First they set up massive barriers such that you’d be insane to start an AI company in Europe. And then they dole up subsidies. Makes no sense. That money will go to waste.
- zirgs 3y ago
  What massive barriers are there?
  - JumpCrisscross 3y ago
    > What massive barriers are there? Starting and scaling a business in Europe is harder than in America. This is true for fundamental reasons, like language and national borders. It’s also true due to regulation and the culture of work and towards commerce in general. Some of those come with reasonable tradeoffs, e.g. in respect of employee protections. Many do not, particularly when it comes to licensing, bureaucracy and the peculiar way most European tax law and regulation tries to compensate for its licensing and bureaucracy by adding more bureaucracy in front of a balancing subsidy.

gentleman11 3y ago

Sorry, could somebody show a source for that? An $11M bill, that’s nuts and just need a reference lol

snypher 3y ago
"But to put things into perspective, GPT-3 175B model required 3.14E23 FLOPS of computing for training. Even at theoretical 28 TFLOPS for V100 and lowest 3 year reserved cloud pricing we could find, this will take 355 GPU-years and cost $4.6M for a single training run." https://lambdalabs.com/blog/demystifying-gpt-3/ https://lambdalabs.com/blog/demystifying-gpt-3/

tyingq 3y ago

There's some compute also, which may not be counted in the 330M "Co-founded by the Iliad group...To do this, the laboratory will use the computing power made available to it by Scaleway, an iliad Group subsidiary. Scaleway’s supercomputer has the highest-performance computing power for AI applications deployed to date in Europe." https://kyutai.org/CP_Kyutai_AI_EN.pdf https://kyutai.org/CP_Kyutai_AI_EN.pdf

coding123 3y ago

Seems like it could be unlimited if people just bought hardware

logicchains 3y ago

> $330M is like 33 training runs of a large LLM. Not sure that would bring open source anywhere near GPT-3.5. Open source is already quite near GPT-3.5; it's reaching GPT-4 level that is the challenge.