7 ms·

French AI startup Mistral secures €2B valuation

107 points by admp 3y ago

rldjbpin 3y ago
what is their moat over other companies releasing "open-source" models, a la stability and meta? all the fanfare during their inception seems to have washed away tbh.
mg 3y ago
Do LLM providers with smaller / less capable LLMs still have a USP or will it be like what we witnessed in the search space, where the #1 company has more than 90% market share, #2 has less than 5% market share and everyone else less than 1%?
- lelag 3y ago
  If the model is closed and served through an API, probably not. But at the moment Mistral seems to focus on smaller and efficient models that you can fine-tuned and run on your own infrastructure. Mistral 7b was free but if they come up with larger capable models, they can offer them with a non-commercial free license and a sell a commercial license for companies that need to fine-tune it. Looking at the performance of small fine-tuned models on specific tasks, I think most company will prefer taking that route rather than trusting OpenAI with their own data, the market is huge I think.
  - jgalt212 3y ago
    > trusting OpenAI with their own data often that's a GDPR violation. there is not such thing as perfect data scrubbing.
    - IanCal 3y ago
      You can definitely transfer data to another processor without breaking GDPR rules, even US ones. You need to take care, but it doesn't require perfect data scrubbing.
  - mark_l_watson 3y ago
    My hope, being fair to Mistral, is that they keep supporting the 7B model series, and release a set of products such as: one off non-commercial and commercial model purchases of more capable 13B, 20B, and 30B models. I would also like to see a subscription service so, if for example, I was using 13B commercially, I could pay a yearly fee for updates and support for that model. EDIT: since I am retired and just obsess about LLMs and in general deep learning as a ‘gentleman scientist’ I hope Mistral has inexpensive non-commercial options for larger, more capable, and commercially supported models.
  - mg 3y ago
    Hmm... I would think the on-premise search for companies falls within the "less than 1% for everybody else" category I mentioned in regards to search. The revenue of Elastic Search is less than 0.5% of Google's revenue.
vinni2 3y ago
https://archive.ph/2023.12.08-165411/https://www.ft.com/content/ea29ddf8-91cb-45e8-86a0-f501ab7ad9bb https://archive.ph/2023.12.08-165411/https://www.ft.com/cont...
cicce19 3y ago
Does anyone know the advantage of mistral over llama2? I understand the founders of mistral previously worked on llama.
- vinni2 3y ago
  Mistral 7b version is slightly better than 7b llama based on elo rating https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
  - nuz 3y ago
    Also less censored
    - vinni2 3y ago
      there is also llama uncensored https://ollama.ai/blog/run-llama2-uncensored-locally https://ollama.ai/blog/run-llama2-uncensored-locally
  - MacsHeadroom 3y ago
    It's better than that. The best Llama-2 finetune is 88th on the 7B leaderboard, behind 87 Mistral finetunes. Mistral 7B is above every Llama-2 finetune, including the 10x larger llama-2 70B, on the Open LLM Leaderboard. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... (Note: Check "architecture" and uncheck precision "?" then ignore "tigerbot" as it was disqualified for cheating.)
shin_lao 3y ago
Isn't the recent AI regulation voted in Europe a problem for this company?
- eurekin 3y ago
  With that valuation, they could pick any country they like
- TacticalCoder 3y ago
  > Isn't the recent AI regulation voted in Europe a problem for this company? Maybe not that big of a problem but the EU legislation that just passed wants the companies building models to hand over technical documentation, comply with EU copyright laws and give detailed information about the data used for training the models. So basically I take it's there's going to be some annoying administrative work to do but it doesn't seem impossible to comply with.
- sailingparrot 3y ago
  "The European Union's landmark Artificial Intelligence (AI) Act may exempt open-source models from strict regulation, according to a leaked compromise proposal seen by Reuters." [1] And Mistral has been betting on an open-source first approach. One might even wonder if there is some causation between the two decisions, one direction or the other. [1]: https://www.reuters.com/technology/eus-ai-act-could-exclude-open-source-models-regulation-2023-12-07/ https://www.reuters.com/technology/eus-ai-act-could-exclude-...
  - mark_l_watson 3y ago
    The EU would have to be crazy to not support smaller local European companies like Mistral (a French company). So good for their economy and securing their own AI infrastructure. I have ranted about this before on HN, so bear with me here: all countries should take reasonable steps that they can afford to secure their own infrastructure: energy, water, food, some local manufacturing, IT, AI, education, etc. This “one world order” stuff is bull-shit. The world is a better place with many cultures, ethnicities, more self-sufficiency, etc. My wife and I like to travel to meet people with very different backgrounds, see new places. I cringe thinking of a “vanilla” world where large corporations and elite interests “homogenize” the world.
Proven 3y ago
[dead]
monlockandkey 3y ago
What is the end game for companies releasing open source models? Any ideas for a sustainable business?
- IanCal 3y ago
  I know someone will hound me on the definition of open source but... I'd bet on one of four things 1. Early access for a fee. Get the latest model at a cost, they're later fully open sourced. 2. Sell commercial use licenses. Get and validate the model works for you easily, then buy a license. Split by company revenue if you want to 3. Sell hosted versions - these things require annoying tech to host, paying for someone else to manage running machines can help (possibly combined with a license around not setting up a resale of the LLM) 4. Sell the service of producing (and possibly hosting) custom models. Prove your worth with the open source ones, so you're a trusted person to come to in order to get a model custom made.
- WhiteNoiz3 3y ago
  I've also wondered this. Aside from just securing funding, I could see: 1. Offering a paid hosted solution down the road 2. Offering bigger / better models that cost money to use commercially 3. Consulting services around their models 4. Licensing their models to companies that want to host them as part of a paid offering IMO the whole advantage of small, open source models is to avoid paying for hosted solutions and additional privacy / security, so it is a bit at odds with commercialization.
- Art9681 3y ago
  The potential for AI is one of those things that I think a lot of people are willing to fund without a return on their investment. The nations and businesses and individuals that leverage this technology will far outpace those that don't. It has value in and of itself to have access to the most powerful model capable of existing. There is a reason governments are starting to pass laws. Clearly this technology is seen as a huge asset to anyone who's paying attention. I am not making a moral argument here. I don't know if what's coming is good or bad. But it's coming with our without my approval. To add to this, it is a valid strategy to give away something for free so attention - and therefore power - is taken away from a competitor. A quick way for everyone to forget OpenAI exists is for another nation or business conglomerate to release a model that is "good enough". This slows growth for businesses trying to profit from it.
deleted 3y ago
[deleted]
ilaksh 3y ago
Did they release the source code or benchmarks for their new 8x model?
- espadrine 3y ago
  No. Without the inference code, the best we can have are guesses on its implementation, so the benchmark figures we can get could be quite wrong. It does seem better than Llama2-70B in my tests[0], which rely on the work done by Dmytro Dzhulgakov[1] and DiscoResearch[2]. But the point of releasing on bittorrent is to see the effervescence in hobbyist research and early attempts at MoE quantization, which are already ongoing[3]. They are benefitting from the community. [0]: https://twitter.com/espadrine/status/1733505332293288256 https://twitter.com/espadrine/status/1733505332293288256 [1]: https://github.com/dzhulgakov/llama-mistral https://github.com/dzhulgakov/llama-mistral [2]: https://huggingface.co/DiscoResearch/mixtral-7b-8expert https://huggingface.co/DiscoResearch/mixtral-7b-8expert [3]: https://github.com/TimDettmers/bitsandbytes/tree/sparse_moe https://github.com/TimDettmers/bitsandbytes/tree/sparse_moe
- mrkramer 3y ago
  Apparently you can download it via torrent[0]. https://venturebeat.com/ai/mistral-ai-bucks-release-trend-by-dropping-torrent-link-to-new-open-source-llm/ https://venturebeat.com/ai/mistral-ai-bucks-release-trend-by...
  - capableweb 3y ago
    First-hand source: https://twitter.com/MistralAI/status/1706877320844509405 https://twitter.com/MistralAI/status/1706877320844509405
- emadm 3y ago
  Mistral doesn't release source code or data just weights. They did include some details of sliding window attention previously
  - ebalit 3y ago
    They released a reference implementation with Mistral 7B and a fork of a framework they probably used to train the Mixture of Experts for the new one. But it's still early and I guess that they kept just enough mystery to kickstart the conversation around this new model!
htrp 3y ago
Can someone explain to me what they're trying to do now? They went from we're going to create a European competitor to OpenAI to we have a 7bn parameter model? From their pitch deck (https://sifted.eu/articles/pitch-deck-mistral https://sifted.eu/articles/pitch-deck-mistral) "By creating mistral.ai, we intend to train state-of-the-art models with counter-positions to closed-model current offerings. Our vision is to become a leading actor in the field, while developing a very valuable business around integrating these models in the European industry and beyond."
- evanchisholm 3y ago
  Have you used any of the Mistral 7b models? They're really good, I think the 7b model just shows what they're capable of. They also dropped an 8x7b mixture of experts model yesterday, that should be very interesting to try.
  - wenc 3y ago
    I have, through Ollama, and it would say their truthfulness is not great. You almost have to treat them with kid gloves and prompt them a certain way to get reasonable results (they're harder to steer than larger models). There's a reddit thread on this where people share tricks on how to handle 7B models. https://www.reddit.com/r/LocalLLaMA/comments/18e929k/prompt_engineering_for_7b_llms/ https://www.reddit.com/r/LocalLLaMA/comments/18e929k/prompt_... My feeling is 7B models should only be used for domains where reasoning and factual correctness are not needed, like summarization or generating creative variations of ideas. Code generation is ok too if one is willing to babysit it. But otherwise, the answers can be misleadingly bad. Even reasoning fine-tuned models like Orca2 7B do very badly on simple math questions, despite Chain of Thought prompting.
- deleted 3y ago
  [deleted]
- ilaksh 3y ago
  They just released a much larger model as a torrent. Probably the first open source mixture-of-experts model. Waiting for them to drop the code and explain it.
mrkramer 3y ago
This is like dot-com bubble all over again...."we have this fancy and powerful new technology, you must invest in us." Good luck competing with Microsoft, Facebook and Google, because they are super-focused now on LLMs and generative AI and they are not going to stop.
- xvector 3y ago
  Mistral's models outperform all of theirs at much smaller sizes and can be self-hosted.
  - mrkramer 3y ago
    Right now, but how long does it take for OpenAI, Facebook and Google's Bard/Gemini to catch up?
    - capableweb 3y ago
      Considering Google been lying in their marketing videos about their AI capabilities, OpenAI seems to be falling apart from the inside and Facebook refuses to release open (not commercially-locked "open") models, there seems to be space for at least one honest and open AI company.
    - baq 3y ago
      Google is trying and just can't? They are pretending they aren't behind despite setting a pile of cash on fire by spending exorbitant amounts on exaggerating their capabilities in basically fake videos. Meanwhile you see Mistral casually dropping magnet links to weights with barely any instructions on how to use them on a Friday afternoon and you see reports of some very happy people finetuning on a civilian 4090 and getting GPT-4-quality performance in blind tests on Saturday morning.
      - sailingparrot 3y ago
        Google is trying to catch up with OpenAI, not with open source 7B models.
gnabgib 3y ago
Previously: OpenAI Rival Mistral Nears $2B Valuation with Andreessen Horowitz Backing (6 days ago, 2 points, 0 comments)[0], OpenAI Rival Mistral Nears $2B Valuation with Andreessen Horowitz Backing (5 days ago, 9 points, 1 comment)[1] [0]: https://news.ycombinator.com/item?id=38522873 https://news.ycombinator.com/item?id=38522873 [1]: https://news.ycombinator.com/item?id=38533725 https://news.ycombinator.com/item?id=38533725

107 points by admp 3y ago

what is their moat over other companies releasing "open-source" models, a la stability and meta? all the fanfare during their inception seems to have washed away tbh.

mg 3y ago

Do LLM providers with smaller / less capable LLMs still have a USP or will it be like what we witnessed in the search space, where the #1 company has more than 90% market share, #2 has less than 5% market share and everyone else less than 1%?

lelag 3y ago
If the model is closed and served through an API, probably not. But at the moment Mistral seems to focus on smaller and efficient models that you can fine-tuned and run on your own infrastructure. Mistral 7b was free but if they come up with larger capable models, they can offer them with a non-commercial free license and a sell a commercial license for companies that need to fine-tune it. Looking at the performance of small fine-tuned models on specific tasks, I think most company will prefer taking that route rather than trusting OpenAI with their own data, the market is huge I think.
- jgalt212 3y ago
  > trusting OpenAI with their own data often that's a GDPR violation. there is not such thing as perfect data scrubbing.
  - IanCal 3y ago
    You can definitely transfer data to another processor without breaking GDPR rules, even US ones. You need to take care, but it doesn't require perfect data scrubbing.
- mark_l_watson 3y ago
  My hope, being fair to Mistral, is that they keep supporting the 7B model series, and release a set of products such as: one off non-commercial and commercial model purchases of more capable 13B, 20B, and 30B models. I would also like to see a subscription service so, if for example, I was using 13B commercially, I could pay a yearly fee for updates and support for that model. EDIT: since I am retired and just obsess about LLMs and in general deep learning as a ‘gentleman scientist’ I hope Mistral has inexpensive non-commercial options for larger, more capable, and commercially supported models.
- mg 3y ago
  Hmm... I would think the on-premise search for companies falls within the "less than 1% for everybody else" category I mentioned in regards to search. The revenue of Elastic Search is less than 0.5% of Google's revenue.

vinni2 3y ago

https://archive.ph/2023.12.08-165411/https://www.ft.com/content/ea29ddf8-91cb-45e8-86a0-f501ab7ad9bb https://archive.ph/2023.12.08-165411/https://www.ft.com/cont...

cicce19 3y ago

Does anyone know the advantage of mistral over llama2? I understand the founders of mistral previously worked on llama.

vinni2 3y ago
Mistral 7b version is slightly better than 7b llama based on elo rating https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
- nuz 3y ago
  Also less censored
  - vinni2 3y ago
    there is also llama uncensored https://ollama.ai/blog/run-llama2-uncensored-locally https://ollama.ai/blog/run-llama2-uncensored-locally
- MacsHeadroom 3y ago
  It's better than that. The best Llama-2 finetune is 88th on the 7B leaderboard, behind 87 Mistral finetunes. Mistral 7B is above every Llama-2 finetune, including the 10x larger llama-2 70B, on the Open LLM Leaderboard. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... (Note: Check "architecture" and uncheck precision "?" then ignore "tigerbot" as it was disqualified for cheating.)

shin_lao 3y ago

Isn't the recent AI regulation voted in Europe a problem for this company?

eurekin 3y ago
With that valuation, they could pick any country they like
TacticalCoder 3y ago
> Isn't the recent AI regulation voted in Europe a problem for this company? Maybe not that big of a problem but the EU legislation that just passed wants the companies building models to hand over technical documentation, comply with EU copyright laws and give detailed information about the data used for training the models. So basically I take it's there's going to be some annoying administrative work to do but it doesn't seem impossible to comply with.
sailingparrot 3y ago
"The European Union's landmark Artificial Intelligence (AI) Act may exempt open-source models from strict regulation, according to a leaked compromise proposal seen by Reuters." [1] And Mistral has been betting on an open-source first approach. One might even wonder if there is some causation between the two decisions, one direction or the other. [1]: https://www.reuters.com/technology/eus-ai-act-could-exclude-open-source-models-regulation-2023-12-07/ https://www.reuters.com/technology/eus-ai-act-could-exclude-...
- mark_l_watson 3y ago
  The EU would have to be crazy to not support smaller local European companies like Mistral (a French company). So good for their economy and securing their own AI infrastructure. I have ranted about this before on HN, so bear with me here: all countries should take reasonable steps that they can afford to secure their own infrastructure: energy, water, food, some local manufacturing, IT, AI, education, etc. This “one world order” stuff is bull-shit. The world is a better place with many cultures, ethnicities, more self-sufficiency, etc. My wife and I like to travel to meet people with very different backgrounds, see new places. I cringe thinking of a “vanilla” world where large corporations and elite interests “homogenize” the world.

Proven 3y ago

[dead]

monlockandkey 3y ago

What is the end game for companies releasing open source models? Any ideas for a sustainable business?

IanCal 3y ago
I know someone will hound me on the definition of open source but... I'd bet on one of four things 1. Early access for a fee. Get the latest model at a cost, they're later fully open sourced. 2. Sell commercial use licenses. Get and validate the model works for you easily, then buy a license. Split by company revenue if you want to 3. Sell hosted versions - these things require annoying tech to host, paying for someone else to manage running machines can help (possibly combined with a license around not setting up a resale of the LLM) 4. Sell the service of producing (and possibly hosting) custom models. Prove your worth with the open source ones, so you're a trusted person to come to in order to get a model custom made.
WhiteNoiz3 3y ago
I've also wondered this. Aside from just securing funding, I could see: 1. Offering a paid hosted solution down the road 2. Offering bigger / better models that cost money to use commercially 3. Consulting services around their models 4. Licensing their models to companies that want to host them as part of a paid offering IMO the whole advantage of small, open source models is to avoid paying for hosted solutions and additional privacy / security, so it is a bit at odds with commercialization.
Art9681 3y ago
The potential for AI is one of those things that I think a lot of people are willing to fund without a return on their investment. The nations and businesses and individuals that leverage this technology will far outpace those that don't. It has value in and of itself to have access to the most powerful model capable of existing. There is a reason governments are starting to pass laws. Clearly this technology is seen as a huge asset to anyone who's paying attention. I am not making a moral argument here. I don't know if what's coming is good or bad. But it's coming with our without my approval. To add to this, it is a valid strategy to give away something for free so attention - and therefore power - is taken away from a competitor. A quick way for everyone to forget OpenAI exists is for another nation or business conglomerate to release a model that is "good enough". This slows growth for businesses trying to profit from it.

deleted 3y ago

[deleted]

ilaksh 3y ago

Did they release the source code or benchmarks for their new 8x model?

espadrine 3y ago
No. Without the inference code, the best we can have are guesses on its implementation, so the benchmark figures we can get could be quite wrong. It does seem better than Llama2-70B in my tests[0], which rely on the work done by Dmytro Dzhulgakov[1] and DiscoResearch[2]. But the point of releasing on bittorrent is to see the effervescence in hobbyist research and early attempts at MoE quantization, which are already ongoing[3]. They are benefitting from the community. [0]: https://twitter.com/espadrine/status/1733505332293288256 https://twitter.com/espadrine/status/1733505332293288256 [1]: https://github.com/dzhulgakov/llama-mistral https://github.com/dzhulgakov/llama-mistral [2]: https://huggingface.co/DiscoResearch/mixtral-7b-8expert https://huggingface.co/DiscoResearch/mixtral-7b-8expert [3]: https://github.com/TimDettmers/bitsandbytes/tree/sparse_moe https://github.com/TimDettmers/bitsandbytes/tree/sparse_moe
mrkramer 3y ago
Apparently you can download it via torrent[0]. https://venturebeat.com/ai/mistral-ai-bucks-release-trend-by-dropping-torrent-link-to-new-open-source-llm/ https://venturebeat.com/ai/mistral-ai-bucks-release-trend-by...
- capableweb 3y ago
  First-hand source: https://twitter.com/MistralAI/status/1706877320844509405 https://twitter.com/MistralAI/status/1706877320844509405
emadm 3y ago
Mistral doesn't release source code or data just weights. They did include some details of sliding window attention previously
- ebalit 3y ago
  They released a reference implementation with Mistral 7B and a fork of a framework they probably used to train the Mixture of Experts for the new one. But it's still early and I guess that they kept just enough mystery to kickstart the conversation around this new model!

htrp 3y ago

Can someone explain to me what they're trying to do now? They went from we're going to create a European competitor to OpenAI to we have a 7bn parameter model? From their pitch deck (https://sifted.eu/articles/pitch-deck-mistral https://sifted.eu/articles/pitch-deck-mistral) "By creating mistral.ai, we intend to train state-of-the-art models with counter-positions to closed-model current offerings. Our vision is to become a leading actor in the field, while developing a very valuable business around integrating these models in the European industry and beyond."

evanchisholm 3y ago
Have you used any of the Mistral 7b models? They're really good, I think the 7b model just shows what they're capable of. They also dropped an 8x7b mixture of experts model yesterday, that should be very interesting to try.
- wenc 3y ago
  I have, through Ollama, and it would say their truthfulness is not great. You almost have to treat them with kid gloves and prompt them a certain way to get reasonable results (they're harder to steer than larger models). There's a reddit thread on this where people share tricks on how to handle 7B models. https://www.reddit.com/r/LocalLLaMA/comments/18e929k/prompt_engineering_for_7b_llms/ https://www.reddit.com/r/LocalLLaMA/comments/18e929k/prompt_... My feeling is 7B models should only be used for domains where reasoning and factual correctness are not needed, like summarization or generating creative variations of ideas. Code generation is ok too if one is willing to babysit it. But otherwise, the answers can be misleadingly bad. Even reasoning fine-tuned models like Orca2 7B do very badly on simple math questions, despite Chain of Thought prompting.
deleted 3y ago
[deleted]
ilaksh 3y ago
They just released a much larger model as a torrent. Probably the first open source mixture-of-experts model. Waiting for them to drop the code and explain it.

mrkramer 3y ago

This is like dot-com bubble all over again...."we have this fancy and powerful new technology, you must invest in us." Good luck competing with Microsoft, Facebook and Google, because they are super-focused now on LLMs and generative AI and they are not going to stop.

xvector 3y ago
Mistral's models outperform all of theirs at much smaller sizes and can be self-hosted.
- mrkramer 3y ago
  Right now, but how long does it take for OpenAI, Facebook and Google's Bard/Gemini to catch up?
  - capableweb 3y ago
    Considering Google been lying in their marketing videos about their AI capabilities, OpenAI seems to be falling apart from the inside and Facebook refuses to release open (not commercially-locked "open") models, there seems to be space for at least one honest and open AI company.
  - baq 3y ago
    Google is trying and just can't? They are pretending they aren't behind despite setting a pile of cash on fire by spending exorbitant amounts on exaggerating their capabilities in basically fake videos. Meanwhile you see Mistral casually dropping magnet links to weights with barely any instructions on how to use them on a Friday afternoon and you see reports of some very happy people finetuning on a civilian 4090 and getting GPT-4-quality performance in blind tests on Saturday morning.
    - sailingparrot 3y ago
      Google is trying to catch up with OpenAI, not with open source 7B models.

gnabgib 3y ago

Previously: OpenAI Rival Mistral Nears $2B Valuation with Andreessen Horowitz Backing (6 days ago, 2 points, 0 comments)[0], OpenAI Rival Mistral Nears $2B Valuation with Andreessen Horowitz Backing (5 days ago, 9 points, 1 comment)[1] [0]: https://news.ycombinator.com/item?id=38522873 https://news.ycombinator.com/item?id=38522873 [1]: https://news.ycombinator.com/item?id=38533725 https://news.ycombinator.com/item?id=38533725