15 ms·
Mistral: Our first AI endpoints are available in early access
- georgehill 3y ago> Mistral-medium. Our highest-quality endpoint currently serves a prototype model, that is currently among the top serviced models available based on standard benchmarks. This is interesting. This model outperforms ChatGPT 3.5. I'm not sure what type of model it is, and it is not open-sourced.
- eurekin 3y agoAh! So that what, we're using in ollama is named the tiny version? > Mistral-tiny. Our most cost-effective endpoint currently serves Mistral 7B Instruct v0.2, a new minor release of Mistral 7B Instruct. Mistral-tiny only works in English. It obtains 7.6 on MT-Bench. The instructed model can be downloaded here.
- rrsp 3y agohttps://docs.mistral.ai/platform/pricing https://docs.mistral.ai/platform/pricing Pricing has been released too. Per 1 million output tokens: Mistral-medium $8 Mistral-small $1.94 gpt-3.5-turbo-1106 $2 gpt-4-1106-preview $30 gpt-4 $60 gpt-4-32k $120 This suggests that they’re reasonably confident that the mistral-medium model is substantially better than gpt3-5
- dcastm 3y agoIf you take input tokens in consideration is more like 5.25 eur vs. 1.5 eur / million tokens overall. Mistral-small seems to be the most direct competitor to gpt-3.5 and it’s cheaper (1.2 eur / million tokens) Note: I’m assuming equal weight for input and output tokens, and cannot see the prices in USD :/
- stavros 3y agoDoes the 8x7B model really perform at a GPT-3.5 level? That means we might see GPT-3.5 models running locally on our phones in a few years.
- anon373839 3y agoThat might be happening in a few weeks. There is a credible claim that this model might be compressible to as little as a 4GB memory footprint.
- stavros 3y agoYou mean the 7B one? That's exciting if true, but if compression means it can do 0.1 token/sec,it doesn't do much for anyone.
- infecto 3y agoNot true. Not everyone is building chat bot or similar interface that requires output with latency low enough for a user. While your example is of course incredibly slow, there are still many interesting things that could be done if it was a little bit quicker.
- stavros 3y agoWhat kind of use cases run in an environment where latency isn't important (some kind of batch process?) but don't have more than 4GB of RAM?
- yzydserd 3y ago"endpoints are available in early access" is in reality "we have a waitlist (of unspecified length) for early access to endpoints" When I try to access: “Access to our API is currently invitation-only, but we'll let you know when you can subscribe to get access to our best models.”
- ur-whale 3y agoThis actually begs the question: Does anyone know the kind of actual infrastructure something like gpt4-32k actually run on? I mean when I actually type something in the prompt, what actually happens behind the scenes? Is the answer computed on a single NVidia GPU? Or is it dedicated H/W not known to the general public? How big is that GPU? How much RAM does it have? Is my conversation run by a single GPU instance that is dedicated to me or is that GPU shared by multiple users? If the latter, how many queries per seconds can a single GPU handle? Where is that GPU? Does it run in an Azure data center? Is the API usage cost actually reflective of the HW cost or is it heavily subsidized? Is a single GPU RAM size the bottleneck for how large a model can be? Is any of that info public ?
- Closi 3y agoWhile we can't be sure of most of those answers, they have stated it is running in Azure. Also we can probably assume the pricing is likely to be somewhat in proportion to the cost to run (possibly subsidised to gain market, but they are unlikely to be taking a giant/unsustainable loss per query here, particularly as they seem to announce price decreases when they increase model performance).
- amir734jj 3y agoAzure VMSS (uniform orchestration) + 2000 to 3000 GPU enabled servers. I'm not sure about what kind of GPU is on these servers.
- dataking 3y ago> Is the answer computed on a single NVidia GPU? Most likely given that one of their open positions for a GPU programmer includes > high technical competence for writing custom CUDA kernels and pushing GPUs to their limits. Edit: only narrows it down to NVidia hardware, IDK if single GPU or not.
- willyyr 3y agoMark Russinovich shares some of it in this recent Ignite session: https://ignite.microsoft.com/en-US/sessions/49347847-9ae4-430b-952f-44df08595c4a?source=/speakers/19ea1daf-562e-4510-9426-cf5943f8ae54 https://ignite.microsoft.com/en-US/sessions/49347847-9ae4-43... *I work at Microsoft but have nothing to do with the datacenter engineering or other insights into the details behind it.
- tarruda 3y ago> Mistral-embed, our embedding endpoint, serves an embedding model with a 1024 embedding dimension. Our embedding model has been designed with retrieval capabilities in mind. It achieves a retrieval score of 55.26 on MTEB. Is there any information if this embedding model is or will be open source?
- infecto 3y agoUntil we can verify, I think it’s safe to place the in the smoke category. It’s invite only so until it hits GA it’s impossible to know if the pricing is real and the true capabilities of what they are offering.
- jacquesm 3y agodupe, see: https://news.ycombinator.com/item?id=38598559 https://news.ycombinator.com/item?id=38598559
- codetrotter 3y agoThey are two different news about one thing. Not dupe.
- InCityDreams 3y ago>They are two different news about one thing. Then they'd be differing opinions?
- jacquesm 3y agoBy the same submitter, minutes apart and both on the homepage.
- deleted 3y ago[deleted]
- georgehill 3y ago> Mistral-Medium outperforms GPT-4 in Winogrande benchmark 88% vs 87.5% from: https://twitter.com/yupiop12/status/1734137238177698106 https://twitter.com/yupiop12/status/1734137238177698106
- ingojoseph 3y agoIt's interesting that many platforms, like Lemonfox.ai, offer Mistral finetunes at lower prices. They also already announced a Mistral 8x7B API. This raises the question of whether they'll still publish future models as open-source (like the Medium version) if they want to make money.
- jumpCastle 3y agoWithout open weights why would anyone care about them? By the time they could compete with gpt4 there's probably be gpt5 already.
- Palmik 3y agoThis is extremely impressive if benchmarks translate to real-world performance [1]. The mistral-medium beats GPT3.5 and also Gemini Pro (Google's best available model) with a huge margin on all available comparable benchmarks: https://screenbud.com/shot/c0d904e3-24a3-4c23-a1e4-2f18bc0215cf/image.png https://screenbud.com/shot/c0d904e3-24a3-4c23-a1e4-2f18bc021... [1] I would expect real world-performance gap to be even larger if Mistral 7B is anything to go by. The fact that safety filters are opt-in is a huge benefit (even for safe applications).
- monlockandkey 3y agoThis is crazy. If they release this new model as open source, that would be very exciting!
- Aerbil313 3y agoThey already did, weights on Twitter.
- JimDabell 3y agoNo, they’ve released the weights for Mistral-small. They haven’t released the weights for Mistral-medium.
- samtheprogram 3y agoThey might’ve been referring to mistral-medium, the larger version, which is not open source. The open-sourced version was the small variant.
- usaar333 3y agoHigher Winogrande score than GPT-4. Surprising given others are more GPT-3ish
- logicchains 3y ago>also Gemini Pro (Google's best available model) Pretty sad for Google if their next big AI thing is already being beaten by a small company with a tiny fraction of their resources.
- ianpurton 3y agoSo, what would be the hardware setup for this? Can it run on 1 GPU and swap between experts.
- LanzVonL 3y agoTwo out of the top three stories today on HN! What an achievement. Is mistral.ai a YC property?
- marviel 3y ago> Our API follows the specifications of the popular chat interface initially proposed by our dearest competitor. I like it, also made me laugh
- mark_l_watson 3y agoI just signed up for the API waiting list. I have been enjoying running Mistral-7B on my home system, and it feels right to give them some of my paid for API business.
- rgbrgb 3y ago> Mistral-tiny. Our most cost-effective endpoint currently serves Mistral 7B Instruct v0.2, a new minor release of Mistral 7B Instruct. Mistral-tiny only works in English. It obtains 7.6 on MT-Bench. The instructed model can be downloaded here. "download here" link is to v0.1 [0]. Oversight or are they holding back the state of the art tiny model? [0]: https://huggingface.co/mistralai/Mistral-7B-v0.1 https://huggingface.co/mistralai/Mistral-7B-v0.1
- Evan-Almloff 3y agoIt's here: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
- rgbrgb 3y agohell yea!
- ComputerGuru 3y agoI’m surprised no one has commented on the context size limitations of these offerings when comparing to the other models. The sliding window technique really does effectively cripple its recall to approximately just 8k tokens which is just plain insufficient for a lot of tasks. All these llama2 derivatives are only effective if you fine tune them, not just because of the parameter count as people keep harping but perhaps even more so because of the tiny context available. A lot of my GPT3.5/4 usage involves “one offs” where it would be faster to do the thing by hand than to train/fine-tune first, made possible because of the generous context window and some amount of modest context stuffing (drives up input token costs but still a big win).
- logicchains 3y ago>The sliding window technique really does effectively cripple its recall to approximately just 8k tokens which is just plain insufficient for a lot of tasks. What are you basing this observation on; personal experience, or is there a benchmark somewhere confirming it?
- ComputerGuru 3y agoReal world testing and experience. If you only need the llm to retain the "gist" of the input tokens in order to return a "related" answer, the sliding window design is fine. But if you need actual technical analysis or tasks that involve verbatim referencing, quoting, recomposing, etc based off parts of the input documents, it doesn't work. I tried using it for "business document" use cases but have ran into this with code as well; the latter might be a better explanation given where we're having this discussion. If you only need the llm to retain the general shape of your inputs so it can reuse them to influence the output, the sliding context is fine. But if you need it to actually reuse code verbatim from the input that you fed it (or to remember the api calls and their surrounding context verbatim to recall from a sample of just one that this api must be called before that api, when the prompt includes instructions to that effect) the "decomposition" of the input tokens with the sliding model is insufficient and the llm completely fails at the assigned task.
- cameroncairns 3y ago
- brandall10 3y agoI'm surprised this isn't firmly attached to the top of HN right now for the entire day. This is a tiny company (appears 30 or so people?) that just scored a 2B valuation, produced easily the most performant 7B model and a 7B*8 MOE model that performs at the level of a 70B requiring the inference power of a 14B. I feel this could be a potential bigger threat to OpenAI than Google or Anthropic. I gather with the huge recent investment they'll be able to a) scale out to a reasonable traffic load in the near future and b) attract the best and brightest researchers put off w/ various chest puffing and drama that has been front and center in this industry.
- huevosabio 3y agoAgreed. By far the most impressive company out of the current AI wave. They managed to put out the gold standard for 7B models in like 6 months, and are quickly moving up the scale. I mocked the funding round back in March as being signs of hype, ($300m for a team of 3 with just an idea?) but clearly I didn't know the details. Really remarkable execution. They may well be on their way to eat all use cases that don't need gpt-4 performance and hopefully soon tackle the big leagues as well. Exciting times!
- appplication 3y agoIn some ways I see this as a sign that LLMs possibly aren’t that hard to iterate on now that the cat is out of the bag. A small team can push the state of the art in just 6 months? It feels like this is more a circumstance of the state of LLMs presently and less a statement on Mistral. I wouldn’t it be surprised if we see 10+ other new $1B+ companies spring up because of how much room for improvement there is in the space.
- a_wild_dandan 3y agoWell said. OpenAI's biggest contribution wasn't serving GPT-4, it was proving that big LLMs are AGIs. Rushing to gold is easy; discovering it ain't. Since we've found a correct path, low-hanging fruit is plentiful. Throw more compute at it. Work smarter not harder (optimize). Graft on simple ideas for big gainz (e.g. RAG, MoE, chain-of-thought, etc). We moved from "GPT Who?" to...all of this, in a year. I now run an AGI at a bookish 13-year-old's level on my MacBook. The trend of hardware/software advances are astonishing. And this is just the beginning.
- davidkunz 3y agoWell done, Mistral! "Show, don't tell" par excellence.
- hospitalJail 3y agoIs there anything Mistral + tuned on ChatGPT4?
- deleted 3y ago[deleted]
- munro 3y agoWow, beating ChatGPT-3.5 is really an accomplishment. Congrats! That's literally the default of OpenAI's product. I had to fallback to GPT-3.5 the other day because I ran out of usage on ChatGPT-4 (playing 20 questions lol). So I really hope someone can come up on GPT-4! For me GPT-3.5 isn't good enough for daily things, it gets too much wrong.
- lioeters 3y agoBy chance I noticed that Fabrice Bellard's TextSynth server has newly added support for Mistral 7B model. > 2023-10-21: CUDA support in the Windows version, mistral model support. Speculative sampling is supported. BNF grammar and JSON schema sampling. > mistral_7B_instruct_q4 - 3.9GB - Mistral 7B chat model https://bellard.org/ts_server/ https://bellard.org/ts_server/
- nojvek 3y agoCompetition is how the world moves forward. I'm super glad small and big players have competitive models. The thing that makes me a bit sad is how announcements show the benchmarks but they way they test is tweaked to make their metrics favorable. They aren't apple to apple benchmarks across different paper publications. Super grateful that they openly share the weights and code with Apache license. 25 shot - is having 25 tries and selecting the best answer. Is anyone working on an open benchmark where they take the major models and compare them apples to apples.