14 ms·
Mistral CEO confirms 'leak' of new open source AI model nearing GPT4 performance
- MallocVoidstar 3y agoCheck out the PR he submitted: https://huggingface.co/miqudev/miqu-1-70b/discussions/10/files https://huggingface.co/miqudev/miqu-1-70b/discussions/10/fil...
- rrr_oh_man 3y agoCynical me smells a PR move
- seydor 3y agoAll their PR moves seem to be like this one. Not complaining
- rubymamis 3y agoTweet by Mistral CEO: https://x.com/arthurmensch/status/1752737462663684344 https://x.com/arthurmensch/status/1752737462663684344 > An over-enthusiastic employee of one of our early access customers leaked a quantised (and watermarked) version of an old model we trained and distributed quite openly. > To quickly start working with a few selected customers, we retrained this model from Llama 2 the minute we got access to our entire cluster — the pretraining finished on the day of Mistral 7B release. > We've made good progress since — stay tuned!
- Timon3 3y agoI love their approach! Mistral seems to be what I long ago hoped OpenAI to be.
- unshavedyak 3y agoMakes me wonder if i should be giving my money to them instead of OpenAI. Do they have a Chat Interface like ChatGPT? Without first signing up, it's looking like they only have endpoints?
- yieldcrv 3y agoLM Studio I use dolphin mixstral 8x7B way more than ChatGPT4 for the past several months if any of this sentence makes sense, I have a M1 with 64gb RAM and I use 5 bit quantizing with metal with 10,000 token context window. Its around 21 tokens/sec which is a little faster than the text speed that ChatGPT4 responds with
- josephg 3y agoWhat is quality like compared to GPT4?
- yieldcrv 3y agofor the kinds of discussions I have it is very close I often have discussions about political topics that I don’t have a complete history on, things that are hard to get a non-emotional non-accusative response from in a forum or in person, licensing questions about obscure professions, liability questions, roleplays without the preachiness about the topic, coding, brand ideas and naming Lots of stuff that I dont want in any cloud or sent online, but then it just became second nature to primarily use it. the hallucinations are heavier. like it will give you specific links that it made up, and never apologize like chatgpt will, it will say “no, thats a real link” so, much more of a bullshitter LM Studio makes it easy to change the temperament though with a variety of premade system prompts, or allowing custom ones very easily I mainly use ChatGPT4 for multimodal like audio conversations, figuring a DIY problem out by sending it a photo of what I’m looking at, having it render photos on how to use something - I was at a gym and had it look at every piece of equipment there and tell me what it was and show me how a human would use it (I notice this is at odds with another comment, I havent used ChatGPT3.5 in nearly a year, but my experience with ChatGPT4 on the aforementioned topics is similar to my output in mixtral)
- moralestapia 3y agoI can answer that from experience (just built a couple Q&A style sites), although a bit subjectively. Compared with the GPTs, Mixtral (and derivatives) perform: * 5% of the time as good as GPT4 * 75% of the time on par with GPT3.5 * 20% of the time a bit worse than GPT3.5 (too chatty and hallucinates with ease, although one could argue a better prompt could improve things a lot) Advantages of Mixtral, for me: * Cost 10-100x cheaper * Faster completion time * More deterministic output, in the sense that if you run the same query several times you get the same answers back (but they could be wrong). GPT almost always gives me an answer of great quality BUT with a lot of variance across them; even with temperature set to zero. This is a PITA when you want some sort of predictable, structured output like a JSON object.
- lordswork 3y agoDoes watermarked imply they can figure out which customer leaked it?
- danpot 3y agoHow do they even watermark a LLM?
- pjerem 3y agoMake it learn that woziboza is a green cloud shaped like a washing machine ?
- coolspot 3y agoWell, this is common knowledge, duh. Just google it.
- declaredapple 3y agoYou can either modify the model weights in a way that doesn't cause any real differences (change a few bits somewhere should be enough), or you could watermark the actual text output. Here's a list of research for watermarking LLMs. https://github.com/hzy312/Awesome-LLM-Watermark?tab=readme-ov-file https://github.com/hzy312/Awesome-LLM-Watermark?tab=readme-o...
- einsum 3y agoScott Aaronson did some work on statistical watermarking with OpenAI: https://www.scottaaronson.com/talks/watermark.ppt https://www.scottaaronson.com/talks/watermark.ppt
- sangnoir 3y agoFinetune your standard model on an small, unique set of made-up shibboleths (one per customer) before distributing? Ask the candidate model where the Fribolouth caves are located.
- haolez 3y agoIt's either that or they can narrow it down to a few people.
- EasyMark 3y agoSeems like they have inferior secops on their end. I tried to give it a go and got "an error occurred" in both brave and firefox. Some message about a cookie wasn't found even though I was logging in with my google id which I use flawlessly with a bunch of other services. I have too many other fires to put out so I'll pass on this one I guess.
- janalsncm 3y ago> Quantization in ML refers to a technique used to make it possible to run certain AI models on less powerful computers and chips by replacing specific long numeric sequences in a model’s architecture with shorter ones. It’s always amusing when the press tries to explain technical concepts. Quantization just means substituting high precision numeric types with lower precision ones. And not specific “numeric sequences”, all numbers.
- SilverBirch 3y agoIt's funny how the original phrase sounds like it was generated by chatgpt. I'd guess the reason they didn't phrase it how you suggest is they probably thought that people who don't know what quantization means aren't going to be happy if you throw precision numerics at them. I don't knwo why they didn't just say "some of the numbers are rounded" though.
- bsaul 3y agoGPT-4 has been out for almost a year now, and it seems that the frantic pace of open-ai releasing new groundbreaking tech every month has come to a halt. Anyone knows what's happening with open AI ? has the recent turmoil with sama caused lag in the company ? or are they working on some superweapon ?
- artninja1988 3y agoProbably getting bogged down by endless safety testing and paperwork by now
- harmmonica 3y agoMaybe tangential to this comment, but I had been using 3.5 to write contracts, but a couple of weeks back, even when using the exact same prompts that would've worked in the past, 3.5 started saying, to paraphrase, "for any contracts you need to get a lawyer." Anyone else had this experience? It seems like they're actually locking down some very helpful use cases, which maybe falls into the "safety" category or, more cynically, in the "we don't want to be sued" category.
- transcriptase 3y agoAs they tack on more and more guardrails it becomes lazier and less helpful. One way it gets around admitting that it has instructed not to help you is by directing you to consult a human expert. I also suspect there’s some dynamic laziness parameter that’s used to counteract increased load on their servers. You can ask the same prompt over and over in a new chat throughout the day and suddenly instead of writing the code you asked for or completing a task, it will do a small part of the work with “add the code to do xyz here” or explain the steps required to complete a task instead of completing it. It happens with v4 as well.
- TeMPOraL 3y ago> some dynamic laziness parameter that’s used to counteract increased load That's a brilliant risk mitigation mechanism! The AI won't recursively self-improve to superhuman levels if it just keeps getting tired of thinking.
- akokanka 3y ago[flagged]
- summarity 3y agoResults look comparable to Medium indeed (I’m using it via Mistrals API, since I got sick of OpenAI switching their stuff up). Medium is pretty great, somewhere between 3.5-turbo and 4-turbo qualitatively. Would be awesome to have out there.
- seydor 3y agoHow does a world where GPTs are like the latest version of apache or mysql, look like? do we go back to the world of millions of web hosts (sorry, AI hosts)
- Tiberium 3y agoFor some more context see my submission from a few days ago - https://news.ycombinator.com/item?id=39175611 https://news.ycombinator.com/item?id=39175611, although it's admittedly not mistral-medium, but llama2 trained on the same dataset (since the outputs do match quite often with the API)
- whimsicalism 3y agohow do we know it’s not mistral medium
- wut42 3y agoIt has been confirmed by the Mistral CEO: > To quickly start working with a few selected customers, we retrained this model from Llama 2 the minute we got access to our entire cluster — the pretraining finished on the day of Mistral 7B release.
- whimsicalism 3y agoThat does not confirm that it is not Mistral Medium. It is written to give that impression, but if it weren’t Mistral Medium, I would expect them to explicitly say so.
- wut42 3y agoI think it does, as Mistral usually do their own models, and also, they couldn't fully commercialise Mistral Medium if it was LLama 2 based.
- whimsicalism 3y agoGood point. Likely Llama trained on similar corpora to what Mistral medium uses
- _boffin_ 3y agoWhat do you mean? Didn’t llama2’s tos say something about if you had something like 600m users at that current moment in time, you can’t use it? If you don’t, you’re good to go.
- brucethemoose2 3y agoIts smart. Definitely keeping it around as my slow/low context local llm. Seems like a great candiate to merge with other 70Bs as well. There aren't a lot of really great 70B training continuations, like CodeLlama or sequelbox's continuations.
- jstummbillig 3y agoThe collective game of still just playing catch-up to GPT4, which was released a year ago, while having apparently no special sauce and full well knowing that OpenAI could come up with something much better at any point must be really exhausting.
- rubymamis 3y ago> "...it appears that not only is Mistral training a version of this so-called “Miqu” model that approaches GPT-4 level performance, but it may, in fact, match or exceed it, if his comments are to be interpreted generously."
- dougmwne 3y agoGPT-4 is an enormous model that took an enormous amount of training. The big news is that smaller teams are getting close to its performance on a small model that can run on a single GPU. No doubt many of these innovations could be scaled up, but pretty much only Google and Microsoft have the compute resources for the behemoth models(Not just the training, but the giant resources required to run the inference for hundreds of millions of users. Google already claims to have surpassed GPT-4 with their unreleased Gemini Ultra model. No doubt OpenAI/Microsoft is sitting on GPT-5 refining it just waiting to leapfrog the competition.
- dom96 3y agoDon't forget about Meta
- anon373839 3y agoAlso, GPT-4 isn’t actually a model per se. It’s a black-box product that uses a model.
- ethanbond 3y agoMaybe if OpenAI also seemed like it was getting stronger, especially organizationally. But if I were Mistral and following this quickly while OAI was tripping over its own shoelaces… that’s gotta be very exciting.
- mysteria 3y agoWhy is this being called an open source model? This is a proprietary model that has been leaked on the internet, and will remain so until Mistral releases it officially. Like Llama 1 they won't care about personal use, but no corp is going to touch this.
- codetrotter 3y agoPresumably because it is planned to be released as open source
- jallmann 3y agoOpen source is really about reproducibility. Most of these model releases are better described as "open weights" because we don't know how exactly they were trained.
- andy99 3y agoNo, open source is about software freedom, see debian free software guidelines from which the "open source definition" derives. https://wiki.debian.org/DebianFreeSoftwareGuidelines https://wiki.debian.org/DebianFreeSoftwareGuidelines Between these and FSF you've got pretty much all the accepted pontificating about free / open source software. Reproducibility is not mentioned because it's not really a consideration for software. Model weights aren't software so there's not an automatic correspondence between the freedoms, but the essential one you might think you need the training data for is freedom to modify and inspect the source. Modification is fine tuning which you're free to do if you have the weights. And the model weights + code fully define a system that can be interrogated to give a practitioner relevant info about how the model works (within our understanding) that the training data isn't needed or relevant for. I don't see that any freedom on use or inspection is violated by not having the data. It could be nice to have it of course, but it's more about using it to learn how, not exercising any freedom. Incidentally, the big freedom that's usually violated is freedom of discrimination against field of endeavor. LLAMA et al list uses and industries they restrict from using them and because of that are not "open".
- londons_explore 3y ago"Nearing GPT-4"... The leaderboard[1] shows that there is a huge gap between GPT4-0314 and GPT4-Turbo. So if you only just are nearing GPT-4-0314, then you're still a year behind the state of the art. [1]: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
- Terretta 3y agoOr you're ahead of it, since the earlier GPT4 models beat the GPT4-Turbo on a variety of technical use cases.
- YetAnotherNick 3y agoThe leaderboard is easy to hack as well.
- londons_explore 3y agoI think you might be thinking of GPT-4-0613? That was pretty crap all round (but was faster)
- int_19h 3y agoNo, they're correct, Turbo is the one that is noticeably inferior. It shows especially with more complicated answers where it's prone to give you an answer with "... (fill in the blank)" type lacunas exactly where the meat of it is supposed to be.
- rrr_oh_man 3y ago> lacuna 1. an unfilled space; a gap. "the journal has filled a lacuna in Middle Eastern studies" 2. (ANATOMY) a cavity or depression, especially in bone. Thank you for this new word!
- tomtom1337 3y ago
- nohat 3y agoIt has been interesting seeing the sleuthing on this one. IMHO it is unfortunate to have this happen to a company that has been very pro open source.
- deleted 3y ago[deleted]
- bee_rider 3y agoSince these models are trained by just scraping the internet, “scraping” this thing and including it in your own model seems like fair game, right?
- zamadatix 3y agoIf you mean training your model on it I'd say yeah, fair game. Mistral is Apache 2.0 licensed though so the question is a bit moot, they'd like you to use it for your own model.
- speedgoose 3y agoIt sounds fair. Just make sure to not involve lawyers.
- dopa42365 3y agoIs this an advertisement for a "promise"? Entirely worthless until there's something to show. It smells.
- a_wild_dandan 3y agoGuess I'll watch TheBloke's page until I can run his Miqu Q5 quant on my MacBook. Mixtral is my daily driver, and if this (or the newer, official) release nears GPT-4, that's a wrap for my OpenAI subscription. The small team at Mistral is putting their competitors to shame. They're what "Open"AI should've been.
- MallocVoidstar 3y agoFull weights aren't available, only Q2, Q4, Q5 quants via miqudev.
- throwaway9274 3y agoUnquantized model is here: https://huggingface.co/152334H/miqu-1-70b-sf https://huggingface.co/152334H/miqu-1-70b-sf This strikes me as less a leak and more clever marketing from Mistral.
- MallocVoidstar 3y agoThat isn't unquantized, it's de-quantized. They went from Q5 to fp16 for use in Pytorch instead of the GGUF ecosystem.
- Taek 3y agoI never thought people would be upscaling models by increasing quantization precision. The rationale makes sense bit its also a goofy outcome.
- nullc 3y agoYou should be able to upscale and fine tune to recover performance, I suppose! Clearly we should train a diffusion model to denoise the weights of LLM transformer models. Yo dawg.
- throwaway9274 3y ago
- chrishare 3y agoI hate when that happens
- accrual 3y agoWhat's wild to me is that this "leak" won't matter in a couple of months. The official model will come out, then an even better model will come out. It's fun to get hyped but just like every other leak, it'll be surpassed by the real thing and its successor in a little bit. The fast pace of things is what has me excited, not any particular model.
- refulgentis 3y agoI don't think so, TFA says it's an early version of an old model that's already been distributed openly. ;)
- deleted 3y ago[deleted]
- 2OEH8eoCRo0 3y agoSeems paradoxical. How does something that is open source leak?
- vulcan01 3y agoAccording to the article, it will be open source, but it's not open source now.
- sharkjacobs 3y agoMistral reminds me of the good old days of pre-2015 when I thought that tech companies were cool.
- syntaxing 3y agoWhat a class act. Going to the leaked model on huggingface, not demanding it to be taken down, and just making a post on the page saying “Might consider attribution” is so damn amazing.
- deleted 3y ago[deleted]
- adampk 3y agoWhat is the best place to use hosted versions of these models? We want to interact with it same as the OpenAI API.