5 ms·
Guess I'll watch TheBloke's page until I can run his Miqu Q5 quant on my MacBook. Mixtral is my daily driver, and if this (or the newer, official) release nears
by a_wild_dandan 3y ago
Guess I'll watch TheBloke's page until I can run his Miqu Q5 quant on my MacBook. Mixtral is my daily driver, and if this (or the newer, official) release nears GPT-4, that's a wrap for my OpenAI subscription.
The small team at Mistral is putting their competitors to shame. They're what "Open"AI should've been.
- MallocVoidstar 3y agoFull weights aren't available, only Q2, Q4, Q5 quants via miqudev.
- throwaway9274 3y agoUnquantized model is here: https://huggingface.co/152334H/miqu-1-70b-sf https://huggingface.co/152334H/miqu-1-70b-sf This strikes me as less a leak and more clever marketing from Mistral.
- MallocVoidstar 3y agoThat isn't unquantized, it's de-quantized. They went from Q5 to fp16 for use in Pytorch instead of the GGUF ecosystem.
- Taek 3y agoI never thought people would be upscaling models by increasing quantization precision. The rationale makes sense bit its also a goofy outcome.
- nullc 3y agoYou should be able to upscale and fine tune to recover performance, I suppose! Clearly we should train a diffusion model to denoise the weights of LLM transformer models. Yo dawg.
- throwaway9274 3y agoYes, that’s correct. Good correction.
- SparkyMcUnicorn 3y agoLooks like the leaked model is already quants in gguf format, so no need to wait for TheBloke. https://huggingface.co/miqudev/miqu-1-70b/tree/main https://huggingface.co/miqudev/miqu-1-70b/tree/main
- brucethemoose2 3y agoThe GGML quants are the only quantization we have lol. They were leaked in Q2K/Q4KM/Q5KM, you can grab them right now. Whats interesting is that Mistral apparently distributed these GGUFs. This is (in my experience) not a good format for production, so I am curious exactly who was wanting to test a model in GGUF.
- whatwhaaaaat 3y agoIt’s by far the easiest to consume and run format no? Spans devices easy. No weights or extra stuff. For “production” maybe not but to get in into the hands of the masses this seems perfect.
- brucethemoose2 3y agoYeah its by far the least trouble. Pretty much any other backend, even a "pytorch free" backend like MLC, is a utter nightmare to install, and that's if it uses a standardized quantization. However, the llama.cpp server is... very buggy. The OpenAI endpoint doesn't work. It hangs and crashes constantly. I don't see how anyone could use it for batched production as of last november/december. The reason I don't use llama.cpp personally is no flash attention (yet) and no 8 bit kv cache, so its not too great at long (32K+) contexts. But this is a niche, and being addressed.
- samstave 3y agoWhat are you needing 32K contexts for, as use case examples plz. From [0] @sdo72 writes:: >>>...32k tokens, 3/4 of 32k is 24k words, each page average is 500 or 0.5k words, so that's basically 24k / .5k = 24 x 2 =~48 pages...." https://news.ycombinator.com/item?id=35841460 https://news.ycombinator.com/item?id=35841460 EDIT: I may be ignorant: does the 32k mean its output context, or single conversation attention span, or how much it can ingest in a prompt?
- fragmede 3y agoFeed it my source code to use as context for how to refactor things. the bigger the context window, the more source code it can "read".
- htrp 3y ago>The small team at Mistral is putting their competitors to shame. Them plus 500 million in funding
- api 3y agoMuch of the cost is probably compute.
- refulgentis 3y agoVery unlikely: note best GPT-4 estimates put at it $45 million
- a_wild_dandan 3y agoAltman said that GPT-4 training was over $100 million. And you need significant additional resources beside just the training run cost.
- tomp 3y agoso like 10x less funding than OpenAI and 100x less than MSFT, AAPL, Google, Facebook, ...
- jtonz 3y agoI think it's fair to say when you hit the hundreds of millions of dollars mark the diminishing returns for making things happen faster have well and truly kicked in. Perhaps the only benefit would be extra computational power yet I would struggle to understand the benefit of jumping from 500 million to 5 billion with such short timeframes.
- Zuiii 3y agoThe ability to truly not think about training run costs, throw random things on the wall to see what sticks. 10x resources is definitely a competitive advantage in LLM training.
- karmasimida 3y agoNear GPT-4 is definitely a stretch. The hype around Mixtral is huge, and my disappointment follows, it doesn't have very good knowledge on books, for example.
- terhechte 3y agoIt's a much smaller model, it didn't ingest the knowledge of the world. It is great at working with the information you give it, but if you want to extract information like a search engine, GPT4 is the king because it is so much bigger.
- avereveard 3y agoThat is a good thing you want the model to process language, knowledge is a side effect of how it gets there. If you rely on training knowledge it's going to be hard to know when you pass the boundary into hallucinations, what you want instead is a model that can pretend reasoning while supporting tools inject knowledge from the world into the context as it iterate toward an answer
- AtmanSeeker 3y agoI agree this is what we want, we should not waste compute and money on memorizing world knowledge, just enough world knowledge to have the best possible understanding of languages.
- karmasimida 3y agoBut here is the thing, if you rely on RAG or any other knowledge injection for recommendation, you essentially make your LLM parrot machine, no better than a search engine, just much slower. Knowledge is one aspect of it, I found its instruction following ability, frustrating as well. I think Ilya Sutskever puts it very well, larger model brings stability to wider range of tasks, what smaller models are not capable of. Even though book recommendation with GPT might a niche, but it is not something really unexpected TBH, thus comes my disappointment.
- chasd00 3y ago
- rmbyrro 3y agoI'm sure their competitors aren't shamed - as they shouldn't. OpenAI, especially, has done tremendously good work. If it wasn't for them, perhaps Mistral wouldn't be news today? Competitors are highly motivated to improve. This is the good side of free markets, when they work well. And why the fearmongering calling for rent-seeking regulation of AI is shameful.
- lukasm 3y agoWhich model is the best to run on M3 max ATM?
- a_wild_dandan 3y agoThat depends on your goal and available memory. I generally use Mixtral for text, LlaVA for multimodal stuff, and Copilot in VSCode.