4 ms·
MistralLite by Amazon Web Services
- brucethemoose2 3y agoYeah, this is a great finetune that kinda flew under the radar. It will indeed take 32K context without hacky context extension (and theoretically should work OK stretched to 48K). It still behaves more like an good unaligned LlamaV2 13B finetune. And it loves to summarize and retrieve facts. In fact, when I give it a huge context and instruct it to storytell, deduct or whatever, sometimes it will summarize the context instead... But the summary is excellent.
- hfjjbf 3y agoIn my experience llama2 13b is fairly worthless in comparison to 70b. I think L70b is the mark to beat, I’m hopeful mistral will take the crown with a smaller model.
- brucethemoose2 3y agoMistral 7B is way better than LlamaV2 13B chat. But some of the 3rd party 13B finetunes/merges are also much better that Meta's chat model.
- hfjjbf 3y agoIf a Mistral 13B beats Llama2 70B it would be a game changer.
- behnamoh 3y agoWhat is Amazon doing in this space? I feel like they have the infra and skills to train their own model and serve it but I don’t see them creating anything.
- mattnewton 3y agoThey’re selling a lot of steel and wooden handles to the shovel makers right now. No doubt if they see an open design like this one they like they’ll iterate on it and offer it too, that seems to be the “bedrock” branded stable diffusion and similar. Maybe they think, why train the foundational model from scratch when you can get some company to rent your sever to do it, and then license it to you for free anyways?
- ttul 3y agoPartly they do this to keep their ML researchers entertained after the glitz of their $1M compensation wears off. And partly they do this to develop products for their customers. Although they partnered with Anthropic, whose arguably SOTA models are vastly superior, AWS has an extremely diverse set of customers with simpler requirements that may be quite thoroughly satisfied with a 7B parameter model.
- brucethemoose2 3y agoA 7B model must be cheaper and faster than Anthropic's huge model. To me, it feels good enough for retrieval/summarization tasks Amazon could sell, especially with grammar based sampling.
- brucethemoose2 3y agoYou can say this about everyone. Stable Diffusion made a fuss a year ago. Everyone is hoarding A100s and H100s like they are the last GPUs ever made, and we have... Maybe 5 good downloadable base model LLMs, and ~4 good diffusion models? And only one from a really large company. shrug
- albert_e 3y agoAmazon is a bit late to the party but they are investing heavily to stay on the podium. They announced BedRock and made it GA directly after a gated preview of just a couple of months. Titan is their trained model. November is their annual gala conference time. Expect a lot of GenAI announcements resounding through late November.
- ReactiveJelly 3y agoSo... this won't work with llama.cpp right? I need a different runtime for it? I'm new to this LLM stuff.
- brucethemoose2 3y agoYeah it works fine. Mistral is a llama architecture llm. There are quantizations already. I would suggest running it on KoboldCPP, or any of the UIs recommended on this page: https://huggingface.co/TheBloke/MistralLite-7B-GGUF https://huggingface.co/TheBloke/MistralLite-7B-GGUF Personally, I run the elx2 quantization with exllamav2 because its faster and more coherent in the same amount of vram. You can do that with Aphrodite or Text-gen-ui: https://huggingface.co/models?search=MistralLite%20exl2 https://huggingface.co/models?search=MistralLite%20exl2
- hfjjbf 3y agoThis account has prepackaged and quantized models for everything, really awesome. https://huggingface.co/TheBloke https://huggingface.co/TheBloke
- brucethemoose2 3y agoIndeed. But TheBloke doesn't (yet) do exl2 quantization. That one is almost better done yourself, as the bpw target is arbitrary, and requires profiling text that should probably fit your intended use (though many just use wikitext as a good default).
- simonw 3y agoCan anyone explain the prompting format for this? <|prompter|>{prompt}</s><|assistant|> Why is that </s> in there? And how would I prompt it with a previous history of the conversation?
- simonw 3y agoI tried running a prompt through it just now: <|prompter|>hello</s><|assistant|> And got back this (truncated): hey there!</s> introducing the first of its kind, a revolutionary AI... Why did that </s> come back as part of the response?
- whoevercares 3y agoHow does SageMaker compare to other hosting solutions? IIRC it was quite high touch yet not very flexible in 2022.
- lappa 3y agoExcellent, appears Amazon has introduced two important things here: - Rope theta of 100,000, likely from the Llama 2 Long paper which found that a large theta helped regulate attention between distant tokens[0] - A 16k (effective 32k) context window, improving upon Mistrals 4k (effective 8k) context window In The Llama 2 Long paper, they saw improvement in short context benchmarks as a result of long context fine tuning. I can't find any of the expected MMLU / HellaSwag / etc benchmarks yet. Benchmarks haven't been submitted to MTEB yet. Some user anecdotally seem to be having trouble with generating quality responses [2][3][4]. I can't find any examples of users getting good results from the model outside of using exact examples from the documentation. [0] https://arxiv.org/pdf/2309.16039.pdf https://arxiv.org/pdf/2309.16039.pdf [2] https://old.reddit.com/r/LocalLLaMA/comments/17jd00g/mistrallite_not_able_to_summarize_text/ https://old.reddit.com/r/LocalLLaMA/comments/17jd00g/mistral... [3] https://old.reddit.com/r/LocalLLaMA/comments/17kzlbl/anyone_get_amazons_longcontext_mistrallite_to/ https://old.reddit.com/r/LocalLLaMA/comments/17kzlbl/anyone_... [4] https://old.reddit.com/r/LocalLLaMA/comments/17b0n8t/llama_2_7b_32k_instruct_summarizes_and_outlines/ https://old.reddit.com/r/LocalLLaMA/comments/17b0n8t/llama_2...
- brucethemoose2 3y agoSummarization works very well for me, but I was testing huge blocks of stories or conversations with 10K+ context.
- syntaxing 3y agoTangentially related, is 13B on the horizon? Misrtral 7B has been such a good all rounder.