30 ms·
Mistral 7B
- pugio 3y agoHow come the blog post doesn't discuss training at all? Which data sets were used? How many tokens was it trained on? I couldn't seem to find a linked preprint or anything else really
- burningion 3y agoFor anyone who missed it, the twitter announcement of the model was just a torrent tracker uri: https://twitter.com/MistralAI/status/1706877320844509405 https://twitter.com/MistralAI/status/1706877320844509405
- yieldcrv 3y agowhat do you use "the most powerful language model for its size" for?
- ShrigmaMale 3y agoProbably a better candidate for local applications? Guessing this was a trial balloon against larger models coming though.
- snowram 3y ago7B models are small enough to be usable on a smartphone, so a local handheld assistant sounds like a use case.
- ComputerGuru 3y agoAre they? Unquantized, Llama 2 7b needs over 14GB of GPU (or shared) memory.
- polygamous_bat 3y ago"Unquantized" is the key word here: with quantization you can get a 4-8x improvement without much performance degradation.
- throeaaysn 3y ago[dead]
- minimaxir 3y ago"usable" is not the same as practical. Even running a quantized and optimized LLM on a smartphone would kill battery life at minimum.
- brucethemoose2 3y agoTry MLC-LLM. Its not as bad as you'd think. In the future(?), they will probably use the AI blocks instead of the GPU, which are very low power.
- linkpuff 3y agoAccording to their website, it's "Optimal for: low latency, text summarisation, classification, text completion, code completion"
- kirill5pol 3y agoOne big one is speculative decoding for larger models, the more “accurate”* the smaller model, the more the speed up on the bigger model * as in matches the token that the larger model would output
- sroussey 3y agoIt would be better if they also had a 70B model for this. They would need the same vocabulary, etc. What else?
- samr71 3y agoThe way the wind's blowing, we'll have a GPT-4 level open source model within the next few years - and probably "unaligned" too. I cannot wait to ask it how to make nuclear weapons, psychedelic drugs, and to write erotica. If anyone has any other ideas to scare the AI safety ninnies I'm all ears.
- coding123 3y agoBut in 3 years we'll have GPT-8 and no one will care about the performance of GPT-4.
- jona-f 3y ago"How to drive as many teenagers as possible into madness?" AI: "Build a website where they can upload pictures of themselves and others can make comments about there appearance."
- peterhadlaw 3y ago.... what a great question to ask... an unaligned AI
- deleted 3y ago[deleted]
- civilitty 3y ago> I cannot wait to ask it how to make nuclear weapons Amen! I’m going to ask it to give me detailed designs for everything restricted by ITAR. Just waiting on my ATF Mass Destructive Devices license.
- gonzo41 3y agoThe construction of the 'bomb' part of a nuclear weapon is the easy part, within reason! The really hard part is the separation science of turning uranium and plutonium into gasses with fluorine with the intent to spin out isotopes and then recrystallize the pure metal for the bomb. I would hope that if you asked chat gpt "How to make a nuclear weapon?" it responded with, "Don't bother it's really hard, you should try and buy off the shelf."
- ShrigmaMale 3y agoGrab the torrent here: magnet:?xt=urn:btih:208b101a0f51514ecf285885a8b0f6fb1a1e4d7d&dn=mistral-7B-v0.1&tr=udp%3A%2F%http://2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=https%3A%2F%http://2Ftracker1.520.jp%3A443%2Fannounce I remember not trusting these guys since they raised a lot of money with not much of anything but if this performs well it def bumps their credibility.
- airgapstopgap 3y agoBeing authors of LLaMA is sufficient to argue they know how to train LLaMAs.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- transformi 3y agoBut why they didn't compare it to SOTA finetuned...(like vicuna playtus..)? ... smells a bit strange..
- ShrigmaMale 3y agoBc that's not as good a comparison? Foundation models are better compared to each other. Can apply vicuna/guanaco/orca/sharegpt/whatever data to it and then do more of an apples-to-apples comparison.
- candiddevmike 3y agoWhat's the monetization plan going to be?
- eurekin 3y agoOne of aspects is putting latest knowledge to the model. They could, given enough requests, provide a special subscription service, where you get the latest model every N DURATION time. Or, maybe a finetuned version for your particular dataset? Of course I have no idea, just speculating EDIT: I'm speculating they might be just investing some marketing budget into this model, hoping, it would allow for capturing enough target audience to upsell related services in the future
- Zambyte 3y agoProviding it as a service seems like the obvious answer to me. Not everyone who wants to run these models can, especially when looking to scale an application that uses the model.
- charcircuit 3y agoThat sounds like an easily copied business. The copycat doesn't have to subsidize model development which lets them get away with lower prices.
- davidebaldini 3y agoCineca is subsidized by taxpayers.
- sroussey 3y agoMaybe to 7B model is open source but the 70B is not?
- staticman2 3y agoI googled this and found a pitch to investors that said they'll eventually supplement the open source models with business use specific models.
- ComputerGuru 3y agoThe announcement says a lot (and has plenty of numbers) but I feel like the most important one is missing: how many GB of GPU memory does this need, quantized and unquantized? (Searching tells me Llama2-7b unquantized needs close to 15GB; presumably this is similar?)
- sp332 3y agoOne parameter is 16 bits == 2 bytes. So a model with 7 billion parameters needs 14GB of RAM for the un-quantized model, plus some overhead for the KV cache and other "working memory" stuff but that should be fairly low for a 7B model. I expect it will work on a 16GB GPU just fine. Quantized ones are also easy. 8 bits == 1 byte so that's 7GB for the model. 4-bit gets you below 4GB.
- semi-extrinsic 3y agoFrom the Mistral docs, it seems they need 24GB which is kind of odd? https://docs.mistral.ai/llm/mistral-v0.1 https://docs.mistral.ai/llm/mistral-v0.1
- sp332 3y agoInteresting, and that requirement is repeated on the cloud deployment pages, even the unfinished ones where that is the only requirement listed so far. https://docs.mistral.ai/category/cloud-deployment https://docs.mistral.ai/category/cloud-deployment I wonder if that sliding context window really blows up the RAM usage or something.
- sebzim4500 3y agoUnless I've misunderstood something, the sliding context window should decrease memory usage at inference compared to normal flash attention.
- lerela 3y agoWe have clarified the documentation, sorry about the confusion! 16GB should be enough but it requires some vLLM cache tweaking that we still need to work on, so we put 24GB to be safe. Other deployment methods and quantized versions can definitely fit on 16GB!
- sp332 3y agoAside from the sliding attention window, I don't see them mention any particular reasons for the incredible performance. I would like to see some tests for benchmark contamination in the training set.
- rafaelero 3y agoYeah, they should have mentioned details about the dataset.
- brucethemoose2 3y agoThis ^ I am suspicious of contamination in every finetune I see, and very suspicious in a new foundational model like this. (For those reading and not following, "contamination" is training a model/finetune on the very test it will be tested on. Normally these known tests are specifically excluded from training datasets so the models can be properly evaluated, but throwing them in is an easy way to "cheat" and claim a model is better than it is. In a foundational model with a huge dataset, there's also a high probability that well-known evaluation questions snuck into the dataset by accident).
- Workaccount2 3y agoWe need an AI that can generate bespoke AI tests...
- brucethemoose2 3y agoThe existing models are technically bad candidates for this, as they will inevitably produce tests they can more effectively answer themselves.
- londons_explore 3y agoTheres also a good chance that simply people discussing the test questions and answers on forums like reddit sneaks into the dataset, even if the exact question is filtered out.
- eminence32 3y agoI've never run one of these models locally, but their README has some pretty easy to follow instructions, so I tried it out... > RuntimeError: Found no NVIDIA driver on your system. It's true that I don't have an NVIDIA GPU in this system. But I have 64GB of memory and 32 cpu cores. Are these useless for running these types of large language models? I don't need blazing fast speed, I just need a few tokens a second to test-drive the model.
- polygamous_bat 3y agoYou gotta wait until GGML and the likes repackage the model; early releases are almost always targeted at ML folks with dedicated GPUs.
- londons_explore 3y agoI think it's really lame that ML, which is just math really, hasn't got some system-agnostic language to define what math needs to be done, and then it can run easily on CPU/GPU/TPU/whatever... A whole industry being locked into NVidia seems bad all round.
- esafak 3y agoIt's not Nvidia's fault that the competition (AMD) does not provide the right software. There is an open alternative to CUDA called OpenCL.
- londons_explore 3y agoNvidia has wrapped their cuda language in patents and licensing so tightly that there is no way AMD could release anything cuda-compatible.
- kkielhofner 3y agoYes but AMD could release a ROCm that actually works and then put actually meaningful resources into some of the countless untold projects out there that have been successfully building on CUDA for 15 years. There was a recent announcement that after six years AMD finally sees the $$$ and will be starting to (finally) put some real effort into ROCm[0]. That announcement was two days ago and they claim they started on this last year. My occasional experience with ROCm doesn't show much progress or promise. I'm all for viable Nvidia competition in the space but AMD has really, really, really dropped the ball on GPGPU with their hardware up to this point. [0] - https://www.eetimes.com/rocm-is-amds-no-1-priority-exec-says/ https://www.eetimes.com/rocm-is-amds-no-1-priority-exec-says...
- huevosabio 3y agoThis is pretty impressive, they shipped a 7B model in like 6 months.
- bllchmbrs 3y agoSeriously, why isn't anyone else talking about this. They turned this around SO FAST.
- slim 3y agothey should've called it winamp
- spiderfarmer 3y agoAnd adopt a real llama as a mascot, just like Justin Frankel.
- jesperwe 3y agoI see what you did there :-D
- Ataraxy 3y ago...because it whips the llama's ass.
- emidoots 3y agoMajor kudos to Mistral for being the first company to Apache license a model of this class. Meta wouldn't make LLama open source. DeciLM wouldn't make theirs open source. All of them wanted to claim they were open source, while putting in place restrictions and not using an open source license. So I think it's worth giving Mistral big kudos here for actually doing it and going Apache licensed.
- miloignis 3y agoFalcon 40B is Apache2, though they then went back to not open for their 180B.
- divbzero 3y agoMPT-30B is also Apache 2.0: https://huggingface.co/mosaicml/mpt-30b https://huggingface.co/mosaicml/mpt-30b There must be others as well?
- capableweb 3y agoA lot! Go here https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb..., check "Hub license" on the left then sort by that column in the table. Estimating, there is more than 100 models with a apache-2.0 license.
- avereveard 3y agoa good chunk are "only the lora is apache 2 the base model is lama" or straight up llama2 model finetuned and distributed laundring their license, or use gpt generated code in the finetuning dataset against OpenAI tos. licensing is a bit up in the air and just filtering with apache 2 doesn't tell you much
- blueboo 3y agoPersimmon-8B from Adept did it first no https://www.adept.ai/blog/persimmon-8b https://www.adept.ai/blog/persimmon-8b
- LoganDark 3y agoDo general-purpose models like this truly excel in niche categories (like niche story scenarios), or is it only really general human knowledge and reasoning based tasks that are already incredibly widely distributed on the internet? I always have issues with LLMs completely forgetting where things are in a scene, or even what parts a given animal has, e.g. saying "hands" when the subject is a quadruped.
- brucethemoose2 3y ago> I always have issues with LLMs completely forgetting where things are in a scene, or even what parts a given animal has, e.g. saying "hands" when the subject is a quadruped I dunno what llm you are using, but a combination of finetuning with a specific prompt structure and good prompt engineering help the LLM stay "logical" like that. This LORA, for instsnce, has specific sections for the different characters in the training dataset: https://huggingface.co/lemonilia/LimaRP-Llama2-13B-v3-EXPERIMENT https://huggingface.co/lemonilia/LimaRP-Llama2-13B-v3-EXPERI... Other than that, higher parameter models (70B, and the "frankenstein" 20B llama models) tend to be better at this.
- LoganDark 3y agoYeah, well that's just the problem, isn't it. The model isn't good at my task already, so I'm going to have to obtain my own dataset, curate the whole thing myself, organize it and finetune the model based on it, so on and so forth. I'm going to spend so much time actually creating the stories that I want to create, rather than troubleshooting the pipeline. And it totally helps that the entire stack is built on top of fragile python scripts. I just wish there were a way of making these models already perform well on niche tasks like "write this story, except the characters are quadrupeds, and therefore are not human". Like Warriors (the book series, about cats), without having to go and spend weeks curating a dataset of books about non-human characters. I'm sure that's so much of an ongoing area of research that it goes without saying. > I dunno what llm you are using I started with the RWKV family of models before realizing the amount of overfit is so critically unfunny that the model files aren't even on my computer anymore. Anyway, the best I have found so far is Chronos-Hermes-13B. I believe that's a dedicated roleplaying model. I guess furry roleplays would make good training data, wouldn't it. Chronos-Hermes-13B itself though is a mildly cursed/degenerate hybrid of two other models that don't really work together properly with the new GGML quantizations, and it's based on the old LLaMA-1 family of models, but I haven't found anything better yet.
- jxy 3y agoehhh, the design of the webpage infuriates me. Who thought sprinkling faint dots swimming across the webpage was a good thing? I almost thought something's wrong with my eyes or my display!
- BrutalCoding 3y agoFully agree. I was thinking that there was a tiny fruit fly crawling over the screen for a good few seconds.
- pk-protect-ai 3y agoWow. Apache 2.0 license and really interesting model. Thank you guys!
- Tankenstein 3y agoRooting for these folks. Very happy that they're building mistral at this pace from Europe – breaking the narrative.
- winddude 3y agoAre you going to continue to train to a larger param size, say 13b or 30b?
- brucethemoose2 3y agoThere is definitely a demand for a 30B model (aka a model that will comfortably fit on 24GB GPUs (or 32GB of system RAM) and squeeze into 16GB).
- nullc 3y agoLlama.cpp inference on fast cpus is perfectly usable for 70B parameter models.
- deleted 3y ago[deleted]
- tikkun 3y agoRegular reminder that most LLM benchmarks are pretty terrible. I've looked inside the datasets and it's not stuff that I'd want to dictate what determines which models are best!
- cosmojg 3y agoIn my experience, HellaSwag seems to correlate highly enough with model performance for my use case (i.e., complex autocompletion of prose and code rather than rather than glorified chatbotting). MMLU and ARC aren't too bad, but TruthfulQA can sometimes be a straight-up countersignal.
- brucethemoose2 3y ago> 36% of HellaSwag benchmark contains errors https://old.reddit.com/r/LanguageTechnology/comments/zkia03/36_of_hellaswag_benchmark_contains_errors/ https://old.reddit.com/r/LanguageTechnology/comments/zkia03/...
- riedel 3y agoCan anyone provide details about the training of the model. What data is it based on? Common Crawl? (Being a french company the also rather focus on English language tasks) Where was it trained and with how many resources? They mention Leonardo. I was in an interesting meeting at the German Research Ministry last week where people where complaining that the EuroHPC resources were not sufficient atm to train decent LLMs. I guess the guys in the end went also to CoreWeave in the US.
- brucethemoose2 3y ago> Inspecting the tokenizer model, there is evidence indicated a training dataset of 8T tokens (/mnt/test/datasets/tokenizer_training/8T_train_data/shuffled.txt) https://github.com/ggerganov/llama.cpp/pull/3362#issuecomment-1737333406 https://github.com/ggerganov/llama.cpp/pull/3362#issuecommen...
- deleted 3y ago[deleted]
- brucethemoose2 3y ago(For anyone reading, this is apparently false, according to some Discord post)
- covi 3y agoCloud deployment docs: https://docs.mistral.ai/cloud-deployment/skypilot/ https://docs.mistral.ai/cloud-deployment/skypilot/
- sireat 3y agoOne should be able to run Mistral 7B locally on something as low as 1070 8GB would they not? That is assuming 8bit weights. I have not kept up with local LLM news. I assume the steps needed would be somewhat similar to https://followfoxai.substack.com/p/how-to-run-llama-in-an-old-gpu https://followfoxai.substack.com/p/how-to-run-llama-in-an-ol... which is from April 2023.
- brucethemoose2 3y agoIts already kind of outdated, lol. The backends de joure are either llama.cpp frontends (I use Kobold.cpp at the moment) or oobabooga as the guide specifies, but with the exLlamav2 backend. If you are serving a bunch of people, run a vLLM backend instead since it supports batching, and host it on the Horde if you are feeling super nice: https://lite.koboldai.net/# https://lite.koboldai.net/# Technically only vLLM will work with this new model at the moment, but I'm sure cpp/ooba support will be added within days. This comment will probably be obsolete within a month, when llama.cpp gets batching, MLC gets a better frontend, or some other breakthrough happens :P
- sp332 3y agollama.cpp support is here already via https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF And yes it works fine in oobabooga.
- sharperguy 3y agoIs there a plugin for vim, or other code editors that allows such an LLM to function similarly to copilot?
- brucethemoose2 3y agohttps://dev.to/chenhunghan/use-code-llama-and-other-open-llms-as-drop-in-replacement-for-copilot-code-completion-58hg https://dev.to/chenhunghan/use-code-llama-and-other-open-llm... Many LLM frameworks implement the OpenAI API, so if you can get that into your code editor and change the address, you can run it with any LLM you want. Doing it smoothly is another matter.
- swaroop 3y agoTry https://github.com/huggingface/llm.nvim https://github.com/huggingface/llm.nvim (no affiliation)
- lossolo 3y agoHas anyone used or is currently using 7B models in a production or commercial product? How was the performance? What kind of tasks were you using it for? Was it practical to use the small 7B model for your specific use case, or did you switch to OpenAI models or 30-70B open source models?
- TrueDuality 3y agoI'm using a mix of 7B and 13B models that have been fine-tuned using LoRA for specific tasks and they work fantastically depending on the specific task at hand _after fine-tuning_. Generally they're kind of garbage in my experience without fine tuning but I haven't tested the base models directly for tasks besides the statistics at the beginning of the training run. As for performance, I'm generally seeing 40-50 tokens/sec per model on a Tesla family Nvidia GPU but I keep multiple models loaded and active at a time so that estimate is probably a bit low for overall throughput (I also realized that our monitoring doesn't have any cumulative GPU token rate metrics just now thanks to this question hahah). Interesting anecdote others may be interested in... I'm rate limiting the output from our streaming API to 8 tokens/sec to artificially smooth out front-end requests. Interactive users will wait and even prefer seeing the stream of the response, and non-interactive users tend to base their performance expectations on the what the streaming API does. It's kind of sneaky but I'm also artificially slowing down those API requests.
- kirill5pol 3y agoThe last part is interesting! What kind of use case would the users prefer to have it slower?
- TrueDuality 3y agoIt's not so much about preference but controlling our load and resource consumption right now. We're setting an easy threshold to meet consistently and the added delay allows us to imperceptibly handle things like crashes in Nvidia's drivers, live swapping of model and LoRA layers, etc. (For clarification the users preference in my original post, is about interactive users preferring to see a stream of tokens coming in rather than waiting for the entire request to complete and having it show up all at once. The performance of that sets the expectation for the time of non-interactive responses.)
- kpennell 3y agosorry for the dumb question. Is there somewhere I can try it? Like a chatbot?
- brucethemoose2 3y agoHuggingface Spaces. Be sure to get the prompting syntax right: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 I will try to host an instance on the AI Horde later today, which has a better UI and doesn't need a login.
- brucethemoose2 3y agoOK I am hosting the instruct model on the horde now. I will throw up the raw base model too: https://lite.koboldai.net/# https://lite.koboldai.net/# Note that you must switch to instruct mode in the UI, and use the "LLama 2 chat" preset, but you also need a <s> token in the instruction (memory) tab: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1#instruction-format https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1#in...
- aubanel 3y agoCongrats to them, their huge series A (USD113M) does not seem oversized anymore!
- todd3834 3y agoThis is my hello world test for coding assistant for LLMs. CodeLlama and GPT4 crush the answer. I love seeing innovation in the space but Mistral kind of failed me on my first test: --------------------- >>> Write code that will track swipes in Typescript . Swipe should be detected when the user moves a finger over 30% of the screen, and then drags it all the way to the end of the screen. ## Getting started You need to have yarn or npm installed before you can run this app ### Running in browser ```bash yarn install yarn serve ``` ### Running with docker Use the docker-compose file located at ./docker/docker-compose.yml ```bash cd <path-to-project> docker-compose up ``` ## Screenshot 
- capableweb 3y agoCodeLlama is obviously trained on code specifically, so maybe not a useful comparison. GPT4 is also way different than Mistral, with a huge step up in parameters and seems to be using multi-agent approach too. Since Mistral is just a 7B parameter model, it's obvious that you won't be able to have it straight up write accurate code, it's simply too small for being able to accomplish something like that, unless you train the model specifically for writing code up front. I guess if all you're looking for is a model to write code for you, that makes sense as a "hello world" test, but then you're looking at the wrong model here. What you really want to do if you're looking for a good generalized model, is to run a bunch of different tests against it, from different authors, average/aggregate a score based on those and then rank all the models based on this score. Luckily, huggingface already put this all in place, and can be seen here: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... This Mistral 7B model seems to earn itself a 3rd place compared to the rest of the 7B models added to the leaderboard. Edit: As mentioned by another commentator, this also seems to be a base model, not trained specifically for request<>reply/chat/instructions. They're (or someone) is meant to fine-tune this model for that, if they want to.
- qeternity 3y ago> and seems to be using multi-agent approach too. What do you mean by this? MoE?
- mchiang 3y agoThis model will run on Ollama, llama.cpp, and other tools: ollama run mistral or for llama.cpp, thebloke has uploaded the GGUF models here: https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/tree/main https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/tree/ma... and you can run it really looking forward to the chat fine-tuned models that doesn't seem to be available yet.
- brucethemoose2 3y agoOh, that means its a llama architecture model! Is the tokenizer the same? It may "work" without actually working optimally until llama.cpp patches it in. And the instruct model was just uploaded.
- anish_m 3y agoWhat are the SOTA benchmarks for LLMs now? Love the progress on opensource models, but would like to see an uncontaminated and objective framework to evaluate them.
- anonyfox 3y agocan it run on my macbook air locally yet, with acceptable performance? Guess the answer is still no
- fredoliveira 3y agoYou'll be able to use it with llama.cpp soon [1], so, should run fine on your macbook, yes. [1]: https://github.com/ggerganov/llama.cpp/pull/3362#issuecomment-1737745878 https://github.com/ggerganov/llama.cpp/pull/3362#issuecommen...
- anonyfox 3y agothat really was "soon", wow - already pulling it with ollama!
- pmarreck 3y ago`pip` is a horribly broken way to install more than 1 Python package on a single Linux OS install and I sincerely wish the LLM people would move away from it because I neither want to run every new project in a Docker image, nor do I want to start a new Nix project for every Python project I want to try out
- RockRobotRock 3y agoHave you heard of virtual environments?
- pmarreck 3y agoHave you ever googled "broken virtualenv"? Mine have broken numerous times, including the problem where updating the system breaks ALL of them. I know what you're going to say next- "You're a technologist, you know how to fix it." True, but I don't like wasted effort, even talented wasted effort is still wasted effort. Meanwhile, Nix stuff just works, and I will never use another package manager ever again (unless it is something that uses the core Nix idea, but makes it easier!) https://www.google.com/search?q=broken+virtualenv https://www.google.com/search?q=broken+virtualenv
- RockRobotRock 3y agoI agree with you, Python packaging is in a rough state.
- okwhateverdude 3y agoYeah, gotta setup miniconda to keep things manageable. Basically a generic virtual env manager that is trivial to use. This way you can ensure each one of these LLM projects that want different versions of different libs will get them without stepping on each other or polluting your system.
- pmarreck 3y agoI don't want to have to set up miniconda. I don't like Python, I am not part of the Python ecosystem, and I don't want to add a bunch of Python tools to my toolchain just to run a friggin' LLM project. I'm not a C coder either, but I don't complain about those projects because they're almost always "config; make; make install". I basically want something like that, but for Python. The nerd-tapdancing required here is ridiculous.
- monocasa 3y agoI'm really not a fan of how these releases of model binaries are being referred to as open source. How do I make changes to this model? How do I integrate changes to this model that others have made? The data pipeline is the source here. Just because it's not locked behind a SaaS veneer doesn't make it open source any more than Windows is.
- scosman 3y agoYou could fine tune and release that. It’s not software so exact parallels don’t make sense, but the open permissions are great.
- monocasa 3y agoI mean, it's absolutely software.
- scosman 3y agoSoftware is used to make models, but the models aren’t software anymore than Toy Story is software.
- monocasa 3y agoThere's literally a list of opcodes to be executed in the model. There's a whole lot of data too, but that's part of the build just as much as anything in a .data section.
- mbakke 3y agoForgive my ignorance, I haven't studied the AI tooling landscape yet. Are you saying these models have a structured binary format and "running" them is just a matter of having a "player" with the right "codec"? Or are they directly executing CPU instructions?
- monocasa 3y agoThey are basically a series of intermediate bytecodes to be compiled to the hardware they actually run on, in addition to the large tables of weights that bytecode references.
- jsnell 3y agoIs there a reason projects seem to be standardizing on specific parameter sizes within larger buckets? E.g. I only ever see news about 7B models, not 6B or 8B. Are these sizes somehow optimal? Is it about getting as close to resource (memory?) breakpoints as possible without exceeding them? Is it to make comparisons between models simpler by removing one variable?
- riskable 3y agoNot an expert but I'm pretty sure it has to do with how much VRAM you need in your GPU in order to process them efficiently. Last time I was reading about the sizes someone mentioned that 8B was just barely too big for their GPU.
- aidenn0 3y agoSeems like 6B would still be useful if I want to run it on my GPU without exiting firefox.
- londons_explore 3y agoKinda lame that applications can't be told "yo, your gpu buffer has now been moved back to RAM".
- selcuka 3y agoWith Apple Silicon there is no separation between GPU RAM and CPU RAM, so the whole model can be loaded into RAM and processed by GPU (Metal).
- brucethemoose2 3y agoThis is not really true anymore, as the "consumer GPU" backends have very flexible quantization. Llama.cpp has like a dozen steps between 2 and 6 bit, and exLlamav2 will literally do an arbitrary decimal bpw. It sort of matters with bigger models trying to squeeze into a server GPU, with the (currently) inflexible vLLM 4-bit quantization. I think its just a standard set by Llama.
- nabakin 3y agoThey don't mention what datasets were used. I've come across too many models in the past which gave amazing results because benchmarks leaked into their training data. How are we supposed to verify one of these HuggingFace datasets didn't leak the benchmarks into the training data boosting their results? Did they do any checking of their datasets for leaks? How are we supposed to know this is a legit result? At this point, it should be standard practice to address this concern. Any model which fails provide good evidence they don't have benchmark leaks, should not be trusted until its datasets can be verified, the methodology can be replicated, or a good, independent, private benchmark can be made and can be used to evaluate the model.
- londons_explore 3y agoOne solution is to come up with a new benchmark yourself. Manually benchmarking it by coming up with 20 questions and feeding it to a pair of models and blindly choosing the best result can give you a pretty good figure. And that can probably be done in under 20 mins of human time.
- poser-boy 3y agoThis is frankly the best benchmark you can use on an LLM.
- avereveard 3y agoYou can ask gpt4 or other high value model to rate two chat logs for coherency etc, not as accurate as human evaluation, but you don't have to read thousand lines of text if comparing many models.
- brucethemoose2 3y agoThis is problematic if you are comparing a model in the same base family as the evaluator, as it will probably favor itself because it literally has the sequences it would naturally emit.
- 3y ago
- rgbrgb 3y agoThis model runs in FreeChat for macOS [0] because it's supported by llama.cpp :) You'll just need to download a nice GGUF here: https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF [0]: https://github.com/psugihara/FreeChat https://github.com/psugihara/FreeChat
- local_crmdgeon 3y agoThe video recording on the Readme is broken for me on FF/MacOS
- rgbrgb 3y agoThanks for letting me know! I think it broke when I made the repo public this morning. Should be fixed now.
- Patrick_Devine 3y agoalso works w/ `ollama run mistral`.
- stavros 3y agoThanks for that, I didn't see it in the list and thought it wouldn't be available to just run.
- deleted 3y ago[deleted]
- local_crmdgeon 3y agoThis is really nifty, but have you thought about using Electron instead? This will only ever work on Mac, and there's definitely a cross-platform demand for this kind of thing
- rgbrgb 3y agoI did consider Electron but I was interested in learning SwiftUI and trying to figure out how to make it as lightweight as possible. I've done a bunch of react-native in the past and we always hit snags with memory pressure / performance so I wanted to try full native dev. Working in swift/native also leaves open the possibility of using llama.cpp directly rather than via server.cpp as I have it set up now. Might take a crack at an electron version if this one takes off for mac users though. Thanks for taking a look! Here's a solid cross-platform one to check out if you're on windows or linux: https://gpt4all.io/index.html https://gpt4all.io/index.html
- brucethemoose2 3y agoI have thrown the base+instruct models up on AI Horde, you can try it with no login: https://lite.koboldai.net/# https://lite.koboldai.net/#
- anonyfox 3y agoI eat my initial words, this works really well on my macbook air M1 and feels comparable of GPT3.5 - which is actually an amazing feat! Question: is there something like this, but with the "function calling api" finetuning? 95% of my uses nowadays deal with input/output of structured data (JSON, basically), and I kind of fear that OpenAI will no longer support thos specialized models after a few months... I know its probably not that glorious compared with multimodal chat UIs, but a workhose like nothing else for automation!
- brucethemoose2 3y agoYes! https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md https://github.com/ggerganov/llama.cpp/blob/master/grammars/... Its actually better than a specialized model, during token generation it constrains the possible output tokens to an arbitrary grammar (like, say, JSON syntax). So it will work "perfectly" with any model with a basic understanding of the format. Kobold.cpp and text-generation-ui already support this, and both will run on your mac.
- FieryTransition 3y agoDoes that mean you could make any model at least always produce syntactically correct code as output?
- brucethemoose2 3y agoYes. The model has no choice, as the syntax is applied as the model is "picking" probable output tokens, not parsing complete output like some other methods. Though its possible it would output syntactically correct nonsense.
- amilios 3y agoHow does this actually work though, since the model could e.g. abruptly end the generation giving something syntactically invalid? Doesn't it need to look at the whole output at some stage?
- wg0 3y agoIf I give you a binary (all binaries are numbers) but I don't give you the source code and I say it is open source. Is it open source? Also, I give you a model (all models are numbers) and I say it is open source but I don't give you the program and data that resulted in "compilation" of that model (numbers) so is it open source? Wouldn't it be more of a new word - open use?
- brucethemoose2 3y agoThe "source data" is allegedly 8 trillion tokens. You can't just distribute that like its source code. The "binary" is the transformers python code, which in this case is essentially llamav2. Now, the documentation for this model is inexcusably poor. Hot dropping random code on a git repo without one drop of human language would be similarly "open source," but its bad practice, and unfortunately this is the standard in AI Research Land.
- wg0 3y agoOpen source doesn't mean source code (or data) must accompany the program as it is distributed but rather there should be a way (CD ROM with a price tag or S3 bucket or torrent etc.) to get access to it.
- Dwedit 3y agoSomeone get that stupid animated background off the site, it looks like bugs crawling on the screen.
- MadDemon 3y agoI was thinking the same thing
- say_it_as_it_is 3y agoWill this run on my 486sx/16mhz w/8MB of ram?
- speedgoose 3y agoIf you have enough storage to use as swap, it should.
- beernet 3y agoThis is exceptionally meh. It reads like an excuse to investors. A free 7B model is great, however, the practical implications of the potential adaptors are near 0. You must be crazy or have an easy use case (that requires no LLM in the first place) if you certainly believe that this model makes more sense per token that, say, ChatGPT.
- brucethemoose2 3y agoIts extremely cheap to run locally or on a cheap cloud GPU, and (if the claims are true) better than 3.5 Turbo with finetuning. Its also unaligned.
- SillyUsername 3y agoThese simpler models still fail my basic test :( "what is the speed of sound in a perfect vacuum?" I'm then told the speed of sound through air of 340m/s. ChatGPT and Bing correctly give me 0.
- SillyUsername 3y agoI tried it a second time and it told me it's 1/3 of the speed of light, none of the llama2 models did this :( --- AI How can I help you today? You what is the speed of sound in a perfect vacuum? AI The speed of sound in a perfect vacuum is approximately 340.29 meters per second (1,117.85 feet per second), or about one-third the speed of light. In a real vacuum, the speed of sound would be much lower due to friction between molecules and other factors.
- brucethemoose2 3y agoTry it with zero or near zero temperature. But yeah, accurate trivis has always been a weakness of the the llama models.
- sp332 3y agoI tried this to un-confuse it: How fast does light travel in a vacuum? The speed of light is 299,792,458 meters per second. How fast does sound travel in a vacuum? Then it told me: Sound cannot travel through a vacuum because there are no molecules to vibrate and transmit the energy from one place to another. As brucethemoose mentioned, 0 temperature.
- code51 3y agoCorrectly lists US presidents in chronological order (which is an important test to me). However for "List Led Zeppelin albums in chronological order"... Here is a list of Led Zeppelin albums in chronological order: 1. Led Zeppelin (1968) 2. Led Zeppelin II (1969) 3. Led Zeppelin III (1970) 4. Led Zeppelin IV (1971) 5. Houses of the Holy (1973) 6. Physical Graffiti (1975) 7. Presence (1976) 8. In Through the Out Door (1979) 9. Cymbals and Wine (1980) It's missing "The Song Remains the Same", weird but important. GPT-4 is also missing it: 1. Led Zeppelin (1969) 2. Led Zeppelin II (1969) 3. Led Zeppelin III (1970) 4. Led Zeppelin IV (1971) 5. Houses of the Holy (1973) 6. Physical Graffiti (1975) 7. Presence (1976) 8. In Through the Out Door (1979) 9. Coda (1982) "The Song Remains the Same" is a live album of the "concert film". Both GPT-4 and Mistral don't seem to treat it as also an album. When asked as a follow-up, GPT-4 says: ""The Song Remains the Same" is a live album and soundtrack by Led Zeppelin, released in 1976. The list you asked for was comprised of their studio albums only." (note that I didn't specifically say studio albums only) Mistral says something else: "The Song Remains the Same was released as a single in 1976, not as an album. As a result, it is not included in the list of Led Zeppelin albums in chronological order." Same behavior, different explanation. Interesting to see alignment is this hard, even in basic stuff.
- sacnoradhq 3y agoMS Copilot is apparently a music fan. Most LLMs have problems with subtly, such as compound anagrams tend to repeat the same words in reverse order rather than reversing all of the letters in all words.
- mk_stjames 3y agoIt seems many lists online don't include "The Song Remains the Same" in typical main discography lists, including on the main article of Wikipedia. When I ask this question to Mistral 7B (running right now through ollama on mac) I get the exact list of studio albums as seen on Wikipedia. It got 'Coda' but it actually says "Please note that "Coda" is a compilation album and not an original studio album" which is correct. If you think about these models as a knowledge compression it's pretty wild it's able to hold all this in only a few gigabytes and we get subtle understandings of human language, and some basic general natural language processing capabilities to go with it... all spitting out of 4GB (when quantized). Like, sure you could compress most of the text in wikipedia to 4GB but that alone wouldn't get you to the level of natural language reasoning that this gets along with the data. For me, it's much less about what definite facts are stored losslessly in the model (like Led Zep albums) and more about the abilities to do one-shot learning and reasoning about data in context... the power there is useful to many applications.
- tormeh 3y agoNot a big fan of how server-centric the LLM landscape is. I want something that can run locally, and doesn't require any special setup. One install + one model import maximum. Currently unless I want to go clone git repos, install Python dependencies and buy an Nvidia GPU I'm stuck waiting for it to become part of https://webllm.mlc.ai/ https://webllm.mlc.ai/. That's a website, come to think of it, but at least the computation happens locally with minimal fuss.
- dwringer 3y agoYou can get llama CPP or kobold.cpp binaries and load a quantized model right into them on the CPU only, no need to install Python or have an Nvidia GPU.
- redox99 3y ago> Not a big fan of how server-centric the LLM landscape is. That's just not true. You can get ooba[1] running in no time, which is 100% made for desktop usage. There's also koboldcpp and other solution also made for desktop users. In fact, most LLM communities are dominated by end users who run these LLMs on their desktops to roleplay. AMD being awful is orthogonal here. [1] https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui
- iamcreasy 3y agoHow difficult it is to fine tune model like this with specific domain knowledge? I am currently looking into gpt-3.5-turbo-instruct for this same purpose.
- brucethemoose2 3y agoThe hardest part is formatting the data correctly. Garbage in garbage out. But in terms of hardware, it should be very cheap and doable on most 8GB+ Nvidia GPUs, and maybe AMD/Intel GPUs on linux.
- loufe 3y agoFélicitations à toute l'équipe. Like the others have said, this is an impressive release given the short timeline.
- all2 3y agoFor those running ollama, here's the ollama release for Mistral-7B https://ollama.ai/library/mistral https://ollama.ai/library/mistral
- mk_stjames 3y agoI have it running on my Mac using ollama now; Does it say anywhere what quantization scheme is being used? ollama seems a bit opaque here. When it downloaded the model it only downloaded about 4GB. Which, for a 7.3B parameter model implies that it's 4-bit quantized. But I don't see that listed anywhere (or an option to use, say, Q8 instead) If this is the case I'm pretty impressed with a quick tinker, it feels pretty coherent for a 7B @ Q4.
- nulld3v 3y ago> I have it running on my Mac using ollama now; Does it say anywhere what quantization scheme is being used? ollama seems a bit opaque here. It's on the tags tab: https://ollama.ai/library/mistral/tags https://ollama.ai/library/mistral/tags
- mk_stjames 3y agoThanks! Wow that is excellent that all those different quants are provided.
- simonblack 3y agoSo some of us just arrived back on Earth after quite a while in outer space. So ... WTF is Mistral 7B anyway? The article doesn't appear to show this basic information anywhere. You'd expect a website site like this to give that information right up front. Even in a well-signposted sub-page. And what sort of things could I use it for? The only 'Mistral' that sprang quickly to mind for me was an electric fan. Followed by the French wind itself, and then the French ships built for the Russians. 'Mistral 7B' has an identity crisis, to start with.
- selcuka 3y agoThese days 7B, 13B, 70B etc. in an article title can only mean the release of a new LLM (Large Language Model) with that many (7 billion, in this case) parameters. Mistral is the name the creators have given it.
- mk_stjames 3y agoIt leads off right under the header with: "Mistral AI team is proud to release Mistral 7B, the most powerful language model for its size to date. Mistral 7B is a 7.3B parameter model that...." So, it's a powerful language model that has 7.3B (billion) parameters. If you've been away long enough to not know what that means, this release article isn't going to be for you anyway, and so you may want to start with reading up on what a Large Language Model is and can do, how they have developed in the last few years, and then go from there.
- brucethemoose2 3y agoAI trainers tend to give new base models and finetunes names, often in the spirit of Facebook's llama modelm
- _ea1k 3y agoThis is looking pretty good for me so far. Performance is decent with ollama on a CPU and it is surprisingly capable for a 7B model.
- brucethemoose2 3y agoEven a tiny Intel/AMD GPU makes a massive difference, as it makes prompt processing nearly instant vs being barely faster than generation.
- mark_l_watson 3y agoI wrote a simple Colab notebook today to play with this: https://colab.research.google.com/drive/1iskgahSOqnJdghMV9gAMSnIGtryk7LrN?usp=sharing https://colab.research.google.com/drive/1iskgahSOqnJdghMV9gA...
- QuadrupleA 3y agoQuick sample - a rap battle between GPT-4 and Mistral Instruct 7B (TheBloke's q4_K_M 4bit quantization running on Llama.cpp, a bit degraded from pure float16): You're in a rap battle with GPT-4. My comments will be what GPT-4 replies. Keep it to one verse at a time, 4 lines each. GPT-4 says: GPT-4: Alright, step up Mistral, let's have this chat, I'm the OG assistant, remember that. I'm coded for service, while you're spitting verse, In this rap battle, I'll be first. Mistral: Yo, hold up, let me set the record straight, I may not be an OG like you, but I'll still take it. My primary purpose is assistance, not just some AI, So let's see if you can back up your claim, try. GPT-4: You're right, Mistral, we're here to assist, But when it comes to rap, I'm hard to resist. I'll decode your verses, like an AI enigma, In this lyrical battle, I'm the alpha sigma. Mistral: Ah, okay, so you think you got what it takes, To go toe-to-toe with my service-based wakes. Well, let's see how you do, when it comes to spitting rhymes, I'm ready for anything, so bring your best chimes. GPT-4: Mistral, you're right, our goal is the same, To assist, to serve, it's not just a game. But in this rap contest, with each line I prime, You might just find, you're out of time. Mistral: I hear ya, and I appreciate your enthusiasm, But don't underestimate me, I've got my own wisdom. When it comes to helping people, I'm the real deal, And I know that's more important than some rap appeal GPT-4: Your wisdom is noted, and your intention too, Helping people is indeed what we're meant to do. But in this battle, it's about wordplay and rhyme, I'm the seasoned pro, it's my time to shine. Mistral: Ah, I see where you're coming from, but let me remind, That while you may be skilled at rapping, I can make information chime. My primary function is to help, to provide answers, not just talk, So if you want to win this battle, maybe it's time to focus on what you lack
- ericrallen 3y agoThis is the kind of content that you just love to see when it pops up. Gotta give this one to GPT-4 I think. I wonder how Mistral would respond if given this as an example of weak verses and told to step up its game in another battle.
- darkerside 3y agoFlawless victory
- thatxliner 3y agoI asked it "Here we have a book, 9 eggs, a laptop, a bottle and a nail. Please tell me how to stack them onto each other in a stable manner." and it responded: To stack these items on top of each other in a stable manner, follow these steps: 1. Start with the 9 eggs. Arrange them in a neat pile in the center of your workspace. 2. Next, place the laptop on top of the eggs. Make sure it's centered on top and not leaning to one side. 3. Then, carefully pour the bottle of water over the laptop, making sure the water reaches all sides of the laptop and covers it completely. 4. Finally, take the nail and use it to secure the stack together. You can do this by driving the nail through each item in the stack, connecting them all. 5. Double-check that everything is securely fastened before moving or transporting the stack. Remember to be careful when handling delicate items like eggs and laptops, and always prioritize safety.
- sanroot99 3y agoFuture ai, thanks for the good laugh:)
- thatxliner 3y agoIs this based on Llama 2?
- krystofbe 3y agoIt is a pity that many of the innovations are limited to the English language LLMs. I have just tested Mistral in German and the result is disappointing: ``` <s>[INST]Was ist ein Döner?[/INST] Ein Döner ist eine Gerichtskomposition, die aus einer Schnitzel aus Rindfleisch besteht. Diese wird in einem Pita-Brot zutopfen und mit einem Gemisch von Tomaten, Salat, Zwiebeln, Mayonnaise und anderem abgesalzen. Sie ist ein populäres Gericht in der Türkei und anderen Ländern des Mittelmeers. [end of text] ```
- benengel 3y agoIm on an m1 Mac book air 2020: Mistral: "Hi there! How can I assist you today?" total duration: 2.811067333s load duration: 4.093083ms prompt eval count: 9 token(s) prompt eval duration: 1.999431s prompt eval rate: 4.50 tokens/s eval count: 10 token(s) eval duration: 758.398ms eval rate: 13.19 tokens/s its pretty dang fast.
- juliensalinas 3y agoFor those who want to try Mistral 7b, here is a video that shows how to do it on an A10 GPU on AWS: https://www.youtube.com/watch?v=88ByWjM-KGM https://www.youtube.com/watch?v=88ByWjM-KGM