6 ms·
I’d like to note that this model’s parameter usage is low enough (13b) to run smoothly at high quality on a 3090 while beating GPT-3.5 on humaneval and sporting
by cuuupid 3y ago
I’d like to note that this model’s parameter usage is low enough (13b) to run smoothly at high quality on a 3090 while beating GPT-3.5 on humaneval and sporting 32k context.
3090s are consumer grade and common on gaming rigs. I’m hoping game devs start experimenting with locally deployed Mixtral in their games. e.g. something like CIV but with each leader powered via LLM
- minimaxir 3y agoThe average gamer doesn't have a 3090-equivalent, or even an Nvidia GPU. Running LLMs locally to create custom dialogue for games is still years away.
- somnic 3y agoVR isn't pragmatically accessible to the average gamer due to hardware requirements and the necessity of setting up the right physical environment but there are still VR games.
- __loam 3y agoThere's a few after almost a decade of vr being a thing.
- Cyphase 3y agoHow did you arrive at "almost a decade"? There's been VR stuff since at least the late 1980s. On the flip side you could say it wasn't "a thing" until just a few years ago. Or that it isn't "a thing" yet.
- __loam 3y agoI think it's pretty obvious that I'm talking about post oculus rift vr and the crop of games that use controllers similar to that device.
- kytazo 3y agoThis is wrong in many ways. First off you don't necessarily need a high-end consumer GPU, look at llama.cpp. Or even better look at something like phi-2. It's likely to go even lower. I'm sure there are people here who can detail more specifics. Its 1/24 and we're already there more or less.
- michaelmrose 3y agoWhy couldn't this be handled remotely and be handled as part of a subscription to the game?
- simion314 3y ago>Running LLMs locally to create custom dialogue for games is still years away I had think about this, you need a small LLM for a game, you do not need it to know about movies, music bands, history, coding and all the text on the internet. I am thinking we need a small model similar to phi-2 , trained only on basic stuff then you would train it on the game world lore. Then the game would also use some "simpler" graphics( we had good looking game graphics decades ago so I do not think you could be limited to text adventures or 2D graphics, just you need some simpler and optimized graphics, it is always interesting when someone shows his unreal demo that uses most RAM/VRAM for a simple demo level then a giant game like GTA5)
- Mashimo 3y ago> or even an Nvidia GPU. What do you mean? Most gamers do have an nvidia GPU. Edit: unless you talk about mobile gamers, and not PC gamers?
- fragmede 3y agoTo wit, the steam report says 74% of its users have Nvidia. https://store.steampowered.com/hwsurvey/Steam-Hardware-Software-Survey-Welcome-to-Steam/ https://store.steampowered.com/hwsurvey/Steam-Hardware-Softw...
- LeoPanthera 3y agoGoogle tells me that the RTX 3090 is priced between US$1,480 and $1,680. You can buy a whole PC for that, I refuse to believe that a GPU priced that highly is "consumer grade" and "common". Are there any GPUs that are good for LLMs or other genAI that aren't absurdly priced? Or ones specifically designed for AI rather than gaming graphics?
- Conasg 3y agoTo be fair, I got a card second hand to play with, and it was only £700 ($900~) and it came with manufacturer warranty. It was a bit of a gamble, but the 24GB VRAM has been a godsend for experimenting with LLMs. And playing video games at 4K!
- minimaxir 3y agotl;dr, no, especially since AMD is lagging behind. Apple is the one doing the best in terms of making consumer-friendly hardware that can perform AI/ML tasks...but that involves a different problem regarding video games.
- alchemist1e9 3y agoGamers and LLM/AI/ML GPU users do not find that absurdly priced. Absurdly priced in our world is $15,000 so your perceptions are off by about a order of magnitude.
- __loam 3y agoI can assure you a $1,500 graphics card is a big luxury for most gamers. https://store.steampowered.com/hwsurvey/Steam-Hardware-Software-Survey-Welcome-to-Steam https://store.steampowered.com/hwsurvey/Steam-Hardware-Softw... 3090 isn't even in the top 30.
- ryanwaggoner 3y agoAren’t there others in that list above the 3090 that are even more expensive?
- 3y ago
- RandomBK 3y agoIt's worth noting that the 4bit quants can run on cpu at ~reading speed, which should unlock many use cases - especially if we can precompute some of the results async.
- minimaxir 3y agoThat assumes full CPU utilization, with no other tasks heavily using the CPU. In the case of high-end video games, that's unlikely.
- RandomBK 3y agoTrue, yet many games are effectively single-threaded anyways. The bigger problem is memory capacity and bandwidth, but I suspect folks will eventually figure out some sort of QoS setup to let the system crunch LLMs using otherwise unused/idle resources.
- somnic 3y agoResource constraints would be a concern, yeah, so if you were developing a game featuring LLMs (which would, at this point in their development and maturity, be a gimmick) you would keep that in mind and keep other demands on resources low.
- ilaksh 3y agoAlthough in my testing the 4bit reasoning was not nearly as good.
- sanjiwatsuki 3y agoThe VRAM usage is closer to a 47B model - although only 2 experts are used at a time for inference, all experts are needed to complete it.
- discordance 3y agoConfirmed. Currently running Mixtral 8x7B gguf (Q8_0) on a Macbook Pro M1 Max w 64GB ram, and RAM usage is sitting at 48.8 GB.
- karolist 3y agoHow many t/s?
- discordance 3y agoAround 15 - 20 t/s
- karolist 3y agoThank you, got the same build M1 Max during Christmas B&H sale and can confirm it's amazing for running local LLMs.
- snickell 3y agoYou can also run Mixtral, at a decent token rate, on a post-2020 Apple Macbook Pro M1/M2/M3 with 32GB+ of RAM. 16GB RAM also works, sort of ok, which I suspect is the same quantization a 3090 is using, but I do notice a difference in the quantization. On my M2 Pro, the token rate and intelligence feels like GPT-3.5turbo. This is the first model I've started actually using (vs playing around with for the love of the tech) instead of GPT-3.5. An Apple M2 Pro with 32GB of RAM is in the same price range as a gaming PC with a 3090, but its another example of normal people with moderately high performance systems "accidentally" being able to run a GPT-3.5 comparable model. If you have an Apple meeting these specs and want to play around, LLM Studio is open source and has made it really easy to get started: https://lmstudio.ai/ https://lmstudio.ai/ I hope to see a LOT more hobby hacking as a result of Mixtral and successors.
- cjbprime 3y agoI don't think it's true that LM Studio is open source. Maybe I'm missing something?
- eyegor 3y agoLmstudio (that they linked) is definitely not open source, and doesn't even offer a pricing model for business use. Llmstudio is, but I suspect that was a typo in their comment. https://github.com/TensorOpsAI/LLMStudio https://github.com/TensorOpsAI/LLMStudio
- nraford 3y agoHow did you get Mixtral to run an a 32gb M1? I tried using Ollama on my machine (same specs as above) and it told me I needed 49gb RAM minimum.
- eurekin 3y agoI'm using: ollama run dolphin-mixtral:8x7b-v2.5-q3_K_S
- 3y ago
- fswd 3y agoyou cannot currently run mixtral with a 32k context on a 3090. Unless am I wrong? I think the largest context I was able to reproduce was around 1500 with 2 or 3 bit, I would have to look at my notes.
- LanternLight83 3y agoI've been working with local models as agents, and anyone interested in trying this needs to know about llama.ccp's "gammers" feature. You can force the model's output to conform to a specific structure, which is not only useful for ensuring that you recieve eg. valid JSON output but more specific stuff like "if you choose to do x, you must also provide y", which can be great for influencing it's thinking (eg. an actor who's planning ahead might be required to resond with three of any of the five W's (it's choice which three), but then it get's to be free-form inside of the JSON string values, which can be used as context for a following selection from a restricted set of actions; or a model might have option of asking for more time to think at the end if it's response, but if it doesn't then it needs to specify it's next action). This does't impact generation speed AFAICT and can be used in vry creative ways, but results can still need re-generated if they're truncated and I had to write a function to stop immediately when the valid JSON obj is closed (ie. at the end) or when more that like five newlines are generated it a row. This will vary by model.