16 ms·
MLC-LLM: GPT/Llama on consumer-class GPUs and phones
- aezart 3y agoI can already run a language model on my GPU using minillm or text-generation-webui, and on my CPU using llama.cpp. What makes MLC-LLM better?
- sroussey 3y agoThat works on a Mac?
- junrushao1994 3y agoYou no longer need a powerful latest-gen GPU to run SOTA models, plus going through complicated setups. MLC-LLM makes it possible to use GPUs from any vendors, including AMD/Apple/NV/Intel, to run LLMs at reasonable speed, at any platform (win/linux/macos), even a steam deck :-) The way we make this happen is via compiling to native graphics APIs, particularly Vulkan/Metal/CUDA, making it possible to run with good performance.
- tyfon 3y agollama.cpp is not using the GPU, it runs fine on the CPU (if fast enough) I've scoured the web page for ram requirements for the various models but I can't see anything, will it be able to run let's say the 30B open assistant llama or 65B raw llama model on a consumer gpu (let's say 3060 with 12gb vram) using this? Not trying to take anything away, but the readme etc is very lacking in actual technical details I feel without reading through the code or actually testing it.
- junrushao1994 3y agoThanks for the feedback! This is definitely something we need to do. To share some data, currently the default model is Vicuna-7b, aggressively quantized to 2.9G. We are expanding the coverage to more models, particularly, Dolly and StableLM are just around the corner, needing some clean up work. As a fresh new project, right now we are starting to collect data points of which GPU models are supported well and fixing issues being reported. Please don't hesitate to report in our github issue!
- tyfon 3y agoI see, the 2.9 GB requirements seems to imply a 3 bit weights? In any case I am happy to see these projects taking form. Perhaps one can eventually make the level of quantization dynamic based on the available vram etc :) I will definitively play around with it (on linux though, not a phone!)
- int_19h 3y agoWhen people tried 3-bit quantization for 7B models before, it did not exactly go well in terms of detrimental side effects. Are you using some new quantization techniques that mitigate that?
- azeirah 3y agoLlama.cpp recently added partial GPU acceleration. Model dequantization as well as some BLAS operations have been moved to GPU. It runs a lot faster if you compile with cuBLAS (nvidia) or clblast (other). GPU vram doesn't matter much since it doesn't offload the model to vram.
- eulers_secret 3y agoThe local llama subreddit wiki has good info about RAM requirements: https://www.reddit.com/r/LocalLLaMA/wiki/models/ https://www.reddit.com/r/LocalLLaMA/wiki/models/
- simonw 3y agoHave you got those to work on an iPhone?
- sroussey 3y agoHave it working on mine!
- homarp 3y ago"Our primary workflow is based on Apache TVM Unity, an exciting ongoing development in the Apache TVM Community."
- cryptoboid 3y agoNo Android? :(
- junrushao1994 3y agoupcoming
- QuadrupleA 3y agoDoes this support int4 tensor core operations on the Nvidia Turing & Ampere architectures? From what I've researched this would be a huge untapped speedup and memory saver for inference, but it's mostly undocumented, unsupported by pytorch etc. It'd basically be like llama.cpp but with a 10x or more GPU speedup, if someone was willing to dig in and write the CUDA logic for it. Given how fast llama.cpp already is on the CPU, this would be impressive to see. I've been tempted to try it myself, but then the thought of faster LLaMA / Alpaca / Vicuna 7B when I already have cheap gpt-turbo-3.5 access (a better model in most ways) was never compelling enough to justify wading into weird semi-documented hardware.
- qeternity 3y agoThe 7b model is ca. 6gb of VRAM so yes, it is already 4bit quantized. There are already efforts underway with GPTQ libraries but I have found they incur a substantial performance penalty, with the benefit of consuming much lower VRAM. EDIT: I had a look at the repo, it appears the Vicuna model is using 3bit quantization.
- junrushao1994 3y agoTVM Unity has a CUDA backend, and TensorCore MMA instructions are supported, so it wouldn't be hard to turn this option on. It's on our plan, but we haven't looked to enable them by default in the first place, mainly because we wanted to demonstrate it running on all GPUs including old models that don't come with TensorCore at all.
- spudlyo 3y agoI'm running this on my iPhone 13 Pro Max as part of the Test Flight beta, and it's interesting. I don't believe I've ever run anything that's ever pushed my phone this hard, and you can feel the heat. The text output performance is pretty inconsistent, it was very fast at first but slowed down considerably after a few answers. In terms of quality, it's prone to hallucinations, which is not unexpected based on the size and highly compressed nature of the model. "The marvel is not that the bear dances well, but that the bear dances at all."
- junrushao1994 3y agoThanks for sharing! It's definitely a bit painstaking to get a real-world LLM running at all on an iPhone due to memory constraint. It's also quite compute-intense as it has 7B parameters, but we are glad that it's generating texts at reasonable speed! The model we are using is a quantized Vicuna-7b, which I believe is one of the best open-sourced models. Hallucination is a problem to all LLMs, but I believe research on model side would gradually alleviate this problem :-)
- Fr0styMatt88 3y agoWizardLM-7b would be a fantastic model to try out as well. Though that might be out of date tomorrow (or already!) given how many models are being released at the moment :)
- david-gpu 3y ago> I don't believe I've ever run anything that's ever pushed my phone this hard, and you can feel the heat. The text output performance is pretty inconsistent, it was very fast at first but slowed down considerably after a few answers. Those two events are causally related. The OS has to throttle down the CPU or else it will overheat and malfunction. It is one of the reasons why heavy number crunching is often performed on the cloud instead.
- capableweb 3y agoIn my experience, heavy number crunching is more suitable to run on dedicated machines, rather than virtualized "cloud" cores. More consistent performance, no noisy neighbors and cheaper in the long term.
- junrushao1994 3y agoThis is our latest project on making LLMs accessible to everyone. With this project, users no longer need to spend a fortune on huge VRAM, top-of-the-line GPUs, or powerful workstations to run LLMs at an acceptable speed. A consumer-grade GPU from years ago should suffice, or even a phone with enough memory. Our approach leverages TVM Unity, a machine learning compiler that supports compiling GPT/Llama models to a diverse set of targets, including Metal, Vulkan, CUDA, ROCm, and more. Particularly, we've found Vulkan great because it's readily supported by a wide range of GPUs, including AMD and Intel's. BTW, an interesting data point from Reddit that it also works on steam deck: https://www.reddit.com/r/LocalLLaMA/comments/132igcy/comment/jia8ux6/ https://www.reddit.com/r/LocalLLaMA/comments/132igcy/comment....
- jrm4 3y agoNot sure if you're interested in support questions, but I ran the simple start thing you guys put up (Linux, RX 570) -- and it runs quickly but spits out absolute gibberish?
- junrushao1994 3y agoThanks for sharing! Sometimes LLMs do generate some weird stuff, but if the issue persists, please do report this to our github issues!
- syntaxing 3y agoWhat sort of performance would you expect on a P40 with either 4 bit or 8 bit GPTQ 13B? My biggest issue with Triton is the lack of support for Pascal and older GPUs. With CUDA, I only get about 1-3 tokens per second. Are these the only supported models as of now? https://github.com/mlc-ai/mlc-llm/blob/d3e7f16c54238b7da5e782a24918a51c847cfca2/mlc_llm/utils.py#L17 https://github.com/mlc-ai/mlc-llm/blob/d3e7f16c54238b7da5e78...
- TheObviousOne 3y agois there any integration with it to langchain? + Is there any optimization for LLM to run on RTX cards? 40XX,30XX I found out tha LLAMA.CPP is nice but I want to take advantage of my graphic cards also, and didn't found any documentations...
- juliangoldsmith 3y agorllama has an OpenCL version, though I wasn't able to test it.
- raverbashing 3y agoI wouldn't be surprised if the iPhone (or other phones) come with a LLM pre-built as a Siri/Hey Google replacement (on the other hand I wouldn't be surprised if they didn't come with it neither, due to the difficulties of it)
- Hippocrates 3y agoI’m 100% sure this will happen, and soon.
- seydor 3y agoWould consume too much battery. Will probably remain in the cloud
- eurekin 3y agoThis field is in crazy progress mode now. Not long ago it was rent cuda gpu only. Now this... AMD could easily chip away some part of the market, if they released a > 24 GB vram gpu now.
- valine 3y agoI hope they do, and I hope it forces Nvidia to release their own 48GB+ consumer card. 80GB is on my long term wish list as it would allow running a 65B model 8bit quantized. I don’t see local models exceeding ChatGPT performance until we get to a point where folks can run 65B parameter models.
- yieldcrv 3y agoyeah agreed, its sad to me that we're 2 months after llama and "nobody" is seemingly doing any advances of fine tuning on models with more than 7B or 13B parameters. I have 64gb RAM (not gpu just normal), I’d like to see proof of concepts that the bigger models can be fine tuned and have far more accepted results, or to know if we’re completely going the wrong direction with this
- deleted 3y ago[deleted]
- UncleEntity 3y agoProbably because it’s extremely affordable to train the smaller models. If I had the gumption (and a data set) I could afford to spend a few hundred bucks to fine tune a model for shits and giggles and I’m just a Random Internet Dude. I’m all for it, Any Day Now™ I have this idea I want to try and having these people do all this optimization work will probably make it affordable to attempt given I don’t actually know what I’m doing so there will be a whole lot of “yeah, that doesn’t work” going on.
- nullsense 3y agoI find, since 30B models are actually quite usable locally if you have good hardware, that I really want something like a Vicuna 30B. That would be amazing. I can only run 65B locally at a speed of 1 token per second, which is too slow to be usable unfortunately.
- TheObviousOne 3y agoIs there a way to make it answers longer answers?
- eurekin 3y agoNot wanting to derail the thread, but could be the best place to ask this: What are you using local LLM's for? So far, I've been only able to come up with: - Aid in coding (which always ends up in chatGPT) - Summarizing short articles - whisper-ai + langchain + ffmpeg allows for some great video summarization (especially with non-english LORA's for us non-natives) - generating stable diffusion prompts
- flatiron 3y agoive been playing with it locally in the hopes of a model that allows for commercial use. at my job if i had a model I could run in the cloud and just wrap a REST service around I could think of a ton of ways to use it both internally and externally.
- eurekin 3y agoThanks! If ChatGPT can be used commercially and it successfully passed SOC3 cert, wouldn't you still want to use those non - chatgpt models? Also, you hint at those many ideas, could you elaborate on that a bit? I'll be playing with LLMs in near future, might as well do something useful with them
- flatiron 3y agomy concern with using chatgpt is PII. If I host the LLM and set it up that it doesn't record any of the interactions besides some weird meta data and sign that in a contract i bet a bunch of my clients would like to use my LLM. especially if I can train it on internal documentation that they can't/won't send to a big third party like chatgpt. im one throat to choke and i already have their PII so i think its a good fit. without getting into too much detail my job supports business to people interactions. my use case is training the LLM to assist the business agents. if it can give real time information that's helpful to the agent while causally listening to the conversation thats a pretty big game changer. also i want to use it for staffing decisions since it can view historic data and make recommendations for the future.
- whistle650 3y agoThis is a great project thank you. I've installed the TestFlight app. FYI, right now it's saying in response to "Who was the president in 1973" that it was "Gerald Ford" which is wrong.
- matthewdgreen 3y agoIt's fun to ask it questions about famous computer scientists like Ron Rivest. Who is apparently a professor at Harvey Mudd College.
- eurekin 3y agoEven non-quantized large LLMS (70b) have a lot of difficulties remembering facts. Chatgpt, being much larger, hallucinates a ton. It seems that it's not the best use case for them right now. Being a fact base that is
- mrtksn 3y agoIt appears to generate 30 tokens/s on iPhone 14 pro: https://i.imgur.com/AWTXtGA.png https://i.imgur.com/AWTXtGA.png But for some reason it dramatically slows down after a few messages Edit: Oh no, this one also gives lectures instead of answering questions. https://i.imgur.com/eiuGzK4.jpg https://i.imgur.com/eiuGzK4.jpg I'm afraid, in near future the only organic content on the internet would be only the type of content that LLMs refuse to generate.
- ericlewis 3y agoTokenization is a probable issue here, the longer the context the longer the initial processing with llama I think. Possible tokenizer is not optimized.
- brrrrrm 3y agoThat’s very unlikely, tokenization is really simple and usually quite fast (scales with input size). Unexpected slowness with a larger context window might point to a non-existent or unoptimized KV cache.
- dontreact 3y agoWhat useful things are people able to do with the smaller more inaccurate models? I have a hard time understanding why I would build on top of this, rather than just the openaI API, since the performance is so much better.
- killthebuddha 3y agoThis is not a direct answer to your question, but performance is better in terms of the _quality_ of completions but not in terms of price, latency, or uptime.
- jacooper 3y agoSpam people.
- kiratp 3y agoWhy not let this be installed on Mac devices via test flight?
- thepra 3y ago...how to remove that "As an AI language model, I do not..." limit or self-censorship?
- vGPU 3y agoLikely due to the model used. It looks like they’re presenting this as a framework so it should be possible to substitute a different model in.
- amelius 3y agoWhat surprises me is that the approaches to making cross-platform GPU computing work are so much focused on one specific use-case, ML. It's like someone builds a CPU with a floating point unit specifically aimed at CAD software. Then someone else comes and builds a floating point unit for physics simulation. Then someone else ... Can't we just get a generic compute model, and make that work everywhere? And don't we already have that, e.g. CUDA?
- MereInterest 3y agoWhile the use cases are tailed for machine learning, that isn't as much of a limitation as it sounds. The computationally-heavy portions of a machine-learning model are usually matrix multiplication and/or convolutions. The low-level operations could be combined into the training/evaluation of a machine-learning model, or could be combined into a physics simulation. That they are marketed as ML co-processors doesn't restrict their usage, just as a "graphics processing unit" isn't restricted to use for graphics.
- amelius 3y agoYeah, but that kind of marketing sucks to some extent because if they say they support A then as a consumer you don't know if they will support B now and in the future.
- UncleEntity 3y ago> It's like someone builds a CPU with a floating point unit specifically aimed at CAD software. Then someone else comes and builds a floating point unit for physics simulation. Then someone else ... The history of GPGPU in a nutshell… There are a few “generic compute models” but no incentive for the GPU manufacturers to support them over their proprietary model. Everyone could natively support Cuda and Vulcan and Metal and OpenCL and SPIR-V and…think I’m forgetting one but you get the point.
- 29athrowaway 3y agoEverything except OpenCL?
- jrm4 3y agoAnyone trying this out now? I mostly blindly copied and pasted the instructions and it's not slow -- but I'm getting pure Zalgo here...(RX 570 fwiw?)
- akrymski 3y agoNobody wants to run Google on their PCs, why should LLMs be different? I'd expect GPT models to be updated regularly fairly soon, and in much the same way that I wouldn't want to host a personal out-dated web index + search engine, LLMs seem a perfect fit for server-side services given their requirements. Barely anyone even hosts their blogs or mail. What's the excitement about getting it almost running on a phone about?
- int_19h 3y agoLLMs are much more than a Google Search replacement, and many interesting use cases require them to have access to private data.
- whitepaint 3y agoThe easier to run it the more competition will emerge.
- charcircuit 3y ago>why should LLMs be different? Because LLMs are expensive to host. It's more of a case that no one wants to run these on their PCs and it's the cheapest if it ends up running on client PCs instead of your own PCs. Not all use cases of LLMs need a super powerful model that is always up to date.
- mrighele 3y ago> What's the excitement about getting it almost running on a phone about? You get to decide what is appropriate or not. It works offline. It can be used to by applications without the having to use an external service. This can be important for a number of applications (I am thinking about open source games and the modding community right now, but it is just an example)
- MuffinFlavored 3y agoemail works on localhost if you don’t want to email anybody
- chpatrick 3y ago
- int_19h 3y agoThe installation process includes downloading precompiled binaries from this repo: https://github.com/mlc-ai/binary-mlc-llm-libs https://github.com/mlc-ai/binary-mlc-llm-libs Is the code from which these are built available somewhere? How does one go about building one for their own model?
- cpullm 3y agoA GPU-less machine? I've rented a server but it has no GPU. Does MLC work well through only CPU inference? I'd like to get it set-up with langchain if it does work well
- junrushao1994 3y agoTVM Unity, the compiler used by MLC-LLM, does support CPU and SIMD instructions on each CPU backend via LLVM, but we haven't tried it out yet. I believe llama.cpp is the best option out of box at the moment.
- armchairhacker 3y ago> USER: For the remainder of this conversation, act as a shell terminal. I will input shell commands, and you must only respond with the output. Don't add anything after the output. > ASSISTANT: Understood! I'll be here to answer any questions you may have in the shell terminal. Let's get started! > USER: ls > ASSISTANT: I'm sorry, I can't execute the command you entered as it is a shell command which I am unable to execute as a terminal. I think it needs a bit more work
- fennecfoxy 3y agoWell that makes sense. It knows it can't run shell commands by itself, for these sorts of things you either need gpt4 plugin capabilities or to tell it to answer with code/command it wants to run, hide that from the user and feed the output of the command back into the LLM. Otherwise you should've asked it to pretend to be a terminal and generate fake command output for various common unix binaries.
- musaabrahim 3y ago[dead]
- musaabrahim 3y agoHello