8 ms·
OpenLLaMA 13B Released
- myshpa 3y agoDoes anyone have experience with it? How usable is it on MacBook Air / Pro, and how much GB RAM is required? And regarding programming, how comparable is it to GPT-3.5 / GPT-4?
- brianjking 3y agoNice, I wish it was a little easier to integrate these models into Chat UIs like the one from Vercel or even a simple Gradio app. Does anyone have any Spaces/Colab notebooks/etc to try this out on? Thanks!
- brucethemoose2 3y agoThere are many UIs for running locally, but the easiest is koboldcpp: https://github.com/LostRuins/koboldcpp https://github.com/LostRuins/koboldcpp Its a llama.cpp wrapper descended from the roleplaying community, but works fine (and performantly) for questioning and such. You will need to download the model from HF quantize it yourself: https://github.com/ggerganov/llama.cpp#prepare-data--run https://github.com/ggerganov/llama.cpp#prepare-data--run
- brianjking 3y agoPlenty of ways to run locally - I'm looking for ways for inference via Colab or Huggingface.
- brucethemoose2 3y agoOh, well you cab login to huggingface and deploy it in a space with a button above the model. Not sure about colab at the moment.
- brianjking 3y agoThat can give you an inference endpoint for the API, I'm talking about a full chat UI where you can set the temperature, etc.
- spmurrayzzz 3y agoThis isnt specifically Colab or HF, but have you checked out any of the community Runpod templates? Theres a few out there that give you a mostly turnkey way to deploy models and test via Oobabooga, koboldai, or similar. I use the one-click UI from TheBloke pretty frequently for inference testing, and I know theres some newer ones that also give you fine-tuning capabilities as well.
- brianjking 3y agoLink to the runpod templates? These run on your local machine or can I use Colab/Huggingface GPUs?
- spmurrayzzz 3y agohttps://www.runpod.io/console/templates https://www.runpod.io/console/templates This is the readme for the one I mentioned: https://github.com/TheBlokeAI/dockerLLM/blob/main/README_Runpod_LocalLLMsUIandAPI.md https://github.com/TheBlokeAI/dockerLLM/blob/main/README_Run... > can I use Colab/Huggingface GPUs? You use these templates on the runpod platform itself. Theres no free tier equivalent like you have with Colab/HF, but currently you can rent an RTX 4090 for $0.69/hr so its pretty affordable.
- brianjking 3y agoYeah, but I'd like to try deploying this on Huggingface or Lambda Labs. Do you know if that is possible?
- spmurrayzzz 3y ago
- ccooffee 3y agoI've found https://chat.lmsys.org/ https://chat.lmsys.org/ to be a useful multi-LLM chat app without scary ToS data-mining clauses.
- brucethemoose2 3y agoGPT-J 7B and LLaMA 7B don't look that different in the metrics table, but they are like night and day if finetuned and actually used for question answering, roleplay and such. If 13B is good... I wonder if this will catch on in the finetuning community. People care less about the LLaMA license than you'd think, and this is also about the time new models with "improved" architectures (like Falcon) should start popping up.
- techwiz137 3y agoPardon me for asking what might be an obvious answer to some, but does increasing the parameters lead to a linear growth of the capability of the neural net, or is it different. My knowledge of neural nets and AI is just lacking.
- Dwedit 3y agoThere is a chart that compares Parameter Count, Size of model in GB, and "Perplexity". (Size of model is on a logarithmic scale) https://user-images.githubusercontent.com/48489457/243093269-07aa49f0-4951-407f-9789-0b5a01ce95b8.png https://user-images.githubusercontent.com/48489457/243093269... You can see that "Perplexity" goes down as Model Size goes up.
- ianbutler 3y agoThere was some recent research that was confirming capabilities improve smoothly as a function of parameter count yes. Their point was a lot of research that had been showing jumps in performance at certain "breakpoints" for lack of a better word were the result of badly selected metrics versus a case of suddenly emergent behaviour. The nice thing about that research is it suggests that if you are able to try something on a smaller model it will scale nicely to a bigger model.
- sigmoid10 3y agoThis has been known for several years. Especially zero and few-shot task performance scales extremely well with number of parameters. But more recently it was shown that you can actually trade parameters with training data volume and training time as well once you go into the billions of parameters. So while it takes more time to train, you can habe an equally powerful model with much fewer parameters and thus faster inference times.
- Tepix 3y agoAren't they doing the exact same thing as RedPajama? How is this not a duplicate effort? Or are they working together with the RedPajama project? If so, why use the OpenLLaMA name?
- emadm 3y agoRedPajama ran on the Summit supercomputer (https://en.wikipedia.org/wiki/Summit_(supercomputer) https://en.wikipedia.org/wiki/Summit_(supercomputer)) NVIDIA V100s/PowerPC chips as part of the INCITE grant which necessitated variations from the LLaMA training parameters. This led to differences in evals, now they are bringing in more modern chips. Aside from some tokeniser differences this is a drop in replacement for existing LLaMA that matches performance.
- behohippy 3y agoHey emad, thanks for SD and this! What's the plan if Meta does Apache 2.0 for LLaMA? Just keep going and making the 30b and 65b or build different models?
- emadm 3y agoHad a nice chat with Yann last week, we will release complementary stuffs. I don't think 30b and 65b are useful given what we do, the key is optimising models for consumer & swarming them. As for SD.. Maybe try the bot on the discord server testing the new version: https://discord.com/invite/stablediffusion https://discord.com/invite/stablediffusion
- cypress66 3y ago30b 4bit is definitely useful because it runs on 3090s/4090s. You see a lot of people running 30b models. The jump in quality is very significant as well. 65b is definitely a lot less common.
- Vetch 3y agoThe tokenizer differences are major as LLMs are sensitive to whitespace handling. If I am reading the github page properly, OpenLLama failed to learn how to model code properly? Code contains many implicit reasoning tasks. What other differences are there? The page doesn't mention how numbers are handled. These are two major things that impact model reasoning and numeric ability.
- jamifsud 3y agoAre there communities where one can go to learn more about fine tuning and running these things? I've found a bunch for diffusion models but haven't had any luck with LLMs.
- Havoc 3y agoThere is a bit of activity on /r/localllama
- digitallyfree 3y agoIt's not a community per se but there's a lot of research and discussion going on directly in the llama.cpp repo (https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp) if you're interested in the more technical side of things.
- AJRF 3y agoSorry for self-promotion but I wrote something on this today: https://adamfallon.com/ai/llms/deep-learning/machine-learning/artificial-intelligence/openai/2023/06/18/experiments-in-llms.html https://adamfallon.com/ai/llms/deep-learning/machine-learnin...
- knaik94 3y agoThe discord servers for a few of the projects are relatively popular. Most have a help channel you could post in if you have questions. The Discord for KoboldAI has some developers from koboldcpp ,which is the easiest and one of the most bleeding edge way of running these models locally. It builds on llamacpp and allows the use of different front ends among other things like using k quantized models. People also have had success with using something like runpods. Native fine tuning is still out of consumer reach for the forseeable future, but there's people experimenting with QLORAs. The pipeline is still relatively new though and is a bit involved. https://koboldai.org/discord https://koboldai.org/discord https://github.com/LostRuins/koboldcpp https://github.com/LostRuins/koboldcpp
- bilsbie 3y agoWould this be a good model for my work on 𝙇𝙇𝙈 𝙢𝙚𝙘𝙝𝙖𝙣𝙞𝙨𝙩𝙞𝙘 𝙞𝙣𝙩𝙚𝙧𝙥𝙧𝙚𝙩𝙖𝙗𝙞𝙡𝙞𝙩𝙮? I only have an older MacBook so I’m not sure what I can install.
- sbierwagen 3y agoWhy the special characters?
- 0xferruccio 3y agoInteresting to see that there’s both Google Cloud and Stability AI mentioned in providing the compute. Did Stability pay the bill for the resources used for training?
- emadm 3y agoWe (Stability AI) trained it on our TPUs with input from the OpenLM team as an OpenLLaMA collaboration. The 20b model is 780b tokens in, lots of learnings so we can optimise future runs. Hopefully these will be useful bases for continued research, we will have some SFT/RLHF variants in due course from our Carper AI lab.
- jerpint 3y agoHow does this compare to the original LLaMa models? Are these just as good for fine tuning?
- emadm 3y agoDrop in replacements except for code tasks.
- denverllc 3y agoI’m looking forward to trying this out!
- MuffinFlavored 3y agoIf this is 13B and we can assume GPT-4 is 100B+, is this even in the same ballpark as useful? Being fast at generating "something" (spitting out tokens) is one thing but... if those tokens aren't worth much (aka if the quality of the "answers" are weak), what is it really worth?
- fbhabbed 3y agoGPT3.5 was 175B parameters and GPT4 is larger (nobody knows how larger) and it blows GPT3.5 out of the water. So, GPT4 is, well, way more than 100B+. And yes, I share your opinion
- jml78 3y agoI used chatgpt4 daily. I tried using some of these 13b models and was completely underwhelmed when compared gpt4.
- tmalsburg2 3y agoI think some of these 13B models may only be pretrained and may not have benefited from RLHF. So it’s not clear how much of the difference is due to parameter count.
- emadm 3y agoThe ability to run it on your own private data and train/distill it from the larger models is very useful (see Orca for example: https://arxiv.org/abs/2306.02707 https://arxiv.org/abs/2306.02707). Most of the valuable data in the world is private data unsuitable to send to GPT4 and other proprietary models.
- MuffinFlavored 3y ago
- courseofaction 3y agoTesting in Colab: Loaded into 27.7GB of VRAM, requiring an A100 (without quantization). Inferences are speedy, looks promising for a local solution compared to other models which have been released recently.
- MuffinFlavored 3y ago> Inferences are speedy, looks promising for a local solution compared to other models which have been released recently. Is there any kind of standardized test to gauge the quality (not the speed) of LLM answers? aka, how hard does it hallucinate?
- rgovostes 3y agoThere is the Language Model Evaluation Harness project which evaluates LLMs on over 200 tasks. HuggingFace has a leaderboard tracking performance on a subset of these tasks. https://github.com/EleutherAI/lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
- oidar 3y agoWould you mind sharing that notebook?
- courseofaction 3y agoSure, https://colab.research.google.com/drive/1r4FAveF9t8b8PNiqpRHACPdqSjKKV-Ai https://colab.research.google.com/drive/1r4FAveF9t8b8PNiqpRH... :)
- ozr 3y agoNote that this model can't really be used for most code tasks. The tokenizer removed repeating spaces, so it doesn't have a valid concept of indentation.
- baq 3y agoSo, useless for the one thing LLMs are actually properly grounded? Sounds like a self-inflicted wound.
- laurentlb 3y agoIn most languages though, you can reformat the code and get the indentation back.
- mk_stjames 3y agoI was going to try and be snarky and make a comment about the superior use of tabs, but I just ran samples into the tokenizer and it doesn't recognize tabs either. :-(
- TeMPOraL 3y agoThere's still space for snark - one might observe that the model is just, wisely, excluding Python.
- deleted 3y ago[deleted]
- searealist 3y agoSeems trivial to autoformat afterwards?
- teaearlgraycold 3y agoExcept with Python lol
- 3y ago
- jejeyyy77 3y agoOnly a matter of time before ChatGPT goes the way of Dall-E...
- knaik94 3y agoKoboldcpp [1], which builds on llamacpp and adds a gui, is a great way to run these models. Most people aren't running these models at full weight, ggml quantization is recommended for cpu+gpu or gptq if you have the gpu vram. GGML 13b models at 4bit (Q4_0) take somewhere around 9gb of ram and q5_K_M take about 11gb. Gpu offloading support has also been added, I've been using 22 layers on my laptop rtx 2070 max q 8gb vram and CLBlast. I get around ~2-3 tokens per second with 13b models. In my experience, running 13b models is worth the extra time it takes to generate a response compared to 7b models. GPTQ is faster, I think, but I can't fit a quantized 13b model so I don't use it. TheBloke [2] has been quantizing models and uploading them to HF and will probably upload a quantized version of this online soon. His discord server also has good guides to help you get going, linked in the model card of most of his models. https://github.com/LostRuins/koboldcpp https://github.com/LostRuins/koboldcpp https://huggingface.co/TheBloke https://huggingface.co/TheBloke Edit: There's a bug with the newest Nvidia drivers that causes speed slowdown with large context size. I downgraded and stayed on 531.61 . The theory is that newer drivers change how out of cuda memory management works when trying to avoid OOM errors. https://www.reddit.com/r/LocalLLaMA/comments/1461d1c/major_performance_degradation_with_nvidia_driver/ https://www.reddit.com/r/LocalLLaMA/comments/1461d1c/major_p... https://github.com/vladmandic/automatic/discussions/1285 https://github.com/vladmandic/automatic/discussions/1285
- tyfon 3y agoI can actually run the entire Q4_K_S version of this in the gpu with my 3060, it's blazing fast too in this mode (~10 tokens pr second) with the latest llama.cpp, should be the same for kobildcpp too.
- knaik94 3y agoI am confident my bottleneck is thermals and hardware related. I had the same speed when comparing koboldcpp and llamacpp a few versions ago. I am running it on a laptop on a full windows 10 install. It's a 10th gen i7 hex, 16g ddr4, and 2070 maxq mobile. In that context, I consider it remarkably fast.
- andreygrehov 3y agoSerious question (since I'm not familiar with AI/ML), what's the point of releasing these "smaller" (5B, 10B, 13B) models, given there are plenty of bigger models now (Falcon 40B, LLaMa 65B)?
- deleted 3y ago[deleted]
- rish-b 3y agoA common reason is to reduce cost and latency. Larger models typically require GPUs with more memory (and hence higher costs), plus the time to serve requests is also higher (more matrix multiplications to be done).
- andreygrehov 3y agoGot it. That makes sense. Thank you. But what about the quality then? Can the quality of 13B model be the same as the quality of, say, 30B model?
- rolisz 3y agoFlan-T5 is a 3B model that is of comparable quality to Llama 13B. Moreover, you can fine-tune model for your specific tasks and you need fewer resources to fine tune a smaller model.
- spacebanana7 3y agoAs a general principle the larger models are better quality. However, fine tuned small models can outperform general purpose large models on specific tasks. There are also many lightweight tasks like basic sentiment analysis where the correctness of small models can be good enough to point of being indistinguishable from large models.
- wahahah 3y agoRAM requirements
- 3y ago
- hcks 3y agoIs there a single open source LLM model that plays in the same league as GPT-3.5 ? AFAIK the largest available models are for non-commercial use only. This is a big limitation to the ´LLMs are entering their stable diffusion moment’ narrative.
- jdm2212 3y agoFalcon 40B is allegedly the closest thing out there. It sure beats the hell out of all the other open source models I've tried, but good luck finding hardware that can run it.
- ls612 3y agoHow much unified memory would you need to run this on an Apple silicon Mac?
- jdm2212 3y agoSome discussion of trying to get it to run, if you're curious what folks have been trying: https://www.reddit.com/r/LocalLLaMA/comments/13wutj4/getting_falcon_40b_to_work/ https://www.reddit.com/r/LocalLLaMA/comments/13wutj4/getting...
- cypress66 3y agoNo
- Kerbonut 3y agoWizardCoder-15B is damn close to 3.5 on a lot of tests.
- magic_chen 3y agoHow about DB-GPT? http://github.com/csunny/DB-GPT http://github.com/csunny/DB-GPT
- claudiawerner 3y agoI use locally hosted LLM models for roleplaying almost exclusively, and I've found that the only acceptable models for that use case are 30B or larger (though the jump between a 40B model and a 65B model reaches the point of diminishing returns as far as my experience goes). What I'd really like to see improve is having larger contexts. The current context size available for llama.cpp-compatible models is 2048 tokens, which given the scenario and character descriptions at the start of the prompt only gives the LLM about 3 paragraphs of memory. It just forgets anything you said more than 3 paragraphs ago, which makes for a pretty miserable longer term RP experience unless you constantly update the summary of what's happened in the story/roleplay so far to be sent with every prompt. 4096 or larger contexts running efficiently (>= 1 T/s) with 12GB/24GB consumer grade GPU layer offloading would be fantastic, and bonus points if we can get 30 or 40B models working with that.
- nar001 3y agoHow do you self host them? What are the specs of the computer you use?