21 ms·
OpenLLM
- visarga 3y agoFine-tuning is the most important part, but it is under intense research today, things change fast. I hope they can streamline this process because these smaller models can only compete with big models when they are fine-tuned.
- uniqueuid 3y agoYes but fine tuning requires a lot more gpu memory and is thus much more expensive, complicated and out of reach of most people. To fine tune a >10B model you still need multiple A100 / H100. Let’s hope that changes with quantized fine tuning, forward pass only etc.
- chaoyu 3y agoThe OpenLLM team is actively exploring those techniques for streamlining the fine-tuning process and making it accessible!
- quickthrower2 3y agoWhat is the $ cost of a fine tune though? $500?
- vczf 3y agoCan you fine tune on an M2 with adequate memory?
- visarga 3y agoYou can fine-tune medium models 3..60B on a single GPU with QLoRA
- comfypotato 3y agoWhat exactly do you mean here that the smaller models can compete with the the larger once they are fine-tuned? What about once the larger models are fine-tuned? Are they then out of reach of the fine-tuned smaller models?
- rmbyrro 3y agoThey're probably referring to fine-tuning on private/proprietary data that is specific to a use case. Say a history of conversation transcripts in a call center. Larger models, like OpenAI's GPT, don't have access to this by default.
- quickthrower2 3y agoOpenAI’s API has fine tuning options for older GPT models: davinci, curie, babbage, and ada
- kohlerm 3y agoIt doesn't for the newer (relevant) ones. Fine-tuning them is expensive and slow, because they are large
- chaoyu 3y agoSmaller models are likely more efficient to run inference and doesn't necessarily need the latest GPU. Larger language model trend to have better performance over more different type of tasks. But for a specific enterprise use case, either distilling a large model or use large model to help with training a smaller model can be quite helpful in getting things to production - where you may need cost-efficiency and lower latency.
- moneywoes 3y agoIs there a good checklist or framework for fine tuning vs using a vector db to increase the context size
- phsource 3y agoCool stuff! How does this compare with Fastchat, which seems like another open source project that helps run LLM models? At a glance, it seems like it's going for lots of similar goals (run LLMs with interoperable APIs): https://github.com/lm-sys/FastChat https://github.com/lm-sys/FastChat
- chaoyu 3y agoOpenLLM in comparison focuses more on building LLM apps for production. For example, the integration with LangChain + BentoML makes it easy to run multiple LLMs in parallel across multiple GPUs/Nodes, or chain LLMs with other type of AI/ML models, and deploy the entire pipeline on Kubernete (via Yatai or BentoCloud). Disclaimer: I helped build BentoML and OpenLLM.
- mjburgess 3y agoStray thought: It would be better to specify NNs in terms of their training-size to weight-size in bytes. Rather than "No. Parameters", or at least, this ratio with the number of parameters. So, eg., I'd imagine ChatGPT would be, say: 100s PB in 0.5TB. The number of parameters is a nearly meaningless metric, consider, eg., that if all the parameters covary then there's one "functional" parameter. The compression ratio tells you the real reason why a NN performs. Ie., you can have 100s bns of parameters, but without that 100s PB -> 0.5TB, which you can't afford, it's all rather pointless.
- moffkalast 3y agoWell it's not a completely meaningless metric as it immediately tells you roughly how much memory you need to load it, which is kind of important?
- mjburgess 3y agoIf you look at my suggestion, it's to state exactly that memory -- rather than to estimate based on bits/parameter.
- moffkalast 3y agoWell then do explain a bit further, I still don't fully grasp what "100s PT in 0.5T" means exactly. 100 petatokens in half a trillion? Half a terrabyte? 100 seconds? Plus afaik base model training tokens don't have the same effect as fine tuning tokens, so there would need to be a way to specify each of those separately.
- aarnphm 3y agoHi all, I'm the main maintainer from the OpenLLM team here. I'm actively developing the fine-tuning feature and will release a PR soon enough. Stay tuned. In the meanwhile, the best way to track the development workflow is at our discord, so feel free to join!!
- moffkalast 3y agoVery cool, btw it's not mentioned in the readme so I assume it's only for running full precision models or do quantized GGML/GPTQ/etc. also work with it?
- user432678 3y agoThanks for the great project! Any chance, your team might consider more open platform than Discord for posting updates? I personally find Discord hard to use, and there’s no way to have sensible subscription (like RSS). Discord is usually muted.
- SimFG 3y agoLooks great! I'm planning to integrate it into my new project(to-chatgpt: https://github.com/SimFG/to-chatgpt https://github.com/SimFG/to-chatgpt), which will provide users of the ChatGPT applications with a wider range of LLM service options.
- chaoyu 3y agoLooking forward to it! OpenLLM is adding a OpenAI-compatible API layer, which will make it even easier to migrate LLM apps built around OpenAI's API spec. Feel free to join our Discord community and discuss more!
- Havoc 3y agoThis sounds promising. Smaller but custom trained/tuned models would be ideal - works for the task without the overhead
- logicchains 3y agoIn my experience it doesn't work like that. It's like if you take an idiot and spend a bunch of time training them, they'll still perform much worse than a moderately intelligent person with much less training. And smaller models can be pretty idiotic.
- sysstemlord 3y agoThen why do Chess AI perform much better than LLMs trying to play chess.
- Gasp0de 3y agoBecause they do completely different things? They literally have nothing to do with each other. Why do planes fly better than ships if ChatGPT can't do math?
- RugnirViking 3y agoProvided that the problem is suited to the strengths of an LLM at all. An example might be a small ai custom trained on documentation for libraries. You ask it a question like "how do I make the background move with parallax effect when you move the cursor". It's a little ambiguous, high-level concept, and probably not a single function. Small ai: likely makes up a function or suggests a single function which isn't sufficient. Refuses to budge from its answer or apologies and gets confused Large LLM: able to actually understand the question, combine several functions. If it doesn't work you can tell it why and it fixes it
- solumunus 3y agoWhy would a language model be good at playing chess?
- rustdeveloper 3y agoDo we know how LLMs available in OpenLLM and other open source LLMs compare to different versions of GPT models? I know there’s a leaderboard on huggingface: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... but it doesn’t contain GPT models.
- rahimnathwani 3y agohttps://lmsys.org/blog/2023-05-25-leaderboard/ https://lmsys.org/blog/2023-05-25-leaderboard/
- tomschwiha 3y agoThe GPT4/ChatGPT ones are in the source visible: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/blob/main/src/assets/hardcoded_evals.py https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
- xvilka 3y agoDoes it work only with text? Or image/video processing too?
- tmalsburg2 3y agoFrom the description of the repo: "An open platform for operating large language models (LLMs) in production."
- chaoyu 3y agoCheck out BentoML, which is the underlying serving framework used by OpenLLM, and it supports other type of models and modality such as images and videos.
- d4rkp4ttern 3y agoSuppose I’ve written code that calls the OpenAI API. Is there some library that helps me easily switch to a local/other LLM. I.e a library that (ideally) provides the same OpenAI interface for several models, or if not then at least the same interface.
- monkeydust 3y agoLangchain might be what you need.
- fortydegrees 3y agoThere's the OpenedAI-API extension for text-generation-webui: https://github.com/oobabooga/text-generation-webui/tree/main/extensions/openai https://github.com/oobabooga/text-generation-webui/tree/main...
- bart_spoon 3y agoLangchain offers abstraction across many LLMs, including OpenAI’s.
- d4rkp4ttern 3y agoFound a couple others that do something like this. Turns out I had bookmarked them a while ago. https://github.com/r2d4/openlm https://github.com/r2d4/openlm https://github.com/hyperonym/basaran https://github.com/hyperonym/basaran
- abrichr 3y agohttps://github.com/nomic-ai/gpt4all https://github.com/nomic-ai/gpt4all
- chaoyu 3y agoOpenLLM plan to provide an OpenAI-compatible API, which allows you to even use OpenAI's python client to talk to OpenLLM, user just need to change to Base URL to point to your OpenLLM server. This feature is working-in-progress.
- cultavix 3y agoAnyone know what terminal/theme they are using ? haha...
- zoom_enh4nce 3y agoThe theme looks like rose-pine-dawn.
- snowcrash123 3y agoThis looks like a very cool project and much needed
- sorenjan 3y agoWhat kind of hardware do I need to run something small scale (1 user concurrently) and get reasonable result? Are we talking about Raspberry Pi, Core i5, Geforce 4090?
- ben_w 3y agoI think that strongly depends on what you count as "reasonable": smaller models take less memory, so there's a trade-off between quality and speed depending on if you can fit it all in graphics VRAM, or system RAM, or virtual memory… Just keep in mind the speed differences between the types of memory. If you've got a 170bn 4-bit parameter model, and you're on virtual memory on a 400 Mbps port, a naive calculation says it will take at best 28 minutes per token unless your architecture lets you skip loading parts of the model. Might take longer if the network has an internal feedback loop in the structure.
- rmbyrro 3y agoWould you mind sharing the calculation memory that reaches at 28 minutes (!) per token?
- ben_w 3y agoIf your model is 170bn 4-bit parameters, you have 85 Gbytes that has to be loaded into the CPU or GPU; if those parameters are on the other side of a 400 Mbps port, that takes 85 Gbyte / 400 Mbps = 1700 seconds = 28 minutes and 20 seconds. If you don't have sufficient real RAM or VRAM, the entire model has to be re-loaded for each step of the process. Assuming no looping (looping makes it longer) and no compartmentalisation in the network structure (if you can make it so an entire fragment might be not-activated, you have the possibility of skipping loading that section; I've not heard of any architecture that does this, it would be analogous to dark silicon or to humans not using every brain cell at the same time (outside of seizures)).
- rootusrootus 3y agoAnecdote: I can tell you that Vicuna-13B, which is a pretty decent model, runs about 4 to 5 tokens/second on an Apple M1 with 16GB. Takes about 10GB of memory when loaded. Friend of mine with a RTX 2070 Super gets comparable results. I'm upgrading my M1 laptop on Thursday to a newer model, M2 MAX with 96GB of memory. I'm totally going to try Falcon-40B on it, though I do not expect it to run that well. But I do expect it (the M2, I mean) will be snappier on the smaller models than my original M1 is.
- hsuduebc2 3y agoHello can someone provide usecase for me as an user of this? Is it better because it is cheaper than commercially available apis?
- tomschwiha 3y agoImo its not cheaper (or has better quality) and is only worth it i) if you want to toy around with it ii) don't want to depend on the api to be available (or don't want to be censored or share sensitive information with a third party) iii) finetune your own model that you need to deploy by yourself
- DrNosferatu 3y agoThe project seems great! However, newcomers (like me) are pretty blind about minimum system requirements. Could you please add them to the models list? For example: what minimum hardware do I need to run Falcon-40b? PS: If you only have a few setups "known to work" (or just one), listing that would be helpful too.
- leetharris 3y agoFor falcon 40b you probably need an A100 40gb or so. Every model is drastically different. If you want to run something on consumer hardware, your best bet is using anything ported to the ggml framework, especially if you're on Apple silicon.
- frodowtf 3y agoDo you know any "standard" way or measures to determine the approximate hardware requirements of a model?
- tyre 3y agoYou want the entire model to fit in memory. So if you’re looking at a download size of, say, 100GB then don’t run on less than that. Your machine will swap to/from disk constantly, which will be slow and wear out an SSD. If you want to train a model, that’s a different story.
- MacsHeadroom 3y agoModel size in GB = VRAM required for uncompressed inference (16bit aka "half precision) plus ~1-4GB for context. For 8bit you need half that. For GPTQ/4bit you need 1/4 that. For example, assuming GPTQ 4bit a 100GB 16bit model needs 26-30GB of VRAM. The smallest video cards which meet this requirement will be 32GB or 40GB cards. (two 24GB cards in parallel work as well, e.g. 2x3090)
- Aerroon 3y agoTo add: usually when you go to download a ggml model you want a quantized version. People like TheBloke will usually have some RAM requirements for running it, eg: https://huggingface.co/TheBloke/vicuna-13b-v1.3-GGML https://huggingface.co/TheBloke/vicuna-13b-v1.3-GGML The number after q determines how many bits the weights are. Eg q4 means that is 4-bit. If you use something like KoboldCPP you can only put some of the layers onto the GPU and be able to run larger models that way. Eg the above linked Vicuna model requires about 10GB of memory at q4, but I have less VRAM than that. I can still run it though.
- moneywoes 3y agoIs this the best? Seeing a lot of these projects somewhat confused on what to go with
- asylteltine 3y agoWhat does it mean to “serve a model”? Where exactly does the request go and how does it interface with the model?
- thecal 3y agoThink of the model as a gigantic compiled binary where you send in strings in a certain format and get back a response. This is a web API wrapper for that so you only need an HTTP client instead of having to run something like llama.cpp yourself.
- sputr 3y agoQuestion: for someone that wants to play around with self-hosted text generation but has a crap laptop – are there any hosting providers (like a VPS) where I can run open source models?
- tomschwiha 3y agoYou can rent pretty easy a server with a GPU using runpod, vast.ai or datacrunch. Or maybe even use something Like Google Colab.
- sputr 3y agoThanks!
- Ajedi32 3y agoI like the idea of having a standard API for interacting with LLMs over the network. Many models need to run on beefy hardware and would benefit from offloading to a remote (possibly self-hosted) server, and I think makes logical sense to separate the code for running LLMs from the UI for accessing them.
- golergka 3y agoIt would be great to have this, but the space is rapidly moving and haven't converged on a set of uniformly accepted practices yet. For example, I'm not aware of a single open source LLM that has something similar to OpenAI's function calls.
- huggingmouth 3y agoWhy doesn't it support llama?
- 0i0 3y agovery cool.
- numpad0 3y agoThough only related to topic wrt LLMs: It seems LLMs occasionally mix up CJK vocabularies and also generate invalid UTF-8 sequences, due to CJK texts having overlapping code points and inputs being processed by tokenizer. Are there developments in that directions? Aren't CJK ideograms essentially tokens?
- YChacker100 3y ago[dead]
- spencerchubb 3y agoWhat is the license like for this? Correct me if I'm wrong, but I think the official Llama has a license that allows research use. Would this have a similar restriction if it had the same model architechture but different parameters?
- chaoyu 3y agoOpenLLM itself is under Apache 2 license, which does NOT restrict commercial use. However, OpenLLM as a framework can be extended to support other LLMs which may come with additional restrictions.