5 ms·
Ask HN: Open-source ChatGPT alternatives?
What's the state of the art in open source GPT models right now, in practical terms? If your typical use case is taking a pretrained model and fine tuning it to a specific task, which LLM would yield the best results while running on consumer hardware? Note that I'm specifically asking for software that I can run on my own hardware, I'm not interested in paying OpenAI $0.02 per API request.
I'll start the recommendations with Karpathy's nanoGPT: https://github.com/karpathy/nanoGPT
What else do we have?
- smoldesu 4y agoThis runs fine in RAM constrained (<2gb) situations: https://huggingface.co/EleutherAI/gpt-neo-125M https://huggingface.co/EleutherAI/gpt-neo-125M It's bigger brother, 1.3b, uses ~5.5gb of memory but yields slightly more GPT-like answers. Both take ~5-20 seconds to generate a response though, so take that into account when building with it.
- bcarambio 4y agoHi from OctoML -- our team just put out some benchmarking results using GPT-J with a few model config/hw combinations that have relatively low latency: /octoml.ai/blog/evaluating-gpt-j-deployment-options/
- bcarambio 4y agoHi from OctoML -- our team just put out some benchmarking results using GPT-J with a few model config/hw combinations that have relatively low latency: https:/octoml.ai/blog/evaluating-gpt-j-deployment-options/
- init0 4y ago[flagged]
- baobabKoodaa 4y agoDid you generate this with Bing? I don't believe this answer is human-written. It describes 2 separate projects with the words "It is available on GitHub and can be used in various research projects".
- speedgoose 4y agoIt’s also not very up to date which is suspicious for such a precise answer.
- franze 4y agoaccording to gptzero AI detection it: ".. text may include parts written by AI"
- flangola7 4y agoFacebook also has OPT, which is one of the largest public pre-trained models
- eric_hui 4y ago[dead]
- speedgoose 4y agoYou also have GPT J 6B and BLOOM but to be honest they are not like ChatGPT. https://huggingface.co/EleutherAI/gpt-j-6B https://huggingface.co/EleutherAI/gpt-j-6B https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom
- baobabKoodaa 4y agoCan you elaborate on how they are not like ChatGPT? I was looking into GPT-JT (built on top of GPT J you mentioned). If I spend the time to actually finetune and run inference with one of these models, am I likely going to be disappointed at the results?
- speedgoose 4y agoIt depends what you use it for. If it’s to classify text and you can fine tune it, it’s probably good enough. For following instructions, ChatGPT is a lot better but GPT J did relatively well if given enough examples on simple tasks. For a chatbot, it’s not really useable.
- smoldesu 4y agoMaybe? GPT-J is closer to the AI-Dungeon model of intelligence. It's able to fill in the blank after what you type, but it's hysterically bad at answering precise questions (to the point that I had to nerf it for fun to see how stupid the output could get). It will handle basic natural language and context clues just fine. It's just not very fast, and the generations probably won't be as thorough as ChatGPT.
- baobabKoodaa 4y agoAm I going to be able to fine tune GPT-J or GPT-JT on consumer hardware?
- deleted 4y ago[deleted]
- 4y ago
- mindcrime 4y agoPrevious related discussions: https://news.ycombinator.com/item?id=34115698 https://news.ycombinator.com/item?id=34115698 https://news.ycombinator.com/item?id=33955125 https://news.ycombinator.com/item?id=33955125 https://news.ycombinator.com/item?id=34163413 https://news.ycombinator.com/item?id=34163413 https://news.ycombinator.com/item?id=34628256 https://news.ycombinator.com/item?id=34628256 https://news.ycombinator.com/item?id=34147281 https://news.ycombinator.com/item?id=34147281 https://news.ycombinator.com/item?id=34445873 https://news.ycombinator.com/item?id=34445873
- baobabKoodaa 4y agoI went through all of these and the only one in there that I found that might be fine-tuneable on consumer hardware seems to be KoboldAI. Not sure yet.
- dieselgate 4y agoI'm not very familiar with this space but would have thought "OpenAI" would be at least somewhat open-source. Is this just naming and not relevant to the product at all?
- baobabKoodaa 4y ago> Is this just naming and not relevant to the product at all? They took funding as an open source non profit. Once they got the money they turned into a closed sourced for-profit censorship machine.
- titaniczero 4y agoYeah, but to be fair GPT 2 is open source and Whisper (a high-quality speech recognition and multilingual translation model) is also open source. A few years ago I needed a good model for transcription for a project and I couldn’t find anything decent. They really have contributed to the open source community. If they keep releasing older models and keep their cutting-edge technology for profit I’m fine with it.
- baobabKoodaa 4y agoFair enough.
- gerash 4y agoOpenAI stopped being a non-profit and hasn't published anything on Chatgpt yet
- astrange 4y agoThey have, although not datasets: https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/ https://openai.com/blog/deep-reinforcement-learning-from-hum... https://openai.com/blog/fine-tuning-gpt-2/ https://openai.com/blog/fine-tuning-gpt-2/ https://openai.com/blog/chatgpt/ https://openai.com/blog/chatgpt/ People like Anthropic have been able to reproduce it.
- PaulHoule 4y agoHere is a Python package that can download transformer embeddings automatically https://www.trychroma.com/ https://www.trychroma.com/ In general a lot of people download models from huggingface, I think that package automates that task.
- baobabKoodaa 4y agoI don't know if there is an implication here that I don't get, but I don't see the connection between this answer and the question I asked.
- PaulHoule 4y agoYou want a large language model. This gives you a large language model.
- baobabKoodaa 4y agoI asked for recommendations on which LLM would run on consumer hardware for the purposes of fine tuning and inference, with good results. You linked a package that can be used to download models? I don't see how these things are related.
- PaulHoule 4y agoWhy don't go ask ChatGPT then? But seriously I am asking a very similar question with a focus on LLMs for classification (e.g. "Is this an article about a sports game?"), information extraction, clustering and such. I am not so interested in generation (Which I am assuming you are.), however the GPT style embeddings and those are are useful for the kind of work I do and are interchangeable with BERT-like and other embeddings. "Good" or "Best" is something you have to define for yourself and the one thing every successful A.I. developer has done is develop a facility for testing if the solution was performing acceptably. With that library you can download a model and start working with the model, again, the successful people all tested at least one model. In the time since your post, a run-of-the-mill Python developer could have made some progress. Learn Python or get a non technical co-founder, For my kind of tasks I want something that handles bigger documents than ChatGPT and when I go shopping for models I cannot find a high quality very large transformer that has been assembled with a tokenizer and trained hard on language tasks. When I look at the literature it seems the very long transformers like reformer wouldn't perform so well if somebody did try to build an LLM so I wait. I am certain that somebody will upload a better model to huggingface someday -- that's the thought process it takes to get an answer for questions like yours. If you look though at the process used to make ChatGPT-3 able to converse there is the GPT-style embedding and then a process of dialog generation trained on totally different principles which is the "Reinforcement Learning from Human Feedback" https://www.assemblyai.com/blog/how-chatgpt-actually-works/ https://www.assemblyai.com/blog/how-chatgpt-actually-works/ and I think you are not going to get that kind of capability open source in that the training data doesn't exist for it. There are many things you have to do once you have that training data, but I think there are many people able to follow that path now that it has been blazed.
- valgaze 4y agoClue on conversation "history"-- "While ChatGPT is able to remember what the user has said earlier in the conversation, there is a limit to how much information it can retain. The model is able to reference up to approximately 3000 words (or 4000 tokens) from the current conversation - any information beyond that is not stored. Please note that ChatGPT is not able to access past conversations to inform its responses." https://help.openai.com/en/articles/6787051-does-chatgpt-remember-what-happened-earlier-in-the-conversation https://help.openai.com/en/articles/6787051-does-chatgpt-rem... Some interesting techniques I've seen involve essentially a ring-buffer and after each turn a call is made to summarize the conversation up to that point and use that as context for subsequent prompt
- roxgib 4y agoPresumably one of the benefits of running your own model is that you can feed extra data into it via training rather than purely through inference? I.e. if you're a software company you could fine-tune it on your codebase, improving its answers without increasing inference time?
- MagicMoonlight 4y agoChatGPT is itself just a GPT that has been finetuned to make it act as a chat bot and try not to be offensive
- ProllyInfamous 4y ago>essentially a ring-buffer and after each turn a call is made to summarize the conversation up to that point and use that as context for subsequent prompt Although people unfamiliar with "active listening" often find it annoying, this is something I attempt to implement within my conversations (quickly summarizing what I understand after somebody explains something technical). Oftentimes people feel like I'm `interrupting them`, but it is really just a way to stay engaged in two-way conversation. Eerie how LLM's benefit from this "focus" as similarly as this autistic commenter.
- SparkyMcUnicorn 4y agoOpen Assistant (started by some of the people that started Stable Diffusion I think?) is very early, but looks very promising. https://open-assistant.io/ https://open-assistant.io/ https://github.com/LAION-AI/Open-Assistant https://github.com/LAION-AI/Open-Assistant
- baobabKoodaa 4y agoI briefly looked at this and it doesn't seem like they provide a model that I can fine tune on consumer hardware?
- Mizza 4y agoHold your damn horses, this technology is brand new, requires a tremendous amount of data gathering and computation, there's a massive volunteer effort already under way, and you're begging for a free home version so you can save $0.02 cents. You want a hole, pick up a shovel.
- baobabKoodaa 4y ago> Hold your damn horses, this technology is brand new, requires a tremendous amount of data gathering and computation, there's a massive volunteer effort already under way, and you're begging for a free home version so you can save $0.02 cents. You want a hole, pick up a shovel. $0.02 cents per API request. What I would like to do is provide people a free service on the internet that using an LLM under the hood. If you are so rich that you can burn $10k on an internet hobby project, good for you, but just know that everybody else is not as rich as you. Also I wasn't "begging for a free home version". In my original question I already provided one option. Sorry, I mean "one free home version". So it's not like I was starved out of options. There are options. I was asking for recommendations.
- junipertea 4y agoEven openAI with the massive success and popularity gain can not sustain keeping it for free. The 0.02$ cents is a bargain for the value provided. OPs point was most of us can't burn 10k on a internet hobby project, and it costs probably 10x-100x more to train. How would you expect to provide people a free service? People are training it to open source right now, so please be patient.
- deleted 4y ago[deleted]
- lopuhin 4y agoIn terms of models which are reasonably fast to run and easy to install, I think Flan-T5 is one of the best: https://huggingface.co/google/flan-t5-xxl https://huggingface.co/google/flan-t5-xxl - although out of the box it's more focused on giving short answers and it's very far from ChatGPT.
- baobabKoodaa 4y agoIt's not clear from the link if it's possible to fine tune Flan-T5 on consumer hardware?
- lopuhin 4y agoThey released different sizes from "small" to "xxl", and at least "base" should be small enough to fine-tune virtually anywhere.
- baobabKoodaa 4y agoI spent several hours on this today. Currently stuck at "out of memory" error trying to fine-tune the base model.
- baobabKoodaa 4y agoWorked a bit with the "small" version of Flan T5 today. I was able to fine tune it, and the results were not complete garbage. About half were non sensical but the other half were passable.
- gigel82 4y agoGPT Neo 1.3B (https://huggingface.co/EleutherAI/gpt-neo-1.3B https://huggingface.co/EleutherAI/gpt-neo-1.3B) is the largest I can run on my 12Gb VRAM GPU, and I'm sorry to say it's output is a joke (nowhere near GPT-3, more like GPT-2 level of BS). However, you can fine tune it; and I'm sure with lots of fine tuning and some jiggling of the parameters you can get a half decent custom-purpose solution.
- aPoCoMiLogin 4y agohttps://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM
- rvz 4y agoWe seriously need more mature and open-source LLMs that are as good as, if not better than ChatGPT. It was a great surprise to see Stable Diffusion disrupt DALLE-2 with open source We need the same for ChatGPT and GPT-4.
- vvipgupta 4y agoYou might also want to check out https://github.com/lucidrains/PaLM-rlhf-pytorch https://github.com/lucidrains/PaLM-rlhf-pytorch
- moomoo11 4y agoI’m only a consumer of GPT. Any pointers as to where someone like me with software engineering experience but literally no AI knowledge can train my own GPT on my own data sets? In my case I have downloaded some public domain databases (1-7gb each) and I would like to get some additional insights out of them. I have been querying them and using them to build my company but I’m curious to know if GPT can help me in that regard.
- v3ss0n 4y agoTried all of them, they are still at gpt2 tier. None of them close to gpt3, so far away from davinci. To run your own laptop and cheap we still need much fine tuned training set with much better algorithm . Right now, most capable ones needs over 120gb of VRAM just inference (run).
- juliensalinas 4y agoSome alternatives are explained in this article: https://nlpcloud.com/chatgpt-open-source-alternatives.html https://nlpcloud.com/chatgpt-open-source-alternatives.html But it will require some prompt engineering in order to get the same level of instruction as ChatGPT.