12 ms·

Ask HN: What's the best self hosted/local alternative to GPT-4?

• Marlon1788 3y ago
  openai not so open. should rebrand to closedai
• airgapstopgap 3y ago
  There really do not exist any alternatives, self-hosted or not. But more importantly, there may never be, what with the rising tide of AI risks and regulations discourse. It seems that soon training and opensourcing or otherwise making accessible a model of that class will be impossible, even as the cost of its production falls.
  • logicchains 3y ago
    The UAE's apparently planning to release a Falcon 180B, which would be the most powerful model you can run at home by far. I don't think they'll bow down to western pressure to lobotomise the models given the country's whole business model is bypassing sanctions and the like.
    • airgapstopgap 3y ago
      That's assuming it's substantially superior to stuff like BLOOM. I sure hope that's the correct assumption to make. Of course, running a 180B dense transformer at home for personal use is utterly impractical.
• MacsHeadroom 3y ago
  Guanaco-65B[0] using Basaran[1] for your OpenAI compatible API. (You can use any ChatGPT front-end which lets you change the OpenAI endpoint URL.) [0] https://huggingface.co/TheBloke/guanaco-65B-HF https://huggingface.co/TheBloke/guanaco-65B-HF A QLoRA finetune of LLaMA-65B by Tim Dettmers from the paper here: https://arxiv.org/abs/2305.14314 https://arxiv.org/abs/2305.14314 [1] https://github.com/hyperonym/basaran https://github.com/hyperonym/basaran
• anotheryou 3y ago
  Here you can try vicunia (and quite a few others) easily https://chat.lmsys.org/ https://chat.lmsys.org/ They also have A/B testing with a leaderboard where vicunia wins for the self-hostable ones: https://chat.lmsys.org/?leaderboard https://chat.lmsys.org/?leaderboard
• meroes 3y ago
  This is like an artist getting used to Adobe’s products before they’re put behind a wall. And borrowing HN’s attitude to that, you apparently deserve it
• boringuser2 3y ago
  I've gone down this rabbit hole and I want to reaffirm what the other commenters are saying: even if you use a massive model and have the compute to back it up at a reasonable pace (you likely don't), it sucks, can't even hold a candle to GPT 3.5
• paulus-saulus 3y ago
  https://huggingface.co/tiiuae/falcon-7b https://huggingface.co/tiiuae/falcon-7b
• deet 3y ago
  In our experimentation, we've found that it really depends what you're looking for. That is you really need to break down down evaluation by task. Local models don't have the power yet to just "do it all well" like GPT4. There are open source models that are fine tuned for different tasks, and if you're able to pick a specific model for a specific use case you'll get better results. --- For example, for chat there are models like `mpt-7b-chat` or `GPT4All-13B-snoozy` or `vicuna` that do okay for chat, but are not great at reasoning or code. Other models are designed for just direct instruction following, but are worse at chat `mpt-7b-instruct` Meanwhile, there are models designed for code completion like from replit and HuggingFace (`starcoder`) that do decently for programming but not other tasks. --- For UI the easiest way to get a feel for quality of each of the models (or, chat models at least) is probably https://gpt4all.io/ https://gpt4all.io/. And as others have mentioned, for providing an API that's compatible with OpenAI, https://github.com/go-skynet/LocalAI https://github.com/go-skynet/LocalAI seems to be the frontrunner at the moment. --- For the project I'm working on (in bio) we're currently struggling with this problem too since we want a nice UI, good performance, and the ability for people to keep their data local. So at least for the moment, there's no single drop-in replacement for all tasks. But things are changing every week and every day, and I believe that open-source and local can be competitive in the end.
• captainmuon 3y ago
  I think you have to distinguish between self-hosted to run on CPU (like LLAMA), on consumer GPU or on big GPUs. I find the market currently very confusing. I'm especially interested since the data center I'm working for is sitting on a bunch of A100 and I get daily requests of people asking for LLMs tuned to specific cases, who can't or won't use OpenAI for various reasons.
• sgd99 3y ago
  Not self-hosted/local but Claude by Anthropic from what I've heard is really good but the API is not publicly available. It's apparently accessible via Poe (https://poe.com https://poe.com) As for open models, HuggingFace has a nice leaderboard to see which ones are decent: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
• kertoip_1 3y ago
  You can check out this leaderboard to see a current state of LLM alternatives to GPT4 https://lmsys.org/blog/2023-05-25-leaderboard/ https://lmsys.org/blog/2023-05-25-leaderboard/ But unfortunately for now it seems there aren't any viable self-hosted options...
• davepeck 3y ago
  There are no viable self-hostable alternatives to GPT-4 or even to GPT3.5. The “best” self-hostable model is a moving target. As of this writing it’s probably one of Vicuña 13B, Wizard 30B, or maybe Guanaco 65B. I’d like to say that Guanaco is wildly better than Vicuña, what with its 5x larger size. But… that seems very task dependent. As anecdata: my experience is that none of these is as good as even GPT3.5 for summarization, extraction, sentiment analysis, or assistance with writing code. Figuring out how to run them is painful. The speed at which their unquantized variants run on any hardware I have access to is painful. Sorting through licensing is… also painful. And again: they’re nowhere close to GPT-4.
• colesantiago 3y ago
  The best self hosted/local alternative to GPT-4 is a (self hosted) GPT-X variant by OpenAI. No kidding, and I am calling it on the record right here. OpenAI will release an 'open source' model to try and recoup their moat in the self hosted / local space. https://www.theinformation.com/briefings/openai-readies-new-open-source-ai-model https://www.theinformation.com/briefings/openai-readies-new-...
  • jfoster 3y ago
    Does it benefit OpenAI if people are using an "open source" OpenAI model versus any other "open source" model?
    • colesantiago 3y ago
      Great question. Yes it does benefit OpenAI because Sam is pushing for and betting on regulatory capture. Meaning AI models that are compliant with AI safety principles would be allowed and considered safe by law should regulations be put in place. Current open source models wouldn't be compliant and would require work to make them compliant, thus creating a moat for OpenAI in the self hosted space. Because there is nothing better than a free model that you can self host that OpenAI has made that is also regulated. Enterprises would love this, since OpenAI has the mindshare already, they can self host a licensed model in their orgs or air gapped environment that they know is regulated. OpenAI may release different model sizes of this GPT-X variant like they did quietly for Whisper, Bigger sizes may require an enterprise license.
• leros 3y ago
  Is anyone using a self hosted thing to assist with parsing?
• zorrobyte 3y ago
  What's the best self hosted for ingesting a local codebase and wiki to ask questions of it? Some of the projects linked here have ingest scripts for doc, pdf files; but it'd be cool to ingest a whole git repo and wiki, have a little chat interface to ask questions about the code.
• DebtDeflation 3y ago
  LLM Leaderboard: https://chat.lmsys.org/?leaderboard https://chat.lmsys.org/?leaderboard The short answer is that nothing self hosted can come close to GPT-4. The only thing that comes close period is Anthropic's Claude.
• TradingPlaces 3y ago
  As people note, you cannot substitute locally for the Azure GPU cloud that GPT-4 runs on. But I believe that will change, and maybe quickly. After years of explosive exponential growth in model size, all of a sudden, small is beautiful. The precipitating factor is that running large models for research is very expensive, but pales in comparison to putting these things into production. Expenses rise exponentially with model size. Everyone is looking for ways to make the models smaller and run at the edge. I will note that PaLM 2 is smaller than PaLM, the first time I can remember something like that happening. The smallest version of PaLM 2 can run at the edge. Small is beautiful.
• RecycledEle 3y ago
  I often worry about aa "The Machine Stops" scenario. GPT AI actually gives me hope. What if we can store and run an AI in a phone-sized-device that is superior to a similarly sized library of books? Can we have a rugged, solar-powered device that could survive the fall of Civilization and help us rebuild? It would certainly have military applications in a warfare. Imagine being the 21ct century equivalent of a 1940's US Marine on Guadal Canal who need to know some survival skills. ChatGPT-on-a-phone would be handy if you could keep the battery charged.
  • JimtheCoder 3y ago
    "Imagine being the 21ct century equivalent of a 1940's US Marine on Guadal Canal who need to know some survival skills. ChatGPT-on-a-phone would be handy if you could keep the battery charged." Would it not make sense to learn survival skills...beforehand? I would much rather learn survival skills in a controlled environment instead waiting until I need them and then whipping out the ChatGPT enabled phone for help... Also, I don't need to have an entire library of knowledge with me, when a simple book of survival skills would be more than adequate... I think pretty much every military in the world agrees with this.
• CSSer 3y ago
  There is a model that was just released called falcon-40B that is available for commercial user. It outperforms every other open LLM model available today. Buyer beware, however, because the license is custom[1] and has restrictions for "attributable revenues" over $1M/year. I'll leave that for you to interpret as you will. [0]: https://huggingface.co/tiiuae/falcon-40b-instruct https://huggingface.co/tiiuae/falcon-40b-instruct [1]: https://huggingface.co/tiiuae/falcon-40b-instruct/blob/main/LICENSE.txt https://huggingface.co/tiiuae/falcon-40b-instruct/blob/main/... EDIT: I just realized you seem to be asking for a fully realized, turn-key commercial solution. Yeah, refer to others who say there's no alternative. It's true. Something like this gives you a lot more power and flexibility, but at the cost of a lot more work building the solution as you try to apply it.
  • yyyk 3y ago
    >Buyer beware, however, because the license is custom and has restrictions for "attributable revenues" over $1M/year. It's more than that, it requires permission in advance and royalty payments (art. 8). See also attribution requirements in art. 5. Arguably, any license could be waived - it should be possible to write to Meta for permission as well, so it's not in a much better state than Llama itself for commercial use.
  • andris9 3y ago
    They changed the license for Falcon 40B today (a day later) to Apache 2.0 https://www.tii.ae/news/uaes-falcon-40b-now-royalty-free https://www.tii.ae/news/uaes-falcon-40b-now-royalty-free
• born-jre 3y ago
  There is so much parallel progress happening left and right at the same time they are not there yet. When things like sparseGPT and models fine-tuned with data with tool ability (not just instruct data) may be soon we get there, as long as there is progress i am hopeful. Some sort of inference optimized hardware would also help.
• nabakin 3y ago
  I would monitor and research each of these top models to determine which best fits your use case. https://lmsys.org/blog/2023-05-25-leaderboard/ https://lmsys.org/blog/2023-05-25-leaderboard/ https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... https://assets-global.website-files.com/61fd4eb76a8d78bc0676b47d/64547b623e779885728099ec_image5.png https://assets-global.website-files.com/61fd4eb76a8d78bc0676... https://www.mosaicml.com/blog/mpt-7b https://www.mosaicml.com/blog/mpt-7b Also keep up to date with r/LocalLLaMA where new best open models are posted all the time.
• Saruto 3y ago
  Falcon 40B
• weystrom 3y ago
  https://github.com/oobabooga/text-generation-webui/ https://github.com/oobabooga/text-generation-webui/ Works on all platforms, but runs much better on Linux. Running this in Docker on my 2080Ti, can barely fit 13B-4bit models into 11G of VRAM, but it works fine, produces around 10-15 tokens/second most of the time. It also has an API, that you can use with something like LangChain. Supports multiple ways to run the models, purely with CUDA (I think AMD support is coming too) or on CPU with llama.cpp (also possible to offload part of the model to GPU VRAM, but the performance is still nowhere near CUDA). Don't expect open-source models to perform as well as ChatGPT though, they're still pretty limited in comparison. Good place to get the models is TheBloke's page - https://huggingface.co/TheBloke https://huggingface.co/TheBloke. Tom converts popular LLM builds into multiple formats that you can use with textgen and he's a pillar of local LLM community. I'm still learning how to fine-tune/train LoRAs, it's pretty finicky, but promising, I'd like to be able to feed personal data into the model and have it reliably answer questions. In my opinion, these developments are way more exciting than whatever OpenAI is doing. No way I'm pushing my chatlogs into some corp datacenter, but running locally and storing checkpoints safely would achieve my end-goal of having it "impersonate" myself on the web.
• FieryTransition 3y ago
  You can fine tune a open source model for your task and achieve better results, at least, instead of just using them directly. But they are still not close to the openai models in generality. Huggingface is the place for exploring models, recently went through a lot of them for my use case, and they are simply not good enough, yet.
• ijk 3y ago
  "Okay with paying for it" gives you a wide range of options. Most of the open source stuff people are talking about is things like running a quantized 33B parameter LLaMA model on a 3090. That can be done on consumer hardware, but isn't quite as good at general purpose queries as GPT-4. Depending on your use case and your ability to fine tune it, that might be sufficient for a number of applications. Partcularly if you've got a very specific task. However, if you're willing to spend, there are bigger models available (e.g. Falcon 40B, LLaMA 65B) that can be run on data server class machines, if you're willing to spend $15-20K. Will that get you GPT-4 level inference? Probably not (though it is difficult to quantify); will it get you a high-quality model that can be further fine-tuned on your own data? Yes. For the smaller models, the fine-tunes for various tasks can be fairly effective; in a few more weeks I expect that they'll have continued to improve significantly. There's new capabilities being added every week. The biggest weakness that's been highlighted in research is that the open source models aren't as good at the wide range of tasks that OpenAI's RLHF has covered; that's partly a data issue and partly a training issue.
• 0xferruccio 3y ago
  Buy a tinybox from tiny corp https://tinygrad.org/ https://tinygrad.org/
• cl42 3y ago
  Have you tried using GPT-4 via Azure? My understanding is that it's faster and more reliable.
• 0xbadc0de5 3y ago
  I'll +1 the votes for Guanaco and Vicuna running with the Oobabooga text-generation-webui. With a 4090, you can get ChatGPT 3.5 level results from Guanaco 33B. Vicuna 13B is a solid performer on more resource-constrained systems. I'd urge the naysayers who tried the OPT and LLaMA models only to give up to note that the the LLM field is moving very quickly - the current set of models are already vastly superior to the LLaMA models from just two months ago. And there is no sign the progress is slowing - in fact, it seems to be accelerating.
• lee101 3y ago
  [dead]
• lee101 3y ago
  [dead]
• f0e4c2f7 3y ago
  Nothing open source is quite as good as GPT-4 yet but the community continues to edge closer. For general use Falcon seems to be the current best: https://huggingface.co/tiiuae https://huggingface.co/tiiuae For code specifically Replit's model seems to be the best: https://huggingface.co/replit/replit-code-v1-3b https://huggingface.co/replit/replit-code-v1-3b
• Ambix 3y ago
  [dead]
• catwalk_moto 3y ago
  [dead]
  • dr_kiszonka 3y ago
    Have you come across any resource that lists hardware requirements for open source models (GPU/VRAM, CPU, RAM)?
  • snet0 3y ago
    > I lay out the whole LLM landscape in this article: No, you don't.
    • ryanklee 3y ago
      Agreed. Information sparse, lays out almost nothing. One would do better by far to just go to /r/LocalLlama and sort all by Top.
  • londons_explore 3y ago
    Worth noting that Llama and derivatives most likely can be used for commercial use, despite what the licensing terms say. There seems to be growing consensus that model weights are not copyrightable, and therefore adhering to the terms of the license isn't required if you downloaded them from a third party. Obviously being legally right doesn't necessarily save you from having Meta's legal team breathing down your neck...
    • jomohke 3y ago
      So if I embed a custom "special sauce" model in my phone app, to do inference faster on the device, my bigger competitors could simply extract it from the bundle and use it too? Perhaps that's okay, I don't know, but it seems strange that the little pieces of client-side javascript are so copyrightable yet this other work isn't.
• MagicMoonlight 3y ago
  OpenAssistant is pretty good. It still has some censorship but nowhere near the levels of commercial models. It’s actually impressive how good it is considering the limited resources they have.
• amilios 3y ago
  How much GPU memory do you have access to? If you can run it, Guanaco-65B is probably as close as you can get in terms of something publicly available. https://github.com/artidoro/qlora https://github.com/artidoro/qlora. But as other comments mention, it's still noticeably worse in my experience.
  • Taek 3y ago
    Another vote from me for guanaco 65b. Here's a link to the model and weights: https://huggingface.co/TheBloke/guanaco-65B-GPTQ https://huggingface.co/TheBloke/guanaco-65B-GPTQ I use it all the time at home. It's decent at things like summaries and writing content. It follows instructions as well as GPT and isn't nearly as pretentious but it's still a bit pretentious. It's not very good at code.
  • npteljes 3y ago
    I was wondering how much GPU memory does G-65B need, and from the docs, it's "48GB".
    • amilios 3y ago
      Is this with the 4-bit quantization? The only issue with it is that inference is incredibly slow with it on right now, but that should be fixed up in the next few weeks I think.
    • netsec_burn 3y ago
      I'm running it on Serge with 51 GB of RAM. I don't think it requires any GPU memory (llama.cpp runs on the CPU), I have a RTX 2060 in the system I'm running it on. Correct me if I'm wrong.
      • forgingahead 3y ago
        How many tokens per second are you getting with your setup?
        • 3y ago
• cypress66 3y ago
  Nothing self hosted is even remotely close to gpt 3.5, let alone gpt4. Wizardlm-uncensored-30B is fun to play with.
  • revskill 3y ago
    How much fun ? Do you have any guidance for me to follow ?
• vs4vijay 3y ago
  You can find more details here - https://old.reddit.com/r/LocalGPT/ https://old.reddit.com/r/LocalGPT/
• Y_Y 3y ago
  You could hire a human to manually respond to the queries
  • samstave 3y ago
    Thank you for calling Movie Phone! Why dont you just tell me which movie you want to see! https://www.youtube.com/watch?v=qM79_itR0Nc https://www.youtube.com/watch?v=qM79_itR0Nc
  • catwalk_moto 3y ago
    [flagged]
  • ceejayoz 3y ago
    Confidently stated responses that are often incorrect or miss the point? Just post it on /r/AskReddit and wait a little bit.
• simonw 3y ago
  The answer to this question changes every week. For compatibility with the OpenAI API one project to consider is https://github.com/go-skynet/LocalAI https://github.com/go-skynet/LocalAI None of the open models are close to GPT-4 yet, but some of the LLaMA derivatives feel similar to GPT3.5. Licenses are a big question though: if you want something you can use for commercial purposes your options are much more limited.
  • fnands 3y ago
    Have you had a chance to look at the Direct Preference Optimization [1] pre-print? It seems like it might help get around RLHF which (as far as I can tell) is the really hard/expensive part of training the best of these models. They said they will release the code soon, so I guess we will found out soon enough. [1] https://arxiv.org/abs/2305.18290 https://arxiv.org/abs/2305.18290
• samwillis 3y ago
  If you want/need to go cpu only then llama.cpp, and the assorted front ends people are building for it, is looking like a good project: https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp
  • grimgrin 3y ago
    I'll append this reference too, as it's related and interesting: https://justine.lol/mmap/ https://justine.lol/mmap/
  • segmondy 3y ago
    your statement sounds like it doesn't support GPU, llama supports GPU as well. However llama is just code to run models. OP needs to find a model that works for them, there are lots of models out there.
    • superkuh 3y ago
      IMO, the best out right now is https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored-GGML https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored... (in llama.cpp ggmlv3 format). A 30B llama fine-tuned on a mix of WizardLM and Vicuna training data. It does conversational interactions pretty well (the wizard) and handles instruction fine too (vicuna). But it is definitely no GPT3.5 or GPT4 replacement. It will not be good for getting "work" done or helping do tasks. It's for recreation. If you want a GPT3.5 level LLM, something akin to text-davinci-002, you'll need to do the SFT and RLHF fine tuning of the 65B llama yourself. And that's no small task. Neither is running a 65B model (even at 4 bits).
• ludovicianul 3y ago
  This is a good candidate: https://github.com/imartinez/privateGPT https://github.com/imartinez/privateGPT
• r_thambapillai 3y ago
  [flagged]
  • outime 3y ago
    I'm all for promoting own solutions if it fits the context but OP is asking for self-hosted alternatives and this is not. This fits the definition of spam.
    • r_thambapillai 3y ago
      Tbf, the OP explicitly said "Preferably self-hosted" which is why I wanted to offer this along with the explicit disclaimer that I am the founder
      • pseg134 3y ago
        Just stop, you are spamming on an unrelated post and now arguing about it. I’m sure people will want to sign up for your service now.
  • deleted 3y ago
    [deleted]
• danpalmer 3y ago
  > Preferably self-hosted (I'm okay with paying for it) The big models, if even available, need >100GB of graphics memory to run and would likely take minutes to warm up. The pricing available via OpenAI/GCP/etc is only effective when you can multi-tenant many users. The cost to run one of these systems for private use would be ~$250k per year.
  • segmondy 3y ago
    ... strange. I'm running 30B models on a 10yr old PC with a $400 RTX 3060. Folks can run the 65B models with 4090 or dual 3090. Usually for about a cost of $2500.
    • redox99 3y ago
      How? 24GB VRAM can only handle 30B 4bit. Are you offloading some layers to the CPU? That kills performance AFAIK.
    • danpalmer 3y ago
      GPT-3.5 is up to 175B parameters, GPT-4 (which is what OP is asking for) has been speculated as having 1T parameters, although that seems a little high to me. It's easy to run a much worse model on much worse hardware, but there's a reason why it's only companies with huge datacenter investments running the top models.
      • ukuina 3y ago
        The 1T number is wild hype. It's likely GPT4 is actually a mixture-of-experts with smaller models under the hood.
    • humanistbot 3y ago
      30B isn't big anymore. GPT-4 is rumored to have 1T parameters.
    • jbellis 3y ago
      Don't 4090 and 3090 have the same amount of vram?
• AndroTux 3y ago
  https://gpt4all.io/ https://gpt4all.io/ works fairly well on my 16 GB M1 Pro MacBook. It's certainly not on a level with ChatGPT, but what is? It's a simple app download and allows you to select from multiple available models. No hacking required.
  • MacsHeadroom 3y ago
    > It's certainly not on a level with ChatGPT, but what is? Guanaco-65B is per https://arxiv.org/abs/2305.14314 https://arxiv.org/abs/2305.14314 CPU Version: https://huggingface.co/TheBloke/guanaco-65B-GGML https://huggingface.co/TheBloke/guanaco-65B-GGML GPU Version: https://huggingface.co/TheBloke/guanaco-65B-HF https://huggingface.co/TheBloke/guanaco-65B-HF 4bit GPU Version: https://huggingface.co/TheBloke/guanaco-65B-GPTQ https://huggingface.co/TheBloke/guanaco-65B-GPTQ
    • VadimPR 3y ago
      How much ram/vram required?
      • MacsHeadroom 3y ago
        At least 48GB of RAM to run with llama.cpp, which now has CUDA GPU Offloading support. The more you can fit on your GPU (in VRAM) the better (for speed), but no GPU is strictly required.
    • balu548411 3y ago
      how can we load this 4bit GPU Version: https://huggingface.co/TheBloke/guanaco-65B-GPTQas https://huggingface.co/TheBloke/guanaco-65B-GPTQas that is in safetensors format
  • Solvency 3y ago
    Why hasn't the community created a distributed Folding@home style GPT4/LLM software, where anyone can ask a question, and all participants machine's are contributing to the computation of the answer? It seems like the ideal way for the open source community. Is it not possible for some reason with the way LLMs work and compute? Or has it simply just not been done yet?
• daryl149 3y ago
  [dead]
• Veen 3y ago
  It depends what you mean by "viable alternatives" and how much money you are prepared to spend on hardware to self-host. As others have mentioned, you can try llama.cpp and LocalAI, but for most ChatGPT-like applications, you won't get anything like as good results. I've found that using GPT-4 via the OpenAI API is somewhat more reliable than ChatGPT, either via the Playground or via a local chat interface like https://github.com/mckaywrigley/chatbot-ui https://github.com/mckaywrigley/chatbot-ui
• Gijs4g 3y ago
  > Preferably self-hosted (I'm okay with paying for it) I'm the founder of Mirage Studio and we created https://www.mirage-studio.io/private_chatgpt https://www.mirage-studio.io/private_chatgpt. A privacy-first ChatGPT alternative that can be hosted on-premise or on a leading EU cloud provider.
  • daryl149 3y ago
    Haha, beat me to it by a minute! But idd, do check out our self hosted option :D
• jonathan-adly 3y ago
  I don't know the licensing and all that jazz (even if you self-host for your personal use it shouldn't matter). But, this paper[0] released a week ago claims " 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" (QLORA). A quick test of the huggingface demo gives reasonable results[1]. The actual model behind the space is here[2], and should be self-hostable with reasonable effort. 0. https://arxiv.org/abs/2305.14314 https://arxiv.org/abs/2305.14314 1. https://huggingface.co/spaces/uwnlp/guanaco-playground-tgi https://huggingface.co/spaces/uwnlp/guanaco-playground-tgi 2. https://huggingface.co/timdettmers/guanaco-33b-merged https://huggingface.co/timdettmers/guanaco-33b-merged
  • MacsHeadroom 3y ago
    You link to Guanaco-33B, but Guanaco-65B is much more capable. CPU Version: https://huggingface.co/TheBloke/guanaco-65B-GGML https://huggingface.co/TheBloke/guanaco-65B-GGML GPU Version: https://huggingface.co/TheBloke/guanaco-65B-HF https://huggingface.co/TheBloke/guanaco-65B-HF 4bit GPU Version: https://huggingface.co/TheBloke/guanaco-65B-GPTQ https://huggingface.co/TheBloke/guanaco-65B-GPTQ
    • wing-_-nuts 3y ago
      It irritates me to no end that people don't list the system requirements of various models. How much ram and vram does one need to run 4,13,33,65B models at a reasonable speed? edit: instead I'll ask this, what's the best model to run on a system with a 24gb 4090 and 64gb of ram?
      • orost 3y ago
        You can just barely fit a 33B GPTQ model in 24GB VRAM. It will be in 4-bit mode, and without maximum context size, but it will be quite fast. Or you can run from RAM+VRAM in GGML format with llama.cpp (or a derivative), which will easily fit 65B models even at 5 or 8 bits, but at much lower speed.
      • evanchisholm 3y ago
• wokwokwok 3y ago
  There is literally no alternative. You’re stuck with openai, and you’re stuck with whatever rules, limitations or changes they give you. There are other models, but specifically if you’re actively using gpt-4 and find gpt-3.5 to be below the quality you require… Too bad. You’re out of luck. Wait for better open source models or wait patiently for someone to release a meaningful competitor, or wait for openai to release a better version. That’s it. Right now, there’s no one else letting people have access to their models which are equivalent to gpt-4.
  • bigmattystyles 3y ago
    The model is the spice then basically?
    • y-curious 3y ago
      Fitting analogy. The Kenyan data transcribers that OpenAI used are the Fremen, and transcripts of erotic Harry Potter fanfiction are the Shai-Hulud.
      • deleted 3y ago
        [deleted]
      • nirav72 3y ago
        Are the kenyan's really going to rise up like the Freman? Most likely not. If anything, they'll be used for cheap labor when the next new thing comes along.
    • ubermon 3y ago
      haha was thinking the same. The other day openAI's api hiccup gave me a small panic attack
  • yankoff 3y ago
    Is this opinion based on some benchmarking you (or someone else) did?
• anon291 3y ago
  I admittedly haven't used GPT-4 yet, but I've replaced several uses of GPT-3 with RWKV on the Raven dataset. I can load it onto my RTX 2060 with 12GB of mem (quantized of course), and use it to whittle down or summarize data for GPT.