5 ms·

From all the experimentation I've done, 7B parameter models just don't seem to be able to produce useful output reliably enough for my use cases. What use case

• jimmyl02 3y ago
  The main use case is that it's probably the only size consumers can run on their personal devices. If you don't want your data going into an external platform like OpenAI it's the only solution even if it's not very usuable.
  • jahewson 3y ago
    These are constraints but not a use case.
    • verwkljdslfklj 3y ago
      Isn't it obvious they're referring to the use case of using a model given those constraints?
      • roughly 3y ago
        Yes, but I think the responder is wondering if there are useable use cases for that - like, what can you actually Do with that model. I’m in the same boat - I don’t want to ship my data to openai, I do want to run local, so I’d love to hear what other folks are Doing with models of that size.
      • sidibe 3y ago
        "Use case of using" :)
  • m00x 3y ago
    You can run big models on the cloud yourself, or with a 3090/4090 quantized. You don't have to go to openai.
    • aryamaan 3y ago
      What’re some models and hardware combos we can run now? I am avoiding to go to OpenAI with my office’s stuff and can use some gpu(s)
      • MINIMAN10000 3y ago
        You would just need a computer which can fit 2 3090s in order to run those to run something like TheBloke/airoboros-65B-gpt4-1.3-GPTQ https://www.reddit.com/r/LocalLLaMA/wiki/models/ https://www.reddit.com/r/LocalLLaMA/wiki/models/ gives you a list of VRAM requirements to load the model into GPU VRAM. the more VRAM the computer has, the larger the model you can load in, thus making 3090s the current consumer grade king due to price to max VRAM. This being said however most models are LLAMA based which all fall under that specific research license. So following the rules, you would be limited to a subset of models which are foundational models which allow for commercial use
      • brucethemoose2 3y ago
        I can easily run LLaMA 13B on my 6GB VRAM/16GB RAM laptop using llama.cpp (specifically Kobold.cpp as the frontend). I can barely run 33B, but anything more than 800 context and I oom. But it would run very comfortably on a bigger GPU or a 24GB+ laptop. Theoretically some phones can comfortably handle 13B on mlc-llm though in practice its not really implemented yet.
      • int_19h 3y ago
        llama-30B (which is actually 33B) and derivatives generally run fine with 4-bit quantization on a single RTX 3090 or 4090, although depending on group size used for quantization you may need to slightly dial down the context size.
• jrflowers 3y ago
  > What use cases do people have for these smaller LLM's? None. Training a functionally useless model and releasing it is a great way to demonstrate that your company is hip and current. That way when prospective clients ask about AI you can vaguely gesture at some model that you released and say you employ cutting edge AI experts.
  • wokwokwok 3y ago
    yep, that’s why it’s free. If it was good, then they’d charge for it. …what they (and everyone) is gonna do is play with smallish models to iterate on the process for relatively small expense and earn karma. Then pay big $$$ to make a really good model for internal use and/or an api that people have to pay for. Tldr; it’s free. It’s by sales force. You should expect it to be a) crippled and b) a loss leader for a paid product. Not judging; it’s a fair strategy. Just saying: salesforce is not a company that just gives hundreds of thousands of dollars away for nothing. If you want a good free open model, you’re kidding yourself if you think a corporate giant is going to kiss you on the head and give it to your for free.
    • Auracle 3y ago
      To be fair, Stable Diffusion (especially the upcoming SDXL) are good and free.
    • jrflowers 3y ago
      > If you want a good free open model, you’re kidding yourself if you think a corporate giant is going to kiss you on the head and give it to your for free. Yep! That makes sense! I would love to be the CEO of the company that does give away an actually useful model and little forehead kisses though. The amount of goodwill that one would generate from that would be astronomical and training costs are getting so low that nearly any company with enough cash could do it. I look forward to waking up and hearing that the Nabisco/Canadian Tire/A&W usefully-tuned model is revolutionizing the economy and seeing the infinite amount of good press that it would generate.
      • 3y ago
• riku_iki 3y ago
  Bert had lots of use cases and this one is supposedly stronger model.
  • TOMDM 3y ago
    I've used BERT in a number of production apps, it feels like a very apples to oranges comparison given how the AI landscape has changed since BERT's release.
    • riku_iki 3y ago
      Then looks like you know already where smaller and weaker models are useful.
    • nextaccountic 3y ago
      What did you use BERT for? Maybe 7B models are up to the task?
• reissbaker 3y ago
  They're only modestly worse than text-davinci-003 in my experience, and you can finetune them cheaply to do simple tasks e.g. triage human input and decide where to send it. But yeah, if you can afford to run a larger model or pay OpenAI for gpt4 calls, that's gonna work a lot better. If you're relying on prompting for the 7B models IMO you're gonna have a bad time — they're mostly toys at that: interesting output but not consistently useful. But finetuning gets better results, and it's cheap to finetune.
• brucethemoose2 3y ago
  7B LLaMA is a terrible general purpose model, but the finetunes are pretty good at very specific roles, like dialogue/roleplay, a dungeon master bot or even code completion. The metrics are good though, perhaps placing this closer to 13B. And 8K context is huge. When you can stuff that much example text in, it gives the model more to "latch onto," and its also the point where you would start worrying about RAM/VRAM consumption for a ~13B model.
  • aryamaan 3y ago
    would appreciate resources on fine tuning
    • brucethemoose2 3y ago
      Honestly, I dunno. I think most people are using lit-llama or EasyLM (on TPUs) for finetuning? QLORA is the gold standard for more affordable training. As for datasets, just look at the open datasets the best-in-class models are using, like Vicuna or https://huggingface.co/NousResearch/Nous-Hermes-13b https://huggingface.co/NousResearch/Nous-Hermes-13b Some model datasets like Manticore, Chronos or the infamous Pygmalion are more "secretive," but you can find the dataset gathering scripts on Github or in community chats.
    • SparkyMcUnicorn 3y ago
      This and/or text-generation-webui training doc are a good place to start. https://github.com/zetavg/LLaMA-LoRA-Tuner https://github.com/zetavg/LLaMA-LoRA-Tuner
    • ttt3ts 3y ago
      https://huggingface.co/blog/stackllama https://huggingface.co/blog/stackllama You can easily finetune 7B or 15B LORA model with that on consumer GPUs.
      • minimaxir 3y ago
        That blog post demonstrates that it's not "easily" finetuneable, just possible to finetune. There's many technical considerations even beyond hardware (dataset formatting, training hyperparameter nuances) that do not make it accessible to the newbie experimenting with LLMs. It's a rabbithole, and unfortunately there's no good shortcuts.
• ftxbro 3y ago
  The use case is people who got some fear of missing out on AI and also they are not capable of seeing the difference between the model outputs themselves. Maybe you will say I'm elitist but it's disturbing to me how many people literally can't tell the difference between gpt 3 and gpt 4 outputs, it's like the "It's the same picture" meme. Maybe that makes me a hipster gpt connoisseur elitist that I can recognize and have opinions about those differences between versions and I hate using the small ones and I think only the very biggest ones are good.
  • ipaddr 3y ago
    It doesn't make you an elitist hipster. It puts you in a category of most people on earth. An elist hipster would swear by a custom smaller parameter model finetuned by a wizard for specific tasks and have a spell book of models.
    • galaxyLogic 3y ago
      Right, that's the Way of the Hipster
• steppi 3y ago
  I did some old school NLP before, but don’t really work in the field anymore. As a generative model. As a general purpose generative model, maybe this isn’t very useful. As a foundation to make models to perform text classification and information extraction tasks, this could be very useful. For these kinds of tasks, you can still get good results with the classic bag of words type approaches people were using 30 years ago even. I remember when transformers first came out, limitations in sequence size made them unusable for some classification tasks.
• makestuff 3y ago
  I feel like this would work well as a local LLM for a home assistant type setup. Fully local instead of having Alexa send everything to the cloud.
• rolisz 3y ago
  I've had very good results with even a 3.5 B model (Fastchat-T5) for retrieval augmented generation (aka putting information into the context window and letting the model rephrase it).