28 ms·
From my experience with llama.cpp and oobaboogas webui I can say they are amazing, at least on my gaming pc. I’m absolutely astonished at the speed and quality
by kbrkbr 4y ago
From my experience with llama.cpp and oobaboogas webui I can say they are amazing, at least on my gaming pc. I’m absolutely astonished at the speed and quality of llama, alpaca, galactica and vicuna (the >10B parameters ones).
Make no mistake, it’s for tinkerers that do not expect each prompt to be answered human like.
I see them as creativity and thought testing tools, also knowledge exploratory.
- deleted 4y ago[deleted]
- synergy20 4y agoare we talking about training or inference for local LLM here? it's hard to do any meaningful training on the edge unless we all carry a heavy gaming pc, even that, the training quality will be subpar?
- qumpis 4y agoInference, even fine-tuning a few layers would be difficult since one needs to use non-quantized model, I'd imagine
- ctoth 4y agoCheckout LoRA and Alpaca LoRA and the whole huge group of people who have already figured this out. I think there was another breakthrough (yesterday?) which is a further adaption of LoRA to touch even less parameters at runtime.
- homarp 4y agoDyLoRA https://news.ycombinator.com/item?id=35514228 https://news.ycombinator.com/item?id=35514228 is what you meant
- whimsicalism 4y agoDisagree that this is a major breakthrough :) but it is probably a marginal improvement.
- whimsicalism 4y agoThey're not training on edge :)
- rubidium 4y agoIs there a "getting started" guide you'd recommend to a newbie to the space?
- kbrkbr 4y agoI started with ggerganov’s llama.cpp GitHub repo, and went from there. But then again, I know some programming, statistics and machine learning, so it may not be for you, I cannot judge that. Models can be found on huggingface.co, and I’d start with eachadea/ggml-vicuna-13b-4bit, but it needs 10G of cpu-ram. It is very friendly to any prompt though. I read on my way (on reddit, when I recall correctly), that there must be some really good intro videos on YouTube.
- boppo1 4y agoadditional point of reference I don't know shit about ML or stats, and am a very weak programmer. I just know how to install programs on linux and use CLI in a basic fashion. I have had no problem getting llama.cpp going by following ggerganov's readme.
- wing-_-nuts 4y agois there a difference in the quality of llm one would be able to train or run on a gpu with 8, 12, 16, all the way up to 24gb? I'm trying to decide whether it's worth while to splurge on a more expensive 4090 vs a 4070 or whatever.
- Garrrrrr 4y agollama.cpp runs on your processor and uses a lot of RAM, so splurging on a GPU isn't going to help the performance
- turmeric_root 4y agoMore VRAM => larger models. IME it is absolutely worth maxing out VRAM for the significant improvement in quality, especially with LLaMA (though even with a 4090, you won't be able to run the largest 65-billion parameter model even with 4-bit quantization). That said, I recommend renting a cloud GPU for a few hours and trying the larger models on them before buying a GPU of your own, just to see if the models meet your requirements.
- sliken 4y agoBut should fit easily on a Apple MBP or Studio with 96GB or 128GB of unified memory.
- whimsicalism 4y agoIt makes 0 economical sense to buy a GPU to train a model. If you want to train a model, train it on the cloud.
- anonzzzies 4y agoCan you calculate that for me on a napkin? Every calculation I make for training, but certainly for inference, makes me break even after well under a year and then it’s vastly less if I buy the hardware myself.
- 4y ago
- whimsicalism 4y agoIf they are amazing, then ChatGPT must be God-like in your view. I've been underwhelmed by Alpaca and Alpaca-LORA and LLaMA all at 13B but I have not tried higher params.
- kbrkbr 4y agoIn my opinion the problem with these is engineering a good prompt. I read of lots of people only getting nonsense or repetitions, and learned a bit from what they shared. These models are not chat bots. Vicuna is more friendly in that regard. But I’m well aware of their limitations also, and I can see how one can be underwhelmed. They are not jacks of all trades
- whimsicalism 4y agoAlpaca & Alpaca-LoRA are literally trained to be chat bots. Their goal is to be instruct-tuned.
- MacsHeadroom 4y agoAlpaca-LoRA, and all LoRas, are garbage. Alpaca is horrible compared to newer finetunes. Even cleaning the Alpaca dataset and retraining a cleaned Alpaca improves its performance greatly. But newer finetunes like Vicuna go well beyond that, including hundreds of thousands of real human conversations with GPT-4 ChatGPT in the dataset (unlike Alpaca's fully synthetic dataset). Vicuna-13B in 16bit is easily comparable to ChatGPT-3.5 in capability. Newer finetunes coming out nearly every day are going beyond chatGPT-3.5 and getting closer and closer to GPT-4 performance. You don't even have to install anything to validate this for yourself. There's a live web demo of Vicuna-13B right here: https://chat.lmsys.org/ https://chat.lmsys.org/ (disable ad blocker if it does not load)
- MacsHeadroom 4y agoAlso, InstructGPT was released in January 2021 and Vicuna-13B blows it out of the water.