MacsHeadroom

40 ms·

601.
by MacsHeadroom 4y ago
They can and they have been look on HuggingFace or search torrent sites. You only actually need about 30GB of VRAM (or unified memory) and no ram to run the largest 65B model.
602.
by MacsHeadroom 4y ago
A 4090 gets 30 tokens/second with LLaMA-30B, which is about 10 times faster than the 300ms/token people are reporting in these comments. (20 tokens/second on a Mac is for the smallest model, ~5x smaller than 30B and 10x small
603.
by MacsHeadroom 4y ago
Strongly disagree. Code is law and failure is personal are core and necessary parts of crypto. Bring on the downvotes for not sharing your opinion.
604.
by MacsHeadroom 4y ago
Yes, text-generation-webui has multiGPU support regardless of your GPU(s). They do not even need to be the same GPU. You can have an AMD 6900xt combined with an RTX 4090, combined with an RTX 3050, if you want. It makes no difference.
605.
by MacsHeadroom 4y ago
>which nothing except an A100 can match LLaMA's GPT-3 175B level model, LLaMA-13B, only requires 8GB of VRAM (and no RAM) to run using pre-quantized 4bit weights. So it's hardly a job for an A100. Even the largest model, LLaMA-
606.
by MacsHeadroom 4y ago
Not per bit or per watt. LLaMA-30B is 16GB and draws 40 watts from a 4090 GPU.
607.
by MacsHeadroom 4y ago
LLaMA-13B 4bit is around 7GB and beats GPT-3.5 175B in benchmarks. It runs at acceptable speeds on a Macbook Air M1 or a $100 consumer video card.
608.
by MacsHeadroom 4y ago
It would cost less than $50 of compute to finetune an existing open source LLM to do the job for another $50 of computer. OpenAI's APIs are incredibly expensive compared to powerful open source LLMs which run on very accessible hardwa
609.
by MacsHeadroom 4y ago
Cards I have seen LLaMA run on in 8bit and 4bit include: RTX 1660, RTX 2060, AMD 5700xt, RTX 3050, RTX 3060, AMD 6900xt, RTX 2060 12GB, RTX 3060 12GB, RTX 3080, P5000, RTX A2000, RTX 3080 20GB, RTX A4500, RTX A5000, RTX 3090, RTX 4090, RTX
610.
by MacsHeadroom 4y ago
Massively diverse community working on the AUTOMATIC1111 of textgen at https://github.com/oobabooga/text-generation-webui/ Ooba's textgen webui runs LLaMA 4bit on 8GB VRAM consumer video cards. KoboldAI ( htt
611.
by MacsHeadroom 4y ago
13B in 4bit runs on your average 8GB VRAM consumer video card at fast enough speeds. The GP's info is outdated. Apparently they don't realize how quickly LLaMA is being developed on. It's just like the early days of SD.
612.
by MacsHeadroom 4y ago
The point of it is to generate arbitrary text locally, like the GPT-3 API does remotely. It can "say"/talk about anything an average IQ person with knowledge of the entire internet and most books in existence could. So if you
613.
by MacsHeadroom 4y ago
Yes, but only minimally. Not enough for any human to notice. However, even this minimal amount can be avoided with GPTQ quantization which maintains uncompressed fp16 performance even at 4bit quantization with 75% less (video)memory overhea
614.
by MacsHeadroom 4y ago
>I suppose that compilers can figure out how to do this in the most efficient way on any type of hardware for which a backend was written. No, that's exactly the problem. Compilers can't because the GPU hardware and the algorit
615.
by MacsHeadroom 4y ago
People have been running LLaMA in 4bit quickly on cheap hardware with a simple GUI for over a week using https://github.com/oobabooga/text-generation-webui Just not on Macs. (that repo does not support Apple Silicon)
616.
by MacsHeadroom 4y ago
The 65B model only needs just over 32GB of VRAM to run. It does not need system RAM to run/use if you use pre-quantized weights which you can find many places already. No need to quantize yourself (besides it takes almost a day to do 4
617.
by MacsHeadroom 4y ago
Judging from downloads of the 4bit file and how many people I've seen post about quantizing it themselves, around 99% of people are just downloading the pre-quantized files. I would not personally call compilation of software part of i
618.
by MacsHeadroom 4y ago
It's about 4 charcters per token. So just over 1 token per word. I just round to 1 token per word since text most people generate does not use larger words and because larger common words are still encoded as one token (e.g. HackerNews
619.
by MacsHeadroom 4y ago
You need two 3090s or 4090s to fit 65B even in 4bit. It's a big one. That said, if you're fine with slower speeds then two P40s could get the job done for $150 each. (Not sure how much slower this would go though.)
620.
by MacsHeadroom 4y ago
It's currently 2048 tokens. But like GPT-3, that can be improved in the future.
621.
by MacsHeadroom 4y ago
It's about as fast as chatGPT when chatGPT first launched. Not as fast as the new "Turbo" version of chatGPT, but much faster than you or anyone can read (so I'm not sure the difference matters).
622.
by MacsHeadroom 4y ago
LLaMA it doesn't require any system RAM to run. It requires some very minimal system RAM to load the model into VRAM and to compile the 4bit quantized weights. But if you use pre-quantized weights (get them from HuggingFace or a friend
623.
by MacsHeadroom 4y ago
You're half right. First, tensor operations are only a small part of modern ML. Second, how you plug all those small operation together is where all the performance difference is had these days between implementations. Different hardwa
624.
by MacsHeadroom 4y ago
* emergency-contraceptives..org * protest-overreach..org
625.
by MacsHeadroom 4y ago
HackerNews is a service by and for the venture capital company Ycombinator. 30% of Ycombinator backed startups use SVB bank, most of them to run their payroll. So no we can't move on. The livelihoods of the people who this site was mad
626.
by MacsHeadroom 4y ago
Yes but it wasn't made to run on a Mac. This project ported LLaMA to Apple Silicon so all the macbook users can finally play with what the rest of us have had access to for the past couple of weeks.
627.
by MacsHeadroom 4y ago
Because that's not how machine learning models work. Machine learning as a field goes through a nearly complete revolution annually. Every new major model is a special snowflake of unique cases. Writing high performance software that h
628.
by MacsHeadroom 4y ago
I get 32 tokens/sec on a 4090 using GPTQ 4bit with streaming off, with the model 5x larger than that. So nowhere close to the 4090, but plenty fast anyway.
629.
by MacsHeadroom 4y ago
Close, fine tuning LLaMA in 8bit has been done. It requires very little VRAM compared to typical fine tuning. https://github.com/zphang/minimal-llama#peft-fine-tuning-wit...
630.
by MacsHeadroom 4y ago
LLaMA-65B fits in 32GB of VRAM using state of the art GPTQ quantization with no output performance loss. https://github.com/qwopqwop200/GPTQ-for-LLaMa

More ›