5 ms·
https://gpt4all.io/ https://gpt4all.io/ works fairly well on my 16 GB M1 Pro MacBook. It's certainly not on a level with ChatGPT, but what is? It's a simple ap
by AndroTux 3y ago
https://gpt4all.io/ https://gpt4all.io/ works fairly well on my 16 GB M1 Pro MacBook. It's certainly not on a level with ChatGPT, but what is?
It's a simple app download and allows you to select from multiple available models. No hacking required.
- MacsHeadroom 3y ago> It's certainly not on a level with ChatGPT, but what is? Guanaco-65B is per https://arxiv.org/abs/2305.14314 https://arxiv.org/abs/2305.14314 CPU Version: https://huggingface.co/TheBloke/guanaco-65B-GGML https://huggingface.co/TheBloke/guanaco-65B-GGML GPU Version: https://huggingface.co/TheBloke/guanaco-65B-HF https://huggingface.co/TheBloke/guanaco-65B-HF 4bit GPU Version: https://huggingface.co/TheBloke/guanaco-65B-GPTQ https://huggingface.co/TheBloke/guanaco-65B-GPTQ
- VadimPR 3y agoHow much ram/vram required?
- MacsHeadroom 3y agoAt least 48GB of RAM to run with llama.cpp, which now has CUDA GPU Offloading support. The more you can fit on your GPU (in VRAM) the better (for speed), but no GPU is strictly required.
- balu548411 3y agohow can we load this 4bit GPU Version: https://huggingface.co/TheBloke/guanaco-65B-GPTQas https://huggingface.co/TheBloke/guanaco-65B-GPTQas that is in safetensors format
- Solvency 3y agoWhy hasn't the community created a distributed Folding@home style GPT4/LLM software, where anyone can ask a question, and all participants machine's are contributing to the computation of the answer? It seems like the ideal way for the open source community. Is it not possible for some reason with the way LLMs work and compute? Or has it simply just not been done yet?
- simonw 3y agoThat exists: https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals
- brookst 3y agoThere's a bunch of stuff on this area. See https://www.fedai.org/ https://www.fedai.org/ for one. I also remember something more directly analogous to @home, but I'm having a hard time finding it. Edit: Petals! Thank you sibling commenter!
- maffydub 3y agoInteresting thought! I think the problem is that it's hard to parallelize this effectively across a network with Internet-scale latency - individual matrix multiplications parallelize very well, but there would need to be coordination of each result, which would be much slower than just doing the computation locally. In the case of Folding@home, evaluating possible folds could be done completely in parallel and only need to be coordinated on discovery of a plausible fold (which is rare), so distributing over a high-latency network is still beneficial.
- shagie 3y agoDistributing very large matrix multiplications doesn't scale across networks well. While this is a MINST classifier (and can be run in the browser) you can get an idea of the math behind it. https://www.3blue1brown.com/lessons/gradient-descent https://www.3blue1brown.com/lessons/gradient-descent and https://www.3blue1brown.com/lessons/neural-network-analysis https://www.3blue1brown.com/lessons/neural-network-analysis When dealing with a LLM, it's being run again and again - token by token. Pick the best next token, append it to the input, run it again. If you want to generate 100 tokens (rather small amount of data when compared to much of the GPT-4 conversations), that means running it 100 times. The network traffic makes the entire system much slower than running it all in one spot. Consider also the "are you the input to everyone?" and the privacy implications of that.
- sounds 3y agoThe model can run across multiple machines but it's so slow that way that nobody bothered to ship that.