16 ms·
Large language models are having their Stable Diffusion moment
- pedrovhb 4y agoOne thing I think will be different and that had totally escaped my radar until recently is just the enormous and diverse community that has been developing around Stable Diffusion, which I think will be less likely to form with language models. I just recently tried out one of the most popular [0] Stable Diffusion WebUIs locally, and I'm positively surprised at how different it is to the rest of the space around ML research/computing. I consider myself to be a competent software engineer, but I still often find it pretty tricky to get e.g. HuggingFace models running and doing what I envision them to do. SpeechT5 for instance is reported to do voice transformations, but it took me a good bit of time and hair-pulling to figure out how to extract voice embeddings from .wav files. I'm sure the way to do this is obvious to most researchers, maybe to the point of feeling like this needs not a mention in the documentation, but it certainly wasn't clear to me. The community around Stable Diffusion is much more inclusive, though. Tools go the extra effort to be easy to use, and documentation for community created models/scripts/tools is so accessible as to be perfectly usable by a non-technical user who is willing to adventure a little bit into the world of hardcore computing by following instructions. Sure, nothing is too polished and you often get the feeling that it's "an ugly thing, but an ugly thing that works", but the point is that it's incredibly accessible. People get to actually use these models to build their stories, fantasy worlds, to work, and things get progressively more impressive as the community builds upon itself (I loved the style of [1] and even effortlessly merged its style with another one in the WebUI, and ControlNet [2] is amazing and gives me ideas for integrating my photography with AI). I think the general interest in creating images is larger than for LLMs with their current limitations (especially in current consumer-available hardware). I do wonder how much this community interest will boost the spaces in the longer run, but right now I can't help but be impressed by the difference in usability and collaborative development between image generative and other types of models. [0] https://github.com/AUTOMATIC1111/stable-diffusion-webui https://github.com/AUTOMATIC1111/stable-diffusion-webui [1] https://civitai.com/models/4998/vivid-watercolors https://civitai.com/models/4998/vivid-watercolors [2] https://github.com/Mikubill/sd-webui-controlnet https://github.com/Mikubill/sd-webui-controlnet
- MacsHeadroom 4y agoMassively diverse community working on the AUTOMATIC1111 of textgen at https://github.com/oobabooga/text-generation-webui/ https://github.com/oobabooga/text-generation-webui/ Ooba's textgen webui runs LLaMA 4bit on 8GB VRAM consumer video cards. KoboldAI (https://github.com/henk717/KoboldAI/ https://github.com/henk717/KoboldAI/) by the creator the AI Horde of stable diffusion fame also supports LLaMA and has forked the Stable Horde for textgen. Currently horde is running half a dozen LLaMA models accessible at http://lite.koboldai.net/ http://lite.koboldai.net/
- kache_ 4y agoDid you know that AUTOMATIC1111 got strapped off of 4chan? Go to 4chan right now, and poke around their technology and video game boards. There's so much chatter about LLaMa. The last time I saw that much chatter about a technology was when eth was 3 dollars a coin. The communities exist, the general public just isn't aware of them.
- homarp 4y agoThere is even has r/LocalLLaMA/
- bilsbie 4y agoHow’s it looking for a six year old MacBook? Not there yet? Does this still use the gpu?
- simonw 4y agoI believe lambda.cpp has been designed for at least an M1 - no idea if there are options for running LLaMA on older hardware.
- astrange 4y agoIt doesn't use CoreML so it should work on Intel machines at some speed. If it used the GPU/ANE and was a true large language model then it would only work on M1 systems because they're unified memory (which nothing except an A100 can match.)
- cromwellian 4y agoI’m wonder how hard it would be to port to OpenXLA which was recently released, that would let you target many CPUs, GPUs, and even TPUs.
- MacsHeadroom 4y ago>which nothing except an A100 can match LLaMA's GPT-3 175B level model, LLaMA-13B, only requires 8GB of VRAM (and no RAM) to run using pre-quantized 4bit weights. So it's hardly a job for an A100. Even the largest model, LLaMA-65B, is only 30GB and inference can be split between two graphics cards with almost no effect on performance.
- pumanoir 4y agoIs there a place to get the prequant'ed model weights?
- Spiwux 4y agoPeople have been running large language models locally for a while now. For now the general consensus is that llama is not fundamentally better than local models with similar resource requirements, and in all the comparisons it falls short of an instruction-tuned model like Chat GPT
- simonw 4y agoMy argument here is that this represents a tipping point. Prior to LLaMA + llama.cpp you could maybe run a large language model locally... if you had the right GPU rig, and if you really knew what you were doing, and were willing to put in a lot of effort to find and figure out how to run a model. My hunch is that the ability to run on a M1/M2 MacBook is going to open this up to a lot more people. (I'm exposing my bias here as a M2 Mac owner.) I think the race is now on to be the first organization to release a good instruction-tuned model that can run on personal hardware.
- stonerri 4y agoAs someone who just got the 7B running on a base MacBook M1/8GB, I strongly agree. The rate of tool development & prompt generation should see the same increase that Stable Diffusion did a few months (weeks?) ago. And given how early the cpp port is, there is likely plenty of performance headroom with more m1/m2-specific optimization.
- Vetch 4y agoIt's not just that it's accessible, it's also significantly higher in quality than previous local runnable causal LMs. I suspect people saying it's not good are prompting it like ChatGPT, not realizing how much trickier a raw model is to prompt. Getting the hyperparameters for good sampling is another stumbling block. The models are very good if you do everything properly.
- reasonabl_human 4y agoInteresting, where can I learn more about prompting, and tuning a raw model?
- rvz 4y agoThere you go and very unsurprising to see that happen very quickly, unless you have a Apple Silicon machine and want to download the model to try it yourself. I still think that open source LLMs models have to be much smaller than 200GB and to be much better than ChatGPT to be more accessible and highly disruptive to OpenAI. It is a great accident needed thanks to Meta. For now one can use it as a service and make it as a SaaS rather than depend fully on OpenAI. Open source (or even free binary only) LLMs will eventually disrupt OpenAI's business plans.
- camjohnson26 4y agoAre there any online communities running these models with non professional hardware? I keep running into issues with poor documentation or outdated scripts with GPT neox, BLOOM, and even stable diffusion 2. Seems like most of the support is either for professionals with clusters of A100s, or consumers who aren’t using code. I have 3 16gb Quadra GPUs but getting this stuff running on them has been surprisingly difficult
- moyix 4y agoThere’s a group of folks on 4chan doing this on gaming class hardware (4080s etc). They have a doc here: https://rentry.org/llama-tard-v2 https://rentry.org/llama-tard-v2
- MacsHeadroom 4y agoCards I have seen LLaMA run on in 8bit and 4bit include: RTX 1660, RTX 2060, AMD 5700xt, RTX 3050, RTX 3060, AMD 6900xt, RTX 2060 12GB, RTX 3060 12GB, RTX 3080, P5000, RTX A2000, RTX 3080 20GB, RTX A4500, RTX A5000, RTX 3090, RTX 4090, RTX 6000, Tesla V100, A100 40GB, A40, RTX A6000, RTX 8000, Titan Ada Mostly using https://github.com/oobabooga/text-generation-webui/ https://github.com/oobabooga/text-generation-webui/, the AUTOMATIC1111 of textgen.
- minimaxir 4y agoCurrently right now there's too many caveats to run even the 7B model per the workflows mentioned in the article. The big difference between it and Stable Diffusion which caused the latter to go megaviral is a) it can run on a typical GPU that gamers likely already have without hitting a perf ceiling and b) it can run easily in a free Colab GPU. Although Hugging Face transformers can run a 7B model on a T4 GPU w/ 8-bit loading, but with its own caveats too. There's a big difference between "can run" and "can run well". VQGAN + CLIP had a lot of friction too and that's partially why AI image generation didn't go megaviral then.
- simonw 4y agoBy caveats do you mean the licensing terms or the difficulty of prompting the model? Unless it's relicensed I don't expect LLaMA to be a long-term foundation model. But it's shown that yes, you can run a GPT-3 class model on an M1 Mac with 8GB of RAM (or maybe 16GB for the 13B one?) I fully expect other models to follow, from other organizations, with better capabilities and more friendly licensing terms.
- zamnos 4y agoBut is anyone actually making money off of StableDiffusion? Maybe the shovel-sellers (runpod.io et al), but afaik no one using it as the foundation for a revenue generating company. I ask, because yes, technically, you can't get LLaMA legally unless you're a researcher and get it directly from Facebook. But that's not going to stop the faithful from finding a copy and working on it.
- logifail 4y ago> is anyone actually making money off of StableDiffusion? We're all still waiting to hear about (non-shovel-selling) successes in this space.
- pmoriarty 4y agoI don't know about Stable Diffusion in particular, but three examples of AI-generated art making money immediately spring to mind: 1 - some guy won hundreds of dollars in an art contest from AI generated art (and this made big news, so it should be easy to find) 2 - one person reported using midjourney's images as a starting point for images that wound up being used in a physical magazine 3 - another artist has used midjourney images that they modify to sell in all sorts of contexts (like background images on stock illustration sites) You'd probably find many other examples in midjourney's #in-the-world discord channel. I'd also be shocked if stock image sites, clipart sites and freelance design/illustration sites weren't already flooded with AI generated images that have been sold for money. That being said, because high questly AI-generated images are so easy to make, the value of images of all types is likely to plummet soon if it hasn't already.
- seydor 4y agoI wonder why we don't have external "neural processing" devices like we once had soundcards. Is anyone working on hardware implementation of transformers? Kudos to Yann lecun for getting his revenge for Galactica
- jhrmnn 4y agohttps://en.wikipedia.org/wiki/Tensor_Processing_Unit https://en.wikipedia.org/wiki/Tensor_Processing_Unit
- seydor 4y agobut those are not for sale, and not transformer-specific. There must be some optimizations that can be done in hardware and transformers are several years old now
- ruuda 4y agoYou likely already bought one. https://blog.google/products/pixel/introducing-google-tensor/ https://blog.google/products/pixel/introducing-google-tensor... https://apple.fandom.com/wiki/Neural_Engine https://apple.fandom.com/wiki/Neural_Engine
- jhrmnn 4y agoComputation-wise, transformers are really just a bunch of matrix multiplications, nothing more to it. (Which is partially why they’re so efficient and scalable.) Also, Nvidia’s GPU architectures are moving in the TPU direction (https://www.nvidia.com/en-us/data-center/tensor-cores/ https://www.nvidia.com/en-us/data-center/tensor-cores/).
- jacooper 4y agoDoes anybody know how to run this on Linux with an AMD GPU? Also do I have to bother with their crappy driver module that doesn't support most GPUs?
- Const-me 4y agollama.cpp doesn't need nor use GPU. You can run on Linux, but you gonna need a CPU with AVX2 support.
- heap_perms 4y agogrep -o avx2 /proc/cpuinfo | uniq If your CPU supports AVX2, the output will be "avx2". If your CPU does not support AVX2, the command will not produce any output.
- nextaccountic 4y agoBut if you have a GPU available, it can't use it? Why not?
- Const-me 4y agoIt can not, that particular library is CPU-only. If you want to run on GPU, use the official Python-based stack, which BTW takes about 10GB of runtime binaries on disk, and only supports nVidia GPUs because CUDA.
- Mathnerd314 4y ago> This all changed yesterday, thanks to the combination of Facebook’s LLaMA model and llama.cpp by Georgi Gerganov. George Hotz was so confident that he was riding the wave with his Python implementation: https://github.com/geohot/tinygrad/blob/master/examples/llama.py https://github.com/geohot/tinygrad/blob/master/examples/llam.... But I guess not, pure C++ seems better.
- quotemstr 4y agoIsn't it more the four bit quantization than the choice of C++ as an orchestrator that's the win? It's not as if in either the C++ or the Python case that high level code is actually doing the matrix multiplications. That basically the whole AI revolution is powered by CPython of all things (not even PyPy) is the 100 megaton nuke that should end language warring forever. That the first AGI will likely be running under a VM so inefficient that it refcounts even integers is God laughing in the face of all the people who've spent the past decades arguing that this language or that language is "faster". Amdahl was right: only inner loops matter.
- minimaxir 4y ago> That basically the whole AI revolution is powered by CPython of all things (not even PyPy) is the 100 megaton nuke that should end language warring forever. And a lot of new AI tooling such as tokenization has been developed for Python using Rust (pyo3)
- Const-me 4y agoThe original llama uses google's tooling for that, written in C++ https://github.com/google/sentencepiece https://github.com/google/sentencepiece
- seydor 4y agoPretty sure the first AGI will design its own VM
- 4y ago
- BaculumMeumEst 4y agowould i have better luck with a gtx 1070 with 8gb of vram or a macbook m1 pro with 16gb of ram?
- techstrategist 4y agoM1 Pro for sure
- rahimnathwani 4y agoThe latter.
- patricktlo 4y agoThat's amazing, any chance of running it on my trusty GTX 1060 6gb, or that's not enough VRAM?
- yacine_ 4y agoFrom my mental math, you can run 4bit quant 7B, but the quality for that might not be what you're looking for.
- deleted 4y ago[deleted]
- ilovefood 4y agoThis is really great, very good write-up. Seems it now also supports AVX2 for x86 architectures too. https://twitter.com/ggerganov/status/1634588951821393922 https://twitter.com/ggerganov/status/1634588951821393922
- novok 4y agoAnyone able to get llama.cpp into some sort chat-bot style mode via a script or similar? What do you need to wrap around it to get it to act that way? Right now it will just go on forever in response to prompts.
- leobg 4y ago“Chat mode” just means that 1) you structure the prompt given to the model for completion as a chat, i.e. a list of lives in the form “NAME: UTTERANCE” and 2) specify as “end” sequence the “NAME:” part for the user (i.e. prevent the model from going on in both roles). On top of that, you may want to do fine-tuning with reinforcement learning. But the above is “all you need” to get into chat mode with a LLM (or any text generation model, really).
- mk_stjames 4y agoGeorge Hotz did a stream yesterday building with Llama and got some good functionality. It's done with some few-shot pre-prompting to give it understanding of 'who it is' and how to answer questions: https://www.youtube.com/watch?v=nctqc8FBJ2U https://www.youtube.com/watch?v=nctqc8FBJ2U
- SergeAx 4y agoMy personal problem with Stable diffusion moment is that nothing practical came up from that moment. Concerning all image generative models I only see them used in generation of OG-images for mostly uninteresting articles on the internet. I can only see the same practical personal use for language models.
- palata 4y ago> I can only see the same practical personal use I can see fake accounts, phishing and disinformation at a whole new scale, too.
- SergeAx 4y agoYes, I meant about the same: I percept uninteresting garbage content as malicious too.
- simonw 4y agoPrior to Stable Diffusion's release very few people outside of the AI research community were paying much attention to generative AI models. Stable Diffusion kicked off a wave of interest which hasn't stopped building yet - albeit massively accelerated by ChatGPT's release in November.
- SergeAx 4y agoYes, this is true and it is hard to ignore. However, I was talking about practical applications, not just general interest. There were several hype-generating stories about diffusion model winning art contest and using LLM to write grading essays, but that was it.
- CyanBird 4y ago> practical applications What do you even mean! It is being used right now on all sorts of creative industries for things like ui/Ux generation, concept art, vfx production etc I myself use it for concepting and color correction on video games pipeline production
- college_physics 4y agoEven if its not the "precise" moment, in the scheme of things does perfect timing really matter? Fact is, this shows beautifully that significant new algorithmic capability is becoming commoditised - fast. And thats a good thing.
- nicbou 4y agoThat was a really good and accessible writeup. Not just the topic, but the way it was written.
- 627467 4y agoIs there a diffusionbee or mochidiffusion equivalent apps for this yet?
- warning26 4y agoThat’s what I’m waiting for — the real game changer for me was just being able to install a package and run it. Hoping we can see that for LLMs soon.
- smeagull 4y agoSurely the situation isn't that bad for Large Language Models.
- heap_perms 4y agoThat's very exciting. I'm building it right now. Can anyone explain how llama.cpp works? On a very high level, I'm interested in how it's possible to run it on CPU.
- woodson 4y agoIt quantizes 32-bit floating point weights to 4-bit integers so that models fit into memory and run reasonably fast on a CPU by using the processor’s vector extensions.
- nico 4y agoIs there a limit for the number of tokens that can be fed to the models when run locally? OpenAI limits prompts to 4096 tokens. If there was no limit, could the LLM be fed a 100 page document in the prompt and then answer questions about it?
- hnhg 4y agoYes. You can break the document up and index each part and then tackle it that way. It works surprisingly well. The 4096 token limit is tied to the attention window, not an API restriction.
- jerpint 4y agoAttention doesn’t have a window, unless you mean there’s a limit to the number of absolute positional embeddings available at train time?
- freezed88 4y agoLlamaIndex offers ways to chunk up your data and store them in data structures for response synthesis: https://gpt-index.readthedocs.io/en/latest/guides/primer.html https://gpt-index.readthedocs.io/en/latest/guides/primer.htm...
- arjvik 4y agoYes, it's a fundamental limitation of their architecture
- ivalm 4y agoContext limit is a property of the model and set at training time. Computational complexity is quadratic with context length. You can potentially recursively summarize/create index to breakdown larger texts into smaller chunks.
- Robotbeat 4y agoHas anyone had luck with doing this sort of thing when asking ChatGPT/GPT3.5 to generate code? Like, ask it to generate functions at a later time?
- zztop44 4y agoJust a note that the link to your Mastodon is broken (missing “n”) :)
- asrin01 4y ago[flagged]
- braingenious 4y agoThis stuff is amazing… now I’ve just got to figure out what I can run on an RTX 3050 Ti
- Taek 4y agoIts worth buying a 3060
- enduser 4y agoI’m getting better results with 30B unquantized (f16) than with 65B at 4 bits, FWIW. On a Mac Studio with 128GB RAM.
- wjessup 4y agopost example prompts and results please?
- koheripbal 4y agoThat's to be expected. 4 bits is too small. But the 8 bit 65b should outperform significantly. Have you tried it?
- sebzim4500 4y agoIs 4 bits too small? Or is the quantization method just not sophisticated enough. See https://arxiv.org/pdf/2210.17323.pdf https://arxiv.org/pdf/2210.17323.pdf
- enduser 4y agoI’ll give it a try this evening.
- sebzim4500 4y agoFrom what I can tell, the quantization being used there is extremely naive. If they were using techniques like GPTQ, presumably it would work much better. https://arxiv.org/pdf/2210.17323.pdf https://arxiv.org/pdf/2210.17323.pdf
- ly3xqhl8g9 4y agoPrediction: sometime by 2033 most consumer CPUs will have the Apple Mx chip architecture with unified memory and the price for 2/4/16 TB of RAM will be similar with today's price for 2/4/16 TB of SSD. Today's price for 1TB of consumer RAM is around 3,000 USD; since it's paramount to be able to run these models locally, especially as they get better the privacy concerns will become even greater, it's time for hardware to step up to the challenge: x10 price reduction in 10 years, a new curve for Gordon Moore's observation.
- brucethemoose2 4y agoAMD"s 7000 laptops already have an Apple-like NPU, and Intel is following shortly after that. IDK about DRAM density breakthroughs though, in fact I kinda worry scaling will slow down over the next decade.
- ketzo 4y agoQuestion for someone smarter than me: What would it take for existing RAM to get that much better? I have a dim sense of the hardware industry, but I know that disk storage has gotten 10-100x cheaper, CPU/GPU power increases quite steadily, but RAM is still quite expensive. Is it even possible for RAM to get much, much better than it is currently? Are there physical limitations, or is there just not the same amount of market interest in more RAM?
- ly3xqhl8g9 4y agoUntil now there wasn't much incentive for a consumer to have 1TB+ of RAM, for any CPU or GPU task: what were you going to do with that much RAM? render 1,000 more Chrome tabs? Beyond the supply/demand economics and also corporate greed [1], there are fabrication issues, just look at the current mess [2], but as new US-based fabs come online from 2025 and beyond [3], they will be solved to some extent. Theoretically, the maximum amount of RAM a 64-bit machine can address is around 16.8 million terabytes [4]: as always with computing, the journey hasn't even begun. [1] Easier and more money in manufacturing RAM for smartphones, https://fossbytes.com/flagship-phones-ram-vs-pc-ram https://fossbytes.com/flagship-phones-ram-vs-pc-ram [2] "EU tells Taiwan to forget about a bilateral investment pact even as bloc seeks more chips", https://www.scmp.com/news/china/article/3213163/eu-tells-taiwan-forget-about-bilateral-investment-pact-even-bloc-seeks-more-chips https://www.scmp.com/news/china/article/3213163/eu-tells-tai... [3] "U.S. Semiconductor Renaissance: All the Upcoming Fabs" (2022), https://www.tomshardware.com/news/new-us-fabs-everything-we-know https://www.tomshardware.com/news/new-us-fabs-everything-we-... [4] https://superuser.com/questions/65191/what-is-the-maximum-amount-of-ram-a-64bit-machine-can-theoretically-address https://superuser.com/questions/65191/what-is-the-maximum-am...
- bayesian_horse 4y agoMy presentations haven't been the same since Stable Diffusion came along.
- jawadch93 4y ago[dead]
- EGreg 4y agoWhere can we download the model and weightz ? Do we still need to fill out a form and say we are a researcher?
- simonw 4y agoThere's a BitTorrent link in this PR: https://github.com/facebookresearch/llama/pull/73 https://github.com/facebookresearch/llama/pull/73
- deleted 4y ago[deleted]