6 ms·
How to make LLMs go fast
- foundval 3y ago[dead]
- orenlindsey 3y agoIf AI has as big of an impact as it seems like it will, we will ultimately have to design consumer-grade chips around neural networks to run it at the edge. (kind of like what https://www.etched.ai https://www.etched.ai is doing) Either that or some massive efficiency breakthrough, probably (hopefully!) both. Otherwise AI will be limited to enormous computing banks at datacenters, and it will be severely limited in terms of privacy (because no one wants to upload their personal data to datacenters, especially not to power some black-box "AI")
- otterley 3y agoIsn’t that what the Neural Engine in Apple Silicon is? I also believe some Pixel phones have TPUs in them as well.
- ColonelPhantom 3y agoIntel Meteor Lake and AMD Phoenix also introduced dedicated AI acceleration hardware.
- kristianp 3y ago> ... we could be running models than handily beat GPT-4 on consumer hardware, The implication there is that consumer hardware will have more and faster RAM than the current defaults. AI features are going to probably require more than the 8gb minimum that macbook airs have. Sure mistral 7b can fit, but I think generally useful models start at 32B, and mixtral 8x7B still is short of GPT-4. Mixtral requires about 28GB 4bit quantised. It will be interesting to see if llama etc drive a step up in consumer hardware defaults, but always connected to cloud and paying a subscription for cloud services is the trend with commercial offerings.
- behnamoh 3y agoI wonder if the transformer will remain the de facto arch for LLMs in a couple years. We already have Mamba, RWKV, etc. which are huge improvements over transformer in terms of memory requirement and speed. I wonder why there's still so much work (and hacks and workarounds) on transformer to make it go faster and consume less VRAM whereas the same effort could be spent on other architectures that solve some of those problems fundamentally.
- uoaei 3y ago> why there's still so much work (and hacks and workarounds) on transformer to make it go faster and consume less VRAM whereas the same effort could be spent on other architectures Probably because the pace of development is now no longer determined by ML researchers but by CS engineers. ML is "easy to pick up" until you have to understand the reasons certain architectures work or don't for different applications.
- cyanydeez 3y agotechnical debt
- __loam 3y agoThe best technology doesn't necessarily win. First to market matters, especially when models on the scale of gpt-4 cost 100mln to train.
- evilantnie 3y ago"best" is subjective, but I do think the industry will eventually converge onto an architecture that is significantly more cost effective than current state of the art. Regardless of who is first to market, everyone is incentivized to continue down this path with their research on improving LLM performance.
- realusername 3y agoWhile the first mover advantage exists, I don't think it's a sufficient moat in the long term. OpenAI's moat is the high cost of training as you mentioned but this might be obsolete in a few papers down the road. They themselves realized this by trying to turn it into a platform but I don't think it's enough.
- tome 3y agoWe use some of these techniques at Groq to make the world's fastest LLM appliance! You can see for yourself how fast it is at https://groq.com/ https://groq.com/ (click "Launch demo" in the top right). The model is Llama 2 70B. Our hardware is an "LPU" (language processing unit). The silicon has a very regular/uniform/homogeneous design with compute located close to memory, which allows us to get a much lower latency for language models than a GPU can. A lot of the smarts is in the compiler toolchain, which can fit high-level model descriptions (say Torch, ONNX) to the hardware without the need for handwritten kernels. I work on the assembler part of the toolchain, which is written in Haskell. EDIT: To the downvoters, I guarantee that once you've tried the demo you'll be so surprised at how fast it is that you'll change your vote.
- llamaInSouth 3y agoFast is great but uncensored is better
- Der_Einzige 3y agoThey had a whole section on guided decoding which can be used to among other thing break censorship/alignment efforts.
- madeofpalk 3y agoJust run/train it yourself.
- llamaInSouth 3y agoThis one is supposed to be uncensored: https://huggingface.co/cognitivecomputations/dolphin-2.6-mixtral-8x7b https://huggingface.co/cognitivecomputations/dolphin-2.6-mix...
- prestonlibby 3y agoFor anyone else who wishes to do so, Fireship has a low-barrier video[1] on training the uncensored model in the sibling comment (dolphin mistral) for local use. [1] https://www.youtube.com/watch?v=GyllRd2E6fg https://www.youtube.com/watch?v=GyllRd2E6fg
- kristiandupont 3y agoWhat do you want to use it for? Out of all the things I use LLM's for, there isn't one situation where I care about political correctness or whatever.
- throwup238 3y agoCourt documents. One mention of rape or murder and there's a 50:50 chance GPT/Claude/Bard will just shut down, when all I want is to extract the entities mentioned in transcripts or opinions.
- 3y ago
- fuddle 3y agoThanks for the blog post, I also enjoyed their previous post on making a transformer by hand: https://vgel.me/posts/handmade-transformer/ https://vgel.me/posts/handmade-transformer/
- altruios 3y agoLLM's generate 'the next token' require a serialization of data, and prevents 'too much' parallelization. I wonder if some sort of diffusion hybrid is being worked on. Something that approximates a complete text answer, but 'increases the resolution' of that text over time. One key benefit to a 'shot-gun/top-down/all-at-once' approach as opposed to a 'guess the next token => add and repeat' is the ability for the tokens at the end of the text (a twist in a surprise mystery story) have a direct effect on the beginning of that story. which (as far as I know) is not possible in current LLM's due to their architecture. how self attention and positional encoding would work in a diffusion model... that's the question I have if that could even work... to be clear, I don't mean stable diffusion rendering text as an image. I mean diffusion run on raw (random) text: turning that into a legible response.
- thatguysaguy 3y agoPeople have been trying to get non-autoregressive generation working for years. There are some methods which work okay but they're always behind AR generation. And yeah they do look very diffusion-esque. One of the issues is that you'd need to be really sold in this being a better paradigm before spending the money to pretrain a huge non-AR model.
- altruios 3y agoI figured the raw approach of diffusion-esque was tried (without techniques lifted from newer LLMs {chatGPT is only a year old}). I am thinking that those attempts were made largely before LLM's were popular. Self attention and positional encoding are the claimed 'secret sauce': are there any models (or attempts) anyone can point to that try to combine SA+PE into a diffusion-like (non-autoregressive) model. What are some models that 'work okay'?
- thatguysaguy 3y agoSelf-attention and positional embeddings are ancient at this point (i.e., more than 5 years old). Here are a couple examples of methods: https://arxiv.org/abs/2112.06749 https://arxiv.org/abs/2112.06749 https://arxiv.org/abs/2205.12558 https://arxiv.org/abs/2205.12558