4 ms·
With code modifications, it should be possible to run this with a very modest machine as long as you're happy for performance to suck. Transformer models typi
by londons_explore 4y ago
With code modifications, it should be possible to run this with a very modest machine as long as you're happy for performance to suck. Transformer models typically need to read all the weights per 'word' output, so if your model is 20GB and you have not enough ram or vram, but have an SSD that reads 1GB/sec, expect 3 words per minute output speed.
However, code changes are necessary to achieve that, although they won't be crazy complex.
- moffkalast 4y agoThe most time/cost optimal solution is probably to buy 32 or 64 gigs of ram. That'll still be slow but most people are already half way there.
- esperent 4y agoDoesn't it need to be GPU ram?
- nl 4y agoThey are saying you can run it on a CPU by doing this: > However, code changes are necessary to achieve that, although they won't be crazy complex. This is technically true. It will be very slow though. However, give it 6 months and I think we might see an order of magnitude increase in speed on CPUs. This will still be too slow to be very useful though.
- koheripbal 4y agoThat will be very VERY slow. Pcie bandwidth is way too slow.
- moffkalast 4y agoShould be like an order of magnitude faster than trying to run it from a NVMe still, no? I've ran some small flan models from RAM and it was fine, but yeah it's not exactly realtime.
- londons_explore 4y agoThere is a neat potential speedup here for the case where the bandwidth to your model weights is the limiting factor. If you have a guess what the model will output, then you can verify that your guess is correct very cheaply, since you can do it in parallel. That means there is the possibility to have a highly quantized small model in RAM, and then use the big model only from time to time. You might be able to get a 10x speedup this way if your small model agrees 90% of the time.
- humanizersequel 4y agoThis is an interesting concept, could you share a paper or some writeup about this?
- ebalit 4y agoIt looks like a description of Speculative Sampling. There's a recent paper from DeepMind about this in the context of LLM [0], although it's not a completely new idea of course. The potential for speedup according to their paper is closer to 2x than 10x however. 0: https://arxiv.org/abs/2302.01318 https://arxiv.org/abs/2302.01318
- bick_nyers 4y agoWhy 3 words per minute as opposed to second? Is that a typo? If you have enough RAM (but not VRAM), does it basically become limited by the PCIE lanes? So for the 112GB model with a Gen 5 GPU (64 GB/s PCIE bandwidth) that would be roughly 2 seconds per word right?
- flangola7 4y agoIf read is 1GB/s then it takes 20s to infer across a 20GB model. That's 3 tokens a minute.
- bick_nyers 4y agoYeah I'm not sure what kind of math I was subscribed to yesterday, thanks
- flangola7 4y ago:-) I keep saying that we don't have to stop AI from hallucinating, we only need to bring the rate to below human level.
- gpm 4y agoI just tried this on the 7B model. Steady state single threaded CPU performance of 23 seconds/token on a Ryzen 5800x (I'm not sure why it's only using a single thread... usually these libraries automatically use more) and 14GB of ram. It used more than double that amount of ram while loading the model, and the first token took 183 seconds (potentially it's doing more work to parse the prompt that I'm not measuring properly).