3 ms·
I wonder if an outdated chip architecture with just a lot of (64GB?) GDDR4 or something would work? Recycle all the previous generation cards to super high VRAM
by coffeebeqn 4y ago
I wonder if an outdated chip architecture with just a lot of (64GB?) GDDR4 or something would work? Recycle all the previous generation cards to super high VRAM units
- lhl 4y agoI've seen reports of people being able to run LLMs at decent speeds on old Nvidia P40s. These are 24GB Pascal GPUs and can be bought for as low as $100 (although more commonly $200) on eBay.
- outofpaper 4y agoLink to report please
- lhl 4y agohttps://www.reddit.com/r/Oobabooga/comments/126dejd/comment/jebvlt0/?utm_source=reddit&utm_medium=web2x&context=3 https://www.reddit.com/r/Oobabooga/comments/126dejd/comment/...
- kiratp 4y agoYou can do it on CPU now. Benchmarks: https://github.com/ggerganov/llama.cpp/issues/34 https://github.com/ggerganov/llama.cpp/issues/34
- atq2119 4y agoIt should work. After the initial prompt processing, token generation is typically limited by memory bandwidth more than by compute.
- xattt 4y agoIntel Larrabee has come into the chat but showed up about decade and a half too early.