2 ms·
re: running on your own hardware.. How? I know very little about ML, but i had assumed the reason models ran on GPUs typically(?) was because of the heavy comp
by unshavedyak 4y ago
re: running on your own hardware.. How?
I know very little about ML, but i had assumed the reason models ran on GPUs typically(?) was because of the heavy compute needed over large sets of in memory data.
Moving it to something cheaper ala general CPU and RAM/Drive would make it prohibitively slow in the standard methodology.
How would we be able to change this to run on users standard hardware? Presuming standard hardware is cheaper, why isn't ChatGPT also running on this cheaper hardware?
Are there significant downsides to using lesser hardware? Or is this some novel approach?
Super curious!
- lairv 4y agoThe goal is not (yet?) to be able to run those models on most of consumers devices (mobile, old laptops etc.), but at least to self-host the model on high-end consumer GPU which is not possible right now. For now you need multiple specialized GPUs like nvidia V100/A100 with a high amount of VRAM, having such models to run on a single rtx40*/rtx30* would already be an achievement