5 ms·
That's on the order of 25 4090 GPUs to run inference. Not a crazy number by any means. We will see consumer robots running that by the end of the decade, mark m
by LarsDu88 3y ago
That's on the order of 25 4090 GPUs to run inference. Not a crazy number by any means. We will see consumer robots running that by the end of the decade, mark my words.
- swyx 3y agoi believe that's George's goal with tinybox
- jerrygenser 3y agoIt's a lot of electricity to power it and heat that gets generated... and electricity to cool it from the heat
- hardware2win 3y ago25 GPUs? A lot of electricity?
- londons_explore 3y agoLanguage models have tiny inputs (words), and tiny outputs (words), and no tight latency requirements (inference takes seconds). That makes them perfect for running where compute is cheap (ie. next to a hydrothermal plant in iceland) and querying them over the internet.
- ChatGTP 3y agoHow are. 254090 GPUs supposed to run inside a robot ?
- deleted 3y ago[deleted]
- fnands 3y agoThey mean 25 x RTX 4090 GPUs. 4090 is a model number
- jasonjmcghee 3y agoAnd to give a bit more context, it's one of the top consumer grade cards available (and has 24GB of RAM). It costs on the order of $1.6k instead of $15-25k of H100.
- jiggawatts 3y agoThat just goes to show how huge the markup is on those H100 cards! Ten 4090 cards have more compute and more memory (240GB!) than a H100 card. The cost of the 4090s is the same or lower for about 4x the compute.
- mrtranscendence 3y agoI think the question stands. You can't fit 25 4090s in a robot (unless we're talking about something massive with an equally massive battery), and even if you could an LLM wouldn't be appropriate for driving a robot. Given the pace of improvements I don't see how you compress 25 4090s into a single GPU in 7 years. A 4090 isn't 25 times the power of a GTX 980, it's closer to maybe three times.
- machiaweliczny 3y agoLight or quantum could possibly deliver by that time
- LarsDu88 3y agoNeither of these are likely nor necessary to deliver the necessary results.
- LarsDu88 3y agoTeslas are already consumer items that rock massive batteries. My 25 count is off. It's probably closer to 75 GPUs right now. Let's say today's models are running vanilla transformers via pytorch without any of Deepmind's Flamingo QKV optimizations. In 10 years algorithmic optimizations, and ML platform improvements push that efficiency up 3-5 fold. We're down in the ballpark of 25 GPUs (again) Now, we ditch the general purpose GPUs entirely and go for specially built AI inference chip. The year is 2033 and specialty inference chips are better and more widespread. Jettison those ray tracing cores, and computer rendering stuff. Another 3x improvement and we're at ~8 GPUs Now you said the 4090 is about 3.5x faster than the 780 from a decade ago. We are now on the order of 2 chips to run inference. These models won't just be running in Teslas, they will be running in agricultural vehicles, military vehicles, and eventually robot baristas.
- sva_ 3y agoIt is my understanding that Nvidia has castrated these cards by removing NVLink?
- swyx 3y agocan you explain the math of how you got to 25 gpus? everyone seems to know these conversions and idk if i missed the memo or something
- coolspot 3y ago220 Bp * 8 = 1760 Bp 1760 Bp takes ~ 1760 Gb of (V)RAM when 8-bit quantized. Plus you will need some memory for state. So you will need at least 1760/24=74 consumer-grade GPUs (3090/4090) or 1760/80=22 professional-grade GPUs (A100/H100).
- swyx 3y agothank you for this! 1Bp ~ 1Gb VRAM 8bit is a very nice equivalence to hold in my head.
- LarsDu88 3y agoI was incorrectly calculating based on 1 weight == 1 transistor which is totally wrong. This figure that you provided is more accurate. We can see that today, the MI300X can already run inference for some open source LLMs: https://www.youtube.com/watch?v=rYVPDQfRcL0 https://www.youtube.com/watch?v=rYVPDQfRcL0 There are almost certainly algorithmic optimizations still available. LLM-scale computing inside of consumer robots should be achievable by the end of the decade. In fact electric cars are probably the best "housing" for this sort of hardware.