5 ms·
> widely-available H100 GPUs Just looked in the parts drawer at home and dont seem to have a $25,000 GPU for some inexplicable reason.
by blitzar 1y ago
> widely-available H100 GPUs
Just looked in the parts drawer at home and dont seem to have a $25,000 GPU for some inexplicable reason.
- KolmogorovComp 1y agoavailable != cheap
- blitzar 1y agoavailable /əˈveɪləbl/ adjective: available able to be used or obtained; at someone's disposal
- swexbe 1y agoYou can rent one from most cloud providers for a few bucks an hour.
- koakuma-chan 1y agoMight as well just use openai api
- deleted 1y ago[deleted]
- Kurtz79 1y agoDoes it even make sense calling them 'GPUs' (I just checked NVIDIA product page for the H100 and it is indeed so)? There should be a quicker way to differentiate between 'consumer-grade hardware that is mainly meant to be used for gaming and can also run LLMs inference in a limited way' and 'business-grade hardware whose main purpose is AI training or running inference for LLMs".
- amelius 1y agoWell, does it come with graphics connectors?
- OliverGuy 1y agoNope, doesn't have any of the required hardware to even process graphics iirc
- diggan 1y agoAlthough the RTX Pro 6000 is not consumer-grade, it does come with graphics ports (four Displayports) and does render graphics like a consumer card :) So seems the difference between the segments is becoming smaller, not bigger.
- simpleintheory 1y agoThat’s because it’s intended as a workstation GPU not one used in servers
- diggan 1y agoSure, but it still sits in the 'business-grade hardware whose main purpose is AI training or running inference for LLMs" segment parent mentioned, yet have graphics connectors so the only thing I'm saying is that just looking at that won't help you understand what segment the GPU goes into.
- lopuhin 1y agoyou can rent them for less then $2/h in a lot of places (maybe not in the drawer)
- dougSF70 1y agoWith Ollama i got the 20B model running on 8 TitanX cards (2015). Ollama distributed the model so that the 15GB of vram required was split evenly accross the 8 cards. The tok/s were faster than reading speed.
- Aurornis 1y agoFor the price of 8 decade old Titan X cards, someone could pick up a single modern GPU with 16GB or more of RAM.
- philipkiely 1y agoThis comment made my day ty! Yeah definitely speaking from a datacenter perspective -- fastest piece of hardware I have in the parts drawer is probably my old iPhone 8.
- Aurornis 1y agoThey’re widely available to rent. Unless you’re running it 24/7 for multiple years, it’s not going to be cost effective to buy the GPU instead of renting a hosted one. For personal use you wouldn’t get a recent generation data center card anyway. You’d get something like a Mac Studio or Strix Halo and deal with the slower speed.
- varispeed 1y agoI rented H100 for training a couple of times and I found that they couldn't do training at all. Same code worked fine on Mac M1 or RTX 5080, but on H100 I was getting completely different results. So I wonder what I could be doing wrong. In the end I just use RTX 5080 as my models fit neatly in the available RAM. * by not working at all, I mean the scripts worked, but results were wrong. As if H100 couldn't do maths properly.
- blueboo 1y agoYou might find $2.50 in change to use one for an hour though
- vonneumannstan 1y ago>Just looked in the parts drawer at home and dont seem to have a $25,000 GPU for some inexplicable reason. It just means you CAN buy one if you want, as in they're in stock and "available", not that you can necessarily afford one.