3 ms·
I didn't spend that much, only $6500 AUD for a GB10 based Asus GX10 which is even slower than OPs, but I spent that because it makes for a great learning platfo
by girvo 5mo ago
I didn't spend that much, only $6500 AUD for a GB10 based Asus GX10 which is even slower than OPs, but I spent that because it makes for a great learning platform. Theres not much else that lets me fiddle with 128GB of RAM for my graphics processor, and it's quite lovely to be able to run things as long as I like without worrying about my cloud instance being shut down.
It's not financially a good idea: renting really does beat owning, and cloud beats both if you're only running inference on these machines. But I'm not just doing inference, and as a thing I can do silly stuff on to learn, it's hard to beat!
- a1o 5mo agoWhen you say you are not just doing inference, you mean you are also training your own llms? I am curious what other things can be done.
- girvo 5mo agoFine tuning, and yeah training my own, experimenting with architectures and learning how it all works. Been a lot of fun
- nostrebored 5mo ago$6500 AUD can get you a good chunk of B200 time on any of the GPU neoclouds :)
- girvo 5mo agoLess than I expected, though! And I get to run this all through the night I do still use Vast and Runpod for things too, but it’s much nicer to test a fine tuning run here to make sure I’m in the ballpark I also did literally say “It's not financially a good idea, renting is better than owning” so I’m confused why I have two people telling me that Also it’s just far more fun to play with something tangible to me :)
- andoando 5mo agoYou could just rent a bare metal server with those specs
- nullsanity 5mo ago[dead]
- girvo 5mo agoYes I could, but that is annoying because of spot pricing and having my instance shut down, and it has fluctuating prices It’s also annoying because then I need to make sure my little “lab” setup is well automated, and I’m lazy :) Also, I literally said “ It's not financially a good idea” so I’m confused why you think I don’t know that.
- adastra22 5mo agoSpot pricing and instance availability don’t apply to on metal hosting. You’d have your own machine dedicated to your own use only, at a locked in price.
- girvo 5mo ago> renting really does beat owning, and cloud beats both
- girvo 5mo agoWait, I went and looked far and wide today and I can’t find anything with ~100GB of VRAM that isn’t $20k a year, what am I missing
- adastra22 5mo agoThat's too small. I was quoted a machine with ~1.5TB of vram, for $10k/mo. This was the minimum node size in the AI data center I was talking with -- they don't make smaller nodes that you can lease as bare metal. There is no public pricing though, and you had to know people to get in.