5 ms·
I would like to use stuff like this as a side-project. Buy a Nvidia Geforce GPU and stick it into my 24/7 server and play around with it in my free time, to see
by qwertox 3y ago
I would like to use stuff like this as a side-project. Buy a Nvidia Geforce GPU and stick it into my 24/7 server and play around with it in my free time, to see what can be done.
The issue with all these AI models is that there's no information on which GPU is enough for which task. I'm absolutely clueless if a single RTX 4000 SFF with its 20GB VRAM and only 70W of max power usage will be a waste of money, or really something great to do experiments on. Like do some ASR with Whisper, images with Stable Diffusion or load a LLM onto it, or this project here from Facebook.
Renting a GPU in the cloud doesn't seem to be a solution for this use case, where you just want to let something run for a couple of days and see if it's useful for something.
- lhl 3y agoThe more VRAM the better if you'd like to run larger LLMs. Old Nvidia P40 (Pascal 24GB) cards are easily available for $200 or less and would be easy/cheap to play. Here's a recent writeup on the LLM performance you can expect for inferencing (training speeds I assume would be similar): https://www.reddit.com/r/LocalLLaMA/comments/13n8bqh/my_results_using_a_tesla_p40/ https://www.reddit.com/r/LocalLLaMA/comments/13n8bqh/my_resu... This repo lists very specific VRAM usage for various LLaMA models (w/ group size, and accounting for context window, which is often missing) - this are all 4-bit GPTQ quantized models: https://github.com/turboderp/exllama https://github.com/turboderp/exllama Note the latest versions of llama.cpp now have decent GPU support and has both a memory tester and lets you load partial models (n-layers) into your GPU. It inferences about 2X slower than exllama from my testing on a RTX 4090, but still about 6X faster than my CPU (Ryzen 5950X). Again this is inferencing. For training, pay attention to 4-bit bitsandbytes, coming soon: https://twitter.com/Tim_Dettmers/status/1657010039679512576 https://twitter.com/Tim_Dettmers/status/1657010039679512576
- KRAKRISMOTT 3y agoUse the A100 as a base reference and work your way down from there.
- nharada 3y agoWait why is renting a GPU in the cloud not a solution? You can even try multiple options and see which ones are capable enough for your use case. Look into some barebones cloud GPU services, for example Lambda Labs which is significantly cheaper than AWS/GCP but offers basically nothing besides the machine with a GPU. You could even try something like Vast in which people rent out their personal GPU machines for cheap. Not something I'd use for uhhh...basically anything corporate, but for a personal project with no data security or uptime issues it would probably work great.
- chaxor 3y agoCloud has a variable price (up to the whim of whatever they decide the price to be that day) so it's uncertain, but typically it is far" more expensive for this type of application. So when faced with 1) probably far more expensive, or 2) single price that will be cheaper, is always available, and has far more uses, I think most would choose 2) for self-hosting. Cloud is very rarely* a good option.
- anonzzzies 3y agoI have not seen many setups that wouldn’t pay itself back (including energy in my case) within a year (sometimes even 6 months) with buying vs renting. For something that pays itself back that fast, and that is without renting it out myself, just training with it, I cannot see how I would want to rent one. Edit; on Lambda labs, the only exception seems to be the H100; it would be 1.5 years or so, but even 2 years would still fast enough. I have an A100 which paid itself back; thinking of getting another one.
- fennecfoxy 3y agoI think the downside to buying hardware as well is that compared to other tech, this LLM/ML stuff is moving very quickly, people are great at quantising now (whereas I only really saw it done before for the coral edge TPUs etc). Someone could buy an H100 to run the biggest and bestest stuff right now, but we could find that a model gets shrunk down to run on a consumer card within a year or two with equivalent performance. I suppose it makes sense if someone wants to be on the bleeding edge all the time.
- deleted 3y ago[deleted]
- ttt3ts 3y agoYou can finetune whisper, stable diffusion, and LLM up to about 15B parameters with 24GB VRAM. Which leads you to what hardware to get. Best bang for the $ right now is definitely a used 3090 at ~$700. If you want more than 24GB vram just rent the hardware as it will be cheaper. If you're not willing to drop $700 don't buy anything just rent. I have had decent luck with vast.ai
- aci_12 3y agoCan we use 3090 or nvidia gpus in general with mac? do people generally have their windows dekstop for the gpus?
- ttt3ts 3y agoNo clue but if you want learn/finetuned ML use a Linux box otherwise you will spend all your time fighting your machine. If you just want to run models Mac might work.
- NBJack 3y agoThere is the world of used Nvidia Teslas, like the M40. Very cheap, but some assembly required.
- ttt3ts 3y agoI own a P100, P4, M40. They either lack ram or speed. Also, unless you are putting them in a server you have to cool them. If your goal is to learn ML don't tinker with very obsolete hardware. Rent or buy something modern.
- flangola7 3y agoWhat is their advantages?
- fswd 3y agoapparently there are a group of folks finetuning with these cards.
- syntaxing 3y agoI would recommend a 3090. It can handle everything a 4000 series can albeit slightly slower, has enough VRAM to handle most things for fun, and can be bought for around $700.
- ra 3y agoI think you just need to educate yourself a bit about the space. These models are very small (the large version is only 1B parameters) so should run on a 4GB gaming GPU.
- slowmovintarget 3y agoHere's a sheet I came across today... https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_install_llama_8bit_and_4bit/ https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_... Granted, it's talking about quantized models, which use less memory. But you can see the 30B models taking 36 GB at 8-bit, and at least 20 GB at 4-bit. The page even lists the recommended cards. But as others have pointed out, you may get more bang "renting" as in purchasing cloud instances able to run these workloads. Buying a system costs about as much as buying instance time for one year. Theoretically, if you only run sporadic workloads when you're playing around it would cost less. If you're training... that's a different story.
- bigdict 3y agoA 3080 Ti is sufficient to run the largest Whisper model (just under 12 GB VRAM).
- segmondy 3y agoJust do it. Spend a few hours doing research and you will find out. With that said, buy as much memory as can. That makes 4090 king if you have the server that can carry it, plus the budget. For me, I settled for 3060, it's a nice compromise between cost and ram. Cheap, 12gb and 170TPW.
- deleted 3y ago[deleted]