3 ms·
Could someone please share a good resource for building a machine from scratch, for doing simple-ish training and running open-source models like Llama? I'd lov
by yawnxyz 3y ago
Could someone please share a good resource for building a machine from scratch, for doing simple-ish training and running open-source models like Llama? I'd love to run some of these and even train them from scratch, and I'd love to use that as an excuse to drop $5k on a new machine...
Would love to run a bunch of models on the machine without dripping $$ to OpenAI, Modal or other providers...
- vonseel 3y agoI am no where near an expert on this subject, and this information is from a few months ago so maybe it's outdated, but people on Reddit[1] are claiming running the llama with 65B parameters would need like 20K+ of GPUs. A 40GB A100 looks like it's almost $8K on Amazon, and I'm sure you could do a lot with just one of those, but that's already beyond your $5K budget. [1] https://www.reddit.com/r/MachineLearning/comments/11i4olx/d_is_it_possible_to_run_metas_llama_65b_model_on/ https://www.reddit.com/r/MachineLearning/comments/11i4olx/d_... I'll let others chime in but you could still probably build something really powerful within your budget that is able to run various AI tasks.
- logicchains 3y agoYou can get around 4-5 tokens per second on the 65B LLaMA with a 32 core 256GB ram Ryzen CPU, not sure how much it costs to build but can rent one from Hetzner for around two hundred bucks a month.
- Joeri 3y agoThere are some threads with hardware recommendations in the LocalLLaMa subreddit. Here’s a recent one: https://www.reddit.com/r/LocalLLaMA/comments/13f5gwn/home_llm_hardware_suggestions/ https://www.reddit.com/r/LocalLLaMA/comments/13f5gwn/home_ll...