23 ms·

Expect to pay $4k-10k - Your RTX 6000 is closer to $10k now - Sparks are creeping into the $4-5k range - AMD Strix are ~3.5k - Apple depends on chipset and

Expect to pay $4k-10k - Your RTX 6000 is closer to $10k now - Sparks are creeping into the $4-5k range - AMD Strix are ~3.5k - Apple depends on chipset and memory. Sweet spot would be 128gb M3 Ultra, probably $6-8k but admittedly haven't been tracking closely. New M5 might come in the fall. You can get a new 128gb M5 Max laptop for ~5-6k today. - a 4x3090 rig would take $5-6k Every platform has tradeoffs, but it's mostly ecosystem, memory bandwidth, and power consumption. They're all slow. The best option is likely to rent hardware on Runpod. The RIO on self-hosting is very low unless you have a specific need or you're ok treating it as a hobby.

anonym29 5mo ago
Bosgame M5 (Strix Halo) w/ 128 GB still goes for $2800 right now. SH systems have surged in price dramatically but quite unevenly. >The best option is likely to rent hardware on Runpod. Vast.ai is much cheaper, but the broader point here is contestable. The only dimension in which cloud GPU rentals win is cost. You lose the confidentiality, integrity, and availability benefits of local deployments.
- ai_fry_ur_brain 5mo ago
  Rentals are priced to pay themselves off in 1-1.5 years (when renting them out per hour, not selling tokens). Its never a better option to rent. Not that I'd encourage anyone to throw large amounts of money to have access to LLMs, but you're definately going to be better off buying something that you can amortize over multiple years with a multi year warranty.
ai_fry_ur_brain 5mo ago
And for what? Spend 10-15k for the slopiest of slop code, non deterministic automations, and the ability to spawn an AI gf? This whole thing is really starting to remind me of the crypto hype phases of 2016-2018 when everyone thought their investment in GPUs was going to make them rich.
- organsnyder 5mo ago
  It is possible to get real work done with LLMs. There are plenty of ethical concerns, and they're definitely over-hyped, but they are exceptionally useful tools when used well.
  - varispeed 5mo ago
    [dead]
- deleted 5mo ago
  [deleted]
- gamander2 5mo ago
  [dead]
  - gcr 5mo ago
    which models do you have in mind? grok from xai?
- dvfjsdhgfv 5mo ago
  I upvoted your comment even though I disagree with you. Yes, LLMs are sloppy, and local models usually more so (but things change fast). But the local ones have one big advantage: they are private. So you can safely feed them the collection of your private documents and things you wouldn't trust people like sama with. The fact that some people do not care is one of the failures of our educational system.
bahmboo 5mo ago
$2600 gets MBP M5 Pro 48gb. 64gb requires a Max which bumps it to $4200 at which point you may as well spend the $800 to go to 128gb.

anonym29 5mo ago

Bosgame M5 (Strix Halo) w/ 128 GB still goes for $2800 right now. SH systems have surged in price dramatically but quite unevenly. >The best option is likely to rent hardware on Runpod. Vast.ai is much cheaper, but the broader point here is contestable. The only dimension in which cloud GPU rentals win is cost. You lose the confidentiality, integrity, and availability benefits of local deployments.

ai_fry_ur_brain 5mo ago
Rentals are priced to pay themselves off in 1-1.5 years (when renting them out per hour, not selling tokens). Its never a better option to rent. Not that I'd encourage anyone to throw large amounts of money to have access to LLMs, but you're definately going to be better off buying something that you can amortize over multiple years with a multi year warranty.

ai_fry_ur_brain 5mo ago

And for what? Spend 10-15k for the slopiest of slop code, non deterministic automations, and the ability to spawn an AI gf? This whole thing is really starting to remind me of the crypto hype phases of 2016-2018 when everyone thought their investment in GPUs was going to make them rich.

organsnyder 5mo ago
It is possible to get real work done with LLMs. There are plenty of ethical concerns, and they're definitely over-hyped, but they are exceptionally useful tools when used well.
- varispeed 5mo ago
  [dead]
deleted 5mo ago
[deleted]
gamander2 5mo ago
[dead]
- gcr 5mo ago
  which models do you have in mind? grok from xai?
dvfjsdhgfv 5mo ago
I upvoted your comment even though I disagree with you. Yes, LLMs are sloppy, and local models usually more so (but things change fast). But the local ones have one big advantage: they are private. So you can safely feed them the collection of your private documents and things you wouldn't trust people like sama with. The fact that some people do not care is one of the failures of our educational system.

bahmboo 5mo ago

$2600 gets MBP M5 Pro 48gb. 64gb requires a Max which bumps it to $4200 at which point you may as well spend the $800 to go to 128gb.